2024-04-15T16:58:11.250 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T16:58:11.250 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} --archive /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020 --verbose --name teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/orig.config.yaml 2024-04-15T16:58:11.299 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T16:58:11.390 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020 branch: main description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} email: ceph-qa@ceph.com first_in_suite: false job_id: '7656020' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: main ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: main sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 owner: scheduled_teuthology@teuthology priority: 951 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 5211 sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 sleep_before_teardown: 0 subset: 28113/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_87dbfb632e974cb2c28d17423071d2d833b53b29/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 87dbfb632e974cb2c28d17423071d2d833b53b29 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpbeu/K9QPf6yYesDG5OyusKkoD0uIcLnn0mMGgpxjgXTzDbqOXkvYZ2OK30e2wC7nGabw2Z0inR0PMJyakmpY= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3V9nOZW6PFZFyIWiZcZE5DSGAtsz/oDq1c7zTcPDTyy2Ph2Iw6Sh1bZRoKE5isR3e7bmPV+yg2fGDlQnhcZLY= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3b44o1gqnrtqTFmqWhWiYplcsw6X/ndzmYAtJE/3ND1uuSSL8NDdR+sP7++axrtcqJj5ohdmpT3KNqD29+sYw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: octopus exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-14_20:00:15 tube: smithi user: teuthology verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T16:58:11.390 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_87dbfb632e974cb2c28d17423071d2d833b53b29/qa; will attempt to use it 2024-04-15T16:58:11.391 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_87dbfb632e974cb2c28d17423071d2d833b53b29/qa/tasks 2024-04-15T16:58:11.391 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T16:58:11.395 INFO:teuthology.task.internal:Saving configuration 2024-04-15T16:58:11.414 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T16:58:11.462 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T16:58:11.490 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.009319', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpbeu/K9QPf6yYesDG5OyusKkoD0uIcLnn0mMGgpxjgXTzDbqOXkvYZ2OK30e2wC7nGabw2Z0inR0PMJyakmpY='} 2024-04-15T16:58:11.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.008178', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3V9nOZW6PFZFyIWiZcZE5DSGAtsz/oDq1c7zTcPDTyy2Ph2Iw6Sh1bZRoKE5isR3e7bmPV+yg2fGDlQnhcZLY='} 2024-04-15T16:58:11.530 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.006834', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3b44o1gqnrtqTFmqWhWiYplcsw6X/ndzmYAtJE/3ND1uuSSL8NDdR+sP7++axrtcqJj5ohdmpT3KNqD29+sYw='} 2024-04-15T16:58:11.530 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T16:58:11.539 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-15T16:58:11.539 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-15T16:58:11.540 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-04-15T16:58:11.540 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T16:58:11.641 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0307936e50>, signals=[15]) 2024-04-15T16:58:11.642 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T16:58:11.650 INFO:teuthology.task.internal:Opening connections... 2024-04-15T16:58:11.650 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-04-15T16:58:11.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T16:58:11.742 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-04-15T16:58:11.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T16:58:11.815 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-04-15T16:58:11.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T16:58:11.887 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T16:58:11.894 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-04-15T16:58:11.918 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-04-15T16:58:11.918 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-04-15T16:58:11.981 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-04-15T16:58:11.981 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-15T16:58:11.982 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T16:58:11.983 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-04-15T16:58:12.002 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-04-15T16:58:12.025 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-04-15T16:58:12.025 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-04-15T16:58:12.085 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-04-15T16:58:12.085 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-15T16:58:12.086 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T16:58:12.087 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-04-15T16:58:12.110 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-04-15T16:58:12.132 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-04-15T16:58:12.133 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-04-15T16:58:12.193 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-15T16:58:12.194 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T16:58:12.195 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-04-15T16:58:12.218 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T16:58:12.228 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T16:58:12.235 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T16:58:12.235 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-04-15T16:58:12.237 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-04-15T16:58:12.240 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-04-15T16:58:12.259 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T16:58:12.265 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T16:58:12.265 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-04-15T16:58:12.299 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-04-15T16:58:12.302 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-04-15T16:58:12.338 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T16:58:12.421 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T16:58:12.435 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-15T16:58:12.435 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-15T16:58:12.436 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T16:58:12.436 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T16:58:12.437 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T16:58:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T16:58:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T16:58:12.456 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-04-15T16:58:12.457 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-04-15T16:58:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T16:58:12.458 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-04-15T16:58:12.518 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-489.el8.x86_64 2024-04-15T16:58:12.518 INFO:teuthology.task.kernel:Running kernel on smithi159: 4.18.0-489.el8.x86_64 2024-04-15T16:58:12.518 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-04-15T16:58:12.519 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-489.el8.x86_64 2024-04-15T16:58:12.519 INFO:teuthology.task.kernel:Running kernel on smithi082: 4.18.0-489.el8.x86_64 2024-04-15T16:58:12.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-04-15T16:58:12.520 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-489.el8.x86_64 2024-04-15T16:58:12.521 INFO:teuthology.task.kernel:Running kernel on smithi049: 4.18.0-489.el8.x86_64 2024-04-15T16:58:12.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-04-15T16:58:17.455 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - AppStream 12 MB/s | 28 MB 00:02 2024-04-15T16:58:19.040 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 7.2 MB/s | 28 MB 00:03 2024-04-15T16:58:20.247 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 5.7 MB/s | 28 MB 00:04 2024-04-15T16:58:24.613 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - BaseOS 4.8 MB/s | 10 MB 00:02 2024-04-15T16:58:26.439 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 4.3 MB/s | 10 MB 00:02 2024-04-15T16:58:26.814 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-04-15T16:58:27.531 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 7.7 kB 00:00 2024-04-15T16:58:27.657 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 4.8 MB/s | 10 MB 00:02 2024-04-15T16:58:28.024 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 8.6 kB/s | 3.5 kB 00:00 2024-04-15T16:58:28.639 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2024-04-15T16:58:29.182 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.7 kB 00:00 2024-04-15T16:58:29.262 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-15T16:58:29.416 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-04-15T16:58:29.790 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2024-04-15T16:58:30.175 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2024-04-15T16:58:30.391 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 7.7 kB 00:00 2024-04-15T16:58:30.660 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-15T16:58:31.937 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-15T16:58:33.050 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 493 kB/s | 24 kB 00:00 2024-04-15T16:58:33.984 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 345 kB/s | 24 kB 00:00 2024-04-15T16:58:35.918 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 576 kB/s | 24 kB 00:00 2024-04-15T16:58:36.089 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-15T16:58:36.089 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-15T16:58:36.192 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T16:58:36.193 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T16:58:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T16:58:36.193 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T16:58:36.193 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T16:58:36.193 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout:Install 3 Packages 2024-04-15T16:58:36.194 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:58:36.195 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 90 M 2024-04-15T16:58:36.195 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 96 M 2024-04-15T16:58:36.195 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T16:58:37.030 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-15T16:58:37.030 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-15T16:58:37.156 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T16:58:37.156 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout:Install 3 Packages 2024-04-15T16:58:37.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:58:37.158 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 90 M 2024-04-15T16:58:37.158 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 96 M 2024-04-15T16:58:37.158 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T16:58:38.643 INFO:teuthology.orchestra.run.smithi159.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 4.3 MB/s | 10 MB 00:02 2024-04-15T16:58:39.148 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-15T16:58:39.149 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-15T16:58:39.263 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T16:58:39.264 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T16:58:39.264 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T16:58:39.264 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T16:58:39.264 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T16:58:39.264 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout:Install 3 Packages 2024-04-15T16:58:39.265 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:58:39.266 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 90 M 2024-04-15T16:58:39.266 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 96 M 2024-04-15T16:58:39.266 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T16:58:41.386 INFO:teuthology.orchestra.run.smithi159.stdout:(2/3): kernel-core-4.18.0-552.el8.x86_64.rpm 8.5 MB/s | 44 MB 00:05 2024-04-15T16:58:43.453 INFO:teuthology.orchestra.run.smithi049.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 2.8 MB/s | 10 MB 00:03 2024-04-15T16:58:50.469 INFO:teuthology.orchestra.run.smithi082.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 826 kB/s | 10 MB 00:12 2024-04-15T16:58:54.760 INFO:teuthology.orchestra.run.smithi049.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 2.4 MB/s | 36 MB 00:15 2024-04-15T16:58:56.726 INFO:teuthology.orchestra.run.smithi049.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 2.6 MB/s | 44 MB 00:16 2024-04-15T16:58:56.726 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T16:58:56.727 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.2 MB/s | 90 MB 00:17 2024-04-15T16:58:57.436 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T16:58:57.906 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T16:58:57.906 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T16:58:59.232 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T16:58:59.264 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T16:59:02.417 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T16:59:02.946 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:06.662 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:07.046 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:10.515 INFO:teuthology.orchestra.run.smithi159.stdout:(3/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 1.1 MB/s | 36 MB 00:34 2024-04-15T16:59:10.515 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T16:59:10.515 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.6 MB/s | 90 MB 00:34 2024-04-15T16:59:10.727 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:11.023 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:11.198 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T16:59:11.639 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T16:59:11.639 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T16:59:12.884 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T16:59:12.920 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T16:59:15.748 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T16:59:16.299 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:18.552 INFO:teuthology.orchestra.run.smithi082.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 911 kB/s | 36 MB 00:40 2024-04-15T16:59:18.927 INFO:teuthology.orchestra.run.smithi082.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 1.1 MB/s | 44 MB 00:41 2024-04-15T16:59:18.928 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T16:59:18.928 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.2 MB/s | 90 MB 00:41 2024-04-15T16:59:19.612 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T16:59:19.888 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:20.044 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T16:59:20.044 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T16:59:20.306 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:21.293 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T16:59:21.327 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T16:59:23.743 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:24.054 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:24.114 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T16:59:24.640 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:28.573 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:32.288 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:34.368 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:34.394 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:34.758 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:35.281 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:59:35.282 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T16:59:35.481 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:35.710 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:35.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:35.711 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-15T16:59:35.711 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T16:59:35.711 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T16:59:35.711 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T16:59:35.711 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T16:59:35.711 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:35.932 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:35.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:35.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.02 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 129.el8 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Group : System Environment/Base 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 9265586 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-15T16:59:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Relocations : (not relocatable) 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : CentOS BuildSystem 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T16:59:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T16:59:35.969 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-04-15T16:59:35.969 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T16:59:35.969 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T16:59:35.970 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T16:59:35.970 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 4.18.0-552.el8.x86_64 2024-04-15T16:59:35.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T16:59:37.554 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-04-15T16:59:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-04-15T16:59:39.897 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-15T16:59:39.932 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.lfu0gneVhF 2024-04-15T16:59:39.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lfu0gneVhF 2024-04-15T16:59:40.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.lfu0gneVhF 2024-04-15T16:59:40.147 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.lfu0gneVhF is 6KB 2024-04-15T16:59:40.195 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.lfu0gneVhF 2024-04-15T16:59:40.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T16:59:40.295 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-15T16:59:40.295 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-15T16:59:40.295 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-15T16:59:40.295 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-15T16:59:40.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-15T16:59:40.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-04-15T16:59:46.097 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:46.121 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:46.595 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:46.595 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:46.595 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:47.144 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:59:47.145 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T16:59:47.302 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:47.519 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:47.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:47.519 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-15T16:59:47.519 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T16:59:47.519 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T16:59:47.519 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-15T16:59:47.520 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T16:59:47.520 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:47.733 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:47.733 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:47.733 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-04-15T16:59:47.766 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-04-15T16:59:47.766 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.02 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 129.el8 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Group : System Environment/Base 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 9265586 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Relocations : (not relocatable) 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : CentOS BuildSystem 2024-04-15T16:59:47.767 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T16:59:47.768 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T16:59:47.769 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T16:59:47.769 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 4.18.0-552.el8.x86_64 2024-04-15T16:59:47.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T16:59:49.158 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-04-15T16:59:51.553 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-04-15T16:59:51.555 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-04-15T16:59:51.589 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.Jzqc0eetgt 2024-04-15T16:59:51.589 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Jzqc0eetgt 2024-04-15T16:59:51.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.Jzqc0eetgt 2024-04-15T16:59:51.823 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.Jzqc0eetgt is 6KB 2024-04-15T16:59:51.872 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.Jzqc0eetgt 2024-04-15T16:59:51.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T16:59:51.981 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-15T16:59:51.981 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-15T16:59:51.981 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-15T16:59:51.982 INFO:teuthology.orchestra.run.smithi159.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-15T16:59:51.983 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-15T16:59:52.101 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-04-15T16:59:54.092 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:54.117 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:54.472 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:54.472 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-15T16:59:54.472 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:59:54.981 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T16:59:55.131 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:55.345 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:55.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:55.346 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-15T16:59:55.346 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T16:59:55.346 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T16:59:55.347 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T16:59:55.347 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T16:59:55.347 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T16:59:55.557 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-15T16:59:55.557 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-15T16:59:55.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2024-04-15T16:59:55.591 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2024-04-15T16:59:55.591 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.02 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 129.el8 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Group : System Environment/Base 2024-04-15T16:59:55.592 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 9265586 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-15T16:59:55.593 INFO:teuthology.orchestra.run.smithi082.stdout:Relocations : (not relocatable) 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : CentOS BuildSystem 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2024-04-15T16:59:55.594 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:59:55.595 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T16:59:55.595 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T16:59:55.595 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T16:59:55.595 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2024-04-15T16:59:55.595 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T16:59:55.596 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T16:59:55.597 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T16:59:55.597 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 4.18.0-552.el8.x86_64 2024-04-15T16:59:55.597 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T16:59:57.507 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-04-15T16:59:59.667 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-04-15T16:59:59.668 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-04-15T16:59:59.697 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.oXPNNUGpxc 2024-04-15T16:59:59.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oXPNNUGpxc 2024-04-15T16:59:59.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.oXPNNUGpxc 2024-04-15T16:59:59.914 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.oXPNNUGpxc is 6KB 2024-04-15T16:59:59.962 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.oXPNNUGpxc 2024-04-15T16:59:59.980 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T17:00:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-15T17:00:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-15T17:00:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-15T17:00:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-15T17:00:00.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-15T17:00:00.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-04-15T17:00:10.499 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T17:00:10.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:00:10.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:22.114 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-15T17:00:22.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:00:22.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:28.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-15T17:00:30.232 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T17:00:30.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:00:30.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:37.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:00:37.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:40.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-04-15T17:00:40.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-15T17:00:48.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-04-15T17:00:49.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:00:49.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:52.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-04-15T17:00:52.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:00:52.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:00:55.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-15T17:00:57.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:00:57.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:00.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-04-15T17:01:04.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:01:04.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:10.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:01:10.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:12.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:01:12.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:14.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-15T17:01:31.295 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-04-15T17:01:32.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:01:32.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:35.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-15T17:01:46.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:01:46.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:01:47.574 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-04-15T17:01:48.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:01:48.034 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-552.el8.x86_64"... 2024-04-15T17:01:48.034 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-04-15T17:01:48.050 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-552.el8.x86_64 2024-04-15T17:01:48.051 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-15T17:01:48.051 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T17:01:48.051 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T17:01:49.052 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T17:01:49.052 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T17:01:49.134 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-04-15T17:01:49.149 DEBUG:teuthology.parallel:result is None 2024-04-15T17:01:56.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:01:56.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:02:04.677 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T17:02:11.798 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-15T17:02:12.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:02:12.069 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-15T17:02:12.070 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-04-15T17:02:12.091 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-552.el8.x86_64 2024-04-15T17:02:12.091 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-15T17:02:12.092 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T17:02:12.092 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T17:02:13.092 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T17:02:13.093 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T17:02:13.151 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-04-15T17:02:13.170 DEBUG:teuthology.parallel:result is None 2024-04-15T17:02:19.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:02:19.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:02:20.021 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-04-15T17:02:20.306 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:02:20.306 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-552.el8.x86_64"... 2024-04-15T17:02:20.306 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-04-15T17:02:20.325 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-552.el8.x86_64 2024-04-15T17:02:20.325 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-15T17:02:20.325 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T17:02:20.325 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T17:02:21.327 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-15T17:02:21.327 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T17:02:21.389 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2024-04-15T17:02:21.402 DEBUG:teuthology.parallel:result is None 2024-04-15T17:02:21.402 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T17:02:21.409 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T17:02:21.409 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T17:02:21.412 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T17:02:21.415 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T17:02:21.437 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T17:02:21.443 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T17:02:21.450 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T17:02:21.452 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T17:02:21.476 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T17:02:21.478 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T17:02:21.647 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T17:02:21.654 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T17:02:21.654 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T17:02:21.657 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T17:02:21.659 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T17:02:21.697 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.702 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.704 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.708 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.714 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.716 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T17:02:21.718 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T17:02:21.725 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T17:02:21.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T17:02:21.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T17:02:21.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T17:02:21.780 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T17:02:21.790 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T17:02:21.790 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T17:02:21.800 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T17:02:21.803 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T17:02:21.840 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:21.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:21.965 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.054 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T17:02:22.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T17:02:22.122 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:22.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:22.226 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.358 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T17:02:22.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T17:02:22.428 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:22.476 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T17:02:22.539 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.586 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T17:02:22.631 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T17:02:22.631 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T17:02:22.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-04-15T17:02:22.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-04-15T17:02:22.706 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-04-15T17:02:22.770 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T17:02:22.774 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T17:02:22.808 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T17:02:23.218 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T17:02:23.226 INFO:teuthology.task.internal:Starting timer... 2024-04-15T17:02:23.226 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T17:02:23.238 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T17:02:23.246 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-15T17:02:23.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-04-15T17:02:23.321 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-04-15T17:02:23.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-04-15T17:02:23.391 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-04-15T17:02:23.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-04-15T17:02:23.477 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-04-15T17:02:23.479 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T17:02:23.479 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-04-15T17:02:23.507 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-04-15T17:02:23.508 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-04-15T17:02:23.607 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-04-15T17:02:23.608 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-04-15T17:02:23.731 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-04-15T17:02:23.731 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-04-15T17:02:23.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T17:02:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T17:02:23.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T17:02:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T17:02:23.792 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T17:02:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T17:02:23.819 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T17:02:23.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-04-15T17:02:23.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-04-15T17:02:23.877 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-04-15T17:02:23.904 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T17:02:23.914 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T17:02:23.933 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T17:02:23.934 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi082.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T17:46:16.816 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-04-15T17:46:16.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:46:16.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:46:16.921 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-15T17:46:16.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-15T17:46:16.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:46:16.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:46:17.021 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-04-15T17:46:17.090 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-15T17:46:17.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:46:17.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T17:46:17.162 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-04-15T17:46:17.188 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-15T17:46:17.188 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T17:46:17.198 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T17:46:17.199 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T17:46:17.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T17:46:17.201 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T17:46:17.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T17:46:17.203 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T17:46:17.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T17:46:17.233 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T17:46:17.235 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T17:46:17.248 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T17:46:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T17:46:17.260 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T17:46:17.274 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T17:46:17.309 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-04-15T17:46:17.320 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-04-15T17:46:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-04-15T17:46:17.335 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T17:46:17.335 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-04-15T17:46:17.340 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-04-15T17:46:17.348 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T17:46:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T17:46:17.351 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-04-15T17:46:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T17:46:17.365 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T17:46:17.378 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T17:46:17.414 INFO:teuthology.orchestra.run.smithi082.stderr:bash: ntpq: command not found 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.418 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.438 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2024-04-15T17:46:17.440 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.442 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.444 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T17:46:17.445 INFO:teuthology.run_tasks:Running task pexec... 2024-04-15T17:46:17.454 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-15T17:46:17.455 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi049.front.sepia.ceph.com 2024-04-15T17:46:17.455 DEBUG:teuthology.orchestra.run.smithi049:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-15T17:46:17.455 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi082.front.sepia.ceph.com 2024-04-15T17:46:17.455 DEBUG:teuthology.orchestra.run.smithi082:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-15T17:46:17.456 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi159.front.sepia.ceph.com 2024-04-15T17:46:17.456 DEBUG:teuthology.orchestra.run.smithi159:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-15T17:46:18.222 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:17:39 ago on Mon 15 Apr 2024 05:28:39 PM UTC. 2024-04-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:48 ago on Mon 15 Apr 2024 05:44:30 PM UTC. 2024-04-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:07:30 ago on Mon 15 Apr 2024 05:38:48 PM UTC. 2024-04-15T17:46:20.370 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout:Resetting modules: 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout: container-tools 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:46:20.371 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:20.372 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:20.394 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:46:20.395 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:20.395 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout:Resetting modules: 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout: container-tools 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:20.396 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:20.593 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout:Resetting modules: 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:20.594 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:20.625 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:46:20.696 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:46:20.864 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:46:21.403 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:17:42 ago on Mon 15 Apr 2024 05:28:39 PM UTC. 2024-04-15T17:46:21.496 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:51 ago on Mon 15 Apr 2024 05:44:30 PM UTC. 2024-04-15T17:46:21.673 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:07:33 ago on Mon 15 Apr 2024 05:38:48 PM UTC. 2024-04-15T17:46:23.498 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout:Installing group/module packages: 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-15T17:46:23.535 INFO:teuthology.orchestra.run.smithi082.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout:Installing module profiles: 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: container-tools/common 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:23.536 INFO:teuthology.orchestra.run.smithi082.stdout:Enabling module streams: 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: container-tools rhel8 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:23.537 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 27 M 2024-04-15T17:46:23.538 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 91 M 2024-04-15T17:46:23.538 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:46:23.643 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:46:23.680 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout:Installing group/module packages: 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-15T17:46:23.681 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:Installing module profiles: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: container-tools/common 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:Enabling module streams: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: container-tools rhel8 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:46:23.682 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2024-04-15T17:46:23.683 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:23.683 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 27 M 2024-04-15T17:46:23.683 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 91 M 2024-04-15T17:46:23.683 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:46:23.732 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 4.9 MB/s | 689 kB 00:00 2024-04-15T17:46:23.765 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 1.1 MB/s | 38 kB 00:00 2024-04-15T17:46:23.799 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 4.9 MB/s | 1.0 MB 00:00 2024-04-15T17:46:23.833 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.8 MB/s | 258 kB 00:00 2024-04-15T17:46:23.877 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 5.0 MB/s | 689 kB 00:00 2024-04-15T17:46:23.883 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.5 MB/s | 180 kB 00:00 2024-04-15T17:46:23.924 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:46:23.934 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 5.2 MB/s | 1.0 MB 00:00 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout:Installing group/module packages: 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-15T17:46:23.965 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Installing module profiles: 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools/common 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling module streams: 2024-04-15T17:46:23.966 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools rhel8 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 27 M 2024-04-15T17:46:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 91 M 2024-04-15T17:46:23.968 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:46:23.968 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 412 kB/s | 38 kB 00:00 2024-04-15T17:46:23.992 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): buildah-1.33.6-2.module_el8+976+1136073 24 MB/s | 9.6 MB 00:00 2024-04-15T17:46:23.994 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 4.2 MB/s | 258 kB 00:00 2024-04-15T17:46:24.025 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): python3-pytoml-0.1.14-5.git7dea353.el8. 174 kB/s | 25 kB 00:00 2024-04-15T17:46:24.036 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 4.2 MB/s | 180 kB 00:00 2024-04-15T17:46:24.050 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-04-15T17:46:24.070 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 740 kB/s | 25 kB 00:00 2024-04-15T17:46:24.129 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-04-15T17:46:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 14 MB/s | 2.5 MB 00:00 2024-04-15T17:46:24.260 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.9 MB/s | 689 kB 00:00 2024-04-15T17:46:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 1.5 MB/s | 38 kB 00:00 2024-04-15T17:46:24.304 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 12 MB/s | 4.1 MB 00:00 2024-04-15T17:46:24.318 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): netavark-1.10.3-1.module_el8+951+32019 7.9 MB/s | 4.1 MB 00:00 2024-04-15T17:46:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 485 kB/s | 53 kB 00:00 2024-04-15T17:46:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2024-04-15T17:46:24.369 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 3.0 MB/s | 258 kB 00:00 2024-04-15T17:46:24.394 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.7 MB/s | 1.0 MB 00:00 2024-04-15T17:46:24.461 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.6 MB/s | 180 kB 00:00 2024-04-15T17:46:24.494 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 741 kB/s | 25 kB 00:00 2024-04-15T17:46:24.504 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 13 MB/s | 9.6 MB 00:00 2024-04-15T17:46:24.545 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-04-15T17:46:24.554 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 10 MB/s | 2.5 MB 00:00 2024-04-15T17:46:24.580 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 699 kB/s | 53 kB 00:00 2024-04-15T17:46:24.605 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2024-04-15T17:46:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 13 MB/s | 8.8 MB 00:00 2024-04-15T17:46:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:46:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:Total 23 MB/s | 27 MB 00:01 2024-04-15T17:46:24.805 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 13 MB/s | 8.8 MB 00:00 2024-04-15T17:46:24.806 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:46:24.806 INFO:teuthology.orchestra.run.smithi159.stdout:Total 24 MB/s | 27 MB 00:01 2024-04-15T17:46:24.845 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 8.6 MB/s | 4.1 MB 00:00 2024-04-15T17:46:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:46:24.999 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:46:24.999 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:46:25.020 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 14 MB/s | 2.5 MB 00:00 2024-04-15T17:46:25.076 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:46:25.098 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:46:25.098 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:46:25.162 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): buildah-1.33.6-2.module_el8+976+113607 8.5 MB/s | 9.6 MB 00:01 2024-04-15T17:46:25.187 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 315 kB/s | 53 kB 00:00 2024-04-15T17:46:25.213 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2024-04-15T17:46:25.264 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:46:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:46:25.359 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:46:25.362 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:46:25.371 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 11 MB/s | 8.8 MB 00:00 2024-04-15T17:46:25.372 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:46:25.372 INFO:teuthology.orchestra.run.smithi049.stdout:Total 20 MB/s | 27 MB 00:01 2024-04-15T17:46:25.648 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:46:25.671 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:46:25.671 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:46:25.888 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:46:25.930 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:46:25.933 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:46:26.003 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:46:26.564 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:46:26.799 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-15T17:46:26.942 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-15T17:46:26.959 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-15T17:46:27.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-15T17:46:27.175 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-15T17:46:27.334 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-15T17:46:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-15T17:46:27.532 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-15T17:46:27.565 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-15T17:46:27.569 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-15T17:46:27.772 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-15T17:46:27.808 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-15T17:46:27.960 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-15T17:46:28.089 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-15T17:46:28.169 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-15T17:46:28.314 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-15T17:46:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-15T17:46:28.979 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-15T17:46:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-15T17:46:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-15T17:46:29.318 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-15T17:46:29.505 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-15T17:46:29.576 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-15T17:46:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-15T17:46:29.946 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-15T17:46:29.989 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-15T17:46:30.118 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:30.203 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-15T17:46:30.270 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-15T17:46:30.300 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:30.469 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-15T17:46:30.495 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:30.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:30.820 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:30.915 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:30.915 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-15T17:46:30.915 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:30.916 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:30.917 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-15T17:46:31.169 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-15T17:46:31.270 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-15T17:46:31.271 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-15T17:46:31.271 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:31.271 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:31.359 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-15T17:46:31.453 INFO:teuthology.orchestra.run.smithi082.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:46:31.454 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:46:31.609 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:31.745 DEBUG:teuthology.parallel:result is None 2024-04-15T17:46:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-15T17:46:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:31.849 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:46:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-15T17:46:31.849 INFO:teuthology.orchestra.run.smithi159.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:46:31.850 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:46:32.052 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-15T17:46:32.052 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-15T17:46:32.052 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-15T17:46:32.053 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-15T17:46:32.130 DEBUG:teuthology.parallel:result is None 2024-04-15T17:46:32.608 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-15T17:46:32.609 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-15T17:46:32.610 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-15T17:46:32.610 INFO:teuthology.orchestra.run.smithi049.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-15T17:46:32.610 INFO:teuthology.orchestra.run.smithi049.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-15T17:46:32.610 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:46:32.610 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:46:32.989 DEBUG:teuthology.parallel:result is None 2024-04-15T17:46:32.989 INFO:teuthology.run_tasks:Running task install... 2024-04-15T17:46:32.997 DEBUG:teuthology.task.install:project ceph 2024-04-15T17:46:32.997 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54'}} 2024-04-15T17:46:32.998 DEBUG:teuthology.task.install:config {'branch': 'octopus', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54'} 2024-04-15T17:46:32.998 INFO:teuthology.task.install:Using flavor: default 2024-04-15T17:46:33.004 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T17:46:33.004 INFO:teuthology.task.install:extra packages: [] 2024-04-15T17:46:33.004 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'octopus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54', 'tag': None, 'wait_for_package': False} 2024-04-15T17:46:33.004 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.004 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.004 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.005 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.005 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T17:46:33.007 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'octopus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54', 'tag': None, 'wait_for_package': False} 2024-04-15T17:46:33.007 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.007 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.007 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.007 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.007 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T17:46:33.008 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'octopus', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54', 'tag': None, 'wait_for_package': False} 2024-04-15T17:46:33.008 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.008 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.009 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.009 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.009 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T17:46:33.215 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/ 2024-04-15T17:46:33.216 INFO:teuthology.task.install.rpm:Package version is 15.2.17-0 2024-04-15T17:46:33.234 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/ 2024-04-15T17:46:33.234 INFO:teuthology.task.install.rpm:Package version is 15.2.17-0 2024-04-15T17:46:33.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T17:46:33.247 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T17:46:33.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T17:46:33.258 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T17:46:33.259 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T17:46:33.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T17:46:33.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-15T17:46:33.276 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.276 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.276 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.276 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.276 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.277 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/octopus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T17:46:33.291 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-15T17:46:33.292 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.292 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.292 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.292 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.292 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.292 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/octopus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T17:46:33.312 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/ 2024-04-15T17:46:33.313 INFO:teuthology.task.install.rpm:Package version is 15.2.17-0 2024-04-15T17:46:33.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T17:46:33.338 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T17:46:33.338 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T17:46:33.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T17:46:33.362 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T17:46:33.372 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-15T17:46:33.372 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T17:46:33.372 INFO:teuthology.packaging:ref: None 2024-04-15T17:46:33.372 INFO:teuthology.packaging:tag: None 2024-04-15T17:46:33.372 INFO:teuthology.packaging:branch: octopus 2024-04-15T17:46:33.372 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T17:46:33.373 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/octopus/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T17:46:33.428 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T17:46:33.441 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T17:46:33.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T17:46:33.495 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-04-15T17:46:33.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-04-15T17:46:33.508 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-04-15T17:46:33.510 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-04-15T17:46:33.522 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T17:46:33.591 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-04-15T17:46:33.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-04-15T17:46:33.780 INFO:teuthology.orchestra.run.smithi049.stdout:78 files removed 2024-04-15T17:46:33.790 INFO:teuthology.orchestra.run.smithi159.stdout:78 files removed 2024-04-15T17:46:33.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-04-15T17:46:33.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2024-04-15T17:46:33.871 INFO:teuthology.orchestra.run.smithi082.stdout:78 files removed 2024-04-15T17:46:33.899 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-04-15T17:46:34.330 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 631 kB/s | 68 kB 00:00 2024-04-15T17:46:34.347 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 618 kB/s | 68 kB 00:00 2024-04-15T17:46:34.377 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 884 kB/s | 68 kB 00:00 2024-04-15T17:46:34.556 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 89 kB/s | 15 kB 00:00 2024-04-15T17:46:34.561 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 103 kB/s | 15 kB 00:00 2024-04-15T17:46:34.573 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2024-04-15T17:46:34.712 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 14 kB/s | 1.7 kB 00:00 2024-04-15T17:46:34.715 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 13 kB/s | 1.7 kB 00:00 2024-04-15T17:46:34.723 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 14 kB/s | 1.7 kB 00:00 2024-04-15T17:46:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 20 MB/s | 28 MB 00:01 2024-04-15T17:46:36.190 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - AppStream 19 MB/s | 28 MB 00:01 2024-04-15T17:46:36.346 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 17 MB/s | 28 MB 00:01 2024-04-15T17:53:24.049 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 26 kB/s | 10 MB 06:42 2024-04-15T17:53:26.006 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2024-04-15T17:53:26.307 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 40 kB/s | 7.7 kB 00:00 2024-04-15T17:53:27.591 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - PowerTools 4.1 MB/s | 5.0 MB 00:01 2024-04-15T17:53:28.676 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-15T17:53:29.987 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-15T17:53:33.819 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 364 kB/s | 24 kB 00:00 2024-04-15T17:53:37.416 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:15.2.17-0.el8 ceph 9.9 M 2024-04-15T17:53:37.419 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:15.2.17-0.el8 ceph 3.5 M 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:15.2.17-0.el8 ceph 1.9 M 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:15.2.17-0.el8 ceph 5.6 M 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:15.2.17-0.el8 ceph 21 M 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:15.2.17-0.el8 ceph 25 k 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-15T17:53:37.420 INFO:teuthology.orchestra.run.smithi082.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:15.2.17-0.el8 ceph 508 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:15.2.17-0.el8 ceph 380 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:15.2.17-0.el8 ceph 3.2 M 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:15.2.17-0.el8 ceph 40 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:15.2.17-0.el8 ceph 77 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:15.2.17-0.el8 ceph 164 k 2024-04-15T17:53:37.421 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:15.2.17-0.el8 ceph 364 k 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:15.2.17-0.el8 ceph 349 k 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:15.2.17-0.el8 ceph 113 k 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout:Install 22 Packages 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-04-15T17:53:37.422 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:53:37.423 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 49 M 2024-04-15T17:53:37.423 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:53:37.869 INFO:teuthology.orchestra.run.smithi082.stdout:(1/24): ceph-base-15.2.17-0.el8.x86_64.rpm 41 MB/s | 5.6 MB 00:00 2024-04-15T17:53:37.895 INFO:teuthology.orchestra.run.smithi082.stdout:(2/24): ceph-selinux-15.2.17-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-15T17:53:37.962 INFO:teuthology.orchestra.run.smithi082.stdout:(3/24): libcephfs2-15.2.17-0.el8.x86_64.rpm 7.5 MB/s | 508 kB 00:00 2024-04-15T17:53:37.995 INFO:teuthology.orchestra.run.smithi082.stdout:(4/24): libradosstriper1-15.2.17-0.el8.x86_64.r 11 MB/s | 380 kB 00:00 2024-04-15T17:53:38.079 INFO:teuthology.orchestra.run.smithi082.stdout:(5/24): librgw2-15.2.17-0.el8.x86_64.rpm 39 MB/s | 3.2 MB 00:00 2024-04-15T17:53:38.104 INFO:teuthology.orchestra.run.smithi082.stdout:(6/24): python3-ceph-argparse-15.2.17-0.el8.x86 1.6 MB/s | 40 kB 00:00 2024-04-15T17:53:38.129 INFO:teuthology.orchestra.run.smithi082.stdout:(7/24): python3-ceph-common-15.2.17-0.el8.x86_6 3.0 MB/s | 77 kB 00:00 2024-04-15T17:53:38.155 INFO:teuthology.orchestra.run.smithi082.stdout:(8/24): python3-cephfs-15.2.17-0.el8.x86_64.rpm 6.4 MB/s | 164 kB 00:00 2024-04-15T17:53:38.263 INFO:teuthology.orchestra.run.smithi082.stdout:(9/24): ceph-radosgw-15.2.17-0.el8.x86_64.rpm 19 MB/s | 9.9 MB 00:00 2024-04-15T17:53:38.289 INFO:teuthology.orchestra.run.smithi082.stdout:(10/24): python3-rados-15.2.17-0.el8.x86_64.rpm 2.7 MB/s | 364 kB 00:00 2024-04-15T17:53:38.314 INFO:teuthology.orchestra.run.smithi082.stdout:(11/24): python3-rbd-15.2.17-0.el8.x86_64.rpm 6.8 MB/s | 349 kB 00:00 2024-04-15T17:53:38.339 INFO:teuthology.orchestra.run.smithi082.stdout:(12/24): python3-rgw-15.2.17-0.el8.x86_64.rpm 2.2 MB/s | 113 kB 00:00 2024-04-15T17:53:38.565 INFO:teuthology.orchestra.run.smithi082.stdout:(13/24): ceph-common-15.2.17-0.el8.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-04-15T17:53:38.590 INFO:teuthology.orchestra.run.smithi082.stdout:(14/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-04-15T17:53:38.682 INFO:teuthology.orchestra.run.smithi082.stdout:(15/24): librdkafka-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 677 kB 00:00 2024-04-15T17:53:38.699 INFO:teuthology.orchestra.run.smithi082.stdout:(16/24): python3-prettytable-0.7.2-14.el8.noarc 328 kB/s | 44 kB 00:00 2024-04-15T17:53:38.791 INFO:teuthology.orchestra.run.smithi082.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.0 MB/s | 116 kB 00:00 2024-04-15T17:53:38.824 INFO:teuthology.orchestra.run.smithi082.stdout:(18/24): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 MB/s | 306 kB 00:00 2024-04-15T17:53:38.858 INFO:teuthology.orchestra.run.smithi082.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 2.6 MB/s | 181 kB 00:00 2024-04-15T17:53:38.875 INFO:teuthology.orchestra.run.smithi082.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2024-04-15T17:53:38.909 INFO:teuthology.orchestra.run.smithi082.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2024-04-15T17:53:38.967 INFO:teuthology.orchestra.run.smithi082.stdout:(22/24): librados2-15.2.17-0.el8.x86_64.rpm 39 MB/s | 3.5 MB 00:00 2024-04-15T17:53:39.009 INFO:teuthology.orchestra.run.smithi082.stdout:(23/24): librbd1-15.2.17-0.el8.x86_64.rpm 19 MB/s | 1.9 MB 00:00 2024-04-15T17:53:41.700 INFO:teuthology.orchestra.run.smithi082.stdout:(24/24): librabbitmq-0.9.0-5.el8.x86_64.rpm 15 kB/s | 47 kB 00:03 2024-04-15T17:53:41.703 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:53:41.703 INFO:teuthology.orchestra.run.smithi082.stdout:Total 11 MB/s | 49 MB 00:04 2024-04-15T17:53:41.746 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:53:41.793 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:53:41.793 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:53:42.092 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:53:42.095 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:53:43.103 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:53:43.312 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2024-04-15T17:53:43.481 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2024-04-15T17:53:43.689 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:53:44.045 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:53:44.205 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:53:44.323 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:53:44.546 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:53:44.705 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:53:44.912 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:53:45.172 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:53:45.223 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:53:45.432 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/26 2024-04-15T17:53:45.833 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/26 2024-04-15T17:53:45.988 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:53:46.063 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:53:46.227 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2024-04-15T17:53:46.414 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:53:46.584 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:53:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:15.2.17-0.el8.x86_64 14/26 2024-04-15T17:53:46.979 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:15.2.17-0.el8.x86_64 15/26 2024-04-15T17:53:47.146 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:53:47.217 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:53:47.448 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2024-04-15T17:53:47.630 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2024-04-15T17:53:47.838 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2024-04-15T17:53:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:15.2.17-0.el8.x86_64 20/26 2024-04-15T17:53:50.283 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:53:50.496 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:53:51.138 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:53:51.414 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:53:51.522 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:53:51.679 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /run 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-04-15T17:54:10.600 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-04-15T17:54:10.601 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-04-15T17:54:10.601 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-04-15T17:54:10.601 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2024-04-15T17:54:10.601 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:11.659 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:54:11.729 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:54:11.730 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:54:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:54:11.968 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:54:12.584 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:54:12.584 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:15.2.17-0.el8.x86_64 1/26 2024-04-15T17:54:12.584 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:15.2.17-0.el8.x86_64 2/26 2024-04-15T17:54:12.584 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:15.2.17-0.el8.x86_64 3/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 8/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:15.2.17-0.el8.x86_64 9/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:15.2.17-0.el8.x86_64 11/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:54:12.585 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2024-04-15T17:54:12.586 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:15.2.17-0.el8.x86_64 25/26 2024-04-15T17:54:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:54:13.507 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:13.507 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:15.2.17-0.el8.x86_64 librbd1-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: fmt-6.2.1-1.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: leveldb-1.22-1.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.508 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:13.509 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:54:13.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-04-15T17:54:14.530 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:41 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:54:16.161 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:15.2.17-0.el8 ceph 42 M 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-15T17:54:16.162 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-15T17:54:16.163 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:16.163 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:54:16.163 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:16.163 INFO:teuthology.orchestra.run.smithi082.stdout:Install 5 Packages 2024-04-15T17:54:16.163 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:16.166 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 43 M 2024-04-15T17:54:16.166 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 155 M 2024-04-15T17:54:16.166 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:54:16.438 INFO:teuthology.orchestra.run.smithi082.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 2.6 MB/s | 203 kB 00:00 2024-04-15T17:54:16.463 INFO:teuthology.orchestra.run.smithi082.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-15T17:54:16.497 INFO:teuthology.orchestra.run.smithi082.stdout:(3/5): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-15T17:54:16.522 INFO:teuthology.orchestra.run.smithi082.stdout:(4/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-15T17:54:16.989 INFO:teuthology.orchestra.run.smithi082.stdout:(5/5): ceph-test-15.2.17-0.el8.x86_64.rpm 67 MB/s | 42 MB 00:00 2024-04-15T17:54:16.990 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:54:16.990 INFO:teuthology.orchestra.run.smithi082.stdout:Total 52 MB/s | 43 MB 00:00 2024-04-15T17:54:17.032 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:54:17.052 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:54:17.052 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:54:17.300 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:54:17.303 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:54:17.755 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:54:17.946 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-15T17:54:18.100 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-15T17:54:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:54:18.326 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:54:22.196 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-15T17:54:22.420 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:54:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:54:22.702 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:15.2.17-0.el8.x86_64 1/5 2024-04-15T17:54:22.705 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-15T17:54:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:54:22.706 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:15.2.17-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-15T17:54:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:23.162 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:54:23.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-04-15T17:54:24.108 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:51 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:54:25.749 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:54:25.752 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:25.752 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-04-15T17:54:25.752 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:25.752 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:54:25.752 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:15.2.17-0.el8 ceph 6.3 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:15.2.17-0.el8 ceph 2.0 M 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:15.2.17-0.el8 ceph 1.4 M 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:15.2.17-0.el8 ceph-noarch 193 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:15.2.17-0.el8 ceph 4.3 M 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:15.2.17-0.el8 ceph 17 M 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-15T17:54:25.753 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-15T17:54:25.754 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:54:25.755 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:54:25.756 INFO:teuthology.orchestra.run.smithi082.stdout:Install 32 Packages 2024-04-15T17:54:25.756 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:54:25.756 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 28 M 2024-04-15T17:54:25.756 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 91 M 2024-04-15T17:54:25.756 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:54:26.306 INFO:teuthology.orchestra.run.smithi082.stdout:(1/32): ceph-15.2.17-0.el8.x86_64.rpm 116 kB/s | 6.3 kB 00:00 2024-04-15T17:54:26.356 INFO:teuthology.orchestra.run.smithi082.stdout:(2/32): ceph-mgr-15.2.17-0.el8.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-04-15T17:54:26.398 INFO:teuthology.orchestra.run.smithi082.stdout:(3/32): ceph-mds-15.2.17-0.el8.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-04-15T17:54:26.424 INFO:teuthology.orchestra.run.smithi082.stdout:(4/32): ceph-mgr-modules-core-15.2.17-0.el8.noa 7.5 MB/s | 193 kB 00:00 2024-04-15T17:54:26.499 INFO:teuthology.orchestra.run.smithi082.stdout:(5/32): ceph-mon-15.2.17-0.el8.x86_64.rpm 23 MB/s | 4.3 MB 00:00 2024-04-15T17:54:26.525 INFO:teuthology.orchestra.run.smithi082.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-04-15T17:54:26.600 INFO:teuthology.orchestra.run.smithi082.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.3 MB/s | 103 kB 00:00 2024-04-15T17:54:26.625 INFO:teuthology.orchestra.run.smithi082.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 309 kB/s | 39 kB 00:00 2024-04-15T17:54:26.650 INFO:teuthology.orchestra.run.smithi082.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-04-15T17:54:26.701 INFO:teuthology.orchestra.run.smithi082.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.9 MB/s | 457 kB 00:00 2024-04-15T17:54:26.868 INFO:teuthology.orchestra.run.smithi082.stdout:(11/32): ceph-osd-15.2.17-0.el8.x86_64.rpm 33 MB/s | 17 MB 00:00 2024-04-15T17:54:48.098 INFO:teuthology.orchestra.run.smithi082.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 5.1 kB/s | 109 kB 00:21 2024-04-15T17:54:48.406 INFO:teuthology.orchestra.run.smithi082.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 141 kB/s | 44 kB 00:00 2024-04-15T17:54:48.615 INFO:teuthology.orchestra.run.smithi082.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 889 kB/s | 185 kB 00:00 2024-04-15T17:54:48.865 INFO:teuthology.orchestra.run.smithi082.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 690 kB/s | 173 kB 00:00 2024-04-15T17:54:49.016 INFO:teuthology.orchestra.run.smithi082.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-04-15T17:54:49.449 INFO:teuthology.orchestra.run.smithi082.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 26 kB/s | 11 kB 00:00 2024-04-15T17:54:50.049 INFO:teuthology.orchestra.run.smithi082.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 29 kB/s | 18 kB 00:00 2024-04-15T17:54:50.308 INFO:teuthology.orchestra.run.smithi082.stdout:(19/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 11 kB/s | 249 kB 00:23 2024-04-15T17:54:50.833 INFO:teuthology.orchestra.run.smithi082.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 62 kB/s | 49 kB 00:00 2024-04-15T17:54:51.275 INFO:teuthology.orchestra.run.smithi082.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 61 kB/s | 59 kB 00:00 2024-04-15T17:54:52.317 INFO:teuthology.orchestra.run.smithi082.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 15 kB/s | 16 kB 00:01 2024-04-15T17:54:52.634 INFO:teuthology.orchestra.run.smithi082.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 157 kB/s | 283 kB 00:01 2024-04-15T17:54:52.927 INFO:teuthology.orchestra.run.smithi082.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 31 kB/s | 19 kB 00:00 2024-04-15T17:54:53.227 INFO:teuthology.orchestra.run.smithi082.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 40 kB/s | 24 kB 00:00 2024-04-15T17:54:53.552 INFO:teuthology.orchestra.run.smithi082.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 47 kB/s | 29 kB 00:00 2024-04-15T17:54:53.836 INFO:teuthology.orchestra.run.smithi082.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 45 kB/s | 27 kB 00:00 2024-04-15T17:54:54.819 INFO:teuthology.orchestra.run.smithi082.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 202 kB/s | 255 kB 00:01 2024-04-15T17:54:55.228 INFO:teuthology.orchestra.run.smithi082.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 180 kB/s | 251 kB 00:01 2024-04-15T17:54:55.303 INFO:teuthology.orchestra.run.smithi082.stdout:(30/32): python3-cryptography-3.2.1-7.el8.x86_6 22 kB/s | 641 kB 00:28 2024-04-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi082.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 95 kB/s | 85 kB 00:00 2024-04-15T17:54:55.820 INFO:teuthology.orchestra.run.smithi082.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 39 kB/s | 23 kB 00:00 2024-04-15T17:54:55.823 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:54:55.824 INFO:teuthology.orchestra.run.smithi082.stdout:Total 968 kB/s | 28 MB 00:30 2024-04-15T17:54:55.901 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:54:55.940 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:54:55.940 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:54:56.246 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:54:56.249 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:54:57.369 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:54:57.569 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-15T17:54:57.822 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-15T17:54:58.105 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-15T17:54:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-15T17:54:58.585 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-15T17:54:58.731 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-15T17:54:58.900 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-15T17:54:59.077 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-15T17:54:59.253 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-15T17:54:59.499 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-15T17:54:59.762 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-15T17:55:00.006 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-15T17:55:00.384 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-15T17:55:00.666 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-15T17:55:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-15T17:55:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-15T17:55:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:55:01.610 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-15T17:55:01.862 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-15T17:55:02.016 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-15T17:55:02.409 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-15T17:55:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-15T17:55:02.852 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-15T17:55:03.185 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-15T17:55:03.532 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-15T17:55:03.867 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 27/32 2024-04-15T17:55:04.139 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:55:05.500 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:55:05.721 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:55:06.110 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:55:06.279 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:55:06.461 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:55:06.670 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:55:06.688 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:55:06.845 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:15.2.17-0.el8.x86_64 1/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:15.2.17-0.el8.x86_64 2/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:15.2.17-0.el8.x86_64 3/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:15.2.17-0.el8.x86_64 4/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:15.2.17-0.el8.x86_64 5/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 6/32 2024-04-15T17:55:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-15T17:55:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-15T17:55:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-15T17:55:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-15T17:55:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-15T17:55:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 2024-04-15T17:55:08.109 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-15T17:55:08.110 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-2.6-1.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:08.111 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:08.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-04-15T17:55:09.165 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:36 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:10.735 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:10.784 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:10.785 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:10.785 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:10.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-04-15T17:55:11.619 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:38 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:13.248 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:15.2.17-0.el8 ceph-noarch 59 k 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:55:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 59 k 2024-04-15T17:55:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 223 k 2024-04-15T17:55:13.250 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:13.296 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-15.2.17-0.el8.noarch.rpm 1.3 MB/s | 59 kB 00:00 2024-04-15T17:55:13.296 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.2 MB/s | 59 kB 00:00 2024-04-15T17:55:13.297 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:13.303 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:13.303 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:13.305 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:13.308 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:13.350 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:13.851 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:55:14.016 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:55:14.240 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:15.2.17-0.el8.noarch 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:14.641 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:14.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-04-15T17:55:15.617 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:42 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:17.239 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:15.2.17-0.el8 ceph 203 k 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:17.240 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:55:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:17.241 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 203 k 2024-04-15T17:55:17.241 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 599 k 2024-04-15T17:55:17.241 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:17.334 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-15.2.17-0.el8.x86_6 2.1 MB/s | 203 kB 00:00 2024-04-15T17:55:17.334 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:17.335 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.1 MB/s | 203 kB 00:00 2024-04-15T17:55:17.335 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:17.345 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:17.345 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:17.393 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:17.396 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:17.508 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:17.693 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:18.101 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:18.455 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:18.456 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:18.456 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:18.456 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:18.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-04-15T17:55:19.687 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:46 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:21.271 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:21.319 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:21.321 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:21.321 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:21.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-04-15T17:55:22.160 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:49 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:23.787 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:23.787 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:23.787 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:15.2.17-0.el8 ceph 572 k 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-04-15T17:55:23.788 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:23.789 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 656 k 2024-04-15T17:55:23.789 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.8 M 2024-04-15T17:55:23.789 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:23.986 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): ceph-fuse-15.2.17-0.el8.x86_64.rpm 9.6 MB/s | 572 kB 00:00 2024-04-15T17:55:27.752 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 22 kB/s | 83 kB 00:03 2024-04-15T17:55:27.752 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:27.753 INFO:teuthology.orchestra.run.smithi082.stdout:Total 165 kB/s | 656 kB 00:03 2024-04-15T17:55:27.764 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:27.786 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:27.787 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:27.872 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:27.875 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:28.088 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:28.311 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-15T17:55:28.490 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T17:55:28.889 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T17:55:28.889 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:15.2.17-0.el8.x86_64 1/2 2024-04-15T17:55:29.399 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-15T17:55:29.399 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:29.399 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:29.399 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:15.2.17-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-15T17:55:29.400 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:29.400 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:29.766 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-04-15T17:55:30.497 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:57 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:32.099 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:55:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:32.101 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 140 k 2024-04-15T17:55:32.101 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 478 k 2024-04-15T17:55:32.101 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:32.179 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-15.2.17-0.el8.x86_64.rpm 1.8 MB/s | 140 kB 00:00 2024-04-15T17:55:32.180 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:32.180 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.7 MB/s | 140 kB 00:00 2024-04-15T17:55:32.180 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:32.186 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:32.186 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:32.232 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:32.235 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:32.350 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:32.530 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:32.824 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:33.180 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:33.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-04-15T17:55:34.142 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:01 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:35.728 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:35.776 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:35.777 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:35.777 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:35.885 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-04-15T17:55:36.606 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:03 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:38.219 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:38.219 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:38.219 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:15.2.17-0.el8 ceph 22 k 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:55:38.220 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:38.221 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 22 k 2024-04-15T17:55:38.221 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 78 k 2024-04-15T17:55:38.221 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:38.261 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-15.2.17-0.el8.x86_64.rpm 544 kB/s | 22 kB 00:00 2024-04-15T17:55:38.261 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:38.261 INFO:teuthology.orchestra.run.smithi082.stdout:Total 528 kB/s | 22 kB 00:00 2024-04-15T17:55:38.262 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:38.265 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:38.265 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:38.266 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:38.270 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:38.318 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:38.453 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:38.753 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:39.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:39.092 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:39.092 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:39.092 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:39.092 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:39.093 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:39.217 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-04-15T17:55:39.948 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:06 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:41.502 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:41.551 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:41.552 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:41.552 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:41.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-04-15T17:55:42.384 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:09 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:43.965 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:44.014 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:44.015 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:44.015 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:44.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-04-15T17:55:44.848 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:11 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:46.425 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:46.473 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:46.475 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:46.475 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:46.590 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-04-15T17:55:47.313 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:14 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:48.889 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:48.940 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:48.941 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:48.941 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:49.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-04-15T17:55:49.778 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:16 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:51.340 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:51.390 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:51.392 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:51.392 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:51.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-04-15T17:55:52.214 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:19 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:53.762 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:55:53.811 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:53.812 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-04-15T17:55:53.812 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:53.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-04-15T17:55:54.668 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:21 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:55:56.292 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:55:56.292 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:15.2.17-0.el8 ceph 73 k 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:55:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:56.294 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 73 k 2024-04-15T17:55:56.294 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 188 k 2024-04-15T17:55:56.294 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:55:56.345 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-15.2.17-0.el8.x86_64.rpm 1.4 MB/s | 73 kB 00:00 2024-04-15T17:55:56.346 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:55:56.346 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.4 MB/s | 73 kB 00:00 2024-04-15T17:55:56.346 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:55:56.355 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:55:56.355 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:55:56.404 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:55:56.407 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:55:56.514 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:55:56.707 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:56.984 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:15.2.17-0.el8.x86_64 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:55:57.384 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:55:57.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-04-15T17:55:58.381 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:25 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:56:00.017 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:15.2.17-0.el8 ceph 2.7 M 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:56:00.018 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:00.019 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.7 M 2024-04-15T17:56:00.019 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 9.8 M 2024-04-15T17:56:00.019 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:56:00.106 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-15.2.17-0.el8.x86_64.rpm 31 MB/s | 2.7 MB 00:00 2024-04-15T17:56:00.106 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:56:00.106 INFO:teuthology.orchestra.run.smithi082.stdout:Total 31 MB/s | 2.7 MB 00:00 2024-04-15T17:56:00.107 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:56:00.118 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:56:00.118 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:56:00.175 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:56:00.178 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:56:00.518 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:56:00.683 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:15.2.17-0.el8.x86_64 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:01.457 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:56:01.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-04-15T17:56:02.385 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:29 ago on Mon 15 Apr 2024 05:53:33 PM UTC. 2024-04-15T17:56:04.013 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-04-15T17:56:04.014 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:04.015 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 140 k 2024-04-15T17:56:04.015 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 418 k 2024-04-15T17:56:04.015 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-04-15T17:56:04.059 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-15.2.17-0.el8.x86_64.rpm 3.1 MB/s | 140 kB 00:00 2024-04-15T17:56:04.060 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:56:04.060 INFO:teuthology.orchestra.run.smithi082.stdout:Total 3.0 MB/s | 140 kB 00:00 2024-04-15T17:56:04.060 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-04-15T17:56:04.069 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-04-15T17:56:04.069 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-04-15T17:56:04.115 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-04-15T17:56:04.118 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-04-15T17:56:04.218 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-04-15T17:56:04.397 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:05.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:56:05.072 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:05.072 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-04-15T17:56:05.072 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:15.2.17-0.el8.x86_64 2024-04-15T17:56:05.073 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-15T17:56:05.073 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-04-15T17:56:05.298 DEBUG:teuthology.parallel:result is None 2024-04-15T17:57:30.279 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 16 kB/s | 10 MB 10:48 2024-04-15T17:57:30.930 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - BaseOS 16 kB/s | 10 MB 10:49 2024-04-15T17:57:32.445 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 37 kB/s | 18 kB 00:00 2024-04-15T17:57:32.746 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 41 kB/s | 7.7 kB 00:00 2024-04-15T17:57:33.364 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras 22 kB/s | 18 kB 00:00 2024-04-15T17:57:33.689 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - Extras common packages 36 kB/s | 7.7 kB 00:00 2024-04-15T17:57:34.130 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - PowerTools 3.8 MB/s | 5.0 MB 00:01 2024-04-15T17:57:34.907 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 8 - PowerTools 4.4 MB/s | 5.0 MB 00:01 2024-04-15T17:57:35.382 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-04-15T17:57:35.992 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-15T17:57:37.635 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-15T17:57:38.485 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 16 MB 00:03 2024-04-15T17:57:41.476 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 419 kB/s | 24 kB 00:00 2024-04-15T17:57:42.550 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 288 kB/s | 24 kB 00:00 2024-04-15T17:57:45.109 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:57:45.111 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:15.2.17-0.el8 ceph 9.9 M 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:15.2.17-0.el8 ceph 3.5 M 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:15.2.17-0.el8 ceph 1.9 M 2024-04-15T17:57:45.112 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:15.2.17-0.el8 ceph 5.6 M 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:15.2.17-0.el8 ceph 21 M 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:15.2.17-0.el8 ceph 25 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:15.2.17-0.el8 ceph 508 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-15T17:57:45.113 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:15.2.17-0.el8 ceph 380 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:15.2.17-0.el8 ceph 3.2 M 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:15.2.17-0.el8 ceph 40 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:15.2.17-0.el8 ceph 77 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:15.2.17-0.el8 ceph 164 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-15T17:57:45.114 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:15.2.17-0.el8 ceph 364 k 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:15.2.17-0.el8 ceph 349 k 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:15.2.17-0.el8 ceph 113 k 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout:Install 22 Packages 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2024-04-15T17:57:45.115 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:57:45.116 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 49 M 2024-04-15T17:57:45.116 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:57:45.794 INFO:teuthology.orchestra.run.smithi159.stdout:(1/24): ceph-base-15.2.17-0.el8.x86_64.rpm 46 MB/s | 5.6 MB 00:00 2024-04-15T17:57:45.828 INFO:teuthology.orchestra.run.smithi159.stdout:(2/24): ceph-selinux-15.2.17-0.el8.x86_64.rpm 774 kB/s | 25 kB 00:00 2024-04-15T17:57:45.870 INFO:teuthology.orchestra.run.smithi159.stdout:(3/24): libcephfs2-15.2.17-0.el8.x86_64.rpm 12 MB/s | 508 kB 00:00 2024-04-15T17:57:45.943 INFO:teuthology.orchestra.run.smithi159.stdout:(4/24): libradosstriper1-15.2.17-0.el8.x86_64.r 5.1 MB/s | 380 kB 00:00 2024-04-15T17:57:46.037 INFO:teuthology.orchestra.run.smithi159.stdout:(5/24): librgw2-15.2.17-0.el8.x86_64.rpm 34 MB/s | 3.2 MB 00:00 2024-04-15T17:57:46.062 INFO:teuthology.orchestra.run.smithi159.stdout:(6/24): python3-ceph-argparse-15.2.17-0.el8.x86 1.6 MB/s | 40 kB 00:00 2024-04-15T17:57:46.087 INFO:teuthology.orchestra.run.smithi159.stdout:(7/24): python3-ceph-common-15.2.17-0.el8.x86_6 3.0 MB/s | 77 kB 00:00 2024-04-15T17:57:46.104 INFO:teuthology.orchestra.run.smithi159.stdout:(8/24): python3-cephfs-15.2.17-0.el8.x86_64.rpm 9.7 MB/s | 164 kB 00:00 2024-04-15T17:57:46.130 INFO:teuthology.orchestra.run.smithi159.stdout:(9/24): python3-rados-15.2.17-0.el8.x86_64.rpm 14 MB/s | 364 kB 00:00 2024-04-15T17:57:46.163 INFO:teuthology.orchestra.run.smithi159.stdout:(10/24): python3-rbd-15.2.17-0.el8.x86_64.rpm 10 MB/s | 349 kB 00:00 2024-04-15T17:57:46.189 INFO:teuthology.orchestra.run.smithi159.stdout:(11/24): python3-rgw-15.2.17-0.el8.x86_64.rpm 4.4 MB/s | 113 kB 00:00 2024-04-15T17:57:46.314 INFO:teuthology.orchestra.run.smithi159.stdout:(12/24): ceph-radosgw-15.2.17-0.el8.x86_64.rpm 15 MB/s | 9.9 MB 00:00 2024-04-15T17:57:46.381 INFO:teuthology.orchestra.run.smithi159.stdout:(13/24): librdkafka-1.6.1-1.el8.x86_64.rpm 3.4 MB/s | 677 kB 00:00 2024-04-15T17:57:46.425 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:57:46.428 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:57:46.428 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-15T17:57:46.428 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:15.2.17-0.el8 ceph 9.9 M 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:15.2.17-0.el8 ceph 3.5 M 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:15.2.17-0.el8 ceph 1.9 M 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:15.2.17-0.el8 ceph 5.6 M 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:15.2.17-0.el8 ceph 21 M 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:15.2.17-0.el8 ceph 25 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:15.2.17-0.el8 ceph 508 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-15T17:57:46.429 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:15.2.17-0.el8 ceph 380 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:15.2.17-0.el8 ceph 3.2 M 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:15.2.17-0.el8 ceph 40 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:15.2.17-0.el8 ceph 77 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:15.2.17-0.el8 ceph 164 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:15.2.17-0.el8 ceph 364 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:15.2.17-0.el8 ceph 349 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:15.2.17-0.el8 ceph 113 k 2024-04-15T17:57:46.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:Install 22 Packages 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 49 M 2024-04-15T17:57:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:57:46.581 INFO:teuthology.orchestra.run.smithi159.stdout:(14/24): ceph-common-15.2.17-0.el8.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-04-15T17:57:46.607 INFO:teuthology.orchestra.run.smithi159.stdout:(15/24): python3-prettytable-0.7.2-14.el8.noarc 195 kB/s | 44 kB 00:00 2024-04-15T17:57:46.632 INFO:teuthology.orchestra.run.smithi159.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 818 kB/s | 259 kB 00:00 2024-04-15T17:57:46.950 INFO:teuthology.orchestra.run.smithi049.stdout:(1/24): ceph-base-15.2.17-0.el8.x86_64.rpm 36 MB/s | 5.6 MB 00:00 2024-04-15T17:57:46.975 INFO:teuthology.orchestra.run.smithi049.stdout:(2/24): ceph-selinux-15.2.17-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-04-15T17:57:47.017 INFO:teuthology.orchestra.run.smithi049.stdout:(3/24): libcephfs2-15.2.17-0.el8.x86_64.rpm 12 MB/s | 508 kB 00:00 2024-04-15T17:57:47.051 INFO:teuthology.orchestra.run.smithi049.stdout:(4/24): libradosstriper1-15.2.17-0.el8.x86_64.r 11 MB/s | 380 kB 00:00 2024-04-15T17:57:47.057 INFO:teuthology.orchestra.run.smithi159.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 258 kB/s | 116 kB 00:00 2024-04-15T17:57:47.143 INFO:teuthology.orchestra.run.smithi049.stdout:(5/24): librgw2-15.2.17-0.el8.x86_64.rpm 35 MB/s | 3.2 MB 00:00 2024-04-15T17:57:47.168 INFO:teuthology.orchestra.run.smithi049.stdout:(6/24): python3-ceph-argparse-15.2.17-0.el8.x86 1.6 MB/s | 40 kB 00:00 2024-04-15T17:57:47.182 INFO:teuthology.orchestra.run.smithi159.stdout:(18/24): gperftools-libs-2.7-9.el8.x86_64.rpm 556 kB/s | 306 kB 00:00 2024-04-15T17:57:47.193 INFO:teuthology.orchestra.run.smithi049.stdout:(7/24): python3-ceph-common-15.2.17-0.el8.x86_6 3.0 MB/s | 77 kB 00:00 2024-04-15T17:57:47.207 INFO:teuthology.orchestra.run.smithi159.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2024-04-15T17:57:47.218 INFO:teuthology.orchestra.run.smithi049.stdout:(8/24): python3-cephfs-15.2.17-0.el8.x86_64.rpm 6.4 MB/s | 164 kB 00:00 2024-04-15T17:57:47.252 INFO:teuthology.orchestra.run.smithi049.stdout:(9/24): python3-rados-15.2.17-0.el8.x86_64.rpm 11 MB/s | 364 kB 00:00 2024-04-15T17:57:47.268 INFO:teuthology.orchestra.run.smithi159.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 687 kB/s | 59 kB 00:00 2024-04-15T17:57:47.277 INFO:teuthology.orchestra.run.smithi049.stdout:(10/24): python3-rbd-15.2.17-0.el8.x86_64.rpm 14 MB/s | 349 kB 00:00 2024-04-15T17:57:47.285 INFO:teuthology.orchestra.run.smithi159.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 967 kB/s | 75 kB 00:00 2024-04-15T17:57:47.336 INFO:teuthology.orchestra.run.smithi049.stdout:(11/24): python3-rgw-15.2.17-0.el8.x86_64.rpm 1.9 MB/s | 113 kB 00:00 2024-04-15T17:57:47.344 INFO:teuthology.orchestra.run.smithi159.stdout:(22/24): librbd1-15.2.17-0.el8.x86_64.rpm 33 MB/s | 1.9 MB 00:00 2024-04-15T17:57:47.403 INFO:teuthology.orchestra.run.smithi159.stdout:(23/24): librados2-15.2.17-0.el8.x86_64.rpm 26 MB/s | 3.5 MB 00:00 2024-04-15T17:57:47.445 INFO:teuthology.orchestra.run.smithi049.stdout:(12/24): ceph-radosgw-15.2.17-0.el8.x86_64.rpm 15 MB/s | 9.9 MB 00:00 2024-04-15T17:57:47.695 INFO:teuthology.orchestra.run.smithi049.stdout:(13/24): ceph-common-15.2.17-0.el8.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-04-15T17:57:47.720 INFO:teuthology.orchestra.run.smithi049.stdout:(14/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 943 kB/s | 259 kB 00:00 2024-04-15T17:57:47.745 INFO:teuthology.orchestra.run.smithi049.stdout:(15/24): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-04-15T17:57:47.762 INFO:teuthology.orchestra.run.smithi049.stdout:(16/24): python3-prettytable-0.7.2-14.el8.noarc 656 kB/s | 44 kB 00:00 2024-04-15T17:57:47.855 INFO:teuthology.orchestra.run.smithi049.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.0 MB/s | 116 kB 00:00 2024-04-15T17:57:47.880 INFO:teuthology.orchestra.run.smithi049.stdout:(18/24): gperftools-libs-2.7-9.el8.x86_64.rpm 2.6 MB/s | 306 kB 00:00 2024-04-15T17:57:47.922 INFO:teuthology.orchestra.run.smithi049.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 2.6 MB/s | 181 kB 00:00 2024-04-15T17:57:47.989 INFO:teuthology.orchestra.run.smithi049.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 541 kB/s | 59 kB 00:00 2024-04-15T17:57:48.014 INFO:teuthology.orchestra.run.smithi049.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 817 kB/s | 75 kB 00:00 2024-04-15T17:57:48.073 INFO:teuthology.orchestra.run.smithi049.stdout:(22/24): librbd1-15.2.17-0.el8.x86_64.rpm 33 MB/s | 1.9 MB 00:00 2024-04-15T17:57:48.131 INFO:teuthology.orchestra.run.smithi049.stdout:(23/24): librados2-15.2.17-0.el8.x86_64.rpm 25 MB/s | 3.5 MB 00:00 2024-04-15T17:57:50.844 INFO:teuthology.orchestra.run.smithi159.stdout:(24/24): librabbitmq-0.9.0-5.el8.x86_64.rpm 11 kB/s | 47 kB 00:04 2024-04-15T17:57:50.848 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:57:50.848 INFO:teuthology.orchestra.run.smithi159.stdout:Total 8.5 MB/s | 49 MB 00:05 2024-04-15T17:57:50.892 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:57:50.939 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:57:50.939 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:57:51.240 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:57:51.243 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:57:52.273 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:57:52.470 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2024-04-15T17:57:52.641 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2024-04-15T17:57:52.849 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:57:53.209 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:57:53.390 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:57:53.439 INFO:teuthology.orchestra.run.smithi049.stdout:(24/24): librabbitmq-0.9.0-5.el8.x86_64.rpm 8.2 kB/s | 47 kB 00:05 2024-04-15T17:57:53.441 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:57:53.442 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.9 MB/s | 49 MB 00:07 2024-04-15T17:57:53.485 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:57:53.486 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:57:53.537 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:57:53.537 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:57:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:57:53.862 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:57:53.866 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:57:53.924 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:57:54.147 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:57:54.407 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:57:54.475 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:57:54.692 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/26 2024-04-15T17:57:54.893 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:57:55.112 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/26 2024-04-15T17:57:55.139 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2024-04-15T17:57:55.299 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:57:55.353 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2024-04-15T17:57:55.390 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:57:55.553 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:57:55.561 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2024-04-15T17:57:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:57:55.922 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:57:55.971 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2024-04-15T17:57:56.114 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:15.2.17-0.el8.x86_64 14/26 2024-04-15T17:57:56.161 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:57:56.239 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:57:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:15.2.17-0.el8.x86_64 15/26 2024-04-15T17:57:56.440 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:57:56.493 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:57:56.620 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:57:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2024-04-15T17:57:56.778 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:57:56.940 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2024-04-15T17:57:57.012 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:57:57.097 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2024-04-15T17:57:57.227 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:57:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:15.2.17-0.el8.x86_64 20/26 2024-04-15T17:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:57:57.507 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/26 2024-04-15T17:57:57.992 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/26 2024-04-15T17:57:58.236 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:57:58.302 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:57:58.482 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2024-04-15T17:57:58.654 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:57:58.873 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:57:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:15.2.17-0.el8.x86_64 14/26 2024-04-15T17:57:59.287 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:15.2.17-0.el8.x86_64 15/26 2024-04-15T17:57:59.435 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:57:59.492 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:57:59.532 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:15.2.17-0.el8.x86_64 16/26 2024-04-15T17:57:59.707 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:57:59.748 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2024-04-15T17:57:59.928 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/26 2024-04-15T17:58:00.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2024-04-15T17:58:00.311 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:15.2.17-0.el8.x86_64 20/26 2024-04-15T17:58:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:58:00.674 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:58:00.784 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:58:00.931 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:02.603 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:58:02.860 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:58:03.568 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:15.2.17-0.el8.x86_64 21/26 2024-04-15T17:58:03.823 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:58:03.933 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:15.2.17-0.el8.x86_64 22/26 2024-04-15T17:58:04.135 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /run 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-04-15T17:58:19.785 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-04-15T17:58:19.786 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /dev 2024-04-15T17:58:19.786 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:20.896 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:58:20.983 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:58:20.983 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:58:21.197 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:58:21.198 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:15.2.17-0.el8.x86_64 1/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:15.2.17-0.el8.x86_64 2/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:15.2.17-0.el8.x86_64 3/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:58:21.932 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 8/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:15.2.17-0.el8.x86_64 9/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:15.2.17-0.el8.x86_64 11/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2024-04-15T17:58:21.933 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2024-04-15T17:58:21.934 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2024-04-15T17:58:21.934 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:21.934 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2024-04-15T17:58:21.934 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:15.2.17-0.el8.x86_64 25/26 2024-04-15T17:58:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:22.796 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-04-15T17:58:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:15.2.17-0.el8.x86_64 librbd1-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.796 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: fmt-6.2.1-1.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: leveldb-1.22-1.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:22.798 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:58:23.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2024-04-15T17:58:23.564 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:23.564 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-04-15T17:58:23.564 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /run 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2024-04-15T17:58:23.565 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:23.871 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:42 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:58:24.703 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:58:24.780 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:15.2.17-0.el8.x86_64 24/26 2024-04-15T17:58:24.780 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:58:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2024-04-15T17:58:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T17:58:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:15.2.17-0.el8 ceph 42 M 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout:Install 5 Packages 2024-04-15T17:58:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:25.492 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 43 M 2024-04-15T17:58:25.492 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 155 M 2024-04-15T17:58:25.492 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:58:25.738 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:25.738 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:15.2.17-0.el8.x86_64 1/26 2024-04-15T17:58:25.738 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:15.2.17-0.el8.x86_64 2/26 2024-04-15T17:58:25.738 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:15.2.17-0.el8.x86_64 3/26 2024-04-15T17:58:25.738 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:15.2.17-0.el8.x86_64 4/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:15.2.17-0.el8.x86_64 5/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:15.2.17-0.el8.x86_64 6/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:15.2.17-0.el8.x86_64 7/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:15.2.17-0.el8.x86_64 8/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:15.2.17-0.el8.x86_64 9/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:15.2.17-0.el8.x86_64 10/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:15.2.17-0.el8.x86_64 11/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:15.2.17-0.el8.x86_64 12/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:15.2.17-0.el8.x86_64 13/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2024-04-15T17:58:25.739 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 19/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:15.2.17-0.el8.x86_64 23/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2024-04-15T17:58:25.740 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:15.2.17-0.el8.x86_64 25/26 2024-04-15T17:58:25.774 INFO:teuthology.orchestra.run.smithi159.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 2.2 MB/s | 203 kB 00:00 2024-04-15T17:58:25.833 INFO:teuthology.orchestra.run.smithi159.stdout:(2/5): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-15T17:58:25.866 INFO:teuthology.orchestra.run.smithi159.stdout:(3/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 71 kB 00:00 2024-04-15T17:58:26.366 INFO:teuthology.orchestra.run.smithi159.stdout:(4/5): ceph-test-15.2.17-0.el8.x86_64.rpm 62 MB/s | 42 MB 00:00 2024-04-15T17:58:26.709 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:15.2.17-0.el8.x86_64 librbd1-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: fmt-6.2.1-1.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: leveldb-1.22-1.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.710 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:15.2.17-0.el8.x86_64 2024-04-15T17:58:26.711 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:26.712 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:58:26.783 INFO:teuthology.orchestra.run.smithi159.stdout:(5/5): oniguruma-6.8.2-3.el8.x86_64.rpm 171 kB/s | 188 kB 00:01 2024-04-15T17:58:26.784 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:58:26.785 INFO:teuthology.orchestra.run.smithi159.stdout:Total 33 MB/s | 43 MB 00:01 2024-04-15T17:58:26.826 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:58:26.845 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:58:26.846 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:58:27.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-04-15T17:58:27.097 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:58:27.101 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:58:27.516 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:58:27.756 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-15T17:58:27.766 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:45 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-15T17:58:28.085 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:28.146 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:29.506 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:58:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:29.507 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T17:58:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:29.507 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:15.2.17-0.el8 ceph 42 M 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2024-04-15T17:58:29.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 43 M 2024-04-15T17:58:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 155 M 2024-04-15T17:58:29.509 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:58:29.797 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): jq-1.6-8.el8.x86_64.rpm 2.4 MB/s | 203 kB 00:00 2024-04-15T17:58:29.822 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-15T17:58:29.847 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2024-04-15T17:58:29.881 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-15T17:58:30.389 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): ceph-test-15.2.17-0.el8.x86_64.rpm 62 MB/s | 42 MB 00:00 2024-04-15T17:58:30.390 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:58:30.391 INFO:teuthology.orchestra.run.smithi049.stdout:Total 49 MB/s | 43 MB 00:00 2024-04-15T17:58:30.435 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:58:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:58:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:58:30.720 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:58:30.723 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:58:31.151 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:58:31.359 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/5 2024-04-15T17:58:31.542 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/5 2024-04-15T17:58:31.713 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:31.778 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-15T17:58:32.333 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:15.2.17-0.el8.x86_64 1/5 2024-04-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-15T17:58:33.054 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-15T17:58:33.054 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:15.2.17-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:33.055 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:58:33.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2024-04-15T17:58:34.040 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:53 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:58:35.680 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:15.2.17-0.el8 ceph 6.3 k 2024-04-15T17:58:35.683 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:15.2.17-0.el8 ceph 2.0 M 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:15.2.17-0.el8 ceph 1.4 M 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:15.2.17-0.el8 ceph-noarch 193 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:15.2.17-0.el8 ceph 4.3 M 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:15.2.17-0.el8 ceph 17 M 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-15T17:58:35.684 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-15T17:58:35.685 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout:Install 32 Packages 2024-04-15T17:58:35.686 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:58:35.687 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 28 M 2024-04-15T17:58:35.687 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 91 M 2024-04-15T17:58:35.687 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:58:35.894 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-8.el8.x86_64 4/5 2024-04-15T17:58:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:58:36.425 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-15.2.17-0.el8.x86_64.rpm 121 kB/s | 6.3 kB 00:00 2024-04-15T17:58:36.475 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-mgr-15.2.17-0.el8.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-04-15T17:58:36.484 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:15.2.17-0.el8.x86_64 5/5 2024-04-15T17:58:36.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:15.2.17-0.el8.x86_64 1/5 2024-04-15T17:58:36.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-8.el8.x86_64 2/5 2024-04-15T17:58:36.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 3/5 2024-04-15T17:58:36.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/5 2024-04-15T17:58:36.526 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): ceph-mds-15.2.17-0.el8.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-04-15T17:58:36.559 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): ceph-mgr-modules-core-15.2.17-0.el8.noa 5.7 MB/s | 193 kB 00:00 2024-04-15T17:58:36.643 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): ceph-mon-15.2.17-0.el8.x86_64.rpm 20 MB/s | 4.3 MB 00:00 2024-04-15T17:58:36.693 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 777 kB/s | 39 kB 00:00 2024-04-15T17:58:36.718 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 988 kB/s | 157 kB 00:00 2024-04-15T17:58:36.752 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2024-04-15T17:58:36.769 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-04-15T17:58:36.836 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.3 MB/s | 457 kB 00:00 2024-04-15T17:58:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 5/5 2024-04-15T17:58:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:36.934 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:58:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:15.2.17-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-15T17:58:36.934 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.8.2-3.el8.x86_64 socat-1.7.4.1-1.el8.x86_64 2024-04-15T17:58:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-15T17:58:36.935 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:36.935 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:58:37.020 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): ceph-osd-15.2.17-0.el8.x86_64.rpm 31 MB/s | 17 MB 00:00 2024-04-15T17:58:37.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-04-15T17:58:37.967 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:55 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:58:39.711 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:15.2.17-0.el8 ceph 6.3 k 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:15.2.17-0.el8 ceph 2.0 M 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:15.2.17-0.el8 ceph 1.4 M 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:15.2.17-0.el8 ceph-noarch 193 k 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:15.2.17-0.el8 ceph 4.3 M 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:15.2.17-0.el8 ceph 17 M 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-15T17:58:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-15T17:58:39.716 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-15T17:58:39.717 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:Install 32 Packages 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 28 M 2024-04-15T17:58:39.718 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 91 M 2024-04-15T17:58:39.719 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:58:40.306 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-15.2.17-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2024-04-15T17:58:40.357 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-mgr-15.2.17-0.el8.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-04-15T17:58:40.398 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): ceph-mds-15.2.17-0.el8.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-04-15T17:58:40.432 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): ceph-mgr-modules-core-15.2.17-0.el8.noa 5.7 MB/s | 193 kB 00:00 2024-04-15T17:58:40.508 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): ceph-mon-15.2.17-0.el8.x86_64.rpm 22 MB/s | 4.3 MB 00:00 2024-04-15T17:58:40.572 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-04-15T17:58:40.616 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.3 MB/s | 103 kB 00:00 2024-04-15T17:58:40.633 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 309 kB/s | 39 kB 00:00 2024-04-15T17:58:40.650 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2024-04-15T17:58:40.701 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.7 MB/s | 457 kB 00:00 2024-04-15T17:58:40.901 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): ceph-osd-15.2.17-0.el8.x86_64.rpm 31 MB/s | 17 MB 00:00 2024-04-15T17:58:47.335 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 11 kB/s | 109 kB 00:10 2024-04-15T17:58:47.677 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 128 kB/s | 44 kB 00:00 2024-04-15T17:58:47.861 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2024-04-15T17:58:47.961 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2024-04-15T17:58:48.136 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2024-04-15T17:58:48.159 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 15 kB/s | 109 kB 00:07 2024-04-15T17:58:48.220 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-04-15T17:58:48.324 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 170 kB/s | 18 kB 00:00 2024-04-15T17:58:48.404 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 610 kB/s | 49 kB 00:00 2024-04-15T17:58:48.480 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 783 kB/s | 59 kB 00:00 2024-04-15T17:58:48.572 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-15T17:58:48.664 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-04-15T17:58:48.667 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2024-04-15T17:58:48.789 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 150 kB/s | 19 kB 00:00 2024-04-15T17:58:48.856 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 359 kB/s | 24 kB 00:00 2024-04-15T17:58:48.939 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 350 kB/s | 29 kB 00:00 2024-04-15T17:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 673 kB/s | 185 kB 00:00 2024-04-15T17:58:49.023 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 330 kB/s | 27 kB 00:00 2024-04-15T17:58:49.132 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-04-15T17:58:49.209 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 647 kB/s | 173 kB 00:00 2024-04-15T17:58:49.224 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-15T17:58:49.315 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 932 kB/s | 85 kB 00:00 2024-04-15T17:58:49.407 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-04-15T17:58:49.768 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 689 kB/s | 384 kB 00:00 2024-04-15T17:58:49.878 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-04-15T17:58:49.985 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-04-15T17:58:50.094 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 450 kB/s | 49 kB 00:00 2024-04-15T17:58:50.311 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 271 kB/s | 59 kB 00:00 2024-04-15T17:58:50.744 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 652 kB/s | 283 kB 00:00 2024-04-15T17:58:50.861 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 134 kB/s | 16 kB 00:00 2024-04-15T17:58:50.962 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-04-15T17:58:51.062 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 239 kB/s | 24 kB 00:00 2024-04-15T17:58:51.162 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2024-04-15T17:58:51.166 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 17 kB/s | 249 kB 00:14 2024-04-15T17:58:51.262 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 275 kB/s | 27 kB 00:00 2024-04-15T17:58:51.604 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 748 kB/s | 255 kB 00:00 2024-04-15T17:58:51.988 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 655 kB/s | 251 kB 00:00 2024-04-15T17:58:52.180 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 446 kB/s | 85 kB 00:00 2024-04-15T17:58:52.305 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-04-15T17:58:54.397 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 18 kB/s | 249 kB 00:13 2024-04-15T17:59:32.218 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): python3-cryptography-3.2.1-7.el8.x86_6 12 kB/s | 641 kB 00:51 2024-04-15T17:59:32.220 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:32.220 INFO:teuthology.orchestra.run.smithi049.stdout:Total 554 kB/s | 28 MB 00:52 2024-04-15T17:59:32.254 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): python3-cryptography-3.2.1-7.el8.x86_6 12 kB/s | 641 kB 00:55 2024-04-15T17:59:32.258 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:32.258 INFO:teuthology.orchestra.run.smithi159.stdout:Total 515 kB/s | 28 MB 00:56 2024-04-15T17:59:32.300 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:59:32.330 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:59:32.342 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:59:32.342 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:59:32.369 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:59:32.369 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:59:32.656 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:59:32.659 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:59:32.671 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:59:32.674 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:59:33.770 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:59:33.902 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:59:33.989 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-15T17:59:34.129 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-15T17:59:34.191 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-15T17:59:34.386 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-15T17:59:34.424 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-15T17:59:34.624 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-15T17:59:34.673 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-15T17:59:34.852 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-15T17:59:34.854 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-15T17:59:35.042 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-15T17:59:35.091 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-15T17:59:35.274 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-15T17:59:35.277 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-15T17:59:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-15T17:59:35.481 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-15T17:59:35.654 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-15T17:59:35.684 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-15T17:59:35.888 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-15T17:59:35.917 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-15T17:59:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-15T17:59:36.172 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-15T17:59:36.386 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-15T17:59:36.446 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-15T17:59:36.749 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-15T17:59:36.779 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-15T17:59:37.003 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-15T17:59:37.199 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-15T17:59:37.226 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-15T17:59:37.484 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-15T17:59:37.490 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-15T17:59:37.731 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-15T17:59:37.743 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:59:38.011 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-15T17:59:38.053 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-15T17:59:38.255 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:59:38.263 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-15T17:59:38.468 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-15T17:59:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-15T17:59:38.614 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-15T17:59:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-15T17:59:38.843 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-15T17:59:39.006 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-15T17:59:39.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-15T17:59:39.175 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-15T17:59:39.278 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-15T17:59:39.428 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-15T17:59:39.591 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-15T17:59:39.758 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-15T17:59:39.942 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-15T17:59:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-15T17:59:40.270 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 27/32 2024-04-15T17:59:40.409 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-15T17:59:40.459 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:59:40.737 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-15T17:59:41.104 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 27/32 2024-04-15T17:59:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:59:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:59:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:59:42.416 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:59:42.599 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:59:42.745 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:15.2.17-0.el8.x86_64 28/32 2024-04-15T17:59:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:59:42.927 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:59:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:59:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:59:43.148 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:59:43.342 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:15.2.17-0.el8.x86_64 29/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:15.2.17-0.el8.x86_64 1/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:15.2.17-0.el8.x86_64 2/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:15.2.17-0.el8.x86_64 3/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:15.2.17-0.el8.x86_64 4/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:15.2.17-0.el8.x86_64 5/32 2024-04-15T17:59:43.562 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 6/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:59:43.563 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-15T17:59:43.564 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-15T17:59:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:59:43.846 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:15.2.17-0.el8.x86_64 30/32 2024-04-15T17:59:44.102 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:59:44.120 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:15.2.17-0.el8.x86_64 31/32 2024-04-15T17:59:44.252 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:44.500 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-15T17:59:44.501 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-2.6-1.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-15T17:59:44.502 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-15T17:59:44.503 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-15T17:59:44.503 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-15T17:59:44.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:44.503 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:59:44.663 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:15.2.17-0.el8.x86_64 32/32 2024-04-15T17:59:44.663 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:15.2.17-0.el8.x86_64 1/32 2024-04-15T17:59:44.663 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:15.2.17-0.el8.x86_64 2/32 2024-04-15T17:59:44.663 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:15.2.17-0.el8.x86_64 3/32 2024-04-15T17:59:44.663 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:15.2.17-0.el8.x86_64 4/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:15.2.17-0.el8.x86_64 5/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 6/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-15T17:59:44.664 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-15T17:59:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-15T17:59:44.813 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:15.2.17-0.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-15T17:59:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-2.6-1.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-15T17:59:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-15T17:59:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:45.492 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:59:45.549 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:04 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:59:45.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-04-15T17:59:46.505 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:04 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:59:47.146 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:59:47.196 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:59:47.197 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T17:59:47.197 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:59:47.317 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2024-04-15T17:59:48.049 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:07 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:59:48.190 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:59:48.243 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:59:48.244 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T17:59:48.244 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:59:48.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-04-15T17:59:49.119 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:07 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:59:49.670 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:59:49.671 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:49.671 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T17:59:49.671 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:49.671 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T17:59:49.671 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:15.2.17-0.el8 ceph-noarch 59 k 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:49.672 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 59 k 2024-04-15T17:59:49.673 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 223 k 2024-04-15T17:59:49.673 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:59:49.718 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-15.2.17-0.el8.noarch.rpm 1.3 MB/s | 59 kB 00:00 2024-04-15T17:59:49.718 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:49.719 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.2 MB/s | 59 kB 00:00 2024-04-15T17:59:49.719 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:59:49.725 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:59:49.726 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:59:49.727 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:59:49.730 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:59:49.776 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:59:50.344 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:50.512 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:50.857 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:59:50.857 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:15.2.17-0.el8 ceph-noarch 59 k 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T17:59:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:50.859 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 59 k 2024-04-15T17:59:50.859 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 223 k 2024-04-15T17:59:50.859 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:59:50.907 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-15.2.17-0.el8.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-15T17:59:50.907 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:50.907 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.2 MB/s | 59 kB 00:00 2024-04-15T17:59:50.908 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:59:50.915 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:59:50.915 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:59:50.917 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:59:50.920 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:59:51.000 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:15.2.17-0.el8.noarch 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:51.125 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:59:51.393 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2024-04-15T17:59:51.502 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:51.701 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:51.938 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:52.136 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:11 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:59:52.367 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:15.2.17-0.el8.noarch 1/1 2024-04-15T17:59:52.367 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:52.368 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:59:52.368 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:15.2.17-0.el8.noarch 2024-04-15T17:59:52.368 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:52.368 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:59:52.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-04-15T17:59:53.374 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:11 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:59:53.771 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:15.2.17-0.el8 ceph 203 k 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T17:59:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:53.773 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 203 k 2024-04-15T17:59:53.773 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 599 k 2024-04-15T17:59:53.773 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T17:59:53.822 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-15.2.17-0.el8.x86_6 4.0 MB/s | 203 kB 00:00 2024-04-15T17:59:53.823 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:53.823 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.9 MB/s | 203 kB 00:00 2024-04-15T17:59:53.823 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T17:59:53.834 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T17:59:53.834 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T17:59:53.882 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T17:59:53.885 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T17:59:54.029 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T17:59:54.214 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:54.614 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:54.999 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:54.999 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:54.999 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T17:59:54.999 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:55.000 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T17:59:55.000 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:59:55.114 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:15.2.17-0.el8 ceph 203 k 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T17:59:55.115 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:55.116 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 203 k 2024-04-15T17:59:55.116 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 599 k 2024-04-15T17:59:55.116 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T17:59:55.170 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-15.2.17-0.el8.x86_6 3.7 MB/s | 203 kB 00:00 2024-04-15T17:59:55.171 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T17:59:55.171 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.6 MB/s | 203 kB 00:00 2024-04-15T17:59:55.171 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T17:59:55.183 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T17:59:55.183 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T17:59:55.231 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2024-04-15T17:59:55.235 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T17:59:55.238 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T17:59:55.356 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T17:59:55.600 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:55.977 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:14 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:59:56.037 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:56.509 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T17:59:56.509 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:56.509 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T17:59:56.509 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:15.2.17-0.el8.x86_64 2024-04-15T17:59:56.510 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T17:59:56.510 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:59:56.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-04-15T17:59:57.555 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:59:57.581 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:15 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T17:59:57.606 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T17:59:57.607 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T17:59:57.607 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T17:59:57.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2024-04-15T17:59:58.468 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:17 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T17:59:59.275 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T17:59:59.327 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T17:59:59.329 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T17:59:59.329 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T17:59:59.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-04-15T18:00:00.114 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:00.114 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:00.114 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:15.2.17-0.el8 ceph 572 k 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:00.115 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:00.116 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:00.116 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-04-15T18:00:00.116 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:00.117 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 656 k 2024-04-15T18:00:00.117 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.8 M 2024-04-15T18:00:00.117 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:00.231 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:18 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:00.416 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): ceph-fuse-15.2.17-0.el8.x86_64.rpm 9.4 MB/s | 572 kB 00:00 2024-04-15T18:00:02.020 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:15.2.17-0.el8 ceph 572 k 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:00:02.021 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:02.022 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-04-15T18:00:02.022 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:02.022 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 656 k 2024-04-15T18:00:02.022 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.8 M 2024-04-15T18:00:02.022 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:00:12.151 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 7.1 kB/s | 83 kB 00:11 2024-04-15T18:00:12.151 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:12.152 INFO:teuthology.orchestra.run.smithi159.stdout:Total 54 kB/s | 656 kB 00:12 2024-04-15T18:00:12.163 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:12.186 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:12.186 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:12.273 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:12.276 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:12.490 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:12.715 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-15T18:00:12.918 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T18:00:13.307 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T18:00:13.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:15.2.17-0.el8.x86_64 1/2 2024-04-15T18:00:14.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-15T18:00:14.307 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:14.307 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:15.2.17-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-15T18:00:14.307 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:14.811 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2024-04-15T18:00:15.556 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:34 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:17.193 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:17.195 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:17.195 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:17.195 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:17.196 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 140 k 2024-04-15T18:00:17.197 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 478 k 2024-04-15T18:00:17.197 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:17.240 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-15.2.17-0.el8.x86_64.rpm 3.1 MB/s | 140 kB 00:00 2024-04-15T18:00:17.240 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:17.241 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.0 MB/s | 140 kB 00:00 2024-04-15T18:00:17.241 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:17.246 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:17.246 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:17.293 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:17.296 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:17.403 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:18.271 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:18.271 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:18.271 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:18.271 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:18.271 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:18.272 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:18.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2024-04-15T18:00:19.243 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:38 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:19.823 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-fuse-15.2.17-0.el8.x86_64.rpm 10 MB/s | 572 kB 00:00 2024-04-15T18:00:20.812 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:20.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2024-04-15T18:00:21.725 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:40 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:23.358 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:15.2.17-0.el8 ceph 22 k 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 22 k 2024-04-15T18:00:23.359 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 78 k 2024-04-15T18:00:23.360 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:23.417 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-15.2.17-0.el8.x86_64.rpm 376 kB/s | 22 kB 00:00 2024-04-15T18:00:23.418 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:23.418 INFO:teuthology.orchestra.run.smithi159.stdout:Total 368 kB/s | 22 kB 00:00 2024-04-15T18:00:23.418 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:23.422 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:23.422 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:23.423 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:23.426 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:23.624 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:24.234 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:24.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:24.235 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:24.235 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:24.235 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:24.235 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:24.373 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2024-04-15T18:00:25.108 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:44 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:26.273 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 13 kB/s | 83 kB 00:06 2024-04-15T18:00:26.273 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:26.273 INFO:teuthology.orchestra.run.smithi049.stdout:Total 27 kB/s | 656 kB 00:24 2024-04-15T18:00:26.286 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:00:26.309 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:00:26.310 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:00:26.405 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:00:26.409 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:00:26.591 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:00:26.697 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:26.747 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:26.748 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:26.749 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:26.879 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2024-04-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-15T18:00:27.086 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T18:00:27.475 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:15.2.17-0.el8.x86_64 2/2 2024-04-15T18:00:27.476 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:15.2.17-0.el8.x86_64 1/2 2024-04-15T18:00:27.614 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:46 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:15.2.17-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:27.858 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:28.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-04-15T18:00:28.892 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:46 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:29.220 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:29.269 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:29.270 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:29.270 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:29.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2024-04-15T18:00:30.162 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:49 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:30.639 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:30.639 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T18:00:30.640 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:30.641 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 140 k 2024-04-15T18:00:30.641 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 478 k 2024-04-15T18:00:30.641 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:00:30.687 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-15.2.17-0.el8.x86_64.rpm 3.1 MB/s | 140 kB 00:00 2024-04-15T18:00:30.687 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:30.687 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.1 MB/s | 140 kB 00:00 2024-04-15T18:00:30.687 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:00:30.691 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:00:30.691 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:00:30.742 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:00:30.745 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:00:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:00:31.034 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:31.313 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:31.725 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:31.737 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:31.786 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:31.787 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:31.787 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:31.914 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2024-04-15T18:00:32.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-04-15T18:00:32.641 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:51 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:32.806 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:50 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:34.203 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:34.251 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:34.252 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:34.252 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:34.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2024-04-15T18:00:34.513 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:34.568 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:34.569 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:34.569 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:34.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-04-15T18:00:35.118 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:54 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:35.465 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:53 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:36.694 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:36.745 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:36.746 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:36.746 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:36.869 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2024-04-15T18:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:15.2.17-0.el8 ceph 22 k 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 22 k 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 78 k 2024-04-15T18:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:00:37.270 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-15.2.17-0.el8.x86_64.rpm 436 kB/s | 22 kB 00:00 2024-04-15T18:00:37.271 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:37.271 INFO:teuthology.orchestra.run.smithi049.stdout:Total 426 kB/s | 22 kB 00:00 2024-04-15T18:00:37.271 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:00:37.275 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:00:37.275 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:00:37.277 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:00:37.280 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:00:37.333 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:00:37.483 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:37.599 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:56 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:37.748 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:38.130 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:38.130 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:38.130 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:00:38.131 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:38.131 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:38.131 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:38.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-04-15T18:00:39.033 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:57 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:39.194 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:39.244 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:39.245 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-15T18:00:39.245 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:39.378 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2024-04-15T18:00:40.110 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:59 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:40.801 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:40.802 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:40.802 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:40.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-04-15T18:00:41.711 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:59 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:41.725 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:41.726 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:41.726 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:41.726 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:41.726 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:15.2.17-0.el8 ceph 73 k 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:41.727 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 73 k 2024-04-15T18:00:41.728 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 188 k 2024-04-15T18:00:41.728 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:41.777 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-15.2.17-0.el8.x86_64.rpm 1.4 MB/s | 73 kB 00:00 2024-04-15T18:00:41.778 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:41.778 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.4 MB/s | 73 kB 00:00 2024-04-15T18:00:41.778 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:41.787 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:41.787 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:41.834 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:41.838 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:41.932 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:42.129 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:42.406 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:42.836 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:43.085 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2024-04-15T18:00:43.398 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:43.450 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:43.451 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:43.451 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:43.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-04-15T18:00:43.817 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:02 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:44.342 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:02 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:45.454 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:45.454 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:45.454 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:15.2.17-0.el8 ceph 2.7 M 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T18:00:45.455 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:45.456 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.7 M 2024-04-15T18:00:45.456 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 9.8 M 2024-04-15T18:00:45.456 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:45.546 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-15.2.17-0.el8.x86_64.rpm 30 MB/s | 2.7 MB 00:00 2024-04-15T18:00:45.547 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:45.547 INFO:teuthology.orchestra.run.smithi159.stdout:Total 30 MB/s | 2.7 MB 00:00 2024-04-15T18:00:45.548 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:45.558 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:45.559 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:45.617 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:45.620 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:46.029 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:46.082 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:46.083 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:46.083 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:46.186 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:46.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-04-15T18:00:46.423 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:46.830 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:46.952 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:04 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:47.227 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:47.227 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:47.227 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:47.227 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:47.228 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:47.228 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:47.447 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2024-04-15T18:00:48.188 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:07 ago on Mon 15 Apr 2024 05:57:41 PM UTC. 2024-04-15T18:00:48.648 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:48.701 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:48.702 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:48.702 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:48.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-04-15T18:00:49.593 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:07 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:49.843 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-15T18:00:49.844 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:49.844 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:49.844 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:49.844 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-15T18:00:49.844 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 140 k 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 418 k 2024-04-15T18:00:49.845 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-15T18:00:49.891 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-15.2.17-0.el8.x86_64.rpm 3.0 MB/s | 140 kB 00:00 2024-04-15T18:00:49.892 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:49.892 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.9 MB/s | 140 kB 00:00 2024-04-15T18:00:49.892 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-15T18:00:49.901 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-15T18:00:49.902 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-15T18:00:49.949 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-15T18:00:49.953 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-15T18:00:50.065 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-15T18:00:50.295 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:50.574 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:00:50.995 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-15T18:00:51.204 DEBUG:teuthology.parallel:result is None 2024-04-15T18:00:51.305 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:51.358 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:51.359 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:51.359 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:51.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-04-15T18:00:52.249 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:10 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:53.939 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:15.2.17-0.el8.x86_64 is already installed. 2024-04-15T18:00:53.992 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:53.993 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-15T18:00:53.993 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:54.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-04-15T18:00:54.874 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:12 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:00:56.638 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:00:56.638 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:56.638 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:00:56.638 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:15.2.17-0.el8 ceph 73 k 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T18:00:56.639 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:56.640 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 73 k 2024-04-15T18:00:56.640 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 188 k 2024-04-15T18:00:56.640 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:00:56.688 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-15.2.17-0.el8.x86_64.rpm 1.5 MB/s | 73 kB 00:00 2024-04-15T18:00:56.689 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:00:56.689 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.4 MB/s | 73 kB 00:00 2024-04-15T18:00:56.689 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:00:56.699 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:00:56.699 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:00:56.750 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:00:56.753 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:00:56.858 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:00:57.113 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:57.782 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:00:57.783 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:57.783 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:00:57.783 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:15.2.17-0.el8.x86_64 2024-04-15T18:00:57.783 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:00:57.783 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:00:58.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-04-15T18:00:58.756 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:16 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:01:00.505 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:15.2.17-0.el8 ceph 2.7 M 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T18:01:00.506 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:00.507 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.7 M 2024-04-15T18:01:00.507 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 9.8 M 2024-04-15T18:01:00.507 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:01:00.591 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-15.2.17-0.el8.x86_64.rpm 33 MB/s | 2.7 MB 00:00 2024-04-15T18:01:00.591 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:01:00.592 INFO:teuthology.orchestra.run.smithi049.stdout:Total 32 MB/s | 2.7 MB 00:00 2024-04-15T18:01:00.592 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:01:00.604 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:01:00.604 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:01:00.665 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:01:00.668 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:01:01.068 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:01:01.472 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:01.909 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:02.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:02.307 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:02.308 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:01:02.308 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:15.2.17-0.el8.x86_64 2024-04-15T18:01:02.308 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:02.308 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:01:02.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-04-15T18:01:03.351 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:21 ago on Mon 15 Apr 2024 05:57:42 PM UTC. 2024-04-15T18:01:05.105 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-15T18:01:05.106 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:05.106 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-15T18:01:05.106 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:05.106 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-15T18:01:05.106 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:15.2.17-0.el8 ceph 140 k 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 140 k 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 418 k 2024-04-15T18:01:05.107 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-15T18:01:05.152 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-15.2.17-0.el8.x86_64.rpm 3.0 MB/s | 140 kB 00:00 2024-04-15T18:01:05.153 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:01:05.153 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.0 MB/s | 140 kB 00:00 2024-04-15T18:01:05.153 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-15T18:01:05.164 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-15T18:01:05.164 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-15T18:01:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-15T18:01:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-15T18:01:05.338 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-15T18:01:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:05.838 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:15.2.17-0.el8.x86_64 1/1 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:15.2.17-0.el8.x86_64 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:01:06.230 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-15T18:01:06.472 DEBUG:teuthology.parallel:result is None 2024-04-15T18:01:06.473 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T18:01:06.473 INFO:teuthology.packaging:ref: None 2024-04-15T18:01:06.473 INFO:teuthology.packaging:tag: None 2024-04-15T18:01:06.473 INFO:teuthology.packaging:branch: octopus 2024-04-15T18:01:06.473 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:06.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T18:01:06.774 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T18:01:06.799 INFO:teuthology.orchestra.run.smithi049.stdout:15.2.17-0.el8 2024-04-15T18:01:06.800 INFO:teuthology.packaging:The installed version of ceph is 15.2.17-0.el8 2024-04-15T18:01:06.800 INFO:teuthology.task.install:The correct ceph version 15.2.17-0 is installed. 2024-04-15T18:01:06.802 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T18:01:06.802 INFO:teuthology.packaging:ref: None 2024-04-15T18:01:06.802 INFO:teuthology.packaging:tag: None 2024-04-15T18:01:06.802 INFO:teuthology.packaging:branch: octopus 2024-04-15T18:01:06.802 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:06.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T18:01:07.026 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T18:01:07.056 INFO:teuthology.orchestra.run.smithi082.stdout:15.2.17-0.el8 2024-04-15T18:01:07.056 INFO:teuthology.packaging:The installed version of ceph is 15.2.17-0.el8 2024-04-15T18:01:07.056 INFO:teuthology.task.install:The correct ceph version 15.2.17-0 is installed. 2024-04-15T18:01:07.058 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T18:01:07.058 INFO:teuthology.packaging:ref: None 2024-04-15T18:01:07.058 INFO:teuthology.packaging:tag: None 2024-04-15T18:01:07.058 INFO:teuthology.packaging:branch: octopus 2024-04-15T18:01:07.058 INFO:teuthology.packaging:sha1: c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:07.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2024-04-15T18:01:07.270 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T18:01:07.297 INFO:teuthology.orchestra.run.smithi159.stdout:15.2.17-0.el8 2024-04-15T18:01:07.298 INFO:teuthology.packaging:The installed version of ceph is 15.2.17-0.el8 2024-04-15T18:01:07.298 INFO:teuthology.task.install:The correct ceph version 15.2.17-0 is installed. 2024-04-15T18:01:07.300 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T18:01:07.300 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:01:07.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T18:01:07.337 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:01:07.338 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T18:01:07.372 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:01:07.372 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T18:01:07.406 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T18:01:07.406 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:01:07.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T18:01:07.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T18:01:07.509 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:01:07.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T18:01:07.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T18:01:07.614 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:01:07.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T18:01:07.646 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T18:01:07.717 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T18:01:07.717 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:01:07.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T18:01:07.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T18:01:07.821 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:01:07.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T18:01:07.852 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T18:01:07.922 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:01:07.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T18:01:07.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T18:01:08.026 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T18:01:08.026 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:01:08.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T18:01:08.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T18:01:08.128 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:01:08.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T18:01:08.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T18:01:08.231 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:01:08.232 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T18:01:08.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T18:01:08.333 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-15T18:01:08.437 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54'} 2024-04-15T18:01:08.437 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:08.437 INFO:tasks.cephadm:Cluster fsid is 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:01:08.437 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-15T18:01:08.438 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.49', 'mon.b': '172.21.15.82', 'mon.c': '172.21.15.159'} 2024-04-15T18:01:08.438 INFO:tasks.cephadm:First mon is mon.a on smithi049 2024-04-15T18:01:08.438 INFO:tasks.cephadm:First mgr is y 2024-04-15T18:01:08.438 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-15T18:01:08.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2024-04-15T18:01:08.473 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-04-15T18:01:08.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo hostname $(hostname -s) 2024-04-15T18:01:08.536 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-15T18:01:08.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:08.776 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/54888/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3080-gc2d8c315', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3080.gc2d8c315'}, 'url': 'https://4.chacra.ceph.com/r/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-13 03:30:48.142221', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-15T18:01:08.919 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref main, sha1 c2d8c3155d854ee204cb7c7261232571229d1d54 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=c2d8c3155d854ee204cb7c7261232571229d1d54 2024-04-15T18:01:08.921 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/x86_64/flavors/default/cephadm 2024-04-15T18:01:08.921 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 15 18:01 /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:09.402 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:10.119 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 15 18:01 /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:10.120 DEBUG:teuthology.orchestra.run.smithi159:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main/c2d8c3155d854ee204cb7c7261232571229d1d54/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:11.047 INFO:teuthology.orchestra.run.smithi159.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2294149 Apr 15 18:01 /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:11.048 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:11.071 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:11.094 DEBUG:teuthology.orchestra.run.smithi159:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T18:01:11.130 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 on all hosts... 2024-04-15T18:01:11.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 pull 2024-04-15T18:01:11.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 pull 2024-04-15T18:01:11.137 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 pull 2024-04-15T18:01:12.094 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54... 2024-04-15T18:01:12.339 INFO:teuthology.orchestra.run.smithi159.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54... 2024-04-15T18:01:12.497 INFO:teuthology.orchestra.run.smithi049.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54... 2024-04-15T18:01:52.728 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:01:52.728 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev)", 2024-04-15T18:01:52.728 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "259793512b732efd8e04a33aa28c6de3675c47d120bc226206521425d246003e", 2024-04-15T18:01:52.728 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-04-15T18:01:52.729 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:55072b3f7eae2e3409906b98d09d882e1c3dc2a9205d85fd5f4c0bc62f82d163" 2024-04-15T18:01:52.729 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:01:52.729 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout: "ceph_version": "ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev)", 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout: "image_id": "259793512b732efd8e04a33aa28c6de3675c47d120bc226206521425d246003e", 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout: "repo_digests": [ 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:55072b3f7eae2e3409906b98d09d882e1c3dc2a9205d85fd5f4c0bc62f82d163" 2024-04-15T18:01:55.049 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:01:55.050 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_version": "ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev)", 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout: "image_id": "259793512b732efd8e04a33aa28c6de3675c47d120bc226206521425d246003e", 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout: "repo_digests": [ 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:55072b3f7eae2e3409906b98d09d882e1c3dc2a9205d85fd5f4c0bc62f82d163" 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:02:01.040 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:02:01.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2024-04-15T18:02:01.099 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-04-15T18:02:01.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph 2024-04-15T18:02:01.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2024-04-15T18:02:01.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-04-15T18:02:01.231 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /etc/ceph 2024-04-15T18:02:01.264 INFO:tasks.cephadm:Writing seed config... 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-04-15T18:02:01.265 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-15T18:02:01.266 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-04-15T18:02:01.267 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-04-15T18:02:01.268 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-15T18:02:01.268 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-15T18:02:01.268 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-15T18:02:01.268 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:02:01.268 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-15T18:02:01.287 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 22c655ba-fb52-11ee-bc8f-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-15T18:02:01.288 DEBUG:teuthology.orchestra.run.smithi049:mon.a> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service 2024-04-15T18:02:01.330 DEBUG:teuthology.orchestra.run.smithi049:mgr.y> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y.service 2024-04-15T18:02:01.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:02:01.358 INFO:tasks.cephadm:Bootstrapping... 2024-04-15T18:02:01.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 -v bootstrap --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:02:01.384 INFO:journalctl@ceph.mgr.y.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:02:01.570 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-15T18:02:01.570 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54', '-v', 'bootstrap', '--fsid', '22c655ba-fb52-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.49', '--skip-admin-label'] 2024-04-15T18:02:01.571 INFO:teuthology.orchestra.run.smithi049.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-15T18:02:01.571 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying podman|docker is present... 2024-04-15T18:02:01.603 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-15T18:02:01.604 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying lvm2 is present... 2024-04-15T18:02:01.604 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying time synchronization is in place... 2024-04-15T18:02:01.611 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-15T18:02:01.611 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T18:02:01.617 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-15T18:02:01.617 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:02:01.623 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-04-15T18:02:01.629 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-04-15T18:02:01.629 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-04-15T18:02:01.629 INFO:teuthology.orchestra.run.smithi049.stdout:Repeating the final host check... 2024-04-15T18:02:01.663 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-15T18:02:01.664 INFO:teuthology.orchestra.run.smithi049.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-15T18:02:01.664 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl is present 2024-04-15T18:02:01.664 INFO:teuthology.orchestra.run.smithi049.stdout:lvcreate is present 2024-04-15T18:02:01.670 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-15T18:02:01.670 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T18:02:01.676 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-15T18:02:01.676 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:02:01.682 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:Host looks OK 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:Cluster fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:Acquiring lock 139723343725120 on /run/cephadm/22c655ba-fb52-11ee-bc8f-c7b262605968.lock 2024-04-15T18:02:01.688 INFO:teuthology.orchestra.run.smithi049.stdout:Lock 139723343725120 acquired on /run/cephadm/22c655ba-fb52-11ee-bc8f-c7b262605968.lock 2024-04-15T18:02:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 3300 ... 2024-04-15T18:02:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 6789 ... 2024-04-15T18:02:01.689 INFO:teuthology.orchestra.run.smithi049.stdout:Base mon IP(s) is [172.21.15.49:3300, 172.21.15.49:6789], mon addrv is [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-04-15T18:02:01.693 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.49 metric 100 2024-04-15T18:02:01.693 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.49 metric 100 2024-04-15T18:02:01.696 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-15T18:02:01.696 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-15T18:02:01.696 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1779sec hoplimit 64 pref medium 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1566/64 scope link 2024-04-15T18:02:01.699 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-15T18:02:01.700 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-04-15T18:02:01.700 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-04-15T18:02:01.700 INFO:teuthology.orchestra.run.smithi049.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-15T18:02:01.700 INFO:teuthology.orchestra.run.smithi049.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-15T18:02:01.701 INFO:teuthology.orchestra.run.smithi049.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54... 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 259793512b732efd8e04a33aa28c6de3675c47d120bc226206521425d246003e 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54... 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Getting image source signatures 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:9e451f54663b179c1b9723e4792c62bd6dec2cebe37d408f5d2cf1c551fbf8b7 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying config sha256:259793512b732efd8e04a33aa28c6de3675c47d120bc226206521425d246003e 2024-04-15T18:02:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-15T18:02:03.314 INFO:teuthology.orchestra.run.smithi049.stdout:ceph: stdout ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev) 2024-04-15T18:02:03.314 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph version: ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev) 2024-04-15T18:02:03.314 INFO:teuthology.orchestra.run.smithi049.stdout:Extracting ceph user uid/gid from container image... 2024-04-15T18:02:04.850 INFO:teuthology.orchestra.run.smithi049.stdout:stat: stdout 167 167 2024-04-15T18:02:04.850 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial keys... 2024-04-15T18:02:06.001 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCdax1mS70wFhAAgwS6Jus2gJu6alBHnAKifQ== 2024-04-15T18:02:07.036 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCeax1m8IPhGBAAgCFVdEb918LncR12cKNoSg== 2024-04-15T18:02:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQCfax1myUt6GxAAHf2GZirM0LvvWpBFVnkLTA== 2024-04-15T18:02:08.163 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial monmap... 2024-04-15T18:02:09.259 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T18:02:09.259 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-15T18:02:09.259 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:09.259 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T18:02:09.259 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool for a [v2:172.21.15.49:3300,v1:172.21.15.49:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T18:02:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-04-15T18:02:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: set fsid to 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T18:02:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:02:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mon... 2024-04-15T18:02:10.362 INFO:teuthology.orchestra.run.smithi049.stdout:create mon.a on 2024-04-15T18:02:10.482 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-15T18:02:10.577 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-15T18:02:10.700 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target. 2024-04-15T18:02:10.700 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target. 2024-04-15T18:02:10.875 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a 2024-04-15T18:02:10.876 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service: Unit ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service not loaded. 2024-04-15T18:02:10.974 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target.wants/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service → /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@.service. 2024-04-15T18:02:11.618 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T18:02:11.618 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-04-15T18:02:11.627 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T18:02:11.627 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:02:11.627 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-04-15T18:02:11.627 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-04-15T18:02:11.627 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon to start... 2024-04-15T18:02:11.628 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon... 2024-04-15T18:02:11.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:11 smithi049 bash[39559]: c8e086cb33731909961ea11c46d2f16c6ffa3ae7cb488965a185045e2f8f09dc 2024-04-15T18:02:11.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:11 smithi049 systemd[1]: Started Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:02:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout cluster: 2024-04-15T18:02:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout id: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-15T18:02:12.764 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout services: 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.630328s) 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout data: 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pgs: 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:mon is available 2024-04-15T18:02:12.765 INFO:teuthology.orchestra.run.smithi049.stdout:Assimilating anything we can from ceph.conf... 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: monmap epoch 1 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: last_changed 2024-04-15T18:02:08.664609+0000 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: min_mon_release 19 (squid) 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: election_strategy: 1 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:02:12.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: fsmap 2024-04-15T18:02:12.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T18:02:12.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: mgrmap e1: no daemons active 2024-04-15T18:02:12.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:12 smithi049 ceph-mon[39596]: from='client.? 172.21.15.49:0/2102724535' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-15T18:02:13.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:13 smithi049 ceph-mon[39596]: from='client.? 172.21.15.49:0/1320459134' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T18:02:13.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:13 smithi049 ceph-mon[39596]: from='client.? 172.21.15.49:0/1320459134' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T18:02:14.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-15T18:02:14.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-15T18:02:14.122 INFO:teuthology.orchestra.run.smithi049.stdout:Generating new minimal ceph.conf... 2024-04-15T18:02:15.183 INFO:teuthology.orchestra.run.smithi049.stdout:Restarting the monitor... 2024-04-15T18:02:15.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:14 smithi049 ceph-mon[39596]: from='client.? 172.21.15.49:0/3858671929' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:02:15.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:15 smithi049 systemd[1]: Stopping Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:02:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:15 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[39573]: 2024-04-15T18:02:15.437+0000 7feb9934e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:02:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:15 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[39573]: 2024-04-15T18:02:15.437+0000 7feb9934e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-15T18:02:16.177 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:15 smithi049 bash[39925]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a 2024-04-15T18:02:16.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 systemd[1]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service: Succeeded. 2024-04-15T18:02:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 systemd[1]: Stopped Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:02:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 systemd[1]: Starting Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:02:16.882 INFO:teuthology.orchestra.run.smithi049.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-15T18:02:17.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev), process ceph-mon, pid 7 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: pidfile_write: ignore empty --pid-file 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: load: jerasure load: lrc 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: RocksDB version: 7.9.2 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Git sha 0 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Compile date 2024-04-12 22:42:42 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: DB SUMMARY 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: DB Session ID: HNKK6E9CNZVYNB7UVZ8D 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: CURRENT file: CURRENT 2024-04-15T18:02:17.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102618 ; 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.error_if_exists: 0 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.create_if_missing: 0 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.paranoid_checks: 1 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.env: 0x55882b03fcc0 2024-04-15T18:02:17.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.fs: PosixFileSystem 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.info_log: 0x55882c431be0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.statistics: (nil) 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.use_fsync: 0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_log_file_size: 0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_fallocate: 1 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T18:02:17.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.use_direct_reads: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.db_log_dir: 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.wal_dir: 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T18:02:17.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.write_buffer_manager: 0x55882c4021e0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.unordered_write: 0 2024-04-15T18:02:17.103 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.row_cache: None 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.wal_filter: None 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.two_write_queues: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.wal_compression: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.atomic_flush: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T18:02:17.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.log_readahead_size: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_background_jobs: 2 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_background_compactions: -1 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_subcompactions: 1 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T18:02:17.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_open_files: -1 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_background_flushes: -1 2024-04-15T18:02:17.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Compression algorithms supported: 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kZSTD supported: 0 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kXpressCompression supported: 0 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kZlibCompression supported: 1 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kSnappyCompression supported: 1 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kLZ4Compression supported: 1 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: kBZip2Compression supported: 0 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T18:02:17.107 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.merge_operator: 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_filter: None 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55882c431dc0) 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks: 1 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_top_level_index_and_filter: 1 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_type: 0 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_index_type: 0 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_shortening: 1 2024-04-15T18:02:17.108 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: checksum: 4 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: no_block_cache: 0 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache: 0x55882c435090 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_name: BinnedLRUCache 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_options: 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: capacity : 536870912 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_shard_bits : 4 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: strict_capacity_limit : 0 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: high_pri_pool_ratio: 0.000 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_compressed: (nil) 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: persistent_cache: (nil) 2024-04-15T18:02:17.109 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size: 4096 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size_deviation: 10 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_restart_interval: 16 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_block_restart_interval: 1 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: metadata_block_size: 4096 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: partition_filters: 0 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: use_delta_encoding: 1 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: filter_policy: bloomfilter 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: whole_key_filtering: 1 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: verify_compression: 0 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: read_amp_bytes_per_bit: 0 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: format_version: 5 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_index_compression: 1 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_align: 0 2024-04-15T18:02:17.110 INFO:journalctl@ceph.mon.a.smithi049.stdout: max_auto_readahead_size: 262144 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout: prepopulate_block_cache: 0 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout: initial_auto_readahead_size: 8192 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression: NoCompression 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.num_levels: 7 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T18:02:17.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T18:02:17.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T18:02:17.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.arena_block_size: 1048576 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T18:02:17.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T18:02:17.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.inplace_update_support: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.bloom_locality: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.max_successive_merges: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.ttl: 2592000 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-15T18:02:17.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enable_blob_files: false 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.min_blob_size: 0 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.blob_file_starting_level: 0 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 50b2878d-0c29-4bf0-8e33-0dcda8cdc739 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204136822991, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-15T18:02:17.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204136824069, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97749, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 312, "table_properties": {"data_size": 95725, "index_size": 280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13648, "raw_average_key_size": 50, "raw_value_size": 88405, "raw_average_value_size": 326, "num_data_blocks": 12, "num_entries": 271, "num_filter_entries": 271, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713204136, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "50b2878d-0c29-4bf0-8e33-0dcda8cdc739", "db_session_id": "HNKK6E9CNZVYNB7UVZ8D", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204136824152, "job": 1, "event": "recovery_finished"} 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55882c524000 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: DB pointer 0x55882c50e000 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** DB Stats ** 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T18:02:17.118 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: L0 2/0 97.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: Sum 2/0 97.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.119 INFO:journalctl@ceph.mon.a.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative compaction: 0.00 GB write, 28.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval compaction: 0.00 GB write, 28.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache BinnedLRUCache@0x55882c435090#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: starting mon.a rank 0 at public addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] at bind addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:17.120 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???) e1 preinit fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).mds e1 new map 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).mds e1 print_map 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: e1 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: legacy client fscid: -1 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout: No filesystems configured 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 bash[40049]: 30fb740b2fdd8095a94725fdc5a1fd86885759bb4021444a50fdc88068eb3e8e 2024-04-15T18:02:17.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 systemd[1]: Started Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: monmap epoch 1 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: last_changed 2024-04-15T18:02:08.664609+0000 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: min_mon_release 19 (squid) 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: election_strategy: 1 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: fsmap 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T18:02:17.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:16 smithi049 ceph-mon[40087]: mgrmap e1: no daemons active 2024-04-15T18:02:17.979 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-15T18:02:18.009 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:02:18.009 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mgr... 2024-04-15T18:02:18.010 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-15T18:02:18.010 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-15T18:02:18.220 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y 2024-04-15T18:02:18.220 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y.service: Unit ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y.service not loaded. 2024-04-15T18:02:18.320 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968.target.wants/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@.service. 2024-04-15T18:02:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3652360568' entity='client.admin' 2024-04-15T18:02:18.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:18 smithi049 systemd[1]: Starting Ceph mgr.y for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:02:19.002 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T18:02:19.002 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-04-15T18:02:19.010 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T18:02:19.010 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:02:19.010 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-04-15T18:02:19.010 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-04-15T18:02:19.018 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T18:02:19.018 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-04-15T18:02:19.027 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T18:02:19.027 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:02:19.027 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-04-15T18:02:19.027 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-15T18:02:19.027 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr to start... 2024-04-15T18:02:19.028 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr... 2024-04-15T18:02:19.262 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:18 smithi049 bash[40325]: 9b4f5ba19cdf60480bc5b148b40626387af07acaa7c2f68e8e7c0942b82b809a 2024-04-15T18:02:19.263 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:18 smithi049 systemd[1]: Started Ceph mgr.y for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:02:19.263 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:19 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:19.074+0000 7f8381953200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T18:02:19.263 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:19 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:19.259+0000 7f8381953200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T18:02:19.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3956394329' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T18:02:19.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:19 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:19.609+0000 7f8381953200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T18:02:20.421 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:20 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:20.156+0000 7f8381953200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T18:02:20.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:20 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:20.417+0000 7f8381953200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T18:02:21.227 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:20 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:20.870+0000 7f8381953200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T18:02:21.889 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:21.896 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-15T18:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-15T18:02:21.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-15T18:02:21.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-15T18:02:21.898 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-15T18:02:21.898 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:21 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:21.623+0000 7f8381953200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T18:02:21.898 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:21 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:21.713+0000 7f8381953200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T18:02:21.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-15T18:02:21.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-15T18:02:21.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-15T18:02:21.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-15T18:02:21.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-04-15T18:02:11.589609+0000", 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:02:21.903 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (1/15)... 2024-04-15T18:02:22.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.343+0000 7f8381953200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T18:02:22.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.431+0000 7f8381953200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T18:02:22.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.523+0000 7f8381953200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T18:02:23.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.774+0000 7f8381953200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T18:02:23.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.870+0000 7f8381953200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T18:02:23.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:22 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:22.967+0000 7f8381953200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T18:02:23.476 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:23 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:23.212+0000 7f8381953200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T18:02:23.477 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:23 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:23.312+0000 7f8381953200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T18:02:23.770 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:23 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:23.506+0000 7f8381953200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T18:02:24.226 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:23 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:23.767+0000 7f8381953200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T18:02:24.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:24 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:24.381+0000 7f8381953200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T18:02:24.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:24 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:24.600+0000 7f8381953200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T18:02:25.226 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:24 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:24.803+0000 7f8381953200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:02:25.898 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:25 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:25.499+0000 7f8381953200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T18:02:25.898 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:25 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:25.706+0000 7f8381953200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:02:26.227 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:25 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:25.895+0000 7f8381953200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T18:02:26.227 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:26.015+0000 7f8381953200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T18:02:26.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: Activating manager daemon y 2024-04-15T18:02:26.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: mgrmap e2: y(active, starting, since 0.00300323s) 2024-04-15T18:02:26.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T18:02:26.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T18:02:26.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T18:02:26.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:02:26.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T18:02:26.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:26 smithi049 ceph-mon[40087]: Manager daemon y is now available 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: from='mgr.14100 172.21.15.49:0/3861259536' entity='mgr.y' 2024-04-15T18:02:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:27 smithi049 ceph-mon[40087]: mgrmap e3: y(active, since 1.00708s) 2024-04-15T18:02:28.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2836621642' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T18:02:29.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:29 smithi049 ceph-mon[40087]: mgrmap e4: y(active, since 2s) 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-15T18:02:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-15T18:02:31.018 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-15T18:02:31.019 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-15T18:02:31.022 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-15T18:02:31.022 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-15T18:02:31.022 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-15T18:02:31.023 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-04-15T18:02:11.589609+0000", 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-15T18:02:31.024 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-04-15T18:02:31.025 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-15T18:02:31.025 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:02:31.025 INFO:teuthology.orchestra.run.smithi049.stdout:mgr is available 2024-04-15T18:02:32.284 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3940924017' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T18:02:32.496 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-15T18:02:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-15T18:02:32.498 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling cephadm module... 2024-04-15T18:02:33.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/42423742' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-15T18:02:33.892 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:33 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: ignoring --setuser ceph since I am not root 2024-04-15T18:02:33.892 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:33 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: ignoring --setgroup ceph since I am not root 2024-04-15T18:02:33.892 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:33 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:33.720+0000 7f7806038200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T18:02:34.221 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:33 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:33.904+0000 7f7806038200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T18:02:34.476 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:34 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:34.244+0000 7f7806038200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T18:02:34.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/42423742' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-15T18:02:34.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:34 smithi049 ceph-mon[40087]: mgrmap e5: y(active, since 7s) 2024-04-15T18:02:35.027 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:34 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:34.765+0000 7f7806038200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T18:02:35.421 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:35 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:35.023+0000 7f7806038200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T18:02:35.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:35 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:35.483+0000 7f7806038200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T18:02:35.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2826705748' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-04-15T18:02:35.944 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 5... 2024-04-15T18:02:36.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:36.251+0000 7f7806038200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T18:02:36.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:36.343+0000 7f7806038200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T18:02:37.226 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:36.970+0000 7f7806038200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T18:02:37.227 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.058+0000 7f7806038200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T18:02:37.227 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.149+0000 7f7806038200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T18:02:37.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.386+0000 7f7806038200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T18:02:37.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.481+0000 7f7806038200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T18:02:37.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.578+0000 7f7806038200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T18:02:38.115 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.820+0000 7f7806038200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T18:02:38.116 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:37 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:37.919+0000 7f7806038200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T18:02:38.116 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:38.112+0000 7f7806038200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T18:02:38.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:38.374+0000 7f7806038200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T18:02:39.411 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:38.985+0000 7f7806038200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T18:02:39.411 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:39 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:39.204+0000 7f7806038200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T18:02:40.476 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:39 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:39.408+0000 7f7806038200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:02:40.477 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:40.099+0000 7f7806038200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T18:02:40.477 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:40.306+0000 7f7806038200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:40.496+0000 7f7806038200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:02:40.616+0000 7f7806038200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: Active manager daemon y restarted 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: Activating manager daemon y 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: osdmap e2: 0 total, 0 up, 0 in 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: mgrmap e6: y(active, starting, since 0.00326454s) 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T18:02:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T18:02:40.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T18:02:40.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T18:02:40.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: Manager daemon y is now available 2024-04-15T18:02:40.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:40.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:40 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: Found migration_current of "None". Setting to last migration. 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:41.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:41 smithi049 ceph-mon[40087]: mgrmap e7: y(active, since 1.0052s) 2024-04-15T18:02:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:02:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-15T18:02:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-15T18:02:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:02:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 5 is available 2024-04-15T18:02:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:Setting orchestrator backend to cephadm... 2024-04-15T18:02:42.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:42 smithi049 ceph-mon[40087]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T18:02:42.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:42 smithi049 ceph-mon[40087]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T18:02:43.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: mgrmap e8: y(active, since 2s) 2024-04-15T18:02:43.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:43.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: [15/Apr/2024:18:02:43] ENGINE Bus STARTING 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: [15/Apr/2024:18:02:43] ENGINE Serving on https://172.21.15.49:7150 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: [15/Apr/2024:18:02:43] ENGINE Serving on http://172.21.15.49:8765 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: [15/Apr/2024:18:02:43] ENGINE Bus STARTED 2024-04-15T18:02:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:43 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:02:44.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-15T18:02:44.904 INFO:teuthology.orchestra.run.smithi049.stdout:Generating ssh key... 2024-04-15T18:02:44.906 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:44 smithi049 ceph-mon[40087]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:45.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-mon[40087]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:45.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-mon[40087]: Generating ssh key... 2024-04-15T18:02:45.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: Generating public/private rsa key pair. 2024-04-15T18:02:45.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: Your identification has been saved in /tmp/tmp6fvx4cq4/key. 2024-04-15T18:02:45.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: Your public key has been saved in /tmp/tmp6fvx4cq4/key.pub. 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: The key fingerprint is: 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: SHA256:VCsJkpim40zFX8JqRgizrI5p1aykMtTYVeHL3/XDuSo ceph-22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: The key's randomart image is: 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: +---[RSA 3072]----+ 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |+ ooo..o. . | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |.+++.+oo o . | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |.+o o.o.+ . | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |+.+++..... | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |=+o= o oS . | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |+++ . . . . o .| 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |=+ . . . = | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: |o. E o| 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: | .... | 2024-04-15T18:02:45.978 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:02:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: +----[SHA256]-----+ 2024-04-15T18:02:46.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:46 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:46.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:46 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:47.498 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:02:47.499 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-15T18:02:47.499 INFO:teuthology.orchestra.run.smithi049.stdout:Adding key to root@localhost authorized_keys... 2024-04-15T18:02:47.499 INFO:teuthology.orchestra.run.smithi049.stdout:Adding host smithi049... 2024-04-15T18:02:47.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:47 smithi049 ceph-mon[40087]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:48.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:48 smithi049 ceph-mon[40087]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:49.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:49 smithi049 ceph-mon[40087]: Deploying cephadm binary to smithi049 2024-04-15T18:02:52.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Added host 'smithi049' with addr '172.21.15.49' 2024-04-15T18:02:52.907 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mon service... 2024-04-15T18:02:53.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:53 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:53 smithi049 ceph-mon[40087]: Added host smithi049 2024-04-15T18:02:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:53 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:02:54.305 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-15T18:02:54.305 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mgr service... 2024-04-15T18:02:54.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:54 smithi049 ceph-mon[40087]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:54.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:54 smithi049 ceph-mon[40087]: Saving service mon spec with placement count:5 2024-04-15T18:02:54.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:54 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:56.087 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-15T18:02:56.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:56 smithi049 ceph-mon[40087]: from='client.14143 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:02:56.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:56 smithi049 ceph-mon[40087]: Saving service mgr spec with placement count:2 2024-04-15T18:02:56.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:56 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:58.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4091370081' entity='client.admin' 2024-04-15T18:02:58.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:58 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:59.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:59 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:02:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:02:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1800615024' entity='client.admin' 2024-04-15T18:03:00.118 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling the dashboard module... 2024-04-15T18:03:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:00 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:03:01.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:00 smithi049 ceph-mon[40087]: from='mgr.14118 172.21.15.49:0/1589217954' entity='mgr.y' 2024-04-15T18:03:02.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:01 smithi049 ceph-mon[40087]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:02.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3836012450' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-15T18:03:02.141 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:01 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: ignoring --setuser ceph since I am not root 2024-04-15T18:03:02.141 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:01 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: ignoring --setgroup ceph since I am not root 2024-04-15T18:03:02.142 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:01 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:01.952+0000 7fb03522b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T18:03:02.142 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:02 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:02.138+0000 7fb03522b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T18:03:02.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:02 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:02.480+0000 7fb03522b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T18:03:03.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3836012450' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-15T18:03:03.050 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:02 smithi049 ceph-mon[40087]: mgrmap e9: y(active, since 21s) 2024-04-15T18:03:03.050 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:03 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:03.012+0000 7fb03522b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T18:03:03.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:03 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:03.269+0000 7fb03522b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T18:03:04.002 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:03 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:03.735+0000 7fb03522b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T18:03:04.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/561834041' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T18:03:04.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:04 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:04.496+0000 7fb03522b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T18:03:04.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:04 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:04.587+0000 7fb03522b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-04-15T18:03:04.738 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 9... 2024-04-15T18:03:05.656 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.232+0000 7fb03522b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T18:03:05.656 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.325+0000 7fb03522b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T18:03:05.657 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.416+0000 7fb03522b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T18:03:05.976 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.652+0000 7fb03522b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T18:03:05.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.749+0000 7fb03522b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T18:03:05.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:05 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:05.847+0000 7fb03522b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T18:03:06.390 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:06 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:06.091+0000 7fb03522b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T18:03:06.390 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:06 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:06.191+0000 7fb03522b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T18:03:06.652 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:06 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:06.386+0000 7fb03522b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T18:03:06.976 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:06 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:06.648+0000 7fb03522b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T18:03:07.697 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:07 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:07.263+0000 7fb03522b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T18:03:07.697 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:07 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:07.484+0000 7fb03522b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T18:03:07.977 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:07 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:07.693+0000 7fb03522b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:03:08.726 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:08.388+0000 7fb03522b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T18:03:08.727 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:08.598+0000 7fb03522b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: Active manager daemon y restarted 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: Activating manager daemon y 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: osdmap e3: 0 total, 0 up, 0 in 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: mgrmap e10: y(active, starting, since 0.00306624s) 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T18:03:09.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-mon[40087]: Manager daemon y is now available 2024-04-15T18:03:09.070 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:08.789+0000 7fb03522b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T18:03:09.070 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:03:08 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:03:08.910+0000 7fb03522b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T18:03:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T18:03:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T18:03:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:09 smithi049 ceph-mon[40087]: mgrmap e11: y(active, since 1.00537s) 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 9 is available 2024-04-15T18:03:10.847 INFO:teuthology.orchestra.run.smithi049.stdout:Generating a dashboard self-signed certificate... 2024-04-15T18:03:11.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:10 smithi049 ceph-mon[40087]: from='client.14155 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T18:03:11.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:10 smithi049 ceph-mon[40087]: from='client.14155 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T18:03:12.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: mgrmap e12: y(active, since 2s) 2024-04-15T18:03:12.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:12.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: [15/Apr/2024:18:03:11] ENGINE Bus STARTING 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: [15/Apr/2024:18:03:11] ENGINE Serving on https://172.21.15.49:7150 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: [15/Apr/2024:18:03:11] ENGINE Serving on http://172.21.15.49:8765 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: [15/Apr/2024:18:03:11] ENGINE Bus STARTED 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: from='client.14163 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:12.412 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-15T18:03:12.412 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial admin user... 2024-04-15T18:03:13.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:14.330 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$pvM6tHgZc/88dqofhH/Q8e5zFsMAeXClwdsyQi8xZlm0xn9Q4gzPm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713204193, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-15T18:03:14.330 INFO:teuthology.orchestra.run.smithi049.stdout:Fetching dashboard port number... 2024-04-15T18:03:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:14 smithi049 ceph-mon[40087]: from='client.14165 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:15.797 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 8443 2024-04-15T18:03:15.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4263933354' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-15T18:03:15.807 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-15T18:03:15.807 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-04-15T18:03:15.816 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-15T18:03:15.816 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-04-15T18:03:15.816 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-04-15T18:03:15.817 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-15T18:03:15.818 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph Dashboard is now available at: 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: URL: https://smithi049.front.sepia.ceph.com:8443/ 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: User: admin 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: Password: 6e8g103g90 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout:Saving cluster configuration to /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config directory 2024-04-15T18:03:15.819 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling autotune for osd_memory_target 2024-04-15T18:03:16.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:16 smithi049 ceph-mon[40087]: mgrmap e13: y(active, since 6s) 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.627 INFO:teuthology.orchestra.run.smithi049.stdout:Or, if you are only running a single cluster on this host: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: ceph telemetry on 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout:For more information see: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:18.628 INFO:teuthology.orchestra.run.smithi049.stdout:Bootstrap complete. 2024-04-15T18:03:18.676 INFO:tasks.cephadm:Fetching config... 2024-04-15T18:03:18.676 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:03:18.676 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-15T18:03:18.695 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-15T18:03:18.695 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:03:18.695 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-15T18:03:18.756 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-15T18:03:18.756 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:03:18.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-15T18:03:18.828 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-15T18:03:18.828 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:03:18.828 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-15T18:03:18.892 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-15T18:03:18.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T18:03:18.969 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:03:18.980 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T18:03:19.024 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:03:19.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T18:03:19.083 INFO:teuthology.orchestra.run.smithi159.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCievz8Pp74b6gVWpE9XJ198dFD3rfDsnFbFU2A58Zte/fPaWHDBX2/C4uYmLaJhYI5hgo/Kd1/zyJWjvUZORlK6uFfZlbKc1pQQObPImdz4wRDjP55Hgj7w/rTEm/1OUuvwxnZgQOxJK4uTT0fLC6GlcXTqnOllU3Wu5jX+CTa2lg1Awp/Adr1nrvl3PguAClKi8rCZ6mxxrZo/CdpctgCd02Wsfao1EIFxbsbW8K9IwBPaOujM7SCx1hOgrR0DCg4MnPdFhky0VDMXBdFe14e6TfdVR7Z1vB6oZoknJZTEhjZ6GjGsRM8chqG9iVWt9mO1+4gAWh0FQqhQjlu+eef5nVnGFQtDGXPt17B2X3Usl5sOEey755oy0rm6lKlIsWvYcID83EM5yK5hP3GLycOhngMr75exG4HBSTgdzAectD7tq3bh4vn+oTk25m9QbhLto5uAPFUDFcXEHpnc0rSTeovUfmIi2YbtREJ5wUe77pGE7rrcVNl/LeGf3dxcWk= ceph-22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:03:19.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-15T18:03:19.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4163362723' entity='client.admin' 2024-04-15T18:03:19.387 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:22.703 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-15T18:03:22.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-15T18:03:22.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1844479560' entity='client.admin' 2024-04-15T18:03:22.993 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi049", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:26.227 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-04-15T18:03:26.228 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:03:26.228 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-04-15T18:03:26.249 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:03:26.249 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:26.310 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-04-15T18:03:26.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch host add smithi082 2024-04-15T18:03:26.604 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: from='client.14175 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: Updating smithi049:/etc/ceph/ceph.conf 2024-04-15T18:03:26.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:26 smithi049 ceph-mon[40087]: Updating smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: Updating smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T18:03:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:27.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:27.905 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:28 smithi049 ceph-mon[40087]: from='client.14177 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:30.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:29 smithi049 ceph-mon[40087]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:30.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:29 smithi049 ceph-mon[40087]: Deploying cephadm binary to smithi082 2024-04-15T18:03:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:31 smithi049 ceph-mon[40087]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:32.497 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-04-15T18:03:33.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T18:03:33.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:33.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:33.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:33 smithi049 ceph-mon[40087]: Added host smithi082 2024-04-15T18:03:33.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:33.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:33.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:34.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:34 smithi049 ceph-mon[40087]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:34.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:34 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:35.291 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:35.291 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-04-15T18:03:35.950 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi159 2024-04-15T18:03:35.950 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:03:35.950 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/etc/ceph/ceph.conf 2024-04-15T18:03:35.972 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:03:35.972 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:36.033 INFO:tasks.cephadm:Adding host smithi159 to orchestrator... 2024-04-15T18:03:36.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch host add smithi159 2024-04-15T18:03:36.303 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:36.765 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:36 smithi049 ceph-mon[40087]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:36.765 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:36 smithi049 ceph-mon[40087]: from='client.14179 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:03:38.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:38.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: Updating smithi082:/etc/ceph/ceph.conf 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:38.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:39.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:39 smithi049 ceph-mon[40087]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T18:03:39.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:39 smithi049 ceph-mon[40087]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi159", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:40.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:40 smithi049 ceph-mon[40087]: Deploying cephadm binary to smithi159 2024-04-15T18:03:40.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:40 smithi049 ceph-mon[40087]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi159' with addr '172.21.15.159' 2024-04-15T18:03:42.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:42 smithi049 ceph-mon[40087]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:42.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:42.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T18:03:43.240 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:43.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:43 smithi049 ceph-mon[40087]: Added host smithi159 2024-04-15T18:03:43.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:43.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:43.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:44 smithi049 ceph-mon[40087]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:45.150 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:03:45.151 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}, {"addr": "172.21.15.159", "hostname": "smithi159", "labels": [], "status": ""}] 2024-04-15T18:03:45.774 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-15T18:03:45.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-15T18:03:46.044 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:03:47.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:46 smithi049 ceph-mon[40087]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:47.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:46 smithi049 ceph-mon[40087]: from='client.14183 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi159", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: Updating smithi159:/etc/ceph/ceph.conf 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: Updating smithi159:/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3690091781' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:48.788 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-04-15T18:03:49.461 INFO:tasks.cephadm:Adding mon.a on smithi049 2024-04-15T18:03:49.461 INFO:tasks.cephadm:Adding mon.b on smithi082 2024-04-15T18:03:49.461 INFO:tasks.cephadm:Adding mon.c on smithi159 2024-04-15T18:03:49.461 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch apply mon '3;smithi049:172.21.15.49=a;smithi082:172.21.15.82=b;smithi159:172.21.15.159=c' 2024-04-15T18:03:49.749 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3690091781' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T18:03:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:49 smithi049 ceph-mon[40087]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:03:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:50 smithi049 ceph-mon[40087]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:51.727 INFO:teuthology.orchestra.run.smithi159.stdout:Scheduled mon update... 2024-04-15T18:03:52.450 DEBUG:teuthology.orchestra.run.smithi082:mon.b> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.b.service 2024-04-15T18:03:52.455 DEBUG:teuthology.orchestra.run.smithi159:mon.c> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.c.service 2024-04-15T18:03:52.458 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T18:03:52.458 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T18:03:52.489 INFO:journalctl@ceph.mon.c.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:03:52.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi082:172.21.15.82=b;smithi159:172.21.15.159=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi082:172.21.15.82=b;smithi159:172.21.15.159=c;count:3 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:52.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:52.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:52.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:03:52.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:52.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:52 smithi049 ceph-mon[40087]: Deploying daemon mon.c on smithi159 2024-04-15T18:03:53.225 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:54.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:54 smithi049 ceph-mon[40087]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:55 smithi159 systemd[1]: Starting Ceph mon.c for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:03:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:03:56.645 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":1,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","modified":"2024-04-15T18:02:08.664609Z","created":"2024-04-15T18:02:08.664609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T18:03:56.647 INFO:teuthology.orchestra.run.smithi159.stderr:dumped monmap epoch 1 2024-04-15T18:03:57.057 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T18:03:57.057 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: ceph version 19.0.0-3080-gc2d8c315 (c2d8c3155d854ee204cb7c7261232571229d1d54) squid (dev), process ceph-mon, pid 7 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pidfile_write: ignore empty --pid-file 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: load: jerasure load: lrc 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: RocksDB version: 7.9.2 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Git sha 0 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Compile date 2024-04-12 22:42:42 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: DB SUMMARY 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: DB Session ID: 4VIZFLLRBZ93H49R4N8Y 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: CURRENT file: CURRENT 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-15T18:03:57.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.error_if_exists: 0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.create_if_missing: 0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.paranoid_checks: 1 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.env: 0x562a59287cc0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.fs: PosixFileSystem 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.info_log: 0x562a5af2dbe0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.statistics: (nil) 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.use_fsync: 0 2024-04-15T18:03:57.059 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_log_file_size: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_fallocate: 1 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.use_direct_reads: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.db_log_dir: 2024-04-15T18:03:57.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.wal_dir: 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.write_buffer_manager: 0x562a5aefe1e0 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T18:03:57.061 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.unordered_write: 0 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.row_cache: None 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.wal_filter: None 2024-04-15T18:03:57.062 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.two_write_queues: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.wal_compression: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.atomic_flush: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.log_readahead_size: 0 2024-04-15T18:03:57.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T18:03:57.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T18:03:57.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T18:03:57.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T18:03:57.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T18:03:57.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_background_jobs: 2 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_background_compactions: -1 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_subcompactions: 1 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_open_files: -1 2024-04-15T18:03:57.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_background_flushes: -1 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Compression algorithms supported: 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kZSTD supported: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kXpressCompression supported: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kZlibCompression supported: 1 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kSnappyCompression supported: 1 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kLZ4Compression supported: 1 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: kBZip2Compression supported: 0 2024-04-15T18:03:57.066 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.merge_operator: 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_filter: None 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a5af2ddc0) 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout: cache_index_and_filter_blocks: 1 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T18:03:57.067 INFO:journalctl@ceph.mon.c.smithi159.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: pin_top_level_index_and_filter: 1 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: index_type: 0 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: data_block_index_type: 0 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: index_shortening: 1 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: checksum: 4 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: no_block_cache: 0 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_cache: 0x562a5af31090 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_cache_name: BinnedLRUCache 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_cache_options: 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: capacity : 536870912 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: num_shard_bits : 4 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: strict_capacity_limit : 0 2024-04-15T18:03:57.068 INFO:journalctl@ceph.mon.c.smithi159.stdout: high_pri_pool_ratio: 0.000 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_cache_compressed: (nil) 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: persistent_cache: (nil) 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_size: 4096 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_size_deviation: 10 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_restart_interval: 16 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: index_block_restart_interval: 1 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: metadata_block_size: 4096 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: partition_filters: 0 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: use_delta_encoding: 1 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: filter_policy: bloomfilter 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: whole_key_filtering: 1 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: verify_compression: 0 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: read_amp_bytes_per_bit: 0 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: format_version: 5 2024-04-15T18:03:57.069 INFO:journalctl@ceph.mon.c.smithi159.stdout: enable_index_compression: 1 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout: block_align: 0 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout: max_auto_readahead_size: 262144 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout: prepopulate_block_cache: 0 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout: initial_auto_readahead_size: 8192 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout: num_file_reads_for_auto_readahead: 2 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression: NoCompression 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.num_levels: 7 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T18:03:57.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T18:03:57.071 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T18:03:57.072 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.arena_block_size: 1048576 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T18:03:57.074 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.inplace_update_support: 0 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T18:03:57.075 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.bloom_locality: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.max_successive_merges: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.ttl: 2592000 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enable_blob_files: false 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.min_blob_size: 0 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T18:03:57.076 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.blob_file_starting_level: 0 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 60aa2549-74bf-4625-af9a-beb3ebd85dfa 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236856424, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236856827, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713204236, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "60aa2549-74bf-4625-af9a-beb3ebd85dfa", "db_session_id": "4VIZFLLRBZ93H49R4N8Y", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236856903, "job": 1, "event": "recovery_finished"} 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-15T18:03:57.077 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562a5b020000 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: DB pointer 0x562a5b00a000 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: ** DB Stats ** 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T18:03:57.078 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: ** Compaction Stats [default] ** 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: ** Compaction Stats [default] ** 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-15T18:03:57.079 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Block cache BinnedLRUCache@0x562a5af31090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-15T18:03:57.080 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: using public_addr v2:172.21.15.159:0/0 -> [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: starting mon.c rank -1 at public addrs [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] at bind addrs [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(???) e0 preinit fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).mds e1 new map 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: e1 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: legacy client fscid: -1 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: 2024-04-15T18:03:57.081 INFO:journalctl@ceph.mon.c.smithi159.stdout: No filesystems configured 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.878824) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236879008, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1125, "num_deletes": 25, "total_data_size": 2297090, "memory_usage": 2321256, "flush_reason": "Manual Compaction"} 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T18:03:57.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Added host smithi082 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.14179 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi082:/etc/ceph/ceph.conf 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.084 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi159", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Deploying cephadm binary to smithi159 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Added host smithi159 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:57.085 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.14183 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi159", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:57.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi159:/etc/ceph/ceph.conf 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi159:/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3690091781' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3690091781' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.14187 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi082:172.21.15.82=b;smithi159:172.21.15.159=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi082:172.21.15.82=b;smithi159:172.21.15.159=c;count:3 2024-04-15T18:03:57.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: Deploying daemon mon.c on smithi159 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/2054992122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236882904, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2015269, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1130, "table_properties": {"data_size": 2009582, "index_size": 2443, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 26861, "raw_average_key_size": 31, "raw_value_size": 1991662, "raw_average_value_size": 2354, "num_data_blocks": 115, "num_entries": 846, "num_filter_entries": 846, "num_deletions": 25, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713204236, "oldest_key_time": 1713204236, "file_creation_time": 1713204236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "60aa2549-74bf-4625-af9a-beb3ebd85dfa", "db_session_id": "4VIZFLLRBZ93H49R4N8Y", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4101 microseconds, and 2721 cpu microseconds. 2024-04-15T18:03:57.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.882929) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2015269 bytes OK 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.882942) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.883052) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.883063) EVENT_LOG_v1 {"time_micros": 1713204236883060, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.883074) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2289250, prev total WAL file size 2289391, number of live WAL files 2. 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: (Original Log Time 2024/04/15-18:03:56.883374) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6B760030' seq:72057594037927935, type:22 .. '6B76003133' seq:0, type:0; will stop at (end) 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1968KB) 8(1648B)] 2024-04-15T18:03:57.089 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:56 smithi159 ceph-mon[45336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713204236883426, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2016917, "oldest_snapshot_seqno": -1} 2024-04-15T18:03:57.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:56 smithi049 ceph-mon[40087]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:03:57.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:03:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/2054992122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:03:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:57 smithi159 bash[45282]: a0a70f0b5f65422c30a879a08368de4f0f9de1c458f5c01bab20f66f4452626c 2024-04-15T18:03:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:03:57 smithi159 systemd[1]: Started Ceph mon.c for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:03:58.573 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T18:03:58.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T18:03:58.845 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: mon.a calling monitor election 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: mon.c calling monitor election 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: monmap epoch 2 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: last_changed 2024-04-15T18:03:56.887126+0000 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: min_mon_release 19 (squid) 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: election_strategy: 1 2024-04-15T18:04:02.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: fsmap 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: mgrmap e13: y(active, since 52s) 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: overall HEALTH_OK 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: mon.a calling monitor election 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: mon.c calling monitor election 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: monmap epoch 2 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: last_changed 2024-04-15T18:03:56.887126+0000 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: min_mon_release 19 (squid) 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: election_strategy: 1 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: fsmap 2024-04-15T18:04:02.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: mgrmap e13: y(active, since 52s) 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: overall HEALTH_OK 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:02.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:03.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:02 smithi159 ceph-mon[45336]: Deploying daemon mon.b on smithi082 2024-04-15T18:04:03.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:02 smithi049 ceph-mon[40087]: Deploying daemon mon.b on smithi082 2024-04-15T18:04:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:03.792 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:04:03.792 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":2,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","modified":"2024-04-15T18:03:56.887126Z","created":"2024-04-15T18:02:08.664609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:3300","nonce":0},{"type":"v1","addr":"172.21.15.159:6789","nonce":0}]},"addr":"172.21.15.159:6789/0","public_addr":"172.21.15.159:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T18:04:03.794 INFO:teuthology.orchestra.run.smithi159.stderr:dumped monmap epoch 2 2024-04-15T18:04:04.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:03 smithi159 ceph-mon[45336]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:04.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/3676330544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:04:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:03 smithi049 ceph-mon[40087]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/3676330544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:04:05.483 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T18:04:05.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T18:04:05.760 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:04:06.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:05 smithi082 bash[44168]: 70a7fc69c8d6c70897f2f5ef859d5fe2a60d22c48bcc37ded0fe92a614fc1528 2024-04-15T18:04:06.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:05 smithi082 systemd[1]: Started Ceph mon.b for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: mon.a calling monitor election 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: mon.c calling monitor election 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:11.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: monmap epoch 3 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: min_mon_release 19 (squid) 2024-04-15T18:04:11.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: election_strategy: 1 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: fsmap 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: mgrmap e13: y(active, since 61s) 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.162 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: mon.a calling monitor election 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: mon.c calling monitor election 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: monmap epoch 3 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: min_mon_release 19 (squid) 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: election_strategy: 1 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: fsmap 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: mgrmap e13: y(active, since 61s) 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.332 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-15T18:04:11.332 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":3,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","modified":"2024-04-15T18:04:05.650761Z","created":"2024-04-15T18:02:08.664609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:3300","nonce":0},{"type":"v1","addr":"172.21.15.159:6789","nonce":0}]},"addr":"172.21.15.159:6789/0","public_addr":"172.21.15.159:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T18:04:11.335 INFO:teuthology.orchestra.run.smithi159.stderr:dumped monmap epoch 3 2024-04-15T18:04:11.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:11.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:11.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/1504473141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:04:11.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/1504473141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:04:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:12.631 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-15T18:04:12.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-15T18:04:12.911 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:04:13.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:13.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: mon.a calling monitor election 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: mon.c calling monitor election 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: monmap epoch 3 2024-04-15T18:04:13.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: min_mon_release 19 (squid) 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: election_strategy: 1 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: fsmap 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: mgrmap e13: y(active, since 61s) 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T18:04:13.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/1504473141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:04:13.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mon.b calling monitor election 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi049:/etc/ceph/ceph.conf 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi082:/etc/ceph/ceph.conf 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi159:/etc/ceph/ceph.conf 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mon.b calling monitor election 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mon.c calling monitor election 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mon.a calling monitor election 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: monmap epoch 3 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: min_mon_release 19 (squid) 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: election_strategy: 1 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:13.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: fsmap 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: mgrmap e13: y(active, since 63s) 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: overall HEALTH_OK 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mon.b calling monitor election 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi049:/etc/ceph/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi082:/etc/ceph/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi159:/etc/ceph/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mon.b calling monitor election 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mon.c calling monitor election 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mon.a calling monitor election 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: monmap epoch 3 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: min_mon_release 19 (squid) 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: election_strategy: 1 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: fsmap 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: mgrmap e13: y(active, since 63s) 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: overall HEALTH_OK 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mon.b calling monitor election 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi049:/etc/ceph/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi082:/etc/ceph/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi159:/etc/ceph/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Updating smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mon.b calling monitor election 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mon.c calling monitor election 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mon.a calling monitor election 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: monmap epoch 3 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: last_changed 2024-04-15T18:04:05.650761+0000 2024-04-15T18:04:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: created 2024-04-15T18:02:08.664609+0000 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: min_mon_release 19 (squid) 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: election_strategy: 1 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: 1: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.c 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: fsmap 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: mgrmap e13: y(active, since 63s) 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: overall HEALTH_OK 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:13.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:14 smithi049 ceph-mon[40087]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:14 smithi049 ceph-mon[40087]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T18:04:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:14 smithi049 ceph-mon[40087]: Reconfiguring daemon mon.a on smithi049 2024-04-15T18:04:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:14 smithi082 ceph-mon[44206]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:14 smithi082 ceph-mon[44206]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T18:04:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:14 smithi082 ceph-mon[44206]: Reconfiguring daemon mon.a on smithi049 2024-04-15T18:04:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:14 smithi159 ceph-mon[45336]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:14 smithi159 ceph-mon[45336]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T18:04:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:14 smithi159 ceph-mon[45336]: Reconfiguring daemon mon.a on smithi049 2024-04-15T18:04:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:14.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.099 INFO:teuthology.orchestra.run.smithi049.stdout:# minimal ceph.conf for 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:16.099 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-04-15T18:04:16.099 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:04:16.099 INFO:teuthology.orchestra.run.smithi049.stdout: mon_host = [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] 2024-04-15T18:04:16.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: Reconfiguring mon.b (monmap changed)... 2024-04-15T18:04:16.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: Reconfiguring daemon mon.b on smithi082 2024-04-15T18:04:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:15 smithi159 ceph-mon[45336]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: Reconfiguring mon.b (monmap changed)... 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: Reconfiguring daemon mon.b on smithi082 2024-04-15T18:04:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:15 smithi049 ceph-mon[40087]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: Reconfiguring mon.b (monmap changed)... 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: Reconfiguring daemon mon.b on smithi082 2024-04-15T18:04:16.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:15 smithi082 ceph-mon[44206]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:16.781 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-15T18:04:16.781 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:04:16.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T18:04:16.814 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:04:16.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:04:16.882 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:04:16.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T18:04:16.917 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:04:16.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:04:16.986 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:04:16.986 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T18:04:17.016 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:04:17.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T18:04:17.083 INFO:tasks.cephadm:Adding mgr.y on smithi049 2024-04-15T18:04:17.083 INFO:tasks.cephadm:Adding mgr.x on smithi082 2024-04-15T18:04:17.083 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch apply mgr '2;smithi049=y;smithi082=x' 2024-04-15T18:04:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2404461011' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: Reconfiguring mon.c (monmap changed)... 2024-04-15T18:04:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:16 smithi159 ceph-mon[45336]: Reconfiguring daemon mon.c on smithi159 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2404461011' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: Reconfiguring mon.c (monmap changed)... 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:16 smithi049 ceph-mon[40087]: Reconfiguring daemon mon.c on smithi159 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2404461011' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: Reconfiguring mon.c (monmap changed)... 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T18:04:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T18:04:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:16 smithi082 ceph-mon[44206]: Reconfiguring daemon mon.c on smithi159 2024-04-15T18:04:17.565 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:04:18.090 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:17 smithi159 ceph-mon[45336]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:18.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:17 smithi049 ceph-mon[40087]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:18.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:17 smithi082 ceph-mon[44206]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:19.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:19.636 INFO:teuthology.orchestra.run.smithi159.stdout:Scheduled mgr update... 2024-04-15T18:04:20.235 DEBUG:teuthology.orchestra.run.smithi082:mgr.x> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.x.service 2024-04-15T18:04:20.238 INFO:tasks.cephadm:Deploying OSDs... 2024-04-15T18:04:20.239 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:04:20.239 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T18:04:20.262 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T18:04:20.262 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-04-15T18:04:20.266 INFO:journalctl@ceph.mgr.x.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:04:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 167383 Links: 1 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-15 18:03:22.915518777 +0000 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-15 17:40:01.263246904 +0000 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-15 17:40:01.263246904 +0000 2024-04-15T18:04:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-15T18:04:20.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T18:04:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-15T18:04:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-15T18:04:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000122937 s, 4.2 MB/s 2024-04-15T18:04:20.397 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T18:04:20.460 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-04-15T18:04:20.521 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 171042 Links: 1 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-15 18:03:22.979517025 +0000 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-15 17:40:01.606237398 +0000 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-15 17:40:01.606237398 +0000 2024-04-15T18:04:20.522 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-15T18:04:20.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T18:04:20.591 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-15T18:04:20.592 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-15T18:04:20.592 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000111137 s, 4.6 MB/s 2024-04-15T18:04:20.592 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T18:04:20.653 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-04-15T18:04:20.713 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 171150 Links: 1 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-15 18:03:23.040515355 +0000 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-15 17:40:01.959227615 +0000 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-15 17:40:01.959227615 +0000 2024-04-15T18:04:20.714 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-15T18:04:20.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T18:04:20.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: Saving service mgr spec with placement smithi049=y;smithi082=x;count:2 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:20 smithi049 ceph-mon[40087]: Deploying daemon mgr.x on smithi082 2024-04-15T18:04:20.754 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-15T18:04:20.755 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-15T18:04:20.755 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000106732 s, 4.8 MB/s 2024-04-15T18:04:20.756 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: Saving service mgr spec with placement smithi049=y;smithi082=x;count:2 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:20.764 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.765 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:20 smithi082 ceph-mon[44206]: Deploying daemon mgr.x on smithi082 2024-04-15T18:04:20.817 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 6h/6d Inode: 173144 Links: 1 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-15 18:03:23.092513931 +0000 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-15 17:40:02.293218359 +0000 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-15 17:40:02.293218359 +0000 2024-04-15T18:04:20.877 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: - 2024-04-15T18:04:20.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T18:04:20.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:20.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:20.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: Saving service mgr spec with placement smithi049=y;smithi082=x;count:2 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:20.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:20 smithi159 ceph-mon[45336]: Deploying daemon mgr.x on smithi082 2024-04-15T18:04:20.944 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-15T18:04:20.944 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-15T18:04:20.944 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 8.4925e-05 s, 6.0 MB/s 2024-04-15T18:04:20.946 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T18:04:21.005 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:04:21.005 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T18:04:21.024 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T18:04:21.024 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-04-15T18:04:21.082 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T18:04:21.082 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 170365 Links: 1 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-15 18:04:05.074606288 +0000 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-15 17:31:05.360397638 +0000 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-15 17:31:05.360397638 +0000 2024-04-15T18:04:21.083 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-15T18:04:21.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T18:04:21.204 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-15T18:04:21.204 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-15T18:04:21.204 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000267375 s, 1.9 MB/s 2024-04-15T18:04:21.205 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T18:04:21.223 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-04-15T18:04:21.280 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T18:04:21.280 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:21.280 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 166543 Links: 1 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-15 18:04:21.154285502 +0000 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-15 17:31:05.682391125 +0000 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-15 17:31:05.682391125 +0000 2024-04-15T18:04:21.281 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-15T18:04:21.281 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T18:04:21.349 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-15T18:04:21.349 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-15T18:04:21.349 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000217503 s, 2.4 MB/s 2024-04-15T18:04:21.350 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T18:04:21.410 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-04-15T18:04:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T18:04:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:21.468 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 166637 Links: 1 2024-04-15T18:04:21.468 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-15 18:04:21.285282889 +0000 2024-04-15T18:04:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-15 17:31:06.026384166 +0000 2024-04-15T18:04:21.469 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-15 17:31:06.026384166 +0000 2024-04-15T18:04:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-15T18:04:21.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T18:04:21.533 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:21 smithi082 systemd[1]: Starting Ceph mgr.x for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:04:21.535 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-15T18:04:21.535 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-15T18:04:21.535 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000176489 s, 2.9 MB/s 2024-04-15T18:04:21.536 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T18:04:21.595 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 169506 Links: 1 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-15 18:04:21.285282889 +0000 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-15 17:31:06.355377512 +0000 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-15 17:31:06.355377512 +0000 2024-04-15T18:04:21.653 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-15T18:04:21.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T18:04:21.721 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-15T18:04:21.721 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-15T18:04:21.721 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000227457 s, 2.3 MB/s 2024-04-15T18:04:21.722 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T18:04:21.781 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:04:21.781 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T18:04:21.803 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T18:04:21.803 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-04-15T18:04:21.865 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T18:04:21.865 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:21.865 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 168507 Links: 1 2024-04-15T18:04:21.865 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:21.866 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:21.866 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-15 18:03:55.446636491 +0000 2024-04-15T18:04:21.866 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-15 17:45:45.330588823 +0000 2024-04-15T18:04:21.866 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-15 17:45:45.330588823 +0000 2024-04-15T18:04:21.866 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-04-15T18:04:21.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T18:04:21.940 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-15T18:04:21.940 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-15T18:04:21.940 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000214431 s, 2.4 MB/s 2024-04-15T18:04:21.941 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T18:04:22.004 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-04-15T18:04:22.065 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T18:04:22.065 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:22.065 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 168562 Links: 1 2024-04-15T18:04:22.065 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:22.065 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:22.066 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-15 18:03:55.447636472 +0000 2024-04-15T18:04:22.066 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-15 17:45:45.705581598 +0000 2024-04-15T18:04:22.066 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-15 17:45:45.705581598 +0000 2024-04-15T18:04:22.066 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-04-15T18:04:22.066 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T18:04:22.136 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-15T18:04:22.136 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-15T18:04:22.136 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000307267 s, 1.7 MB/s 2024-04-15T18:04:22.137 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T18:04:22.198 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-04-15T18:04:22.222 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:21 smithi082 ceph-mgr[45885]: -- 172.21.15.82:0/1293113521 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4ed9cc5a0 con 0x55e4ed9a0c00 2024-04-15T18:04:22.222 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:21 smithi082 bash[45706]: 632886f8260c49cdfe000f756ee71821ecaa4580258a5be04bd7537b3b46458a 2024-04-15T18:04:22.222 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:21 smithi082 systemd[1]: Started Ceph mgr.x for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:04:22.222 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:22.052+0000 7fcfae973200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T18:04:22.223 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:22.219+0000 7fcfae973200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T18:04:22.257 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T18:04:22.257 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 170334 Links: 1 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-15 18:03:55.447636472 +0000 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-15 17:45:46.039575162 +0000 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-15 17:45:46.039575162 +0000 2024-04-15T18:04:22.258 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-04-15T18:04:22.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T18:04:22.326 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-15T18:04:22.326 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-15T18:04:22.326 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000207404 s, 2.5 MB/s 2024-04-15T18:04:22.327 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T18:04:22.387 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 6h/6d Inode: 165483 Links: 1 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T18:04:22.445 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-15 18:03:55.447636472 +0000 2024-04-15T18:04:22.446 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-15 17:45:46.402568169 +0000 2024-04-15T18:04:22.446 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-15 17:45:46.402568169 +0000 2024-04-15T18:04:22.446 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: - 2024-04-15T18:04:22.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T18:04:22.511 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-15T18:04:22.511 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-15T18:04:22.511 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000212366 s, 2.4 MB/s 2024-04-15T18:04:22.512 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T18:04:22.571 INFO:tasks.cephadm:Deploying osd.0 on smithi049 with /dev/vg_nvme/lv_4... 2024-04-15T18:04:22.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T18:04:22.855 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:04:22.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:22.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:22.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:22.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:22.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:23.020 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:22.576+0000 7fcfae973200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T18:04:23.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:23.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:23.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:23.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:23.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:23.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:23.290 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:23.110+0000 7fcfae973200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T18:04:23.790 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:23.358+0000 7fcfae973200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T18:04:24.290 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:23.802+0000 7fcfae973200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:24.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:24.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: Reconfiguring daemon mgr.y on smithi049 2024-04-15T18:04:24.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.790 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:24.514+0000 7fcfae973200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T18:04:24.790 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:24.598+0000 7fcfae973200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T18:04:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: Reconfiguring daemon mgr.y on smithi049 2024-04-15T18:04:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: Reconfiguring daemon mgr.y on smithi049 2024-04-15T18:04:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:25.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.185+0000 7fcfae973200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T18:04:25.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.269+0000 7fcfae973200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T18:04:25.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.356+0000 7fcfae973200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00813765 s, 1.3 GB/s 2024-04-15T18:04:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-04-15T18:04:26.005 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.587+0000 7fcfae973200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T18:04:26.006 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.677+0000 7fcfae973200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T18:04:26.006 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:25.769+0000 7fcfae973200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T18:04:26.286 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:26.003+0000 7fcfae973200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T18:04:26.287 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:26.097+0000 7fcfae973200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T18:04:26.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_4 2024-04-15T18:04:26.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:26.283+0000 7fcfae973200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T18:04:26.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:26.532+0000 7fcfae973200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T18:04:26.620 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:04:27.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.113 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.114 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:27.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:27.517 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:27.110+0000 7fcfae973200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T18:04:27.517 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:27.321+0000 7fcfae973200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T18:04:27.790 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:27.514+0000 7fcfae973200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:04:28.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:27 smithi159 ceph-mon[45336]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:28.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:27 smithi082 ceph-mon[44206]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:27 smithi049 ceph-mon[40087]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:28.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:28.172+0000 7fcfae973200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T18:04:28.540 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:28.369+0000 7fcfae973200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T18:04:28.855 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:28.548+0000 7fcfae973200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T18:04:28.855 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x[45813]: 2024-04-15T18:04:28.662+0000 7fcfae973200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T18:04:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: Standby manager daemon x started 2024-04-15T18:04:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T18:04:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T18:04:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T18:04:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:28 smithi159 ceph-mon[45336]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T18:04:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: Standby manager daemon x started 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T18:04:29.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:28 smithi049 ceph-mon[40087]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T18:04:29.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:29.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:29.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:29.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: Standby manager daemon x started 2024-04-15T18:04:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T18:04:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T18:04:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T18:04:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:28 smithi082 ceph-mon[44206]: from='mgr.? 172.21.15.82:0/817593441' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T18:04:29.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:29 smithi049 ceph-mon[40087]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-15T18:04:29.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T18:04:29.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:29 smithi049 ceph-mon[40087]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:30.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:29 smithi159 ceph-mon[45336]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-15T18:04:30.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T18:04:30.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:29 smithi159 ceph-mon[45336]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:29 smithi082 ceph-mon[44206]: mgrmap e14: y(active, since 79s), standbys: x 2024-04-15T18:04:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T18:04:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:29 smithi082 ceph-mon[44206]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]: dispatch 2024-04-15T18:04:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]': finished 2024-04-15T18:04:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T18:04:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:32.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3249007522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:04:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]: dispatch 2024-04-15T18:04:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]': finished 2024-04-15T18:04:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T18:04:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3249007522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]: dispatch 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/321838151' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9"}]': finished 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3249007522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:04:34.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:33 smithi049 ceph-mon[40087]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:34.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:33 smithi082 ceph-mon[44206]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:34.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:33 smithi159 ceph-mon[45336]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:36.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:36 smithi082 ceph-mon[44206]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:36.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:36 smithi159 ceph-mon[45336]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:36 smithi049 ceph-mon[40087]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:38 smithi082 ceph-mon[44206]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T18:04:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:38 smithi082 ceph-mon[44206]: Deploying daemon osd.0 on smithi049 2024-04-15T18:04:38.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:38 smithi159 ceph-mon[45336]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:38.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T18:04:38.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:38.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:38 smithi159 ceph-mon[45336]: Deploying daemon osd.0 on smithi049 2024-04-15T18:04:38.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:38 smithi049 ceph-mon[40087]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:38.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T18:04:38.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:38.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:38 smithi049 ceph-mon[40087]: Deploying daemon osd.0 on smithi049 2024-04-15T18:04:40.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:40 smithi049 ceph-mon[40087]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:40 smithi082 ceph-mon[44206]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:40 smithi159 ceph-mon[45336]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:42.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:42 smithi082 ceph-mon[44206]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:42.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:42 smithi049 ceph-mon[40087]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:42 smithi159 ceph-mon[45336]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:44 smithi082 ceph-mon[44206]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:44.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:44 smithi159 ceph-mon[45336]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:44.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:44 smithi049 ceph-mon[40087]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:46.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:46.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:46.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:46.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:46.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:46.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.277 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 0 on host 'smithi049' 2024-04-15T18:04:47.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:47 smithi049 ceph-mon[40087]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T18:04:47.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:47.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:47 smithi159 ceph-mon[45336]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T18:04:47.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:47.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:47 smithi082 ceph-mon[44206]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T18:04:47.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:47.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:04:47.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:48.035 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.0.service 2024-04-15T18:04:48.038 INFO:tasks.cephadm:Deploying osd.1 on smithi049 with /dev/vg_nvme/lv_3... 2024-04-15T18:04:48.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T18:04:48.071 INFO:journalctl@ceph.osd.0.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:04:48.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-mon[40087]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:48.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-mon[40087]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T18:04:48.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-mon[40087]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T18:04:48.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-mon[40087]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:04:48.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:48.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:48 smithi159 ceph-mon[45336]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:48.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:48 smithi159 ceph-mon[45336]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T18:04:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:48 smithi159 ceph-mon[45336]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T18:04:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:48 smithi159 ceph-mon[45336]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:04:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:48.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:48 smithi082 ceph-mon[44206]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:48.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:48 smithi082 ceph-mon[44206]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T18:04:48.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:48 smithi082 ceph-mon[44206]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T18:04:48.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:48 smithi082 ceph-mon[44206]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:04:48.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:48.828 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:04:48.977 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0[59449]: 2024-04-15T18:04:48.565+0000 7f7a384d4700 -1 osd.0 0 waiting for initial osdmap 2024-04-15T18:04:48.977 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:04:48 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0[59449]: 2024-04-15T18:04:48.567+0000 7f7a302bf700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:04:49.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:49 smithi049 ceph-mon[40087]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:04:49.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:49 smithi049 ceph-mon[40087]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T18:04:49.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:49 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:49 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:49 smithi049 ceph-mon[40087]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' 2024-04-15T18:04:49.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:49 smithi159 ceph-mon[45336]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:04:49.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:49 smithi159 ceph-mon[45336]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T18:04:49.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:49 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:49 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:49 smithi159 ceph-mon[45336]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' 2024-04-15T18:04:49.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:49 smithi082 ceph-mon[44206]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:04:49.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:49 smithi082 ceph-mon[44206]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T18:04:49.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:49 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:49 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:49.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:49 smithi082 ceph-mon[44206]: from='osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154]' entity='osd.0' 2024-04-15T18:04:50.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:04:50.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:04:50.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154] boot 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T18:04:50.560 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:50 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:50.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:04:50.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154] boot 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T18:04:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:50 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:04:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:04:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: osd.0 [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154] boot 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T18:04:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:50 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T18:04:51.633 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T18:04:51.634 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T18:04:51.634 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-04-15T18:04:51.634 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-04-15T18:04:51.634 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00841964 s, 1.2 GB/s 2024-04-15T18:04:51.634 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-04-15T18:04:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:52 smithi049 ceph-mon[40087]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:52 smithi049 ceph-mon[40087]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T18:04:52.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_3 2024-04-15T18:04:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:52 smithi159 ceph-mon[45336]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:52 smithi159 ceph-mon[45336]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T18:04:52.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:52 smithi082 ceph-mon[44206]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:52.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:52 smithi082 ceph-mon[44206]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T18:04:53.014 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi049 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:04:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi049 to 17122M 2024-04-15T18:04:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:54.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi049 2024-04-15T18:04:54.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:04:54.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi049 to 17122M 2024-04-15T18:04:54.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:54.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:54.316 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi049 2024-04-15T18:04:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi049 to 17122M 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:04:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:04:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:55 smithi082 ceph-mon[44206]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:55 smithi082 ceph-mon[44206]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:56.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:56.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:55 smithi049 ceph-mon[40087]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:56.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:55 smithi049 ceph-mon[40087]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:56.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:56.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:56.385 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:55 smithi159 ceph-mon[45336]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:55 smithi159 ceph-mon[45336]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:04:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:04:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:04:56.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:04:58.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:57 smithi159 ceph-mon[45336]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:57 smithi049 ceph-mon[40087]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:57 smithi082 ceph-mon[44206]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:04:59.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]: dispatch 2024-04-15T18:04:59.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]': finished 2024-04-15T18:04:59.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:58 smithi159 ceph-mon[45336]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T18:04:59.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:04:59.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1488450873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:04:59.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]: dispatch 2024-04-15T18:04:59.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]': finished 2024-04-15T18:04:59.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:58 smithi049 ceph-mon[40087]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T18:04:59.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:04:59.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1488450873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:04:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]: dispatch 2024-04-15T18:04:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1225992887' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595"}]': finished 2024-04-15T18:04:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:58 smithi082 ceph-mon[44206]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T18:04:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:04:59.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1488450873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:00.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:04:59 smithi159 ceph-mon[45336]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:00.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:04:59 smithi049 ceph-mon[40087]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:04:59 smithi082 ceph-mon[44206]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:02 smithi082 ceph-mon[44206]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:02.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:02 smithi049 ceph-mon[40087]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:02 smithi159 ceph-mon[45336]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:04.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:04 smithi049 ceph-mon[40087]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:04 smithi082 ceph-mon[44206]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:04.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:04 smithi159 ceph-mon[45336]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T18:05:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:05 smithi082 ceph-mon[44206]: Deploying daemon osd.1 on smithi049 2024-04-15T18:05:05.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T18:05:05.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:05.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:05 smithi049 ceph-mon[40087]: Deploying daemon osd.1 on smithi049 2024-04-15T18:05:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T18:05:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:05 smithi159 ceph-mon[45336]: Deploying daemon osd.1 on smithi049 2024-04-15T18:05:06.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:06 smithi049 ceph-mon[40087]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:06 smithi082 ceph-mon[44206]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:06 smithi159 ceph-mon[45336]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:08.199 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:08 smithi049 ceph-mon[40087]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:08 smithi082 ceph-mon[44206]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:08 smithi159 ceph-mon[45336]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:10 smithi049 ceph-mon[40087]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:10.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:10 smithi082 ceph-mon[44206]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:10 smithi159 ceph-mon[45336]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:11.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:11.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:12 smithi049 ceph-mon[40087]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:12.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:12 smithi159 ceph-mon[45336]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:12 smithi082 ceph-mon[44206]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:13 smithi049 ceph-mon[40087]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T18:05:13.746 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 1 on host 'smithi049' 2024-04-15T18:05:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:13 smithi159 ceph-mon[45336]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T18:05:14.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:14.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:14.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:13 smithi082 ceph-mon[44206]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T18:05:14.715 DEBUG:teuthology.orchestra.run.smithi049:osd.1> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.1.service 2024-04-15T18:05:14.717 INFO:tasks.cephadm:Deploying osd.2 on smithi049 with /dev/vg_nvme/lv_2... 2024-04-15T18:05:14.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T18:05:14.751 INFO:journalctl@ceph.osd.1.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:05:14.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T18:05:14.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:14.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:14.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:15.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T18:05:15.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:15.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:15.515 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:05:15.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:15 smithi049 ceph-mon[40087]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:15.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:15 smithi049 ceph-mon[40087]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T18:05:15.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:15.900 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:15.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:15 smithi159 ceph-mon[45336]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:15.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:15 smithi159 ceph-mon[45336]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T18:05:15.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:15.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:15 smithi082 ceph-mon[44206]: from='osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:16.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:15 smithi082 ceph-mon[44206]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T18:05:16.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046] boot 2024-04-15T18:05:16.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T18:05:16.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:16.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046] boot 2024-04-15T18:05:16.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T18:05:16.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:17.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:05:17.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:05:17.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:17.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-15T18:05:17.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:17.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: osd.1 [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046] boot 2024-04-15T18:05:17.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T18:05:17.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T18:05:17.800 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:17 smithi049 ceph-mon[40087]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T18:05:17.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:17 smithi159 ceph-mon[45336]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T18:05:18.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:17 smithi082 ceph-mon[44206]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-04-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-04-15T18:05:18.451 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00804993 s, 1.3 GB/s 2024-04-15T18:05:18.451 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-04-15T18:05:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:18 smithi049 ceph-mon[40087]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:18.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:18 smithi159 ceph-mon[45336]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:19.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:18 smithi082 ceph-mon[44206]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:19.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_2 2024-04-15T18:05:19.794 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi049 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi049 to 8561M 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:20.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi049 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi049 to 8561M 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi049 2024-04-15T18:05:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi049 to 8561M 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:20.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:21.069 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:20 smithi049 ceph-mon[40087]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:21.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:20 smithi159 ceph-mon[45336]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:20 smithi082 ceph-mon[44206]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:22.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:21 smithi159 ceph-mon[45336]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:22.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:21 smithi049 ceph-mon[40087]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:21 smithi082 ceph-mon[44206]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:23.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:23.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:23.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:23 smithi049 ceph-mon[40087]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:24.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:23 smithi159 ceph-mon[45336]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:24.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:23 smithi082 ceph-mon[44206]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:25.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]: dispatch 2024-04-15T18:05:25.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]': finished 2024-04-15T18:05:25.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:24 smithi159 ceph-mon[45336]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T18:05:25.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:25.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]: dispatch 2024-04-15T18:05:25.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]': finished 2024-04-15T18:05:25.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:24 smithi049 ceph-mon[40087]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T18:05:25.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:25.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]: dispatch 2024-04-15T18:05:25.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3838009979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8"}]': finished 2024-04-15T18:05:25.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:24 smithi082 ceph-mon[44206]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T18:05:25.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:26.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:25 smithi159 ceph-mon[45336]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:26.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2200827680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:26.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:25 smithi049 ceph-mon[40087]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:26.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2200827680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:25 smithi082 ceph-mon[44206]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2200827680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:28.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:28 smithi049 ceph-mon[40087]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:28 smithi082 ceph-mon[44206]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:28 smithi159 ceph-mon[45336]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:30.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:30 smithi049 ceph-mon[40087]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:30 smithi082 ceph-mon[44206]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:30.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:30 smithi159 ceph-mon[45336]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:32 smithi082 ceph-mon[44206]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T18:05:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:32 smithi082 ceph-mon[44206]: Deploying daemon osd.2 on smithi049 2024-04-15T18:05:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:32 smithi159 ceph-mon[45336]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T18:05:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:32 smithi159 ceph-mon[45336]: Deploying daemon osd.2 on smithi049 2024-04-15T18:05:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:32 smithi049 ceph-mon[40087]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T18:05:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:32 smithi049 ceph-mon[40087]: Deploying daemon osd.2 on smithi049 2024-04-15T18:05:34.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:34 smithi049 ceph-mon[40087]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:34.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:34 smithi082 ceph-mon[44206]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:34.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:34 smithi159 ceph-mon[45336]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:36.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:36 smithi082 ceph-mon[44206]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:36.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:36 smithi049 ceph-mon[40087]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:36.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:36 smithi159 ceph-mon[45336]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:37 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:37 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:37.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:37 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:37.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:37 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:37.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:37 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:37.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:37 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:37 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:37 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:37 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:38.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:38 smithi049 ceph-mon[40087]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:38.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:38 smithi159 ceph-mon[45336]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:38 smithi082 ceph-mon[44206]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:40.217 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 2 on host 'smithi049' 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:05:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:40.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:40.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:40.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-15T18:05:40.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:40.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:41.027 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T18:05:41.029 INFO:tasks.cephadm:Deploying osd.3 on smithi049 with /dev/vg_nvme/lv_1... 2024-04-15T18:05:41.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T18:05:41.063 INFO:journalctl@ceph.osd.2.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:05:41.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:05:41.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-mon[40087]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T18:05:41.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:41.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:41.838 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:05:41.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:41 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:05:41.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:41 smithi159 ceph-mon[45336]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T18:05:41.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:41 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:41.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:41.977 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[74744]: 2024-04-15T18:05:41.604+0000 7f0cfe72a700 -1 osd.2 0 waiting for initial osdmap 2024-04-15T18:05:41.977 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:41 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[74744]: 2024-04-15T18:05:41.611+0000 7f0cfa51d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:05:42.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:41 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:05:42.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:41 smithi082 ceph-mon[44206]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T18:05:42.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:41 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:05:42.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:42.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:42.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:42.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T18:05:42.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:42.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:42.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:42.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:42.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618] boot 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:43.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:43.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:05:43.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:05:43.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:43.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618] boot 2024-04-15T18:05:43.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T18:05:43.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:43.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:44.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:05:44.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:05:44.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:05:44.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: osd.2 [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618] boot 2024-04-15T18:05:44.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T18:05:44.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:05:44.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:44.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:44 smithi159 ceph-mon[45336]: pgmap v85: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:44.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:44.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:44 smithi159 ceph-mon[45336]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T18:05:44.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:44 smithi049 ceph-mon[40087]: pgmap v85: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:44 smithi049 ceph-mon[40087]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T18:05:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:45.019 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T18:05:45.019 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T18:05:45.019 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-04-15T18:05:45.019 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-04-15T18:05:45.020 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00818643 s, 1.3 GB/s 2024-04-15T18:05:45.020 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-04-15T18:05:45.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:44 smithi082 ceph-mon[44206]: pgmap v85: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:45.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:45.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:44 smithi082 ceph-mon[44206]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T18:05:45.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:45.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:45.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:45.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi049:vg_nvme/lv_1 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:45.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:46.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:05:46.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T18:05:46.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:05:46.167 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79672]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79672]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79672]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79672]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79675]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79675]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79675]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.227 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:05:45 smithi049 sudo[79675]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.630 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79760]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:46.630 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79760]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79760]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79763]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79763]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79763]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79763]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79766]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79766]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79766]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79766]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.631 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:46.632 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.632 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79769]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.632 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79769]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79781]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79795]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79795]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79795]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 sudo[79795]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47916]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47916]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47916]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47916]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47919]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47919]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47919]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:46.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 sudo[47919]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi049 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi049 to 5707M 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.911 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:46 smithi159 ceph-mon[45336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:46.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:46.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi049 2024-04-15T18:05:46.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi049 to 5707M 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:46.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:46 smithi049 ceph-mon[40087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:47.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46906]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 sudo[46909]: pam_unix(sudo:session): session closed for user root 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi049 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi049 to 5707M 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:47.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:47.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T18:05:47.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:05:47.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:05:47.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:05:47.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:46 smithi082 ceph-mon[44206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T18:05:47.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:47.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:47 smithi049 ceph-mon[40087]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:47 smithi159 ceph-mon[45336]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:05:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:47 smithi082 ceph-mon[44206]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:49.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:48 smithi159 ceph-mon[45336]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T18:05:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:48 smithi159 ceph-mon[45336]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:05:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:48 smithi049 ceph-mon[40087]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T18:05:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:48 smithi049 ceph-mon[40087]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:05:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:48 smithi082 ceph-mon[44206]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T18:05:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:48 smithi082 ceph-mon[44206]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:05:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:05:49.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:05:49.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:50.088 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:05:50.088 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:05:50.088 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:49 smithi049 ceph-mon[40087]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T18:05:50.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:49 smithi049 ceph-mon[40087]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:05:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:05:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:49 smithi082 ceph-mon[44206]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T18:05:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:49 smithi082 ceph-mon[44206]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:05:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:05:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:49 smithi159 ceph-mon[45336]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T18:05:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:49 smithi159 ceph-mon[45336]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]: dispatch 2024-04-15T18:05:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]': finished 2024-04-15T18:05:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:51 smithi082 ceph-mon[44206]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T18:05:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:51 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:05:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]: dispatch 2024-04-15T18:05:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]': finished 2024-04-15T18:05:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:51 smithi159 ceph-mon[45336]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T18:05:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:51 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:05:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]: dispatch 2024-04-15T18:05:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3425421428' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2"}]': finished 2024-04-15T18:05:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:51 smithi049 ceph-mon[40087]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T18:05:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:51 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:05:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:52 smithi082 ceph-mon[44206]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1868047532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:52.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:52 smithi159 ceph-mon[45336]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:52.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1868047532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:52 smithi049 ceph-mon[40087]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:05:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1868047532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:05:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:54 smithi049 ceph-mon[40087]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-15T18:05:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:54 smithi082 ceph-mon[44206]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-15T18:05:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:54 smithi159 ceph-mon[45336]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-15T18:05:56.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:56 smithi049 ceph-mon[40087]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T18:05:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:56 smithi082 ceph-mon[44206]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T18:05:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:56 smithi159 ceph-mon[45336]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T18:05:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:58 smithi082 ceph-mon[44206]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 64 KiB/s wr, 4 op/s 2024-04-15T18:05:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T18:05:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:05:58 smithi082 ceph-mon[44206]: Deploying daemon osd.3 on smithi049 2024-04-15T18:05:58.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:58 smithi049 ceph-mon[40087]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 64 KiB/s wr, 4 op/s 2024-04-15T18:05:58.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T18:05:58.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:58.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:05:58 smithi049 ceph-mon[40087]: Deploying daemon osd.3 on smithi049 2024-04-15T18:05:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:58 smithi159 ceph-mon[45336]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 64 KiB/s wr, 4 op/s 2024-04-15T18:05:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T18:05:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:05:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:05:58 smithi159 ceph-mon[45336]: Deploying daemon osd.3 on smithi049 2024-04-15T18:06:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:00 smithi082 ceph-mon[44206]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:00.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:00 smithi049 ceph-mon[40087]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:00 smithi159 ceph-mon[45336]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:02.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:02 smithi049 ceph-mon[40087]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:02 smithi082 ceph-mon[44206]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:02 smithi159 ceph-mon[45336]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-15T18:06:04.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:04 smithi159 ceph-mon[45336]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 939 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-15T18:06:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:04 smithi049 ceph-mon[40087]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 939 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-15T18:06:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:04 smithi082 ceph-mon[44206]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail; 939 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-15T18:06:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:06.913 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 3 on host 'smithi049' 2024-04-15T18:06:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:06 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:06 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:06 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:06:07.589 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T18:06:07.591 INFO:tasks.cephadm:Deploying osd.4 on smithi082 with /dev/vg_nvme/lv_4... 2024-04-15T18:06:07.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T18:06:07.624 INFO:journalctl@ceph.osd.3.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:06:07.861 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:06:08.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:06:08.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T18:06:08.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:06:08.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:08.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:08.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:07 smithi082 ceph-mon[44206]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:08.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:06:08.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T18:06:08.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:06:08.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:08.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:08.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:07 smithi159 ceph-mon[45336]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:08.193 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[82102]: 2024-04-15T18:06:07.859+0000 7f7099dd4700 -1 osd.3 0 waiting for initial osdmap 2024-04-15T18:06:08.193 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[82102]: 2024-04-15T18:06:07.864+0000 7f7095bc7700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:06:08.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:06:08.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T18:06:08.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:06:08.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:08.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:08.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:08.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:07 smithi049 ceph-mon[40087]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:09.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:08 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:06:09.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:08 smithi082 ceph-mon[44206]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T18:06:09.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:08 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:06:09.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:08 smithi159 ceph-mon[45336]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T18:06:09.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:08 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-04-15T18:06:09.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:08 smithi049 ceph-mon[40087]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T18:06:09.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:09.734 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T18:06:09.735 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T18:06:09.735 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-04-15T18:06:09.735 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-04-15T18:06:09.735 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123459 s, 849 MB/s 2024-04-15T18:06:09.735 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-04-15T18:06:10.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:06:10.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:06:10.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:10.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081] boot 2024-04-15T18:06:10.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T18:06:10.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:10.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:09 smithi082 ceph-mon[44206]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:10.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081] boot 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:10.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:09 smithi159 ceph-mon[45336]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:10.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:06:10.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:06:10.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:10.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: osd.3 [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081] boot 2024-04-15T18:06:10.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T18:06:10.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:06:10.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:09 smithi049 ceph-mon[40087]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-15T18:06:10.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-04-15T18:06:10.629 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:06:11.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:10 smithi159 ceph-mon[45336]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T18:06:11.180 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:10 smithi082 ceph-mon[44206]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T18:06:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:10 smithi049 ceph-mon[40087]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T18:06:12.118 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi049 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi049 to 4280M 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.119 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:11 smithi082 ceph-mon[44206]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-15T18:06:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi049 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi049 to 4280M 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:11 smithi159 ceph-mon[45336]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi049 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi049 to 4280M 2024-04-15T18:06:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:12.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:12.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:12.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:11 smithi049 ceph-mon[40087]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 358 GiB / 358 GiB avail 2024-04-15T18:06:13.186 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:12 smithi082 ceph-mon[44206]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:13.187 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:13.187 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:13.187 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:12 smithi049 ceph-mon[40087]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:13.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:12 smithi159 ceph-mon[45336]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:13.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:13.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:14.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:13 smithi082 ceph-mon[44206]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:13 smithi049 ceph-mon[40087]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:14.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:13 smithi159 ceph-mon[45336]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/1093561547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]': finished 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/3271985014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/1093561547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]': finished 2024-04-15T18:06:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T18:06:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/3271985014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/1093561547' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]: dispatch 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928"}]': finished 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/3271985014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:18.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:18 smithi082 ceph-mon[44206]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:18.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:18 smithi159 ceph-mon[45336]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:18.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:18 smithi049 ceph-mon[40087]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:20.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:20 smithi082 ceph-mon[44206]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:20.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:20 smithi159 ceph-mon[45336]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:20.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:20 smithi049 ceph-mon[40087]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:22 smithi082 ceph-mon[44206]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T18:06:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:22 smithi082 ceph-mon[44206]: Deploying daemon osd.4 on smithi082 2024-04-15T18:06:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:22 smithi159 ceph-mon[45336]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T18:06:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:22 smithi159 ceph-mon[45336]: Deploying daemon osd.4 on smithi082 2024-04-15T18:06:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:22 smithi049 ceph-mon[40087]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T18:06:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:22 smithi049 ceph-mon[40087]: Deploying daemon osd.4 on smithi082 2024-04-15T18:06:24.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:24 smithi082 ceph-mon[44206]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:24.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:24 smithi159 ceph-mon[45336]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:24.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:24 smithi049 ceph-mon[40087]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:26 smithi082 ceph-mon[44206]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:26.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:26 smithi159 ceph-mon[45336]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:26 smithi049 ceph-mon[40087]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:28 smithi082 ceph-mon[44206]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:28.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:28 smithi159 ceph-mon[45336]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:28.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:28 smithi049 ceph-mon[40087]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:30.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:30.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.743 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:30.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:30.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:30.744 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:30 smithi082 ceph-mon[44206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:30.780 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 4 on host 'smithi082' 2024-04-15T18:06:30.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:30.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:30.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:30.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:30.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:30 smithi159 ceph-mon[45336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:30.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:30 smithi049 ceph-mon[40087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T18:06:31.456 DEBUG:teuthology.orchestra.run.smithi082:osd.4> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.4.service 2024-04-15T18:06:31.462 INFO:tasks.cephadm:Deploying osd.5 on smithi082 with /dev/vg_nvme/lv_3... 2024-04-15T18:06:31.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T18:06:31.489 INFO:journalctl@ceph.osd.4.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:31.712 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:06:31.558+0000 7f703fc5b700 -1 osd.4 0 waiting for initial osdmap 2024-04-15T18:06:31.712 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:06:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:06:31.565+0000 7f703ba4e700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:06:31.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T18:06:31.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:32.244 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649] boot 2024-04-15T18:06:32.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T18:06:32.687 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649] boot 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T18:06:32.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: osd.4 [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649] boot 2024-04-15T18:06:32.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T18:06:32.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T18:06:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:06:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:06:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:33 smithi159 ceph-mon[45336]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T18:06:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:33 smithi159 ceph-mon[45336]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:06:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:06:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:33 smithi049 ceph-mon[40087]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T18:06:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:33 smithi049 ceph-mon[40087]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:34.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:06:34.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:06:34.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:34.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:33 smithi082 ceph-mon[44206]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T18:06:34.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:33 smithi082 ceph-mon[44206]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:35.155 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T18:06:35.155 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T18:06:35.155 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-04-15T18:06:35.156 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-04-15T18:06:35.156 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116164 s, 903 MB/s 2024-04-15T18:06:35.156 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-04-15T18:06:35.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:06:35.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:06:35.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:34 smithi159 ceph-mon[45336]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T18:06:35.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:06:35.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:06:35.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:34 smithi082 ceph-mon[44206]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T18:06:35.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:06:35.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:06:35.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:34 smithi049 ceph-mon[40087]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi082 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi082 to 17122M 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:36.006 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:36.006 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.030 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-04-15T18:06:36.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T18:06:36.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi082 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi082 to 17122M 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:36.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi082 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi082 to 17122M 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:36.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:36.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:36.290 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:06:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:36 smithi082 ceph-mon[44206]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T18:06:37.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:36 smithi159 ceph-mon[45336]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T18:06:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:36 smithi049 ceph-mon[40087]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T18:06:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:37 smithi082 ceph-mon[44206]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-15T18:06:38.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:37 smithi159 ceph-mon[45336]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-15T18:06:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:37 smithi049 ceph-mon[40087]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-15T18:06:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:38 smithi082 ceph-mon[44206]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:39.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:38 smithi159 ceph-mon[45336]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:39.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:39.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:39.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:39.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:38 smithi049 ceph-mon[40087]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:06:39.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:06:39.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:06:39.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:40.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:39 smithi159 ceph-mon[45336]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:06:40.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:39 smithi049 ceph-mon[40087]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:06:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:39 smithi082 ceph-mon[44206]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:06:41.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/4119421179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:40 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:40 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]': finished 2024-04-15T18:06:41.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:40 smithi159 ceph-mon[45336]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T18:06:41.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/4119421179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:40 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:40 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]': finished 2024-04-15T18:06:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:40 smithi049 ceph-mon[40087]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T18:06:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:41.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/4119421179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:40 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]: dispatch 2024-04-15T18:06:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:40 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a"}]': finished 2024-04-15T18:06:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:40 smithi082 ceph-mon[44206]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T18:06:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:42.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:41 smithi049 ceph-mon[40087]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:06:42.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/1383223678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:42.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:41 smithi082 ceph-mon[44206]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:06:42.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/1383223678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:41 smithi159 ceph-mon[45336]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:06:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/1383223678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:06:44.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:44 smithi082 ceph-mon[44206]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:06:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:44 smithi159 ceph-mon[45336]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:06:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:44 smithi049 ceph-mon[40087]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:06:46.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:46 smithi082 ceph-mon[44206]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-15T18:06:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:46 smithi159 ceph-mon[45336]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-15T18:06:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:46 smithi049 ceph-mon[40087]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-15T18:06:47.166 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:47 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:06:47.166 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:47 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:06:47.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:47 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:06:47.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:47 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:06:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:47 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:06:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:47 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:06:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:48 smithi082 ceph-mon[44206]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T18:06:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:48 smithi082 ceph-mon[44206]: Deploying daemon osd.5 on smithi082 2024-04-15T18:06:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:48 smithi159 ceph-mon[45336]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T18:06:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:48 smithi159 ceph-mon[45336]: Deploying daemon osd.5 on smithi082 2024-04-15T18:06:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:48 smithi049 ceph-mon[40087]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T18:06:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:48 smithi049 ceph-mon[40087]: Deploying daemon osd.5 on smithi082 2024-04-15T18:06:50.062 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:50 smithi082 ceph-mon[44206]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:50 smithi159 ceph-mon[45336]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:50.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:50 smithi049 ceph-mon[40087]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:52 smithi082 ceph-mon[44206]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:52.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:52 smithi159 ceph-mon[45336]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:52 smithi049 ceph-mon[40087]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:53 smithi049 ceph-mon[40087]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:54.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:53 smithi082 ceph-mon[44206]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:53 smithi159 ceph-mon[45336]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:56.575 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 5 on host 'smithi082' 2024-04-15T18:06:56.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:56.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:56.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:56.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:57.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:06:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:06:57.339 DEBUG:teuthology.orchestra.run.smithi082:osd.5> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T18:06:57.341 INFO:tasks.cephadm:Deploying osd.6 on smithi082 with /dev/vg_nvme/lv_2... 2024-04-15T18:06:57.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T18:06:57.370 INFO:journalctl@ceph.osd.5.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:06:58.216 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:06:58.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:58.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:06:58.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T18:06:58.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:58.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:58.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:58 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:58.721 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[57379]: 2024-04-15T18:06:58.329+0000 7fc24efb0700 -1 osd.5 0 waiting for initial osdmap 2024-04-15T18:06:58.721 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[57379]: 2024-04-15T18:06:58.334+0000 7fc24ada3700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:06:58.721 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:58.721 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:06:58.722 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T18:06:58.722 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:58.722 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:58.722 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:58 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:58.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:06:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:06:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T18:06:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: from='osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:58 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:06:59.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992] boot 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T18:06:59.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:06:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:06:59.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:06:59.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992] boot 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T18:06:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:06:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:06:59.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T18:06:59.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:06:59.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: osd.5 [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992] boot 2024-04-15T18:06:59.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T18:06:59.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:06:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:07:00.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:00 smithi159 ceph-mon[45336]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:07:00.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:00 smithi159 ceph-mon[45336]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110524 s, 949 MB/s 2024-04-15T18:07:00.679 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-04-15T18:07:00.716 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:00 smithi082 ceph-mon[44206]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:07:00.717 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:00 smithi082 ceph-mon[44206]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T18:07:00.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:00 smithi049 ceph-mon[40087]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-15T18:07:00.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:00 smithi049 ceph-mon[40087]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T18:07:01.509 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:07:01.509 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:07:01.509 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:01 smithi159 ceph-mon[45336]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T18:07:01.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:07:01.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:07:01.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:01 smithi082 ceph-mon[44206]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T18:07:01.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-04-15T18:07:01.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:07:01.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:07:01.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:01 smithi049 ceph-mon[40087]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T18:07:02.316 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:07:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: pgmap v146: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: pgmap v146: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: pgmap v146: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:03.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:03 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi082 2024-04-15T18:07:03.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:03 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi082 to 8561M 2024-04-15T18:07:03.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:03 smithi082 ceph-mon[44206]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T18:07:03.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:03 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi082 2024-04-15T18:07:03.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:03 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi082 to 8561M 2024-04-15T18:07:03.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:03 smithi159 ceph-mon[45336]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T18:07:03.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:03 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi082 2024-04-15T18:07:03.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:03 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi082 to 8561M 2024-04-15T18:07:03.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:03 smithi049 ceph-mon[40087]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T18:07:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:04 smithi159 ceph-mon[45336]: pgmap v149: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:04 smithi159 ceph-mon[45336]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T18:07:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:04.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:04.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:04 smithi049 ceph-mon[40087]: pgmap v149: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:04 smithi049 ceph-mon[40087]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T18:07:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:04.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:04.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:04 smithi082 ceph-mon[44206]: pgmap v149: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:04.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:04 smithi082 ceph-mon[44206]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T18:07:04.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:04.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:04.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:05.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:05 smithi159 ceph-mon[45336]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:05.685 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:05 smithi082 ceph-mon[44206]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:05.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:05 smithi049 ceph-mon[40087]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:06.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:06 smithi159 ceph-mon[45336]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-15T18:07:06.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:06 smithi049 ceph-mon[40087]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-15T18:07:06.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:06 smithi082 ceph-mon[44206]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-15T18:07:07.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/1087514645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:07 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:07 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]': finished 2024-04-15T18:07:07.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:07 smithi159 ceph-mon[45336]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T18:07:07.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:07.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/1087514645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:07 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:07 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]': finished 2024-04-15T18:07:07.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:07 smithi049 ceph-mon[40087]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T18:07:07.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:07.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/1087514645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:07 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]: dispatch 2024-04-15T18:07:07.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:07 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783"}]': finished 2024-04-15T18:07:07.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:07 smithi082 ceph-mon[44206]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T18:07:07.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:08.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:08 smithi159 ceph-mon[45336]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-15T18:07:08.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/9671672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:08.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:08 smithi049 ceph-mon[40087]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-15T18:07:08.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/9671672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:08.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:08 smithi082 ceph-mon[44206]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-15T18:07:08.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/9671672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:10.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:10 smithi159 ceph-mon[45336]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-15T18:07:10.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:10 smithi049 ceph-mon[40087]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-15T18:07:10.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:10 smithi082 ceph-mon[44206]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-15T18:07:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:12 smithi082 ceph-mon[44206]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:12.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:12 smithi159 ceph-mon[45336]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:12 smithi049 ceph-mon[40087]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T18:07:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T18:07:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T18:07:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:14 smithi159 ceph-mon[45336]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-15T18:07:14.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:14 smithi159 ceph-mon[45336]: Deploying daemon osd.6 on smithi082 2024-04-15T18:07:14.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:14 smithi049 ceph-mon[40087]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-15T18:07:14.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:14 smithi049 ceph-mon[40087]: Deploying daemon osd.6 on smithi082 2024-04-15T18:07:14.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:14 smithi082 ceph-mon[44206]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-15T18:07:14.746 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:14 smithi082 ceph-mon[44206]: Deploying daemon osd.6 on smithi082 2024-04-15T18:07:16.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:16 smithi159 ceph-mon[45336]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:16.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:16 smithi049 ceph-mon[40087]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:16.772 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:16 smithi082 ceph-mon[44206]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:18.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:18 smithi159 ceph-mon[45336]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:18 smithi049 ceph-mon[40087]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:18 smithi082 ceph-mon[44206]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:19.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:19.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:19.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:19.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:20.902 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:20 smithi082 ceph-mon[44206]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:20.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:20 smithi159 ceph-mon[45336]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:20.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:20 smithi049 ceph-mon[40087]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:22.224 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 6 on host 'smithi082' 2024-04-15T18:07:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:22 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:22 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:22.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:22.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:22.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:22.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:22 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:07:23.011 DEBUG:teuthology.orchestra.run.smithi082:osd.6> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6.service 2024-04-15T18:07:23.014 INFO:tasks.cephadm:Deploying osd.7 on smithi082 with /dev/vg_nvme/lv_1... 2024-04-15T18:07:23.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T18:07:23.044 INFO:journalctl@ceph.osd.6.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:07:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:07:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T18:07:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:23 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:23.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:07:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T18:07:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:23 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:23.790 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[65336]: 2024-04-15T18:07:23.653+0000 7fcd70111700 -1 osd.6 0 waiting for initial osdmap 2024-04-15T18:07:23.790 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:07:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[65336]: 2024-04-15T18:07:23.660+0000 7fcd6bf04700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:07:24.448 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:07:24.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:24 smithi082 ceph-mon[44206]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:24.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:24 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:24.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:24 smithi082 ceph-mon[44206]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T18:07:24.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:24.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:24 smithi159 ceph-mon[45336]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:24 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:24 smithi159 ceph-mon[45336]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T18:07:24.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:24.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:24 smithi049 ceph-mon[40087]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-15T18:07:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:24 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:24 smithi049 ceph-mon[40087]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T18:07:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:25.941 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:07:25.941 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:07:25.941 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:25.941 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039] boot 2024-04-15T18:07:25.942 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T18:07:25.942 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039] boot 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T18:07:25.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:26.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:07:26.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:07:26.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:26.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: osd.6 [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039] boot 2024-04-15T18:07:26.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T18:07:26.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124373 s, 843 MB/s 2024-04-15T18:07:26.869 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-04-15T18:07:26.967 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:26 smithi082 ceph-mon[44206]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:26.967 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:26 smithi082 ceph-mon[44206]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T18:07:26.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:26 smithi049 ceph-mon[40087]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:26.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:26 smithi049 ceph-mon[40087]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T18:07:27.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:26 smithi159 ceph-mon[45336]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:27.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:26 smithi159 ceph-mon[45336]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T18:07:27.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:27.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_1 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:27.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:27.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:27.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:28.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:28.212 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:07:28.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:28 smithi082 ceph-mon[44206]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:28.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:28 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi082 2024-04-15T18:07:28.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:28 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi082 to 5707M 2024-04-15T18:07:28.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:28 smithi082 ceph-mon[44206]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T18:07:28.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:28 smithi049 ceph-mon[40087]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:28.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:28 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi082 2024-04-15T18:07:28.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:28 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi082 to 5707M 2024-04-15T18:07:28.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:28 smithi049 ceph-mon[40087]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T18:07:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:28 smithi159 ceph-mon[45336]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:28 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi082 2024-04-15T18:07:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:28 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi082 to 5707M 2024-04-15T18:07:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:28 smithi159 ceph-mon[45336]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T18:07:29.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:29 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:07:29.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:29 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:07:30.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:29 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:07:30.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:29 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:07:30.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:29 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:07:30.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:29 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:07:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:30 smithi049 ceph-mon[40087]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:30 smithi049 ceph-mon[40087]: osdmap e52: 7 total, 7 up, 7 in 2024-04-15T18:07:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:30.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:31.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:30 smithi082 ceph-mon[44206]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:31.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:30 smithi082 ceph-mon[44206]: osdmap e52: 7 total, 7 up, 7 in 2024-04-15T18:07:31.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:31.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:31.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:31.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:30 smithi159 ceph-mon[45336]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:31.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:30 smithi159 ceph-mon[45336]: osdmap e52: 7 total, 7 up, 7 in 2024-04-15T18:07:31.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:31.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:31.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:31.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:31 smithi082 ceph-mon[44206]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:31 smithi049 ceph-mon[40087]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:32.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:31 smithi159 ceph-mon[45336]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:32 smithi049 ceph-mon[40087]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-15T18:07:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:32 smithi082 ceph-mon[44206]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-15T18:07:33.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:32 smithi159 ceph-mon[45336]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/3419489385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]': finished 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:33.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/17797830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:34.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/3419489385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:34.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:34.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]': finished 2024-04-15T18:07:34.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T18:07:34.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:34.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/17797830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:34.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/3419489385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]: dispatch 2024-04-15T18:07:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2"}]': finished 2024-04-15T18:07:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T18:07:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:34.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/17797830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:34 smithi049 ceph-mon[40087]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:07:35.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:34 smithi082 ceph-mon[44206]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:07:35.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:34 smithi159 ceph-mon[45336]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:07:36.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:36 smithi049 ceph-mon[40087]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:07:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:36 smithi082 ceph-mon[44206]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:07:37.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:36 smithi159 ceph-mon[45336]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:07:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:38 smithi049 ceph-mon[40087]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:38 smithi082 ceph-mon[44206]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:39.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:38 smithi159 ceph-mon[45336]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:07:39.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T18:07:39.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T18:07:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:40.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T18:07:40.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:40 smithi049 ceph-mon[40087]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T18:07:40.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:40 smithi049 ceph-mon[40087]: Deploying daemon osd.7 on smithi082 2024-04-15T18:07:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:40 smithi082 ceph-mon[44206]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T18:07:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:40 smithi082 ceph-mon[44206]: Deploying daemon osd.7 on smithi082 2024-04-15T18:07:41.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:40 smithi159 ceph-mon[45336]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T18:07:41.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:40 smithi159 ceph-mon[45336]: Deploying daemon osd.7 on smithi082 2024-04-15T18:07:42.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:42 smithi049 ceph-mon[40087]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:43.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:42 smithi082 ceph-mon[44206]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:43.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:42 smithi159 ceph-mon[45336]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:44.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:44 smithi049 ceph-mon[40087]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:44.988 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:44 smithi082 ceph-mon[44206]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:44 smithi159 ceph-mon[45336]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:46 smithi159 ceph-mon[45336]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:46 smithi049 ceph-mon[40087]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:46 smithi082 ceph-mon[44206]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:48.648 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 7 on host 'smithi082' 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:49.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:49.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:49.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:07:49.337 DEBUG:teuthology.orchestra.run.smithi082:osd.7> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.7.service 2024-04-15T18:07:49.340 INFO:tasks.cephadm:Deploying osd.8 on smithi159 with /dev/vg_nvme/lv_4... 2024-04-15T18:07:49.340 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T18:07:49.368 INFO:journalctl@ceph.osd.7.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:07:49.609 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:07:50.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:07:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:49 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:07:50.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:49 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:07:50.290 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:07:49.896+0000 7f85d4a7f700 -1 osd.7 0 waiting for initial osdmap 2024-04-15T18:07:50.290 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:07:49.906+0000 7f85d0872700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: from='osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:07:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:49 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:07:50.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:50 smithi082 ceph-mon[44206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:50.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:50 smithi082 ceph-mon[44206]: osdmap e55: 8 total, 7 up, 8 in 2024-04-15T18:07:50.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:50 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:50.996 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:50 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:51.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:50 smithi159 ceph-mon[45336]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:51.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:50 smithi159 ceph-mon[45336]: osdmap e55: 8 total, 7 up, 8 in 2024-04-15T18:07:51.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:50 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:51.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:50 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:50 smithi049 ceph-mon[40087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-04-15T18:07:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:50 smithi049 ceph-mon[40087]: osdmap e55: 8 total, 7 up, 8 in 2024-04-15T18:07:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:50 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:50 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:51.785 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T18:07:51.786 INFO:teuthology.orchestra.run.smithi159.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T18:07:51.786 INFO:teuthology.orchestra.run.smithi159.stderr: stderr: 10+0 records in 2024-04-15T18:07:51.786 INFO:teuthology.orchestra.run.smithi159.stderr:10+0 records out 2024-04-15T18:07:51.786 INFO:teuthology.orchestra.run.smithi159.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112532 s, 932 MB/s 2024-04-15T18:07:51.786 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping successful for: 2024-04-15T18:07:52.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805] boot 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:52.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:51 smithi159 ceph-mon[45336]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805] boot 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:51 smithi049 ceph-mon[40087]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T18:07:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:07:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: osd.7 [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805] boot 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:51 smithi082 ceph-mon[44206]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T18:07:52.409 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi159:vg_nvme/lv_4 2024-04-15T18:07:52.668 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:07:53.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi082 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi082 to 4280M 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:53.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:53 smithi159 ceph-mon[45336]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T18:07:54.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi082 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi082 to 4280M 2024-04-15T18:07:54.042 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:54.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:54.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:53 smithi082 ceph-mon[44206]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T18:07:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi082 2024-04-15T18:07:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi082 to 4280M 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:07:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:53 smithi049 ceph-mon[40087]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T18:07:54.994 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:54 smithi159 ceph-mon[45336]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:54.994 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:54 smithi159 ceph-mon[45336]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T18:07:54.994 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:54.994 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:54.994 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:55.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:54 smithi082 ceph-mon[44206]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:55.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:54 smithi082 ceph-mon[44206]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T18:07:55.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:55.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:55.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:55.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:54 smithi049 ceph-mon[40087]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:55.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:54 smithi049 ceph-mon[40087]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T18:07:55.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:07:55.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:07:55.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:07:55.846 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:55 smithi159 ceph-mon[45336]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:55.846 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:55 smithi159 ceph-mon[45336]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T18:07:56.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:55 smithi082 ceph-mon[44206]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:56.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:55 smithi082 ceph-mon[44206]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T18:07:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:55 smithi049 ceph-mon[40087]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:07:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:55 smithi049 ceph-mon[40087]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T18:07:57.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:56 smithi082 ceph-mon[44206]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:57.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:56 smithi159 ceph-mon[45336]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:57.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:56 smithi049 ceph-mon[40087]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:07:58.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/2435237503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]': finished 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: osdmap e61: 9 total, 8 up, 9 in 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:07:58.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:07:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/452229036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/2435237503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]': finished 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: osdmap e61: 9 total, 8 up, 9 in 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:07:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:07:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/452229036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:07:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-15T18:07:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/2435237503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]: dispatch 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59"}]': finished 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: osdmap e61: 9 total, 8 up, 9 in 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:07:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:07:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/452229036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:00 smithi082 ceph-mon[44206]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:08:00.332 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:00 smithi159 ceph-mon[45336]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:08:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:00 smithi049 ceph-mon[40087]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T18:08:02.283 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:02 smithi159 ceph-mon[45336]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:02 smithi049 ceph-mon[40087]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:02 smithi082 ceph-mon[44206]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:04.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:04.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:04.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-15T18:08:04.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: osdmap e62: 9 total, 8 up, 9 in 2024-04-15T18:08:04.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:04.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-15T18:08:04.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:04.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:04 smithi159 ceph-mon[45336]: Deploying daemon osd.8 on smithi159 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: osdmap e62: 9 total, 8 up, 9 in 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:04 smithi049 ceph-mon[40087]: Deploying daemon osd.8 on smithi159 2024-04-15T18:08:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-15T18:08:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: osdmap e62: 9 total, 8 up, 9 in 2024-04-15T18:08:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-15T18:08:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:04 smithi082 ceph-mon[44206]: Deploying daemon osd.8 on smithi159 2024-04-15T18:08:06.380 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:06 smithi159 ceph-mon[45336]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:06 smithi049 ceph-mon[40087]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:06 smithi082 ceph-mon[44206]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:08.091 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:07 smithi159 ceph-mon[45336]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:07 smithi049 ceph-mon[40087]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:07 smithi082 ceph-mon[44206]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:10.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:09 smithi082 ceph-mon[44206]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:09 smithi159 ceph-mon[45336]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:10.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:09 smithi049 ceph-mon[40087]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:10.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:10.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:12.377 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:12 smithi159 ceph-mon[45336]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:12 smithi082 ceph-mon[44206]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:12.712 INFO:teuthology.orchestra.run.smithi159.stdout:Created osd(s) 8 on host 'smithi159' 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:12 smithi049 ceph-mon[40087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-15T18:08:13.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: osdmap e63: 9 total, 8 up, 9 in 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:13.471 DEBUG:teuthology.orchestra.run.smithi159:osd.8> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.8.service 2024-04-15T18:08:13.473 INFO:tasks.cephadm:Deploying osd.9 on smithi159 with /dev/vg_nvme/lv_3... 2024-04-15T18:08:13.474 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T18:08:13.505 INFO:journalctl@ceph.osd.8.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: osdmap e63: 9 total, 8 up, 9 in 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: osdmap e63: 9 total, 8 up, 9 in 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:13.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:14.158 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:08:13.682+0000 7fc8f9c25700 -1 osd.8 0 waiting for initial osdmap 2024-04-15T18:08:14.159 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:08:13.683+0000 7fc8ed39b700 -1 osd.8 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-15T18:08:14.159 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:08:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:08:13.695+0000 7fc8f1a10700 -1 osd.8 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:08:14.249 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:08:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: osdmap e64: 9 total, 8 up, 9 in 2024-04-15T18:08:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:14 smithi082 ceph-mon[44206]: from='osd.8 ' entity='osd.8' 2024-04-15T18:08:14.588 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:14.589 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:14.589 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: osdmap e64: 9 total, 8 up, 9 in 2024-04-15T18:08:14.589 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.589 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.589 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:14 smithi159 ceph-mon[45336]: from='osd.8 ' entity='osd.8' 2024-04-15T18:08:14.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:14.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:14.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: osdmap e64: 9 total, 8 up, 9 in 2024-04-15T18:08:14.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:14.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:14 smithi049 ceph-mon[40087]: from='osd.8 ' entity='osd.8' 2024-04-15T18:08:15.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130] boot 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: osdmap e65: 9 total, 9 up, 9 in 2024-04-15T18:08:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:15.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130] boot 2024-04-15T18:08:15.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: osdmap e65: 9 total, 9 up, 9 in 2024-04-15T18:08:15.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: osd.8 [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130] boot 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: osdmap e65: 9 total, 9 up, 9 in 2024-04-15T18:08:15.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-15T18:08:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:16 smithi082 ceph-mon[44206]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:16 smithi082 ceph-mon[44206]: osdmap e66: 9 total, 9 up, 9 in 2024-04-15T18:08:16.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:16 smithi159 ceph-mon[45336]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:16.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:16 smithi159 ceph-mon[45336]: osdmap e66: 9 total, 9 up, 9 in 2024-04-15T18:08:16.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:16 smithi049 ceph-mon[40087]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-15T18:08:16.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:16 smithi049 ceph-mon[40087]: osdmap e66: 9 total, 9 up, 9 in 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr: stderr: 10+0 records in 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr:10+0 records out 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0210639 s, 498 MB/s 2024-04-15T18:08:16.950 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping successful for: 2024-04-15T18:08:17.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:17 smithi082 ceph-mon[44206]: osdmap e67: 9 total, 9 up, 9 in 2024-04-15T18:08:17.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:17 smithi082 ceph-mon[44206]: osdmap e68: 9 total, 9 up, 9 in 2024-04-15T18:08:17.586 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:17 smithi159 ceph-mon[45336]: osdmap e67: 9 total, 9 up, 9 in 2024-04-15T18:08:17.586 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:17 smithi159 ceph-mon[45336]: osdmap e68: 9 total, 9 up, 9 in 2024-04-15T18:08:17.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:17 smithi049 ceph-mon[40087]: osdmap e67: 9 total, 9 up, 9 in 2024-04-15T18:08:17.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:17 smithi049 ceph-mon[40087]: osdmap e68: 9 total, 9 up, 9 in 2024-04-15T18:08:17.793 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi159:vg_nvme/lv_3 2024-04-15T18:08:18.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:18 smithi159 ceph-mon[45336]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:18.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:18 smithi159 ceph-mon[45336]: osdmap e69: 9 total, 9 up, 9 in 2024-04-15T18:08:18.500 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:08:18.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:18 smithi082 ceph-mon[44206]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:18.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:18 smithi082 ceph-mon[44206]: osdmap e69: 9 total, 9 up, 9 in 2024-04-15T18:08:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:18 smithi049 ceph-mon[40087]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:18 smithi049 ceph-mon[40087]: osdmap e69: 9 total, 9 up, 9 in 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi159 to 21218M 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:19.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:19 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi159 to 21218M 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:19 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:19.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi159 to 21218M 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:19 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:20.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:20 smithi049 ceph-mon[40087]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:20.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:20.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:20.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:20.985 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:20 smithi159 ceph-mon[45336]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:20.985 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:20.985 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:20.986 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:21.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:20 smithi082 ceph-mon[44206]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:21.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:21.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:21.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:21.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:21 smithi159 ceph-mon[45336]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:21.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:21.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:21.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:21 smithi049 ceph-mon[40087]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:21.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:21.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:22.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:21 smithi082 ceph-mon[44206]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:22.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:22.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:22.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:22 smithi049 ceph-mon[40087]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-15T18:08:22.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:22 smithi049 ceph-mon[40087]: osdmap e70: 9 total, 9 up, 9 in 2024-04-15T18:08:23.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:22 smithi082 ceph-mon[44206]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-15T18:08:23.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:22 smithi082 ceph-mon[44206]: osdmap e70: 9 total, 9 up, 9 in 2024-04-15T18:08:23.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:22 smithi159 ceph-mon[45336]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-15T18:08:23.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:22 smithi159 ceph-mon[45336]: osdmap e70: 9 total, 9 up, 9 in 2024-04-15T18:08:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/2387391446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:23 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:23 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]': finished 2024-04-15T18:08:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:23 smithi049 ceph-mon[40087]: osdmap e71: 10 total, 9 up, 10 in 2024-04-15T18:08:23.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:23 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:24.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/2387391446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:24.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:23 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:24.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:23 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]': finished 2024-04-15T18:08:24.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:23 smithi082 ceph-mon[44206]: osdmap e71: 10 total, 9 up, 10 in 2024-04-15T18:08:24.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:23 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:24.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/2387391446' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:24.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:23 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]: dispatch 2024-04-15T18:08:24.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:23 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d"}]': finished 2024-04-15T18:08:24.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:23 smithi159 ceph-mon[45336]: osdmap e71: 10 total, 9 up, 10 in 2024-04-15T18:08:24.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:23 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:24 smithi049 ceph-mon[40087]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:08:24.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/3601871449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:25.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:24 smithi082 ceph-mon[44206]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:08:25.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/3601871449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:25.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:24 smithi159 ceph-mon[45336]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-15T18:08:25.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/3601871449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:26.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:26 smithi159 ceph-mon[45336]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:26.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:26 smithi049 ceph-mon[40087]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:27.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:26 smithi082 ceph-mon[44206]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T18:08:28.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:27 smithi159 ceph-mon[45336]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:08:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:27 smithi049 ceph-mon[40087]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:08:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:27 smithi082 ceph-mon[44206]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T18:08:30.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:30 smithi159 ceph-mon[45336]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:30.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-15T18:08:30.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:30.304 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:30 smithi159 ceph-mon[45336]: Deploying daemon osd.9 on smithi159 2024-04-15T18:08:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:30 smithi049 ceph-mon[40087]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-15T18:08:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:30 smithi049 ceph-mon[40087]: Deploying daemon osd.9 on smithi159 2024-04-15T18:08:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:30 smithi082 ceph-mon[44206]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-15T18:08:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:30 smithi082 ceph-mon[44206]: Deploying daemon osd.9 on smithi159 2024-04-15T18:08:31.248 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:31.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:31.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:31 smithi159 ceph-mon[45336]: osdmap e72: 10 total, 9 up, 10 in 2024-04-15T18:08:31.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:31 smithi049 ceph-mon[40087]: osdmap e72: 10 total, 9 up, 10 in 2024-04-15T18:08:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:31 smithi082 ceph-mon[44206]: osdmap e72: 10 total, 9 up, 10 in 2024-04-15T18:08:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:32.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:32 smithi159 ceph-mon[45336]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:32 smithi049 ceph-mon[40087]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:32 smithi082 ceph-mon[44206]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:34.367 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:34 smithi159 ceph-mon[45336]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:34 smithi049 ceph-mon[40087]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:34 smithi082 ceph-mon[44206]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:36 smithi049 ceph-mon[40087]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:36 smithi082 ceph-mon[44206]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:36 smithi159 ceph-mon[45336]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: osdmap e73: 10 total, 9 up, 10 in 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:38.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:38 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: osdmap e73: 10 total, 9 up, 10 in 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:38 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: osdmap e73: 10 total, 9 up, 10 in 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.562 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:38.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:38.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:38 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:38.951 INFO:teuthology.orchestra.run.smithi159.stdout:Created osd(s) 9 on host 'smithi159' 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:39.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-15T18:08:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:08:39.693 DEBUG:teuthology.orchestra.run.smithi159:osd.9> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.9.service 2024-04-15T18:08:39.695 INFO:tasks.cephadm:Deploying osd.10 on smithi159 with /dev/vg_nvme/lv_2... 2024-04-15T18:08:39.695 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T18:08:39.727 INFO:journalctl@ceph.osd.9.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:08:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-15T18:08:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: osdmap e74: 10 total, 9 up, 10 in 2024-04-15T18:08:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-mon[45336]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: osdmap e74: 10 total, 9 up, 10 in 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:40 smithi049 ceph-mon[40087]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:40.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-15T18:08:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: osdmap e74: 10 total, 9 up, 10 in 2024-04-15T18:08:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: from='osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:40 smithi082 ceph-mon[44206]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:08:40.794 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:08:40.795 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:08:40.689+0000 7f79e4c04700 -1 osd.9 0 waiting for initial osdmap 2024-04-15T18:08:40.795 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:08:40.689+0000 7f79d837a700 -1 osd.9 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-15T18:08:40.795 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:08:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:08:40.700+0000 7f79dc9ef700 -1 osd.9 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:08:41.342 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:41 smithi159 ceph-mon[45336]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:41.342 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:41 smithi159 ceph-mon[45336]: osdmap e75: 10 total, 9 up, 10 in 2024-04-15T18:08:41.342 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.342 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.342 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:41 smithi159 ceph-mon[45336]: from='osd.9 ' entity='osd.9' 2024-04-15T18:08:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:41 smithi049 ceph-mon[40087]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:41 smithi049 ceph-mon[40087]: osdmap e75: 10 total, 9 up, 10 in 2024-04-15T18:08:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:41 smithi049 ceph-mon[40087]: from='osd.9 ' entity='osd.9' 2024-04-15T18:08:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:41 smithi082 ceph-mon[44206]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:08:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:41 smithi082 ceph-mon[44206]: osdmap e75: 10 total, 9 up, 10 in 2024-04-15T18:08:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:41.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:41 smithi082 ceph-mon[44206]: from='osd.9 ' entity='osd.9' 2024-04-15T18:08:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:08:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233] boot 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: osdmap e76: 10 total, 10 up, 10 in 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:42 smithi082 ceph-mon[44206]: osdmap e77: 10 total, 10 up, 10 in 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233] boot 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: osdmap e76: 10 total, 10 up, 10 in 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:42 smithi159 ceph-mon[45336]: osdmap e77: 10 total, 10 up, 10 in 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 249 MiB used, 804 GiB / 805 GiB avail 2024-04-15T18:08:42.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: osd.9 [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233] boot 2024-04-15T18:08:42.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: osdmap e76: 10 total, 10 up, 10 in 2024-04-15T18:08:42.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-15T18:08:42.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:42 smithi049 ceph-mon[40087]: osdmap e77: 10 total, 10 up, 10 in 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr: stderr: 10+0 records in 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr:10+0 records out 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110659 s, 948 MB/s 2024-04-15T18:08:43.835 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping successful for: 2024-04-15T18:08:44.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:44.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi159 2024-04-15T18:08:44.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi159 to 10609M 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:44 smithi159 ceph-mon[45336]: osdmap e78: 10 total, 10 up, 10 in 2024-04-15T18:08:44.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi159 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi159 to 10609M 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:44 smithi082 ceph-mon[44206]: osdmap e78: 10 total, 10 up, 10 in 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi159 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi159 to 10609M 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:08:44.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:44 smithi049 ceph-mon[40087]: osdmap e78: 10 total, 10 up, 10 in 2024-04-15T18:08:44.647 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi159:vg_nvme/lv_2 2024-04-15T18:08:44.909 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:08:46.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:46 smithi159 ceph-mon[45336]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:46.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:46 smithi082 ceph-mon[44206]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:46 smithi049 ceph-mon[40087]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:47.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:47 smithi082 ceph-mon[44206]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:47.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:47.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:47.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:47.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:47 smithi082 ceph-mon[44206]: osdmap e79: 10 total, 10 up, 10 in 2024-04-15T18:08:47.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:47 smithi159 ceph-mon[45336]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:47.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:47.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:47.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:47.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:47 smithi159 ceph-mon[45336]: osdmap e79: 10 total, 10 up, 10 in 2024-04-15T18:08:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:47 smithi049 ceph-mon[40087]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:08:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:08:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:08:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:47 smithi049 ceph-mon[40087]: osdmap e79: 10 total, 10 up, 10 in 2024-04-15T18:08:48.251 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:48 smithi159 ceph-mon[45336]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:48 smithi082 ceph-mon[44206]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:48 smithi049 ceph-mon[40087]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/3618598520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]': finished 2024-04-15T18:08:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: osdmap e80: 11 total, 10 up, 11 in 2024-04-15T18:08:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:50 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/3618598520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]': finished 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: osdmap e80: 11 total, 10 up, 11 in 2024-04-15T18:08:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:50 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/3618598520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]: dispatch 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9731146-d494-4a3c-a205-37064c685816"}]': finished 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: osdmap e80: 11 total, 10 up, 11 in 2024-04-15T18:08:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:50 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/3291045108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/3291045108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/3291045108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:08:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:52 smithi082 ceph-mon[44206]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:52 smithi082 ceph-mon[44206]: osdmap e81: 11 total, 10 up, 11 in 2024-04-15T18:08:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:52 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:52.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:52 smithi159 ceph-mon[45336]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:52.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:52 smithi159 ceph-mon[45336]: osdmap e81: 11 total, 10 up, 11 in 2024-04-15T18:08:52.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:52 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:52 smithi049 ceph-mon[40087]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:52 smithi049 ceph-mon[40087]: osdmap e81: 11 total, 10 up, 11 in 2024-04-15T18:08:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:52 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:54 smithi082 ceph-mon[44206]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:54 smithi159 ceph-mon[45336]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:54.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:54 smithi049 ceph-mon[40087]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:56 smithi082 ceph-mon[44206]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-15T18:08:56.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:56 smithi159 ceph-mon[45336]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-15T18:08:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:56 smithi049 ceph-mon[40087]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-15T18:08:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:08:57.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:57 smithi082 ceph-mon[44206]: Deploying daemon osd.10 on smithi159 2024-04-15T18:08:57.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:57 smithi082 ceph-mon[44206]: osdmap e82: 11 total, 10 up, 11 in 2024-04-15T18:08:57.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:57 smithi159 ceph-mon[45336]: Deploying daemon osd.10 on smithi159 2024-04-15T18:08:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:57 smithi159 ceph-mon[45336]: osdmap e82: 11 total, 10 up, 11 in 2024-04-15T18:08:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:57.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:57 smithi049 ceph-mon[40087]: Deploying daemon osd.10 on smithi159 2024-04-15T18:08:57.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:57 smithi049 ceph-mon[40087]: osdmap e82: 11 total, 10 up, 11 in 2024-04-15T18:08:57.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:08:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:08:58 smithi082 ceph-mon[44206]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:08:58 smithi159 ceph-mon[45336]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:08:58.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:08:58 smithi049 ceph-mon[40087]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:00 smithi082 ceph-mon[44206]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:00 smithi159 ceph-mon[45336]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:00 smithi049 ceph-mon[40087]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:02.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:02.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:02.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:02.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:02.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:03.710 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:03 smithi159 ceph-mon[45336]: osdmap e83: 11 total, 10 up, 11 in 2024-04-15T18:09:03.710 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:03.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:03 smithi049 ceph-mon[40087]: osdmap e83: 11 total, 10 up, 11 in 2024-04-15T18:09:03.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:03.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:03 smithi082 ceph-mon[44206]: osdmap e83: 11 total, 10 up, 11 in 2024-04-15T18:09:03.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:04.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:04.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:04 smithi159 ceph-mon[45336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:05.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:05.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:05.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:05.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:05.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:04 smithi049 ceph-mon[40087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:04 smithi082 ceph-mon[44206]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-15T18:09:05.407 INFO:teuthology.orchestra.run.smithi159.stdout:Created osd(s) 10 on host 'smithi159' 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: osdmap e84: 11 total, 10 up, 11 in 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.129 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.130 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:06.163 DEBUG:teuthology.orchestra.run.smithi159:osd.10> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.10.service 2024-04-15T18:09:06.165 INFO:tasks.cephadm:Deploying osd.11 on smithi159 with /dev/vg_nvme/lv_1... 2024-04-15T18:09:06.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T18:09:06.198 INFO:journalctl@ceph.osd.10.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: osdmap e84: 11 total, 10 up, 11 in 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-15T18:09:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: osdmap e84: 11 total, 10 up, 11 in 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:06.408 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:09:06.273+0000 7f53fe371700 -1 osd.10 0 waiting for initial osdmap 2024-04-15T18:09:06.409 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:09:06.274+0000 7f53f1ae7700 -1 osd.10 0 failed to load OSD map for epoch 10, got 0 bytes 2024-04-15T18:09:06.409 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:09:06.287+0000 7f53f615c700 -1 osd.10 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:09:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: osdmap e85: 11 total, 10 up, 11 in 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:06 smithi159 ceph-mon[45336]: from='osd.10 ' entity='osd.10' 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: osdmap e85: 11 total, 10 up, 11 in 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:06 smithi049 ceph-mon[40087]: from='osd.10 ' entity='osd.10' 2024-04-15T18:09:07.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: osdmap e85: 11 total, 10 up, 11 in 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:06 smithi082 ceph-mon[44206]: from='osd.10 ' entity='osd.10' 2024-04-15T18:09:07.397 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:09:07.963 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187] boot 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: osdmap e86: 11 total, 11 up, 11 in 2024-04-15T18:09:07.964 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187] boot 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: osdmap e86: 11 total, 11 up, 11 in 2024-04-15T18:09:08.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: osd.10 [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187] boot 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: osdmap e86: 11 total, 11 up, 11 in 2024-04-15T18:09:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-15T18:09:09.104 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:08 smithi159 ceph-mon[45336]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:09.104 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:08 smithi159 ceph-mon[45336]: osdmap e87: 11 total, 11 up, 11 in 2024-04-15T18:09:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:08 smithi049 ceph-mon[40087]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:08 smithi049 ceph-mon[40087]: osdmap e87: 11 total, 11 up, 11 in 2024-04-15T18:09:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:08 smithi082 ceph-mon[44206]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 894 GiB / 894 GiB avail 2024-04-15T18:09:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:08 smithi082 ceph-mon[44206]: osdmap e87: 11 total, 11 up, 11 in 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr: stderr: 10+0 records in 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr:10+0 records out 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111838 s, 938 MB/s 2024-04-15T18:09:10.151 INFO:teuthology.orchestra.run.smithi159.stderr:--> Zapping successful for: 2024-04-15T18:09:10.878 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:10.878 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:10.878 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:10.878 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:10.878 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:10.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:10.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi159:vg_nvme/lv_1 2024-04-15T18:09:11.163 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:11.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:11.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:11.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:11.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:11.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:11 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi159 2024-04-15T18:09:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:11 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi159 to 7072M 2024-04-15T18:09:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:11 smithi159 ceph-mon[45336]: osdmap e88: 11 total, 11 up, 11 in 2024-04-15T18:09:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:11 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi159 2024-04-15T18:09:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:11 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi159 to 7072M 2024-04-15T18:09:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:11 smithi049 ceph-mon[40087]: osdmap e88: 11 total, 11 up, 11 in 2024-04-15T18:09:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:11 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi159 2024-04-15T18:09:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:11 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi159 to 7072M 2024-04-15T18:09:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:11 smithi082 ceph-mon[44206]: osdmap e88: 11 total, 11 up, 11 in 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=53,logm=12} crc {auth=4105388244,config=1548763834,health=3897446751,kv=3952097922,logm=4135420642}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053546506}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262995054}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603596451}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061624899}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444144461}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136241786}) 2024-04-15T18:09:12.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689151476}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707358963}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2192234141}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183051059}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339180085}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784311653}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460123291}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630896456}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873982037}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=21} crc {logm=997188538,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1331630137,mon_config_key=1563093113}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4132404736}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2211996771}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=57,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=26} crc {mon_config_key=591522155,monmap=1029477071,osd_metadata=2980568239,osd_pg_creating=3162366595,osdmap=422789363}) 2024-04-15T18:09:12.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:12 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=94072402}) 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=53,logm=12} crc {auth=4105388244,config=1548763834,health=3897446751,kv=3952097922,logm=4135420642}) 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053546506}) 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262995054}) 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603596451}) 2024-04-15T18:09:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061624899}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444144461}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136241786}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689151476}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707358963}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2192234141}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183051059}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339180085}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784311653}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460123291}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630896456}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873982037}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=21} crc {logm=997188538,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1331630137,mon_config_key=1563093113}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4132404736}) 2024-04-15T18:09:13.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2211996771}) 2024-04-15T18:09:13.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=57,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=26} crc {mon_config_key=591522155,monmap=1029477071,osd_metadata=2980568239,osd_pg_creating=3162366595,osdmap=422789363}) 2024-04-15T18:09:13.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:12 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=94072402}) 2024-04-15T18:09:13.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:13.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=53,logm=12} crc {auth=4105388244,config=1548763834,health=3897446751,kv=3952097922,logm=4135420642}) 2024-04-15T18:09:13.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1053546506}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262995054}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603596451}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061624899}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3444144461}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4136241786}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689151476}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707358963}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2192234141}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3183051059}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339180085}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784311653}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460123291}) 2024-04-15T18:09:13.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2630896456}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873982037}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=44,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=21} crc {logm=997188538,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1331630137,mon_config_key=1563093113}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4132404736}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2211996771}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=57,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=26} crc {mon_config_key=591522155,monmap=1029477071,osd_metadata=2980568239,osd_pg_creating=3162366595,osdmap=422789363}) 2024-04-15T18:09:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:12 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=94072402}) 2024-04-15T18:09:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:09:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:09:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:09:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:09:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:14.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T18:09:14.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T18:09:14.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:14.974 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:14 smithi159 ceph-mon[45336]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:14.974 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:14 smithi159 ceph-mon[45336]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:09:15.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:14 smithi049 ceph-mon[40087]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:15.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:14 smithi049 ceph-mon[40087]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:09:15.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:14 smithi082 ceph-mon[44206]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:15.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:14 smithi082 ceph-mon[44206]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi159:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T18:09:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/3106409229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]': finished 2024-04-15T18:09:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:15 smithi159 ceph-mon[45336]: osdmap e89: 12 total, 11 up, 12 in 2024-04-15T18:09:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/3106409229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]': finished 2024-04-15T18:09:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:15 smithi049 ceph-mon[40087]: osdmap e89: 12 total, 11 up, 12 in 2024-04-15T18:09:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/3106409229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]: dispatch 2024-04-15T18:09:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118"}]': finished 2024-04-15T18:09:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:15 smithi082 ceph-mon[44206]: osdmap e89: 12 total, 11 up, 12 in 2024-04-15T18:09:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:16 smithi159 ceph-mon[45336]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/823410103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:09:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:16 smithi049 ceph-mon[40087]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/823410103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:09:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:16 smithi082 ceph-mon[44206]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/823410103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T18:09:19.031 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:18 smithi159 ceph-mon[45336]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:19.031 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:18 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:09:19.031 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:18 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:09:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:18 smithi049 ceph-mon[40087]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:18 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:09:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:18 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:09:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:18 smithi082 ceph-mon[44206]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:18 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:09:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:18 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:09:21.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:20 smithi159 ceph-mon[45336]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:21.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:20 smithi049 ceph-mon[40087]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:20 smithi082 ceph-mon[44206]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:23.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:22 smithi159 ceph-mon[45336]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-15T18:09:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:22 smithi049 ceph-mon[40087]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-15T18:09:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:22 smithi082 ceph-mon[44206]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:23.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-15T18:09:23.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:24.046 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:23 smithi159 ceph-mon[45336]: Deploying daemon osd.11 on smithi159 2024-04-15T18:09:24.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:23 smithi049 ceph-mon[40087]: Deploying daemon osd.11 on smithi159 2024-04-15T18:09:24.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:23 smithi082 ceph-mon[44206]: Deploying daemon osd.11 on smithi159 2024-04-15T18:09:25.139 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:24 smithi159 ceph-mon[45336]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:25.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:24 smithi049 ceph-mon[40087]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:25.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:24 smithi082 ceph-mon[44206]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:27.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:26 smithi159 ceph-mon[45336]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:27.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:27.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:26 smithi049 ceph-mon[40087]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:27.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:27.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:27.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:26 smithi082 ceph-mon[44206]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:27.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:27.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:28.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:27 smithi159 ceph-mon[45336]: osdmap e90: 12 total, 11 up, 12 in 2024-04-15T18:09:28.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:27 smithi049 ceph-mon[40087]: osdmap e90: 12 total, 11 up, 12 in 2024-04-15T18:09:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:27 smithi082 ceph-mon[44206]: osdmap e90: 12 total, 11 up, 12 in 2024-04-15T18:09:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:29.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:28 smithi159 ceph-mon[45336]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:28 smithi159 ceph-mon[45336]: osdmap e91: 12 total, 11 up, 12 in 2024-04-15T18:09:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:28 smithi049 ceph-mon[40087]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:28 smithi049 ceph-mon[40087]: osdmap e91: 12 total, 11 up, 12 in 2024-04-15T18:09:29.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:28 smithi082 ceph-mon[44206]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:28 smithi082 ceph-mon[44206]: osdmap e91: 12 total, 11 up, 12 in 2024-04-15T18:09:29.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:30.209 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:29 smithi159 ceph-mon[45336]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:30.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:29 smithi049 ceph-mon[40087]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:29 smithi082 ceph-mon[44206]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:31.394 INFO:teuthology.orchestra.run.smithi159.stdout:Created osd(s) 11 on host 'smithi159' 2024-04-15T18:09:31.784 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.784 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.784 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:31.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:32.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:32.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:32.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:32.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:09:32.151 DEBUG:teuthology.orchestra.run.smithi159:osd.11> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11.service 2024-04-15T18:09:32.153 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-15T18:09:32.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-15T18:09:32.184 INFO:journalctl@ceph.osd.11.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:09:32.440 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:32.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:09:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: osdmap e92: 12 total, 11 up, 12 in 2024-04-15T18:09:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:32.654 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:09:32.599+0000 7fde27363700 -1 osd.11 0 waiting for initial osdmap 2024-04-15T18:09:32.654 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:09:32.599+0000 7fde1eae1700 -1 osd.11 0 failed to load OSD map for epoch 37, got 0 bytes 2024-04-15T18:09:32.654 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:09:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:09:32.610+0000 7fde23156700 -1 osd.11 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: osdmap e92: 12 total, 11 up, 12 in 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:32.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:32 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:33.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:09:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: osdmap e92: 12 total, 11 up, 12 in 2024-04-15T18:09:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: from='osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:33.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:32 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:09:33.624 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:33 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:33.625 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:33 smithi159 ceph-mon[45336]: osdmap e93: 12 total, 11 up, 12 in 2024-04-15T18:09:33.625 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:33.625 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:33.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:33 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:33.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:33 smithi049 ceph-mon[40087]: osdmap e93: 12 total, 11 up, 12 in 2024-04-15T18:09:33.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:33.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:34.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:33 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]': finished 2024-04-15T18:09:34.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:33 smithi082 ceph-mon[44206]: osdmap e93: 12 total, 11 up, 12 in 2024-04-15T18:09:34.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:34.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:34.681 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:09:34.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647] boot 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: osdmap e94: 12 total, 12 up, 12 in 2024-04-15T18:09:34.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:34 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647] boot 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: osdmap e94: 12 total, 12 up, 12 in 2024-04-15T18:09:34.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:34 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:35.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:35.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:35.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:09:35.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:09:35.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: osd.11 [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647] boot 2024-04-15T18:09:35.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: osdmap e94: 12 total, 12 up, 12 in 2024-04-15T18:09:35.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:34 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:09:35.563 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":95,"num_osds":12,"num_up_osds":12,"osd_up_since":1713204573,"num_in_osds":12,"osd_in_since":1713204555,"num_remapped_pgs":1} 2024-04-15T18:09:35.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:09:35.838 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: osdmap e95: 12 total, 12 up, 12 in 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4287950526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:35.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:35.910 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: osdmap e95: 12 total, 12 up, 12 in 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4287950526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:35.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:36.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: osdmap e95: 12 total, 12 up, 12 in 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4287950526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:09:36.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:09:36.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:36 smithi159 ceph-mon[45336]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:36.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:36 smithi159 ceph-mon[45336]: Detected new or changed devices on smithi159 2024-04-15T18:09:36.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:36 smithi159 ceph-mon[45336]: Adjusting osd_memory_target on smithi159 to 5304M 2024-04-15T18:09:36.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:36 smithi159 ceph-mon[45336]: osdmap e96: 12 total, 12 up, 12 in 2024-04-15T18:09:36.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:36 smithi049 ceph-mon[40087]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:36.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:36 smithi049 ceph-mon[40087]: Detected new or changed devices on smithi159 2024-04-15T18:09:36.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:36 smithi049 ceph-mon[40087]: Adjusting osd_memory_target on smithi159 to 5304M 2024-04-15T18:09:36.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:36 smithi049 ceph-mon[40087]: osdmap e96: 12 total, 12 up, 12 in 2024-04-15T18:09:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:36 smithi082 ceph-mon[44206]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:36 smithi082 ceph-mon[44206]: Detected new or changed devices on smithi159 2024-04-15T18:09:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:36 smithi082 ceph-mon[44206]: Adjusting osd_memory_target on smithi159 to 5304M 2024-04-15T18:09:37.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:36 smithi082 ceph-mon[44206]: osdmap e96: 12 total, 12 up, 12 in 2024-04-15T18:09:37.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:37 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:37.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:37 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:37.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:37 smithi049 ceph-mon[40087]: osdmap e97: 12 total, 12 up, 12 in 2024-04-15T18:09:37.892 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:09:37.892 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":98,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:09:37.617455+0000","last_up_change":"2024-04-15T18:09:33.592748+0000","last_in_change":"2024-04-15T18:09:15.827520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6819","nonce":1108239618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6821","nonce":1108239618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6825","nonce":1108239618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6823","nonce":1108239618}]},"public_addr":"172.21.15.49:6819/1108239618","cluster_addr":"172.21.15.49:6821/1108239618","heartbeat_back_addr":"172.21.15.49:6825/1108239618","heartbeat_front_addr":"172.21.15.49:6823/1108239618","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6827","nonce":4051312081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6829","nonce":4051312081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6833","nonce":4051312081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6831","nonce":4051312081}]},"public_addr":"172.21.15.49:6827/4051312081","cluster_addr":"172.21.15.49:6829/4051312081","heartbeat_back_addr":"172.21.15.49:6833/4051312081","heartbeat_front_addr":"172.21.15.49:6831/4051312081","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6809","nonce":1943742992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6811","nonce":1943742992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6815","nonce":1943742992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6813","nonce":1943742992}]},"public_addr":"172.21.15.82:6809/1943742992","cluster_addr":"172.21.15.82:6811/1943742992","heartbeat_back_addr":"172.21.15.82:6815/1943742992","heartbeat_front_addr":"172.21.15.82:6813/1943742992","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:27.844918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:09.854112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:36.130986+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:37.083118+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:02.278752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:26.073684+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:38.644149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.394204+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:13.191298+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:39.347535+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.593932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:31.827232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:09:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:37 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:37 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:37 smithi159 ceph-mon[45336]: osdmap e97: 12 total, 12 up, 12 in 2024-04-15T18:09:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:37 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:37 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:38.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:37 smithi082 ceph-mon[44206]: osdmap e97: 12 total, 12 up, 12 in 2024-04-15T18:09:38.666 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T18:09:38.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-15T18:09:38.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:38 smithi159 ceph-mon[45336]: pgmap v281: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:38.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:38 smithi159 ceph-mon[45336]: osdmap e98: 12 total, 12 up, 12 in 2024-04-15T18:09:38.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4171880114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:38.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:38.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:38.940 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:38 smithi049 ceph-mon[40087]: pgmap v281: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:38 smithi049 ceph-mon[40087]: osdmap e98: 12 total, 12 up, 12 in 2024-04-15T18:09:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4171880114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:38.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:38 smithi082 ceph-mon[44206]: pgmap v281: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:39.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:38 smithi082 ceph-mon[44206]: osdmap e98: 12 total, 12 up, 12 in 2024-04-15T18:09:39.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4171880114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:39.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:39.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:40.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:39 smithi049 ceph-mon[40087]: osdmap e99: 12 total, 12 up, 12 in 2024-04-15T18:09:40.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:39 smithi049 ceph-mon[40087]: pgmap v284: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:39 smithi082 ceph-mon[44206]: osdmap e99: 12 total, 12 up, 12 in 2024-04-15T18:09:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:39 smithi082 ceph-mon[44206]: pgmap v284: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:39 smithi159 ceph-mon[45336]: osdmap e99: 12 total, 12 up, 12 in 2024-04-15T18:09:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:39 smithi159 ceph-mon[45336]: pgmap v284: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T18:09:41.129 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-04-15T18:09:41.949 INFO:tasks.cephadm:Setting up client nodes... 2024-04-15T18:09:41.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T18:09:42.226 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:42.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:42 smithi049 ceph-mon[40087]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-15T18:09:42.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3939741958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T18:09:42.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:42 smithi159 ceph-mon[45336]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-15T18:09:42.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3939741958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T18:09:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:42 smithi082 ceph-mon[44206]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-15T18:09:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3939741958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T18:09:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:44 smithi159 ceph-mon[45336]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:44 smithi049 ceph-mon[40087]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:44 smithi082 ceph-mon[44206]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:44.692 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-04-15T18:09:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBobR1mpeAPKRAA08XawDI91fENRh377R2szQ== 2024-04-15T18:09:45.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:45.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:45.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2495256367' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:45.603 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-15T18:09:45.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-15T18:09:45.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T18:09:45.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T18:09:45.922 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:09:46.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:46 smithi049 ceph-mon[40087]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:47.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:46 smithi082 ceph-mon[44206]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:46 smithi159 ceph-mon[45336]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:48.022 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2024-04-15T18:09:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQBsbR1mcF/3ABAAVIRbQWAzmSpLFLnef3Bakw== 2024-04-15T18:09:48.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:47 smithi049 ceph-mon[40087]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:47 smithi082 ceph-mon[44206]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:47 smithi159 ceph-mon[45336]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:48.771 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-15T18:09:48.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-15T18:09:48.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T18:09:48.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T18:09:49.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.82:0/2850065336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:48 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:48 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:49.091 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:09:49.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.82:0/2850065336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:48 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:48 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.82:0/2850065336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:48 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:48 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:49 smithi049 ceph-mon[40087]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:50.275 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:49 smithi159 ceph-mon[45336]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:49 smithi082 ceph-mon[44206]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:51.153 INFO:teuthology.orchestra.run.smithi159.stdout:[client.2] 2024-04-15T18:09:51.153 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBvbR1mJoClCBAAvI+I1D/n9gKHbwto6p9fUQ== 2024-04-15T18:09:51.834 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-15T18:09:51.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-04-15T18:09:51.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-15T18:09:51.874 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T18:09:51.874 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-15T18:09:51.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-15T18:09:52.161 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:52 smithi049 ceph-mon[40087]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.159:0/2314399717' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:52.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:52 smithi159 ceph-mon[45336]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.159:0/2314399717' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:52.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:52 smithi082 ceph-mon[44206]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:52.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.159:0/2314399717' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T18:09:52.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T18:09:54.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:54 smithi049 ceph-mon[40087]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:54.393 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:09:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:54 smithi159 ceph-mon[45336]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:54.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:54 smithi082 ceph-mon[44206]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:55.266 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":15,"flags":0,"active_gid":14151,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":1934058225},{"type":"v1","addr":"172.21.15.49:6801","nonce":1934058225}]},"active_addr":"172.21.15.49:6801/1934058225","active_change":"2024-04-15T18:03:08.925984+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.49:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1644848147}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2496137843}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1177648564}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1753058100}]}]} 2024-04-15T18:09:55.270 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-15T18:09:55.270 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:09:55.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:09:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:55 smithi159 ceph-mon[45336]: osdmap e100: 12 total, 12 up, 12 in 2024-04-15T18:09:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4177868459' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T18:09:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:55 smithi049 ceph-mon[40087]: osdmap e100: 12 total, 12 up, 12 in 2024-04-15T18:09:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4177868459' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T18:09:55.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:55.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:55.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:55 smithi082 ceph-mon[44206]: osdmap e100: 12 total, 12 up, 12 in 2024-04-15T18:09:55.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4177868459' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T18:09:55.545 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:09:56.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:56 smithi159 ceph-mon[45336]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:56 smithi049 ceph-mon[40087]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:56 smithi082 ceph-mon[44206]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:09:57.828 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":101,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:09:57.077270+0000","last_up_change":"2024-04-15T18:09:33.592748+0000","last_in_change":"2024-04-15T18:09:15.827520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6819","nonce":1108239618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6821","nonce":1108239618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6825","nonce":1108239618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6823","nonce":1108239618}]},"public_addr":"172.21.15.49:6819/1108239618","cluster_addr":"172.21.15.49:6821/1108239618","heartbeat_back_addr":"172.21.15.49:6825/1108239618","heartbeat_front_addr":"172.21.15.49:6823/1108239618","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6827","nonce":4051312081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6829","nonce":4051312081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6833","nonce":4051312081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6831","nonce":4051312081}]},"public_addr":"172.21.15.49:6827/4051312081","cluster_addr":"172.21.15.49:6829/4051312081","heartbeat_back_addr":"172.21.15.49:6833/4051312081","heartbeat_front_addr":"172.21.15.49:6831/4051312081","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6809","nonce":1943742992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6811","nonce":1943742992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6815","nonce":1943742992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6813","nonce":1943742992}]},"public_addr":"172.21.15.82:6809/1943742992","cluster_addr":"172.21.15.82:6811/1943742992","heartbeat_back_addr":"172.21.15.82:6815/1943742992","heartbeat_front_addr":"172.21.15.82:6813/1943742992","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:27.844918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:09.854112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:36.130986+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:56.691645+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:02.278752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:26.073684+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:38.644149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.394204+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:38.021619+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:53.677079+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.593932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:31.827232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:09:58.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:09:58.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:09:58.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:58 smithi049 ceph-mon[40087]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:58.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:58 smithi049 ceph-mon[40087]: osdmap e101: 12 total, 12 up, 12 in 2024-04-15T18:09:58.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:09:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2226256177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:09:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:09:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:58 smithi159 ceph-mon[45336]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:58 smithi159 ceph-mon[45336]: osdmap e101: 12 total, 12 up, 12 in 2024-04-15T18:09:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:09:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2226256177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:58.512 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:09:58.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:09:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:09:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:09:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:58 smithi082 ceph-mon[44206]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:09:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:58 smithi082 ceph-mon[44206]: osdmap e101: 12 total, 12 up, 12 in 2024-04-15T18:09:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:09:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2226256177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:09:58.788 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:00 smithi159 ceph-mon[45336]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:00 smithi159 ceph-mon[45336]: overall HEALTH_OK 2024-04-15T18:10:00.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:00 smithi049 ceph-mon[40087]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:00.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:00 smithi049 ceph-mon[40087]: overall HEALTH_OK 2024-04-15T18:10:00.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:00 smithi082 ceph-mon[44206]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:00.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:00 smithi082 ceph-mon[44206]: overall HEALTH_OK 2024-04-15T18:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":101,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:09:57.077270+0000","last_up_change":"2024-04-15T18:09:33.592748+0000","last_in_change":"2024-04-15T18:09:15.827520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6819","nonce":1108239618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6821","nonce":1108239618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6825","nonce":1108239618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6823","nonce":1108239618}]},"public_addr":"172.21.15.49:6819/1108239618","cluster_addr":"172.21.15.49:6821/1108239618","heartbeat_back_addr":"172.21.15.49:6825/1108239618","heartbeat_front_addr":"172.21.15.49:6823/1108239618","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6827","nonce":4051312081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6829","nonce":4051312081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6833","nonce":4051312081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6831","nonce":4051312081}]},"public_addr":"172.21.15.49:6827/4051312081","cluster_addr":"172.21.15.49:6829/4051312081","heartbeat_back_addr":"172.21.15.49:6833/4051312081","heartbeat_front_addr":"172.21.15.49:6831/4051312081","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6809","nonce":1943742992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6811","nonce":1943742992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6815","nonce":1943742992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6813","nonce":1943742992}]},"public_addr":"172.21.15.82:6809/1943742992","cluster_addr":"172.21.15.82:6811/1943742992","heartbeat_back_addr":"172.21.15.82:6815/1943742992","heartbeat_front_addr":"172.21.15.82:6813/1943742992","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:27.844918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:09.854112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:36.130986+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:56.691645+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:02.278752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:26.073684+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:08:38.644149+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.394204+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:38.021619+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:53.677079+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:05.593932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:09:31.827232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:10:01.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3044163403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:10:01.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3044163403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:10:01.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3044163403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:10:01.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T18:10:01.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T18:10:01.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T18:10:01.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T18:10:01.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T18:10:01.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T18:10:01.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-15T18:10:01.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-15T18:10:01.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-15T18:10:01.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-15T18:10:01.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-15T18:10:01.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-15T18:10:02.235 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:02 smithi049 ceph-mon[40087]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:02 smithi159 ceph-mon[45336]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:02.435 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:02 smithi082 ceph-mon[44206]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:04.025 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:04.027 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:04.031 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:04.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:04 smithi049 ceph-mon[40087]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:04.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:04 smithi159 ceph-mon[45336]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:04 smithi082 ceph-mon[44206]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:06 smithi159 ceph-mon[45336]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:06 smithi049 ceph-mon[40087]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:06 smithi082 ceph-mon[44206]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:06.549 INFO:teuthology.orchestra.run.smithi049.stdout:55834574910 2024-04-15T18:10:06.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T18:10:06.878 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:06.878 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:06.880 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:07.607 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:07.608 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:08.210 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:07 smithi049 ceph-mon[40087]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:07 smithi082 ceph-mon[44206]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:07 smithi159 ceph-mon[45336]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:08.828 INFO:teuthology.orchestra.run.smithi049.stdout:240518168605 2024-04-15T18:10:08.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-15T18:10:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:08 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:08 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:08 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:08 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:09.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:08 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:09.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:08 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:09.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:09.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:09.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:09 smithi049 ceph-mon[40087]: osdmap e102: 12 total, 12 up, 12 in 2024-04-15T18:10:09.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:09 smithi049 ceph-mon[40087]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:10.158 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:10.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:09 smithi082 ceph-mon[44206]: osdmap e102: 12 total, 12 up, 12 in 2024-04-15T18:10:10.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:09 smithi082 ceph-mon[44206]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:09 smithi159 ceph-mon[45336]: osdmap e102: 12 total, 12 up, 12 in 2024-04-15T18:10:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:09 smithi159 ceph-mon[45336]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: osdmap e103: 12 total, 12 up, 12 in 2024-04-15T18:10:12.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:11 smithi049 ceph-mon[40087]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: osdmap e103: 12 total, 12 up, 12 in 2024-04-15T18:10:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:11 smithi082 ceph-mon[44206]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:12.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: osdmap e103: 12 total, 12 up, 12 in 2024-04-15T18:10:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:11 smithi159 ceph-mon[45336]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:14.421 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:14 smithi159 ceph-mon[45336]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:14 smithi049 ceph-mon[40087]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:14 smithi082 ceph-mon[44206]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:14.679 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:15.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:15.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:15.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:15 smithi049 ceph-mon[40087]: osdmap e104: 12 total, 12 up, 12 in 2024-04-15T18:10:15.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:15.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:15.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:15 smithi159 ceph-mon[45336]: osdmap e104: 12 total, 12 up, 12 in 2024-04-15T18:10:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:15 smithi082 ceph-mon[44206]: osdmap e104: 12 total, 12 up, 12 in 2024-04-15T18:10:15.868 INFO:teuthology.orchestra.run.smithi049.stdout:133143986221 2024-04-15T18:10:15.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T18:10:16.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:16 smithi049 ceph-mon[40087]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:16 smithi159 ceph-mon[45336]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:16 smithi082 ceph-mon[44206]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:18.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:18 smithi049 ceph-mon[40087]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:18 smithi159 ceph-mon[45336]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:18.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:18 smithi082 ceph-mon[44206]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:20.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:20 smithi049 ceph-mon[40087]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:20.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:20 smithi159 ceph-mon[45336]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:20.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:20 smithi082 ceph-mon[44206]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:21.517 INFO:teuthology.orchestra.run.smithi049.stdout:167503724585 2024-04-15T18:10:21.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T18:10:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:22 smithi159 ceph-mon[45336]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:22.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:22 smithi049 ceph-mon[40087]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:22 smithi082 ceph-mon[44206]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:24.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:24 smithi049 ceph-mon[40087]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:24.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4233994182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:10:24.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:24 smithi159 ceph-mon[45336]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:24.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4233994182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:10:24.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:24 smithi082 ceph-mon[44206]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:24.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4233994182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:10:24.765 INFO:teuthology.orchestra.run.smithi049.stdout:403726925837 2024-04-15T18:10:24.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-15T18:10:25.975 INFO:teuthology.orchestra.run.smithi049.stdout:55834574914 2024-04-15T18:10:26.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:26 smithi049 ceph-mon[40087]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:26.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:26 smithi159 ceph-mon[45336]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:26.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:26 smithi082 ceph-mon[44206]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:26.846 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574914 for osd.1 2024-04-15T18:10:26.846 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:27.129 INFO:teuthology.orchestra.run.smithi049.stdout:369367187474 2024-04-15T18:10:27.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-15T18:10:28.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:27 smithi049 ceph-mon[40087]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:28.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:27 smithi082 ceph-mon[44206]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:27 smithi159 ceph-mon[45336]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:28.723 INFO:teuthology.orchestra.run.smithi049.stdout:279172874269 2024-04-15T18:10:28.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-15T18:10:29.299 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:30.021 INFO:teuthology.orchestra.run.smithi049.stdout:77309411387 2024-04-15T18:10:30.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T18:10:30.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:30 smithi159 ceph-mon[45336]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:30 smithi049 ceph-mon[40087]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:30.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:30 smithi082 ceph-mon[44206]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:31.148 INFO:teuthology.orchestra.run.smithi049.stdout:111669149750 2024-04-15T18:10:31.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T18:10:31.538 INFO:teuthology.orchestra.run.smithi049.stdout:326417514521 2024-04-15T18:10:31.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-15T18:10:32.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:32 smithi049 ceph-mon[40087]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:32 smithi159 ceph-mon[45336]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:32 smithi082 ceph-mon[44206]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:32.569 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:32.623 INFO:teuthology.orchestra.run.smithi049.stdout:201863462951 2024-04-15T18:10:32.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-15T18:10:32.971 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:33 smithi049 ceph-mon[40087]: osdmap e105: 12 total, 12 up, 12 in 2024-04-15T18:10:33.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:33 smithi159 ceph-mon[45336]: osdmap e105: 12 total, 12 up, 12 in 2024-04-15T18:10:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:33 smithi082 ceph-mon[44206]: osdmap e105: 12 total, 12 up, 12 in 2024-04-15T18:10:34.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:34 smithi159 ceph-mon[45336]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:34.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:34 smithi049 ceph-mon[40087]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:34 smithi082 ceph-mon[44206]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:35.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:35 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:35.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:35 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:35.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:35 smithi159 ceph-mon[45336]: osdmap e106: 12 total, 12 up, 12 in 2024-04-15T18:10:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:35 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:35 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:35 smithi049 ceph-mon[40087]: osdmap e106: 12 total, 12 up, 12 in 2024-04-15T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:35 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:35 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:35.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:35 smithi082 ceph-mon[44206]: osdmap e106: 12 total, 12 up, 12 in 2024-04-15T18:10:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:35.809 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:35.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:35.819 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:35.920 INFO:teuthology.orchestra.run.smithi049.stdout:167503724589 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2316424898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:10:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2316424898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:10:36.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:10:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:10:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2316424898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:10:36.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:10:36.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:10:36.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:10:37.235 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724585 got 167503724589 for osd.5 2024-04-15T18:10:37.235 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:38.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:38 smithi049 ceph-mon[40087]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:38.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:38 smithi159 ceph-mon[45336]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:38 smithi082 ceph-mon[44206]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:40 smithi159 ceph-mon[45336]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:40 smithi049 ceph-mon[40087]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:40.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:40 smithi082 ceph-mon[44206]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:40.583 INFO:teuthology.orchestra.run.smithi049.stdout:240518168612 2024-04-15T18:10:41.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1409707642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:10:41.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1409707642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:10:41.413 INFO:teuthology.orchestra.run.smithi049.stdout:133143986227 2024-04-15T18:10:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1409707642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:10:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:42 smithi159 ceph-mon[45336]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:42.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1530831543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:10:42.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:42 smithi049 ceph-mon[40087]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:42.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1530831543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:10:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:42 smithi082 ceph-mon[44206]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1530831543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:10:43.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:43.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:43.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:43.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:43.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:43.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:44.184 INFO:teuthology.orchestra.run.smithi049.stdout:77309411391 2024-04-15T18:10:44.263 INFO:teuthology.orchestra.run.smithi049.stdout:369367187479 2024-04-15T18:10:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:44 smithi159 ceph-mon[45336]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:44.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:44 smithi159 ceph-mon[45336]: osdmap e107: 12 total, 12 up, 12 in 2024-04-15T18:10:44.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:44 smithi049 ceph-mon[40087]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:44.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:44 smithi049 ceph-mon[40087]: osdmap e107: 12 total, 12 up, 12 in 2024-04-15T18:10:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:44 smithi082 ceph-mon[44206]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:44 smithi082 ceph-mon[44206]: osdmap e107: 12 total, 12 up, 12 in 2024-04-15T18:10:45.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:45 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:10:45.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:45 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:10:45.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/526555756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:10:45.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4013941005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:10:45.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:45 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:10:45.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:45 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:10:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/526555756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:10:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4013941005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:10:45.497 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411387 got 77309411391 for osd.2 2024-04-15T18:10:45.498 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:45 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:10:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:45 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:10:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/526555756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:10:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4013941005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:10:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:46 smithi159 ceph-mon[45336]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:46 smithi049 ceph-mon[40087]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:46 smithi082 ceph-mon[44206]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:47.374 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187474 got 369367187479 for osd.10 2024-04-15T18:10:47.374 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:47.720 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168605 got 240518168612 for osd.7 2024-04-15T18:10:47.720 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:48.015 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:48.019 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:48.051 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986221 got 133143986227 for osd.4 2024-04-15T18:10:48.051 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:48.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:47 smithi049 ceph-mon[40087]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:47 smithi082 ceph-mon[44206]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:47 smithi159 ceph-mon[45336]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:48.774 INFO:teuthology.orchestra.run.smithi049.stdout:279172874273 2024-04-15T18:10:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/284736069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:10:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/795559753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:10:49.149 INFO:teuthology.orchestra.run.smithi049.stdout:34359738440 2024-04-15T18:10:49.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T18:10:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/284736069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:10:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/795559753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:10:49.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/284736069' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:10:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/795559753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:10:49.550 INFO:teuthology.orchestra.run.smithi049.stdout:403726925842 2024-04-15T18:10:49.563 INFO:teuthology.orchestra.run.smithi049.stdout:111669149754 2024-04-15T18:10:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:49 smithi049 ceph-mon[40087]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3103495482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:10:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:49 smithi082 ceph-mon[44206]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3103495482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:10:50.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:49 smithi159 ceph-mon[45336]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3103495482' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:10:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:51.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:51.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149750 got 111669149754 for osd.3 2024-04-15T18:10:51.636 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:51 smithi049 ceph-mon[40087]: osdmap e108: 12 total, 12 up, 12 in 2024-04-15T18:10:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:10:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:10:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:51 smithi049 ceph-mon[40087]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:51 smithi082 ceph-mon[44206]: osdmap e108: 12 total, 12 up, 12 in 2024-04-15T18:10:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:10:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:10:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:51 smithi082 ceph-mon[44206]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:51 smithi159 ceph-mon[45336]: osdmap e108: 12 total, 12 up, 12 in 2024-04-15T18:10:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:10:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:10:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:51 smithi159 ceph-mon[45336]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:52.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 403726925837 got 403726925842 for osd.11 2024-04-15T18:10:52.730 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:53.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:52 smithi049 ceph-mon[40087]: osdmap e109: 12 total, 12 up, 12 in 2024-04-15T18:10:53.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:52 smithi082 ceph-mon[44206]: osdmap e109: 12 total, 12 up, 12 in 2024-04-15T18:10:53.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:52 smithi159 ceph-mon[45336]: osdmap e109: 12 total, 12 up, 12 in 2024-04-15T18:10:53.792 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874269 got 279172874273 for osd.8 2024-04-15T18:10:53.792 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:54.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:53 smithi049 ceph-mon[40087]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:54.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2658414213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:10:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:53 smithi082 ceph-mon[44206]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:54.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2658414213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:10:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:53 smithi159 ceph-mon[45336]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:54.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2658414213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:10:54.475 INFO:teuthology.orchestra.run.smithi049.stdout:326417514525 2024-04-15T18:10:55.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514521 got 326417514525 for osd.9 2024-04-15T18:10:55.106 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:55.222 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:10:56.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:56 smithi159 ceph-mon[45336]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:56 smithi049 ceph-mon[40087]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:56 smithi082 ceph-mon[44206]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:57.162 INFO:teuthology.orchestra.run.smithi049.stdout:201863462957 2024-04-15T18:10:57.823 INFO:teuthology.orchestra.run.smithi049.stdout:34359738443 2024-04-15T18:10:57.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:57 smithi049 ceph-mon[40087]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:57.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/879956640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:10:57.980 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:10:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/439448279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:10:58.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462951 got 201863462957 for osd.6 2024-04-15T18:10:58.151 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:57 smithi082 ceph-mon[44206]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/879956640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:10:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:10:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/439448279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:10:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:57 smithi159 ceph-mon[45336]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:10:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/879956640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:10:58.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:10:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/439448279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:10:58.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738443 for osd.0 2024-04-15T18:10:58.878 DEBUG:teuthology.parallel:result is None 2024-04-15T18:10:58.878 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:10:58.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:10:59.161 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:00 smithi159 ceph-mon[45336]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:00 smithi049 ceph-mon[40087]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:00.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:00 smithi082 ceph-mon[44206]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:11:01.260 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:11:02.041 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-15T18:11:01.022844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338880,"kb_used_data":10440,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124602944,"statfs":{"total":1151940427776,"available":1151593414656,"internally_reserved":0,"allocated":10690560,"data_stored":6471952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001822"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":377,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-15T18:10:52.220440+0000","last_change":"2024-04-15T18:10:43.312774+0000","last_active":"2024-04-15T18:10:52.220440+0000","last_peered":"2024-04-15T18:10:52.220440+0000","last_clean":"2024-04-15T18:10:52.220440+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:10:52.220440+0000","last_undegraded":"2024-04-15T18:10:52.220440+0000","last_fullsized":"2024-04-15T18:10:52.220440+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:10:43.312652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:10:43.312652+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:12:09.543116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187482,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514527,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31924,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:11:02.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:11:02.322 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:02 smithi159 ceph-mon[45336]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:02 smithi159 ceph-mon[45336]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:02 smithi049 ceph-mon[40087]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:02 smithi049 ceph-mon[40087]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:02 smithi082 ceph-mon[44206]: pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:02 smithi082 ceph-mon[44206]: from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:04.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:04 smithi049 ceph-mon[40087]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:04.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:04 smithi159 ceph-mon[45336]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:04.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:11:04.432 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:11:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:04 smithi082 ceph-mon[44206]: pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:05.203 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-04-15T18:11:03.023224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338880,"kb_used_data":10440,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124602944,"statfs":{"total":1151940427776,"available":1151593414656,"internally_reserved":0,"allocated":10690560,"data_stored":6471952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.065048"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":377,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-15T18:10:52.220440+0000","last_change":"2024-04-15T18:10:43.312774+0000","last_active":"2024-04-15T18:10:52.220440+0000","last_peered":"2024-04-15T18:10:52.220440+0000","last_clean":"2024-04-15T18:10:52.220440+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:10:52.220440+0000","last_undegraded":"2024-04-15T18:10:52.220440+0000","last_fullsized":"2024-04-15T18:10:52.220440+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:10:43.312652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:10:43.312652+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:12:09.543116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925845,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31924,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986232,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:11:05.204 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:11:05.204 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T18:11:05.204 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-15T18:11:05.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-15T18:11:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:05 smithi159 ceph-mon[45336]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:05 smithi049 ceph-mon[40087]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:05.479 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:05 smithi082 ceph-mon[44206]: from='client.14643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:06 smithi159 ceph-mon[45336]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:06.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:06 smithi049 ceph-mon[40087]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:06 smithi082 ceph-mon[44206]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:07.737 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:11:07.737 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T18:11:08.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:07 smithi049 ceph-mon[40087]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:08.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2744880451' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T18:11:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:07 smithi082 ceph-mon[44206]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2744880451' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T18:11:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:07 smithi159 ceph-mon[45336]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2744880451' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T18:11:08.533 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-15T18:11:08.533 INFO:tasks.cephadm:Setup complete, yielding 2024-04-15T18:11:08.534 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-15T18:11:08.545 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-15T18:11:08.545 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2d8c3155d854ee204cb7c7261232571229d1d54'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'sha1': '87dbfb632e974cb2c28d17423071d2d833b53b29'}} 2024-04-15T18:11:08.545 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-04-15T18:11:08.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-15T18:11:08.821 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:10 smithi159 ceph-mon[45336]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:10.429 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:10 smithi049 ceph-mon[40087]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:10 smithi082 ceph-mon[44206]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:11.662 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-15T18:11:11.662 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:11:11.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:11:11.937 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:11.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2632830364' entity='client.admin' 2024-04-15T18:11:11.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:11:11.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:11:11.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:11:11.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:11:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2632830364' entity='client.admin' 2024-04-15T18:11:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:11:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:11:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:11:12.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:11:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2632830364' entity='client.admin' 2024-04-15T18:11:12.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:11:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:11:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:11:12.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:11:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:12 smithi159 ceph-mon[45336]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:13.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:12 smithi049 ceph-mon[40087]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:13.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:12 smithi082 ceph-mon[44206]: pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:14.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:11:14.035 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:11:14.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:14.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:14.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:14.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:14.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:14.642 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-15T18:11:13.883344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338880,"kb_used_data":10440,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124602944,"statfs":{"total":1151940427776,"available":1151593414656,"internally_reserved":0,"allocated":10690560,"data_stored":6471952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.860120"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":377,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-15T18:10:52.220440+0000","last_change":"2024-04-15T18:10:43.312774+0000","last_active":"2024-04-15T18:10:52.220440+0000","last_peered":"2024-04-15T18:10:52.220440+0000","last_clean":"2024-04-15T18:10:52.220440+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:10:52.220440+0000","last_undegraded":"2024-04-15T18:10:52.220440+0000","last_fullsized":"2024-04-15T18:10:52.220440+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:10:43.312652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:10:43.312652+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:12:09.543116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925847,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1290240,"data_stored":936027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31924,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724596,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:11:14.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:11:14.916 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:15.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:14 smithi159 ceph-mon[45336]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:15.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:14 smithi159 ceph-mon[45336]: osdmap e110: 12 total, 12 up, 12 in 2024-04-15T18:11:15.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:14 smithi049 ceph-mon[40087]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:15.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:14 smithi049 ceph-mon[40087]: osdmap e110: 12 total, 12 up, 12 in 2024-04-15T18:11:15.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:14 smithi082 ceph-mon[44206]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:15.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:14 smithi082 ceph-mon[44206]: osdmap e110: 12 total, 12 up, 12 in 2024-04-15T18:11:16.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:15 smithi159 ceph-mon[45336]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:16.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:16.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:15 smithi049 ceph-mon[40087]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:16.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:16.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:15 smithi082 ceph-mon[44206]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:11:17.018 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:11:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:16 smithi159 ceph-mon[45336]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:17.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:16 smithi159 ceph-mon[45336]: osdmap e111: 12 total, 12 up, 12 in 2024-04-15T18:11:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:16 smithi049 ceph-mon[40087]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:17.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:16 smithi049 ceph-mon[40087]: osdmap e111: 12 total, 12 up, 12 in 2024-04-15T18:11:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:16 smithi082 ceph-mon[44206]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:17.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:16 smithi082 ceph-mon[44206]: osdmap e111: 12 total, 12 up, 12 in 2024-04-15T18:11:17.745 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-15T18:11:16.896607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338836,"kb_used_data":9696,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124602988,"statfs":{"total":1151940427776,"available":1151593459712,"internally_reserved":0,"allocated":9928704,"data_stored":6146669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.872233"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":379,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-15T18:11:14.136759+0000","last_change":"2024-04-15T18:10:43.312774+0000","last_active":"2024-04-15T18:11:14.136759+0000","last_peered":"2024-04-15T18:11:14.136759+0000","last_clean":"2024-04-15T18:11:14.136759+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:11:14.136759+0000","last_undegraded":"2024-04-15T18:11:14.136759+0000","last_fullsized":"2024-04-15T18:11:14.136759+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:10:43.312652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:10:43.312652+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:12:09.543116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":888477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":359569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":888497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1179648,"data_stored":888497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":366996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31932,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":647168,"data_stored":359569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27772,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":647168,"data_stored":359569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":757760,"data_stored":407099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:11:17.745 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:11:17.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump 2024-04-15T18:11:17.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:17.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:17.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:17 smithi049 ceph-mon[40087]: osdmap e112: 12 total, 12 up, 12 in 2024-04-15T18:11:18.020 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:18.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:18.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:18.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:17 smithi159 ceph-mon[45336]: osdmap e112: 12 total, 12 up, 12 in 2024-04-15T18:11:18.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:18.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:17 smithi082 ceph-mon[44206]: osdmap e112: 12 total, 12 up, 12 in 2024-04-15T18:11:19.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:18 smithi049 ceph-mon[40087]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:19.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:18 smithi049 ceph-mon[40087]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:18 smithi082 ceph-mon[44206]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:19.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:18 smithi082 ceph-mon[44206]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:19.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:18 smithi159 ceph-mon[45336]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:11:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:18 smithi159 ceph-mon[45336]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:20.003 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 113 2024-04-15T18:11:20.003 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-04-15T18:02:11.588977+0000 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-04-15T18:11:18.901022+0000 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:crush_version 26 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:full_ratio 0.95 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:backfillfull_ratio 0.9 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:nearfull_ratio 0.85 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:require_min_compat_client luminous 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:min_compat_client jewel 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:require_osd_release squid 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:stretch_mode_enabled false 2024-04-15T18:11:20.004 INFO:teuthology.orchestra.run.smithi049.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:max_osd 12 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154] [v2:172.21.15.49:6804/306402154,v1:172.21.15.49:6805/306402154] exists,up c6740db9-c8a9-4083-9ef3-6ac938f861a9 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046] [v2:172.21.15.49:6812/1404755046,v1:172.21.15.49:6813/1404755046] exists,up a5f0a887-6af8-4537-b37c-3d9ed493b595 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618] [v2:172.21.15.49:6820/1108239618,v1:172.21.15.49:6821/1108239618] exists,up 57bad304-da93-49aa-9320-13cf3d1ab1a8 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081] [v2:172.21.15.49:6828/4051312081,v1:172.21.15.49:6829/4051312081] exists,up 8db13188-c0af-4b82-9333-e0659a026aa2 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649] [v2:172.21.15.82:6802/1223630649,v1:172.21.15.82:6803/1223630649] exists,up 6f66dfc2-7e1a-4545-81f8-246ff27e5928 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 up in weight 1 up_from 39 up_thru 42 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992] [v2:172.21.15.82:6810/1943742992,v1:172.21.15.82:6811/1943742992] exists,up fafc8df6-bd72-4306-9b73-7ce797f5744a 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 up in weight 1 up_from 47 up_thru 50 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039] [v2:172.21.15.82:6818/1896143039,v1:172.21.15.82:6819/1896143039] exists,up 2ac61adb-a5b8-4e27-a648-756453c35783 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 up in weight 1 up_from 56 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805] [v2:172.21.15.82:6826/457966805,v1:172.21.15.82:6827/457966805] exists,up a8940a14-d060-46ed-aeb9-d5c43ff421c2 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130] [v2:172.21.15.159:6802/403815130,v1:172.21.15.159:6803/403815130] exists,up 07798de8-8c74-4948-8860-a3e8c07d7e59 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233] [v2:172.21.15.159:6810/1558317233,v1:172.21.15.159:6811/1558317233] exists,up 43cab3c4-4f27-44ba-b8a4-0f6909e3001d 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187] [v2:172.21.15.159:6818/2653197187,v1:172.21.15.159:6819/2653197187] exists,up e9731146-d494-4a3c-a205-37064c685816 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 up in weight 1 up_from 94 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647] [v2:172.21.15.159:6826/4245248647,v1:172.21.15.159:6827/4245248647] exists,up f8de423a-9e36-45ae-97e7-0384858a8118 2024-04-15T18:11:20.005 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3052240311 expires 2024-04-16T18:03:08.925868+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/2417929619 expires 2024-04-16T18:03:08.925868+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/2417929619 expires 2024-04-16T18:03:08.925868+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/769044034 expires 2024-04-16T18:03:08.925868+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/3639973481 expires 2024-04-16T18:02:40.629638+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/1867631398 expires 2024-04-16T18:03:08.925868+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/3639973481 expires 2024-04-16T18:02:40.629638+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/2669325064 expires 2024-04-16T18:02:40.629638+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/530504890 expires 2024-04-16T18:02:40.629638+0000 2024-04-15T18:11:20.006 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/453126905 expires 2024-04-16T18:02:40.629638+0000 2024-04-15T18:11:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:19 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:19 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:19 smithi049 ceph-mon[40087]: osdmap e113: 12 total, 12 up, 12 in 2024-04-15T18:11:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:19 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:19 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:19 smithi082 ceph-mon[44206]: osdmap e113: 12 total, 12 up, 12 in 2024-04-15T18:11:20.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:19 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:20.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:19 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:19 smithi159 ceph-mon[45336]: osdmap e113: 12 total, 12 up, 12 in 2024-04-15T18:11:20.617 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 58 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/306402154,v1:172.21.15.49:6803/306402154] [v2:172.21.15.49:6804/306402154,v1:172.21.15.49:6805/306402154] exists,up c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'osd.1 up in weight 1 up_from 13 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/1404755046,v1:172.21.15.49:6811/1404755046] [v2:172.21.15.49:6812/1404755046,v1:172.21.15.49:6813/1404755046] exists,up a5f0a887-6af8-4537-b37c-3d9ed493b595', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/1108239618,v1:172.21.15.49:6819/1108239618] [v2:172.21.15.49:6820/1108239618,v1:172.21.15.49:6821/1108239618] exists,up 57bad304-da93-49aa-9320-13cf3d1ab1a8', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/4051312081,v1:172.21.15.49:6827/4051312081] [v2:172.21.15.49:6828/4051312081,v1:172.21.15.49:6829/4051312081] exists,up 8db13188-c0af-4b82-9333-e0659a026aa2', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/1223630649,v1:172.21.15.82:6801/1223630649] [v2:172.21.15.82:6802/1223630649,v1:172.21.15.82:6803/1223630649] exists,up 6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'osd.5 up in weight 1 up_from 39 up_thru 42 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/1943742992,v1:172.21.15.82:6809/1943742992] [v2:172.21.15.82:6810/1943742992,v1:172.21.15.82:6811/1943742992] exists,up fafc8df6-bd72-4306-9b73-7ce797f5744a', 'osd.6 up in weight 1 up_from 47 up_thru 50 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/1896143039,v1:172.21.15.82:6817/1896143039] [v2:172.21.15.82:6818/1896143039,v1:172.21.15.82:6819/1896143039] exists,up 2ac61adb-a5b8-4e27-a648-756453c35783', 'osd.7 up in weight 1 up_from 56 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/457966805,v1:172.21.15.82:6825/457966805] [v2:172.21.15.82:6826/457966805,v1:172.21.15.82:6827/457966805] exists,up a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'osd.8 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6800/403815130,v1:172.21.15.159:6801/403815130] [v2:172.21.15.159:6802/403815130,v1:172.21.15.159:6803/403815130] exists,up 07798de8-8c74-4948-8860-a3e8c07d7e59', 'osd.9 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6808/1558317233,v1:172.21.15.159:6809/1558317233] [v2:172.21.15.159:6810/1558317233,v1:172.21.15.159:6811/1558317233] exists,up 43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'osd.10 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6816/2653197187,v1:172.21.15.159:6817/2653197187] [v2:172.21.15.159:6818/2653197187,v1:172.21.15.159:6819/2653197187] exists,up e9731146-d494-4a3c-a205-37064c685816', 'osd.11 up in weight 1 up_from 94 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.159:6824/4245248647,v1:172.21.15.159:6825/4245248647] [v2:172.21.15.159:6826/4245248647,v1:172.21.15.159:6827/4245248647] exists,up f8de423a-9e36-45ae-97e7-0384858a8118'] 2024-04-15T18:11:20.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon mon.a config show 2024-04-15T18:11:20.891 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:20 smithi049 ceph-mon[40087]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1739817271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-15T18:11:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:20 smithi082 ceph-mon[44206]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:21.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1739817271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-15T18:11:21.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:20 smithi159 ceph-mon[45336]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:21.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1739817271' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-15T18:11:22.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:21 smithi049 ceph-mon[40087]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:22.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:21 smithi082 ceph-mon[44206]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:21 smithi159 ceph-mon[45336]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:22.698 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mon.a", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster": "ceph", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket_mode": "", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_client_required": "cephx, none", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_cluster_required": "cephx", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_debug": "true", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-15T18:11:22.699 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_required": "cephx", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_supported": "", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio": "true", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_poll_ms": "250", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_reap_max": "16", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_async_discard_threads": "0", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_block_size": "4096", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio": "false", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_inflight_ios": "false", 2024-04-15T18:11:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_enable_discard": "false", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry": "3", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash": "0", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring": "false", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_hipri": "false", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_type": "", 2024-04-15T18:11:22.701 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_alloc_size": "1048576", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_allocator": "hybrid", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_buffered_io": "true", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_for_zeros": "false", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_compact_log_sync": "false", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_log_runway": "4194304", 2024-04-15T18:11:22.702 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_prefetch": "1048576", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_flush_size": "524288", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_log_runway": "1048576", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery": "false", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_sync_write": "false", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocation_from_file": "true", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocator": "hybrid", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-15T18:11:22.703 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_create": "true", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_create": "false", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_path": "", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_size": "0", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_path": "", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_preallocate_file": "false", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_size": "107374182400", 2024-04-15T18:11:22.704 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_create": "false", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_path": "", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_size": "100663296", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs": "true", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune": "true", 2024-04-15T18:11:22.705 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size": "0", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_type": "2q", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_clone_cow": "true", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-15T18:11:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_mode": "none", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_csum_type": "crc32c", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_freelist": "false", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-15T18:11:22.707 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_small_allocations": "0", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_read": "true", 2024-04-15T18:11:22.708 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_write": "false", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fail_eio": "false", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-15T18:11:22.709 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-15T18:11:22.710 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_ignore_data_csum": "false", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_op_age": "5.000000", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_alloc_size": "0", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size": "0", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_deferred_txc": "32", 2024-04-15T18:11:22.711 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size": "0", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_nid_prealloc": "1024", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_retry_disk_reads": "3", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cf": "true", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-15T18:11:22.712 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_mem": "512", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_tracing": "false", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-15T18:11:22.713 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_zero_block_detection": "false", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-15T18:11:22.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_signatures": "false", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_version": "2", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_signatures": "false", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_version": "2", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_signatures": "false", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_version": "2", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_sign_messages": "true", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "chdir": "", 2024-04-15T18:11:22.715 INFO:teuthology.orchestra.run.smithi049.stdout: "client_acl_type": "", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_asio_thread_count": "2", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_mid": "0.750000", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_size": "16384", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_caps_release_delay": "5", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_check_pool_perm": "true", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_force_sync_read": "false", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_getattr_caps": "false", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_inject_tick_delay": "0", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_remount": "false", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_dirsize_rbytes": "true", 2024-04-15T18:11:22.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_force_lazyio": "false", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_fs": "", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_release_failure": "false", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_inline_size": "4096", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mds_namespace": "", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_metadata": "", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_gid": "-1", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_timeout": "300", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_uid": "-1", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mountpoint": "/", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_notify_timeout": "10", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc": "true", 2024-04-15T18:11:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty": "104857600", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_objects": "1000", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_size": "209715200", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_target_dirty": "8388608", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_permissions": "true", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota": "true", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota_df": "true", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_bytes": "0", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_periods": "4", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_min": "131072", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_reconnect_stale": "false", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_shutdown_timeout": "30", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_snapdir": ".snap", 2024-04-15T18:11:22.718 INFO:teuthology.orchestra.run.smithi049.stdout: "client_tick_interval": "1", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "client_trace": "", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "client_try_dentry_invalidate": "false", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_faked_inos": "false", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_random_mds": "false", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog": "false", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_port": "12201", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_monitors": "default=true", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog": "false", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_level": "info", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "-", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network": "", 2024-04-15T18:11:22.719 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network_interface": "", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_isal": "false", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_level": "5", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_winsize": "-15", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zstd_level": "1", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:55072b3f7eae2e3409906b98d09d882e1c3dc2a9205d85fd5f4c0bc62f82d163", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_op_num_threads": "6", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_seastar_cpu_cores": "", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location": "", 2024-04-15T18:11:22.720 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook": "", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook_timeout": "10", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "daemonize": "false", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "daos_pool": "tank", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_alienstore": "0/5", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_allow_any_pool_priority": "false", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok": "1/5", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok_assert_abort": "false", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asserts_on_shutdown": "true", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_auth": "1/5", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bdev": "1/3", 2024-04-15T18:11:22.721 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluefs": "1/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluestore": "1/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_buffer": "0/1", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ceph_exporter": "1/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephfs_mirror": "0/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephsqlite": "0/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_client": "0/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_compressor": "1/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_context": "0/1", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crimson_interrupt": "0/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crush": "1/1", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crypto": "1/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cyanstore": "0/5", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_deliberately_leak_memory": "false", 2024-04-15T18:11:22.722 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_disable_randomized_ping": "false", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_dpdk": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_eventtrace": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filer": "0/1", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filestore": "1/3", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_finisher": "1/1", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_fuse": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeat_testing_span": "0", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeatmap": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_javaclient": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journal": "1/3", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journaler": "0/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_kstore": "1/5", 2024-04-15T18:11:22.723 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_lockdep": "0/1", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mclock": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_balancer": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_locker": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log_expire": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_migrator": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_quiesce": "3/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_memstore": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgr": "2/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgrc": "1/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mon": "20/20", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_monc": "0/10", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ms": "1/1", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_none": "0/5", 2024-04-15T18:11:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objclass": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objectcacher": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objecter": "0/1", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_optracker": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_osd": "1/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_paxos": "20/20", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_perfcounter": "1/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_prioritycache": "1/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rados": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_mirror": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_pwl": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_replay": "0/5", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_refs": "0/0", 2024-04-15T18:11:22.725 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_reserver": "1/1", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_access": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_datacache": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_dbstore": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_flight": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_notification": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_sync": "1/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rocksdb": "4/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore": "0/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_backref": "0/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cache": "0/5", 2024-04-15T18:11:22.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cleaner": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_device": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_epm": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_journal": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_lba": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_odata": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_omap": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_onode": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_t": "0/5", 2024-04-15T18:11:22.727 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_tm": "0/5", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_striper": "0/1", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_test": "0/5", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_throttle": "1/1", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_timer": "0/1", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_tp": "0/5", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_trace": "1/5", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "device_failure_prediction_mode": "none", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_graylog": "false", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_journald": "false", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_stderr": "true", 2024-04-15T18:11:22.728 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_syslog": "false", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event_tracing": "false", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_addr": "0.0.0.0", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_http_port": "9926", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_prio_limit": "5", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sort_metrics": "true", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_stats_period": "5", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "fake_statfs_for_testing": "0", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "fatal_signal_handlers": "true", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_purge_ops": "10", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_truncate_ops": "128", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_apply_finisher_threads": "1", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_blackhole": "false", 2024-04-15T18:11:22.729 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_clone_range": "true", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_snap": "true", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_caller_concurrency": "10", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_collect_device_partition_information": "true", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_commit_timeout": "600.000000", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_inject_read_err": "false", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_omap_check": "false", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_verify_split": "false", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_dump_file": "", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fadvise": "true", 2024-04-15T18:11:22.730 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fail_eio": "true", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_shards": "16", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_size": "128", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap": "false", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap_threshold": "4096", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_inject_stall": "0", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_parallel": "false", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_trailing": "false", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_writeahead": "false", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_kill_at": "0", 2024-04-15T18:11:22.731 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs": "0", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size": "0", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-15T18:11:22.732 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_merge_threshold": "-10", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_odsync_write": "false", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend": "rocksdb", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend_path": "", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_timeout": "60", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_threads": "2", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_punch_hole": "false", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-15T18:11:22.733 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_ops": "50", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_seek_data_hole": "false", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc": "false", 2024-04-15T18:11:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_splice": "false", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_multiple": "2", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_rand_factor": "20", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_update_to": "1000", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_enable": "true", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-15T18:11:22.735 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_xfs_extsize": "false", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_zfs_snap": "false", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fio_dir": "/tmp/fio", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_allow_other": "true", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_atomic_o_trunc": "true", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_big_writes": "true", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_debug": "false", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_default_permissions": "false", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_disable_pagecache": "false", 2024-04-15T18:11:22.736 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_max_write": "0", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_multithreaded": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_require_active_mds": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_set_user_groups": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_move": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_read": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_write": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_use_invalidate_cb": "true", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_target_name": "ceph", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_file": "", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_inject_failure": "0", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_interval": "5", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "host": "smithi049", 2024-04-15T18:11:22.737 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-15T18:11:22.738 INFO:teuthology.orchestra.run.smithi049.stdout: "inject_early_sigterm": "false", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_agent_port": "6799", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_tracing_enable": "false", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_aio": "true", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_align_min_size": "65536", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_align": "true", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_size": "4096", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_dio": "true", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_discard": "false", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_force_aio": "false", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_ignore_corruption": "false", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_bytes": "10485760", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_entries": "100", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_replay_from": "0", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-15T18:11:22.739 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_write_header_frequency": "0", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_zero_on_create": "false", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prefetch_periods": "10", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prezero_periods": "5", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_write_head_interval": "15", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "keyfile": "", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_backend": "rocksdb", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_default_stripe_size": "65536", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount": "false", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_bytes": "67108864", 2024-04-15T18:11:22.740 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_ops": "512", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_nid_prealloc": "1024", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_onode_map_size": "1024", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_submit_transaction": "false", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_transaction": "false", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "librados_thread_count": "2", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep": "false", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep_force_backtrace": "false", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_coarse_timestamps": "true", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_flush_on_exit": "false", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_host": "127.0.0.1", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_port": "12201", 2024-04-15T18:11:22.741 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_new": "1000", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_recent": "10000", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stderr_prefix": "", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stop_at_utilization": "0.970000", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": "true", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_graylog": "false", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_journald": "false", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_stderr": "false", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_syslog": "false", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "max_rotating_auth_attempts": "10", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_action_on_write_error": "1", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_alternate_name_max": "8192", 2024-04-15T18:11:22.742 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_asio_thread_count": "2", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_export_pin": "true", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_dirs": "true", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_interval": "5", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_interval": "10", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max": "-1", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max_until": "-1", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_merge_size": "50", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_midchunk": "0.300000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_start": "0.200000", 2024-04-15T18:11:22.743 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_minchunk": "0.001000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_mode": "0", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_max": "1.200000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_min": "0.800000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_overload_epochs": "2", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_bits": "3", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_size": "10000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_target_decay": "10.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-15T18:11:22.744 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_grace": "15.000000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_interval": "4.000000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_mid": "0.700000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_delay": "0", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_release_free_interval": "10", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_reservation": "0.050000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_interval": "1", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_threshold": "262144", 2024-04-15T18:11:22.745 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_prealloc_inos": "1000", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_connect_bootstrapping": "false", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_damage_table_max_entries": "10000", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_auth_pins": "false", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_frag": "false", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_scatterstat": "false", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_subtrees": "false", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_decay_halflife": "5.000000", 2024-04-15T18:11:22.746 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_default_dir_hash": "2", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_defer_session_stale": "true", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_deny_all_reconnect": "false", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_keys_per_op": "16384", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_commit_size": "10", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_entries": "0", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_prefetch": "true", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_on_map": "false", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_early_reply": "true", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enable_op_tracker": "true", 2024-04-15T18:11:22.747 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enforce_unique_name": "true", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random": "true", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_summarize_threshold": "10", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-15T18:11:22.748 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_health_dummy": "false", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_migrator_session_race": "false", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_join_fs": "", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_journal_format": "1", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_create_at": "0", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_export_at": "0", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_import_at": "0", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_at": "0", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_expire_at": "0", 2024-04-15T18:11:22.749 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_replay_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_link_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_mdstable_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_openc_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_rename_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_shutdown_at": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_event_large_threshold": "524288", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_events_per_segment": "1024", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_events": "-1", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_segments": "128", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_pause": "false", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_segment_size": "0", 2024-04-15T18:11:22.750 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_threshold": "128", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_warn_factor": "2.000000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_caps_per_client": "1048576", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_flushes": "100000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_requests": "100000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_export_size": "20971520", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_file_recover": "32", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_files": "64", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops": "8192", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-15T18:11:22.751 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_snaps_per_dir": "100", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_metrics_update_interval": "2", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_per_client": "100", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_working_set": "10000", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_numa_node": "-1", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_complaint_time": "30.000000", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_duration": "600", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_size": "20", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_size": "20", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_log_threshold": "5", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_grace": "15", 2024-04-15T18:11:22.752 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_interval": "5", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_caps": "30000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_threshold": "262144", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_interval": "1.000000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_gid": "0", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_uid": "0", 2024-04-15T18:11:22.753 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_shutdown_check": "0", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_skip_ino": "0", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_max_uid": "4294967294", 2024-04-15T18:11:22.754 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_min_uid": "0", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_rstat": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_standby_replay_damaged": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_symlink_recovery": "true", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_exports": "0", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_fragments": "0", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_tick_interval": "5.000000", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_valgrind_exit": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_backtrace": "1", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_scatter": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_sessions": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool_debug": "false", 2024-04-15T18:11:22.755 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_device_bytes": "1073741824", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_set": "false", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_size": "65536", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_bytes": "134217728", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_messages": "512", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_disabled_modules": "", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_creating": "1024", 2024-04-15T18:11:22.756 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_num_change": "128", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_bytes": "134217728", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_messages": "128", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_bytes": "134217728", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_messages": "128", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_bytes": "536870912", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_messages": "8192", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_pool": "true", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_standby_modules": "true", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_period": "5", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_threshold": "5", 2024-04-15T18:11:22.757 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_tick_period": "2", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_delete": "true", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_size_one": "false", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_auth_validate_all_caps": "true", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_bytes": "104857600", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_directed_command_retry": "5", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-15T18:11:22.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_parallel": "3", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_log_interval": "1.000000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_interval": "10.000000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_target_rank": "-1", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_file_level": "debug", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_file": "true", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-15T18:11:22.759 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_journald": "true", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_bootstrap": "false", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_start": "false", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_trim": "true", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cpu_threads": "4", 2024-04-15T18:11:22.760 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_crush_min_required_version": "hammer", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_daemon_bytes": "419430400", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_crit": "5", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_warn": "5", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_size_warn": "16106127360", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_json": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_transactions": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_extra_checks": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-15T18:11:22.761 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_reef": "false", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_squid": "false", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_mkfs_grace": "60", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_uptime_grace": "60", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_default_strategy": "3", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_timeout": "5.000000", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_divisor": "2", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_enable_op_tracker": "true", 2024-04-15T18:11:22.762 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_fake_pool_delete": "false", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_force_quorum_join": "false", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_globalid_prealloc": "10000", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_detail_to_clog": "true", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_log_update_period": "5", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_max_detail": "50", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog": "true", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_interval": "600", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host": "[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0]", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host_override": "", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_initial_members": "", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-15T18:11:22.763 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease": "5.000000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_full_interval": "50", 2024-04-15T18:11:22.764 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max": "10000", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max_summary": "50", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_epochs": "500", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_osd": "10000", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pg_per_osd": "10000", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pool_pg_num": "65536", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_force_trim_to": "0", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_skip_sanity": "false", 2024-04-15T18:11:22.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_autotune": "true", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_target": "2147483648", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_beacon_grace": "30", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_digest_period": "5", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_inactive_grace": "60", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_min_osdmap_epochs": "50", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_complaint_time": "30", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_duration": "600", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_size": "20", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_size": "20", 2024-04-15T18:11:22.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_log_threshold": "5", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_in": "false", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size": "500", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-15T18:11:22.767 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_interval": "600", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_force_trim_to": "0", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_down_reporters": "2", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-15T18:11:22.768 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_report_timeout": "900", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-15T18:11:22.769 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_stuck_threshold": "60", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_probe_timeout": "2.000000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_change": "0.050000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_osds": "4", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-15T18:11:22.770 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_interval": "300", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_max_keys": "100", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_timeout": "300", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_session_timeout": "300", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_smart_report_timeout": "5", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stat_smooth_intervals": "6", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_min_size": "2", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_size": "4", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_debug": "false", 2024-04-15T18:11:22.771 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_provider_kill_at": "0", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_requester_kill_at": "0", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_timeout": "60.000000", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_target_pg_per_osd": "100", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_tick_interval": "5", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_interval": "300.000000", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_use_min_delay_socket": "false", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_older_version_delay": "604800", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-15T18:11:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_misplaced": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_older_version": "true", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-15T18:11:22.773 INFO:teuthology.orchestra.run.smithi049.stdout: "monmap": "", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_tracing_enabled": "false", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_op_threads": "3", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_cm": "false", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_device_name": "", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_dscp": "96", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-15T18:11:22.774 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_local_gid": "", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_port_num": "1", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_sl": "3", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_support_srq": "true", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_type": "ib", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_reap_threshold": "5", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_before_connect": "false", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv4": "true", 2024-04-15T18:11:22.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv6": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr1": "true", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr2": "true", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_max": "7568", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_min": "6800", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_count": "3", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_delay": "5", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_client": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mds": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mgr": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mon": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_osd": "false", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_client_mode": "crc secure", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_mode": "crc secure", 2024-04-15T18:11:22.776 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_type": "", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_compress_secure": "false", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_idle_timeout": "900", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_ready_timeout": "10", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_data": "true", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_header": "true", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bad_msg": "false", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bug": "true", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_old_message": "true", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_skipped_message": "false", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_coremask": "0xF", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-15T18:11:22.777 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_enable_tso": "true", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hugepages": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_lro": "true", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_memory_channel": "4", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_pmd": "", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_port_id": "0", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-15T18:11:22.778 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_on_send": "false", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_initial_backoff": "0.200000", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_max": "1.000000", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_probability": "0.005000", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_type": "osd", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_internal_delays": "0.002000", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_network_congestion": "0", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_socket_failures": "2500", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_learn_addr_from_peer": "true", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_accept_failures": "4", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_backoff": "15.000000", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_client_mode": "secure crc", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_service_mode": "secure crc", 2024-04-15T18:11:22.779 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_min_size": "1024", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_mode": "none", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_min_cost": "65536", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_public_type": "", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_service_mode": "crc secure", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_listen_backlog": "512", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_nodelay": "true", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_rcvbuf": "0", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_type": "async+posix", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "no_config_file": "false", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_completion_locks_per_session": "32", 2024-04-15T18:11:22.780 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_ops": "1024", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_tick_interval": "5.000000", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_timeout": "10.000000", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_blackhole": "false", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "openssl_engine_opts": "", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_delay_time": "5.000000", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_hist_halflife": "1000", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_low_ops": "2", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_ops": "4", 2024-04-15T18:11:22.781 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_slop": "0.020000", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_asio_thread_count": "2", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_async_recovery_min_cost": "100", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_max": "512", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_min": "64", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_degraded": "false", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_peering": "false", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_unfound": "true", 2024-04-15T18:11:22.782 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_beacon_report_interval": "300", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_duration": "30", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_max_block_size": "67108864", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blkin_trace_all": "false", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_for_log_corruption": "false", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-15T18:11:22.783 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_update_on_start": "true", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_cap": "256", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_size_cap": "524288000", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_op_priority": "63", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_watch_timeout": "30", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_command_max_records": "256", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_compact_on_start": "false", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-15T18:11:22.784 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_on_start": "true", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_weight_set": "true", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_feed_pullee": "-1", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_misdirected_ops": "false", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_acting_change": "false", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_purge_strays": "false", 2024-04-15T18:11:22.785 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_op_order": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_shutdown": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_trim_objects": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_snaps": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-15T18:11:22.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_keys": "1024", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_stride": "524288", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_notify_timeout": "30", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep": "0.000000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_discard_disconnected_ops": "true", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_enable_op_tracker": "true", 2024-04-15T18:11:22.787 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_info": "true", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown": "true", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_function_tracing": "false", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_grace": "20", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_interval": "6", 2024-04-15T18:11:22.788 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_peers": "10", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_size": "2000", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_stale": "600", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_max_size": "100000", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_min_size": "1000", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_size": "5120", 2024-04-15T18:11:22.789 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_kill_backfill_at": "0", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_cache_size": "50", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_dedup": "true", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max": "40", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_share_max_epochs": "40", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_name_len": "100", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_size": "0", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_backfills": "1", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_count": "5", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_period": "600", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_name_len": "2048", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_namespace_len": "256", 2024-04-15T18:11:22.790 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_size": "134217728", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_blocked_by": "16", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_log_entries": "10000", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pgls": "1024", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_cost": "8388608", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_objects": "10", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_scrubs": "3", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_trimming_pgs": "2", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_op_reply_len": "64", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_size": "90", 2024-04-15T18:11:22.791 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_profile": "balanced", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-15T18:11:22.792 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_base": "805306368", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_min": "134217728", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target": "4294967296", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_autotune": "false", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_pg_log_entries": "250", 2024-04-15T18:11:22.793 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_recovery_priority": "0", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_interval": "5", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_cache_shards": "32", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_op_tracker_shard": "32", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_auto_affinity": "true", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_node": "-1", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_prefer_iface": "true", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objecter_finishers": "1", 2024-04-15T18:11:22.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore": "bluestore", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_fuse": "false", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_tracing": "false", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_complaint_time": "30.000000", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_duration": "600", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_size": "20", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_size": "20", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_log_threshold": "5", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards": "0", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_hdd": "5", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_ssd": "8", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-15T18:11:22.795 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_min_cost": "65536", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue_cut_off": "high", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_timeout": "15", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_open_classes_on_start": "true", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_os_flags": "0", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_peering_op_priority": "255", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_cost": "1048576", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_priority": "5", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-15T18:11:22.796 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_max": "10000", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_min": "100", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crimson": "false", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-15T18:11:22.797 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flags": "0", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_min_size": "2", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_num": "32", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pgp_num": "0", 2024-04-15T18:11:22.798 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_ratio": "70", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_size": "3", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_type": "replicated", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_push_per_object_cost": "1000", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap": "true", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_cost": "20971520", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active": "0", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-15T18:11:22.799 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_single_start": "1", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_priority": "3", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_priority": "5", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep": "0.000000", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_repair_during_recovery": "false", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_requested_scrub_priority": "120", 2024-04-15T18:11:22.800 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair": "false", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_hour": "0", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_week_day": "0", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_max": "25", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_min": "5", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_cost": "52428800", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_during_recovery": "false", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_hour": "0", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_week_day": "0", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_event_cost": "4096", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-15T18:11:22.801 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_invalid_stats": "true", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_preemptions": "5", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_priority": "5", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_sleep": "0.000000", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_data_digest": "false", 2024-04-15T18:11:22.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_smart_report_timeout": "5", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_cost": "1048576", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_priority": "5", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_transaction_size": "30", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-15T18:11:22.803 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tracing": "false", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_use_stale_snap": "false", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "osdc_blkin_trace_all": "false", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_kill_at": "0", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_max_join_drift": "10", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min": "500", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min_wait": "0.050000", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_propose_interval": "1.000000", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max": "500", 2024-04-15T18:11:22.804 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_min": "10", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_stash_full_interval": "25", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_max": "500", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_min": "250", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "perf": "true", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "pid_file": "", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "-", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrv": "", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "public_bind_addr": "-", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network": "", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network_interface": "", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_enabled": "false", 2024-04-15T18:11:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_session_max_number": "256", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_mon_op_timeout": "0", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_osd_op_timeout": "0", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_tracing": "false", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_atime_update_interval": "60", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_parent_reads": "false", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_snap_reads": "false", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blkin_trace_all": "false", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache": "true", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-15T18:11:22.806 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_policy": "writearound", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_size": "33554432", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_clone_copy_on_read": "false", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_compression_hint": "none", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_concurrent_management_ops": "10", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_clone_format": "auto", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_data_pool": "", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_features": "61", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_format": "2", 2024-04-15T18:11:22.807 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_map_options": "", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_order": "22", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_pool": "rbd", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_count": "0", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_unit": "0", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_enable_alloc_hint": "true", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler": "simple", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-15T18:11:22.808 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_order": "24", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_pool": "", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_splay_width": "4", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_parent_reads": "false", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_snap_reads": "false", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-15T18:11:22.809 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-15T18:11:22.810 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-15T18:11:22.811 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mtime_update_interval": "60", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_non_blocking_aio": "true", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_thread_timeout": "60", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_threads": "1", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_parent_cache_enabled": "false", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_plugins": "", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst": "0", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_limit": "0", 2024-04-15T18:11:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_exclude_ops": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_limit": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-15T18:11:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_read_from_replica_policy": "default", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_skip_partial_discard": "true", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_tracing": "false", 2024-04-15T18:11:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_names": "true", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_pool": "true", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_base_url": "", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_log_level": "", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_acl_grants_max_num": "100", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_admin_entry": "admin", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_backend_store": "rados", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_barbican_url": "", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_beast_enable_async": "true", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache": "false", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-15T18:11:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_enabled": "true", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_expiry_interval": "900", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_lru_size": "10000", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_config_store": "rados", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_content_length_compat": "false", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress": "true", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cors_rules_max_num": "100", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cross_domain_policy": "", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-15T18:11:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_addr": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_password": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_username": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_require_ssl": "true", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-15T18:11:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_addr": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_auth": "token", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_namespace": "", 2024-04-15T18:11:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_prefix": "", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_token_file": "", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_buffersize": "524288", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_time": "300", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-15T18:11:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_port": "6379", 2024-04-15T18:11:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_changes_size": "1000", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_num_shards": "128", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_window": "30", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-15T18:11:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-15T18:11:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_name": "", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_s3website_name": "", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding": "true", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_gc_threads": "true", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_lc_threads": "true", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_ops_log": "false", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_quota_threads": "true", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_static_website": "false", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_usage_log": "false", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enforce_swift_acls": "true", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_exit_timeout_secs": "120", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_expose_bucket": "false", 2024-04-15T18:11:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_extended_http_attrs": "", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_filter": "none", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontends": "beast port=7480", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred": "50", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_objs": "32", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_period": "3600", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-15T18:11:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_init_timeout": "300", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_connection_idle": "300", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_domain": "", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password": "", 2024-04-15T18:11:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password_path": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_project": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_tenant": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token_path": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_user": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_api_version": "2", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_domain": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_password": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_project": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_user": "", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-15T18:11:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_url": "", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_debug_interval": "-1", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_lock_max_time": "90", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_objs": "32", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_rules": "1000", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_worker": "3", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_thread_delay": "0", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-15T18:11:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_dnattr": "uid", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchfilter": "", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_uri": "", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_http_headers": "", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-15T18:11:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name_utc": "false", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_name_len": "0", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_size": "0", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_chunk_size": "4194304", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_listing_results": "1000", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_notify_retries": "10", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_param_size": "1048576", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_size": "5368709120", 2024-04-15T18:11:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_slo_entries": "1000", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_log_max_shards": "64", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mp_lock_max_time": "600", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-15T18:11:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_max_gc": "300", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_async_rados_threads": "32", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_control_oids": "8", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_numa_node": "-1", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-15T18:11:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_chunk_size": "100", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_gc_interval": "600", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_timeout": "600", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_tracing": "false", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_token": "", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_url": "", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_verify_ssl": "true", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-15T18:11:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_rados": "false", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_socket_path": "", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval": "2.000000", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lanes": "3", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-15T18:11:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_partitions": "3", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_continue": "true", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_tracing": "false", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm": "", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_id": "", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region": "", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-15T18:11:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_request_uri": "", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_batch_size": "64", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_max_aio": "128", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_num_logs": "16", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_thread_interval": "600", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_resolve_cname": "false", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_run_sync_thread": "true", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-15T18:11:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_scheduler_type": "throttler", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_script_uri": "", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_service_provider_name": "", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_id": "", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_secret": "", 2024-04-15T18:11:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_entry": "sts", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_key": "sts", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_min_session_duration": "900", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_token_introspection_url": "", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_account_in_url": "false", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_entry": "auth", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_url": "", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_custom_header": "", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_need_stats": "true", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_tenant_name": "", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_token_expiration": "86400", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url": "", 2024-04-15T18:11:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url_prefix": "swift", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_lease_period": "120", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_thread_pool_size": "512", 2024-04-15T18:11:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_comment": "", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_createby": "", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_encoding": "", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_flag": "false", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_origin": "", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_tracker": "", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_trust_forwarded_https": "false", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-15T18:11:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_shards": "32", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_user_shards": "1", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_use_opa_authz": "false", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache": "false", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_max_buckets": "1000", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_policies_max_num": "100", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_unique_email": "true", 2024-04-15T18:11:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_verify_ssl": "true", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone": "", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_id": "", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup": "", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_id": "", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_auto_session_clear": "true", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_block_size": "4096", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-15T18:11:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_size": "536870912", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_index_type": "binary_search", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-15T18:11:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_partition_filters": "false", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_perf": "false", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "run_dir": "/var/run/ceph", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_block_create": "true", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cache_lru_size": "67108864", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_max_object_size": "16777216", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-15T18:11:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_device_size": "53687091200", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_capacity": "16", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_main_device_type": "SSD", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_segment_size": "67108864", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "set_keepcaps": "false", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "setgroup": "ceph", 2024-04-15T18:11:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser": "ceph", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser_match_path": "", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "thp": "false", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_default_timeout": "60", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "throttler_perf_counter": "true" 2024-04-15T18:11:22.845 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:11:23.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-15T18:11:23.737 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:24.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:24 smithi049 ceph-mon[40087]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:24.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:24 smithi159 ceph-mon[45336]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:24.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:24 smithi082 ceph-mon[44206]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:25.640 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-04-15T18:11:25.656 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-04-15T18:11:25.671 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-04-15T18:11:25.672 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:11:25.672 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:11:25.672 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:11:26.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:26 smithi049 ceph-mon[40087]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:26.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3721540377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:11:26.386 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-15T18:11:26.396 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:11:26.397 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:11:26.397 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-15T18:11:26.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-04-15T18:11:26.398 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-15T18:11:26.398 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-15T18:11:26.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 0 1 2024-04-15T18:11:26.399 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-15T18:11:26.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:26.399 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-15T18:11:26.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:11:26.400 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-15T18:11:26.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:11:26.400 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-15T18:11:26.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:11:26.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:26 smithi159 ceph-mon[45336]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3721540377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:11:26.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:26 smithi082 ceph-mon[44206]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:26.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3721540377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:11:26.436 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:26.457 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:26.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:26.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:26.598 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:26.619 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:26.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:26.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:26.737 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:26.757 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:26.761 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:26.778 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:26.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:26.796 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:26.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:26.910 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:27.008 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:27.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:27.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:27.145 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:27.167 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:27.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:27.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:11:27.302 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:27.765 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:28.050 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:11:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:11:28.153 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:28.185 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.206 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:28.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:27 smithi082 ceph-mon[44206]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:28.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:27 smithi049 ceph-mon[40087]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:28.310 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:28.340 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.361 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:27 smithi159 ceph-mon[45336]: pgmap v352: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:28.465 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.513 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:28.616 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:28.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.663 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:28.767 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:28.793 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.812 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:28.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:28.947 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:28.969 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:28.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:29.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:29.102 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.122 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:29.226 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:29.254 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.274 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:29.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:29.409 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.437 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:29.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:29.570 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.589 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:29.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:29.722 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.742 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:29.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:29.882 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:29.903 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:29.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:30.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:30.041 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.062 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:30.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:30 smithi159 ceph-mon[45336]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:30.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:30.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:30 smithi082 ceph-mon[44206]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:30.325 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:30.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:30 smithi049 ceph-mon[40087]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:30.429 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:30.461 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.481 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:30.586 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:30.621 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.643 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:30.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:11:30.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:11:30.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:11:30.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:11:30.733 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:11:30.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:30.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:11:30.778 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.783 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:11:30.799 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:11:30.834 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:11:30.859 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:11:30.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:30.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:30.954 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:30.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:31.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:31.086 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:11:31.091 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:31.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:11:31.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:11:31.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:11:31.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2656606889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:31.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:31.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2656606889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2656606889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.095 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' created 2024-04-15T18:11:32.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-15T18:11:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: pgmap v354: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-15T18:11:33.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-15T18:11:33.243 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:33.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:33.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:33.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-15T18:11:33.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:33 smithi049 ceph-mon[40087]: osdmap e114: 12 total, 12 up, 12 in 2024-04-15T18:11:33.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:33 smithi049 ceph-mon[40087]: osdmap e115: 12 total, 12 up, 12 in 2024-04-15T18:11:33.374 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:33.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-15T18:11:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:33 smithi159 ceph-mon[45336]: osdmap e114: 12 total, 12 up, 12 in 2024-04-15T18:11:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:33 smithi159 ceph-mon[45336]: osdmap e115: 12 total, 12 up, 12 in 2024-04-15T18:11:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2441083681' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-15T18:11:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:33 smithi082 ceph-mon[44206]: osdmap e114: 12 total, 12 up, 12 in 2024-04-15T18:11:33.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:33 smithi082 ceph-mon[44206]: osdmap e115: 12 total, 12 up, 12 in 2024-04-15T18:11:33.724 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:33.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:33.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:33.860 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:33.882 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:33.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:33.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:34.018 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:34.039 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:34.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:34.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:34.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:34 smithi159 ceph-mon[45336]: pgmap v356: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:34.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:34 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:11:34.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:34 smithi159 ceph-mon[45336]: osdmap e116: 12 total, 12 up, 12 in 2024-04-15T18:11:34.173 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:34.193 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:34.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:34.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:34 smithi082 ceph-mon[44206]: pgmap v356: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:34.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:34 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:11:34.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:34 smithi082 ceph-mon[44206]: osdmap e116: 12 total, 12 up, 12 in 2024-04-15T18:11:34.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:34.334 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:34.354 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:34.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:34.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:34 smithi049 ceph-mon[40087]: pgmap v356: 17 pgs: 3 creating+peering, 1 active+clean, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:34 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:11:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:34 smithi049 ceph-mon[40087]: osdmap e116: 12 total, 12 up, 12 in 2024-04-15T18:11:34.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:11:34.490 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:34.511 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:34.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:34.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:11:34.644 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:35.113 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-04-15T18:11:35.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:11:35.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:11:35.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:11:35.796 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:11:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:35.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:11:35.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:35.930 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:36.108 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:36.140 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.161 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:36.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:36.297 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.318 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:36.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:36 smithi159 ceph-mon[45336]: pgmap v359: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:11:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:36 smithi159 ceph-mon[45336]: osdmap e117: 12 total, 12 up, 12 in 2024-04-15T18:11:36.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:36 smithi049 ceph-mon[40087]: pgmap v359: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:36.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:11:36.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:36 smithi049 ceph-mon[40087]: osdmap e117: 12 total, 12 up, 12 in 2024-04-15T18:11:36.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:11:36.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:36 smithi082 ceph-mon[44206]: pgmap v359: 17 pgs: 3 creating+peering, 6 active+clean, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:36.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2380153705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:11:36.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:36 smithi082 ceph-mon[44206]: osdmap e117: 12 total, 12 up, 12 in 2024-04-15T18:11:36.456 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.478 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:36.582 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:36.614 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.635 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:36.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:36.769 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:36.790 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:36.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:36.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:36.924 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:11:37.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:37 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:37.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:37 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:37.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:37 smithi159 ceph-mon[45336]: osdmap e118: 12 total, 12 up, 12 in 2024-04-15T18:11:37.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:11:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:37 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:37 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:37 smithi049 ceph-mon[40087]: osdmap e118: 12 total, 12 up, 12 in 2024-04-15T18:11:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:11:37.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:37 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:37.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:37 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:37.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:37 smithi082 ceph-mon[44206]: osdmap e118: 12 total, 12 up, 12 in 2024-04-15T18:11:37.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:11:37.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 1 1 2024-04-15T18:11:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:38 smithi049 ceph-mon[40087]: pgmap v362: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:38 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:11:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:38 smithi049 ceph-mon[40087]: osdmap e119: 12 total, 12 up, 12 in 2024-04-15T18:11:38.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:38 smithi159 ceph-mon[45336]: pgmap v362: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:38.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:38 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:38.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:11:38.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:38 smithi159 ceph-mon[45336]: osdmap e119: 12 total, 12 up, 12 in 2024-04-15T18:11:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:38 smithi082 ceph-mon[44206]: pgmap v362: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:38 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:38.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3763818252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:11:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:38 smithi082 ceph-mon[44206]: osdmap e119: 12 total, 12 up, 12 in 2024-04-15T18:11:38.546 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:38.574 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:38.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:38.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:11:38.591 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:38.612 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:38.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:38.712 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:38.734 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:38.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:38.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:38.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:38.887 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:38.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:38.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:39.020 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.168 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:39.274 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:39.306 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.327 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:39.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:39.460 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.480 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:39.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:39.616 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.638 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:39.742 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:39.771 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.791 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:39.896 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:39.927 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:39.948 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:39.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:40.052 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:40.082 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:40.102 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:40.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:40.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:40.234 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: pgmap v364: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: osdmap e120: 12 total, 12 up, 12 in 2024-04-15T18:11:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:40.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:11:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: pgmap v364: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: osdmap e120: 12 total, 12 up, 12 in 2024-04-15T18:11:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: pgmap v364: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: osdmap e120: 12 total, 12 up, 12 in 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:40.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:40.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:40.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:40.831 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:40.888 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:40.921 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:40.941 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:40.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:41.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:41.078 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:41.098 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:41.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:41.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:41.233 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:41.257 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:41.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:41.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:41 smithi049 ceph-mon[40087]: osdmap e121: 12 total, 12 up, 12 in 2024-04-15T18:11:41.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:41.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:41 smithi159 ceph-mon[45336]: osdmap e121: 12 total, 12 up, 12 in 2024-04-15T18:11:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:41 smithi082 ceph-mon[44206]: osdmap e121: 12 total, 12 up, 12 in 2024-04-15T18:11:42.197 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:42 smithi159 ceph-mon[45336]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:42.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:42 smithi049 ceph-mon[40087]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:42.443 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:42.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:42 smithi082 ceph-mon[44206]: pgmap v367: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:42.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:42.579 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:42.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:42.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:42.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:42.736 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:42.757 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:42.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:42.862 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:42.894 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:42.916 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:42.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:43.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:43.051 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:44.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:44.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:44.204 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:44.237 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:44.257 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:44.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:44.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:44.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:44 smithi049 ceph-mon[40087]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:44.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:11:44.393 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:44 smithi159 ceph-mon[45336]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:44.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:11:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:44 smithi082 ceph-mon[44206]: pgmap v368: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:11:44.661 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:11:44.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:11:44.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:11:44.733 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:11:44.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:11:44.783 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:11:44.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:11:45.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:11:45.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:11:45.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:11:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-15T18:11:45.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:11:45.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:11:45.141 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:11:45.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: osdmap e122: 12 total, 12 up, 12 in 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1879647663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:11:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: osdmap e122: 12 total, 12 up, 12 in 2024-04-15T18:11:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1879647663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.523 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:45.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/104609822' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:11:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: osdmap e122: 12 total, 12 up, 12 in 2024-04-15T18:11:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1879647663' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:45.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:45.660 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.113 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:46.152 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-04-15T18:11:46.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:46.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:11:46.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:11:46.255 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.294 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:46.398 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:11:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:11:46.430 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.451 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: pgmap v370: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:11:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:11:46.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:11:46.583 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.602 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:11:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:11:46.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:46.738 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.759 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:46.863 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:46.896 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:46.917 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:46.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:47.006 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-15T18:11:47.009 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:11:47.009 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:11:47.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:47.047 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:47.138 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:11:47.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:11:47.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_80260 2024-04-15T18:11:47.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:11:47.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:11:47.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:11:47.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:47 smithi049 ceph-mon[40087]: osdmap e123: 12 total, 12 up, 12 in 2024-04-15T18:11:47.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:11:47.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:47 smithi159 ceph-mon[45336]: osdmap e123: 12 total, 12 up, 12 in 2024-04-15T18:11:47.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3880062436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:11:47.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:47 smithi082 ceph-mon[44206]: osdmap e123: 12 total, 12 up, 12 in 2024-04-15T18:11:47.801 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:47.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:47.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:47.944 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:48.045 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:48.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:48.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 1895 1879 117.431 117.438 0.0580213 0.00825211 2024-04-15T18:11:48.150 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:48.160 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:11:48.175 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:48.194 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:48.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:48.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:48 smithi049 ceph-mon[40087]: pgmap v372: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:48.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/202869033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:48 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:48.335 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:48 smithi159 ceph-mon[45336]: pgmap v372: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/202869033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:48 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.461 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:11:48.461 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:11:48.462 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:11:48.462 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:11:48.462 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:11:48.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:48 smithi082 ceph-mon[44206]: pgmap v372: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:11:48.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/202869033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:48 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:11:48.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:11:48.990 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:49.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:49.132 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 3440 3424 106.993 96.5625 0.00132012 0.00910391 2024-04-15T18:11:49.152 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:49 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:11:49.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:49 smithi049 ceph-mon[40087]: osdmap e124: 12 total, 12 up, 12 in 2024-04-15T18:11:49.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:49.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.303 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:49.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:11:49.408 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:49.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:49 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:11:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:49 smithi159 ceph-mon[45336]: osdmap e124: 12 total, 12 up, 12 in 2024-04-15T18:11:49.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:49 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:11:49.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:11:49.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:49 smithi082 ceph-mon[44206]: osdmap e124: 12 total, 12 up, 12 in 2024-04-15T18:11:49.438 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.461 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:49.565 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:49.594 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.615 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:49.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 2 1 2024-04-15T18:11:49.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:49.759 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.840 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:49.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:49.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:49.972 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:49.999 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:50.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:50.131 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 4627 4611 96.0561 74.1875 0.0276494 0.0103455 2024-04-15T18:11:50.151 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:50.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:50.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:50.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:50.280 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:50 smithi159 ceph-mon[45336]: pgmap v374: 17 pgs: 17 active+clean; 11 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 26 op/s 2024-04-15T18:11:50.282 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.301 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:50.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:50.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:50 smithi049 ceph-mon[40087]: pgmap v374: 17 pgs: 17 active+clean; 11 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 26 op/s 2024-04-15T18:11:50.442 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.488 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:50.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:50 smithi082 ceph-mon[44206]: pgmap v374: 17 pgs: 17 active+clean; 11 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 26 op/s 2024-04-15T18:11:50.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:50.621 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.643 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:50.747 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:50.776 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.796 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:50.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:50.930 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:50.950 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:50.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:51.054 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:51.086 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:51.110 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:51.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:51.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 5987 5971 93.2901 85 0.00900774 0.0106836 2024-04-15T18:11:51.215 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:51.243 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:51.263 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:51.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:51.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:11:51.402 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:52.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 6933 6917 86.4562 59.125 0.0262849 0.0114953 2024-04-15T18:11:52.155 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:11:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:52.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:11:52.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:11:52.217 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:52.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:52 smithi049 ceph-mon[40087]: pgmap v375: 17 pgs: 17 active+clean; 188 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-04-15T18:11:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:52 smithi049 ceph-mon[40087]: osdmap e125: 12 total, 12 up, 12 in 2024-04-15T18:11:52.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:52 smithi082 ceph-mon[44206]: pgmap v375: 17 pgs: 17 active+clean; 188 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-04-15T18:11:52.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:52 smithi082 ceph-mon[44206]: osdmap e125: 12 total, 12 up, 12 in 2024-04-15T18:11:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:52 smithi159 ceph-mon[45336]: pgmap v375: 17 pgs: 17 active+clean; 188 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 436 op/s 2024-04-15T18:11:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:52 smithi159 ceph-mon[45336]: osdmap e125: 12 total, 12 up, 12 in 2024-04-15T18:11:52.677 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:52.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:52.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:11:52.812 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:52.835 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:11:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:52.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:11:52.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:52.972 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:53.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 8000 7984 83.1611 66.6875 0.0289479 0.0120033 2024-04-15T18:11:53.172 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:53.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:53.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:11:53.552 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:53.572 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:11:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:53.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:11:53.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:53.706 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:53.727 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:53.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:53.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:11:53.741+0000 7f6abf59e700 0 --2- 172.21.15.49:0/315686337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ac0107df0 0x7f6ac019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:11:53.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:11:53.808 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:11:53.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:53.835 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:11:53.860 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:53.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:11:53.881 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:53.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:53.898 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:11:53.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:11:53.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:11:53.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:53.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:11:54.016 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.025 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:11:54.037 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:54.055 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:11:54.082 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:11:54.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:11:54.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:11:54.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:11:54.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:11:54.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:11:54.114 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:11:54.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:11:54.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 9346 9330 83.2978 84.125 0.00559538 0.011991 2024-04-15T18:11:54.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:11:54.172 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.196 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:11:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:11:54.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:54.330 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.351 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:54.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:54 smithi049 ceph-mon[40087]: pgmap v377: 17 pgs: 17 active+clean; 341 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:11:54.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/315686337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:54.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:54.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:11:54.481 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:54 smithi082 ceph-mon[44206]: pgmap v377: 17 pgs: 17 active+clean; 341 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:11:54.481 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/315686337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:54.481 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:54.483 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.505 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:11:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:11:54.576 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:11:54.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:54.635 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:54 smithi159 ceph-mon[45336]: pgmap v377: 17 pgs: 17 active+clean; 341 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:11:54.635 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/315686337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:11:54.635 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:54.637 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.657 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:54.762 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:54.791 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.811 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:11:54.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:54.942 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:54.960 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:54.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:55.064 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:11:55.090 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:55.109 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:11:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:55.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:11:55.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 10636 10620 82.9629 80.625 0.00136976 0.0119092 2024-04-15T18:11:55.212 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:11:55.240 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:55.264 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:11:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:55.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:11:55.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:11:55.396 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.416 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:55.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:11:55.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:11:55.559 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:56.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 12307 12291 85.3481 104.438 0.0109972 0.0117003 2024-04-15T18:11:56.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:11:56.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:11:56.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:56 smithi049 ceph-mon[40087]: pgmap v378: 17 pgs: 17 active+clean; 460 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-04-15T18:11:56.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:56 smithi049 ceph-mon[40087]: osdmap e126: 12 total, 12 up, 12 in 2024-04-15T18:11:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:11:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:11:56.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:56 smithi082 ceph-mon[44206]: pgmap v378: 17 pgs: 17 active+clean; 460 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-04-15T18:11:56.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:56 smithi082 ceph-mon[44206]: osdmap e126: 12 total, 12 up, 12 in 2024-04-15T18:11:56.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:11:56.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:11:56.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:56 smithi159 ceph-mon[45336]: pgmap v378: 17 pgs: 17 active+clean; 460 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-04-15T18:11:56.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:56 smithi159 ceph-mon[45336]: osdmap e126: 12 total, 12 up, 12 in 2024-04-15T18:11:56.938 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:11:56.938 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:11:57.139 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 13967 13951 87.1875 103.75 0.0330962 0.0114507 2024-04-15T18:11:57.308 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:11:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:57.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:11:57.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:11:57.440 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:57.459 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:11:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:57.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:11:57.564 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:11:57.593 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:57.614 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:11:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:57.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:11:57.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:11:57.750 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:57 smithi049 ceph-mon[40087]: osdmap e127: 12 total, 12 up, 12 in 2024-04-15T18:11:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:57 smithi049 ceph-mon[40087]: pgmap v381: 17 pgs: 17 active+clean; 662 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:11:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2457113691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:57 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:57.963 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-04-15T18:11:58.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 14942 14926 84.8 60.9375 0.00383609 0.0117746 2024-04-15T18:11:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:57 smithi082 ceph-mon[44206]: osdmap e127: 12 total, 12 up, 12 in 2024-04-15T18:11:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:57 smithi082 ceph-mon[44206]: pgmap v381: 17 pgs: 17 active+clean; 662 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:11:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2457113691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:58.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:57 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:57 smithi159 ceph-mon[45336]: osdmap e127: 12 total, 12 up, 12 in 2024-04-15T18:11:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:57 smithi159 ceph-mon[45336]: pgmap v381: 17 pgs: 17 active+clean; 662 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:11:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2457113691' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:57 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:11:59.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 16211 16195 84.3421 79.3125 0.137913 0.0117458 2024-04-15T18:11:59.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:11:59.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:58 smithi049 ceph-mon[40087]: osdmap e128: 12 total, 12 up, 12 in 2024-04-15T18:11:59.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:11:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:11:59.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:58 smithi082 ceph-mon[44206]: osdmap e128: 12 total, 12 up, 12 in 2024-04-15T18:11:59.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:11:59.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:58 smithi159 ceph-mon[45336]: osdmap e128: 12 total, 12 up, 12 in 2024-04-15T18:11:59.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:11:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:11:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:59.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:11:59.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:11:59.941 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:11:59.963 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:11:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:11:59.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:00.066 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:00.097 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.119 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:00.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 17436 17420 83.7433 76.5625 0.00405878 0.011932 2024-04-15T18:12:00.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:11:59 smithi049 ceph-mon[40087]: pgmap v383: 17 pgs: 17 active+clean; 743 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:12:00.224 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:00.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 3 1 2024-04-15T18:12:00.252 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:11:59 smithi082 ceph-mon[44206]: pgmap v383: 17 pgs: 17 active+clean; 743 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:12:00.254 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:00.276 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:00.380 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:11:59 smithi159 ceph-mon[45336]: pgmap v383: 17 pgs: 17 active+clean; 743 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:12:00.413 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.440 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:00.544 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:00.573 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.593 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:00.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:00.728 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.753 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:00.857 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:00.889 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:00.912 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:00.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:00.983 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:12:01.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:01.046 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/368207345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:01.046 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:01.048 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:01.067 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:01.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:01.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/368207345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:01.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:01.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 19043 19027 84.9351 100.438 0.00439001 0.0117377 2024-04-15T18:12:01.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:01.200 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:01.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/368207345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:01.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:02.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 20313 20297 84.564 79.375 0.0133152 0.0118154 2024-04-15T18:12:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:01 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:01 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:01 smithi049 ceph-mon[40087]: osdmap e129: 12 total, 12 up, 12 in 2024-04-15T18:12:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:01 smithi049 ceph-mon[40087]: pgmap v385: 17 pgs: 17 active+clean; 1004 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:12:02.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:01 smithi049 ceph-mon[40087]: osdmap e130: 12 total, 12 up, 12 in 2024-04-15T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:01 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:01 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:01 smithi082 ceph-mon[44206]: osdmap e129: 12 total, 12 up, 12 in 2024-04-15T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:01 smithi082 ceph-mon[44206]: pgmap v385: 17 pgs: 17 active+clean; 1004 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:12:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:01 smithi082 ceph-mon[44206]: osdmap e130: 12 total, 12 up, 12 in 2024-04-15T18:12:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:01 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:02.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:01 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:01 smithi159 ceph-mon[45336]: osdmap e129: 12 total, 12 up, 12 in 2024-04-15T18:12:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:01 smithi159 ceph-mon[45336]: pgmap v385: 17 pgs: 17 active+clean; 1004 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:12:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:01 smithi159 ceph-mon[45336]: osdmap e130: 12 total, 12 up, 12 in 2024-04-15T18:12:03.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 21374 21358 83.423 66.3125 0.00260225 0.0119754 2024-04-15T18:12:03.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:03.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:03.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:03.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:03.579 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:03.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:03.588 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:03.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:03.714 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:03.734 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:03.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:03.839 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:03.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:03.888 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:03.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:03.992 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:04.018 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:04.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:04.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:04.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 22302 22286 81.9272 58 0.0435843 0.0121609 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: osdmap e131: 12 total, 12 up, 12 in 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:04.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:04.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:04.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:04.170 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:04.195 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:04.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: osdmap e131: 12 total, 12 up, 12 in 2024-04-15T18:12:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:04.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:04.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:04.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:04.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:04.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:04.327 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:04.347 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:04.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: osdmap e131: 12 total, 12 up, 12 in 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:04.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:04.484 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:04.509 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:04.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:04.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:04.645 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:04.668 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:04.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:04.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:04.818 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:05.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 24151 24135 83.7955 115.562 0.00255302 0.0119277 2024-04-15T18:12:05.178 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:05.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:05.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:05.327 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:05.714 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:05.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:05.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:05.853 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:05.873 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:05.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:05.978 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:06.009 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:06.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:06.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:12:06.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:12:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:12:06.132 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:12:06.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:06.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 25545 25529 83.9704 87.125 0.00177056 0.0118818 2024-04-15T18:12:06.160 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:06.179 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:06.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:06.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:12:06.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:06 smithi082 ceph-mon[44206]: pgmap v389: 17 pgs: 17 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:12:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4089463731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:06 smithi049 ceph-mon[40087]: pgmap v389: 17 pgs: 17 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:12:06.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4089463731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:06.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:06 smithi159 ceph-mon[45336]: pgmap v389: 17 pgs: 17 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:12:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4089463731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:06.475 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:06.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:06.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:06.628 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:06.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:06.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:06.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:06.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:06.795 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.055 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:26787 2.c e132/114 2:39e28d25:::benchmark_data_smithi159_80260_object26786:head v 132'1655, mlcod=132'1653)", 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:07.089047+0000", 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00056821400000000005, 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00058904200000000002, 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:12:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089047+0000", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089047+0000", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089048+0000", 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6199999999999997e-07 2024-04-15T18:12:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089149+0000", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100745 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089186+0000", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7741000000000003e-05 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089194+0000", 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4619999999999999e-06 2024-04-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089247+0000", 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3492000000000002e-05 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.089266+0000", 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8221000000000001e-05 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24511.0:26782 2.6 2:6a2427a6:::benchmark_data_smithi159_80260_object26781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:07.086293+0000", 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0033227590000000002, 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003389712, 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24511", 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 26782 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:12:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086293+0000", 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086293+0000", 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086293+0000", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4700000000000001e-07 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086414+0000", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120952 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086428+0000", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3308e-05 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:12:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086435+0000", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4100000000000002e-06 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086486+0000", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0967000000000002e-05 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086597+0000", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110777 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.086737+0000", 2024-04-15T18:12:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000139822 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:07.087860+0000", 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011234350000000001 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-04-15T18:12:07.101 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:12:07.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:12:07.138524+0000 min lat: 0.00105031 max lat: 0.415161 avg lat: 0.0119023 2024-04-15T18:12:07.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:12:07.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 26888 26872 83.9684 83.9375 0.00262831 0.0119023 2024-04-15T18:12:07.162 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:07.193 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.215 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:07.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:07 smithi082 ceph-mon[44206]: osdmap e132: 12 total, 12 up, 12 in 2024-04-15T18:12:07.352 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.374 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:07.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:07 smithi159 ceph-mon[45336]: osdmap e132: 12 total, 12 up, 12 in 2024-04-15T18:12:07.478 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:07.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:07 smithi049 ceph-mon[40087]: osdmap e132: 12 total, 12 up, 12 in 2024-04-15T18:12:07.510 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.533 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:07.637 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:07.674 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.705 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:07.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:07.849 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:07.958 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:07.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:08.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:08.102 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:08.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 28576 28560 84.9933 105.5 0.00583686 0.0117533 2024-04-15T18:12:08.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:08.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:08.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:12:08.300 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:08.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:08 smithi049 ceph-mon[40087]: pgmap v391: 17 pgs: 17 active+clean; 1.4 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:12:08.362 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:08.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:08 smithi159 ceph-mon[45336]: pgmap v391: 17 pgs: 17 active+clean; 1.4 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:12:08.398 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:08.422 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:08.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:08.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:08.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:08 smithi082 ceph-mon[44206]: pgmap v391: 17 pgs: 17 active+clean; 1.4 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:12:08.557 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:08.577 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:08.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:08.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:08.709 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:08.732 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:08.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:08.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:12:08.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:08.869 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:08.892 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:08.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:08.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:09.031 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:09.052 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:09.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:09.140 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 15 30105 30090 85.4762 95.625 0.00187794 0.0116867 2024-04-15T18:12:09.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:09.188 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:09.210 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:09.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:09.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:09.351 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:09.377 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:09.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:09.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:09.512 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:09.534 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:09.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:09.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:09.671 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:09.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:09.748 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:09.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:09.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:09.896 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:10.028 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:10.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:10.113 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-04-15T18:12:10.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:10.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 31505 31489 85.5612 87.4375 0.00577229 0.0116804 2024-04-15T18:12:10.162 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:10.184 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:10.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:10.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:10.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:10 smithi049 ceph-mon[40087]: pgmap v392: 17 pgs: 17 active+clean; 1.5 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-15T18:12:10.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1368000552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:10.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:10 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:10.332 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:10 smithi159 ceph-mon[45336]: pgmap v392: 17 pgs: 17 active+clean; 1.5 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-15T18:12:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1368000552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:10.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:10 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:10 smithi082 ceph-mon[44206]: pgmap v392: 17 pgs: 17 active+clean; 1.5 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-15T18:12:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1368000552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:10 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:12:11.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 32901 32885 85.6313 87.25 0.00526465 0.0116678 2024-04-15T18:12:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:11 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:12:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:11 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:12:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:11 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:12:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:11 smithi049 ceph-mon[40087]: osdmap e133: 12 total, 12 up, 12 in 2024-04-15T18:12:11.397 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:11.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:11.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:11 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:12:11.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:11 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:12:11.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:11 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:12:11.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:11 smithi159 ceph-mon[45336]: osdmap e133: 12 total, 12 up, 12 in 2024-04-15T18:12:11.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 4 1 2024-04-15T18:12:11.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:11 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:12:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:11 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:12:11.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:11 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:12:11.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:11 smithi082 ceph-mon[44206]: osdmap e133: 12 total, 12 up, 12 in 2024-04-15T18:12:11.547 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:11.589 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:11.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:11.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:11.728 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:11.746 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:11.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:11.850 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:11.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:11.910 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:11.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:12.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:12.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 34199 34183 85.4507 81.125 0.00269881 0.0116915 2024-04-15T18:12:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:12 smithi159 ceph-mon[45336]: pgmap v394: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:12.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:12 smithi159 ceph-mon[45336]: osdmap e134: 12 total, 12 up, 12 in 2024-04-15T18:12:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:12 smithi049 ceph-mon[40087]: pgmap v394: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:12 smithi049 ceph-mon[40087]: osdmap e134: 12 total, 12 up, 12 in 2024-04-15T18:12:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:12 smithi082 ceph-mon[44206]: pgmap v394: 17 pgs: 17 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:12 smithi082 ceph-mon[44206]: osdmap e134: 12 total, 12 up, 12 in 2024-04-15T18:12:12.623 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:12.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:12.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:12.764 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:13.117 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:13.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:13.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 35558 35542 85.4307 84.9375 0.00270049 0.0116951 2024-04-15T18:12:13.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:13.943 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:13.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:13.953 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:14.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:14.077 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:14.100 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:14.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:14.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 36583 36567 84.6391 64.0625 0.0111572 0.0118079 2024-04-15T18:12:14.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:14.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:14 smithi049 ceph-mon[40087]: pgmap v396: 17 pgs: 17 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:12:14.242 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:14.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:14 smithi159 ceph-mon[45336]: pgmap v396: 17 pgs: 17 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:12:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:14 smithi082 ceph-mon[44206]: pgmap v396: 17 pgs: 17 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:12:14.617 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:12:14.617 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:12:14.617 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:12:14.617 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:12:14.617 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:12:14.782 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:14.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:14.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:14.929 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:14.967 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:14.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:15.071 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:15.109 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:15.132 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:12:15.135 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:15.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:15.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 38156 38140 85.1272 98.3125 0.0207624 0.0117395 2024-04-15T18:12:15.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:15.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:15.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1224189081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:15.281 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:15.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1224189081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1224189081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:15.568 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:15.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:15.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:12:15.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:15.702 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:15.725 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:15.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:15.831 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:15.859 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:15.880 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:15.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:15.984 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:16.013 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:16.034 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:16.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:16.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:16.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 39697 39681 85.5126 96.3125 0.00391796 0.011683 2024-04-15T18:12:16.169 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:16.191 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:16.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:16 smithi082 ceph-mon[44206]: pgmap v397: 17 pgs: 17 active+clean; 2.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:12:16.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:16 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:12:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:16 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:12:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:16.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:16 smithi082 ceph-mon[44206]: osdmap e135: 12 total, 12 up, 12 in 2024-04-15T18:12:16.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:16.335 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:16 smithi159 ceph-mon[45336]: pgmap v397: 17 pgs: 17 active+clean; 2.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:12:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:16 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:12:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:16 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:12:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:16 smithi159 ceph-mon[45336]: osdmap e135: 12 total, 12 up, 12 in 2024-04-15T18:12:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:16 smithi049 ceph-mon[40087]: pgmap v397: 17 pgs: 17 active+clean; 2.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:12:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:16 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:12:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:16 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:12:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:16 smithi049 ceph-mon[40087]: osdmap e135: 12 total, 12 up, 12 in 2024-04-15T18:12:16.541 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:16.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:16.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:16.674 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:16.693 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:16.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:16.797 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:16.824 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:16.843 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:16.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:16.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:16.982 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:17.005 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:17.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:17.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:17.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 41010 40994 85.3974 82.0625 0.0285849 0.011704 2024-04-15T18:12:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:17.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:17.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:17.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:17.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:17 smithi159 ceph-mon[45336]: osdmap e136: 12 total, 12 up, 12 in 2024-04-15T18:12:17.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:17 smithi049 ceph-mon[40087]: osdmap e136: 12 total, 12 up, 12 in 2024-04-15T18:12:17.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:17.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:17.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:17 smithi082 ceph-mon[44206]: osdmap e136: 12 total, 12 up, 12 in 2024-04-15T18:12:17.495 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:17.514 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:17.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:17.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:17.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:17.661 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:17.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:17.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:17.811 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:18.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:18.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:18.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:18.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 42094 42078 84.8279 67.75 0.00576172 0.0117811 2024-04-15T18:12:18.163 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:18.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:18.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:18.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:18.333 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:18.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:18 smithi159 ceph-mon[45336]: pgmap v400: 17 pgs: 17 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:12:18.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:12:18.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:12:18.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:18 smithi049 ceph-mon[40087]: pgmap v400: 17 pgs: 17 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:12:18.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:12:18.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:12:18.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:18 smithi082 ceph-mon[44206]: pgmap v400: 17 pgs: 17 active+clean; 2.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:12:18.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:12:18.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:12:18.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:12:18.622 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:12:18.656 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:12:18.681 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:18.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:18.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:12:18.704 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:18.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:12:18.775 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:12:18.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:18.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:12:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:12:18.811 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:12:18.828 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:19.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 43450 43434 84.8252 84.75 0.00301123 0.0117812 2024-04-15T18:12:19.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/141790479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/141790479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/141790479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:19.996 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:20.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:20.137 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 44774 44758 84.7621 82.75 0.00590771 0.0117912 2024-04-15T18:12:20.169 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-04-15T18:12:20.188 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: pgmap v401: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2075530987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:20.330 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.358 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: pgmap v401: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2075530987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: pgmap v401: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2075530987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.486 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:12:20.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.507 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:20.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:20.645 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.670 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:20.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:20.801 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.823 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:20.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:20.956 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:20.976 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:20.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:21.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:21.120 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:21.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 15 45936 45921 84.4068 72.6875 0.00266275 0.0118379 2024-04-15T18:12:21.165 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:21.165 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:21.165 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:12:21.165 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:21 smithi082 ceph-mon[44206]: osdmap e137: 12 total, 12 up, 12 in 2024-04-15T18:12:21.321 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:21.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:21.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:21.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:12:21.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:21 smithi049 ceph-mon[40087]: osdmap e137: 12 total, 12 up, 12 in 2024-04-15T18:12:21.580 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:21.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:21.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:21.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:21.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:12:21.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:21 smithi159 ceph-mon[45336]: osdmap e137: 12 total, 12 up, 12 in 2024-04-15T18:12:21.686 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:21.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:12:21.713 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:21.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:21.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:21.883 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:21.973 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:21.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:22.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:22.130 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:22.141 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 47062 47046 84.004 70.3125 0.0540093 0.0118984 2024-04-15T18:12:22.160 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:12:22.160 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:12:22.171 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:22.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:22.313 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:22 smithi049 ceph-mon[40087]: pgmap v403: 17 pgs: 17 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-15T18:12:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:22 smithi049 ceph-mon[40087]: osdmap e138: 12 total, 12 up, 12 in 2024-04-15T18:12:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:22 smithi082 ceph-mon[44206]: pgmap v403: 17 pgs: 17 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-15T18:12:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:22 smithi082 ceph-mon[44206]: osdmap e138: 12 total, 12 up, 12 in 2024-04-15T18:12:22.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:22 smithi159 ceph-mon[45336]: pgmap v403: 17 pgs: 17 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-04-15T18:12:22.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:22 smithi159 ceph-mon[45336]: osdmap e138: 12 total, 12 up, 12 in 2024-04-15T18:12:22.767 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:22.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:22.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:22.903 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:23.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 48348 48332 83.9029 80.375 0.0173635 0.0119027 2024-04-15T18:12:24.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 49612 49596 83.7701 79 0.00823977 0.0119263 2024-04-15T18:12:24.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:24 smithi049 ceph-mon[40087]: pgmap v405: 17 pgs: 17 active+clean; 2.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-15T18:12:24.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:24 smithi082 ceph-mon[44206]: pgmap v405: 17 pgs: 17 active+clean; 2.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-15T18:12:24.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:24 smithi159 ceph-mon[45336]: pgmap v405: 17 pgs: 17 active+clean; 2.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-04-15T18:12:25.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 50658 50642 83.2857 65.375 0.0138041 0.0119773 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:25 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:25.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:25.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:25 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:25.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:12:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:25 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:12:26.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 51550 51534 82.5796 55.75 0.0190295 0.0121026 2024-04-15T18:12:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:26 smithi049 ceph-mon[40087]: pgmap v406: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-15T18:12:26.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:26 smithi082 ceph-mon[44206]: pgmap v406: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-15T18:12:26.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:26 smithi159 ceph-mon[45336]: pgmap v406: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-04-15T18:12:27.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:12:27.140377+0000 min lat: 0.00105031 max lat: 0.415161 avg lat: 0.0121093 2024-04-15T18:12:27.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:12:27.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 52802 52786 82.4711 78.25 0.00244514 0.0121093 2024-04-15T18:12:28.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 53982 53966 82.2582 73.75 0.00935957 0.0121485 2024-04-15T18:12:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:28 smithi049 ceph-mon[40087]: pgmap v407: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:28.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:28 smithi082 ceph-mon[44206]: pgmap v407: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:28.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:28.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:28.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:28 smithi159 ceph-mon[45336]: pgmap v407: 17 pgs: 17 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:12:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:29.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 55133 55117 82.0123 71.9375 0.0108573 0.0121866 2024-04-15T18:12:29.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:29 smithi049 ceph-mon[40087]: osdmap e139: 12 total, 12 up, 12 in 2024-04-15T18:12:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:29 smithi082 ceph-mon[44206]: osdmap e139: 12 total, 12 up, 12 in 2024-04-15T18:12:29.600 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:29.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:29.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 5 1 2024-04-15T18:12:29.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:29 smithi159 ceph-mon[45336]: osdmap e139: 12 total, 12 up, 12 in 2024-04-15T18:12:29.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:29.737 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:29.760 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:29.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:29.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:12:29.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:29.932 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:29.994 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:30.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:30.101 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:30.129 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:30.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 56396 56380 81.9407 78.9375 0.00476809 0.0121976 2024-04-15T18:12:30.149 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:30.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:30.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:30 smithi049 ceph-mon[40087]: pgmap v409: 17 pgs: 17 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:12:30.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:30.305 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:30.338 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:30.381 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:30 smithi082 ceph-mon[44206]: pgmap v409: 17 pgs: 17 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:12:30.619 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:30.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:30 smithi159 ceph-mon[45336]: pgmap v409: 17 pgs: 17 active+clean; 3.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:12:31.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 57248 57232 81.2884 53.25 0.197848 0.0122255 2024-04-15T18:12:31.213 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:12:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:31 smithi049 ceph-mon[40087]: osdmap e140: 12 total, 12 up, 12 in 2024-04-15T18:12:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3075858082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:31.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:31 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:31 smithi082 ceph-mon[44206]: osdmap e140: 12 total, 12 up, 12 in 2024-04-15T18:12:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3075858082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:31 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:31 smithi159 ceph-mon[45336]: osdmap e140: 12 total, 12 up, 12 in 2024-04-15T18:12:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3075858082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:31 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:12:32.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 15 58331 58316 80.9874 67.75 0.0210734 0.0123414 2024-04-15T18:12:32.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:32 smithi049 ceph-mon[40087]: pgmap v411: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:32.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:32 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:32.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:12:32.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:32 smithi049 ceph-mon[40087]: osdmap e141: 12 total, 12 up, 12 in 2024-04-15T18:12:32.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:32 smithi049 ceph-mon[40087]: osdmap e142: 12 total, 12 up, 12 in 2024-04-15T18:12:32.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:32 smithi082 ceph-mon[44206]: pgmap v411: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:32.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:32 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:32.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:12:32.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:32 smithi082 ceph-mon[44206]: osdmap e141: 12 total, 12 up, 12 in 2024-04-15T18:12:32.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:32 smithi082 ceph-mon[44206]: osdmap e142: 12 total, 12 up, 12 in 2024-04-15T18:12:32.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:32 smithi159 ceph-mon[45336]: pgmap v411: 17 pgs: 17 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:32.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:32 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:12:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:32 smithi159 ceph-mon[45336]: osdmap e141: 12 total, 12 up, 12 in 2024-04-15T18:12:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:32 smithi159 ceph-mon[45336]: osdmap e142: 12 total, 12 up, 12 in 2024-04-15T18:12:32.854 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:12:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:12:32.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:12:32.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:12:32.992 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:12:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:12:33.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:12:33.074 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:12:33.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:12:33.141 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:12:33.142 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 59387 59371 80.6601 65.9375 0.00823085 0.0123923 2024-04-15T18:12:33.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:12:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:12:33.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:12:33.220 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:12:33.220 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:12:33.220 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:12:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3881313995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3881313995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:33.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3881313995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:33.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:33.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 60501 60485 80.4252 69.625 0.0231164 0.0124283 2024-04-15T18:12:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: pgmap v414: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:34 smithi049 ceph-mon[40087]: osdmap e143: 12 total, 12 up, 12 in 2024-04-15T18:12:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: pgmap v414: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:34 smithi082 ceph-mon[44206]: osdmap e143: 12 total, 12 up, 12 in 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: pgmap v414: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:34.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:34 smithi159 ceph-mon[45336]: osdmap e143: 12 total, 12 up, 12 in 2024-04-15T18:12:34.861 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:34.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:34.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:35.002 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.026 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:35.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:35.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 61733 61717 80.3537 77 0.0180293 0.0124386 2024-04-15T18:12:35.162 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:35.290 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:35.321 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.344 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:35.448 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:35.476 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.497 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:35.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:35.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:35.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:35.788 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.815 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:35.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:35.947 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:35.966 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:35.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:36.070 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24511.0:62211 2.1 2:8b830cda:::benchmark_data_smithi159_80260_object62210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.584481+0000", 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.021479853, 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021513003999999999, 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:12:36.081 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24511", 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62211 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584481+0000", 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:12:36.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584481+0000", 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584482+0000", 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2299999999999999e-07 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584585+0000", 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102785 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584598+0000", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2728000000000001e-05 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584605+0000", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8439999999999998e-06 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584650+0000", 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4165000000000002e-05 2024-04-15T18:12:36.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584733+0000", 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3671000000000006e-05 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.584823+0000", 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9390999999999998e-05 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:12:36.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.587821+0000", 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029981460000000001 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.604638+0000", 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016817542000000001 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.086 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24511.0:62225 2.e 2:79fcd3bd:::benchmark_data_smithi159_80260_object62224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.596926+0000", 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0090348630000000006, 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0091334220000000004, 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24511", 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62225 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.596926+0000", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.596926+0000", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.596927+0000", 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3400000000000001e-06 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597058+0000", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013038399999999999 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597072+0000", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.467e-05 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597080+0000", 2024-04-15T18:12:36.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0450000000000003e-06 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597125+0000", 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5253999999999997e-05 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597230+0000", 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010479 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:12:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.597327+0000", 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7277999999999999e-05 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.600541+0000", 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003214466 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24511.0:62233 2.e 2:79381cbe:::benchmark_data_smithi159_80260_object62232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.601566+0000", 2024-04-15T18:12:36.091 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0043944479999999996, 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045407690000000001, 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24511", 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 62233 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601566+0000", 2024-04-15T18:12:36.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601566+0000", 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601568+0000", 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3060000000000001e-06 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:12:36.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601685+0000", 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116722 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601699+0000", 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4763000000000001e-05 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601708+0000", 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5339999999999999e-06 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601775+0000", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7453999999999995e-05 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.601919+0000", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143683 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.602079+0000", 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015998400000000001 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:12:36.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.604152+0000", 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020733219999999998 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 3 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:12:36.096 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:36.114 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:36.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:36.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 62489 62473 79.6779 47.25 0.00274824 0.0125034 2024-04-15T18:12:36.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:36.245 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:36 smithi049 ceph-mon[40087]: pgmap v416: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:12:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:36 smithi082 ceph-mon[44206]: pgmap v416: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:12:36.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:12:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:36 smithi159 ceph-mon[45336]: pgmap v416: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:12:36.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:12:36.800 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:36.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:36.903 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:36.931 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:36.950 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:36.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:37.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 62534 62518 78.1405 2.8125 0.0025969 0.012532 2024-04-15T18:12:37.154 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:37.182 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:37.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:37.287 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.305 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:37.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:37.439 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.459 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:37.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:37.597 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.618 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:37.722 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:37.750 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.771 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:37.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:12:37.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:37.906 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:37.927 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:37.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:38.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:38.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:37 smithi049 ceph-mon[40087]: osdmap e144: 12 total, 12 up, 12 in 2024-04-15T18:12:38.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:37 smithi049 ceph-mon[40087]: pgmap v418: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:12:38.060 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:38.113 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:38.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:38.116 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:38.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 62534 62518 76.6083 0 - 0.012532 2024-04-15T18:12:38.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:37 smithi159 ceph-mon[45336]: osdmap e144: 12 total, 12 up, 12 in 2024-04-15T18:12:38.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:37 smithi159 ceph-mon[45336]: pgmap v418: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:12:38.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:38.243 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:37 smithi082 ceph-mon[44206]: osdmap e144: 12 total, 12 up, 12 in 2024-04-15T18:12:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:37 smithi082 ceph-mon[44206]: pgmap v418: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:12:38.592 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:38.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:38.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:38.728 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:38.749 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:38.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:38.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:38.884 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:38.906 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:38.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:39.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:39.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:39.141 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-04-15T18:12:39.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 62534 62518 75.1351 0 - 0.012532 2024-04-15T18:12:39.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:39.195 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.216 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:39.312 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:12:39.319 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:39.336 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:12:39.351 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:12:39.372 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:39.384 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:12:39.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:12:39.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:12:39.433 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:12:39.458 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:12:39.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:39.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:12:39.483 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:12:39.507 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:39.507 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:12:39.529 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:39.540 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:12:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:12:39.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:12:39.590 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:12:39.592 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:12:39.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:39.666 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.687 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:39.791 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:39.823 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:39.844 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:39.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:39.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:39.981 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:40.002 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:40.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:40.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:40.135 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:40.143 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 62534 62518 73.7174 0 - 0.012532 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: pgmap v419: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osdmap e145: 12 total, 12 up, 12 in 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3293144272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: pgmap v419: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osdmap e145: 12 total, 12 up, 12 in 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3293144272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 6 1 2024-04-15T18:12:40.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: pgmap v419: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4241583553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osdmap e145: 12 total, 12 up, 12 in 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3293144272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 62534 62518 72.3522 0 - 0.012532 2024-04-15T18:12:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.486 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:41.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:41.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:41.639 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:41.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:41.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:41.775 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:41.795 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:41.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:41.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:41.931 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:41.952 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:41.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:42.055 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:42.085 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:42.104 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:42.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:42.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 62534 62518 71.0366 0 - 0.012532 2024-04-15T18:12:42.208 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:42.240 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:42.261 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:42.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:42.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:42 smithi049 ceph-mon[40087]: pgmap v421: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 706 op/s 2024-04-15T18:12:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:42 smithi049 ceph-mon[40087]: osdmap e146: 12 total, 12 up, 12 in 2024-04-15T18:12:42.394 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:42 smithi159 ceph-mon[45336]: pgmap v421: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 706 op/s 2024-04-15T18:12:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:42 smithi159 ceph-mon[45336]: osdmap e146: 12 total, 12 up, 12 in 2024-04-15T18:12:42.467 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:42.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:42.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:42 smithi082 ceph-mon[44206]: pgmap v421: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 706 op/s 2024-04-15T18:12:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:42 smithi082 ceph-mon[44206]: osdmap e146: 12 total, 12 up, 12 in 2024-04-15T18:12:42.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:42.597 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:42.937 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:42.945 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:42.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:42.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:43.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:43.083 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:43.103 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:43.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:43.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:12:43.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 62534 62518 69.768 0 - 0.012532 2024-04-15T18:12:43.208 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:43.240 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:43.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:43.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:43.366 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:43.398 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:43.419 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:43.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:43.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:43.554 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:44.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 62534 62518 68.5439 0 - 0.012532 2024-04-15T18:12:44.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:44 smithi049 ceph-mon[40087]: pgmap v423: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 218 op/s 2024-04-15T18:12:44.385 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:12:44.386 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:12:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:44 smithi159 ceph-mon[45336]: pgmap v423: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 218 op/s 2024-04-15T18:12:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:44.507 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:44.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:44 smithi082 ceph-mon[44206]: pgmap v423: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 218 op/s 2024-04-15T18:12:44.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:44.637 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:44.766 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:44.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:44.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:44.902 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:44.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:44.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:45.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:45.060 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.081 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:45.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 62534 62518 67.3621 0 - 0.012532 2024-04-15T18:12:45.165 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:12:45.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:45.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:12:45.215 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.234 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:45.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:45.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:45.367 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.386 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:45.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:45.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:45.517 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.536 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:12:45.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:45.672 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.693 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:45.797 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:45.824 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:45.842 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:45.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:45.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:45.957 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:45.979 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:46.144 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 62534 62518 66.2204 0 - 0.012532 2024-04-15T18:12:46.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:46 smithi049 ceph-mon[40087]: pgmap v424: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 166 op/s 2024-04-15T18:12:46.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:46 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:46.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:46.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:46 smithi049 ceph-mon[40087]: osdmap e147: 12 total, 12 up, 12 in 2024-04-15T18:12:46.392 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:46 smithi159 ceph-mon[45336]: pgmap v424: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 166 op/s 2024-04-15T18:12:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:46 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:46 smithi159 ceph-mon[45336]: osdmap e147: 12 total, 12 up, 12 in 2024-04-15T18:12:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:46 smithi082 ceph-mon[44206]: pgmap v424: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 166 op/s 2024-04-15T18:12:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:46 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2696525849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:12:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:46 smithi082 ceph-mon[44206]: osdmap e147: 12 total, 12 up, 12 in 2024-04-15T18:12:47.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:12:47.142708+0000 min lat: 0.00105031 max lat: 0.610448 avg lat: 0.012532 2024-04-15T18:12:47.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:12:47.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 62534 62518 65.1167 0 - 0.012532 2024-04-15T18:12:47.978 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-04-15T18:12:48.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 62534 62518 64.0491 0 - 0.012532 2024-04-15T18:12:48.200 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:47 smithi049 ceph-mon[40087]: osdmap e148: 12 total, 12 up, 12 in 2024-04-15T18:12:48.200 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:47 smithi049 ceph-mon[40087]: pgmap v427: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s wr, 0 op/s 2024-04-15T18:12:48.200 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:12:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:47 smithi082 ceph-mon[44206]: osdmap e148: 12 total, 12 up, 12 in 2024-04-15T18:12:48.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:47 smithi082 ceph-mon[44206]: pgmap v427: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s wr, 0 op/s 2024-04-15T18:12:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:12:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:47 smithi159 ceph-mon[45336]: osdmap e148: 12 total, 12 up, 12 in 2024-04-15T18:12:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:47 smithi159 ceph-mon[45336]: pgmap v427: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s wr, 0 op/s 2024-04-15T18:12:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:12:48.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:12:49.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 62534 62518 63.016 0 - 0.012532 2024-04-15T18:12:49.147 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:12:49.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:48 smithi049 ceph-mon[40087]: osdmap e149: 12 total, 12 up, 12 in 2024-04-15T18:12:49.170 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:49.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:49.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:12:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:12:49.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:48 smithi082 ceph-mon[44206]: osdmap e149: 12 total, 12 up, 12 in 2024-04-15T18:12:49.305 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3426795983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:12:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:48 smithi159 ceph-mon[45336]: osdmap e149: 12 total, 12 up, 12 in 2024-04-15T18:12:49.428 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:12:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:49.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:12:49.534 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:49.566 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:49.587 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:49.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:49.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:49.698 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:12:49.698 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:12:49.720 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:49.969 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:49.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:50.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:50.105 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:50.126 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:50.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:50.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 62534 62518 62.0157 0 - 0.012532 2024-04-15T18:12:50.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:49 smithi049 ceph-mon[40087]: pgmap v429: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:50.229 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:12:50.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:49 smithi159 ceph-mon[45336]: pgmap v429: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:50.261 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:50.283 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:12:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:50.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:12:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:49 smithi082 ceph-mon[44206]: pgmap v429: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:50.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:50.417 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:50.438 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:50.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:50.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:50.578 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:50.619 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:50.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:50.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 7 1 2024-04-15T18:12:50.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:50.756 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:50.777 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:50.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:50.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:50.912 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:51.141 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:12:51.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 62878 62862 61.3826 1.53571 0.00918593 0.016285 2024-04-15T18:12:51.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:12:51.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:12:51.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:12:51.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1024306146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:51.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:12:51.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:12:51.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:12:51.376 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:12:51.405 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:12:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1024306146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:51.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:12:51.471 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:51.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:51.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:12:51.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:12:51.513 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:12:51.523 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:12:51.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:12:51.525 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:51.531 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:51.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1024306146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:12:51.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:12:51.608 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:51.628 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:12:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:51.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:12:51.733 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:51.762 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:51.784 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:51.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:51.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:51.921 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:51.943 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:51.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:52.048 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:52.079 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:52.105 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:52.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:52.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 63992 63976 61.5093 69.625 0.0141967 0.0162526 2024-04-15T18:12:52.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: pgmap v430: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:52 smithi049 ceph-mon[40087]: osdmap e150: 12 total, 12 up, 12 in 2024-04-15T18:12:52.258 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: pgmap v430: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:52 smithi082 ceph-mon[44206]: osdmap e150: 12 total, 12 up, 12 in 2024-04-15T18:12:52.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:52.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: pgmap v430: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:52.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:52 smithi159 ceph-mon[45336]: osdmap e150: 12 total, 12 up, 12 in 2024-04-15T18:12:52.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:52.476 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:52.583 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:52.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:52.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:52.721 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:52.746 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:52.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:52.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:52.912 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:53.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 65296 65280 61.8121 81.5 0.0204269 0.0161724 2024-04-15T18:12:53.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:12:53.831 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:53.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:53.939 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:53.971 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:53.996 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:53.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:54.100 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:54.128 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:54.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 66327 66311 61.8512 64.4375 0.0153832 0.0161622 2024-04-15T18:12:54.151 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:54.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:54.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:54 smithi049 ceph-mon[40087]: pgmap v432: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s wr, 88 op/s 2024-04-15T18:12:54.256 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:54.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:54 smithi159 ceph-mon[45336]: pgmap v432: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s wr, 88 op/s 2024-04-15T18:12:54.290 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:54.316 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:54.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:54.345 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:54.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:12:54.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:54.470 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:54.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:54 smithi082 ceph-mon[44206]: pgmap v432: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s wr, 88 op/s 2024-04-15T18:12:55.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:55.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:55.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 67683 67667 62.1879 84.75 0.0238156 0.0160753 2024-04-15T18:12:55.247 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:55.282 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:55.307 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:55.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:55.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:55.439 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:55.459 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:55.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:55.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:55.590 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:55.614 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:55.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:55.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:55.753 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:55.776 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:55.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:55.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:55.925 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:56.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 68726 68710 62.2313 65.1875 0.0380219 0.0160634 2024-04-15T18:12:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:56 smithi049 ceph-mon[40087]: pgmap v433: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 238 op/s 2024-04-15T18:12:56.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:12:56.272 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.293 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:12:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:12:56.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:56.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:56 smithi159 ceph-mon[45336]: pgmap v433: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 238 op/s 2024-04-15T18:12:56.433 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.458 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:56 smithi082 ceph-mon[44206]: pgmap v433: 17 pgs: 2 active+clean+laggy, 15 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s wr, 238 op/s 2024-04-15T18:12:56.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:56.595 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.617 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:56.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:56.749 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.770 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:56.799 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:12:56.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:56.903 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:56.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:57.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:12:57.063 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:57.121 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:12:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:57.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:12:57.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 69628 69612 62.1475 56.375 0.00378965 0.0160827 2024-04-15T18:12:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:12:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-04-15T18:12:57.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:57.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:57.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:57.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1698223936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3054014795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:57.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:57 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:57.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:57.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:57.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1698223936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3054014795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:57.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:57 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:57.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:57.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:57.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1698223936' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:12:57.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3054014795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:57.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:57 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:12:58.140 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:58.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:58.145 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 70723 70707 62.236 68.4375 0.0492578 0.0160479 2024-04-15T18:12:58.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:58.272 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:58.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:58.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:12:58.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: pgmap v434: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-04-15T18:12:58.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:58.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:12:58.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:58.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: osdmap e151: 12 total, 12 up, 12 in 2024-04-15T18:12:58.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:12:58.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:12:58.438 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:58.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: pgmap v434: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-04-15T18:12:58.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:58.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:12:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: osdmap e151: 12 total, 12 up, 12 in 2024-04-15T18:12:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:12:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:12:58.462 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:12:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:58.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:12:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: pgmap v434: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-04-15T18:12:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:12:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: osdmap e151: 12 total, 12 up, 12 in 2024-04-15T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:12:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:12:58.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:12:58.609 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:58.764 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:12:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:58.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:12:58.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 8 1 2024-04-15T18:12:58.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:58.894 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:58.917 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:58.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:59.022 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:59.050 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.073 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:59.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 71792 71776 62.2995 66.8125 0.0316283 0.0160436 2024-04-15T18:12:59.176 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:12:59.204 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.224 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:12:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:12:59.331 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:12:59.365 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.384 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:12:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:12:59.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:12:59.531 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.561 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:12:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:12:59.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:12:59.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.717 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:12:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:12:59.820 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:12:59.848 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:12:59.868 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:12:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:12:59.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:12:59.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:00.007 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.030 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:00.134 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:00.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 72794 72778 62.3039 62.625 0.0167385 0.0160446 2024-04-15T18:13:00.164 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:13:00.186 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:00.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:59 smithi082 ceph-mon[44206]: 2.1 scrub starts 2024-04-15T18:13:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:59 smithi082 ceph-mon[44206]: osdmap e152: 12 total, 12 up, 12 in 2024-04-15T18:13:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:12:59 smithi082 ceph-mon[44206]: pgmap v437: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 794 op/s 2024-04-15T18:13:00.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:59 smithi159 ceph-mon[45336]: 2.1 scrub starts 2024-04-15T18:13:00.322 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:59 smithi159 ceph-mon[45336]: osdmap e152: 12 total, 12 up, 12 in 2024-04-15T18:13:00.322 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:12:59 smithi159 ceph-mon[45336]: pgmap v437: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 794 op/s 2024-04-15T18:13:00.324 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.328 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:13:00.328 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:13:00.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.328 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:72619 2.d e152/114 2:bc8a46b7:::benchmark_data_smithi159_80260_object72618:head v 152'4641, mlcod=152'4639)", 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:59.965272+0000", 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0027482409999999998, 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002779176, 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965272+0000", 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:00.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965272+0000", 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965273+0000", 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.125e-06 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965355+0000", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2241000000000001e-05 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965388+0000", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3031000000000003e-05 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965396+0000", 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5220000000000001e-06 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965446+0000", 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0757000000000002e-05 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.965467+0000", 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1189999999999999e-05 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:72620 2.1 e152/114 2:8890c118:::benchmark_data_smithi159_80260_object72619:head v 152'4509, mlcod=152'4507)", 2024-04-15T18:13:00.332 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:59.967782+0000", 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00023814600000000001, 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00033565500000000002, 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967782+0000", 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967782+0000", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967782+0000", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-07 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967804+0000", 2024-04-15T18:13:00.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1641000000000002e-05 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967840+0000", 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5542e-05 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967847+0000", 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9840000000000004e-06 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967912+0000", 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5030999999999995e-05 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:59.967931+0000", 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9372e-05 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-04-15T18:13:00.336 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:13:00.348 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:00.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:59 smithi049 ceph-mon[40087]: 2.1 scrub starts 2024-04-15T18:13:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:59 smithi049 ceph-mon[40087]: osdmap e152: 12 total, 12 up, 12 in 2024-04-15T18:13:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:12:59 smithi049 ceph-mon[40087]: pgmap v437: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 794 op/s 2024-04-15T18:13:00.478 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.501 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:00.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:00.632 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.654 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:00.760 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:00.794 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.818 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:00.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:00.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:00.981 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:00.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:01.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:01.121 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:01.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 73662 73646 62.195 54.25 0.0763158 0.0160711 2024-04-15T18:13:01.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:00 smithi159 ceph-mon[45336]: 2.c scrub starts 2024-04-15T18:13:01.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:00 smithi159 ceph-mon[45336]: 2.1 scrub ok 2024-04-15T18:13:01.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:00 smithi159 ceph-mon[45336]: 2.c scrub ok 2024-04-15T18:13:01.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:00 smithi082 ceph-mon[44206]: 2.c scrub starts 2024-04-15T18:13:01.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:00 smithi082 ceph-mon[44206]: 2.1 scrub ok 2024-04-15T18:13:01.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:00 smithi082 ceph-mon[44206]: 2.c scrub ok 2024-04-15T18:13:01.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:00 smithi049 ceph-mon[40087]: 2.c scrub starts 2024-04-15T18:13:01.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:00 smithi049 ceph-mon[40087]: 2.1 scrub ok 2024-04-15T18:13:01.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:00 smithi049 ceph-mon[40087]: 2.c scrub ok 2024-04-15T18:13:02.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 15 74705 74690 62.2356 65.25 0.0231654 0.0160618 2024-04-15T18:13:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:01 smithi082 ceph-mon[44206]: osdmap e153: 12 total, 12 up, 12 in 2024-04-15T18:13:02.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:01 smithi082 ceph-mon[44206]: pgmap v439: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-15T18:13:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:01 smithi159 ceph-mon[45336]: osdmap e153: 12 total, 12 up, 12 in 2024-04-15T18:13:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:01 smithi159 ceph-mon[45336]: pgmap v439: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-15T18:13:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:01 smithi049 ceph-mon[40087]: osdmap e153: 12 total, 12 up, 12 in 2024-04-15T18:13:02.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:01 smithi049 ceph-mon[40087]: pgmap v439: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-04-15T18:13:03.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 75793 75777 62.3106 67.9375 0.00519212 0.0160438 2024-04-15T18:13:03.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:02 smithi082 ceph-mon[44206]: 2.7 scrub starts 2024-04-15T18:13:03.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:02 smithi159 ceph-mon[45336]: 2.7 scrub starts 2024-04-15T18:13:03.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:02 smithi049 ceph-mon[40087]: 2.7 scrub starts 2024-04-15T18:13:03.721 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:03.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:03.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:13:03.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:03.895 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:04.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 76788 76772 62.3089 62.1875 0.00839262 0.0160188 2024-04-15T18:13:04.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:04 smithi082 ceph-mon[44206]: 2.3 scrub starts 2024-04-15T18:13:04.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:04 smithi082 ceph-mon[44206]: pgmap v440: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:13:04.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:04 smithi082 ceph-mon[44206]: 2.7 scrub ok 2024-04-15T18:13:04.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:04 smithi159 ceph-mon[45336]: 2.3 scrub starts 2024-04-15T18:13:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:04 smithi159 ceph-mon[45336]: pgmap v440: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:13:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:04 smithi159 ceph-mon[45336]: 2.7 scrub ok 2024-04-15T18:13:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:04 smithi049 ceph-mon[40087]: 2.3 scrub starts 2024-04-15T18:13:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:04 smithi049 ceph-mon[40087]: pgmap v440: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:13:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:04 smithi049 ceph-mon[40087]: 2.7 scrub ok 2024-04-15T18:13:05.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 77707 77691 62.2464 57.4375 0.0211539 0.0160594 2024-04-15T18:13:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: 2.8 scrub starts 2024-04-15T18:13:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: 2.d scrub starts 2024-04-15T18:13:05.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: 2.e scrub starts 2024-04-15T18:13:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:05.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:05 smithi082 ceph-mon[44206]: 2.a scrub starts 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: 2.8 scrub starts 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: 2.d scrub starts 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: 2.e scrub starts 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:05.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:05 smithi049 ceph-mon[40087]: 2.a scrub starts 2024-04-15T18:13:05.366 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:05.371 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:05.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: 2.8 scrub starts 2024-04-15T18:13:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: 2.d scrub starts 2024-04-15T18:13:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: 2.e scrub starts 2024-04-15T18:13:05.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:05 smithi159 ceph-mon[45336]: 2.a scrub starts 2024-04-15T18:13:05.472 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:05.500 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:05.522 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:05.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:05.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:05.669 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:05.697 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:05.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:05.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:05.845 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.115 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:06.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 78609 78593 62.172 56.375 0.00973701 0.0160782 2024-04-15T18:13:06.200 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:06.221 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:06.241 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:06.249 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.271 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:06.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.3 scrub ok 2024-04-15T18:13:06.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.8 scrub ok 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: osdmap e154: 12 total, 12 up, 12 in 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: pgmap v442: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.d scrub ok 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.9 scrub starts 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.e scrub ok 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.4 scrub starts 2024-04-15T18:13:06.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:06 smithi082 ceph-mon[44206]: 2.a scrub ok 2024-04-15T18:13:06.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.3 scrub ok 2024-04-15T18:13:06.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.8 scrub ok 2024-04-15T18:13:06.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: osdmap e154: 12 total, 12 up, 12 in 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: pgmap v442: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.d scrub ok 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.9 scrub starts 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.e scrub ok 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.4 scrub starts 2024-04-15T18:13:06.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:06 smithi049 ceph-mon[40087]: 2.a scrub ok 2024-04-15T18:13:06.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:06.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:06.377 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:06.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.3 scrub ok 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.8 scrub ok 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: osdmap e154: 12 total, 12 up, 12 in 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: pgmap v442: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.d scrub ok 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.9 scrub starts 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.e scrub ok 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.4 scrub starts 2024-04-15T18:13:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:06 smithi159 ceph-mon[45336]: 2.a scrub ok 2024-04-15T18:13:06.419 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.426 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:06.446 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:06.460 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:06.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:06.528 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:06.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:06.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:06.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:06.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:06.625 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:06.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:06.763 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:06.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:06.917 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:06.939 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:06.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:07.044 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:07.074 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:07.101 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:07.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:07.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:13:07.144614+0000 min lat: 0.00105031 max lat: 15.0146 avg lat: 0.0161423 2024-04-15T18:13:07.146 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:13:07.147 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 79215 79199 61.8683 37.875 0.00255735 0.0161423 2024-04-15T18:13:07.205 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:07.229 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: 2.2 scrub starts 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: 2.f scrub starts 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1045459839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.231 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:07.252 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:07.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: 2.2 scrub starts 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: 2.f scrub starts 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1045459839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:07.402 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:07.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: 2.2 scrub starts 2024-04-15T18:13:07.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: 2.f scrub starts 2024-04-15T18:13:07.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1045459839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:07.507 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:07.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:07.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:07.646 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:07.667 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:07.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:07.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:07.807 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:07.832 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:07.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:07.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:07.992 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:08.147 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 80214 80198 61.8753 62.4375 0.00534118 0.0161562 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: 2.9 scrub ok 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: 2.0 scrub starts 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: pgmap v443: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: 2.4 scrub ok 2024-04-15T18:13:08.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:08 smithi049 ceph-mon[40087]: 2.f scrub ok 2024-04-15T18:13:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: 2.9 scrub ok 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: 2.0 scrub starts 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: pgmap v443: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: 2.4 scrub ok 2024-04-15T18:13:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:08 smithi082 ceph-mon[44206]: 2.f scrub ok 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: 2.9 scrub ok 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: 2.0 scrub starts 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: pgmap v443: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: 2.4 scrub ok 2024-04-15T18:13:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:08 smithi159 ceph-mon[45336]: 2.f scrub ok 2024-04-15T18:13:08.462 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:08.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:08.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:08.612 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:08.745 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:08.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:08.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:08.886 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:08.911 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:08.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:08.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:09.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:09.044 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.063 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:09.147 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 81314 81298 61.959 68.75 0.00566752 0.0161281 2024-04-15T18:13:09.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:09 smithi159 ceph-mon[45336]: 2.2 scrub ok 2024-04-15T18:13:09.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:09 smithi159 ceph-mon[45336]: 2.0 scrub ok 2024-04-15T18:13:09.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:09.196 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:09 smithi082 ceph-mon[44206]: 2.2 scrub ok 2024-04-15T18:13:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:09 smithi082 ceph-mon[44206]: 2.0 scrub ok 2024-04-15T18:13:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:09.295 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:09 smithi049 ceph-mon[40087]: 2.2 scrub ok 2024-04-15T18:13:09.295 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:09 smithi049 ceph-mon[40087]: 2.0 scrub ok 2024-04-15T18:13:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:09.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:09.429 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.452 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:09.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:09.585 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.604 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:09.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:09.743 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.766 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:09.870 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:09.899 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:09.918 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:09.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:10.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:10.053 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:10.074 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:10.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:10.147 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 81936 81920 61.6808 38.875 0.00861763 0.0162054 2024-04-15T18:13:10.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:10 smithi159 ceph-mon[45336]: pgmap v444: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:13:10.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:10.228 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:10.259 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:10.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:10.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:10.364 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:10.400 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:10.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:10 smithi049 ceph-mon[40087]: pgmap v444: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:13:10.428 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:10.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:10.533 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:10 smithi082 ceph-mon[44206]: pgmap v444: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:13:10.573 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:10.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:10.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:10.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:10.751 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:10.963 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:10.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:11.057 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-04-15T18:13:11.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:11.103 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:11.127 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:11.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:11.147 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 82870 82854 61.6414 58.375 0.0216507 0.0162167 2024-04-15T18:13:11.232 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:11.277 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:11.296 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:11.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:11.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:13:11.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:11 smithi049 ceph-mon[40087]: 2.5 scrub starts 2024-04-15T18:13:11.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:11 smithi049 ceph-mon[40087]: 2.6 scrub starts 2024-04-15T18:13:11.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:13:11.402 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:11.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:11 smithi159 ceph-mon[45336]: 2.5 scrub starts 2024-04-15T18:13:11.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:11 smithi159 ceph-mon[45336]: 2.6 scrub starts 2024-04-15T18:13:11.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:13:11.429 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:11 smithi082 ceph-mon[44206]: 2.5 scrub starts 2024-04-15T18:13:11.429 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:11 smithi082 ceph-mon[44206]: 2.6 scrub starts 2024-04-15T18:13:11.429 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:13:11.431 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:11.453 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:11.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:11.558 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:11.596 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:11.620 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:11.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:11.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:11.763 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:11.791 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:11.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:11.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:12.148 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 83869 83853 61.6499 62.4375 0.0114476 0.0162136 2024-04-15T18:13:12.165 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:12.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:12.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 9 1 2024-04-15T18:13:12.271 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:12.302 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:12.324 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:13:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:12 smithi159 ceph-mon[45336]: osdmap e155: 12 total, 12 up, 12 in 2024-04-15T18:13:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:12 smithi159 ceph-mon[45336]: pgmap v446: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-04-15T18:13:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:12 smithi159 ceph-mon[45336]: 2.5 scrub ok 2024-04-15T18:13:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:12 smithi159 ceph-mon[45336]: 2.b scrub starts 2024-04-15T18:13:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:12.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:12.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:13:12.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:12 smithi049 ceph-mon[40087]: osdmap e155: 12 total, 12 up, 12 in 2024-04-15T18:13:12.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:12 smithi049 ceph-mon[40087]: pgmap v446: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-04-15T18:13:12.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:12 smithi049 ceph-mon[40087]: 2.5 scrub ok 2024-04-15T18:13:12.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:12 smithi049 ceph-mon[40087]: 2.b scrub starts 2024-04-15T18:13:12.428 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:12.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/163751838' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:13:12.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:12 smithi082 ceph-mon[44206]: osdmap e155: 12 total, 12 up, 12 in 2024-04-15T18:13:12.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:12 smithi082 ceph-mon[44206]: pgmap v446: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-04-15T18:13:12.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:12 smithi082 ceph-mon[44206]: 2.5 scrub ok 2024-04-15T18:13:12.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:12 smithi082 ceph-mon[44206]: 2.b scrub starts 2024-04-15T18:13:12.458 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:12.480 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:12.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:12.583 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:12.617 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:12.638 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:12.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:12.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:12.785 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:12.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:12.891 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:13:12.891 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:13:12.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:12.967 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:13.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:13 smithi049 ceph-mon[40087]: 2.6 scrub ok 2024-04-15T18:13:13.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:13 smithi049 ceph-mon[40087]: 2.b scrub ok 2024-04-15T18:13:13.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1219228149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:13 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.080 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:13:13.148 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 84865 84849 61.6568 62.25 0.00795402 0.0162131 2024-04-15T18:13:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:13 smithi159 ceph-mon[45336]: 2.6 scrub ok 2024-04-15T18:13:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:13 smithi159 ceph-mon[45336]: 2.b scrub ok 2024-04-15T18:13:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1219228149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:13 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:13 smithi082 ceph-mon[44206]: 2.6 scrub ok 2024-04-15T18:13:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:13 smithi082 ceph-mon[44206]: 2.b scrub ok 2024-04-15T18:13:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1219228149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:13 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:13.730 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:13.732 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:13.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:13.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:13.872 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:13.892 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:13.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:13.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:14.044 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:14.072 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:14.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:14.148 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 85904 85888 61.6944 64.9375 0.0166446 0.016204 2024-04-15T18:13:14.160 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:14 smithi159 ceph-mon[45336]: pgmap v447: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-04-15T18:13:14.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:14 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:14.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:14 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:13:14.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:14 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:14.161 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:14 smithi159 ceph-mon[45336]: osdmap e156: 12 total, 12 up, 12 in 2024-04-15T18:13:14.177 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:14.207 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:14.230 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:14.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:13:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:14.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:14.334 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:14.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:14 smithi082 ceph-mon[44206]: pgmap v447: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-04-15T18:13:14.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:14 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:14.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:14 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:13:14.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:14 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:14.359 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:14 smithi082 ceph-mon[44206]: osdmap e156: 12 total, 12 up, 12 in 2024-04-15T18:13:14.361 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:14.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:14.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:14 smithi049 ceph-mon[40087]: pgmap v447: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-04-15T18:13:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:14 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:14 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:13:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:14 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:14 smithi049 ceph-mon[40087]: osdmap e156: 12 total, 12 up, 12 in 2024-04-15T18:13:14.489 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:14.526 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:14.551 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:14.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:14.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:14.698 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:14.720 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:14.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:14.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:15.148 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 86956 86940 61.7405 65.75 0.0327582 0.0161909 2024-04-15T18:13:15.729 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:15.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:15.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:15.874 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:16.148 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 87992 87976 61.7742 64.75 0.00745943 0.0161816 2024-04-15T18:13:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:16.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:16.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:16.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:16 smithi049 ceph-mon[40087]: pgmap v449: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 851 op/s 2024-04-15T18:13:16.387 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:16.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:16 smithi159 ceph-mon[45336]: pgmap v449: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 851 op/s 2024-04-15T18:13:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:16 smithi082 ceph-mon[44206]: pgmap v449: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 851 op/s 2024-04-15T18:13:16.592 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:16.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:16.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:16.742 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:16.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:16.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:17.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 5 89110 89105 61.8718 70.5625 0.0123121 0.0161571 2024-04-15T18:13:17.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:17.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:17.295 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:17.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:17.342 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:17.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:13:17.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:17.375 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:17.402 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:17.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:17.420 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:17.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:17.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:17.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:17.486 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:17.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:17.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:17.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:17.564 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.697 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 89110 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 61.4064 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 31.2023 2024-04-15T18:13:18.149 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 117.438 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 982 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 499.249 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 1879 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 0 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0161754 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.199439 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 15.0146 2024-04-15T18:13:18.150 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00105031 2024-04-15T18:13:18.154 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:13:18.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: pgmap v450: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3776846678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: pgmap v450: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3776846678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: pgmap v450: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3776846678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:18.507 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:18.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:18.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:18.663 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:19.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:20.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:19 smithi049 ceph-mon[40087]: pgmap v451: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 830 op/s 2024-04-15T18:13:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:20.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:20.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:20.179 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:20.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:19 smithi082 ceph-mon[44206]: pgmap v451: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 830 op/s 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:1873 2.1 e123/114 2:8c372ad1:::benchmark_data_smithi159_80260_object1872:head v 123'100, mlcod=123'98)", 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:11:48.095109+0000", 2024-04-15T18:13:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 92.232146408000006, 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.117116788, 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.095109+0000", 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.095109+0000", 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.095109+0000", 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4600000000000004e-07 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.095129+0000", 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9177999999999998e-05 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.211724+0000", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116595281 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.211730+0000", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1029999999999998e-06 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.211828+0000", 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8432999999999998e-05 2024-04-15T18:13:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.211852+0000", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3713000000000001e-05 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.212186+0000", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000333519 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:48.212225+0000", 2024-04-15T18:13:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9814999999999997e-05 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:4643 2.a e124/114 2:56799576:::benchmark_data_smithi159_80260_object4642:head v 124'301, mlcod=124'295)", 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:11:50.151727+0000", 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 90.175528463999996, 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.1464752, 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.151727+0000", 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.151727+0000", 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.151727+0000", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.151827+0000", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9471999999999999e-05 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.297678+0000", 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.145851113 2024-04-15T18:13:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.297697+0000", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9032e-05 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.297775+0000", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8087e-05 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.297791+0000", 2024-04-15T18:13:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5695000000000001e-05 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.298171+0000", 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000380415 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:50.298202+0000", 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0732999999999999e-05 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:7183 2.1 e125/114 2:87516a67:::benchmark_data_smithi159_80260_object7182:head v 125'432, mlcod=125'430)", 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:11:52.355263+0000", 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.971992192000002, 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14407605400000001, 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.355263+0000", 2024-04-15T18:13:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.355263+0000", 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.355264+0000", 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4900000000000004e-07 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.355290+0000", 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5899999999999999e-05 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.498781+0000", 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.143491279 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.498788+0000", 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0450000000000003e-06 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.498843+0000", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4564000000000001e-05 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.498858+0000", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5784e-05 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.499306+0000", 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000447701 2024-04-15T18:13:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.499339+0000", 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2832e-05 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:8600 2.e e125/114 2:7ccd7577:::benchmark_data_smithi159_80260_object8599:head v 125'551, mlcod=125'547)", 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:11:53.509416+0000", 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 86.817838921999993, 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.098560894999999996, 2024-04-15T18:13:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.509416+0000", 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.509416+0000", 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.509417+0000", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.68e-07 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.509434+0000", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7529000000000001e-05 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607321+0000", 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.097887232000000005 2024-04-15T18:13:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607329+0000", 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5429999999999996e-06 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607387+0000", 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7917999999999997e-05 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607416+0000", 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9077999999999998e-05 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607933+0000", 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051685900000000005 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.607977+0000", 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4168000000000001e-05 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:25503 2.d e131/114 2:bed950ad:::benchmark_data_smithi159_80260_object25502:head v 131'1638, mlcod=131'1635)", 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:06.054082+0000", 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 74.273172690999999, 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.165567568, 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.054082+0000", 2024-04-15T18:13:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.054082+0000", 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.054084+0000", 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3349999999999999e-06 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.054176+0000", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2078999999999997e-05 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.218726+0000", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16455055199999999 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.218732+0000", 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3340000000000001e-06 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.218787+0000", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5791999999999997e-05 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.218812+0000", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4845999999999999e-05 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.219608+0000", 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079538799999999995 2024-04-15T18:13:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.219650+0000", 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2242e-05 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:25669 2.d e131/114 2:b3c936f4:::benchmark_data_smithi159_80260_object25668:head v 131'1643, mlcod=131'1640)", 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:06.316317+0000", 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 74.010938272000004, 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.078873734000000001, 2024-04-15T18:13:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.316317+0000", 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.316317+0000", 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.316317+0000", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.316421+0000", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103705 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.394311+0000", 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.077889949999999999 2024-04-15T18:13:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.394319+0000", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7619999999999995e-06 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.394363+0000", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3741e-05 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.394376+0000", 2024-04-15T18:13:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3254e-05 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.395142+0000", 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076652000000000003 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:06.395191+0000", 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8149000000000003e-05 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:31281 2.d e132/114 2:b87281b7:::benchmark_data_smithi159_80260_object31280:head v 132'2027, mlcod=132'2023)", 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:09.956696+0000", 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 70.370559421999999, 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.082064498999999999, 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:09.956696+0000", 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:09.956696+0000", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:09.956696+0000", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-07 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:09.956827+0000", 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013062499999999999 2024-04-15T18:13:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.037530+0000", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.080702784 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.037536+0000", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7089999999999996e-06 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.037586+0000", 2024-04-15T18:13:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9292000000000002e-05 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.037600+0000", 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4697e-05 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.038721+0000", 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001120539 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.038760+0000", 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9368e-05 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:36204 2.a e134/114 2:576979be:::benchmark_data_smithi159_80260_object36203:head v 134'2288, mlcod=134'2285)", 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:13.797358+0000", 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 66.529896645999997, 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.110295144, 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.797358+0000", 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.797358+0000", 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.797359+0000", 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0299999999999999e-07 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.797421+0000", 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2173000000000003e-05 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.906404+0000", 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.108982515 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.906423+0000", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9269000000000001e-05 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.906582+0000", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015894999999999999 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.906622+0000", 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0475000000000001e-05 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.907621+0000", 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0009987520000000001 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.907654+0000", 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2407000000000003e-05 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:41211 2.1 e136/114 2:84ef20b0:::benchmark_data_smithi159_80260_object41210:head v 136'2578, mlcod=136'2575)", 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:17.296789+0000", 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 63.030466310999998, 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13613392499999999, 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.296789+0000", 2024-04-15T18:13:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.296789+0000", 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.296790+0000", 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2500000000000004e-07 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.296815+0000", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5607000000000001e-05 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.431535+0000", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13471977600000001 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.431544+0000", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1079999999999999e-06 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.431606+0000", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1821000000000003e-05 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.431630+0000", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3992999999999999e-05 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.432892+0000", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001262338 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.432923+0000", 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0357000000000001e-05 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:41305 2.e e136/114 2:784c3e8b:::benchmark_data_smithi159_80260_object41304:head v 136'2613, mlcod=136'2611)", 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:17.504537+0000", 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 62.822718211000002, 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.188969004, 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.504537+0000", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.504537+0000", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.504538+0000", 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8800000000000002e-07 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.504560+0000", 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2821999999999999e-05 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.691730+0000", 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18716967100000001 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.691738+0000", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.691816+0000", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7384000000000003e-05 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.691836+0000", 2024-04-15T18:13:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0282999999999999e-05 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.693471+0000", 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001634716 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.693506+0000", 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5308999999999997e-05 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:60285 2.1 e143/114 2:8c969d58:::benchmark_data_smithi159_80260_object60284:head v 143'3765, mlcod=143'3762)", 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:33.819172+0000", 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 46.508083474000003, 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15189261100000001, 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819172+0000", 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819172+0000", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819172+0000", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819196+0000", 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3260000000000001e-05 2024-04-15T18:13:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.969323+0000", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15012778900000001 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.969332+0000", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1820000000000006e-06 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.969385+0000", 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3375999999999999e-05 2024-04-15T18:13:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.969400+0000", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5353e-05 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.971016+0000", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016159049999999999 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.971064+0000", 2024-04-15T18:13:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8037000000000001e-05 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:60293 2.6 e143/114 2:6fd774c9:::benchmark_data_smithi159_80260_object60292:head v 143'3808, mlcod=143'3805)", 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:33.844113+0000", 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 46.483141621999998, 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13822585800000001, 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.844113+0000", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.844113+0000", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.844114+0000", 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7300000000000005e-07 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.844134+0000", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9553999999999999e-05 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.980565+0000", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13643085699999999 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.980571+0000", 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6959999999999999e-06 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.980613+0000", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1627e-05 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.980633+0000", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9820999999999999e-05 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.982311+0000", 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016784460000000001 2024-04-15T18:13:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.982339+0000", 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7983999999999998e-05 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:62465 2.9 e143/114 2:9fc5d0d7:::benchmark_data_smithi159_80260_object62464:head v 143'3848, mlcod=143'3845)", 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.923394+0000", 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 44.403861081999999, 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13110708800000001, 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.923394+0000", 2024-04-15T18:13:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.923394+0000", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.923395+0000", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0999999999999998e-07 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.923414+0000", 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9599e-05 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.052787+0000", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.129372774 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.052792+0000", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.215e-06 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.052865+0000", 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2599000000000001e-05 2024-04-15T18:13:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.052880+0000", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5613000000000001e-05 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.054467+0000", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001586868 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.054501+0000", 2024-04-15T18:13:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3810000000000003e-05 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:63983 2.d e150/114 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head v 150'4098, mlcod=150'4094)", 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:52.137441+0000", 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 28.189813858000001, 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12634198499999999, 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.137441+0000", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.137441+0000", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.137442+0000", 2024-04-15T18:13:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.82e-07 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.137570+0000", 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128558 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.256341+0000", 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11877027699999999 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.256349+0000", 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2009999999999999e-06 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.257307+0000", 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095841399999999999 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.257329+0000", 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1500000000000001e-05 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.263720+0000", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006391295 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.263783+0000", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3257999999999996e-05 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:65797 2.d e150/114 2:b05c7ed4:::benchmark_data_smithi159_80260_object65796:head v 150'4206, mlcod=150'4202)", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:53.600649+0000", 2024-04-15T18:13:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.726606201999999, 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.079369114000000004, 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.600649+0000", 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.600649+0000", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.600650+0000", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.600759+0000", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109845 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678490+0000", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.077730627999999996 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678496+0000", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8039999999999998e-06 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678535+0000", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8714999999999998e-05 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678548+0000", 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3688e-05 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.679987+0000", 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014383829999999999 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.680018+0000", 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1359999999999998e-05 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:65833 2.d e150/114 2:bf3a5e6e:::benchmark_data_smithi159_80260_object65832:head v 150'4207, mlcod=150'4204)", 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:53.678389+0000", 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.648866248000001, 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.079048418999999995, 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678389+0000", 2024-04-15T18:13:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678389+0000", 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678389+0000", 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3e-07 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.678476+0000", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6293999999999997e-05 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.755720+0000", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.077244550999999995 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.755728+0000", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3980000000000002e-06 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.755784+0000", 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6693000000000001e-05 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.755801+0000", 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6523000000000001e-05 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.757403+0000", 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001601845 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:53.757437+0000", 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4484999999999999e-05 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:70711 2.1 e151/114 2:8b7e6f39:::benchmark_data_smithi159_80260_object70710:head v 151'4399, mlcod=151'4396)", 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:58.065485+0000", 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.261770286000001, 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.124496591, 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.065485+0000", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.065485+0000", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.065485+0000", 2024-04-15T18:13:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2799999999999996e-07 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.065505+0000", 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9432999999999998e-05 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.187681+0000", 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122176277 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.187687+0000", 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3180000000000003e-06 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.187747+0000", 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9213999999999999e-05 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.187764+0000", 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7350000000000002e-05 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.189948+0000", 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021844059999999998 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:58.189981+0000", 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2965e-05 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:75121 2.d e153/114 2:bb1783f6:::benchmark_data_smithi159_80260_object75120:head v 153'4801, mlcod=153'4799)", 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:02.481816+0000", 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 17.845438621, 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.081350428000000002, 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.481816+0000", 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.481816+0000", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.481818+0000", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0950000000000001e-06 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.481913+0000", 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5888000000000001e-05 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.559960+0000", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.078046883999999997 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.559968+0000", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5530000000000004e-06 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.560009+0000", 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0679000000000001e-05 2024-04-15T18:13:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.560024+0000", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5466999999999999e-05 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.563130+0000", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031060699999999998 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:02.563167+0000", 2024-04-15T18:13:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6792000000000003e-05 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:77693 2.d e154/114 2:b3e63456:::benchmark_data_smithi159_80260_object77692:head v 154'4950, mlcod=154'4948)", 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:05.131038+0000", 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 15.196217455999999, 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11956539300000001, 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.131038+0000", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.131038+0000", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.131038+0000", 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4399999999999999e-07 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.131121+0000", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3100000000000001e-05 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.246955+0000", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11583321000000001 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.246961+0000", 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3729999999999998e-06 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.247021+0000", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9960999999999997e-05 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.247039+0000", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8128e-05 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.250558+0000", 2024-04-15T18:13:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003518799 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.250603+0000", 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5077999999999998e-05 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:77932 2.d e154/114 2:b0d4af62:::benchmark_data_smithi159_80260_object77931:head v 154'4957, mlcod=154'4954)", 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:05.339879+0000", 2024-04-15T18:13:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 14.987376086999999, 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.088475324999999994, 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.339879+0000", 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.339879+0000", 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:13:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.339880+0000", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0299999999999998e-07 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.340059+0000", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000179371 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.424797+0000", 2024-04-15T18:13:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.084737460000000001 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.424806+0000", 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9279999999999999e-06 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.424868+0000", 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2293999999999997e-05 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:13:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.424891+0000", 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3155000000000001e-05 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.428313+0000", 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034215930000000001 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:05.428354+0000", 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1721000000000003e-05 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:13:20.423 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:13:20.424 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:19 smithi159 ceph-mon[45336]: pgmap v451: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 830 op/s 2024-04-15T18:13:20.689 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:21.007 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-04-15T18:13:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:21.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:20 smithi082 ceph-mon[44206]: osdmap e157: 12 total, 12 up, 12 in 2024-04-15T18:13:21.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3590613194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:20 smithi049 ceph-mon[40087]: osdmap e157: 12 total, 12 up, 12 in 2024-04-15T18:13:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3590613194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:21.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:21.405 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:13:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:20 smithi159 ceph-mon[45336]: osdmap e157: 12 total, 12 up, 12 in 2024-04-15T18:13:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3590613194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:13:21.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:21.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:13:21.516 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:21.656 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:21.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:21.718 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:21.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:21.795 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:22 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:13:22.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:22 smithi082 ceph-mon[44206]: osdmap e158: 12 total, 12 up, 12 in 2024-04-15T18:13:22.043 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:22 smithi082 ceph-mon[44206]: pgmap v454: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 982 op/s 2024-04-15T18:13:22.046 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:22.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:22.191 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.219 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:22.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:22.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:22 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:13:22.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:22 smithi049 ceph-mon[40087]: osdmap e158: 12 total, 12 up, 12 in 2024-04-15T18:13:22.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:22 smithi049 ceph-mon[40087]: pgmap v454: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 982 op/s 2024-04-15T18:13:22.364 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.406 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:22 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:13:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:22 smithi159 ceph-mon[45336]: osdmap e158: 12 total, 12 up, 12 in 2024-04-15T18:13:22.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:22 smithi159 ceph-mon[45336]: pgmap v454: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 982 op/s 2024-04-15T18:13:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:22.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 10 1 2024-04-15T18:13:22.511 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:22.544 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.564 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:22.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:22.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.723 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:22.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:22.861 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:22.881 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:22.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:22.988 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:23.013 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:23.032 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:23.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:23.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:23 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:13:23.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:23 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:13:23.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:23 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:13:23.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:23 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:13:23.164 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:23.188 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:23.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:23.292 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:23.326 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:23.348 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:23.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:23 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:13:23.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:23 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:13:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:23.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:23.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:23.459 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:23.497 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:23.917 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:24.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:24.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:24.178 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:95615 2.f e158/114 2:f9f96df2:::benchmark_data_smithi159_80260_object6502:head v 158'5945, mlcod=158'5943)", 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:24.174705+0000", 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0014661170000000001, 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014897860000000001, 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174705+0000", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174705+0000", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174706+0000", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3600000000000002e-07 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:24.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174712+0000", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7819999999999996e-06 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174740+0000", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7248000000000001e-05 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174746+0000", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8130000000000001e-06 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:24.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174796+0000", 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0186999999999999e-05 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:24.174818+0000", 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2387000000000001e-05 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 1 2024-04-15T18:13:24.182 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:24.197 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:24 smithi082 ceph-mon[44206]: pgmap v455: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 38 MiB/s wr, 1.00k op/s 2024-04-15T18:13:24.203 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:24.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:24.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:24.342 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:24.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:24 smithi159 ceph-mon[45336]: pgmap v455: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 38 MiB/s wr, 1.00k op/s 2024-04-15T18:13:24.442 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:24.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:24.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:24 smithi049 ceph-mon[40087]: pgmap v455: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 38 MiB/s wr, 1.00k op/s 2024-04-15T18:13:24.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:24.584 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:24.838 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:24.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:24.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:24.973 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:24.978 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:13:24.995 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:24.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:25.054 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:13:25.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:25.147 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3295551381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:25 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.151 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:25.284 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:25.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:25.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3295551381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:25 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.295 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:25.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:25.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3295551381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:25 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:13:25.428 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:25.462 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:25.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:25.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:25.598 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:25.623 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:25.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:25.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: osdmap e159: 12 total, 12 up, 12 in 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: pgmap v457: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 16 MiB/s wr, 1.03k op/s 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.033 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:13:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: osdmap e159: 12 total, 12 up, 12 in 2024-04-15T18:13:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: pgmap v457: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 16 MiB/s wr, 1.03k op/s 2024-04-15T18:13:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:13:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.392 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:26.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:26.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:26.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:13:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: osdmap e159: 12 total, 12 up, 12 in 2024-04-15T18:13:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: pgmap v457: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 16 MiB/s wr, 1.03k op/s 2024-04-15T18:13:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:13:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:26.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:26.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:26.675 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:26.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:26.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:26.808 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:26.830 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:26.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:26.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:27.005 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:27.165 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:27.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:27.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:27.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:27.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:13:27.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:13:27.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:27.296 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:27.296 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:13:27.296 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:13:27.296 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:13:27.296 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:27.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:13:27.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:13:27.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:27.311 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:27.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:13:27.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:13:27.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:13:27.571 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:27.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:27.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:27.755 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:27.973 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:27.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:28.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:28.126 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: pgmap v458: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 145 B/s rd, 5.5 MiB/s wr, 1.01k op/s 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: osdmap e160: 12 total, 12 up, 12 in 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/918063075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:28 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.176 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:13:28.216 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:28.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: pgmap v458: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 145 B/s rd, 5.5 MiB/s wr, 1.01k op/s 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: osdmap e160: 12 total, 12 up, 12 in 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/918063075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:28 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.300 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:28.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:28.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:28.366 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:28.376 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:28.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:28.405 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: pgmap v458: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 145 B/s rd, 5.5 MiB/s wr, 1.01k op/s 2024-04-15T18:13:28.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: osdmap e160: 12 total, 12 up, 12 in 2024-04-15T18:13:28.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/918063075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:28 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:13:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:28.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:28.437 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:28.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:28.484 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:28.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:28.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:28.547 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:28.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:28.555 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-04-15T18:13:28.578 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:28.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:28.587 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:28.617 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:28.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:28.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:28.703 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:28.705 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:28.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:28.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:28.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:28.718 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:28.737 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:28.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:28.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:28.875 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:28.906 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:28.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:29.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:29.058 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/71301168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: osdmap e161: 12 total, 12 up, 12 in 2024-04-15T18:13:29.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/71301168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: osdmap e161: 12 total, 12 up, 12 in 2024-04-15T18:13:29.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/71301168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:13:29.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: osdmap e161: 12 total, 12 up, 12 in 2024-04-15T18:13:29.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:29.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 11 1 2024-04-15T18:13:30.006 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:30.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:30.110 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:30.115 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:30.138 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:30 smithi049 ceph-mon[40087]: pgmap v461: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:13:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:30 smithi159 ceph-mon[45336]: pgmap v461: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:13:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:30.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:30 smithi082 ceph-mon[44206]: pgmap v461: 17 pgs: 17 active+clean; 5.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:13:31.293 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:31.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:31.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:31.431 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:31.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:31.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:13:31.566 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:31.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:31.619 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:31.641 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:31.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:31.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:31.776 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:31.797 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:31.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:31.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:31.930 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:31.953 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:31.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:32.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:32.116 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:32.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:32 smithi049 ceph-mon[40087]: pgmap v462: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-15T18:13:32.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:32 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:32.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:32 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:32 smithi159 ceph-mon[45336]: pgmap v462: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-15T18:13:32.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:32 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:32.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:32 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:32 smithi082 ceph-mon[44206]: pgmap v462: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-04-15T18:13:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:32 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:32 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:32.871 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:13:32.871 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:13:33.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:33 smithi082 ceph-mon[44206]: osdmap e162: 12 total, 12 up, 12 in 2024-04-15T18:13:33.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:33 smithi159 ceph-mon[45336]: osdmap e162: 12 total, 12 up, 12 in 2024-04-15T18:13:33.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:33 smithi049 ceph-mon[40087]: osdmap e162: 12 total, 12 up, 12 in 2024-04-15T18:13:33.482 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:33.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:33.589 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:33.631 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:33.660 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:33.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:33.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:13:33.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:33.795 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:33.820 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:33.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:33.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:33.960 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:33.983 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:33.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:34.055 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:34.086 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:34.120 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:34.199 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:34.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:34.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:34 smithi049 ceph-mon[40087]: pgmap v464: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-04-15T18:13:34.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:34 smithi082 ceph-mon[44206]: pgmap v464: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-04-15T18:13:34.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:34.338 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:34.361 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:34.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:34.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:34 smithi159 ceph-mon[45336]: pgmap v464: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-04-15T18:13:34.464 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:34.493 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:34.814 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:34.945 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:34.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:35.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:35.085 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:35.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:13:35.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:35.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:35.456 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:35.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:35.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:35.593 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:35.845 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:35.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:35.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:35.979 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:36.079 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:36.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:36.093 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:13:36.093 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:13:36.186 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:36.216 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:36.341 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:36.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:36 smithi082 ceph-mon[44206]: pgmap v465: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-04-15T18:13:36.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:36 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:36.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:36.343 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:36 smithi082 ceph-mon[44206]: osdmap e163: 12 total, 12 up, 12 in 2024-04-15T18:13:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:36.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:36.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:36 smithi159 ceph-mon[45336]: pgmap v465: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-04-15T18:13:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:36 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:36 smithi159 ceph-mon[45336]: osdmap e163: 12 total, 12 up, 12 in 2024-04-15T18:13:36.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:36.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:36 smithi049 ceph-mon[40087]: pgmap v465: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s 2024-04-15T18:13:36.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:36 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:36.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2895893111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:36.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:36 smithi049 ceph-mon[40087]: osdmap e163: 12 total, 12 up, 12 in 2024-04-15T18:13:36.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:36.596 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:36.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:36.716 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:36.759 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:36.790 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:36.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:36.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:36.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:13:36.933 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:37.100 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:13:37.100 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:37 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:37.100 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:37 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:37.101 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-04-15T18:13:37.218 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:13:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:37 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:37 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:37.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:13:37.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:37 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:37.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:37 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:37.387 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:37.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:37.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:37.534 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:37.693 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:37.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:37.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:37.833 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:38.037 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:38.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:38.139 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-15T18:13:38.139 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-15T18:13:38.139 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:13:38.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:13:38.143 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:38.185 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:38.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:38.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:38.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:38.347 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:38.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:38 smithi049 ceph-mon[40087]: pgmap v467: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-15T18:13:38.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:13:38.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:38 smithi049 ceph-mon[40087]: osdmap e164: 12 total, 12 up, 12 in 2024-04-15T18:13:38.414 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:38 smithi159 ceph-mon[45336]: pgmap v467: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-15T18:13:38.414 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:13:38.414 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:38 smithi159 ceph-mon[45336]: osdmap e164: 12 total, 12 up, 12 in 2024-04-15T18:13:38.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:38 smithi082 ceph-mon[44206]: pgmap v467: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s 2024-04-15T18:13:38.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1955859114' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:13:38.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:38 smithi082 ceph-mon[44206]: osdmap e164: 12 total, 12 up, 12 in 2024-04-15T18:13:38.549 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:38.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:38.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:38.707 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:38.750 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:38.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:38.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:38.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:38.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:38.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:38.903 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:38.904 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:38.934 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:38.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:38.940 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:38.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:39.019 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:39.038 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:39.065 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:39.087 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:39.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:39.108 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:39.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:39.171 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:39.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:39.196 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:39.196 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:13:39.196 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:13:39.196 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:13:39.196 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:39.209 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:39.214 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/205715886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:39.218 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:39.218 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:39.218 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:39.236 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:39.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/205715886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:39.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/205715886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:39.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:13:39.941 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:13:40.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: pgmap v469: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osdmap e165: 12 total, 12 up, 12 in 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.271 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: pgmap v469: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osdmap e165: 12 total, 12 up, 12 in 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.293 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:40.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:40.356 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:40.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:40.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: pgmap v469: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osdmap e165: 12 total, 12 up, 12 in 2024-04-15T18:13:40.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:40.443 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:40.470 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:40.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:40.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:40.617 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:40.649 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:40.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:40.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:40.807 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:40.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:40.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:41.011 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:41.089 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:41.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:41.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:41.224 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:41.247 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:41.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:41.352 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:41.381 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:41.582 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:41.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:41.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:41.719 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:41.874 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:41.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:41.984 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:42.015 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:42.036 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:42.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:42.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:4189 2.3 2:cf308da2:::benchmark_data_smithi159_80260_object4188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.628811+0000", 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.550868998, 2024-04-15T18:13:42.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20672693, 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 4189 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628811+0000", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628811+0000", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628812+0000", 2024-04-15T18:13:42.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4799999999999998e-07 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628855+0000", 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2463000000000003e-05 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628860+0000", 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8200000000000002e-06 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.628865+0000", 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6280000000000004e-06 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.629530+0000", 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066542299999999995 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.629617+0000", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6310999999999994e-05 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.629706+0000", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9198999999999998e-05 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.630155+0000", 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00044912199999999999 2024-04-15T18:13:42.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.630342+0000", 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018660899999999999 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835426+0000", 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.205084291 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835494+0000", 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7643000000000003e-05 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835538+0000", 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4873000000000002e-05 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:4193 2.3 2:c3a89f7e:::benchmark_data_smithi159_80260_object4192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.548045273, 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29915336799999998, 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 4193 2024-04-15T18:13:42.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:13:42.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631636+0000", 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631699+0000", 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3040000000000006e-05 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631709+0000", 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8169999999999994e-06 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631714+0000", 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4399999999999996e-06 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631749+0000", 2024-04-15T18:13:42.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4291000000000001e-05 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631839+0000", 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0038999999999996e-05 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631923+0000", 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3978000000000001e-05 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632245+0000", 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032240399999999998 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835583+0000", 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203337675 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930731+0000", 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.095148014000000003 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930764+0000", 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3596999999999997e-05 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930789+0000", 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4340000000000001e-05 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:4189 2.3 e124/114 ondisk, result = 0)", 2024-04-15T18:13:42.201 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.632442+0000", 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.547238616, 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203102849, 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632442+0000", 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632442+0000", 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632442+0000", 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7100000000000002e-07 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632446+0000", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4750000000000002e-06 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835364+0000", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.202918547 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835369+0000", 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7720000000000002e-06 2024-04-15T18:13:42.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835412+0000", 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2296999999999998e-05 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835420+0000", 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2900000000000002e-06 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835545+0000", 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012489700000000001 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6450 2.3 2:ce1e541e:::benchmark_data_smithi159_80260_object6449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.514950+0000", 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 110.66473033299999, 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21478255399999999, 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.206 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6450 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514950+0000", 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514950+0000", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514951+0000", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4099999999999997e-07 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514977+0000", 2024-04-15T18:13:42.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6410999999999998e-05 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514988+0000", 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0241000000000001e-05 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514995+0000", 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4950000000000002e-06 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515038+0000", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2994000000000002e-05 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515119+0000", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0828000000000006e-05 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515203+0000", 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4153999999999994e-05 2024-04-15T18:13:42.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515684+0000", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048043599999999999 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729638+0000", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21395408899999999 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729669+0000", 2024-04-15T18:13:42.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1534e-05 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729706+0000", 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6897999999999998e-05 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729733+0000", 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6633000000000001e-05 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:6450 2.3 e124/114 ondisk, result = 0)", 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.515878+0000", 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 110.66380266, 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.213769136, 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515878+0000", 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515878+0000", 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515879+0000", 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1399999999999996e-07 2024-04-15T18:13:42.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.515882+0000", 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6349999999999999e-06 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729579+0000", 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21369682300000001 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729584+0000", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9760000000000003e-06 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729624+0000", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9697000000000003e-05 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729632+0000", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9189999999999999e-06 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.729647+0000", 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5372000000000001e-05 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:33662 2.2 2:41f39727:::benchmark_data_smithi159_80260_object33661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:11.632068+0000", 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 90.547612174999998, 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.206177636, 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.217 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 33662 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632068+0000", 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632068+0000", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632069+0000", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.75e-07 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632090+0000", 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0701000000000001e-05 2024-04-15T18:13:42.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632098+0000", 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9549999999999992e-06 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632103+0000", 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1710000000000001e-06 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632137+0000", 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4592999999999999e-05 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632217+0000", 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9611999999999996e-05 2024-04-15T18:13:42.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.632312+0000", 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5444000000000003e-05 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.633208+0000", 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00089517100000000005 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.633407+0000", 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000199409 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.838184+0000", 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.204777299 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.838220+0000", 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5484000000000001e-05 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.838246+0000", 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6122000000000002e-05 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:65717 2.2 2:4ab79f1f:::benchmark_data_smithi159_80260_object65716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.695305165000001, 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27045140699999998, 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 65717 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484376+0000", 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9599999999999998e-07 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484427+0000", 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1204e-05 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484439+0000", 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1138000000000001e-05 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484444+0000", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8250000000000001e-06 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484492+0000", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7579e-05 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484623+0000", 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013083499999999999 2024-04-15T18:13:42.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484784+0000", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000161008 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486198+0000", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014138919999999999 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519765+0000", 2024-04-15T18:13:42.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033567630000000001 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754699+0000", 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23493340300000001 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754759+0000", 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0655000000000001e-05 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754827+0000", 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7341999999999993e-05 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65717 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.693378289000002, 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.268538165, 2024-04-15T18:13:42.231 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486303+0000", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3799999999999997e-07 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486306+0000", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5760000000000002e-06 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753379+0000", 2024-04-15T18:13:42.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26707288800000001 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753389+0000", 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4569999999999994e-06 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754686+0000", 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001297749 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754693+0000", 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6730000000000003e-06 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754840+0000", 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014728399999999999 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:65740 2.2 2:41578887:::benchmark_data_smithi159_80260_object65739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:13:42.235 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.662305826000001, 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23761864199999999, 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 65740 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517376+0000", 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.288e-06 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517435+0000", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9138000000000002e-05 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517458+0000", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2416e-05 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517465+0000", 2024-04-15T18:13:42.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6739999999999997e-06 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517511+0000", 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5875999999999999e-05 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517608+0000", 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6705000000000003e-05 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517706+0000", 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7916000000000002e-05 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519639+0000", 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019331750000000001 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754588+0000", 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.234948666 2024-04-15T18:13:42.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754886+0000", 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000298001 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754939+0000", 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3599e-05 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754993+0000", 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4187999999999997e-05 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65740 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.659990417000003, 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23491751399999999, 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.242 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519691+0000", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0999999999999998e-07 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519695+0000", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8770000000000003e-06 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.751599+0000", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23190482500000001 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.751605+0000", 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4070000000000001e-06 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754573+0000", 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002967819 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754581+0000", 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1349999999999992e-06 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754608+0000", 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6741e-05 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65740 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.659766914000002, 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.235089255, 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.246 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9999999999999997e-07 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519918+0000", 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.388e-06 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753511+0000", 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23359376100000001 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753519+0000", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3590000000000002e-06 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754874+0000", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013550649999999999 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754880+0000", 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1859999999999997e-06 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.755003+0000", 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122896 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:68954 2.2 2:4d9d74ec:::benchmark_data_smithi159_80260_object68953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:13:42.250 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.801308202000001, 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23112603400000001, 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 68954 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378373+0000", 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378440+0000", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6950000000000001e-05 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378456+0000", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6177000000000001e-05 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378463+0000", 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3100000000000003e-06 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378509+0000", 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5544999999999998e-05 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378587+0000", 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8337999999999994e-05 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378701+0000", 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113859 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.380314+0000", 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016132430000000001 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.380987+0000", 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067303000000000003 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609420+0000", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.228432627 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609456+0000", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6075000000000003e-05 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609498+0000", 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2230000000000001e-05 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.256 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:68954 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.798507798999999, 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22833046300000001, 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8999999999999998e-07 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381175+0000", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3520000000000001e-06 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609375+0000", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22819928 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609380+0000", 2024-04-15T18:13:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9200000000000003e-06 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609412+0000", 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2533999999999997e-05 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609417+0000", 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6260000000000003e-06 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609503+0000", 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6460999999999998e-05 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:68957 2.2 2:4af888cc:::benchmark_data_smithi159_80260_object68956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.798392589000002, 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23726344099999999, 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 68957 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381289+0000", 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699999999999997e-07 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381393+0000", 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010438300000000001 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381402+0000", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4200000000000007e-06 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381406+0000", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6569999999999996e-06 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381434+0000", 2024-04-15T18:13:42.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7229999999999998e-05 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381520+0000", 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6051999999999995e-05 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381604+0000", 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4414999999999997e-05 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.383060+0000", 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014556459999999999 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.383296+0000", 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000236689 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618491+0000", 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23519432200000001 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618525+0000", 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4384000000000001e-05 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618551+0000", 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6325999999999998e-05 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:68957 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 45.794861363000003, 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23373741200000001, 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384820+0000", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.37e-07 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384823+0000", 2024-04-15T18:13:42.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.393e-06 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618437+0000", 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23361385500000001 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618442+0000", 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2449999999999998e-06 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618483+0000", 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0392999999999998e-05 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618487+0000", 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7809999999999996e-06 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618557+0000", 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9208000000000006e-05 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.271 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:72915 2.2 2:40cc6233:::benchmark_data_smithi159_80260_object72914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:00.271755+0000", 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.907925810000002, 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21482610499999999, 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.272 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 72915 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271755+0000", 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271755+0000", 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271756+0000", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.11e-06 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271853+0000", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7634999999999996e-05 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271866+0000", 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.259e-05 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271874+0000", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.498e-06 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.271914+0000", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0720999999999999e-05 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.272026+0000", 2024-04-15T18:13:42.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011190499999999999 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.272151+0000", 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012451500000000001 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.273884+0000", 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017336319999999999 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.281352+0000", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074672599999999999 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486490+0000", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20513836799999999 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486527+0000", 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7002999999999998e-05 2024-04-15T18:13:42.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486581+0000", 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3868e-05 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:72921 2.2 2:4862ee1c:::benchmark_data_smithi159_80260_object72920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:00.278695+0000", 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 41.900985890999998, 2024-04-15T18:13:42.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20801935899999999, 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 72921 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278695+0000", 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278695+0000", 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278696+0000", 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.176e-06 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278787+0000", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0801e-05 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278798+0000", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1729000000000001e-05 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278805+0000", 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4180000000000002e-06 2024-04-15T18:13:42.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.278864+0000", 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9772999999999998e-05 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.279008+0000", 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014341199999999999 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.279194+0000", 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018636 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.281725+0000", 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025306830000000002 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.283123+0000", 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013981919999999999 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486635+0000", 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20351143699999999 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486673+0000", 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8099999999999998e-05 2024-04-15T18:13:42.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.486714+0000", 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1278e-05 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:89108 2.3 e156/114 ondisk, result = 0)", 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.041048686, 2024-04-15T18:13:42.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40252157700000002, 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:13:42.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6599999999999996e-07 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138636+0000", 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.348e-06 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540769+0000", 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40213289499999999 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540782+0000", 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3692999999999999e-05 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540945+0000", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162227 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540962+0000", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7819999999999999e-05 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.541153+0000", 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019102799999999999 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.289 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:97764 2.2 2:452aef37:::benchmark_data_smithi159_80260_object8651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 15.572903131, 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22587837799999999, 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.290 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 97764 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606778+0000", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7499999999999998e-07 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606784+0000", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1930000000000001e-06 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606796+0000", 2024-04-15T18:13:42.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2588999999999999e-05 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606802+0000", 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2940000000000002e-06 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606876+0000", 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3916000000000002e-05 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.607010+0000", 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013359799999999999 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.607143+0000", 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013261800000000001 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.609507+0000", 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023643539999999999 2024-04-15T18:13:42.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.610155+0000", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00064824200000000005 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832542+0000", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22238657100000001 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832605+0000", 2024-04-15T18:13:42.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3528999999999994e-05 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832656+0000", 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0498999999999999e-05 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:97776 2.2 2:45c2f118:::benchmark_data_smithi159_80260_object8663:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:26.630976+0000", 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 15.548704578000001, 2024-04-15T18:13:42.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20183261399999999, 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 97776 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:42.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.630976+0000", 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.630976+0000", 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.630977+0000", 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.048e-06 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.630983+0000", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5840000000000002e-06 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.630997+0000", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4687e-05 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.631006+0000", 2024-04-15T18:13:42.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5890000000000003e-06 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.631056+0000", 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9758e-05 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.631188+0000", 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013234 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:13:42.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.631313+0000", 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012499700000000001 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.633448+0000", 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021350420000000002 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.634038+0000", 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058959300000000002 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832712+0000", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19867478499999999 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832766+0000", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3769e-05 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832809+0000", 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2422000000000001e-05 2024-04-15T18:13:42.302 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.303 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:42.303 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:42.303 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:42.304 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:42 smithi082 ceph-mon[44206]: pgmap v471: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-15T18:13:42.304 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3074455179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:13:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:42 smithi049 ceph-mon[40087]: pgmap v471: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-15T18:13:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3074455179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:13:42.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:42 smithi159 ceph-mon[45336]: pgmap v471: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-04-15T18:13:42.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3074455179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:13:42.418 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:42.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:42.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:42.536 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:13:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":165,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:13:39.098599+0000","last_up_change":"2024-04-15T18:09:33.592748+0000","last_in_change":"2024-04-15T18:09:15.827520+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2024-04-15T18:11:31.642096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6819","nonce":1108239618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6821","nonce":1108239618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6825","nonce":1108239618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6823","nonce":1108239618}]},"public_addr":"172.21.15.49:6819/1108239618","cluster_addr":"172.21.15.49:6821/1108239618","heartbeat_back_addr":"172.21.15.49:6825/1108239618","heartbeat_front_addr":"172.21.15.49:6823/1108239618","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6827","nonce":4051312081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6829","nonce":4051312081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6833","nonce":4051312081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":4051312081},{"type":"v1","addr":"172.21.15.49:6831","nonce":4051312081}]},"public_addr":"172.21.15.49:6827/4051312081","cluster_addr":"172.21.15.49:6829/4051312081","heartbeat_back_addr":"172.21.15.49:6833/4051312081","heartbeat_front_addr":"172.21.15.49:6831/4051312081","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6809","nonce":1943742992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6811","nonce":1943742992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6815","nonce":1943742992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6813","nonce":1943742992}]},"public_addr":"172.21.15.82:6809/1943742992","cluster_addr":"172.21.15.82:6811/1943742992","heartbeat_back_addr":"172.21.15.82:6815/1943742992","heartbeat_front_addr":"172.21.15.82:6813/1943742992","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:04.075611+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:12:32.867277+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:19.904803+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:36.226339+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:38.958826+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:12:58.104642+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:12:27.085808+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:12:56.288405+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:27.029074+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:12:32.840239+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:00.710892+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:13:31.730695+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:13:42.572 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:42.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:13:43.003 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:13:43.134 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:43.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:43.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:43 smithi049 ceph-mon[40087]: osdmap e166: 12 total, 12 up, 12 in 2024-04-15T18:13:43.236 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:43.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:43.280 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:13:43.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:43.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:43.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:43 smithi082 ceph-mon[44206]: osdmap e166: 12 total, 12 up, 12 in 2024-04-15T18:13:43.316 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:43.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:43.371 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:43.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:43.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:43.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:43 smithi159 ceph-mon[45336]: osdmap e166: 12 total, 12 up, 12 in 2024-04-15T18:13:43.600 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:43.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:43.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:43.746 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:43.775 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:43.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:43.839 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:43.879 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:43.920 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:44.171 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:44.172 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:44 smithi082 ceph-mon[44206]: pgmap v473: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s 2024-04-15T18:13:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:44.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:44.274 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:44.303 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:44.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:44 smithi159 ceph-mon[45336]: pgmap v473: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s 2024-04-15T18:13:44.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:44 smithi049 ceph-mon[40087]: pgmap v473: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s 2024-04-15T18:13:44.755 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:44.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:44.862 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:44.893 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:44.915 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:44.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:45.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:45.053 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:45.077 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:45.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1727812219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.165 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:13:45.181 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:45.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1727812219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:45.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:45.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:45.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1727812219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:13:45.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:45.344 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:116288 2.7 2:ecd05aa0:::benchmark_data_smithi159_80260_object27175:head [delete] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:45.340261+0000", 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0019356340000000001, 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001958535, 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:45.345 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 116288 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340261+0000", 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340261+0000", 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340263+0000", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5939999999999999e-06 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340269+0000", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.669e-06 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340284+0000", 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5130000000000001e-05 2024-04-15T18:13:45.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340292+0000", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.678e-06 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340347+0000", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5086000000000001e-05 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340439+0000", 2024-04-15T18:13:45.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1759e-05 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:45.340520+0000", 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1330999999999998e-05 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 1 2024-04-15T18:13:45.349 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:45.380 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:45.404 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:45.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:45.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:45.556 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:45.584 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:45.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:45.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:45.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:45.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:45.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:45.987 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:46.076 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:46.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:46.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:13:46.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:46.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:46.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:46.241 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: pgmap v474: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: osdmap e167: 12 total, 12 up, 12 in 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:46.242 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2343987408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:46.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:46.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:46.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:46.398 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: pgmap v474: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s 2024-04-15T18:13:46.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:13:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: osdmap e167: 12 total, 12 up, 12 in 2024-04-15T18:13:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:46.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2343987408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:46.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:46.450 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:46.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:46.513 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: pgmap v474: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: osdmap e167: 12 total, 12 up, 12 in 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2343987408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:46.559 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:46.600 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:46.610 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:46.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:46.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:46.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:46.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:46.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:46.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:46.636 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:46.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:46.752 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:46.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:46.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:47.023 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:47.276 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osdmap e168: 12 total, 12 up, 12 in 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:47.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:47.381 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osdmap e168: 12 total, 12 up, 12 in 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.410 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:47.433 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:47.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osdmap e168: 12 total, 12 up, 12 in 2024-04-15T18:13:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:47.544 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:47.572 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:47.598 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:47.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:47.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:47.729 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:47.806 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:47.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:47.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:47.942 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:48.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:48.105 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:48.145 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.169 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:48.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:48.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:48 smithi082 ceph-mon[44206]: pgmap v477: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s 2024-04-15T18:13:48.320 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.398 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:48.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:48 smithi159 ceph-mon[45336]: pgmap v477: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s 2024-04-15T18:13:48.443 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:13:48.443 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:13:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:48.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:48 smithi049 ceph-mon[40087]: pgmap v477: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s 2024-04-15T18:13:48.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:48.539 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.564 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:48.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:48.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:13:48.671 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:48.702 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.724 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:48.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:48.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:48.858 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:49.112 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:49.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:49.129 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:13:49.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:49.258 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:49.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:13:49.467 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:49.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:49.473 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:49.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:49.607 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:49.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:49.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:49.909 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:50 smithi082 ceph-mon[44206]: pgmap v478: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:13:50.360 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:50.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:50.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:50 smithi049 ceph-mon[40087]: pgmap v478: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:13:50.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:50 smithi159 ceph-mon[45336]: pgmap v478: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:13:50.493 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:50.517 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:50.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:50.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:50.651 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:50.905 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:50.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:51.012 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:51.042 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:51.132 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:13:51.244 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:51.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:51.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:51.427 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:51.460 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:51.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:51.541 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:13:51.616 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:13:51.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:13:51.651 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:51.677 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:13:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:51.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:13:51.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:51.828 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:52.036 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:52.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:52.143 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:52.177 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:52.205 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:13:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:52.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:52.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:52 smithi082 ceph-mon[44206]: pgmap v479: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-04-15T18:13:52.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:52.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:13:52.310 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:52.343 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:52.364 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:52.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:52.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:52 smithi049 ceph-mon[40087]: pgmap v479: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-04-15T18:13:52.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:52.496 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:52.596 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:13:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:52.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:13:52.608 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:52 smithi159 ceph-mon[45336]: pgmap v479: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 962 op/s 2024-04-15T18:13:52.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:13:52.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:52.749 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:52.835 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:52.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:52.940 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:52.974 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:53.003 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:53.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:53.108 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:13:53.140 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:53.167 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:13:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:53.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:53.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:53 smithi049 ceph-mon[40087]: osdmap e169: 12 total, 12 up, 12 in 2024-04-15T18:13:53.270 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:53.311 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:53.346 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:53.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:53.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:13:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:13:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:13:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:53 smithi159 ceph-mon[45336]: osdmap e169: 12 total, 12 up, 12 in 2024-04-15T18:13:53.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:13:53.496 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:13:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:13:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:13:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:13:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1985392563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:13:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:53 smithi082 ceph-mon[44206]: osdmap e169: 12 total, 12 up, 12 in 2024-04-15T18:13:53.640 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:13:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:53.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:13:53.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:53.788 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:53.850 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:53.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:53.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:13:53.989 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:54.015 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:13:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:54.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:13:54.030 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:13:54.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:13:54.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:13:54.111 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:13:54.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:54.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:13:54.153 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:54.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:13:54.179 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:54.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:54.206 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:13:54.243 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:13:54.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:13:54.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:54.314 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:54.319 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:13:54.337 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:54.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:54.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:13:54.386 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:13:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:13:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:13:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:13:54.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:13:54.393 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:13:54.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:54 smithi049 ceph-mon[40087]: pgmap v481: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-15T18:13:54.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3881092809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:54.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:54.482 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:54.511 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:54.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:54.535 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:124555 2.9 e169/114 2:9c92c9f5:::benchmark_data_smithi159_80260_object35442:head v 169'7691, mlcod=169'7689)", 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:54.530502+0000", 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0025198590000000002, 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025411079999999998, 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:54.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530502+0000", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530502+0000", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530503+0000", 2024-04-15T18:13:54.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9599999999999999e-07 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530508+0000", 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7849999999999999e-06 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530526+0000", 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8709e-05 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530530+0000", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1269999999999996e-06 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530583+0000", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2182999999999998e-05 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.530608+0000", 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5454000000000001e-05 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:54.539 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:124547 2.8 2:157ade2e:::benchmark_data_smithi159_80260_object35434:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:54.522634+0000", 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.010388342, 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01045286, 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:54.540 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124547 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522634+0000", 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522634+0000", 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522635+0000", 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1119999999999999e-06 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522640+0000", 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8799999999999999e-06 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:54.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522651+0000", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1307e-05 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522657+0000", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.28e-06 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522707+0000", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9372000000000001e-05 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:54.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522799+0000", 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2174000000000002e-05 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522872+0000", 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2663000000000005e-05 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.529246+0000", 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0063748390000000002 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.529402+0000", 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015565300000000001 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:124548 2.8 2:1b8be884:::benchmark_data_smithi159_80260_object35435:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:54.522676+0000", 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.010345872000000001, 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01049782, 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:13:54.545 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124548 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522676+0000", 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522676+0000", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522677+0000", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0100000000000005e-07 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522679+0000", 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6900000000000001e-06 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522684+0000", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8659999999999997e-06 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.522688+0000", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4489999999999999e-06 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.526351+0000", 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036631139999999999 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.526424+0000", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3601000000000002e-05 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.526532+0000", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000107643 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.529395+0000", 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028627710000000001 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:13:54.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.529407+0000", 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1956e-05 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 3 2024-04-15T18:13:54.550 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:54.551 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:54 smithi082 ceph-mon[44206]: pgmap v481: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-15T18:13:54.551 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3881092809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:54.616 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:13:54.649 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:54.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:54 smithi159 ceph-mon[45336]: pgmap v481: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s 2024-04-15T18:13:54.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3881092809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:13:54.848 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:13:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:54.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:13:54.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:13:54.982 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:55.204 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.204 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:13:55.335 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:13:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:55.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:13:55.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:13:55.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:55.467 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.467 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.469 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:55.494 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:55.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:55.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:13:55.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:55.641 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:13:55.660 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:13:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:55.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:13:55.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:55.807 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:55.836 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:55.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:55.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:55.942 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:13:55.975 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:55.999 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:13:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:56.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:13:56.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:13:56.135 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:56.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:56.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:56 smithi082 ceph-mon[44206]: pgmap v482: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-04-15T18:13:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:56 smithi049 ceph-mon[40087]: pgmap v482: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-04-15T18:13:56.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:13:56.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:56 smithi159 ceph-mon[45336]: pgmap v482: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-04-15T18:13:57.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:13:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:57.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:13:57.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:13:57.336 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:57.365 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:13:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:57.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:13:57.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:13:57.501 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:57.711 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:13:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:13:57.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:13:57.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:13:57.850 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:13:57.964 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:13:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:13:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:13:57.965 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:13:57.965 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:13:58.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:13:57 smithi049 ceph-mon[40087]: pgmap v483: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s 2024-04-15T18:13:58.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:13:57 smithi082 ceph-mon[44206]: pgmap v483: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s 2024-04-15T18:13:58.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:13:57 smithi159 ceph-mon[45336]: pgmap v483: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s 2024-04-15T18:13:58.740 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:13:59.051 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:14:00.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:00 smithi082 ceph-mon[44206]: pgmap v484: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-04-15T18:14:00.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:00 smithi159 ceph-mon[45336]: pgmap v484: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-04-15T18:14:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:00 smithi049 ceph-mon[40087]: pgmap v484: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-04-15T18:14:01.041 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:14:01.041 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:14:01.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:14:01.041 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:14:01.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4193 2.3 e124/114 2:c3a89f7e:::benchmark_data_smithi159_80260_object4192:head v 124'273, mlcod=124'269)", 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.632002+0000", 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 131.403683656, 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203317842, 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632002+0000", 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632002+0000", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632003+0000", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7400000000000003e-07 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632024+0000", 2024-04-15T18:14:01.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0854999999999999e-05 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834866+0000", 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.202841824 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834871+0000", 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4890000000000003e-06 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834921+0000", 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9536999999999997e-05 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834947+0000", 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6716e-05 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835292+0000", 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034516899999999999 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835320+0000", 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7577999999999999e-05 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4220 2.b e124/114 2:dd4bfb9c:::benchmark_data_smithi159_80260_object4219:head v 124'252, mlcod=124'249)", 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.649307+0000", 2024-04-15T18:14:01.046 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 131.386378971, 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.185115896, 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.649307+0000", 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.649307+0000", 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.649307+0000", 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8800000000000002e-07 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.649332+0000", 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4615999999999999e-05 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834003+0000", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18467093500000001 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834009+0000", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2190000000000001e-06 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834049+0000", 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0244999999999998e-05 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834063+0000", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3974999999999999e-05 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834392+0000", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032822199999999998 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.834423+0000", 2024-04-15T18:14:01.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1096000000000002e-05 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4637 2.3 e124/114 2:cf814ddf:::benchmark_data_smithi159_80260_object4636:head v 124'303, mlcod=124'299)", 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:50.148692+0000", 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.88699419599999, 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.182871059, 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.148692+0000", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.148692+0000", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.148693+0000", 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.029e-06 2024-04-15T18:14:01.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.148713+0000", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0729000000000001e-05 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331106+0000", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.182392797 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331112+0000", 2024-04-15T18:14:01.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.127e-06 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331157+0000", 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5025999999999999e-05 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331172+0000", 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4569999999999999e-05 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331523+0000", 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035134599999999999 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.331563+0000", 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9434999999999998e-05 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6449 2.8 2:15139674:::benchmark_data_smithi159_80260_object6448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.513524+0000", 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 129.52216143499999, 2024-04-15T18:14:01.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.159718465, 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6449 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513524+0000", 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513524+0000", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513525+0000", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8400000000000004e-07 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513583+0000", 2024-04-15T18:14:01.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8334000000000001e-05 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513593+0000", 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0144e-05 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513599+0000", 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1399999999999999e-06 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513633+0000", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4764000000000001e-05 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513711+0000", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7267000000000004e-05 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.513804+0000", 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3392000000000003e-05 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514194+0000", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038958299999999999 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.514574+0000", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038049999999999998 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.673168+0000", 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.158594349 2024-04-15T18:14:01.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.673201+0000", 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2626000000000002e-05 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.673243+0000", 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1782000000000001e-05 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:7809 2.8 2:152bc777:::benchmark_data_smithi159_80260_object7808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e125)", 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:52.893827+0000", 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 128.14185889999999, 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18124950400000001, 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7809 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.062 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893827+0000", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893827+0000", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893828+0000", 2024-04-15T18:14:01.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3099999999999997e-07 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893906+0000", 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8338999999999997e-05 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893915+0000", 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2459999999999992e-06 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.893920+0000", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1610000000000002e-06 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.894010+0000", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9851000000000004e-05 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.894083+0000", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2426000000000001e-05 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.894178+0000", 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5742999999999995e-05 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.894570+0000", 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039177299999999999 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:52.904169+0000", 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0095987569999999994 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:53.075017+0000", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.170848529 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:53.075051+0000", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4128000000000003e-05 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:53.075076+0000", 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4919999999999999e-05 2024-04-15T18:14:01.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:33638 2.8 2:1c68108a:::benchmark_data_smithi159_80260_object33637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:11.608738+0000", 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 109.426947459, 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.214346078, 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.068 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 33638 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608738+0000", 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608738+0000", 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608739+0000", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6499999999999998e-07 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608805+0000", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6347999999999996e-05 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608814+0000", 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9789999999999999e-06 2024-04-15T18:14:01.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608822+0000", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1300000000000003e-06 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608862+0000", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0327000000000002e-05 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.608980+0000", 2024-04-15T18:14:01.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117871 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.609127+0000", 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014742999999999999 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.609929+0000", 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080177700000000005 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.610615+0000", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00068634999999999996 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823008+0000", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.212392988 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823044+0000", 2024-04-15T18:14:01.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6121000000000002e-05 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823084+0000", 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9991999999999999e-05 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:33657 2.8 2:14ca00c2:::benchmark_data_smithi159_80260_object33656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:11.625609+0000", 2024-04-15T18:14:01.074 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 109.410077141, 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.198070301, 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 33657 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625609+0000", 2024-04-15T18:14:01.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625609+0000", 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625610+0000", 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1340000000000001e-06 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625684+0000", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3770999999999996e-05 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625694+0000", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0057999999999999e-05 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625699+0000", 2024-04-15T18:14:01.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3630000000000002e-06 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625739+0000", 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9895999999999999e-05 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625832+0000", 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2981999999999998e-05 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.625922+0000", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9708000000000002e-05 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.626883+0000", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00096166899999999996 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.627692+0000", 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000809156 2024-04-15T18:14:01.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823620+0000", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19592779499999999 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823654+0000", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3593999999999997e-05 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823679+0000", 2024-04-15T18:14:01.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5174999999999999e-05 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:33638 2.8 e133/114 ondisk, result = 0)", 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:11.627499+0000", 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 109.40818638099999, 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19559105500000001, 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:14:01.081 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.627499+0000", 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.627499+0000", 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.627500+0000", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1399999999999997e-07 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.627503+0000", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.016e-06 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.822654+0000", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19515070000000001 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.822659+0000", 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.682e-06 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.822999+0000", 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033908099999999997 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823005+0000", 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0700000000000003e-06 2024-04-15T18:14:01.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:11.823090+0000", 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5891999999999997e-05 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:41156 2.8 2:1aa6b45d:::benchmark_data_smithi159_80260_object41155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e136)", 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:17.237143+0000", 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 103.798542399, 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216278309, 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.085 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 41156 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237143+0000", 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237143+0000", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237145+0000", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1459999999999999e-06 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237203+0000", 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8913000000000003e-05 2024-04-15T18:14:01.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237215+0000", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1103e-05 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237221+0000", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9730000000000002e-06 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237264+0000", 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3517999999999999e-05 2024-04-15T18:14:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237351+0000", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7406999999999998e-05 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.237439+0000", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7386999999999995e-05 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.238867+0000", 2024-04-15T18:14:01.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014276829999999999 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.239123+0000", 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000256142 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.453319+0000", 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21419681900000001 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.453374+0000", 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4401999999999999e-05 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:17.453422+0000", 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7815999999999998e-05 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:60277 2.8 2:1f76c904:::benchmark_data_smithi159_80260_object60276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:33.811560+0000", 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.224126049000006, 2024-04-15T18:14:01.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19606552599999999, 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 60277 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811560+0000", 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811560+0000", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811561+0000", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.409e-06 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811616+0000", 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.499e-05 2024-04-15T18:14:01.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811630+0000", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3740000000000001e-05 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811638+0000", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8210000000000002e-06 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811685+0000", 2024-04-15T18:14:01.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7558000000000001e-05 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811790+0000", 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104745 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.811888+0000", 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8079999999999996e-05 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.813421+0000", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001532425 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.822318+0000", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0088974480000000005 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007563+0000", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18524521499999999 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007598+0000", 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5148999999999998e-05 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007625+0000", 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6945999999999999e-05 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:60277 2.8 e143/114 ondisk, result = 0)", 2024-04-15T18:14:01.097 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:33.814031+0000", 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.221654432999998, 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19359959600000001, 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.814031+0000", 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.814031+0000", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.814032+0000", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3600000000000003e-07 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:33.814036+0000", 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8820000000000003e-06 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007513+0000", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19347714499999999 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007518+0000", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8339999999999998e-06 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007555+0000", 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7299999999999999e-05 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007560+0000", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7489999999999998e-06 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:34.007631+0000", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1050000000000006e-05 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:72950 2.0 2:048e7642:::benchmark_data_smithi159_80260_object72949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e152)", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:00.312659+0000", 2024-04-15T18:14:01.101 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 60.723027055000003, 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17770656300000001, 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 72950 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312659+0000", 2024-04-15T18:14:01.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312659+0000", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312659+0000", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4700000000000001e-07 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312716+0000", 2024-04-15T18:14:01.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6375000000000001e-05 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312726+0000", 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0403e-05 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312732+0000", 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8669999999999999e-06 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312770+0000", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7703000000000001e-05 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312862+0000", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1858e-05 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.312945+0000", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2806000000000006e-05 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.314854+0000", 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00190913 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.316922+0000", 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020686189999999998 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.490301+0000", 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.173378538 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.490337+0000", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6018e-05 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:00.490365+0000", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8498999999999999e-05 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:76730 2.8 2:19395dcd:::benchmark_data_smithi159_80260_object76729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:03.963939+0000", 2024-04-15T18:14:01.107 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.071746587, 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22101702200000001, 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 76730 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.963939+0000", 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.963939+0000", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.963940+0000", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5099999999999999e-07 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.964013+0000", 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3061999999999999e-05 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.964023+0000", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7589999999999992e-06 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.964028+0000", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3380000000000004e-06 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970099+0000", 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0060704629999999999 2024-04-15T18:14:01.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970219+0000", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120146 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970365+0000", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014595400000000001 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.974606+0000", 2024-04-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042408560000000003 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.974618+0000", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2638e-05 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184868+0000", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21025018100000001 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184903+0000", 2024-04-15T18:14:01.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4362000000000001e-05 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184956+0000", 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3511999999999999e-05 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:76732 2.0 2:0ab39dd4:::benchmark_data_smithi159_80260_object76731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:03.965785+0000", 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.069900933, 2024-04-15T18:14:01.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21915143100000001, 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 76732 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965785+0000", 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965785+0000", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965786+0000", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5799999999999998e-07 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965873+0000", 2024-04-15T18:14:01.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7390000000000002e-05 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965890+0000", 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7022999999999999e-05 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.965899+0000", 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2059999999999996e-06 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970109+0000", 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042100820000000004 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970218+0000", 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108936 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.970365+0000", 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014633100000000001 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.974274+0000", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039088480000000004 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.974598+0000", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032414400000000001 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184845+0000", 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.210247777 2024-04-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184895+0000", 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9592000000000002e-05 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184936+0000", 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1244e-05 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:76736 2.8 2:18a7bb72:::benchmark_data_smithi159_80260_object76735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:03.972568+0000", 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.063118187999997, 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21250302500000001, 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 76736 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972568+0000", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972568+0000", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972569+0000", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3400000000000001e-06 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972668+0000", 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8904e-05 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972677+0000", 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3400000000000004e-06 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972683+0000", 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5690000000000004e-06 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972765+0000", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2634999999999997e-05 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972891+0000", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125647 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.972989+0000", 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8414999999999999e-05 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.974627+0000", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00163737 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.976703+0000", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020756989999999999 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.184996+0000", 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20829377299999999 2024-04-15T18:14:01.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.185031+0000", 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4978999999999998e-05 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.185071+0000", 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9354000000000003e-05 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.125 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:76752 2.8 2:1da57f5a:::benchmark_data_smithi159_80260_object76751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e153)", 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:03.989754+0000", 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.045931715999998, 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20140059199999999, 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 76752 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989754+0000", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989754+0000", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989756+0000", 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4500000000000001e-06 2024-04-15T18:14:01.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989848+0000", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1970000000000002e-05 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989860+0000", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2404e-05 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989868+0000", 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3780000000000001e-06 2024-04-15T18:14:01.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.989920+0000", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1937000000000001e-05 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.990029+0000", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108381 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.990201+0000", 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017204199999999999 2024-04-15T18:14:01.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.992285+0000", 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020838919999999999 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:03.994244+0000", 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019593330000000002 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.191061+0000", 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.196817452 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.191102+0000", 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0818999999999997e-05 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.191155+0000", 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2534000000000002e-05 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.131 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:76765 2.9 e153/114 2:97849de7:::benchmark_data_smithi159_80260_object76764:head v 153'4763, mlcod=153'4758)", 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:04.016186+0000", 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.019499334000002, 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17846125299999999, 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.016186+0000", 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.016186+0000", 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.016187+0000", 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4299999999999999e-07 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.016244+0000", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7244999999999998e-05 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.192220+0000", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17597569299999999 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.192225+0000", 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9779999999999996e-06 2024-04-15T18:14:01.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.192268+0000", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3519000000000001e-05 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.192288+0000", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0000000000000002e-05 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.194617+0000", 2024-04-15T18:14:01.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023284529999999999 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.194648+0000", 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1121999999999998e-05 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:76774 2.9 e153/114 2:9897c163:::benchmark_data_smithi159_80260_object76773:head v 153'4764, mlcod=153'4760)", 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:04.032448+0000", 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 57.003237335999998, 2024-04-15T18:14:01.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17451255099999999, 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.032448+0000", 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.032448+0000", 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.032474+0000", 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5578000000000001e-05 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.032504+0000", 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0136000000000001e-05 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.204092+0000", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17158747399999999 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.204096+0000", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5689999999999998e-06 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.204145+0000", 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8337999999999997e-05 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.204165+0000", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0933000000000001e-05 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.206915+0000", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027497860000000002 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:04.206961+0000", 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5736999999999999e-05 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.140 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:124400 2.8 2:1668ff5a:::benchmark_data_smithi159_80260_object35287:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:54.343114+0000", 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 6.6925716389999996, 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16493423800000001, 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124400 2024-04-15T18:14:01.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343114+0000", 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343114+0000", 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343115+0000", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4799999999999997e-07 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343119+0000", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0080000000000004e-06 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343129+0000", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0414000000000001e-05 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343135+0000", 2024-04-15T18:14:01.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4380000000000003e-06 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343202+0000", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7255000000000006e-05 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343282+0000", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9763000000000002e-05 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.343346+0000", 2024-04-15T18:14:01.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4542999999999995e-05 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.345871+0000", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025249719999999999 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.348238+0000", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023669210000000001 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.507918+0000", 2024-04-15T18:14:01.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15968006900000001 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.507978+0000", 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9679999999999998e-05 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:54.508048+0000", 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0327000000000006e-05 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:127469 2.9 e169/114 2:9b9827ef:::benchmark_data_smithi159_80260_object38356:head v 169'7842, mlcod=169'7839)", 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:57.825502+0000", 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.2101833559999999, 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19811111200000001, 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:57.825502+0000", 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:57.825502+0000", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:57.825503+0000", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2399999999999997e-07 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:57.825509+0000", 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.195e-06 2024-04-15T18:14:01.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.019997+0000", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19448748199999999 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.020003+0000", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1909999999999997e-06 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.020069+0000", 2024-04-15T18:14:01.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6304999999999996e-05 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.020088+0000", 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8732e-05 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.023582+0000", 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003494203 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:14:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:58.023614+0000", 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1279999999999999e-05 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:14:01.151 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:14:01.308 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:01.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:01.354 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:01.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:01.458 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:01.482 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:01.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:01.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:01.621 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:01.871 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:01.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:01.888 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:14:01.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:02.022 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:02.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:02.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:02.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:02 smithi049 ceph-mon[40087]: pgmap v485: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-04-15T18:14:02.135 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:02 smithi049 ceph-mon[40087]: osdmap e170: 12 total, 12 up, 12 in 2024-04-15T18:14:02.230 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:02.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:02.255 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:02.335 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:02.367 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:02.367 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:02.367 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:02 smithi159 ceph-mon[45336]: pgmap v485: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-04-15T18:14:02.367 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:02 smithi159 ceph-mon[45336]: osdmap e170: 12 total, 12 up, 12 in 2024-04-15T18:14:02.373 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:02.503 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:02.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:02 smithi082 ceph-mon[44206]: pgmap v485: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-04-15T18:14:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:02 smithi082 ceph-mon[44206]: osdmap e170: 12 total, 12 up, 12 in 2024-04-15T18:14:02.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:02.639 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:02.664 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:02.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:02.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:02.802 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:03.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:14:03.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:14:03.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:14:03.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:03.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:03.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:03 smithi049 ceph-mon[40087]: osdmap e171: 12 total, 12 up, 12 in 2024-04-15T18:14:03.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:03.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:03.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:03 smithi159 ceph-mon[45336]: osdmap e171: 12 total, 12 up, 12 in 2024-04-15T18:14:03.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:03.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:03.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:03 smithi082 ceph-mon[44206]: osdmap e171: 12 total, 12 up, 12 in 2024-04-15T18:14:03.756 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:03.757 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:03.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:03.828 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:03.863 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:03.900 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:03.986 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:03.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:04.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:04.137 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:04.165 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:14:04.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:04 smithi049 ceph-mon[40087]: 2.c scrub starts 2024-04-15T18:14:04.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:04 smithi049 ceph-mon[40087]: pgmap v488: 17 pgs: 17 active+clean; 3.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s 2024-04-15T18:14:04.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:04 smithi049 ceph-mon[40087]: 2.c scrub ok 2024-04-15T18:14:04.306 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/207104699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:04.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:04 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:04.324 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:132711 2.b 2:d8206eb1:::benchmark_data_smithi159_80260_object43598:head [delete] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:04.302059+0000", 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.019724598999999999, 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019752412, 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 132711 2024-04-15T18:14:04.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302059+0000", 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302059+0000", 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:04.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302060+0000", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1739999999999999e-06 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302064+0000", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6149999999999999e-06 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302077+0000", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2517999999999999e-05 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:04.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302084+0000", 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5689999999999999e-06 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302137+0000", 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2796000000000001e-05 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302261+0000", 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012357399999999999 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.302387+0000", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012583599999999999 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.305717+0000", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033303870000000002 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.307140+0000", 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0014228960000000001 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:04.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:132720 2.b 2:dbe41c08:::benchmark_data_smithi159_80260_object43607:head [delete] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:04.318457+0000", 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0033264710000000001, 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034178870000000001, 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 132720 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.330 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318457+0000", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318457+0000", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318458+0000", 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.31e-06 2024-04-15T18:14:04.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318464+0000", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0360000000000003e-06 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318477+0000", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3407000000000001e-05 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318483+0000", 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0290000000000002e-06 2024-04-15T18:14:04.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318528+0000", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4576000000000001e-05 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318621+0000", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3429000000000003e-05 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:04.318694+0000", 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2342000000000006e-05 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:04.333 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:14:04.334 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 2 2024-04-15T18:14:04.334 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:04.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:04 smithi159 ceph-mon[45336]: 2.c scrub starts 2024-04-15T18:14:04.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:04 smithi159 ceph-mon[45336]: pgmap v488: 17 pgs: 17 active+clean; 3.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s 2024-04-15T18:14:04.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:04 smithi159 ceph-mon[45336]: 2.c scrub ok 2024-04-15T18:14:04.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/207104699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:04.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:04 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:04 smithi082 ceph-mon[44206]: 2.c scrub starts 2024-04-15T18:14:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:04 smithi082 ceph-mon[44206]: pgmap v488: 17 pgs: 17 active+clean; 3.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s 2024-04-15T18:14:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:04 smithi082 ceph-mon[44206]: 2.c scrub ok 2024-04-15T18:14:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/207104699' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:04 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:05.171 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:14:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:05 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:05 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:05 smithi159 ceph-mon[45336]: osdmap e172: 12 total, 12 up, 12 in 2024-04-15T18:14:05.468 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:05 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:05 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:05 smithi049 ceph-mon[40087]: osdmap e172: 12 total, 12 up, 12 in 2024-04-15T18:14:05.517 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:05.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:05 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:05 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:05 smithi082 ceph-mon[44206]: osdmap e172: 12 total, 12 up, 12 in 2024-04-15T18:14:05.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:05.658 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:05.684 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:05.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:05.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:05.830 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:05.854 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:05.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:05.960 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:06.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:06.372 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:06 smithi159 ceph-mon[45336]: pgmap v490: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-04-15T18:14:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:06 smithi049 ceph-mon[40087]: pgmap v490: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-04-15T18:14:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:06 smithi082 ceph-mon[44206]: pgmap v490: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 907 op/s 2024-04-15T18:14:06.867 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:06.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:06.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:07.015 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:07.147 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:07.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:07.287 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:07.309 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:07.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:07.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:07.452 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:07.484 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:07.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:07.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:14:07.590 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:07.616 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:07.634 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:07.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:07.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:07.814 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:07.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:07.949 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:07.949 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:07.964 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:07.967 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:07.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:08.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:08.041 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:08.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:08.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:08.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:08.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:08.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:14:08.076 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:08.085 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:14:08.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:08 smithi049 ceph-mon[40087]: pgmap v491: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-15T18:14:08.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3184793254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:08.089 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:08.127 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:08 smithi082 ceph-mon[44206]: pgmap v491: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-15T18:14:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3184793254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:08.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:08 smithi159 ceph-mon[45336]: pgmap v491: 17 pgs: 17 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-04-15T18:14:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3184793254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:08.718 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:08.718 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:14:08.719 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:14:08.719 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:14:08.719 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:09 smithi049 ceph-mon[40087]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:09 smithi082 ceph-mon[44206]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:09.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:09 smithi159 ceph-mon[45336]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:09.987 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-04-15T18:14:07.062762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2949251640,"num_objects":44996,"num_object_clones":0,"num_object_copies":134988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44996,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":133409,"num_write_kb":5705979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133365,"ondisk_log_size":133365,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9460620,"kb_used_data":8775492,"kb_used_omap":18,"kb_used_meta":684973,"kb_avail":1115481204,"statfs":{"total":1151940427776,"available":1142252752896,"internally_reserved":0,"allocated":8986103808,"data_stored":8982625634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":701412665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,3,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-323158016,"num_objects":-4931,"num_object_clones":0,"num_object_copies":-14793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001216"},"pg_stats":[{"pgid":"2.f","version":"172'8282","reported_seq":8466,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:05.199525+0000","last_change":"2024-04-15T18:13:07.121893+0000","last_active":"2024-04-15T18:14:05.199525+0000","last_peered":"2024-04-15T18:14:05.199525+0000","last_clean":"2024-04-15T18:14:05.199525+0000","last_became_active":"2024-04-15T18:11:33.100145+0000","last_became_peered":"2024-04-15T18:11:33.100145+0000","last_unstale":"2024-04-15T18:14:05.199525+0000","last_undegraded":"2024-04-15T18:14:05.199525+0000","last_fullsized":"2024-04-15T18:14:05.199525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4926","last_scrub_stamp":"2024-04-15T18:13:07.121782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.121782+0000","objects_scrubbed":4918,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:13.567121+0000","scrub_duration":1003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":354176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"172'8476","reported_seq":8662,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:06.702437+0000","last_change":"2024-04-15T18:13:07.069832+0000","last_active":"2024-04-15T18:14:06.702437+0000","last_peered":"2024-04-15T18:14:06.702437+0000","last_clean":"2024-04-15T18:14:06.702437+0000","last_became_active":"2024-04-15T18:11:33.230674+0000","last_became_peered":"2024-04-15T18:11:33.230674+0000","last_unstale":"2024-04-15T18:14:06.702437+0000","last_undegraded":"2024-04-15T18:14:06.702437+0000","last_fullsized":"2024-04-15T18:14:06.702437+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'5007","last_scrub_stamp":"2024-04-15T18:13:07.069711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.069711+0000","objects_scrubbed":4976,"log_size":8476,"log_dups_size":0,"ondisk_log_size":8476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:13.240340+0000","scrub_duration":1832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8476,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"172'8099","reported_seq":8282,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.373318+0000","last_change":"2024-04-15T18:13:07.360699+0000","last_active":"2024-04-15T18:14:04.373318+0000","last_peered":"2024-04-15T18:14:04.373318+0000","last_clean":"2024-04-15T18:14:04.373318+0000","last_became_active":"2024-04-15T18:11:33.099128+0000","last_became_peered":"2024-04-15T18:11:33.099128+0000","last_unstale":"2024-04-15T18:14:04.373318+0000","last_undegraded":"2024-04-15T18:14:04.373318+0000","last_fullsized":"2024-04-15T18:14:04.373318+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4836","last_scrub_stamp":"2024-04-15T18:13:07.360568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.360568+0000","objects_scrubbed":4806,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:30.621360+0000","scrub_duration":1308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":349696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"171'8172","reported_seq":8349,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-15T18:14:02.679065+0000","last_change":"2024-04-15T18:13:00.209536+0000","last_active":"2024-04-15T18:14:02.679065+0000","last_peered":"2024-04-15T18:14:02.679065+0000","last_clean":"2024-04-15T18:14:02.679065+0000","last_became_active":"2024-04-15T18:11:33.232175+0000","last_became_peered":"2024-04-15T18:11:33.232175+0000","last_unstale":"2024-04-15T18:14:02.679065+0000","last_undegraded":"2024-04-15T18:14:02.679065+0000","last_fullsized":"2024-04-15T18:14:02.679065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4521","last_scrub_stamp":"2024-04-15T18:13:00.209243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:00.209243+0000","objects_scrubbed":4488,"log_size":8172,"log_dups_size":0,"ondisk_log_size":8172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:11.171237+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":8658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":353856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"172'8421","reported_seq":8603,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.338100+0000","last_change":"2024-04-15T18:13:04.626201+0000","last_active":"2024-04-15T18:14:04.338100+0000","last_peered":"2024-04-15T18:14:04.338100+0000","last_clean":"2024-04-15T18:14:04.338100+0000","last_became_active":"2024-04-15T18:11:33.097184+0000","last_became_peered":"2024-04-15T18:11:33.097184+0000","last_unstale":"2024-04-15T18:14:04.338100+0000","last_undegraded":"2024-04-15T18:14:04.338100+0000","last_fullsized":"2024-04-15T18:14:04.338100+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4916","last_scrub_stamp":"2024-04-15T18:13:04.626108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:04.626108+0000","objects_scrubbed":4867,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:12.478910+0000","scrub_duration":1619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":362496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":521,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.148728+0000","last_change":"2024-04-15T18:13:21.859558+0000","last_active":"2024-04-15T18:14:04.148728+0000","last_peered":"2024-04-15T18:14:04.148728+0000","last_clean":"2024-04-15T18:14:04.148728+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:14:04.148728+0000","last_undegraded":"2024-04-15T18:14:04.148728+0000","last_fullsized":"2024-04-15T18:14:04.148728+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:13:21.859468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:13:21.859468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:49.623977+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"172'8259","reported_seq":8445,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:05.915054+0000","last_change":"2024-04-15T18:13:07.614724+0000","last_active":"2024-04-15T18:14:05.915054+0000","last_peered":"2024-04-15T18:14:05.915054+0000","last_clean":"2024-04-15T18:14:05.915054+0000","last_became_active":"2024-04-15T18:11:33.230198+0000","last_became_peered":"2024-04-15T18:11:33.230198+0000","last_unstale":"2024-04-15T18:14:05.915054+0000","last_undegraded":"2024-04-15T18:14:05.915054+0000","last_fullsized":"2024-04-15T18:14:05.915054+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4935","last_scrub_stamp":"2024-04-15T18:13:07.614548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.614548+0000","objects_scrubbed":4911,"log_size":8259,"log_dups_size":0,"ondisk_log_size":8259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:22.135685+0000","scrub_duration":706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176881664,"num_objects":2699,"num_object_clones":0,"num_object_copies":8097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":350656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"172'8256","reported_seq":8441,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:05.264541+0000","last_change":"2024-04-15T18:13:11.048746+0000","last_active":"2024-04-15T18:14:05.264541+0000","last_peered":"2024-04-15T18:14:05.264541+0000","last_clean":"2024-04-15T18:14:05.264541+0000","last_became_active":"2024-04-15T18:11:33.230675+0000","last_became_peered":"2024-04-15T18:11:33.230675+0000","last_unstale":"2024-04-15T18:14:05.264541+0000","last_undegraded":"2024-04-15T18:14:05.264541+0000","last_fullsized":"2024-04-15T18:14:05.264541+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'5138","last_scrub_stamp":"2024-04-15T18:13:11.048581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:11.048581+0000","objects_scrubbed":5103,"log_size":8256,"log_dups_size":0,"ondisk_log_size":8256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:35.701816+0000","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"172'8464","reported_seq":8652,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:06.689414+0000","last_change":"2024-04-15T18:13:11.928691+0000","last_active":"2024-04-15T18:14:06.689414+0000","last_peered":"2024-04-15T18:14:06.689414+0000","last_clean":"2024-04-15T18:14:06.689414+0000","last_became_active":"2024-04-15T18:11:33.233286+0000","last_became_peered":"2024-04-15T18:11:33.233286+0000","last_unstale":"2024-04-15T18:14:06.689414+0000","last_undegraded":"2024-04-15T18:14:06.689414+0000","last_fullsized":"2024-04-15T18:14:06.689414+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'5250","last_scrub_stamp":"2024-04-15T18:13:11.928500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:11.928500+0000","objects_scrubbed":5200,"log_size":8464,"log_dups_size":0,"ondisk_log_size":8464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:14.574074+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176947200,"num_objects":2700,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8464,"num_write_kb":357248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"172'8289","reported_seq":8470,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.878787+0000","last_change":"2024-04-15T18:13:03.394418+0000","last_active":"2024-04-15T18:14:04.878787+0000","last_peered":"2024-04-15T18:14:04.878787+0000","last_clean":"2024-04-15T18:14:04.878787+0000","last_became_active":"2024-04-15T18:11:33.230359+0000","last_became_peered":"2024-04-15T18:11:33.230359+0000","last_unstale":"2024-04-15T18:14:04.878787+0000","last_undegraded":"2024-04-15T18:14:04.878787+0000","last_fullsized":"2024-04-15T18:14:04.878787+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4725","last_scrub_stamp":"2024-04-15T18:13:03.394223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:03.394223+0000","objects_scrubbed":4680,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:13.066150+0000","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183828480,"num_objects":2805,"num_object_clones":0,"num_object_copies":8415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8289,"num_write_kb":355008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"172'8352","reported_seq":8535,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:05.930326+0000","last_change":"2024-04-15T18:13:04.996518+0000","last_active":"2024-04-15T18:14:05.930326+0000","last_peered":"2024-04-15T18:14:05.930326+0000","last_clean":"2024-04-15T18:14:05.930326+0000","last_became_active":"2024-04-15T18:11:33.230216+0000","last_became_peered":"2024-04-15T18:11:33.230216+0000","last_unstale":"2024-04-15T18:14:05.930326+0000","last_undegraded":"2024-04-15T18:14:05.930326+0000","last_fullsized":"2024-04-15T18:14:05.930326+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4825","last_scrub_stamp":"2024-04-15T18:13:04.996348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:04.996348+0000","objects_scrubbed":4784,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:19.394686+0000","scrub_duration":1113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182976512,"num_objects":2792,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"172'8309","reported_seq":8492,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:06.635511+0000","last_change":"2024-04-15T18:13:06.364251+0000","last_active":"2024-04-15T18:14:06.635511+0000","last_peered":"2024-04-15T18:14:06.635511+0000","last_clean":"2024-04-15T18:14:06.635511+0000","last_became_active":"2024-04-15T18:11:33.232432+0000","last_became_peered":"2024-04-15T18:11:33.232432+0000","last_unstale":"2024-04-15T18:14:06.635511+0000","last_undegraded":"2024-04-15T18:14:06.635511+0000","last_fullsized":"2024-04-15T18:14:06.635511+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4885","last_scrub_stamp":"2024-04-15T18:13:06.364130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:06.364130+0000","objects_scrubbed":4850,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:11.610143+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8309,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"172'8429","reported_seq":8614,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.581654+0000","last_change":"2024-04-15T18:13:05.338257+0000","last_active":"2024-04-15T18:14:04.581654+0000","last_peered":"2024-04-15T18:14:04.581654+0000","last_clean":"2024-04-15T18:14:04.581654+0000","last_became_active":"2024-04-15T18:11:33.232105+0000","last_became_peered":"2024-04-15T18:11:33.232105+0000","last_unstale":"2024-04-15T18:14:04.581654+0000","last_undegraded":"2024-04-15T18:14:04.581654+0000","last_fullsized":"2024-04-15T18:14:04.581654+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4936","last_scrub_stamp":"2024-04-15T18:13:05.338106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.338106+0000","objects_scrubbed":4907,"log_size":8429,"log_dups_size":0,"ondisk_log_size":8429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:34.463184+0000","scrub_duration":1140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":8691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":362432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"172'8484","reported_seq":8672,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:05.235517+0000","last_change":"2024-04-15T18:13:12.027697+0000","last_active":"2024-04-15T18:14:05.235517+0000","last_peered":"2024-04-15T18:14:05.235517+0000","last_clean":"2024-04-15T18:14:05.235517+0000","last_became_active":"2024-04-15T18:11:33.099322+0000","last_became_peered":"2024-04-15T18:11:33.099322+0000","last_unstale":"2024-04-15T18:14:05.235517+0000","last_undegraded":"2024-04-15T18:14:05.235517+0000","last_fullsized":"2024-04-15T18:14:05.235517+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'5299","last_scrub_stamp":"2024-04-15T18:13:12.027378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:12.027378+0000","objects_scrubbed":5277,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:17.640226+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177602560,"num_objects":2710,"num_object_clones":0,"num_object_copies":8130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8484,"num_write_kb":358208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"172'8369","reported_seq":8587,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.595333+0000","last_change":"2024-04-15T18:14:03.729519+0000","last_active":"2024-04-15T18:14:04.595333+0000","last_peered":"2024-04-15T18:14:04.595333+0000","last_clean":"2024-04-15T18:14:04.595333+0000","last_became_active":"2024-04-15T18:11:33.231023+0000","last_became_peered":"2024-04-15T18:11:33.231023+0000","last_unstale":"2024-04-15T18:14:04.595333+0000","last_undegraded":"2024-04-15T18:14:04.595333+0000","last_fullsized":"2024-04-15T18:14:04.595333+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'8326","last_scrub_stamp":"2024-04-15T18:14:03.729221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:14:03.729221+0000","objects_scrubbed":2979,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:15:32.647150+0000","scrub_duration":753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8369,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"171'8370","reported_seq":8552,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-15T18:14:02.207556+0000","last_change":"2024-04-15T18:13:05.101199+0000","last_active":"2024-04-15T18:14:02.207556+0000","last_peered":"2024-04-15T18:14:02.207556+0000","last_clean":"2024-04-15T18:14:02.207556+0000","last_became_active":"2024-04-15T18:11:33.232041+0000","last_became_peered":"2024-04-15T18:11:33.232041+0000","last_unstale":"2024-04-15T18:14:02.207556+0000","last_undegraded":"2024-04-15T18:14:02.207556+0000","last_fullsized":"2024-04-15T18:14:02.207556+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4949","last_scrub_stamp":"2024-04-15T18:13:05.101059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.101059+0000","objects_scrubbed":4908,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:19.543442+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198246424,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8370,"num_write_kb":364609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"171'8197","reported_seq":8378,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-15T18:14:02.690866+0000","last_change":"2024-04-15T18:13:05.192950+0000","last_active":"2024-04-15T18:14:02.690866+0000","last_peered":"2024-04-15T18:14:02.690866+0000","last_clean":"2024-04-15T18:14:02.690866+0000","last_became_active":"2024-04-15T18:11:33.232176+0000","last_became_peered":"2024-04-15T18:11:33.232176+0000","last_unstale":"2024-04-15T18:14:02.690866+0000","last_undegraded":"2024-04-15T18:14:02.690866+0000","last_fullsized":"2024-04-15T18:14:02.690866+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4842","last_scrub_stamp":"2024-04-15T18:13:05.192815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.192815+0000","objects_scrubbed":4809,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:11.357439+0000","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":8541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":353408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2948661272,"num_objects":44994,"num_object_clones":0,"num_object_copies":134982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133228,"num_write_kb":5703041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8909959168,"data_stored":8909946952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133228,"ondisk_log_size":133228,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925882,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602488,"kb_used_data":557420,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93142664,"statfs":{"total":95995035648,"available":95378087936,"internally_reserved":0,"allocated":570798080,"data_stored":570508104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187519,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603184,"kb_used_data":558116,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93141968,"statfs":{"total":95995035648,"available":95377375232,"internally_reserved":0,"allocated":571510784,"data_stored":571224380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514564,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231400,"kb_used_data":1155804,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92513752,"statfs":{"total":95995035648,"available":94734082048,"internally_reserved":0,"allocated":1183543296,"data_stored":1183257366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874313,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776060,"kb_used_data":708784,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92969092,"statfs":{"total":95995035648,"available":95200350208,"internally_reserved":0,"allocated":725794816,"data_stored":725503816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168654,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956548,"kb_used_data":889272,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92788604,"statfs":{"total":95995035648,"available":95015530496,"internally_reserved":0,"allocated":910614528,"data_stored":910319976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462995,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":391992,"kb_used_data":364972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93353160,"statfs":{"total":95995035648,"available":95593635840,"internally_reserved":0,"allocated":373731328,"data_stored":373441372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950332,"kb_used_data":874736,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92794820,"statfs":{"total":95995035648,"available":95021895680,"internally_reserved":0,"allocated":895729664,"data_stored":895438664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962364,"kb_used_data":899248,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92782788,"statfs":{"total":95995035648,"available":95009574912,"internally_reserved":0,"allocated":920829952,"data_stored":920538952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":824176,"kb_used_data":748580,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92920976,"statfs":{"total":95995035648,"available":95151079424,"internally_reserved":0,"allocated":766545920,"data_stored":766259516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151164,"kb_used_data":1079732,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92593988,"statfs":{"total":95995035648,"available":94816243712,"internally_reserved":0,"allocated":1105645568,"data_stored":1105350496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611508,"kb_used_data":566440,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93133644,"statfs":{"total":95995035648,"available":95368851456,"internally_reserved":0,"allocated":580034560,"data_stored":579744084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399404,"kb_used_data":372388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93345748,"statfs":{"total":95995035648,"available":95586045952,"internally_reserved":0,"allocated":381325312,"data_stored":381038908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":920059904,"data_stored":920059904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":894500864,"data_stored":894500864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":762380288,"data_stored":762380288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1102450688,"data_stored":1102446616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":578359296,"data_stored":578355224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375914496,"data_stored":375914496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":366411776,"data_stored":366411776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":897449984,"data_stored":897449984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":721813504,"data_stored":721813504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1159532544,"data_stored":1159528472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":569638912,"data_stored":569638912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":561446912,"data_stored":561446912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:14:09.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:14:10.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:10 smithi049 ceph-mon[40087]: pgmap v492: 17 pgs: 17 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s 2024-04-15T18:14:10.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:10 smithi159 ceph-mon[45336]: pgmap v492: 17 pgs: 17 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s 2024-04-15T18:14:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:10 smithi082 ceph-mon[44206]: pgmap v492: 17 pgs: 17 active+clean; 2.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s 2024-04-15T18:14:10.828 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:11.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:14:12.122 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:12 smithi159 ceph-mon[45336]: pgmap v493: 17 pgs: 17 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:14:12.122 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:12.129 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:14:12.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:14:12.191 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:12.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:12.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:12.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:12.346 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:12.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:12.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:12.473 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:12 smithi082 ceph-mon[44206]: pgmap v493: 17 pgs: 17 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:14:12.474 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:12.475 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:12 smithi049 ceph-mon[40087]: pgmap v493: 17 pgs: 17 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s 2024-04-15T18:14:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:12.494 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:12.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:12.539 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:12.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:12.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:12.663 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:12.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:12.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:12.811 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:12.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:12.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:12.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:13.012 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:13.234 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:13.261 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:13.287 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:13.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:13.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: osdmap e173: 12 total, 12 up, 12 in 2024-04-15T18:14:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=4144007323,config=1981513354,health=179246142,kv=1779864009}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2765593934,logm=4046250033}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436186014}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049695012}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856976045}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61000226}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505613043}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821721428}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2318271616}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297855993}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471609071}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234399472}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648091667}) 2024-04-15T18:14:13.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229793102}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517278515}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930049728}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060950475}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700814546}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923153742}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908606418}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796315007}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527693671}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176531866}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483894473}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095398974}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260630172}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=75,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=1} crc {logm=1282351375,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=618275649}) 2024-04-15T18:14:13.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=15,mon_config_key=85} crc {mgrstat=1514251075,mon_config_key=1238432791}) 2024-04-15T18:14:13.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971493241}) 2024-04-15T18:14:13.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=3365641613,monmap=2972940104}) 2024-04-15T18:14:13.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2479268061,osd_metadata=2798619708,osd_pg_creating=2002984280,osdmap=3115828538}) 2024-04-15T18:14:13.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=282369781}) 2024-04-15T18:14:13.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:13.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: osdmap e173: 12 total, 12 up, 12 in 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=4144007323,config=1981513354,health=179246142,kv=1779864009}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2765593934,logm=4046250033}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436186014}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049695012}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856976045}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61000226}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505613043}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821721428}) 2024-04-15T18:14:13.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2318271616}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297855993}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471609071}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234399472}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648091667}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229793102}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517278515}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930049728}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060950475}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700814546}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923153742}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908606418}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796315007}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527693671}) 2024-04-15T18:14:13.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176531866}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483894473}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095398974}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260630172}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=75,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=1} crc {logm=1282351375,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=618275649}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=15,mon_config_key=85} crc {mgrstat=1514251075,mon_config_key=1238432791}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971493241}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=3365641613,monmap=2972940104}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2479268061,osd_metadata=2798619708,osd_pg_creating=2002984280,osdmap=3115828538}) 2024-04-15T18:14:13.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=282369781}) 2024-04-15T18:14:13.434 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1110431021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: osdmap e173: 12 total, 12 up, 12 in 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=4144007323,config=1981513354,health=179246142,kv=1779864009}) 2024-04-15T18:14:13.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2765593934,logm=4046250033}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436186014}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1049695012}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1856976045}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61000226}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1505613043}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821721428}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2318271616}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=297855993}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471609071}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234399472}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648091667}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4229793102}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517278515}) 2024-04-15T18:14:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2930049728}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060950475}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1700814546}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923153742}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3908606418}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=796315007}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3527693671}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=176531866}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1483894473}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095398974}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3260630172}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=75,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=1} crc {logm=1282351375,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=618275649}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=15,mon_config_key=85} crc {mgrstat=1514251075,mon_config_key=1238432791}) 2024-04-15T18:14:13.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971493241}) 2024-04-15T18:14:13.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=3365641613,monmap=2972940104}) 2024-04-15T18:14:13.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2479268061,osd_metadata=2798619708,osd_pg_creating=2002984280,osdmap=3115828538}) 2024-04-15T18:14:13.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=282369781}) 2024-04-15T18:14:13.590 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:13.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:13.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:13.731 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:14.315 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:14 smithi159 ceph-mon[45336]: pgmap v495: 17 pgs: 17 active+clean; 2.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-04-15T18:14:14.389 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:14.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:14 smithi049 ceph-mon[40087]: pgmap v495: 17 pgs: 17 active+clean; 2.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-04-15T18:14:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:14.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:14.496 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:14 smithi082 ceph-mon[44206]: pgmap v495: 17 pgs: 17 active+clean; 2.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-04-15T18:14:14.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:14.581 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:14.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:14.661 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:14:14.661 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:14:14.684 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:14.715 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:14.738 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:14.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:14.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:14.885 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:15.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:14:15.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:15.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:15.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:15.332 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:15.353 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:15.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:15.366 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:14:15.379 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:14:15.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:14:15.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:15.493 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:15.520 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:15.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:15.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:15.712 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:15.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:15.814 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:15.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:15.847 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:15.872 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:15.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:15.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:16.011 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:16.061 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:16.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:16.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:16.218 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:16.260 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-04-15T18:14:15.065717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2548499000,"num_objects":38881,"num_object_clones":0,"num_object_copies":116643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38881,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":139524,"num_write_kb":5705979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139480,"ondisk_log_size":139480,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8233692,"kb_used_data":7531228,"kb_used_omap":18,"kb_used_meta":701613,"kb_avail":1116708132,"statfs":{"total":1151940427776,"available":1143509127168,"internally_reserved":0,"allocated":7711977472,"data_stored":7708785336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":718452025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-537264128,"num_objects":-8198,"num_object_clones":0,"num_object_copies":-24594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003329"},"pg_stats":[{"pgid":"2.f","version":"172'8554","reported_seq":8738,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:10.317450+0000","last_change":"2024-04-15T18:13:07.121893+0000","last_active":"2024-04-15T18:14:10.317450+0000","last_peered":"2024-04-15T18:14:10.317450+0000","last_clean":"2024-04-15T18:14:10.317450+0000","last_became_active":"2024-04-15T18:11:33.100145+0000","last_became_peered":"2024-04-15T18:11:33.100145+0000","last_unstale":"2024-04-15T18:14:10.317450+0000","last_undegraded":"2024-04-15T18:14:10.317450+0000","last_fullsized":"2024-04-15T18:14:10.317450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4926","last_scrub_stamp":"2024-04-15T18:13:07.121782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.121782+0000","objects_scrubbed":4918,"log_size":8554,"log_dups_size":0,"ondisk_log_size":8554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:13.567121+0000","scrub_duration":1003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164757504,"num_objects":2514,"num_object_clones":0,"num_object_copies":7542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":354176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"172'8742","reported_seq":8928,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:11.701853+0000","last_change":"2024-04-15T18:13:07.069832+0000","last_active":"2024-04-15T18:14:11.701853+0000","last_peered":"2024-04-15T18:14:11.701853+0000","last_clean":"2024-04-15T18:14:11.701853+0000","last_became_active":"2024-04-15T18:11:33.230674+0000","last_became_peered":"2024-04-15T18:11:33.230674+0000","last_unstale":"2024-04-15T18:14:11.701853+0000","last_undegraded":"2024-04-15T18:14:11.701853+0000","last_fullsized":"2024-04-15T18:14:11.701853+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'5007","last_scrub_stamp":"2024-04-15T18:13:07.069711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.069711+0000","objects_scrubbed":4976,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:13.240340+0000","scrub_duration":1832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8742,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"173'8629","reported_seq":8815,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:14.339480+0000","last_change":"2024-04-15T18:13:07.360699+0000","last_active":"2024-04-15T18:14:14.339480+0000","last_peered":"2024-04-15T18:14:14.339480+0000","last_clean":"2024-04-15T18:14:14.339480+0000","last_became_active":"2024-04-15T18:11:33.099128+0000","last_became_peered":"2024-04-15T18:11:33.099128+0000","last_unstale":"2024-04-15T18:14:14.339480+0000","last_undegraded":"2024-04-15T18:14:14.339480+0000","last_fullsized":"2024-04-15T18:14:14.339480+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4836","last_scrub_stamp":"2024-04-15T18:13:07.360568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.360568+0000","objects_scrubbed":4806,"log_size":8629,"log_dups_size":0,"ondisk_log_size":8629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:30.621360+0000","scrub_duration":1308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150667264,"num_objects":2299,"num_object_clones":0,"num_object_copies":6897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8629,"num_write_kb":349696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"173'8661","reported_seq":8843,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:12.678657+0000","last_change":"2024-04-15T18:13:00.209536+0000","last_active":"2024-04-15T18:14:12.678657+0000","last_peered":"2024-04-15T18:14:12.678657+0000","last_clean":"2024-04-15T18:14:12.678657+0000","last_became_active":"2024-04-15T18:11:33.232175+0000","last_became_peered":"2024-04-15T18:11:33.232175+0000","last_unstale":"2024-04-15T18:14:12.678657+0000","last_undegraded":"2024-04-15T18:14:12.678657+0000","last_fullsized":"2024-04-15T18:14:12.678657+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4521","last_scrub_stamp":"2024-04-15T18:13:00.209243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:00.209243+0000","objects_scrubbed":4488,"log_size":8661,"log_dups_size":0,"ondisk_log_size":8661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:16.480197+0000","scrub_duration":1282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157089792,"num_objects":2397,"num_object_clones":0,"num_object_copies":7191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":353856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"173'8945","reported_seq":9129,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:14.373957+0000","last_change":"2024-04-15T18:13:04.626201+0000","last_active":"2024-04-15T18:14:14.373957+0000","last_peered":"2024-04-15T18:14:14.373957+0000","last_clean":"2024-04-15T18:14:14.373957+0000","last_became_active":"2024-04-15T18:11:33.097184+0000","last_became_peered":"2024-04-15T18:11:33.097184+0000","last_unstale":"2024-04-15T18:14:14.373957+0000","last_undegraded":"2024-04-15T18:14:14.373957+0000","last_fullsized":"2024-04-15T18:14:14.373957+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4916","last_scrub_stamp":"2024-04-15T18:13:04.626108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:04.626108+0000","objects_scrubbed":4867,"log_size":8945,"log_dups_size":0,"ondisk_log_size":8945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:17.586568+0000","scrub_duration":1619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156172288,"num_objects":2383,"num_object_clones":0,"num_object_copies":7149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8945,"num_write_kb":362496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":521,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:04.148728+0000","last_change":"2024-04-15T18:13:21.859558+0000","last_active":"2024-04-15T18:14:04.148728+0000","last_peered":"2024-04-15T18:14:04.148728+0000","last_clean":"2024-04-15T18:14:04.148728+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:14:04.148728+0000","last_undegraded":"2024-04-15T18:14:04.148728+0000","last_fullsized":"2024-04-15T18:14:04.148728+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:13:21.859468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:13:21.859468+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:49.623977+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"172'8516","reported_seq":8700,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:10.963662+0000","last_change":"2024-04-15T18:13:07.614724+0000","last_active":"2024-04-15T18:14:10.963662+0000","last_peered":"2024-04-15T18:14:10.963662+0000","last_clean":"2024-04-15T18:14:10.963662+0000","last_became_active":"2024-04-15T18:11:33.230198+0000","last_became_peered":"2024-04-15T18:11:33.230198+0000","last_unstale":"2024-04-15T18:14:10.963662+0000","last_undegraded":"2024-04-15T18:14:10.963662+0000","last_fullsized":"2024-04-15T18:14:10.963662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4935","last_scrub_stamp":"2024-04-15T18:13:07.614548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:07.614548+0000","objects_scrubbed":4911,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:22.135685+0000","scrub_duration":706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160038912,"num_objects":2442,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":350656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"172'8524","reported_seq":8709,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:10.280298+0000","last_change":"2024-04-15T18:13:11.048746+0000","last_active":"2024-04-15T18:14:10.280298+0000","last_peered":"2024-04-15T18:14:10.280298+0000","last_clean":"2024-04-15T18:14:10.280298+0000","last_became_active":"2024-04-15T18:11:33.230675+0000","last_became_peered":"2024-04-15T18:11:33.230675+0000","last_unstale":"2024-04-15T18:14:10.280298+0000","last_undegraded":"2024-04-15T18:14:10.280298+0000","last_fullsized":"2024-04-15T18:14:10.280298+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'5138","last_scrub_stamp":"2024-04-15T18:13:11.048581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:11.048581+0000","objects_scrubbed":5103,"log_size":8524,"log_dups_size":0,"ondisk_log_size":8524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:35.701816+0000","scrub_duration":989,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162136064,"num_objects":2474,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"172'8726","reported_seq":8914,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:11.722515+0000","last_change":"2024-04-15T18:13:11.928691+0000","last_active":"2024-04-15T18:14:11.722515+0000","last_peered":"2024-04-15T18:14:11.722515+0000","last_clean":"2024-04-15T18:14:11.722515+0000","last_became_active":"2024-04-15T18:11:33.233286+0000","last_became_peered":"2024-04-15T18:11:33.233286+0000","last_unstale":"2024-04-15T18:14:11.722515+0000","last_undegraded":"2024-04-15T18:14:11.722515+0000","last_fullsized":"2024-04-15T18:14:11.722515+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'5250","last_scrub_stamp":"2024-04-15T18:13:11.928500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:11.928500+0000","objects_scrubbed":5200,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:14.574074+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159776768,"num_objects":2438,"num_object_clones":0,"num_object_copies":7314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8726,"num_write_kb":357248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"173'8793","reported_seq":8975,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:14.888713+0000","last_change":"2024-04-15T18:13:03.394418+0000","last_active":"2024-04-15T18:14:14.888713+0000","last_peered":"2024-04-15T18:14:14.888713+0000","last_clean":"2024-04-15T18:14:14.888713+0000","last_became_active":"2024-04-15T18:11:33.230359+0000","last_became_peered":"2024-04-15T18:11:33.230359+0000","last_unstale":"2024-04-15T18:14:14.888713+0000","last_undegraded":"2024-04-15T18:14:14.888713+0000","last_fullsized":"2024-04-15T18:14:14.888713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4725","last_scrub_stamp":"2024-04-15T18:13:03.394223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:03.394223+0000","objects_scrubbed":4680,"log_size":8793,"log_dups_size":0,"ondisk_log_size":8793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1362,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798336,"num_objects":2301,"num_object_clones":0,"num_object_copies":6903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":355008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"172'8611","reported_seq":8793,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:10.954049+0000","last_change":"2024-04-15T18:13:04.996518+0000","last_active":"2024-04-15T18:14:10.954049+0000","last_peered":"2024-04-15T18:14:10.954049+0000","last_clean":"2024-04-15T18:14:10.954049+0000","last_became_active":"2024-04-15T18:11:33.230216+0000","last_became_peered":"2024-04-15T18:11:33.230216+0000","last_unstale":"2024-04-15T18:14:10.954049+0000","last_undegraded":"2024-04-15T18:14:10.954049+0000","last_fullsized":"2024-04-15T18:14:10.954049+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'4825","last_scrub_stamp":"2024-04-15T18:13:04.996348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:04.996348+0000","objects_scrubbed":4784,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:19.394686+0000","scrub_duration":1113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":7599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8611,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"172'8546","reported_seq":8729,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:11.668421+0000","last_change":"2024-04-15T18:13:06.364251+0000","last_active":"2024-04-15T18:14:11.668421+0000","last_peered":"2024-04-15T18:14:11.668421+0000","last_clean":"2024-04-15T18:14:11.668421+0000","last_became_active":"2024-04-15T18:11:33.232432+0000","last_became_peered":"2024-04-15T18:11:33.232432+0000","last_unstale":"2024-04-15T18:14:11.668421+0000","last_undegraded":"2024-04-15T18:14:11.668421+0000","last_fullsized":"2024-04-15T18:14:11.668421+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4885","last_scrub_stamp":"2024-04-15T18:13:06.364130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:06.364130+0000","objects_scrubbed":4850,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:16.666419+0000","scrub_duration":2125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159645696,"num_objects":2436,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"173'8942","reported_seq":9129,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:14.579867+0000","last_change":"2024-04-15T18:13:05.338257+0000","last_active":"2024-04-15T18:14:14.579867+0000","last_peered":"2024-04-15T18:14:14.579867+0000","last_clean":"2024-04-15T18:14:14.579867+0000","last_became_active":"2024-04-15T18:11:33.232105+0000","last_became_peered":"2024-04-15T18:11:33.232105+0000","last_unstale":"2024-04-15T18:14:14.579867+0000","last_undegraded":"2024-04-15T18:14:14.579867+0000","last_fullsized":"2024-04-15T18:14:14.579867+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4936","last_scrub_stamp":"2024-04-15T18:13:05.338106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.338106+0000","objects_scrubbed":4907,"log_size":8942,"log_dups_size":0,"ondisk_log_size":8942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:34.463184+0000","scrub_duration":1140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156237824,"num_objects":2384,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":362432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"172'8722","reported_seq":8910,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-04-15T18:14:10.245322+0000","last_change":"2024-04-15T18:13:12.027697+0000","last_active":"2024-04-15T18:14:10.245322+0000","last_peered":"2024-04-15T18:14:10.245322+0000","last_clean":"2024-04-15T18:14:10.245322+0000","last_became_active":"2024-04-15T18:11:33.099322+0000","last_became_peered":"2024-04-15T18:11:33.099322+0000","last_unstale":"2024-04-15T18:14:10.245322+0000","last_undegraded":"2024-04-15T18:14:10.245322+0000","last_fullsized":"2024-04-15T18:14:10.245322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'5299","last_scrub_stamp":"2024-04-15T18:13:12.027378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:12.027378+0000","objects_scrubbed":5277,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:17.640226+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":358208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"173'8887","reported_seq":9107,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:14.598005+0000","last_change":"2024-04-15T18:14:03.729519+0000","last_active":"2024-04-15T18:14:14.598005+0000","last_peered":"2024-04-15T18:14:14.598005+0000","last_clean":"2024-04-15T18:14:14.598005+0000","last_became_active":"2024-04-15T18:11:33.231023+0000","last_became_peered":"2024-04-15T18:11:33.231023+0000","last_unstale":"2024-04-15T18:14:14.598005+0000","last_undegraded":"2024-04-15T18:14:14.598005+0000","last_fullsized":"2024-04-15T18:14:14.598005+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'8326","last_scrub_stamp":"2024-04-15T18:14:03.729221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:14:03.729221+0000","objects_scrubbed":2979,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:15:32.647150+0000","scrub_duration":753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157483008,"num_objects":2403,"num_object_clones":0,"num_object_copies":7209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8887,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"173'8862","reported_seq":9049,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:12.263454+0000","last_change":"2024-04-15T18:13:05.101199+0000","last_active":"2024-04-15T18:14:12.263454+0000","last_peered":"2024-04-15T18:14:12.263454+0000","last_clean":"2024-04-15T18:14:12.263454+0000","last_became_active":"2024-04-15T18:11:33.232041+0000","last_became_peered":"2024-04-15T18:11:33.232041+0000","last_unstale":"2024-04-15T18:14:12.263454+0000","last_undegraded":"2024-04-15T18:14:12.263454+0000","last_fullsized":"2024-04-15T18:14:12.263454+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4949","last_scrub_stamp":"2024-04-15T18:13:05.101059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.101059+0000","objects_scrubbed":4908,"log_size":8862,"log_dups_size":0,"ondisk_log_size":8862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:19.543442+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002712,"num_objects":2534,"num_object_clones":0,"num_object_copies":7602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8862,"num_write_kb":364609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"173'8683","reported_seq":8868,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-04-15T18:14:12.683063+0000","last_change":"2024-04-15T18:13:05.192950+0000","last_active":"2024-04-15T18:14:12.683063+0000","last_peered":"2024-04-15T18:14:12.683063+0000","last_clean":"2024-04-15T18:14:12.683063+0000","last_became_active":"2024-04-15T18:11:33.232176+0000","last_became_peered":"2024-04-15T18:11:33.232176+0000","last_unstale":"2024-04-15T18:14:12.683063+0000","last_undegraded":"2024-04-15T18:14:12.683063+0000","last_fullsized":"2024-04-15T18:14:12.683063+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'4842","last_scrub_stamp":"2024-04-15T18:13:05.192815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:11:32.086185+0000","last_clean_scrub_stamp":"2024-04-15T18:13:05.192815+0000","objects_scrubbed":4809,"log_size":8683,"log_dups_size":0,"ondisk_log_size":8683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:14:16.480236+0000","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154730496,"num_objects":2361,"num_object_clones":0,"num_object_copies":7083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8683,"num_write_kb":353408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2547908632,"num_objects":38879,"num_object_clones":0,"num_object_copies":116637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139343,"num_write_kb":5703041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7629123584,"data_stored":7629111368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139343,"ondisk_log_size":139343,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511688,"kb_used_data":458304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93233464,"statfs":{"total":95995035648,"available":95471067136,"internally_reserved":0,"allocated":469303296,"data_stored":469054832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":54655428},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187521,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":517320,"kb_used_data":463800,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93227832,"statfs":{"total":95995035648,"available":95465299968,"internally_reserved":0,"allocated":474931200,"data_stored":474686308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514566,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1028412,"kb_used_data":956980,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92716740,"statfs":{"total":95995035648,"available":94941941760,"internally_reserved":0,"allocated":979947520,"data_stored":979652448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874314,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714296,"kb_used_data":642864,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93030856,"statfs":{"total":95995035648,"available":95263596544,"internally_reserved":0,"allocated":658292736,"data_stored":658001736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882112,"kb_used_data":810616,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92863040,"statfs":{"total":95995035648,"available":95091752960,"internally_reserved":0,"allocated":830070784,"data_stored":829776232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462997,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325576,"kb_used_data":294400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93419576,"statfs":{"total":95995035648,"available":95661645824,"internally_reserved":0,"allocated":301465600,"data_stored":301217156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853816,"kb_used_data":790704,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92891336,"statfs":{"total":95995035648,"available":95120728064,"internally_reserved":0,"allocated":809680896,"data_stored":809389896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900600,"kb_used_data":833264,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92844552,"statfs":{"total":95995035648,"available":95072821248,"internally_reserved":0,"allocated":853262336,"data_stored":852971336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":688648,"kb_used_data":621240,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93056504,"statfs":{"total":95995035648,"available":95289860096,"internally_reserved":0,"allocated":636149760,"data_stored":635904868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954628,"kb_used_data":887356,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92790524,"statfs":{"total":95995035648,"available":95017496576,"internally_reserved":0,"allocated":908652544,"data_stored":908403580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520076,"kb_used_data":466492,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93225076,"statfs":{"total":95995035648,"available":95462477824,"internally_reserved":0,"allocated":477687808,"data_stored":477438844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336520,"kb_used_data":305208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93408632,"statfs":{"total":95995035648,"available":95650439168,"internally_reserved":0,"allocated":312532992,"data_stored":312288100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":832438272,"data_stored":832438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":807927808,"data_stored":807927808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":632553472,"data_stored":632553472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":907743232,"data_stored":907739160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":476516352,"data_stored":476512280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":306839552,"data_stored":306839552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":300089344,"data_stored":300089344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":816185344,"data_stored":816185344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":653066240,"data_stored":653066240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":960368640,"data_stored":960364568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":472383488,"data_stored":472383488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":463011840,"data_stored":463011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:14:16.261 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:14:16.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:16 smithi159 ceph-mon[45336]: pgmap v496: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:14:16.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:16 smithi159 ceph-mon[45336]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:16.355 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:16.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:16.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:16 smithi049 ceph-mon[40087]: pgmap v496: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:14:16.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:16 smithi049 ceph-mon[40087]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:16.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:16.529 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:16 smithi082 ceph-mon[44206]: pgmap v496: 17 pgs: 17 active+clean; 2.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:14:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:16 smithi082 ceph-mon[44206]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:14:16.686 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:16.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:16.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:16.791 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:16.819 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:16.843 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:16.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:16.918 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:16.946 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:16.950 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:16.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:16.985 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:17.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:14:17.052 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:17.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:17.141 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:17.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:17.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:17.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:17.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:17.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:17.291 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:17.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:17.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:17.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/281087597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:17.340 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:17.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/281087597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:17.480 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:17.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:17.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/281087597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:17.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:17.614 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:17.634 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:17.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:17.740 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:17.768 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:17.789 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:17.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:143571 2.d 2:bd723247:::benchmark_data_smithi159_80260_object54458:head [delete] snapc 0=[] ondisk+write+known_if_redirected e174)", 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:17.795471+0000", 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0025409080000000001, 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025734659999999999, 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:14:17.801 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 143571 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795471+0000", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795471+0000", 2024-04-15T18:14:17.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795472+0000", 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.406e-06 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795478+0000", 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4280000000000001e-06 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:17.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795493+0000", 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4718e-05 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795501+0000", 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8879999999999997e-06 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795559+0000", 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7544000000000003e-05 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795681+0000", 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000122767 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:17.795791+0000", 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109683 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 1 2024-04-15T18:14:17.805 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:17.894 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:17.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.036 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:18.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:18.169 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.191 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:18.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:18.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.344 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:18.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: pgmap v497: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osdmap e174: 12 total, 12 up, 12 in 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: pgmap v497: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osdmap e174: 12 total, 12 up, 12 in 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.448 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:18.479 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: pgmap v497: 17 pgs: 17 active+clean; 2.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osdmap e174: 12 total, 12 up, 12 in 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.480 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.481 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.481 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:18.481 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.508 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:18.612 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:18.641 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.662 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:18.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:14:18.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:18.815 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.839 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:18.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:18.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:18.975 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:18.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:19.042 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:19.101 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:19.130 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:19.150 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:19.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:19.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:19.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:19.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:19.446 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:19.525 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:19.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:19.690 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:19.723 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:19.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:19.871 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:19.909 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:19.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:20.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:20.045 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:20.069 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:20.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:20.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:20.177 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:14:20.209 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:20 smithi159 ceph-mon[45336]: pgmap v499: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:14:20.209 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:20.214 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:20.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:14:20.413 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:20.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:20.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:20 smithi049 ceph-mon[40087]: pgmap v499: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:14:20.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:20.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:20.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:20 smithi082 ceph-mon[44206]: pgmap v499: 17 pgs: 17 active+clean; 2.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:14:20.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:20.561 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:20.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:20.996 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:21.031 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:21.071 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:21.081 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:21.081 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:14:21.081 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:14:21.082 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:14:21.082 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:21.099 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:21.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:21.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:21 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:21.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:21.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:21 smithi049 ceph-mon[40087]: osdmap e175: 12 total, 12 up, 12 in 2024-04-15T18:14:21.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:21 smithi049 ceph-mon[40087]: 2.d scrub starts 2024-04-15T18:14:21.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:21.239 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:21.308 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:21.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:21.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:21 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:21.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:21 smithi159 ceph-mon[45336]: osdmap e175: 12 total, 12 up, 12 in 2024-04-15T18:14:21.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:21 smithi159 ceph-mon[45336]: 2.d scrub starts 2024-04-15T18:14:21.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:21.455 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:21.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:21 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:21.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3947758424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:21.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:21 smithi082 ceph-mon[44206]: osdmap e175: 12 total, 12 up, 12 in 2024-04-15T18:14:21.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:21 smithi082 ceph-mon[44206]: 2.d scrub starts 2024-04-15T18:14:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:21.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:21.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:21.707 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:21.913 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:21.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:21.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:14:22.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:22.054 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:22.081 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:22.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:22.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:22.227 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:22.233 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:22 smithi159 ceph-mon[45336]: pgmap v501: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-04-15T18:14:22.234 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:22 smithi159 ceph-mon[45336]: 2.f scrub starts 2024-04-15T18:14:22.234 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:22 smithi159 ceph-mon[45336]: 2.d scrub ok 2024-04-15T18:14:22.263 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:22.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:22.367 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:22.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:22.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:22 smithi049 ceph-mon[40087]: pgmap v501: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-04-15T18:14:22.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:22 smithi049 ceph-mon[40087]: 2.f scrub starts 2024-04-15T18:14:22.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:22 smithi049 ceph-mon[40087]: 2.d scrub ok 2024-04-15T18:14:22.423 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:22 smithi082 ceph-mon[44206]: pgmap v501: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-04-15T18:14:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:22 smithi082 ceph-mon[44206]: 2.f scrub starts 2024-04-15T18:14:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:22 smithi082 ceph-mon[44206]: 2.d scrub ok 2024-04-15T18:14:22.683 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:22.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:22.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:22.830 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:22.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:14:23.283 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:23.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:23.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:23.421 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:23.443 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:23.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:23 smithi082 ceph-mon[44206]: 2.f scrub ok 2024-04-15T18:14:23.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:23 smithi082 ceph-mon[44206]: 2.0 scrub starts 2024-04-15T18:14:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:23.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:23.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:23 smithi049 ceph-mon[40087]: 2.f scrub ok 2024-04-15T18:14:23.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:23 smithi049 ceph-mon[40087]: 2.0 scrub starts 2024-04-15T18:14:23.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:23.551 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:23.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:23.583 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:23 smithi159 ceph-mon[45336]: 2.f scrub ok 2024-04-15T18:14:23.583 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:23 smithi159 ceph-mon[45336]: 2.0 scrub starts 2024-04-15T18:14:23.585 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:23.618 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:23.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:23.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:14:23.726 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:23.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:23.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:23.856 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:23.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:23.963 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:23.998 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:24.037 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:24.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:24.120 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:24.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:24.200 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:24.221 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:24.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:24.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:24.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:24.252 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:24.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:24.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:24.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: 2.9 scrub starts 2024-04-15T18:14:24.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: 2.0 scrub ok 2024-04-15T18:14:24.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: pgmap v502: 17 pgs: 17 active+clean; 2.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s 2024-04-15T18:14:24.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: 2.b scrub starts 2024-04-15T18:14:24.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1865838189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:24.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.396 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:24.528 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:14:24.528 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:14:24.530 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:24.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: 2.9 scrub starts 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: 2.0 scrub ok 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: pgmap v502: 17 pgs: 17 active+clean; 2.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: 2.b scrub starts 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1865838189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.550 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: 2.9 scrub starts 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: 2.0 scrub ok 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: pgmap v502: 17 pgs: 17 active+clean; 2.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: 2.b scrub starts 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1865838189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.551 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:24.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:24.682 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:24.834 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:24.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:24.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:24.986 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:25.412 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: 2.9 scrub ok 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: 2.b scrub ok 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: 2.7 scrub starts 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:25 smithi049 ceph-mon[40087]: 2.7 scrub ok 2024-04-15T18:14:25.441 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:25.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: 2.9 scrub ok 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: 2.b scrub ok 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: 2.7 scrub starts 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:25 smithi082 ceph-mon[44206]: 2.7 scrub ok 2024-04-15T18:14:25.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: 2.9 scrub ok 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: 2.b scrub ok 2024-04-15T18:14:25.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: 2.7 scrub starts 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:25.661 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:25 smithi159 ceph-mon[45336]: 2.7 scrub ok 2024-04-15T18:14:25.729 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:25.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:25.782 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:25.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:25.873 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:26.083 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:26.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:26.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:26.235 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:26.451 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:26.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:26 smithi049 ceph-mon[40087]: 2.4 scrub starts 2024-04-15T18:14:26.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:26 smithi049 ceph-mon[40087]: pgmap v503: 17 pgs: 17 active+clean; 1.9 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:14:26.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:26 smithi049 ceph-mon[40087]: 2.8 scrub starts 2024-04-15T18:14:26.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:26 smithi049 ceph-mon[40087]: 2.4 scrub ok 2024-04-15T18:14:26.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:26 smithi049 ceph-mon[40087]: 2.1 scrub starts 2024-04-15T18:14:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:26.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:26.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:26 smithi082 ceph-mon[44206]: 2.4 scrub starts 2024-04-15T18:14:26.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:26 smithi082 ceph-mon[44206]: pgmap v503: 17 pgs: 17 active+clean; 1.9 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:14:26.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:26 smithi082 ceph-mon[44206]: 2.8 scrub starts 2024-04-15T18:14:26.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:26 smithi082 ceph-mon[44206]: 2.4 scrub ok 2024-04-15T18:14:26.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:26 smithi082 ceph-mon[44206]: 2.1 scrub starts 2024-04-15T18:14:26.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:26.622 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:26 smithi159 ceph-mon[45336]: 2.4 scrub starts 2024-04-15T18:14:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:26 smithi159 ceph-mon[45336]: pgmap v503: 17 pgs: 17 active+clean; 1.9 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:14:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:26 smithi159 ceph-mon[45336]: 2.8 scrub starts 2024-04-15T18:14:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:26 smithi159 ceph-mon[45336]: 2.4 scrub ok 2024-04-15T18:14:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:26 smithi159 ceph-mon[45336]: 2.1 scrub starts 2024-04-15T18:14:26.661 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:26.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:26.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:26.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:27.241 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:14:27.258 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:27.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:27.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:27.395 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:27.415 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:27.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: 2.8 scrub ok 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: 2.1 scrub ok 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:27 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.522 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:27.533 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: 2.8 scrub ok 2024-04-15T18:14:27.533 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: 2.1 scrub ok 2024-04-15T18:14:27.533 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:14:27.534 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:27.534 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.534 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:27 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: 2.8 scrub ok 2024-04-15T18:14:27.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: 2.1 scrub ok 2024-04-15T18:14:27.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:14:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:27 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:27.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:27.588 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:27.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:27.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:27.755 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:27.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:14:27.845 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:14:27.845 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:14:27.989 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:27.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:28.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:28.145 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:28.191 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:28.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:28.248 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:28.298 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:28.345 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: pgmap v504: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: osdmap e176: 12 total, 12 up, 12 in 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: 2.3 scrub starts 2024-04-15T18:14:28.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: 2.3 scrub ok 2024-04-15T18:14:28.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:14:28.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:14:28.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: pgmap v504: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: osdmap e176: 12 total, 12 up, 12 in 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: 2.3 scrub starts 2024-04-15T18:14:28.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: 2.3 scrub ok 2024-04-15T18:14:28.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:14:28.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:14:28.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: pgmap v504: 17 pgs: 17 active+clean; 1.8 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-04-15T18:14:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:14:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:14:28.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1901991733' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: osdmap e176: 12 total, 12 up, 12 in 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: 2.3 scrub starts 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: 2.3 scrub ok 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:14:28.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:14:28.703 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:28.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:14:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:28.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:28.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:28.856 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:29.049 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:29 smithi082 ceph-mon[44206]: 2.6 deep-scrub starts 2024-04-15T18:14:29.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:29 smithi082 ceph-mon[44206]: 2.e scrub starts 2024-04-15T18:14:29.584 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:29 smithi159 ceph-mon[45336]: 2.6 deep-scrub starts 2024-04-15T18:14:29.585 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:29 smithi159 ceph-mon[45336]: 2.e scrub starts 2024-04-15T18:14:29.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:29 smithi049 ceph-mon[40087]: 2.6 deep-scrub starts 2024-04-15T18:14:29.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:29 smithi049 ceph-mon[40087]: 2.e scrub starts 2024-04-15T18:14:29.736 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:29.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:29.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:29.899 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:30.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:14:30.021 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:30.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:30.126 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:30.153 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:30.172 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:30.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:30.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:30.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:14:30.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:30 smithi049 ceph-mon[40087]: pgmap v506: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:14:30.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:30 smithi049 ceph-mon[40087]: 2.e scrub ok 2024-04-15T18:14:30.346 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:30.397 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:14:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:30.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:14:30.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:30.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:30.503 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:30.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:30 smithi082 ceph-mon[44206]: pgmap v506: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:14:30.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:30 smithi082 ceph-mon[44206]: 2.e scrub ok 2024-04-15T18:14:30.558 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:30.562 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:30.608 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:14:30.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:30.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:30 smithi159 ceph-mon[45336]: pgmap v506: 17 pgs: 17 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:14:30.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:30 smithi159 ceph-mon[45336]: 2.e scrub ok 2024-04-15T18:14:30.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:30.731 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:30.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:30.808 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:30.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:30.870 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:30.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:30.890 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:30.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:30.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:30.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:30.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:30.980 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:31.008 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:31.030 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:31.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:31.123 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:14:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:14:31.123 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:14:31.123 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:31.134 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:31.161 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:31.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:31.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:31.321 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:31.365 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:31.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:31.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2629810606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:31.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:31.497 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:31.518 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:31.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2629810606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.622 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:31.659 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:31.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2629810606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:31.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:31.979 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:14:32.018 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:32.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:32.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:32.152 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:32.172 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:32.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:32.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:32.304 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:32.323 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:32.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:32.378 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:32.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:32.495 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:32 smithi082 ceph-mon[44206]: pgmap v507: 17 pgs: 17 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:32 smithi159 ceph-mon[45336]: pgmap v507: 17 pgs: 17 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:32.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:32 smithi049 ceph-mon[40087]: pgmap v507: 17 pgs: 17 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:14:34.392 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:34.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:34.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:34 smithi159 ceph-mon[45336]: pgmap v508: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:14:34.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:34.412 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:4214 2.4 e124/114 2:28ffab64:::benchmark_data_smithi159_80260_object4213:head v 124'280, mlcod=124'277)", 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:49.643980+0000", 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 164.79752598100001, 2024-04-15T18:14:34.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.15182206200000001, 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.643980+0000", 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.643980+0000", 2024-04-15T18:14:34.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.643981+0000", 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.644061+0000", 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9733000000000004e-05 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795216+0000", 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.151155228 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795229+0000", 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2775999999999999e-05 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795328+0000", 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9772999999999995e-05 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795351+0000", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2968000000000001e-05 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795764+0000", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00041306900000000001 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795802+0000", 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7793000000000002e-05 2024-04-15T18:14:34.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:4241 2.4 e124/114 2:2c03f583:::benchmark_data_smithi159_80260_object4240:head v 124'281, mlcod=124'278)", 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:49.795121+0000", 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 164.64638536699999, 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.128088166, 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795121+0000", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795121+0000", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795129+0000", 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4500000000000004e-06 2024-04-15T18:14:34.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.795180+0000", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0748999999999998e-05 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.922674+0000", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.127494044 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.922680+0000", 2024-04-15T18:14:34.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2079999999999997e-06 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.922741+0000", 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1315999999999994e-05 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.922756+0000", 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5012e-05 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.923180+0000", 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042354099999999999 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.923209+0000", 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8846000000000001e-05 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:7218 2.4 e125/114 2:28eca49e:::benchmark_data_smithi159_80260_object7217:head v 125'474, mlcod=125'471)", 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:52.390115+0000", 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 162.05139142799999, 2024-04-15T18:14:34.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.106219086, 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.390115+0000", 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.390115+0000", 2024-04-15T18:14:34.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.390115+0000", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7899999999999998e-07 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.390144+0000", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8731999999999999e-05 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.495634+0000", 2024-04-15T18:14:34.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10549030199999999 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.495641+0000", 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0729999999999999e-06 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.495726+0000", 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4870000000000006e-05 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.495750+0000", 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3906000000000001e-05 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.496286+0000", 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00053584100000000001 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:52.496334+0000", 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7682999999999997e-05 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.460 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:36243 2.4 e134/114 2:2119bcbe:::benchmark_data_smithi159_80260_object36242:head v 134'2279, mlcod=134'2275)", 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:13.869019+0000", 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 140.572487153, 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.088684472, 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.869019+0000", 2024-04-15T18:14:34.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.869019+0000", 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.869020+0000", 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.869050+0000", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0049e-05 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.956669+0000", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.087618791000000001 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.956672+0000", 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.647e-06 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.956721+0000", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8471999999999999e-05 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.956745+0000", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4012999999999999e-05 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.957660+0000", 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091562700000000002 2024-04-15T18:14:34.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.957703+0000", 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3089e-05 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:36256 2.4 e134/114 2:2cad7e7a:::benchmark_data_smithi159_80260_object36255:head v 134'2280, mlcod=134'2276)", 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:13.894389+0000", 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 140.547117031, 2024-04-15T18:14:34.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10146965300000001, 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.894389+0000", 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.894389+0000", 2024-04-15T18:14:34.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.894390+0000", 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.029e-06 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.894424+0000", 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3698999999999997e-05 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.994767+0000", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10034354299999999 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.994773+0000", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3489999999999999e-06 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.994823+0000", 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0170000000000002e-05 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.994845+0000", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2668e-05 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.995822+0000", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00097602400000000003 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:13.995859+0000", 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7171e-05 2024-04-15T18:14:34.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:60262 2.4 e143/114 2:246e6008:::benchmark_data_smithi159_80260_object60261:head v 143'3801, mlcod=143'3798)", 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:33.789749+0000", 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 120.651757414, 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.169075897, 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.789749+0000", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.789749+0000", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.789750+0000", 2024-04-15T18:14:34.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.192e-06 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.789800+0000", 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0599999999999997e-05 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.957416+0000", 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16761559100000001 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.957421+0000", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6469999999999997e-06 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.957464+0000", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3319999999999999e-05 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.957478+0000", 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3886999999999999e-05 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.958782+0000", 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001304567 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:33.958825+0000", 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2092999999999998e-05 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:63233 2.7 e149/114 2:e318b3ee:::benchmark_data_smithi159_80260_object63232:head v 149'3919, mlcod=149'3914)", 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:51.436539+0000", 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.004967365, 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.093202333999999998, 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.436539+0000", 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.436539+0000", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.436539+0000", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.436614+0000", 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4350000000000005e-05 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.528153+0000", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.091538921999999995 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.528160+0000", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.528206+0000", 2024-04-15T18:14:34.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6397999999999999e-05 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.528224+0000", 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7218e-05 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.529712+0000", 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00148847 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:51.529741+0000", 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8920000000000001e-05 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:70721 2.7 e151/114 2:eb221dcb:::benchmark_data_smithi159_80260_object70720:head v 151'4393, mlcod=151'4390)", 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.091007+0000", 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 96.350499064000005, 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.122937791, 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.479 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.091007+0000", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.091007+0000", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.091008+0000", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0999999999999998e-07 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.091103+0000", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5210999999999994e-05 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.212289+0000", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.12118625399999999 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.212296+0000", 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9160000000000004e-06 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.212342+0000", 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6167e-05 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.212367+0000", 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4606000000000001e-05 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.213911+0000", 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015441739999999999 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.213945+0000", 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3853000000000003e-05 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:72950 2.0 e152/114 2:048e7642:::benchmark_data_smithi159_80260_object72949:head v 152'4556, mlcod=152'4554)", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:00.313069+0000", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.128437414999993, 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.177104329, 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.313069+0000", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.313069+0000", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.313071+0000", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.979e-06 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.313122+0000", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1856e-05 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.487831+0000", 2024-04-15T18:14:34.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17470835100000001 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.487839+0000", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7200000000000006e-06 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.487898+0000", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9787999999999997e-05 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.487922+0000", 2024-04-15T18:14:34.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3555000000000001e-05 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.490134+0000", 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022124610000000002 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:00.490173+0000", 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8618999999999997e-05 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:76761 2.4 e153/114 2:28ffbad1:::benchmark_data_smithi159_80260_object76760:head v 153'4847, mlcod=153'4844)", 2024-04-15T18:14:34.487 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:04.010134+0000", 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 90.431372437999997, 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18630422999999999, 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.010134+0000", 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.010134+0000", 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.010135+0000", 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.579e-06 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.010174+0000", 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9078000000000001e-05 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.194558+0000", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18438365600000001 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.194564+0000", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3380000000000001e-06 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.194614+0000", 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9381e-05 2024-04-15T18:14:34.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.194637+0000", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3337e-05 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.196401+0000", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001764238 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.196438+0000", 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6622999999999998e-05 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.491 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:77348 2.0 e153/114 2:0eea22b8:::benchmark_data_smithi159_80260_object77347:head v 153'4800, mlcod=153'4793)", 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:04.772272+0000", 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.669233610999996, 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10256072500000001, 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.772272+0000", 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.772272+0000", 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.772273+0000", 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.13e-07 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.772372+0000", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8504000000000004e-05 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.872501+0000", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.100129631 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.872509+0000", 2024-04-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8290000000000008e-06 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.872599+0000", 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9930000000000001e-05 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.872621+0000", 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1702999999999999e-05 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.874767+0000", 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002146345 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:04.874833+0000", 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6169999999999998e-05 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.496 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:97718 2.0 e159/114 2:0c7a2dd6:::benchmark_data_smithi159_80260_object8605:head v 159'6021, mlcod=159'6018)", 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.473954+0000", 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.967551924999995, 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.110912492, 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.473954+0000", 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.473954+0000", 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.473955+0000", 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9299999999999997e-07 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.473962+0000", 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2320000000000004e-06 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582686+0000", 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10872364399999999 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582694+0000", 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9980000000000003e-06 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582752+0000", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7859000000000003e-05 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582773+0000", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1557999999999999e-05 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.584837+0000", 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020642360000000001 2024-04-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.584867+0000", 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9272e-05 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:97756 2.0 e159/114 2:03c6d7c8:::benchmark_data_smithi159_80260_object8643:head v 159'6022, mlcod=159'6019)", 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.582657+0000", 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.858848683000005, 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10342254100000001, 2024-04-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582657+0000", 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582657+0000", 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582658+0000", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999998e-07 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.582666+0000", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0469999999999994e-06 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.683803+0000", 2024-04-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.10113711 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.683813+0000", 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3440000000000007e-06 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.683866+0000", 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3189999999999997e-05 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.683883+0000", 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7278999999999998e-05 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.686047+0000", 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021638859999999998 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.686080+0000", 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2864999999999998e-05 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:97812 2.0 e159/114 2:00af3f0c:::benchmark_data_smithi159_80260_object8699:head v 159'6026, mlcod=159'6023)", 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.714030+0000", 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.727476514000003, 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.098787844999999999, 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.714030+0000", 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.714030+0000", 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.714030+0000", 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6099999999999999e-07 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.714036+0000", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3000000000000001e-06 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.810430+0000", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.096394194000000002 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.810438+0000", 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4270000000000008e-06 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.810516+0000", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7411999999999997e-05 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.810532+0000", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6716999999999999e-05 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.812779+0000", 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022466640000000002 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.812817+0000", 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8269999999999998e-05 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:128311 2.7 e169/114 2:ef66df44:::benchmark_data_smithi159_80260_object39198:head v 169'8003, mlcod=169'8001)", 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.945827+0000", 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.495678794, 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.186620183, 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.510 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.945827+0000", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.945827+0000", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.945828+0000", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6899999999999996e-07 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.945835+0000", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8040000000000004e-06 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.128277+0000", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18244153299999999 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.128291+0000", 2024-04-15T18:14:34.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4569000000000001e-05 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.128344+0000", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2778000000000002e-05 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.128370+0000", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5605999999999998e-05 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.132417+0000", 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040478839999999999 2024-04-15T18:14:34.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:59.132447+0000", 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.004e-05 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:129561 2.4 e169/114 2:23cc4558:::benchmark_data_smithi159_80260_object40448:head v 169'8169, mlcod=169'8166)", 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:00.545725+0000", 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.895781223999997, 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.095028293999999999, 2024-04-15T18:14:34.514 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.545725+0000", 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.545725+0000", 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.545726+0000", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2049999999999999e-06 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.545735+0000", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6619999999999994e-06 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.637206+0000", 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.091471781000000002 2024-04-15T18:14:34.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.637214+0000", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4660000000000002e-06 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.637268+0000", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3742999999999997e-05 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.637290+0000", 2024-04-15T18:14:34.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1993000000000001e-05 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.640724+0000", 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034341250000000001 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:00.640753+0000", 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9319000000000002e-05 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:131614 2.0 e171/114 2:0f0136ab:::benchmark_data_smithi159_80260_object42501:head v 171'8132, mlcod=171'8128)", 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:02.841865+0000", 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 31.599640692000001, 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.099267682999999995, 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.841865+0000", 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.841865+0000", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.841866+0000", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9500000000000001e-07 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.841872+0000", 2024-04-15T18:14:34.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8150000000000002e-06 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.937181+0000", 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.095309185000000005 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.937187+0000", 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7799999999999997e-06 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.937241+0000", 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3773999999999997e-05 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.937265+0000", 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.429e-05 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.941085+0000", 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038200069999999998 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:02.941133+0000", 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7936999999999999e-05 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:133614 2.7 e172/114 2:ef127ea4:::benchmark_data_smithi159_80260_object44501:head v 172'8316, mlcod=172'8313)", 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.177607460000001, 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19239903699999999, 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.523 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263900+0000", 2024-04-15T18:14:34.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1039999999999999e-06 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263909+0000", 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1810000000000007e-06 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453017+0000", 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18910781300000001 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453026+0000", 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3239999999999992e-06 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453087+0000", 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0713e-05 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453111+0000", 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4711000000000001e-05 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.456254+0000", 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031426890000000002 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.456298+0000", 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3501999999999998e-05 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:146409 2.0 e175/114 2:0952d99d:::benchmark_data_smithi159_80260_object57296:head v 175'9055, mlcod=175'9051)", 2024-04-15T18:14:34.527 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:21.519425+0000", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 12.922081542000001, 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.092958915000000003, 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.519425+0000", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.519425+0000", 2024-04-15T18:14:34.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.519426+0000", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1429999999999999e-06 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.519438+0000", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2736e-05 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.609022+0000", 2024-04-15T18:14:34.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.089583852000000005 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.609029+0000", 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8210000000000004e-06 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.609079+0000", 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0105000000000002e-05 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.609097+0000", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8185999999999998e-05 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.612337+0000", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032397530000000002 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:21.612383+0000", 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6319000000000002e-05 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.531 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:154895 2.0 e176/114 2:081b6fcb:::benchmark_data_smithi159_80260_object65782:head v 176'9588, mlcod=176'9585)", 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:34.087750+0000", 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.35375635599999999, 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.104033376, 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.087750+0000", 2024-04-15T18:14:34.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.087750+0000", 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.087751+0000", 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.119e-06 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.087759+0000", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6289999999999999e-06 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.188399+0000", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.100639938 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.188408+0000", 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.188459+0000", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0683999999999998e-05 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.188478+0000", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9089999999999998e-05 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.191736+0000", 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032585840000000001 2024-04-15T18:14:34.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:34.191783+0000", 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6672999999999999e-05 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:14:34.536 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:34.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:34 smithi082 ceph-mon[44206]: pgmap v508: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:14:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:34.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:34.567 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:34.593 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:34.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:34.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:34.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:34 smithi049 ceph-mon[40087]: pgmap v508: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:14:34.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:34.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:34.738 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:35.303 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:35.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:35.313 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:14:35.408 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:35.438 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:35.459 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:35.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:35.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:35.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:35 smithi049 ceph-mon[40087]: osdmap e177: 12 total, 12 up, 12 in 2024-04-15T18:14:35.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:35 smithi049 ceph-mon[40087]: 2.2 scrub starts 2024-04-15T18:14:35.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:35 smithi049 ceph-mon[40087]: 2.2 scrub ok 2024-04-15T18:14:35.613 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:35.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:35 smithi159 ceph-mon[45336]: osdmap e177: 12 total, 12 up, 12 in 2024-04-15T18:14:35.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:35 smithi159 ceph-mon[45336]: 2.2 scrub starts 2024-04-15T18:14:35.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:35 smithi159 ceph-mon[45336]: 2.2 scrub ok 2024-04-15T18:14:35.693 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:35.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:35 smithi082 ceph-mon[44206]: osdmap e177: 12 total, 12 up, 12 in 2024-04-15T18:14:35.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:35 smithi082 ceph-mon[44206]: 2.2 scrub starts 2024-04-15T18:14:35.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:35 smithi082 ceph-mon[44206]: 2.2 scrub ok 2024-04-15T18:14:36.262 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-15T18:14:36.262 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-15T18:14:36.263 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-15T18:14:36.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3 2024-04-15T18:14:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:36 smithi159 ceph-mon[45336]: pgmap v510: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s 2024-04-15T18:14:36.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:36 smithi159 ceph-mon[45336]: 2.5 deep-scrub starts 2024-04-15T18:14:36.727 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:14:36 smithi049 systemd[1]: Stopping Ceph osd.3 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:14:36.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:36 smithi049 ceph-mon[40087]: pgmap v510: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s 2024-04-15T18:14:36.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:36 smithi049 ceph-mon[40087]: 2.5 deep-scrub starts 2024-04-15T18:14:36.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:36 smithi082 ceph-mon[44206]: pgmap v510: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s 2024-04-15T18:14:36.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:36 smithi082 ceph-mon[44206]: 2.5 deep-scrub starts 2024-04-15T18:14:37.887 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:37.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:37.994 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:37.995 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:14:37.995 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:14:37.995 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:37.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:38.023 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:38.046 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:38.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:38.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:38.184 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:38.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:38 smithi049 ceph-mon[40087]: pgmap v511: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-04-15T18:14:38.227 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:14:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[82102]: 2024-04-15T18:14:38.093+0000 7f709add6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:14:38.227 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:14:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[82102]: 2024-04-15T18:14:38.093+0000 7f709add6700 -1 osd.3 177 *** Got signal Terminated *** 2024-04-15T18:14:38.227 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:14:38 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[82102]: 2024-04-15T18:14:38.093+0000 7f709add6700 -1 osd.3 177 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:14:38.242 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:38.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:38 smithi082 ceph-mon[44206]: pgmap v511: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-04-15T18:14:38.346 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:38.357 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:38 smithi159 ceph-mon[45336]: pgmap v511: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s 2024-04-15T18:14:38.377 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:38.734 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:38.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:38.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:14:38.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:38.867 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:38.891 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:38.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:38.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:39.028 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:39.046 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:14:39.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:39.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:39.124 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:39.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:39.189 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:39 smithi082 ceph-mon[44206]: 2.6 deep-scrub ok 2024-04-15T18:14:39.189 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:39 smithi082 ceph-mon[44206]: osd.3 marked itself down and dead 2024-04-15T18:14:39.189 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:39.191 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:39.210 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:39.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:39.313 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:39.340 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:39.358 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:39.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:39.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:39 smithi159 ceph-mon[45336]: 2.6 deep-scrub ok 2024-04-15T18:14:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:39 smithi159 ceph-mon[45336]: osd.3 marked itself down and dead 2024-04-15T18:14:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:39.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:39 smithi049 ceph-mon[40087]: 2.6 deep-scrub ok 2024-04-15T18:14:39.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:39 smithi049 ceph-mon[40087]: osd.3 marked itself down and dead 2024-04-15T18:14:39.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:14:39.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:39.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:39.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:39.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:39.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:39.642 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:39.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:14:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:40 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:40 smithi082 ceph-mon[44206]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:14:40.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:40 smithi082 ceph-mon[44206]: osdmap e178: 12 total, 11 up, 12 in 2024-04-15T18:14:40.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:40 smithi082 ceph-mon[44206]: pgmap v513: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s 2024-04-15T18:14:40.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:40 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:40.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:40 smithi049 ceph-mon[40087]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:14:40.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:40.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:40 smithi049 ceph-mon[40087]: osdmap e178: 12 total, 11 up, 12 in 2024-04-15T18:14:40.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:40 smithi049 ceph-mon[40087]: pgmap v513: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s 2024-04-15T18:14:40.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:40 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:40.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:40 smithi159 ceph-mon[45336]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:14:40.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2543159188' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:14:40.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:40 smithi159 ceph-mon[45336]: osdmap e178: 12 total, 11 up, 12 in 2024-04-15T18:14:40.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:40 smithi159 ceph-mon[45336]: pgmap v513: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 515 op/s 2024-04-15T18:14:40.392 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:40.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:40.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:40.530 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:40.552 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:40.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:40.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:40.679 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:40.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:40.729 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:40.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:40.839 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:40.871 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:40.893 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:40.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:40.977 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:14:40 smithi049 bash[116765]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3 2024-04-15T18:14:41.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:41.054 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:41 smithi159 ceph-mon[45336]: osdmap e179: 12 total, 11 up, 12 in 2024-04-15T18:14:41.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:41.107 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:41.108 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:14:41.108 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:14:41.108 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:14:41.108 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:14:41.108 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:14:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:41.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:41.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:41.259 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:41.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:41 smithi082 ceph-mon[44206]: osdmap e179: 12 total, 11 up, 12 in 2024-04-15T18:14:41.297 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:41.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:41.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:41 smithi049 ceph-mon[40087]: osdmap e179: 12 total, 11 up, 12 in 2024-04-15T18:14:41.406 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:41.438 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:41.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:41.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:41.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:41.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:41.620 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:41.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:41.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:41.759 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:41.782 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:41.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:41.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:41.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:14:41.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:14:41.935 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:42.162 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:42 smithi159 ceph-mon[45336]: pgmap v515: 17 pgs: 1 peering, 2 active+clean+scrubbing+deep, 14 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-04-15T18:14:42.283 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:14:42.295 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:42.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:42.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:42.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:42 smithi049 ceph-mon[40087]: pgmap v515: 17 pgs: 1 peering, 2 active+clean+scrubbing+deep, 14 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-04-15T18:14:42.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:42 smithi082 ceph-mon[44206]: pgmap v515: 17 pgs: 1 peering, 2 active+clean+scrubbing+deep, 14 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-04-15T18:14:42.594 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:42.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:42.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:42.752 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:43.160 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:43.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:43 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 5809/59556 objects degraded (9.754%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:43.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:43.378 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:43.412 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:43.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:43.499 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:43.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:43.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:43 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 5809/59556 objects degraded (9.754%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:43.577 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:43.616 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:43.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:43.663 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:43 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 5809/59556 objects degraded (9.754%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:43.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:43.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:43.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:43.714 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:44.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-15T18:14:44.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:14:44.173 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-15T18:14:44.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd down 3 2024-04-15T18:14:44.318 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:44.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:44.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: pgmap v516: 17 pgs: 5 active+undersized+degraded, 1 peering, 1 active+clean+scrubbing+deep, 10 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 5809/59556 objects degraded (9.754%) 2024-04-15T18:14:44.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1146045891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:44.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:44.471 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:44.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: pgmap v516: 17 pgs: 5 active+undersized+degraded, 1 peering, 1 active+clean+scrubbing+deep, 10 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 5809/59556 objects degraded (9.754%) 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1146045891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: pgmap v516: 17 pgs: 5 active+undersized+degraded, 1 peering, 1 active+clean+scrubbing+deep, 10 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 5809/59556 objects degraded (9.754%) 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1146045891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:14:44.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:14:44.931 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:44.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:45.038 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:45.085 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:45.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:45.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:45.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:45.735 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:46.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:46 smithi049 ceph-mon[40087]: 2.5 deep-scrub ok 2024-04-15T18:14:46.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:46 smithi049 ceph-mon[40087]: pgmap v517: 17 pgs: 6 active+undersized+degraded, 1 active+clean+scrubbing+deep, 10 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 6942/56346 objects degraded (12.320%) 2024-04-15T18:14:46.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:46 smithi049 ceph-mon[40087]: osdmap e180: 12 total, 11 up, 12 in 2024-04-15T18:14:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:46 smithi082 ceph-mon[44206]: 2.5 deep-scrub ok 2024-04-15T18:14:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:46 smithi082 ceph-mon[44206]: pgmap v517: 17 pgs: 6 active+undersized+degraded, 1 active+clean+scrubbing+deep, 10 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 6942/56346 objects degraded (12.320%) 2024-04-15T18:14:46.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:46 smithi082 ceph-mon[44206]: osdmap e180: 12 total, 11 up, 12 in 2024-04-15T18:14:46.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:46 smithi159 ceph-mon[45336]: 2.5 deep-scrub ok 2024-04-15T18:14:46.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:46 smithi159 ceph-mon[45336]: pgmap v517: 17 pgs: 6 active+undersized+degraded, 1 active+clean+scrubbing+deep, 10 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 503 op/s; 6942/56346 objects degraded (12.320%) 2024-04-15T18:14:46.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:46 smithi159 ceph-mon[45336]: osdmap e180: 12 total, 11 up, 12 in 2024-04-15T18:14:46.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:14:47.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:47.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:47.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:47.394 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:47.478 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:47.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:47.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:47.614 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:47.640 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:47.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:47.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:47.784 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:47.989 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:47.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:48.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:48 smithi082 ceph-mon[44206]: pgmap v519: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 6430/50307 objects degraded (12.782%) 2024-04-15T18:14:48.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:48.041 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:14:48.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:48 smithi049 ceph-mon[40087]: pgmap v519: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 6430/50307 objects degraded (12.782%) 2024-04-15T18:14:48.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:48.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:48.132 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:48.157 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:48.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:48.261 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:48.289 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:48.316 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:48.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:48 smithi159 ceph-mon[45336]: pgmap v519: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 6430/50307 objects degraded (12.782%) 2024-04-15T18:14:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:14:48.409 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:14:48.410 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:14:48.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:48.448 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:48.471 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:48.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:48.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:48.614 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:48.820 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:48.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:48.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:48.964 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:49 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:49.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:49.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:49 smithi049 ceph-mon[40087]: osdmap e181: 12 total, 11 up, 12 in 2024-04-15T18:14:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:49 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:49.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:49 smithi159 ceph-mon[45336]: osdmap e181: 12 total, 11 up, 12 in 2024-04-15T18:14:49.178 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:49.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:49.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:14:49.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:49 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:49.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2158784868' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:14:49.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:49 smithi082 ceph-mon[44206]: osdmap e181: 12 total, 11 up, 12 in 2024-04-15T18:14:49.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:49.319 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.345 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:49.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:49.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:49.478 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.502 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:49.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:49.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:49.635 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.658 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:49.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:49.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:49.808 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.839 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:49.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3 is already down. 2024-04-15T18:14:49.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:49.975 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:49.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:50.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:50.102 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:50.131 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:50.152 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:50.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:50.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:50 smithi159 ceph-mon[45336]: pgmap v521: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 999 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 5643/47946 objects degraded (11.769%) 2024-04-15T18:14:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/360192933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.200 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:50.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:49 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:14:49.924+0000 7f8015e7e700 -1 mon.a@0(leader).osd e181 definitely_dead 0 2024-04-15T18:14:50.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:50 smithi049 ceph-mon[40087]: pgmap v521: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 999 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 5643/47946 objects degraded (11.769%) 2024-04-15T18:14:50.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/360192933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:50.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:50 smithi082 ceph-mon[44206]: pgmap v521: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 999 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 5643/47946 objects degraded (11.769%) 2024-04-15T18:14:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/360192933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["3"]}]: dispatch 2024-04-15T18:14:50.297 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:50.328 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:50.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:50.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:50.470 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:50.726 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:50.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:50.740 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-04-15T18:14:50.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd out 3 2024-04-15T18:14:50.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:50.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:14:50.859 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:50.882 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:50.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:50.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:51.019 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:51.044 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:51.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:51.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:51.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:51 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5643/47946 objects degraded (11.769%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:51.180 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:51.204 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:51.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:51.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:51 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5643/47946 objects degraded (11.769%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:51.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:51.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:51 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5643/47946 objects degraded (11.769%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:51.353 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:51.473 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:51.565 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:51.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:51.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:51.721 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:52 smithi049 ceph-mon[40087]: pgmap v522: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 833 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 5339/39942 objects degraded (13.367%) 2024-04-15T18:14:52.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:14:52.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:14:52.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:52 smithi159 ceph-mon[45336]: pgmap v522: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 833 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 5339/39942 objects degraded (13.367%) 2024-04-15T18:14:52.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:14:52.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:14:52.483 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:14:52.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:14:52.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:52 smithi082 ceph-mon[44206]: pgmap v522: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 833 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 817 op/s; 5339/39942 objects degraded (13.367%) 2024-04-15T18:14:52.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:14:52.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:14:52.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:14:52.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:14:52.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:14:52.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:14:52.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:14:52.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:14:52.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: osdmap e182: 12 total, 11 up, 12 in 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1159432104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: osdmap e182: 12 total, 11 up, 12 in 2024-04-15T18:14:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1159432104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: osdmap e182: 12 total, 11 up, 12 in 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1159432104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:53.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.286 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:54.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:54.313 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:54.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:54 smithi159 ceph-mon[45336]: pgmap v524: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 734 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 4041/35199 objects degraded (11.480%) 2024-04-15T18:14:54.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:54 smithi049 ceph-mon[40087]: pgmap v524: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 734 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 4041/35199 objects degraded (11.480%) 2024-04-15T18:14:54.435 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:14:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:54 smithi082 ceph-mon[44206]: pgmap v524: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 734 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 893 op/s; 4041/35199 objects degraded (11.480%) 2024-04-15T18:14:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:14:55.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:14:55.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:14:55.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:14:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:14:55.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:14:55.501 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:55.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:14:55.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:14:55.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:14:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:55.899 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:14:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:55.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:14:56.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:14:56.050 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:56 smithi049 ceph-mon[40087]: pgmap v525: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 642 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 3758/30798 objects degraded (12.202%) 2024-04-15T18:14:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:56 smithi049 ceph-mon[40087]: osdmap e183: 12 total, 11 up, 12 in 2024-04-15T18:14:56.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:56 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3758/30798 objects degraded (12.202%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:56.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:56 smithi159 ceph-mon[45336]: pgmap v525: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 642 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 3758/30798 objects degraded (12.202%) 2024-04-15T18:14:56.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:56 smithi159 ceph-mon[45336]: osdmap e183: 12 total, 11 up, 12 in 2024-04-15T18:14:56.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:56 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3758/30798 objects degraded (12.202%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:56.515 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:14:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:56.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:14:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:56 smithi082 ceph-mon[44206]: pgmap v525: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 642 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 3758/30798 objects degraded (12.202%) 2024-04-15T18:14:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:56 smithi082 ceph-mon[44206]: osdmap e183: 12 total, 11 up, 12 in 2024-04-15T18:14:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:56 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3758/30798 objects degraded (12.202%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:14:56.622 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:56.674 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:56.699 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:56.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:56.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:56.833 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:56.856 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:56.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:56.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:14:57.004 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:57.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:14:57.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:57.261 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:14:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:57.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:14:57.369 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:57.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:57.400 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:57.403 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:14:57.427 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:57.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:57.534 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:57.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:14:57.580 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:57.615 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:57.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:57.720 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:57.759 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:57.790 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:57.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:57.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:57.937 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:57.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:57.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:58.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:14:58.098 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.121 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:14:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:14:58.226 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:58.259 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:58.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:58 smithi159 ceph-mon[45336]: pgmap v527: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 500 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 3213/23988 objects degraded (13.394%) 2024-04-15T18:14:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:58 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:58 smithi159 ceph-mon[45336]: osdmap e184: 12 total, 11 up, 12 in 2024-04-15T18:14:58.426 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.457 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:14:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:14:58.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:58 smithi049 ceph-mon[40087]: pgmap v527: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 500 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 3213/23988 objects degraded (13.394%) 2024-04-15T18:14:58.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:58 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:58.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:58.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:58 smithi049 ceph-mon[40087]: osdmap e184: 12 total, 11 up, 12 in 2024-04-15T18:14:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:58 smithi082 ceph-mon[44206]: pgmap v527: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 500 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 998 op/s; 3213/23988 objects degraded (13.394%) 2024-04-15T18:14:58.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:58 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:14:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/561244387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:14:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:58 smithi082 ceph-mon[44206]: osdmap e184: 12 total, 11 up, 12 in 2024-04-15T18:14:58.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:58.593 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.615 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:58.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:58.762 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.793 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:58.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:58.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:58.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:58.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:58.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:58 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:14:58.792+0000 7f8015e7e700 -1 mon.a@0(leader).osd e184 definitely_dead 0 2024-04-15T18:14:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:14:59.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:173757 2.c e184/114 2:3626edd4:::benchmark_data_smithi159_80260_object84644:head v 184'10959, mlcod=184'10957)", 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:59.068973+0000", 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.005331166, 2024-04-15T18:14:59.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053542099999999999, 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.068973+0000", 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.068973+0000", 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.068973+0000", 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.068980+0000", 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6100000000000002e-06 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.069011+0000", 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1168999999999999e-05 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.069020+0000", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4670000000000004e-06 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.069080+0000", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0396000000000002e-05 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.069103+0000", 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2558e-05 2024-04-15T18:14:59.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:173758 2.c e184/114 2:36611e25:::benchmark_data_smithi159_80260_object84645:head v 184'10960, mlcod=184'10957)", 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:59.070371+0000", 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.003932835, 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039993490000000001, 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:14:59.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070371+0000", 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070371+0000", 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:14:59.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070371+0000", 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1799999999999995e-07 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070377+0000", 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9009999999999999e-06 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070408+0000", 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0552999999999997e-05 2024-04-15T18:14:59.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.070415+0000", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9369999999999998e-06 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073060+0000", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026456330000000001 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073082+0000", 2024-04-15T18:14:59.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1747999999999999e-05 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:173764 2.1 e184/178 2:85857685:::benchmark_data_smithi159_80260_object84651:head v 184'10785, mlcod=184'10783)", 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:59.072968+0000", 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001335522, 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014383930000000001, 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:14:59.089 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.072968+0000", 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.072968+0000", 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:14:59.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.072969+0000", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.072976+0000", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2479999999999999e-06 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073010+0000", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4004999999999997e-05 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:14:59.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073017+0000", 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2230000000000001e-06 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073120+0000", 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010247399999999999 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:59.073144+0000", 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4224999999999999e-05 2024-04-15T18:14:59.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 3 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:14:59.093 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.106 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.3. 2024-04-15T18:14:59.113 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:14:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:14:59.217 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:14:59.243 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.261 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:14:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:14:59.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3921516653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:14:59 smithi049 ceph-mon[40087]: Client client.admin marked osd.3 out, after it was down for 19 seconds 2024-04-15T18:14:59.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:14:59.390 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3921516653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.391 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.391 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:14:59 smithi159 ceph-mon[45336]: Client client.admin marked osd.3 out, after it was down for 19 seconds 2024-04-15T18:14:59.392 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.411 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:14:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:14:59.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:14:59.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3921516653' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-04-15T18:14:59.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:14:59 smithi082 ceph-mon[44206]: Client client.admin marked osd.3 out, after it was down for 19 seconds 2024-04-15T18:14:59.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.559 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:14:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:14:59.662 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:14:59.688 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.706 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:14:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:14:59.810 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:14:59.841 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:14:59.860 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:14:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:14:59.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:14:59.965 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:14:59.991 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:00.010 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:00.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:00.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:00.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:00.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:15:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:00 smithi159 ceph-mon[45336]: pgmap v529: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 449 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s; 2395/21534 objects degraded (11.122%) 2024-04-15T18:15:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:00 smithi159 ceph-mon[45336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:15:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-15T18:15:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:00 smithi159 ceph-mon[45336]: osdmap e185: 12 total, 11 up, 11 in 2024-04-15T18:15:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:00 smithi159 ceph-mon[45336]: osdmap e186: 12 total, 11 up, 11 in 2024-04-15T18:15:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:00 smithi049 ceph-mon[40087]: pgmap v529: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 449 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s; 2395/21534 objects degraded (11.122%) 2024-04-15T18:15:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:00 smithi049 ceph-mon[40087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:15:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-15T18:15:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:00 smithi049 ceph-mon[40087]: osdmap e185: 12 total, 11 up, 11 in 2024-04-15T18:15:00.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:00 smithi049 ceph-mon[40087]: osdmap e186: 12 total, 11 up, 11 in 2024-04-15T18:15:00.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:00 smithi082 ceph-mon[44206]: pgmap v529: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 449 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 759 op/s; 2395/21534 objects degraded (11.122%) 2024-04-15T18:15:00.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:00 smithi082 ceph-mon[44206]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:15:00.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-04-15T18:15:00.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:00 smithi082 ceph-mon[44206]: osdmap e185: 12 total, 11 up, 11 in 2024-04-15T18:15:00.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:00 smithi082 ceph-mon[44206]: osdmap e186: 12 total, 11 up, 11 in 2024-04-15T18:15:00.558 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-04-15T18:15:00.558 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi049.front.sepia.ceph.com 2024-04-15T18:15:00.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.3 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-04-15T18:15:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:01.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:01.315 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.3/config 2024-04-15T18:15:01.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:01.471 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:01.628 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:01.633 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:01.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:01.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:15:01.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:01.798 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:01.960 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:01.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:02.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:02.120 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:02.146 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:02.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: pgmap v532: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 315 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 872 op/s; 1849/15093 objects degraded (12.251%) 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: osdmap e187: 12 total, 11 up, 11 in 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.9 starting backfill to osd.2 from (0'0,0'0] MAX to 184'10726 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.6 starting backfill to osd.4 from (0'0,0'0] MAX to 184'12371 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 184'10786 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 184'10774 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.d starting backfill to osd.2 from (0'0,0'0] MAX to 184'11103 2024-04-15T18:15:02.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: 2.a starting backfill to osd.8 from (0'0,0'0] MAX to 184'11026 2024-04-15T18:15:02.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1849/15093 objects degraded (12.251%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:02.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:02.300 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:02.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: pgmap v532: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 315 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 872 op/s; 1849/15093 objects degraded (12.251%) 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: osdmap e187: 12 total, 11 up, 11 in 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.9 starting backfill to osd.2 from (0'0,0'0] MAX to 184'10726 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.6 starting backfill to osd.4 from (0'0,0'0] MAX to 184'12371 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 184'10786 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 184'10774 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.d starting backfill to osd.2 from (0'0,0'0] MAX to 184'11103 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: 2.a starting backfill to osd.8 from (0'0,0'0] MAX to 184'11026 2024-04-15T18:15:02.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1849/15093 objects degraded (12.251%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:02.465 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:02.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: pgmap v532: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 315 MiB data, 1.6 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 872 op/s; 1849/15093 objects degraded (12.251%) 2024-04-15T18:15:02.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: osdmap e187: 12 total, 11 up, 11 in 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.9 starting backfill to osd.2 from (0'0,0'0] MAX to 184'10726 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.6 starting backfill to osd.4 from (0'0,0'0] MAX to 184'12371 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.1 starting backfill to osd.8 from (0'0,0'0] MAX to 184'10786 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.e starting backfill to osd.7 from (0'0,0'0] MAX to 184'10774 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.d starting backfill to osd.2 from (0'0,0'0] MAX to 184'11103 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: 2.a starting backfill to osd.8 from (0'0,0'0] MAX to 184'11026 2024-04-15T18:15:02.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1849/15093 objects degraded (12.251%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:02.498 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:02.522 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:02.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:02.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:02.655 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:02.680 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:02.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:15:02.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:15:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:02.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:02.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:15:02.744 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:15:02.780 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:15:02.788 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:02.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:15:02.817 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:02.838 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:02.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:02.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:15:02.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:15:02.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:15:02.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:02.963 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:15:02.993 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:03.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:15:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:15:03.029 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:03.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:03.100 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/478963314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:03.100 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.137 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:03.166 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:03.185 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:03.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:03.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:03.314 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/478963314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:03.314 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.314 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.316 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:03.338 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:03.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:03.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:03.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/478963314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:03.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.473 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:03.496 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:03.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:03.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:03.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:03.631 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:15:04.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: pgmap v534: 17 pgs: 1 active+undersized+remapped, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 261 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 735/12486 objects degraded (5.887%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:15:04.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:04 smithi159 ceph-mon[45336]: osdmap e188: 12 total, 11 up, 11 in 2024-04-15T18:15:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: pgmap v534: 17 pgs: 1 active+undersized+remapped, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 261 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 735/12486 objects degraded (5.887%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:15:04.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:04 smithi049 ceph-mon[40087]: osdmap e188: 12 total, 11 up, 11 in 2024-04-15T18:15:04.540 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: pgmap v534: 17 pgs: 1 active+undersized+remapped, 1 peering, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 261 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 735/12486 objects degraded (5.887%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:15:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:04 smithi082 ceph-mon[44206]: osdmap e188: 12 total, 11 up, 11 in 2024-04-15T18:15:04.982 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:04.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:05.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:05 smithi159 ceph-mon[45336]: osdmap e189: 12 total, 11 up, 11 in 2024-04-15T18:15:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:05 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:15:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:05 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:15:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:05 smithi049 ceph-mon[40087]: osdmap e189: 12 total, 11 up, 11 in 2024-04-15T18:15:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:05 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:15:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:05 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:15:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:05 smithi082 ceph-mon[44206]: osdmap e189: 12 total, 11 up, 11 in 2024-04-15T18:15:05.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:05 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:15:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:05 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:15:05.557 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:15:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:05.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:05.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:06.001 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:06.019 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:06.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:06.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:15:06.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:06.155 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:06.187 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:06.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:06.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:06 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:06 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:06 smithi159 ceph-mon[45336]: pgmap v537: 17 pgs: 1 peering, 1 active+undersized+remapped, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 225 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 472 op/s; 735/10761 objects degraded (6.830%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:15:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:06 smithi159 ceph-mon[45336]: osdmap e190: 12 total, 11 up, 11 in 2024-04-15T18:15:06.452 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:06.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:06 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:06.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:06 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:06.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:06 smithi049 ceph-mon[40087]: pgmap v537: 17 pgs: 1 peering, 1 active+undersized+remapped, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 225 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 472 op/s; 735/10761 objects degraded (6.830%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:15:06.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:06 smithi049 ceph-mon[40087]: osdmap e190: 12 total, 11 up, 11 in 2024-04-15T18:15:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:06.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:06 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:06 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:06 smithi082 ceph-mon[44206]: pgmap v537: 17 pgs: 1 peering, 1 active+undersized+remapped, 4 active+undersized+degraded+remapped+backfilling, 11 active+clean; 225 MiB data, 1.4 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 472 op/s; 735/10761 objects degraded (6.830%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:15:06.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:06 smithi082 ceph-mon[44206]: osdmap e190: 12 total, 11 up, 11 in 2024-04-15T18:15:06.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:07.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:07 smithi049 ceph-mon[40087]: osdmap e191: 12 total, 11 up, 11 in 2024-04-15T18:15:07.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 735/10761 objects degraded (6.830%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:07.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:07 smithi049 ceph-mon[40087]: osdmap e192: 12 total, 11 up, 11 in 2024-04-15T18:15:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:07 smithi159 ceph-mon[45336]: osdmap e191: 12 total, 11 up, 11 in 2024-04-15T18:15:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 735/10761 objects degraded (6.830%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:07.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:07 smithi159 ceph-mon[45336]: osdmap e192: 12 total, 11 up, 11 in 2024-04-15T18:15:07.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:07 smithi082 ceph-mon[44206]: osdmap e191: 12 total, 11 up, 11 in 2024-04-15T18:15:07.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 735/10761 objects degraded (6.830%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:15:07.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:07 smithi082 ceph-mon[44206]: osdmap e192: 12 total, 11 up, 11 in 2024-04-15T18:15:07.708 INFO:teuthology.orchestra.run.smithi049.stdout:2.6 2024-04-15T18:15:07.709 INFO:teuthology.orchestra.run.smithi049.stdout:2.a 2024-04-15T18:15:07.709 INFO:teuthology.orchestra.run.smithi049.stdout:2.9 2024-04-15T18:15:07.709 INFO:teuthology.orchestra.run.smithi049.stdout:2.d 2024-04-15T18:15:07.709 INFO:teuthology.orchestra.run.smithi049.stdout:2.1 2024-04-15T18:15:07.709 INFO:teuthology.orchestra.run.smithi049.stdout:2.e 2024-04-15T18:15:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:15:07.926 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:15:08.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:08 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:08.231 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:08 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:08.231 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:08 smithi049 ceph-mon[40087]: pgmap v541: 17 pgs: 1 activating, 1 peering, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean; 176 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 338 op/s; 347/8448 objects degraded (4.107%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:15:08.231 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:08 smithi049 ceph-mon[40087]: osdmap e193: 12 total, 11 up, 11 in 2024-04-15T18:15:08.240 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:08.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:08.348 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:08.386 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:08 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:08.386 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:08 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:08.386 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:08 smithi159 ceph-mon[45336]: pgmap v541: 17 pgs: 1 activating, 1 peering, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean; 176 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 338 op/s; 347/8448 objects degraded (4.107%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:15:08.386 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:08 smithi159 ceph-mon[45336]: osdmap e193: 12 total, 11 up, 11 in 2024-04-15T18:15:08.389 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:08.417 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:08.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:08.523 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:08.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:08 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:08.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:08 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:08.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:08 smithi082 ceph-mon[44206]: pgmap v541: 17 pgs: 1 activating, 1 peering, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean; 176 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 338 op/s; 347/8448 objects degraded (4.107%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:15:08.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:08 smithi082 ceph-mon[44206]: osdmap e193: 12 total, 11 up, 11 in 2024-04-15T18:15:08.572 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:08.607 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:08.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:08.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:08.795 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:08.836 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:08.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:08.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:09.006 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.037 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:15:09.141 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:09.173 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.196 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:09.301 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:09.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:09.339 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.372 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:09.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:09.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:09.511 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:09.512 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.533 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:09.640 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:09.675 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.703 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:09.807 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:09.842 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:09.863 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:09.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:09.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:10.016 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:10.132 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:10 smithi159 ceph-mon[45336]: pgmap v543: 17 pgs: 1 activating, 2 peering, 14 active+clean; 162 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 251 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:15:10.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:10 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 347/8448 objects degraded (4.107%), 2 pgs degraded) 2024-04-15T18:15:10.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:10 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:15:10.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:10.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:10 smithi159 ceph-mon[45336]: osdmap e194: 12 total, 11 up, 11 in 2024-04-15T18:15:10.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:10 smithi049 ceph-mon[40087]: pgmap v543: 17 pgs: 1 activating, 2 peering, 14 active+clean; 162 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 251 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:15:10.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:10 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 347/8448 objects degraded (4.107%), 2 pgs degraded) 2024-04-15T18:15:10.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:10 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:15:10.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:10.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:10 smithi049 ceph-mon[40087]: osdmap e194: 12 total, 11 up, 11 in 2024-04-15T18:15:10.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:15:10.486 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:10.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:10 smithi082 ceph-mon[44206]: pgmap v543: 17 pgs: 1 activating, 2 peering, 14 active+clean; 162 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 251 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:15:10.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:10 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 347/8448 objects degraded (4.107%), 2 pgs degraded) 2024-04-15T18:15:10.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:10 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:15:10.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1493060595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:10.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:10 smithi082 ceph-mon[44206]: osdmap e194: 12 total, 11 up, 11 in 2024-04-15T18:15:10.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:10.618 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:10.637 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:10.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:10.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:10.768 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:11.117 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 89110 objects 2024-04-15T18:15:11.117 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :112.962 2024-04-15T18:15:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:12 smithi159 ceph-mon[45336]: pgmap v545: 17 pgs: 1 activating, 1 peering, 15 active+clean; 75 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:15:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:12 smithi159 ceph-mon[45336]: osdmap e195: 12 total, 11 up, 11 in 2024-04-15T18:15:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:12 smithi049 ceph-mon[40087]: pgmap v545: 17 pgs: 1 activating, 1 peering, 15 active+clean; 75 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:15:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:12.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:12 smithi049 ceph-mon[40087]: osdmap e195: 12 total, 11 up, 11 in 2024-04-15T18:15:12.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:12 smithi082 ceph-mon[44206]: pgmap v545: 17 pgs: 1 activating, 1 peering, 15 active+clean; 75 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 374 op/s; 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:15:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:12.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:12 smithi082 ceph-mon[44206]: osdmap e195: 12 total, 11 up, 11 in 2024-04-15T18:15:12.750 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:13.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:13 smithi049 ceph-mon[40087]: 2.a scrub starts 2024-04-15T18:15:13.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:13 smithi049 ceph-mon[40087]: 2.a scrub ok 2024-04-15T18:15:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:13 smithi159 ceph-mon[45336]: 2.a scrub starts 2024-04-15T18:15:13.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:13 smithi159 ceph-mon[45336]: 2.a scrub ok 2024-04-15T18:15:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:13 smithi082 ceph-mon[44206]: 2.a scrub starts 2024-04-15T18:15:13.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:13 smithi082 ceph-mon[44206]: 2.a scrub ok 2024-04-15T18:15:14.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:14 smithi159 ceph-mon[45336]: pgmap v547: 17 pgs: 17 active+clean; 20 MiB data, 863 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-04-15T18:15:14.456 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-15T18:15:14.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-15T18:15:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:14 smithi049 ceph-mon[40087]: pgmap v547: 17 pgs: 17 active+clean; 20 MiB data, 863 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-04-15T18:15:14.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:14 smithi082 ceph-mon[44206]: pgmap v547: 17 pgs: 17 active+clean; 20 MiB data, 863 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 416 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-04-15T18:15:14.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.3 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 2.d --file /var/log/ceph/exp.2.d.3 2024-04-15T18:15:14.620 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:14.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:14.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:14.758 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:14.779 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:14.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:14.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:14.917 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:14.938 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:14.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:15.035 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.3/config 2024-04-15T18:15:15.041 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:15.068 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:15.074 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:15.095 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:15.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:15.200 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:15.230 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:15.250 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:15.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:15.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:15.408 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:15.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:15.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:15.540 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:15.788 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:15.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:15.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:15:15.891 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:15.926 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:15.948 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:15.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:16.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:16.083 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:16.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:16 smithi159 ceph-mon[45336]: pgmap v548: 17 pgs: 17 active+clean; 2.4 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 362 op/s 2024-04-15T18:15:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:16 smithi049 ceph-mon[40087]: pgmap v548: 17 pgs: 17 active+clean; 2.4 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 362 op/s 2024-04-15T18:15:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:16 smithi082 ceph-mon[44206]: pgmap v548: 17 pgs: 17 active+clean; 2.4 MiB data, 836 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 362 op/s 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:15:17.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:15:17.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:15:17.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:15:17.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/551100368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:17.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/551100368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:15:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:15:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:15:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:15:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:15:17.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/551100368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:18.143 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:18.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: pgmap v549: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 322 op/s 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:18.281 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:18.302 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:18.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:18.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: pgmap v549: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 322 op/s 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.438 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:18.512 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:18.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:18.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: pgmap v549: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 322 op/s 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:18.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:18.650 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:19.075 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:15:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:15:19.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14157.0:37 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'23, mlcod=20'14)", 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:05:44.737894+0000", 2024-04-15T18:15:19.590 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 574.01941919800004, 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.985572542, 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:44.737894+0000", 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:44.737894+0000", 2024-04-15T18:15:19.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:44.737894+0000", 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3200000000000005e-07 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:44.737900+0000", 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.336e-06 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.722884+0000", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98498356099999995 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.722932+0000", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7695999999999999e-05 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.723103+0000", 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017127900000000001 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.723124+0000", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0647000000000001e-05 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.723428+0000", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000304147 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:05:45.723466+0000", 2024-04-15T18:15:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8244000000000002e-05 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:7809 2.8 e125/114 2:152bc777:::benchmark_data_smithi159_80260_object7808:head v 125'458, mlcod=125'454)", 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:11:52.894285+0000", 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 205.863028181, 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18062999399999999, 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.894285+0000", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.894285+0000", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.894285+0000", 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3799999999999997e-07 2024-04-15T18:15:19.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:52.894355+0000", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9667999999999998e-05 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074449+0000", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18009423699999999 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074454+0000", 2024-04-15T18:15:19.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6750000000000001e-06 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074494+0000", 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0615000000000003e-05 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074509+0000", 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4312e-05 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074885+0000", 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000376459 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:11:53.074915+0000", 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9490000000000001e-05 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:32496 2.1 e133/114 2:84d154fa:::benchmark_data_smithi159_80260_object32495:head v 133'2037, mlcod=133'2034)", 2024-04-15T18:15:19.599 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:10.754613+0000", 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 188.00270014399999, 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14535515299999999, 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.754613+0000", 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.754613+0000", 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.754613+0000", 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9599999999999999e-07 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.754631+0000", 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8139e-05 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.898663+0000", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.144032302 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.898669+0000", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1449999999999999e-06 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.898719+0000", 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0791999999999998e-05 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.898741+0000", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1682e-05 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.899922+0000", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011805520000000001 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:10.899968+0000", 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6044999999999997e-05 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.603 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:33657 2.8 e133/114 2:14ca00c2:::benchmark_data_smithi159_80260_object33656:head v 133'2072, mlcod=133'2068)", 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:11.626043+0000", 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 187.13126966900001, 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19655206, 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.626043+0000", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.626043+0000", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.626044+0000", 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.178e-06 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.626127+0000", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2387999999999998e-05 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.821554+0000", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19542784499999999 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.821562+0000", 2024-04-15T18:15:19.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.464e-06 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.821613+0000", 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0664000000000001e-05 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.821628+0000", 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5296999999999999e-05 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.822571+0000", 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00094271700000000003 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:11.822595+0000", 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4507000000000001e-05 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:35452 2.8 e134/114 2:116f8125:::benchmark_data_smithi159_80260_object35451:head v 134'2178, mlcod=134'2175)", 2024-04-15T18:15:19.608 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:12.990077+0000", 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 185.76723525099999, 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.129566083, 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:12.990077+0000", 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:12.990077+0000", 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:12.990078+0000", 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2099999999999996e-07 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:12.990156+0000", 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8199000000000001e-05 2024-04-15T18:15:19.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.118503+0000", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.128346659 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.118508+0000", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0640000000000001e-06 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.118560+0000", 2024-04-15T18:15:19.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1748e-05 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.118580+0000", 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0574999999999999e-05 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.119599+0000", 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010188790000000001 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:13.119644+0000", 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4237999999999999e-05 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:41156 2.8 e136/114 2:1aa6b45d:::benchmark_data_smithi159_80260_object41155:head v 136'2532, mlcod=136'2529)", 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:17.237543+0000", 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 181.51976973000001, 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.215641788, 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.613 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.237543+0000", 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.237543+0000", 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.237544+0000", 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7300000000000005e-07 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.237653+0000", 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010922899999999999 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.452010+0000", 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21435680900000001 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.452015+0000", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.339e-06 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.452062+0000", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6428999999999999e-05 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.452076+0000", 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4103e-05 2024-04-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.453152+0000", 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001075849 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:17.453185+0000", 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3256999999999997e-05 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:60285 2.1 e143/114 2:8c969d58:::benchmark_data_smithi159_80260_object60284:head v 143'3765, mlcod=143'3762)", 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:33.819152+0000", 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.93816078099999, 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.169232418, 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819152+0000", 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819152+0000", 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819153+0000", 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.02e-07 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.819177+0000", 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3893e-05 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.986795+0000", 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16761862799999999 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.986803+0000", 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2450000000000001e-06 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.986849+0000", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6798999999999997e-05 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.986865+0000", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5520999999999999e-05 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.988355+0000", 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014898330000000001 2024-04-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:33.988384+0000", 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9597000000000001e-05 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:62472 2.1 e143/114 2:81d12c07:::benchmark_data_smithi159_80260_object62471:head v 143'3896, mlcod=143'3893)", 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 162.82739185099999, 2024-04-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38137517700000001, 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929922+0000", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3099999999999997e-07 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929945+0000", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3850999999999999e-05 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308461+0000", 2024-04-15T18:15:19.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37851550900000003 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308470+0000", 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7199999999999995e-06 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308524+0000", 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4682999999999998e-05 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308547+0000", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447999999999999e-05 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.311264+0000", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027170699999999998 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.311296+0000", 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2165000000000001e-05 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:63167 2.8 e149/114 2:16c4c22e:::benchmark_data_smithi159_80260_object63166:head v 149'3913, mlcod=149'3910)", 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:51.367699+0000", 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 147.38961352699999, 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15288375500000001, 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.367699+0000", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.367699+0000", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.367700+0000", 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.68e-07 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.367808+0000", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010841 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.513958+0000", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14614988300000001 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.513964+0000", 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2829999999999998e-06 2024-04-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.514024+0000", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9926999999999997e-05 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.514052+0000", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8061000000000001e-05 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.520538+0000", 2024-04-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064858279999999999 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:51.520583+0000", 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4694999999999998e-05 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:64634 2.5 e150/114 2:a365c0f6:::benchmark_data_smithi159_80260_object64633:head v 150'4022, mlcod=150'4019)", 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:52.604996+0000", 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 146.15231673900001, 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12194490199999999, 2024-04-15T18:15:19.631 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.604996+0000", 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.604996+0000", 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.604997+0000", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0699999999999996e-07 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.607843+0000", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028465249999999999 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.725197+0000", 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11735416899999999 2024-04-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.725206+0000", 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5120000000000008e-06 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.725279+0000", 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2874999999999996e-05 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.725305+0000", 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6092999999999999e-05 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.726911+0000", 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016060930000000001 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:52.726941+0000", 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9828e-05 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:97475 2.5 e159/114 2:ad8d4387:::benchmark_data_smithi159_80260_object8362:head v 159'6011, mlcod=159'6005)", 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.644118646, 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25188648699999999, 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:15:19.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113195+0000", 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7100000000000002e-07 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113200+0000", 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9060000000000001e-06 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357485+0000", 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24428497299999999 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357492+0000", 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8350000000000004e-06 2024-04-15T18:15:19.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357561+0000", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8725000000000004e-05 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357580+0000", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9048000000000001e-05 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365036+0000", 2024-04-15T18:15:19.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0074560140000000004 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365081+0000", 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4415000000000001e-05 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:97538 2.5 e159/114 2:aaf52e94:::benchmark_data_smithi159_80260_object8425:head v 159'6012, mlcod=159'6007)", 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:26.176432+0000", 2024-04-15T18:15:19.640 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.58088112999999, 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18876969800000001, 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.176432+0000", 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.176432+0000", 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.176433+0000", 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1880000000000001e-06 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.176440+0000", 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0570000000000004e-06 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.361658+0000", 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.185217887 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.361666+0000", 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8339999999999999e-06 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.361739+0000", 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3323000000000003e-05 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.361781+0000", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2416000000000002e-05 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365178+0000", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003396558 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365201+0000", 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3434999999999998e-05 2024-04-15T18:15:19.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:97810 2.5 e159/114 2:ae860221:::benchmark_data_smithi159_80260_object8697:head v 159'6027, mlcod=159'6024)", 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:26.710825+0000", 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.04648736599999, 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.126268673, 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.710825+0000", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.710825+0000", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.710826+0000", 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6499999999999998e-07 2024-04-15T18:15:19.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.710832+0000", 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7540000000000003e-06 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.834961+0000", 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.124128689 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.834968+0000", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1690000000000003e-06 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.835019+0000", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1054000000000003e-05 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.835035+0000", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6646999999999998e-05 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.837065+0000", 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020299189999999998 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.837094+0000", 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8676000000000001e-05 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:124128 2.4 e169/114 2:2d06eb6f:::benchmark_data_smithi159_80260_object35015:head v 169'7818, mlcod=169'7814)", 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:53.894539+0000", 2024-04-15T18:15:19.649 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 84.862773656000002, 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14605886800000001, 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:53.894539+0000", 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:53.894539+0000", 2024-04-15T18:15:19.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:53.894540+0000", 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:53.894546+0000", 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7239999999999997e-06 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.037253+0000", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14270692200000001 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.037261+0000", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5990000000000002e-06 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.037322+0000", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0408000000000002e-05 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.037344+0000", 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2819999999999998e-05 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.040556+0000", 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032110200000000002 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.040598+0000", 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2407000000000002e-05 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.653 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:124395 2.8 e169/114 2:1fe47acc:::benchmark_data_smithi159_80260_object35282:head v 169'7744, mlcod=169'7740)", 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:54.338547+0000", 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 84.418766132000002, 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14046874400000001, 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.338547+0000", 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.338547+0000", 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.338548+0000", 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6500000000000008e-07 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.338554+0000", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7569999999999998e-06 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.475897+0000", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137342468 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.475902+0000", 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.475950+0000", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7876e-05 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.475965+0000", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4783000000000001e-05 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.478971+0000", 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030062909999999999 2024-04-15T18:15:19.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.479015+0000", 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3912000000000002e-05 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:124400 2.8 e169/114 2:1668ff5a:::benchmark_data_smithi159_80260_object35287:head v 169'7745, mlcod=169'7741)", 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:54.343434+0000", 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 84.413879160999997, 2024-04-15T18:15:19.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.164328314, 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.343434+0000", 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.343434+0000", 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.343434+0000", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3499999999999996e-07 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.343440+0000", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6069999999999998e-06 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.504431+0000", 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16099092000000001 2024-04-15T18:15:19.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.504438+0000", 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7570000000000004e-06 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.504491+0000", 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3001999999999998e-05 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.504514+0000", 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2855000000000001e-05 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.507713+0000", 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031984980000000001 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:54.507762+0000", 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9039999999999998e-05 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.662 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:129225 2.4 e169/114 2:2ec21f18:::benchmark_data_smithi159_80260_object40112:head v 169'8148, mlcod=169'8143)", 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:00.049968+0000", 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 78.707344663000001, 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13455234899999999, 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.049968+0000", 2024-04-15T18:15:19.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.049968+0000", 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.049969+0000", 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7800000000000001e-07 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.049975+0000", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9789999999999998e-06 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.181523+0000", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.131548374 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.181528+0000", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4260000000000002e-06 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.181576+0000", 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7113000000000001e-05 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.181591+0000", 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5719999999999999e-05 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.184475+0000", 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028837210000000001 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:00.184520+0000", 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5337999999999999e-05 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:133604 2.c e172/114 2:313518bc:::benchmark_data_smithi159_80260_object44491:head v 172'8414, mlcod=172'8411)", 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.503696787999999, 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23120621699999999, 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.667 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253617+0000", 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253623+0000", 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5470000000000002e-06 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477897+0000", 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.224274311 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477906+0000", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3059999999999998e-06 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477964+0000", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7972e-05 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477989+0000", 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5466e-05 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.484780+0000", 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067909049999999999 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.484822+0000", 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2033999999999998e-05 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.671 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:133619 2.5 e172/114 2:a2753e15:::benchmark_data_smithi159_80260_object44506:head v 172'8256, mlcod=172'8253)", 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:05.269106+0000", 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 73.488206773000002, 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15677485399999999, 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.269106+0000", 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.269106+0000", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.269107+0000", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.16e-07 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.269114+0000", 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.61e-06 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.422236+0000", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15312136000000001 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.422242+0000", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8490000000000002e-06 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.422292+0000", 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0717999999999998e-05 2024-04-15T18:15:19.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.422315+0000", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2619000000000001e-05 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.425849+0000", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035344679999999998 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.425881+0000", 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1414000000000001e-05 2024-04-15T18:15:19.675 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(osd.8.0:30013 2.5 e178/114 2:a65bf038:::benchmark_data_smithi159_80260_object79506:head v 178'10267, mlcod=178'10241)", 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:39.528998+0000", 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 39.228314228999999, 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20579220300000001, 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.528998+0000", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.528998+0000", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.528999+0000", 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9399999999999995e-07 2024-04-15T18:15:19.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.529004+0000", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2830000000000001e-06 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.729796+0000", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20079187400000001 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.729807+0000", 2024-04-15T18:15:19.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0519e-05 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.729890+0000", 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2912999999999997e-05 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.729922+0000", 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2573000000000001e-05 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:15:19.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.734729+0000", 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048073129999999997 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:39.734791+0000", 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1234000000000004e-05 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:15:19.680 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:15:19.773 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:19.799 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:20.235 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:20 smithi049 ceph-mon[40087]: pgmap v550: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 259 op/s 2024-04-15T18:15:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:20 smithi159 ceph-mon[45336]: pgmap v550: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 259 op/s 2024-04-15T18:15:20.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:20 smithi082 ceph-mon[44206]: pgmap v550: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 259 op/s 2024-04-15T18:15:21.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:21.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:21.445 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:21.466 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:21.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:15:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:21.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:21.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:21.600 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 2.d info 2.d( v 177'10080 (0'0,177'10080] local-lis/les=114/115 n=1316 ec=114/114 lis/c=114/114 les/c/f=115/115/0 sis=114) 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0018e1d:::benchmark_data_smithi159_80260_object85038:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0064ed6:::benchmark_data_smithi159_80260_object69177:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b00753ad:::benchmark_data_smithi159_80260_object81077:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b008fbe7:::benchmark_data_smithi159_80260_object79761:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b01448d4:::benchmark_data_smithi159_80260_object68972:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b01b34de:::benchmark_data_smithi159_80260_object86470:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b01fa662:::benchmark_data_smithi159_80260_object82254:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b022bda3:::benchmark_data_smithi159_80260_object87448:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0282be4:::benchmark_data_smithi159_80260_object80448:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b02af4ee:::benchmark_data_smithi159_80260_object72237:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b031ef8b:::benchmark_data_smithi159_80260_object71584:head# 2024-04-15T18:15:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b032be7f:::benchmark_data_smithi159_80260_object70616:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b035b7eb:::benchmark_data_smithi159_80260_object77595:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b038c14d:::benchmark_data_smithi159_80260_object73805:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b039c4b4:::benchmark_data_smithi159_80260_object81154:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b03da32a:::benchmark_data_smithi159_80260_object88985:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b03f0dcb:::benchmark_data_smithi159_80260_object76660:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b04c72d8:::benchmark_data_smithi159_80260_object82833:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b04f8586:::benchmark_data_smithi159_80260_object83811:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b052c116:::benchmark_data_smithi159_80260_object83119:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0543daa:::benchmark_data_smithi159_80260_object86104:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0577563:::benchmark_data_smithi159_80260_object79034:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0581559:::benchmark_data_smithi159_80260_object82149:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b05b4508:::benchmark_data_smithi159_80260_object75693:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b05f1cb4:::benchmark_data_smithi159_80260_object78161:head# 2024-04-15T18:15:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b06860a5:::benchmark_data_smithi159_80260_object77578:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b069a512:::benchmark_data_smithi159_80260_object83468:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b06b53d1:::benchmark_data_smithi159_80260_object77848:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b06d0469:::benchmark_data_smithi159_80260_object82646:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b06e3a72:::benchmark_data_smithi159_80260_object71610:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b072f85d:::benchmark_data_smithi159_80260_object71676:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b083f16f:::benchmark_data_smithi159_80260_object74511:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0850401:::benchmark_data_smithi159_80260_object86138:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b086db25:::benchmark_data_smithi159_80260_object74523:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b089a0ee:::benchmark_data_smithi159_80260_object86090:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b08b8e44:::benchmark_data_smithi159_80260_object75016:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b08c2f70:::benchmark_data_smithi159_80260_object78858:head# 2024-04-15T18:15:21.744 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b08e6ec9:::benchmark_data_smithi159_80260_object72751:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0907c5a:::benchmark_data_smithi159_80260_object71758:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0950324:::benchmark_data_smithi159_80260_object82906:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b095185a:::benchmark_data_smithi159_80260_object79032:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09559e4:::benchmark_data_smithi159_80260_object85788:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0958696:::benchmark_data_smithi159_80260_object81810:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b095c739:::benchmark_data_smithi159_80260_object77584:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b097445d:::benchmark_data_smithi159_80260_object80159:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09765a9:::benchmark_data_smithi159_80260_object83388:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09b6274:::benchmark_data_smithi159_80260_object75262:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09bc8f4:::benchmark_data_smithi159_80260_object88874:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09c9bd2:::benchmark_data_smithi159_80260_object81246:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b09e4de6:::benchmark_data_smithi159_80260_object68555:head# 2024-04-15T18:15:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0a156af:::benchmark_data_smithi159_80260_object76768:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0a175ff:::benchmark_data_smithi159_80260_object78065:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0a27b8f:::benchmark_data_smithi159_80260_object86771:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0a62532:::benchmark_data_smithi159_80260_object87694:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0a65e8f:::benchmark_data_smithi159_80260_object78522:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0aa5fa9:::benchmark_data_smithi159_80260_object83769:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0ab2f68:::benchmark_data_smithi159_80260_object80946:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0ab388e:::benchmark_data_smithi159_80260_object75809:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b050cf:::benchmark_data_smithi159_80260_object88479:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b0feb8:::benchmark_data_smithi159_80260_object84917:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b36c0c:::benchmark_data_smithi159_80260_object80135:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b385ee:::benchmark_data_smithi159_80260_object74191:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b59c24:::benchmark_data_smithi159_80260_object70802:head# 2024-04-15T18:15:21.746 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b6b749:::benchmark_data_smithi159_80260_object79656:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b7584c:::benchmark_data_smithi159_80260_object82103:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0b9e759:::benchmark_data_smithi159_80260_object79089:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0ba628e:::benchmark_data_smithi159_80260_object68968:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0bba882:::benchmark_data_smithi159_80260_object87999:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0be17f6:::benchmark_data_smithi159_80260_object83562:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0be4415:::benchmark_data_smithi159_80260_object88857:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0c5e708:::benchmark_data_smithi159_80260_object70010:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0ce2326:::benchmark_data_smithi159_80260_object84474:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0d01183:::benchmark_data_smithi159_80260_object70861:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0d171fb:::benchmark_data_smithi159_80260_object85472:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0d4af62:::benchmark_data_smithi159_80260_object77931:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0d69d60:::benchmark_data_smithi159_80260_object81268:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0dab7ba:::benchmark_data_smithi159_80260_object82466:head# 2024-04-15T18:15:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0dd4f10:::benchmark_data_smithi159_80260_object70862:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0deba0b:::benchmark_data_smithi159_80260_object83977:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0dfbba6:::benchmark_data_smithi159_80260_object87014:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0e08272:::benchmark_data_smithi159_80260_object76913:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0e163b9:::benchmark_data_smithi159_80260_object76665:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0eedca5:::benchmark_data_smithi159_80260_object79025:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0ef3c5a:::benchmark_data_smithi159_80260_object77916:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0f06416:::benchmark_data_smithi159_80260_object85484:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0f3973b:::benchmark_data_smithi159_80260_object71787:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0f7bb07:::benchmark_data_smithi159_80260_object69883:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0f853a6:::benchmark_data_smithi159_80260_object70953:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b0fbff9d:::benchmark_data_smithi159_80260_object72907:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1055e4c:::benchmark_data_smithi159_80260_object83815:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b10872f3:::benchmark_data_smithi159_80260_object71900:head# 2024-04-15T18:15:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b11fb7a2:::benchmark_data_smithi159_80260_object76283:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1233d02:::benchmark_data_smithi159_80260_object85024:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1275d86:::benchmark_data_smithi159_80260_object79268:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b128645c:::benchmark_data_smithi159_80260_object84364:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b12ac9e3:::benchmark_data_smithi159_80260_object75736:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b13018ca:::benchmark_data_smithi159_80260_object81195:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b133bc13:::benchmark_data_smithi159_80260_object86804:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1389a6f:::benchmark_data_smithi159_80260_object70121:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b13b3020:::benchmark_data_smithi159_80260_object76091:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b13c9a7d:::benchmark_data_smithi159_80260_object73010:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1427f61:::benchmark_data_smithi159_80260_object75834:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b14a394e:::benchmark_data_smithi159_80260_object71619:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b14e6664:::benchmark_data_smithi159_80260_object77310:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b152d21c:::benchmark_data_smithi159_80260_object72511:head# 2024-04-15T18:15:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1548ffa:::benchmark_data_smithi159_80260_object87454:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b155811a:::benchmark_data_smithi159_80260_object85361:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b155eebf:::benchmark_data_smithi159_80260_object70363:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b158ad43:::benchmark_data_smithi159_80260_object70746:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b158ec27:::benchmark_data_smithi159_80260_object73050:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b15e8679:::benchmark_data_smithi159_80260_object80252:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b166b510:::benchmark_data_smithi159_80260_object77841:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b169109c:::benchmark_data_smithi159_80260_object69409:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1694808:::benchmark_data_smithi159_80260_object71923:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b16d191c:::benchmark_data_smithi159_80260_object80543:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1726fe6:::benchmark_data_smithi159_80260_object73952:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b175b3fe:::benchmark_data_smithi159_80260_object75453:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1767a31:::benchmark_data_smithi159_80260_object74903:head# 2024-04-15T18:15:21.750 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1778b8c:::benchmark_data_smithi159_80260_object84751:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b179af7c:::benchmark_data_smithi159_80260_object73503:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b17ff248:::benchmark_data_smithi159_80260_object70367:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1827dc1:::benchmark_data_smithi159_80260_object76741:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b18920d6:::benchmark_data_smithi159_80260_object78854:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b190d645:::benchmark_data_smithi159_80260_object85771:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1931181:::benchmark_data_smithi159_80260_object76707:head# 2024-04-15T18:15:21.751 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b194e006:::benchmark_data_smithi159_80260_object80700:head# 2024-04-15T18:15:21.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b19c2e57:::benchmark_data_smithi159_80260_object83961:head# 2024-04-15T18:15:21.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b19e06cc:::benchmark_data_smithi159_80260_object84937:head# 2024-04-15T18:15:21.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1a30e97:::benchmark_data_smithi159_80260_object72767:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1a6bc61:::benchmark_data_smithi159_80260_object86607:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1aac9fb:::benchmark_data_smithi159_80260_object74574:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1ad1eae:::benchmark_data_smithi159_80260_object86994:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1aecf7d:::benchmark_data_smithi159_80260_object81266:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b03c41:::benchmark_data_smithi159_80260_object70069:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b46d4a:::benchmark_data_smithi159_80260_object86995:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b517c0:::benchmark_data_smithi159_80260_object88925:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b5f2f2:::benchmark_data_smithi159_80260_object69476:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b78e71:::benchmark_data_smithi159_80260_object77822:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b8128b:::benchmark_data_smithi159_80260_object83741:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b96ed4:::benchmark_data_smithi159_80260_object69133:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1b98984:::benchmark_data_smithi159_80260_object72672:head# 2024-04-15T18:15:21.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1ba674c:::benchmark_data_smithi159_80260_object78971:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1bd1914:::benchmark_data_smithi159_80260_object73699:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1bf9ba2:::benchmark_data_smithi159_80260_object79332:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1bfc3df:::benchmark_data_smithi159_80260_object73761:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1bffa80:::benchmark_data_smithi159_80260_object83800:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1c71b5b:::benchmark_data_smithi159_80260_object76710:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1c72b5f:::benchmark_data_smithi159_80260_object69291:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1ccdbaa:::benchmark_data_smithi159_80260_object73672:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1ce2786:::benchmark_data_smithi159_80260_object87739:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1cfffeb:::benchmark_data_smithi159_80260_object78811:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1d648b8:::benchmark_data_smithi159_80260_object85411:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1d8c2e7:::benchmark_data_smithi159_80260_object78663:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1d97047:::benchmark_data_smithi159_80260_object79683:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1d9d9aa:::benchmark_data_smithi159_80260_object73243:head# 2024-04-15T18:15:21.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1dd6cc2:::benchmark_data_smithi159_80260_object73426:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1df3b52:::benchmark_data_smithi159_80260_object82318:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1e0a51e:::benchmark_data_smithi159_80260_object87121:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1e3c037:::benchmark_data_smithi159_80260_object73338:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1e89113:::benchmark_data_smithi159_80260_object70575:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1ea4db0:::benchmark_data_smithi159_80260_object85883:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1eef2d2:::benchmark_data_smithi159_80260_object80081:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1f02f00:::benchmark_data_smithi159_80260_object84798:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1f52342:::benchmark_data_smithi159_80260_object86192:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1f5cc5c:::benchmark_data_smithi159_80260_object77189:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1f7939f:::benchmark_data_smithi159_80260_object83404:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b1fbf95f:::benchmark_data_smithi159_80260_object71749:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2011ac0:::benchmark_data_smithi159_80260_object79139:head# 2024-04-15T18:15:21.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b20348db:::benchmark_data_smithi159_80260_object75865:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b203a6b0:::benchmark_data_smithi159_80260_object85221:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2085f88:::benchmark_data_smithi159_80260_object77051:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b20bcc0f:::benchmark_data_smithi159_80260_object70603:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b20bfbf9:::benchmark_data_smithi159_80260_object80862:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b20d63db:::benchmark_data_smithi159_80260_object79035:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b20e0397:::benchmark_data_smithi159_80260_object84438:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b217319e:::benchmark_data_smithi159_80260_object75401:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21b431c:::benchmark_data_smithi159_80260_object84319:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21c4e13:::benchmark_data_smithi159_80260_object81364:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21d21ea:::benchmark_data_smithi159_80260_object77534:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21d352d:::benchmark_data_smithi159_80260_object72070:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21e83f1:::benchmark_data_smithi159_80260_object88681:head# 2024-04-15T18:15:21.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b21e8674:::benchmark_data_smithi159_80260_object70279:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b22506f5:::benchmark_data_smithi159_80260_object88607:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b225cf52:::benchmark_data_smithi159_80260_object80243:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b22985af:::benchmark_data_smithi159_80260_object77559:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b229a360:::benchmark_data_smithi159_80260_object78902:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b22e3959:::benchmark_data_smithi159_80260_object72939:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b22e9afd:::benchmark_data_smithi159_80260_object79462:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b23044f1:::benchmark_data_smithi159_80260_object80607:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b23094b0:::benchmark_data_smithi159_80260_object84451:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b239b69f:::benchmark_data_smithi159_80260_object87760:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b23d3eb6:::benchmark_data_smithi159_80260_object83539:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b23f24b1:::benchmark_data_smithi159_80260_object75615:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2418b35:::benchmark_data_smithi159_80260_object73371:head# 2024-04-15T18:15:21.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b24284b3:::benchmark_data_smithi159_80260_object70309:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b24990be:::benchmark_data_smithi159_80260_object89058:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2499472:::benchmark_data_smithi159_80260_object84855:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b24b8bbb:::benchmark_data_smithi159_80260_object77232:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b24eb5a1:::benchmark_data_smithi159_80260_object85339:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2562a2d:::benchmark_data_smithi159_80260_object73017:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2578325:::benchmark_data_smithi159_80260_object85368:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b25c1fc9:::benchmark_data_smithi159_80260_object68583:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b25e7da1:::benchmark_data_smithi159_80260_object86580:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b25ee349:::benchmark_data_smithi159_80260_object74807:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b25f04cb:::benchmark_data_smithi159_80260_object86848:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2625fb6:::benchmark_data_smithi159_80260_object85761:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b26ab841:::benchmark_data_smithi159_80260_object78541:head# 2024-04-15T18:15:21.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b26be00f:::benchmark_data_smithi159_80260_object73907:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b26c2d46:::benchmark_data_smithi159_80260_object71327:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b270d4f9:::benchmark_data_smithi159_80260_object82983:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b27177ef:::benchmark_data_smithi159_80260_object71678:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b273bf48:::benchmark_data_smithi159_80260_object88235:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b275bb38:::benchmark_data_smithi159_80260_object86260:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2765a6a:::benchmark_data_smithi159_80260_object72367:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b282d5e8:::benchmark_data_smithi159_80260_object88723:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b286806b:::benchmark_data_smithi159_80260_object68988:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b288940f:::benchmark_data_smithi159_80260_object78842:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2902c99:::benchmark_data_smithi159_80260_object82518:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b290893b:::benchmark_data_smithi159_80260_object82533:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b290a0a4:::benchmark_data_smithi159_80260_object87781:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2915f0c:::benchmark_data_smithi159_80260_object81358:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b291b5aa:::benchmark_data_smithi159_80260_object77001:head# 2024-04-15T18:15:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b291b897:::benchmark_data_smithi159_80260_object85213:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b294d9be:::benchmark_data_smithi159_80260_object77017:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b29823e8:::benchmark_data_smithi159_80260_object88743:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b29e55f1:::benchmark_data_smithi159_80260_object73011:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a3d259:::benchmark_data_smithi159_80260_object68716:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a45e95:::benchmark_data_smithi159_80260_object84225:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a6e949:::benchmark_data_smithi159_80260_object88225:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a7122f:::benchmark_data_smithi159_80260_object80277:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a838fc:::benchmark_data_smithi159_80260_object77039:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a8e13a:::benchmark_data_smithi159_80260_object68576:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a92a1e:::benchmark_data_smithi159_80260_object87386:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2a99896:::benchmark_data_smithi159_80260_object79678:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2ad7b1f:::benchmark_data_smithi159_80260_object75812:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2aeabfc:::benchmark_data_smithi159_80260_object69136:head# 2024-04-15T18:15:21.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2b73eff:::benchmark_data_smithi159_80260_object79195:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2bbcb1c:::benchmark_data_smithi159_80260_object83845:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2be1d5e:::benchmark_data_smithi159_80260_object82395:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2c8f4c9:::benchmark_data_smithi159_80260_object83876:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2cbc138:::benchmark_data_smithi159_80260_object81139:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2cc5a84:::benchmark_data_smithi159_80260_object74385:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d1ecce:::benchmark_data_smithi159_80260_object86153:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d20e03:::benchmark_data_smithi159_80260_object81353:head# 2024-04-15T18:15:21.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d3ae29:::benchmark_data_smithi159_80260_object76972:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d48024:::benchmark_data_smithi159_80260_object70151:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d4899c:::benchmark_data_smithi159_80260_object87161:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2d69744:::benchmark_data_smithi159_80260_object84517:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2e58fa8:::benchmark_data_smithi159_80260_object80554:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2e77d4d:::benchmark_data_smithi159_80260_object83531:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2e9f3a5:::benchmark_data_smithi159_80260_object82820:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2eadfb7:::benchmark_data_smithi159_80260_object84398:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2eec898:::benchmark_data_smithi159_80260_object68672:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2f4bfe4:::benchmark_data_smithi159_80260_object82765:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2f5de32:::benchmark_data_smithi159_80260_object68900:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2fac832:::benchmark_data_smithi159_80260_object71975:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2fbf7da:::benchmark_data_smithi159_80260_object88999:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b2fc5dbc:::benchmark_data_smithi159_80260_object74349:head# 2024-04-15T18:15:21.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b302477b:::benchmark_data_smithi159_80260_object69768:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b307689d:::benchmark_data_smithi159_80260_object68857:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b307ace9:::benchmark_data_smithi159_80260_object73626:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b308005c:::benchmark_data_smithi159_80260_object81876:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b30a214c:::benchmark_data_smithi159_80260_object74193:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b30dfcf3:::benchmark_data_smithi159_80260_object78698:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b311277e:::benchmark_data_smithi159_80260_object78320:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3158ad9:::benchmark_data_smithi159_80260_object80818:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3162bfa:::benchmark_data_smithi159_80260_object79749:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b31f0847:::benchmark_data_smithi159_80260_object80611:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3228714:::benchmark_data_smithi159_80260_object81472:head# 2024-04-15T18:15:21.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b32488a1:::benchmark_data_smithi159_80260_object72414:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3372469:::benchmark_data_smithi159_80260_object71215:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b33a00b4:::benchmark_data_smithi159_80260_object68779:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b33f7abb:::benchmark_data_smithi159_80260_object68991:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3407745:::benchmark_data_smithi159_80260_object74694:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3441301:::benchmark_data_smithi159_80260_object81459:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3441ace:::benchmark_data_smithi159_80260_object89004:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b345916a:::benchmark_data_smithi159_80260_object85206:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b34ab825:::benchmark_data_smithi159_80260_object83178:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b34abd51:::benchmark_data_smithi159_80260_object84351:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b34cadad:::benchmark_data_smithi159_80260_object73418:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b34ea192:::benchmark_data_smithi159_80260_object71200:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b351184b:::benchmark_data_smithi159_80260_object69234:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b351f9db:::benchmark_data_smithi159_80260_object78264:head# 2024-04-15T18:15:21.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3570d83:::benchmark_data_smithi159_80260_object85139:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3574f0f:::benchmark_data_smithi159_80260_object81376:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b35a26b2:::benchmark_data_smithi159_80260_object87151:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b35a9b0d:::benchmark_data_smithi159_80260_object85872:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b35c6c5b:::benchmark_data_smithi159_80260_object72194:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b35d5fc3:::benchmark_data_smithi159_80260_object85631:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b35df4b8:::benchmark_data_smithi159_80260_object80945:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b367820b:::benchmark_data_smithi159_80260_object70475:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b36a0ebe:::benchmark_data_smithi159_80260_object85116:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b36c9f6f:::benchmark_data_smithi159_80260_object88088:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b374547a:::benchmark_data_smithi159_80260_object82967:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3750cd9:::benchmark_data_smithi159_80260_object76740:head# 2024-04-15T18:15:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b378ae0f:::benchmark_data_smithi159_80260_object72917:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b37a7a30:::benchmark_data_smithi159_80260_object86833:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b37ff229:::benchmark_data_smithi159_80260_object87188:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38106f2:::benchmark_data_smithi159_80260_object88749:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b381ee86:::benchmark_data_smithi159_80260_object80552:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38959d1:::benchmark_data_smithi159_80260_object88884:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3896e83:::benchmark_data_smithi159_80260_object72115:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38d6fbf:::benchmark_data_smithi159_80260_object76255:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38e899a:::benchmark_data_smithi159_80260_object87961:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38f6c90:::benchmark_data_smithi159_80260_object70649:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b38f7aac:::benchmark_data_smithi159_80260_object87938:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b390a6a0:::benchmark_data_smithi159_80260_object71763:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b39889c8:::benchmark_data_smithi159_80260_object83412:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b39c0507:::benchmark_data_smithi159_80260_object80643:head# 2024-04-15T18:15:21.767 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3a2f2ce:::benchmark_data_smithi159_80260_object76946:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3a38d98:::benchmark_data_smithi159_80260_object79689:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3a50b0e:::benchmark_data_smithi159_80260_object81410:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3a8d051:::benchmark_data_smithi159_80260_object82825:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3a9d057:::benchmark_data_smithi159_80260_object85887:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3ac3c76:::benchmark_data_smithi159_80260_object83073:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b0ed47:::benchmark_data_smithi159_80260_object80840:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b1f74f:::benchmark_data_smithi159_80260_object84260:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b22861:::benchmark_data_smithi159_80260_object86339:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b3dc30:::benchmark_data_smithi159_80260_object76254:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b3e622:::benchmark_data_smithi159_80260_object79516:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3b76758:::benchmark_data_smithi159_80260_object88606:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3bb751d:::benchmark_data_smithi159_80260_object78136:head# 2024-04-15T18:15:21.768 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3be3ccb:::benchmark_data_smithi159_80260_object72242:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3c10440:::benchmark_data_smithi159_80260_object78044:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3cfe65a:::benchmark_data_smithi159_80260_object83755:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3d3b631:::benchmark_data_smithi159_80260_object88209:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3d4393c:::benchmark_data_smithi159_80260_object83868:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3d56726:::benchmark_data_smithi159_80260_object71809:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3d6245f:::benchmark_data_smithi159_80260_object79343:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3ddc8b2:::benchmark_data_smithi159_80260_object82883:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3df8d2c:::benchmark_data_smithi159_80260_object83898:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e0c714:::benchmark_data_smithi159_80260_object83932:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e12367:::benchmark_data_smithi159_80260_object72748:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e2f805:::benchmark_data_smithi159_80260_object73356:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e35f0a:::benchmark_data_smithi159_80260_object71882:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e427c8:::benchmark_data_smithi159_80260_object71460:head# 2024-04-15T18:15:21.769 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3e63456:::benchmark_data_smithi159_80260_object77692:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3ea0614:::benchmark_data_smithi159_80260_object71211:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3ecb32a:::benchmark_data_smithi159_80260_object80980:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3efc054:::benchmark_data_smithi159_80260_object73120:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3f4ca25:::benchmark_data_smithi159_80260_object81767:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3f5aac5:::benchmark_data_smithi159_80260_object84217:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3f80045:::benchmark_data_smithi159_80260_object72198:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b3fb458a:::benchmark_data_smithi159_80260_object72605:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b40074b7:::benchmark_data_smithi159_80260_object85796:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4093ebd:::benchmark_data_smithi159_80260_object81589:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b409e499:::benchmark_data_smithi159_80260_object81028:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b40adfd0:::benchmark_data_smithi159_80260_object68815:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b416f4d4:::benchmark_data_smithi159_80260_object78218:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4170b6e:::benchmark_data_smithi159_80260_object79215:head# 2024-04-15T18:15:21.770 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4195c41:::benchmark_data_smithi159_80260_object79706:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b41bc71a:::benchmark_data_smithi159_80260_object75477:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b41dd147:::benchmark_data_smithi159_80260_object69035:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b41e5937:::benchmark_data_smithi159_80260_object75397:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b421b9d5:::benchmark_data_smithi159_80260_object88297:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b42580d9:::benchmark_data_smithi159_80260_object81514:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4280ff5:::benchmark_data_smithi159_80260_object68637:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b428140e:::benchmark_data_smithi159_80260_object75621:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b42acf9f:::benchmark_data_smithi159_80260_object77429:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b42be092:::benchmark_data_smithi159_80260_object69865:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b42f3955:::benchmark_data_smithi159_80260_object76563:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4320c68:::benchmark_data_smithi159_80260_object71363:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b43561b9:::benchmark_data_smithi159_80260_object79888:head# 2024-04-15T18:15:21.771 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b437daa7:::benchmark_data_smithi159_80260_object86620:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b43a25b6:::benchmark_data_smithi159_80260_object70889:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b43d91ab:::benchmark_data_smithi159_80260_object87580:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b440fe05:::benchmark_data_smithi159_80260_object82678:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4453328:::benchmark_data_smithi159_80260_object84733:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b447cb12:::benchmark_data_smithi159_80260_object77108:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b44efc4d:::benchmark_data_smithi159_80260_object80947:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b454c1ab:::benchmark_data_smithi159_80260_object77631:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b45a3a57:::benchmark_data_smithi159_80260_object73884:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b45bb2d0:::benchmark_data_smithi159_80260_object73901:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b460b12b:::benchmark_data_smithi159_80260_object69887:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b46649ea:::benchmark_data_smithi159_80260_object76264:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b46be99e:::benchmark_data_smithi159_80260_object79372:head# 2024-04-15T18:15:21.772 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b470b612:::benchmark_data_smithi159_80260_object70890:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b47d5470:::benchmark_data_smithi159_80260_object71208:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b47f18f4:::benchmark_data_smithi159_80260_object73993:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b47f7845:::benchmark_data_smithi159_80260_object83750:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b47f8171:::benchmark_data_smithi159_80260_object73886:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b480a40a:::benchmark_data_smithi159_80260_object86244:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4864e15:::benchmark_data_smithi159_80260_object87795:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b48813a7:::benchmark_data_smithi159_80260_object83200:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4890cc8:::benchmark_data_smithi159_80260_object71498:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4964acd:::benchmark_data_smithi159_80260_object78834:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4983a3c:::benchmark_data_smithi159_80260_object78064:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b49abfaa:::benchmark_data_smithi159_80260_object72938:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b49d6711:::benchmark_data_smithi159_80260_object76057:head# 2024-04-15T18:15:21.773 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b49dd236:::benchmark_data_smithi159_80260_object70064:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b49fcf91:::benchmark_data_smithi159_80260_object82569:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a3cf13:::benchmark_data_smithi159_80260_object84181:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a5a2f2:::benchmark_data_smithi159_80260_object78980:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a69355:::benchmark_data_smithi159_80260_object76965:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a73624:::benchmark_data_smithi159_80260_object89055:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a83190:::benchmark_data_smithi159_80260_object72416:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a87952:::benchmark_data_smithi159_80260_object77436:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4a97a9e:::benchmark_data_smithi159_80260_object85821:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4aee3a0:::benchmark_data_smithi159_80260_object84024:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4b11527:::benchmark_data_smithi159_80260_object83929:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4b232a6:::benchmark_data_smithi159_80260_object85049:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4c48c1c:::benchmark_data_smithi159_80260_object83514:head# 2024-04-15T18:15:21.774 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4c67142:::benchmark_data_smithi159_80260_object77995:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4c72405:::benchmark_data_smithi159_80260_object82895:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4cc87c6:::benchmark_data_smithi159_80260_object86755:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4d71731:::benchmark_data_smithi159_80260_object71786:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4e30680:::benchmark_data_smithi159_80260_object82289:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4e7e7e9:::benchmark_data_smithi159_80260_object74679:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4ee2c45:::benchmark_data_smithi159_80260_object74879:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f04957:::benchmark_data_smithi159_80260_object84546:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f38030:::benchmark_data_smithi159_80260_object76389:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f40fe7:::benchmark_data_smithi159_80260_object75464:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f448c2:::benchmark_data_smithi159_80260_object82363:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f5eb7a:::benchmark_data_smithi159_80260_object83756:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f6cfa9:::benchmark_data_smithi159_80260_object72587:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f804e1:::benchmark_data_smithi159_80260_object71821:head# 2024-04-15T18:15:21.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4f9d86d:::benchmark_data_smithi159_80260_object72326:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4fdde49:::benchmark_data_smithi159_80260_object78421:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b4fed900:::benchmark_data_smithi159_80260_object68763:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b502a161:::benchmark_data_smithi159_80260_object82667:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b50ac17f:::benchmark_data_smithi159_80260_object68711:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b50e0189:::benchmark_data_smithi159_80260_object80376:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b50e5490:::benchmark_data_smithi159_80260_object74150:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b51108d7:::benchmark_data_smithi159_80260_object81612:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b51218e2:::benchmark_data_smithi159_80260_object69437:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b51c3807:::benchmark_data_smithi159_80260_object84627:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b51c6405:::benchmark_data_smithi159_80260_object86807:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5207e8f:::benchmark_data_smithi159_80260_object87157:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b523772b:::benchmark_data_smithi159_80260_object75449:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b52692ff:::benchmark_data_smithi159_80260_object68642:head# 2024-04-15T18:15:21.776 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b528b1a8:::benchmark_data_smithi159_80260_object74873:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b530cfe8:::benchmark_data_smithi159_80260_object71939:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b531c148:::benchmark_data_smithi159_80260_object85602:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b53635c5:::benchmark_data_smithi159_80260_object76090:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b536d05c:::benchmark_data_smithi159_80260_object84775:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5385a9c:::benchmark_data_smithi159_80260_object75202:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b53947b2:::benchmark_data_smithi159_80260_object77582:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b53bdaf5:::benchmark_data_smithi159_80260_object81937:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b53f591a:::benchmark_data_smithi159_80260_object78723:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b54003f1:::benchmark_data_smithi159_80260_object78695:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5452191:::benchmark_data_smithi159_80260_object85534:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b546c36c:::benchmark_data_smithi159_80260_object87936:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b548ca29:::benchmark_data_smithi159_80260_object86510:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b54bd427:::benchmark_data_smithi159_80260_object84464:head# 2024-04-15T18:15:21.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b553bdad:::benchmark_data_smithi159_80260_object87099:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5607902:::benchmark_data_smithi159_80260_object86435:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5608e10:::benchmark_data_smithi159_80260_object74956:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b560f9aa:::benchmark_data_smithi159_80260_object72456:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b564cfce:::benchmark_data_smithi159_80260_object69673:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b568eca1:::benchmark_data_smithi159_80260_object82675:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5696f1c:::benchmark_data_smithi159_80260_object85650:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b575ad28:::benchmark_data_smithi159_80260_object70305:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b579834c:::benchmark_data_smithi159_80260_object74489:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b585d0ae:::benchmark_data_smithi159_80260_object74984:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b58d060d:::benchmark_data_smithi159_80260_object70245:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b58e55c5:::benchmark_data_smithi159_80260_object73257:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b591ae46:::benchmark_data_smithi159_80260_object73840:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b592d7de:::benchmark_data_smithi159_80260_object72891:head# 2024-04-15T18:15:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b595d08c:::benchmark_data_smithi159_80260_object85714:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b597be13:::benchmark_data_smithi159_80260_object84762:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b59a3856:::benchmark_data_smithi159_80260_object81390:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b59fca2e:::benchmark_data_smithi159_80260_object85165:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5a2b56c:::benchmark_data_smithi159_80260_object74269:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5a32bb3:::benchmark_data_smithi159_80260_object84257:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5a5de6b:::benchmark_data_smithi159_80260_object84708:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5a9a553:::benchmark_data_smithi159_80260_object81714:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5aba595:::benchmark_data_smithi159_80260_object76854:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5b1f7eb:::benchmark_data_smithi159_80260_object84595:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5b618cb:::benchmark_data_smithi159_80260_object88179:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5bae9e2:::benchmark_data_smithi159_80260_object68985:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5bc31a4:::benchmark_data_smithi159_80260_object87654:head# 2024-04-15T18:15:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5bcdb53:::benchmark_data_smithi159_80260_object69583:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5cfcd93:::benchmark_data_smithi159_80260_object79397:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5d154f3:::benchmark_data_smithi159_80260_object71083:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5d6ea91:::benchmark_data_smithi159_80260_object74911:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5dbafb1:::benchmark_data_smithi159_80260_object75917:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5e36686:::benchmark_data_smithi159_80260_object69990:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5e43bd0:::benchmark_data_smithi159_80260_object73352:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5e4dfaf:::benchmark_data_smithi159_80260_object83415:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5eab3b4:::benchmark_data_smithi159_80260_object82634:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5ee6553:::benchmark_data_smithi159_80260_object71994:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5f386fb:::benchmark_data_smithi159_80260_object86539:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b5ff4954:::benchmark_data_smithi159_80260_object86931:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b603469e:::benchmark_data_smithi159_80260_object88595:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6049561:::benchmark_data_smithi159_80260_object84528:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6062e9b:::benchmark_data_smithi159_80260_object89041:head# 2024-04-15T18:15:21.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b606444a:::benchmark_data_smithi159_80260_object77370:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6085634:::benchmark_data_smithi159_80260_object79428:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6086ae5:::benchmark_data_smithi159_80260_object79875:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b60972f1:::benchmark_data_smithi159_80260_object78573:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b60e4424:::benchmark_data_smithi159_80260_object79299:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b60ef1b1:::benchmark_data_smithi159_80260_object80860:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6120f42:::benchmark_data_smithi159_80260_object81179:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b614e5ba:::benchmark_data_smithi159_80260_object69028:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6150c6d:::benchmark_data_smithi159_80260_object70218:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b618de02:::benchmark_data_smithi159_80260_object78643:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b61aea4c:::benchmark_data_smithi159_80260_object70887:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b61d093b:::benchmark_data_smithi159_80260_object86574:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b61d1744:::benchmark_data_smithi159_80260_object88875:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b61d8030:::benchmark_data_smithi159_80260_object81601:head# 2024-04-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b61dbeac:::benchmark_data_smithi159_80260_object70932:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6208c04:::benchmark_data_smithi159_80260_object85692:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6214ced:::benchmark_data_smithi159_80260_object75842:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b62216e0:::benchmark_data_smithi159_80260_object86271:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b624e45a:::benchmark_data_smithi159_80260_object79961:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b62615e7:::benchmark_data_smithi159_80260_object85051:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b62a236a:::benchmark_data_smithi159_80260_object71378:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b62ed19d:::benchmark_data_smithi159_80260_object88190:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b62f0787:::benchmark_data_smithi159_80260_object81726:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6306172:::benchmark_data_smithi159_80260_object78253:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b632620e:::benchmark_data_smithi159_80260_object73428:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b63352c3:::benchmark_data_smithi159_80260_object71561:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6344319:::benchmark_data_smithi159_80260_object79012:head# 2024-04-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b634dc3a:::benchmark_data_smithi159_80260_object82110:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b639b73b:::benchmark_data_smithi159_80260_object73373:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b63e8380:::benchmark_data_smithi159_80260_object70195:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6426778:::benchmark_data_smithi159_80260_object69739:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b645ffe5:::benchmark_data_smithi159_80260_object89043:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6472c10:::benchmark_data_smithi159_80260_object88193:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6472e26:::benchmark_data_smithi159_80260_object71152:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b647ed1c:::benchmark_data_smithi159_80260_object73555:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b64975f6:::benchmark_data_smithi159_80260_object87197:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b64f3de5:::benchmark_data_smithi159_80260_object72754:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b64f904b:::benchmark_data_smithi159_80260_object87763:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6531752:::benchmark_data_smithi159_80260_object78563:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b656305d:::benchmark_data_smithi159_80260_object77119:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b658014c:::benchmark_data_smithi159_80260_object87775:head# 2024-04-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b65b1d80:::benchmark_data_smithi159_80260_object83278:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b660a2a8:::benchmark_data_smithi159_80260_object73833:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6621e92:::benchmark_data_smithi159_80260_object86203:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6623f8d:::benchmark_data_smithi159_80260_object73711:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6667ad1:::benchmark_data_smithi159_80260_object81083:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b66b9986:::benchmark_data_smithi159_80260_object70247:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b66f1491:::benchmark_data_smithi159_80260_object85670:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6709d78:::benchmark_data_smithi159_80260_object78277:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6732203:::benchmark_data_smithi159_80260_object71132:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b67375ef:::benchmark_data_smithi159_80260_object76879:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b675f9d6:::benchmark_data_smithi159_80260_object77976:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b67a3fbf:::benchmark_data_smithi159_80260_object80704:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b67a7e5f:::benchmark_data_smithi159_80260_object71446:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b67dcc9f:::benchmark_data_smithi159_80260_object85346:head# 2024-04-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b67e0ff1:::benchmark_data_smithi159_80260_object88320:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b682e62d:::benchmark_data_smithi159_80260_object85960:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6872022:::benchmark_data_smithi159_80260_object89063:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b687532a:::benchmark_data_smithi159_80260_object80499:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b68ea4bf:::benchmark_data_smithi159_80260_object88071:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b691822f:::benchmark_data_smithi159_80260_object87138:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6919121:::benchmark_data_smithi159_80260_object73866:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b696efba:::benchmark_data_smithi159_80260_object73463:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6980615:::benchmark_data_smithi159_80260_object85375:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b69a5825:::benchmark_data_smithi159_80260_object85617:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b69dc834:::benchmark_data_smithi159_80260_object80646:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b69e0948:::benchmark_data_smithi159_80260_object82162:head# 2024-04-15T18:15:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6a2a1ac:::benchmark_data_smithi159_80260_object80155:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6a8114a:::benchmark_data_smithi159_80260_object78095:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6b1aed6:::benchmark_data_smithi159_80260_object85431:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6ba3ed7:::benchmark_data_smithi159_80260_object86294:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6be3b4c:::benchmark_data_smithi159_80260_object86919:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6c298f9:::benchmark_data_smithi159_80260_object72604:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6c688b1:::benchmark_data_smithi159_80260_object68845:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6c7ef8a:::benchmark_data_smithi159_80260_object76904:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6ccbbc1:::benchmark_data_smithi159_80260_object88781:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6d1ed25:::benchmark_data_smithi159_80260_object80702:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6d681a3:::benchmark_data_smithi159_80260_object73877:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6d7b022:::benchmark_data_smithi159_80260_object85804:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6d873fc:::benchmark_data_smithi159_80260_object84299:head# 2024-04-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6d87acf:::benchmark_data_smithi159_80260_object87423:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6da3978:::benchmark_data_smithi159_80260_object72354:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6db77eb:::benchmark_data_smithi159_80260_object81911:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6dd04fd:::benchmark_data_smithi159_80260_object85817:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6df3a21:::benchmark_data_smithi159_80260_object73112:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6e1b45a:::benchmark_data_smithi159_80260_object75575:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6ec065f:::benchmark_data_smithi159_80260_object70961:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6f0abf1:::benchmark_data_smithi159_80260_object81086:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6f9ed59:::benchmark_data_smithi159_80260_object73456:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6fb660b:::benchmark_data_smithi159_80260_object83241:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b6fdf2b5:::benchmark_data_smithi159_80260_object75683:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b707c4bb:::benchmark_data_smithi159_80260_object77167:head# 2024-04-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b70ce5f6:::benchmark_data_smithi159_80260_object78452:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b70d40d7:::benchmark_data_smithi159_80260_object80726:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7109e14:::benchmark_data_smithi159_80260_object68540:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b716c6d8:::benchmark_data_smithi159_80260_object82219:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b71d36f4:::benchmark_data_smithi159_80260_object82637:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b723d6a1:::benchmark_data_smithi159_80260_object70222:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b728a228:::benchmark_data_smithi159_80260_object84857:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b72b4a80:::benchmark_data_smithi159_80260_object88842:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b72e2d18:::benchmark_data_smithi159_80260_object86736:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7304100:::benchmark_data_smithi159_80260_object75541:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b731e344:::benchmark_data_smithi159_80260_object73486:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b734f0be:::benchmark_data_smithi159_80260_object85818:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b73573cc:::benchmark_data_smithi159_80260_object77643:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7375314:::benchmark_data_smithi159_80260_object84370:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b73cc004:::benchmark_data_smithi159_80260_object71528:head# 2024-04-15T18:15:21.788 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b73f256b:::benchmark_data_smithi159_80260_object82258:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7437496:::benchmark_data_smithi159_80260_object85539:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7448d86:::benchmark_data_smithi159_80260_object73590:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b745e0e3:::benchmark_data_smithi159_80260_object74204:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b74934bd:::benchmark_data_smithi159_80260_object73955:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b74aa096:::benchmark_data_smithi159_80260_object73975:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b74bf7f8:::benchmark_data_smithi159_80260_object76799:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7561d0b:::benchmark_data_smithi159_80260_object78254:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b75671ff:::benchmark_data_smithi159_80260_object80365:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7568d81:::benchmark_data_smithi159_80260_object77518:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b75ce383:::benchmark_data_smithi159_80260_object70982:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b75f9070:::benchmark_data_smithi159_80260_object84280:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7617bf1:::benchmark_data_smithi159_80260_object78619:head# 2024-04-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b76237f7:::benchmark_data_smithi159_80260_object71076:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b764215c:::benchmark_data_smithi159_80260_object78088:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7644690:::benchmark_data_smithi159_80260_object69420:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b76476bf:::benchmark_data_smithi159_80260_object73034:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7676f1c:::benchmark_data_smithi159_80260_object70512:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b768c365:::benchmark_data_smithi159_80260_object80494:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b76f2a2e:::benchmark_data_smithi159_80260_object78762:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b770ca89:::benchmark_data_smithi159_80260_object81327:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b772b856:::benchmark_data_smithi159_80260_object70132:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b77430ef:::benchmark_data_smithi159_80260_object68868:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7743cbd:::benchmark_data_smithi159_80260_object88978:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7745f48:::benchmark_data_smithi159_80260_object78804:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7792583:::benchmark_data_smithi159_80260_object87817:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b77aae13:::benchmark_data_smithi159_80260_object78149:head# 2024-04-15T18:15:21.790 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b77b7eb0:::benchmark_data_smithi159_80260_object77181:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b77fe83a:::benchmark_data_smithi159_80260_object84779:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b780a870:::benchmark_data_smithi159_80260_object69745:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b782d107:::benchmark_data_smithi159_80260_object85829:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b783f73d:::benchmark_data_smithi159_80260_object76388:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78468bf:::benchmark_data_smithi159_80260_object71236:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7860447:::benchmark_data_smithi159_80260_object80203:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b787bb0a:::benchmark_data_smithi159_80260_object83970:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78ab854:::benchmark_data_smithi159_80260_object86928:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78ade07:::benchmark_data_smithi159_80260_object72200:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78b14e0:::benchmark_data_smithi159_80260_object69164:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78bc9e1:::benchmark_data_smithi159_80260_object77836:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78d06c9:::benchmark_data_smithi159_80260_object72911:head# 2024-04-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78e3b6f:::benchmark_data_smithi159_80260_object74989:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b78f8a90:::benchmark_data_smithi159_80260_object82961:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b79462a0:::benchmark_data_smithi159_80260_object84602:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7983770:::benchmark_data_smithi159_80260_object83121:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b79e137b:::benchmark_data_smithi159_80260_object71655:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7a028c7:::benchmark_data_smithi159_80260_object88378:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7a5d4a5:::benchmark_data_smithi159_80260_object75836:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7a683e8:::benchmark_data_smithi159_80260_object83366:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7a7017f:::benchmark_data_smithi159_80260_object88713:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7a9150a:::benchmark_data_smithi159_80260_object88420:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7af6bca:::benchmark_data_smithi159_80260_object75356:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7b2b8e1:::benchmark_data_smithi159_80260_object78749:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7bc1f69:::benchmark_data_smithi159_80260_object86776:head# 2024-04-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7bcddce:::benchmark_data_smithi159_80260_object82458:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7c02fef:::benchmark_data_smithi159_80260_object76779:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7c03cd6:::benchmark_data_smithi159_80260_object82061:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7c64794:::benchmark_data_smithi159_80260_object72294:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7c800bd:::benchmark_data_smithi159_80260_object85257:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7c831bf:::benchmark_data_smithi159_80260_object84731:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7ca3761:::benchmark_data_smithi159_80260_object69805:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7ca4fa8:::benchmark_data_smithi159_80260_object85832:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7cc1727:::benchmark_data_smithi159_80260_object68740:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7cf436f:::benchmark_data_smithi159_80260_object73623:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7d259d4:::benchmark_data_smithi159_80260_object80298:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7d33602:::benchmark_data_smithi159_80260_object83003:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7d637f4:::benchmark_data_smithi159_80260_object80257:head# 2024-04-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7dc29d9:::benchmark_data_smithi159_80260_object69733:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7df47c7:::benchmark_data_smithi159_80260_object70296:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e10141:::benchmark_data_smithi159_80260_object83805:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e50b8d:::benchmark_data_smithi159_80260_object80384:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e64b53:::benchmark_data_smithi159_80260_object69966:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e6c990:::benchmark_data_smithi159_80260_object84417:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e740ba:::benchmark_data_smithi159_80260_object88754:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7e7ee7f:::benchmark_data_smithi159_80260_object69332:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7eaff47:::benchmark_data_smithi159_80260_object74999:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7ee4878:::benchmark_data_smithi159_80260_object86557:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7f29612:::benchmark_data_smithi159_80260_object78712:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7f32bf0:::benchmark_data_smithi159_80260_object84549:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7f55952:::benchmark_data_smithi159_80260_object83126:head# 2024-04-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7f5dbb0:::benchmark_data_smithi159_80260_object87394:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7fc0f4a:::benchmark_data_smithi159_80260_object72603:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b7ff0c15:::benchmark_data_smithi159_80260_object73720:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b80049a5:::benchmark_data_smithi159_80260_object69558:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8009e3a:::benchmark_data_smithi159_80260_object73090:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8016844:::benchmark_data_smithi159_80260_object70803:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b801c7a7:::benchmark_data_smithi159_80260_object86732:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8052cbb:::benchmark_data_smithi159_80260_object87003:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b80d978c:::benchmark_data_smithi159_80260_object68931:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8112f37:::benchmark_data_smithi159_80260_object75725:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8154869:::benchmark_data_smithi159_80260_object73973:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b81871d0:::benchmark_data_smithi159_80260_object76638:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8189f58:::benchmark_data_smithi159_80260_object74594:head# 2024-04-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b819fdee:::benchmark_data_smithi159_80260_object69470:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b81e70c2:::benchmark_data_smithi159_80260_object86572:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b823edc7:::benchmark_data_smithi159_80260_object79070:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b82727d4:::benchmark_data_smithi159_80260_object69522:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8292a79:::benchmark_data_smithi159_80260_object78327:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b829931d:::benchmark_data_smithi159_80260_object87094:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b82d7425:::benchmark_data_smithi159_80260_object70031:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b82ea432:::benchmark_data_smithi159_80260_object76051:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b831bd05:::benchmark_data_smithi159_80260_object69622:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b834b151:::benchmark_data_smithi159_80260_object76943:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b83d7b92:::benchmark_data_smithi159_80260_object83914:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b83f6f7a:::benchmark_data_smithi159_80260_object85697:head# 2024-04-15T18:15:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b841d9c0:::benchmark_data_smithi159_80260_object78084:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b847cd38:::benchmark_data_smithi159_80260_object78486:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b848d4dd:::benchmark_data_smithi159_80260_object77346:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b849cebc:::benchmark_data_smithi159_80260_object79224:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b84fc5b3:::benchmark_data_smithi159_80260_object80363:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b850e21c:::benchmark_data_smithi159_80260_object77361:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b85a556b:::benchmark_data_smithi159_80260_object71535:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b85c8fa5:::benchmark_data_smithi159_80260_object86111:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b86079d6:::benchmark_data_smithi159_80260_object89022:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8615e71:::benchmark_last_metadata:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b861bc82:::benchmark_data_smithi159_80260_object84683:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b862817d:::benchmark_data_smithi159_80260_object73507:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b862e0a8:::benchmark_data_smithi159_80260_object69528:head# 2024-04-15T18:15:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8669500:::benchmark_data_smithi159_80260_object81300:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b867b9a3:::benchmark_data_smithi159_80260_object78649:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b867c4d8:::benchmark_data_smithi159_80260_object81132:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b86ba2df:::benchmark_data_smithi159_80260_object88801:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b871c890:::benchmark_data_smithi159_80260_object86706:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b873a5f5:::benchmark_data_smithi159_80260_object75264:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b873d38b:::benchmark_data_smithi159_80260_object88989:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b874fa6c:::benchmark_data_smithi159_80260_object86746:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b877de02:::benchmark_data_smithi159_80260_object82158:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b87a3e1c:::benchmark_data_smithi159_80260_object73996:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b87e9a7c:::benchmark_data_smithi159_80260_object77142:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b88b41ea:::benchmark_data_smithi159_80260_object80044:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8978b23:::benchmark_data_smithi159_80260_object73062:head# 2024-04-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b899c15d:::benchmark_data_smithi159_80260_object74730:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b89c1549:::benchmark_data_smithi159_80260_object73890:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8a3cc9f:::benchmark_data_smithi159_80260_object72896:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8a50ba5:::benchmark_data_smithi159_80260_object73174:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8a8e63d:::benchmark_data_smithi159_80260_object82730:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8aa2f11:::benchmark_data_smithi159_80260_object84168:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8ad7be0:::benchmark_data_smithi159_80260_object87530:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8b0cefb:::benchmark_data_smithi159_80260_object84385:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8ba9dbf:::benchmark_data_smithi159_80260_object80843:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8bb758d:::benchmark_data_smithi159_80260_object86521:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8bc705f:::benchmark_data_smithi159_80260_object85503:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8be59d1:::benchmark_data_smithi159_80260_object68862:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8c05df4:::benchmark_data_smithi159_80260_object79231:head# 2024-04-15T18:15:21.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8c4b800:::benchmark_data_smithi159_80260_object70917:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8c516cd:::benchmark_data_smithi159_80260_object86293:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8c86e3a:::benchmark_data_smithi159_80260_object74878:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8c915e2:::benchmark_data_smithi159_80260_object73974:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d27650:::benchmark_data_smithi159_80260_object75766:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d5a1e9:::benchmark_data_smithi159_80260_object75087:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d73ffa:::benchmark_data_smithi159_80260_object79786:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d785a0:::benchmark_data_smithi159_80260_object83001:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d8098b:::benchmark_data_smithi159_80260_object71161:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8d814b6:::benchmark_data_smithi159_80260_object71588:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8dad803:::benchmark_data_smithi159_80260_object71419:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8dc6814:::benchmark_data_smithi159_80260_object73989:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8dd29d4:::benchmark_data_smithi159_80260_object69595:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8e2e392:::benchmark_data_smithi159_80260_object79024:head# 2024-04-15T18:15:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8e857d7:::benchmark_data_smithi159_80260_object78656:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8eb47fd:::benchmark_data_smithi159_80260_object88330:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8ef6661:::benchmark_data_smithi159_80260_object80148:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8ef9bf0:::benchmark_data_smithi159_80260_object74723:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f00f80:::benchmark_data_smithi159_80260_object84891:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f0f44e:::benchmark_data_smithi159_80260_object69085:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f283b7:::benchmark_data_smithi159_80260_object86105:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f2b6fd:::benchmark_data_smithi159_80260_object85831:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f3000b:::benchmark_data_smithi159_80260_object72108:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f84f26:::benchmark_data_smithi159_80260_object84717:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f8b782:::benchmark_data_smithi159_80260_object81509:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f90fc6:::benchmark_data_smithi159_80260_object72249:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f9166a:::benchmark_data_smithi159_80260_object79942:head# 2024-04-15T18:15:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8f9bcb0:::benchmark_data_smithi159_80260_object71794:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8fdbddd:::benchmark_data_smithi159_80260_object72052:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8fe2fca:::benchmark_data_smithi159_80260_object79290:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b8ffc387:::benchmark_data_smithi159_80260_object81460:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b90953c7:::benchmark_data_smithi159_80260_object79396:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b90cd23d:::benchmark_data_smithi159_80260_object69037:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b90df283:::benchmark_data_smithi159_80260_object83461:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b91289a6:::benchmark_data_smithi159_80260_object79769:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b91958b3:::benchmark_data_smithi159_80260_object83463:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b919ccdd:::benchmark_data_smithi159_80260_object74737:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b91ba3fc:::benchmark_data_smithi159_80260_object69438:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b91f40f1:::benchmark_data_smithi159_80260_object83250:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9219bb6:::benchmark_data_smithi159_80260_object86612:head# 2024-04-15T18:15:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b926d346:::benchmark_data_smithi159_80260_object68768:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9332f7b:::benchmark_data_smithi159_80260_object70182:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b933525b:::benchmark_data_smithi159_80260_object81497:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b93dcbef:::benchmark_data_smithi159_80260_object88917:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b93f0b43:::benchmark_data_smithi159_80260_object82914:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9438126:::benchmark_data_smithi159_80260_object86713:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b943aaaf:::benchmark_data_smithi159_80260_object83080:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b944bb40:::benchmark_data_smithi159_80260_object72175:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b94514b9:::benchmark_data_smithi159_80260_object78116:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b94c3e75:::benchmark_data_smithi159_80260_object85971:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9532218:::benchmark_data_smithi159_80260_object78091:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b953900e:::benchmark_data_smithi159_80260_object84556:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9588af0:::benchmark_data_smithi159_80260_object75217:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b958e866:::benchmark_data_smithi159_80260_object73598:head# 2024-04-15T18:15:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b95bb733:::benchmark_data_smithi159_80260_object75980:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b95d74d8:::benchmark_data_smithi159_80260_object86624:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b966e5ad:::benchmark_data_smithi159_80260_object71017:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b968f7a1:::benchmark_data_smithi159_80260_object76207:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b96b3ddc:::benchmark_data_smithi159_80260_object84104:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b96c958e:::benchmark_data_smithi159_80260_object72100:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b96d084e:::benchmark_data_smithi159_80260_object69690:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b96ef195:::benchmark_data_smithi159_80260_object71589:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b96f947e:::benchmark_data_smithi159_80260_object84713:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9704ded:::benchmark_data_smithi159_80260_object79859:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9724ae5:::benchmark_data_smithi159_80260_object85527:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b97ee1ff:::benchmark_data_smithi159_80260_object81013:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9802563:::benchmark_data_smithi159_80260_object81791:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98466eb:::benchmark_data_smithi159_80260_object80439:head# 2024-04-15T18:15:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b984e24d:::benchmark_data_smithi159_80260_object68853:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b985d00f:::benchmark_data_smithi159_80260_object74446:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98905ad:::benchmark_data_smithi159_80260_object85014:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98affd3:::benchmark_data_smithi159_80260_object70465:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98e8929:::benchmark_data_smithi159_80260_object88389:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98ea6b7:::benchmark_data_smithi159_80260_object70822:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b98f04ec:::benchmark_data_smithi159_80260_object85202:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b99d0a88:::benchmark_data_smithi159_80260_object76071:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b99f291d:::benchmark_data_smithi159_80260_object78419:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9a285d4:::benchmark_data_smithi159_80260_object87504:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9a7f611:::benchmark_data_smithi159_80260_object79715:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9a8c009:::benchmark_data_smithi159_80260_object76274:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9ade017:::benchmark_data_smithi159_80260_object75080:head# 2024-04-15T18:15:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9afd176:::benchmark_data_smithi159_80260_object80307:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9b56e27:::benchmark_data_smithi159_80260_object76236:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9b68424:::benchmark_data_smithi159_80260_object76835:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9b912f4:::benchmark_data_smithi159_80260_object83257:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9be45a8:::benchmark_data_smithi159_80260_object72904:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9c626b0:::benchmark_data_smithi159_80260_object78503:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9c63f2c:::benchmark_data_smithi159_80260_object81632:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9c70ecc:::benchmark_data_smithi159_80260_object80485:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9c9230f:::benchmark_data_smithi159_80260_object81317:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9cc891d:::benchmark_data_smithi159_80260_object86044:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9ccc506:::benchmark_data_smithi159_80260_object76455:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9cd876a:::benchmark_data_smithi159_80260_object68591:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9cdaba0:::benchmark_data_smithi159_80260_object84642:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9d160a2:::benchmark_data_smithi159_80260_object79340:head# 2024-04-15T18:15:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9d1a79a:::benchmark_data_smithi159_80260_object78546:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9dc5dd2:::benchmark_data_smithi159_80260_object86663:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9dca4df:::benchmark_data_smithi159_80260_object87228:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9e46b8c:::benchmark_data_smithi159_80260_object75196:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9e73c3d:::benchmark_data_smithi159_80260_object84928:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9e8374f:::benchmark_data_smithi159_80260_object82099:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9ed729f:::benchmark_data_smithi159_80260_object84107:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9f38bab:::benchmark_data_smithi159_80260_object77345:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9fb20a7:::benchmark_data_smithi159_80260_object68747:head# 2024-04-15T18:15:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9fb89a6:::benchmark_data_smithi159_80260_object70344:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9fd7e5a:::benchmark_data_smithi159_80260_object75279:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:b9ff6c20:::benchmark_data_smithi159_80260_object73113:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba059fae:::benchmark_data_smithi159_80260_object80654:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba0df2ef:::benchmark_data_smithi159_80260_object82668:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba156314:::benchmark_data_smithi159_80260_object88982:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba219d9f:::benchmark_data_smithi159_80260_object75782:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba2787cb:::benchmark_data_smithi159_80260_object86575:head# 2024-04-15T18:15:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba2f36b4:::benchmark_data_smithi159_80260_object84141:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba304a2e:::benchmark_data_smithi159_80260_object71274:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba32db12:::benchmark_data_smithi159_80260_object84230:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba351ecc:::benchmark_data_smithi159_80260_object78774:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba377daa:::benchmark_data_smithi159_80260_object72418:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba3807a9:::benchmark_data_smithi159_80260_object69336:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba38e0ac:::benchmark_data_smithi159_80260_object88153:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba3ad026:::benchmark_data_smithi159_80260_object74259:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba3b14a4:::benchmark_data_smithi159_80260_object76443:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba457244:::benchmark_data_smithi159_80260_object71310:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba4b9993:::benchmark_data_smithi159_80260_object75103:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba4fb327:::benchmark_data_smithi159_80260_object83197:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba4fc8f9:::benchmark_data_smithi159_80260_object70013:head# 2024-04-15T18:15:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba51151a:::benchmark_data_smithi159_80260_object78673:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba51fb03:::benchmark_data_smithi159_80260_object82218:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba55c2d1:::benchmark_data_smithi159_80260_object86132:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba566499:::benchmark_data_smithi159_80260_object87472:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba58b342:::benchmark_data_smithi159_80260_object74232:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba590cd5:::benchmark_data_smithi159_80260_object84951:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba5a2b6b:::benchmark_data_smithi159_80260_object86937:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba601434:::benchmark_data_smithi159_80260_object87034:head# 2024-04-15T18:15:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba6420b2:::benchmark_data_smithi159_80260_object80151:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba67e8d4:::benchmark_data_smithi159_80260_object86941:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba68c0e7:::benchmark_data_smithi159_80260_object76481:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba6acd7d:::benchmark_data_smithi159_80260_object86973:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba6d4664:::benchmark_data_smithi159_80260_object86179:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba74ecae:::benchmark_data_smithi159_80260_object82107:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba79ecf8:::benchmark_data_smithi159_80260_object85885:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba7a398e:::benchmark_data_smithi159_80260_object82375:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba7e7cd1:::benchmark_data_smithi159_80260_object76923:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba7ef82a:::benchmark_data_smithi159_80260_object76049:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba7f63cf:::benchmark_data_smithi159_80260_object82629:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba7f9e18:::benchmark_data_smithi159_80260_object71594:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba8754aa:::benchmark_data_smithi159_80260_object82457:head# 2024-04-15T18:15:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba895351:::benchmark_data_smithi159_80260_object78989:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba8c3d67:::benchmark_data_smithi159_80260_object80793:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba94c056:::benchmark_data_smithi159_80260_object69157:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba980c4e:::benchmark_data_smithi159_80260_object75432:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba9b3355:::benchmark_data_smithi159_80260_object70443:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:ba9f04ae:::benchmark_data_smithi159_80260_object75748:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa051ec:::benchmark_data_smithi159_80260_object82250:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa0646d:::benchmark_data_smithi159_80260_object82127:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa192b9:::benchmark_data_smithi159_80260_object71666:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa33656:::benchmark_data_smithi159_80260_object81828:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa3a8ba:::benchmark_data_smithi159_80260_object81445:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baa6bc5e:::benchmark_data_smithi159_80260_object79724:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baad60a6:::benchmark_data_smithi159_80260_object76945:head# 2024-04-15T18:15:21.812 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bab4db6c:::benchmark_data_smithi159_80260_object73397:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bab5d921:::benchmark_data_smithi159_80260_object74637:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bab837f4:::benchmark_data_smithi159_80260_object77426:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:babcff57:::benchmark_data_smithi159_80260_object81956:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:babd1127:::benchmark_data_smithi159_80260_object81284:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:babfb117:::benchmark_data_smithi159_80260_object68881:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bac12417:::benchmark_data_smithi159_80260_object81770:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bac5e6bf:::benchmark_data_smithi159_80260_object78760:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bac8fdd7:::benchmark_data_smithi159_80260_object76605:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baca5bba:::benchmark_data_smithi159_80260_object72958:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bacbb841:::benchmark_data_smithi159_80260_object75434:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bacc5a26:::benchmark_data_smithi159_80260_object70112:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bace419c:::benchmark_data_smithi159_80260_object80236:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bad6622c:::benchmark_data_smithi159_80260_object72454:head# 2024-04-15T18:15:21.813 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bad95daa:::benchmark_data_smithi159_80260_object86060:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:badaba57:::benchmark_data_smithi159_80260_object70213:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:badcdcc8:::benchmark_data_smithi159_80260_object73553:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:badcf7f1:::benchmark_data_smithi159_80260_object87301:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bae18762:::benchmark_data_smithi159_80260_object75461:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baeba2b2:::benchmark_data_smithi159_80260_object85703:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baf3f90b:::benchmark_data_smithi159_80260_object88524:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:baf96371:::benchmark_data_smithi159_80260_object81286:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bafb47f6:::benchmark_data_smithi159_80260_object73052:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bafd5026:::benchmark_data_smithi159_80260_object81788:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb028b06:::benchmark_data_smithi159_80260_object87791:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb0389b3:::benchmark_data_smithi159_80260_object79997:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb0390d0:::benchmark_data_smithi159_80260_object84266:head# 2024-04-15T18:15:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb074b80:::benchmark_data_smithi159_80260_object69192:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb0aa2a8:::benchmark_data_smithi159_80260_object68984:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb0bd15b:::benchmark_data_smithi159_80260_object85404:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1143b2:::benchmark_data_smithi159_80260_object85179:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb162d4a:::benchmark_data_smithi159_80260_object83292:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1783f6:::benchmark_data_smithi159_80260_object75120:head# 2024-04-15T18:15:21.815 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1beb1a:::benchmark_data_smithi159_80260_object71756:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1caf82:::benchmark_data_smithi159_80260_object80065:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1cd39f:::benchmark_data_smithi159_80260_object77247:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1d93a6:::benchmark_data_smithi159_80260_object70436:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb1f3511:::benchmark_data_smithi159_80260_object83916:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb2b56ce:::benchmark_data_smithi159_80260_object75499:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb2dc771:::benchmark_data_smithi159_80260_object86131:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb33fe22:::benchmark_data_smithi159_80260_object87762:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb3491a8:::benchmark_data_smithi159_80260_object74086:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb34e4a7:::benchmark_data_smithi159_80260_object70095:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb35a816:::benchmark_data_smithi159_80260_object77652:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb374520:::benchmark_data_smithi159_80260_object87700:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb379763:::benchmark_data_smithi159_80260_object78958:head# 2024-04-15T18:15:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb4142f0:::benchmark_data_smithi159_80260_object73253:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb4209a6:::benchmark_data_smithi159_80260_object76676:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb4c026a:::benchmark_data_smithi159_80260_object86999:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb4cf232:::benchmark_data_smithi159_80260_object88248:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb5348f5:::benchmark_data_smithi159_80260_object85911:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb58eeac:::benchmark_data_smithi159_80260_object83434:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb5b1f40:::benchmark_data_smithi159_80260_object82563:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb5fa63f:::benchmark_data_smithi159_80260_object84433:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb6368d1:::benchmark_data_smithi159_80260_object68838:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb66fae9:::benchmark_data_smithi159_80260_object76513:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb72e3dd:::benchmark_data_smithi159_80260_object73027:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb73ff45:::benchmark_data_smithi159_80260_object83334:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb7442aa:::benchmark_data_smithi159_80260_object73850:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb7e2ac0:::benchmark_data_smithi159_80260_object76824:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb803626:::benchmark_data_smithi159_80260_object73706:head# 2024-04-15T18:15:21.817 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb81945e:::benchmark_data_smithi159_80260_object69589:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb81b0d6:::benchmark_data_smithi159_80260_object69078:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb84c1f6:::benchmark_data_smithi159_80260_object69757:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb858f7a:::benchmark_data_smithi159_80260_object68830:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb89ee80:::benchmark_data_smithi159_80260_object78947:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb8c585b:::benchmark_data_smithi159_80260_object71741:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb8ce154:::benchmark_data_smithi159_80260_object77105:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb8ed53c:::benchmark_data_smithi159_80260_object70628:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb913c8e:::benchmark_data_smithi159_80260_object71876:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb93333a:::benchmark_data_smithi159_80260_object88814:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb978b6b:::benchmark_data_smithi159_80260_object69044:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb9b8a1c:::benchmark_data_smithi159_80260_object69018:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb9cfc23:::benchmark_data_smithi159_80260_object84719:head# 2024-04-15T18:15:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bb9f1acf:::benchmark_data_smithi159_80260_object88714:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bba4a332:::benchmark_data_smithi159_80260_object87295:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bba5d18a:::benchmark_data_smithi159_80260_object74610:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbaa7714:::benchmark_data_smithi159_80260_object78115:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbabda09:::benchmark_data_smithi159_80260_object75975:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbb34e33:::benchmark_data_smithi159_80260_object87774:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbbb8eb9:::benchmark_data_smithi159_80260_object81326:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbbd485b:::benchmark_data_smithi159_80260_object79524:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbc6f446:::benchmark_data_smithi159_80260_object75204:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbc9c83c:::benchmark_data_smithi159_80260_object88655:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbd52326:::benchmark_data_smithi159_80260_object73446:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbd77c78:::benchmark_data_smithi159_80260_object83477:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbd92ff4:::benchmark_data_smithi159_80260_object77298:head# 2024-04-15T18:15:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbde07cb:::benchmark_data_smithi159_80260_object70740:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbe55c5c:::benchmark_data_smithi159_80260_object72147:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbe85188:::benchmark_data_smithi159_80260_object74212:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbef86ef:::benchmark_data_smithi159_80260_object76576:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbf0b88d:::benchmark_data_smithi159_80260_object84183:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbfb7b4f:::benchmark_data_smithi159_80260_object83405:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbfbd0d9:::benchmark_data_smithi159_80260_object69736:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bbff97f1:::benchmark_data_smithi159_80260_object72770:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc0203d6:::benchmark_data_smithi159_80260_object74629:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc02d3a6:::benchmark_data_smithi159_80260_object69145:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc03fe4f:::benchmark_data_smithi159_80260_object78739:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc053aa1:::benchmark_data_smithi159_80260_object81567:head# 2024-04-15T18:15:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc0b398f:::benchmark_data_smithi159_80260_object70679:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc0d1d63:::benchmark_data_smithi159_80260_object87668:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc1077a6:::benchmark_data_smithi159_80260_object81403:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc10f2bd:::benchmark_data_smithi159_80260_object71908:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc13df20:::benchmark_data_smithi159_80260_object83247:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc142963:::benchmark_data_smithi159_80260_object81481:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc16603b:::benchmark_data_smithi159_80260_object79642:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc1874b0:::benchmark_data_smithi159_80260_object69608:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc21fa53:::benchmark_data_smithi159_80260_object82612:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc2273d0:::benchmark_data_smithi159_80260_object82735:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc246960:::benchmark_data_smithi159_80260_object87542:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc246ba3:::benchmark_data_smithi159_80260_object83240:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc252ef0:::benchmark_data_smithi159_80260_object70078:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc271577:::benchmark_data_smithi159_80260_object74490:head# 2024-04-15T18:15:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc2d7fb4:::benchmark_data_smithi159_80260_object81490:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc323226:::benchmark_data_smithi159_80260_object86544:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc34e5f8:::benchmark_data_smithi159_80260_object74197:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc365b31:::benchmark_data_smithi159_80260_object87044:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc3766cb:::benchmark_data_smithi159_80260_object73467:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc38d04a:::benchmark_data_smithi159_80260_object78812:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc4627d7:::benchmark_data_smithi159_80260_object72502:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc4702f6:::benchmark_data_smithi159_80260_object75027:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc4b42e3:::benchmark_data_smithi159_80260_object73144:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc6203d4:::benchmark_data_smithi159_80260_object86951:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc65a41d:::benchmark_data_smithi159_80260_object82297:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc6a78b4:::benchmark_data_smithi159_80260_object81114:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc6b347e:::benchmark_data_smithi159_80260_object82707:head# 2024-04-15T18:15:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc6be3a0:::benchmark_data_smithi159_80260_object83983:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc6c23ae:::benchmark_data_smithi159_80260_object86623:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc7360c6:::benchmark_data_smithi159_80260_object74618:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc757243:::benchmark_data_smithi159_80260_object84324:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc76f4fe:::benchmark_data_smithi159_80260_object73965:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc7b5536:::benchmark_data_smithi159_80260_object78040:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc7ccaa0:::benchmark_data_smithi159_80260_object72997:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc7fc5ce:::benchmark_data_smithi159_80260_object78123:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc862b6c:::benchmark_data_smithi159_80260_object84359:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc8a46b7:::benchmark_data_smithi159_80260_object72618:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc8d22f9:::benchmark_data_smithi159_80260_object85052:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc93e914:::benchmark_data_smithi159_80260_object80763:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc94833c:::benchmark_data_smithi159_80260_object84622:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc96eec0:::benchmark_data_smithi159_80260_object80046:head# 2024-04-15T18:15:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bc9c8c84:::benchmark_data_smithi159_80260_object83172:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca0156a:::benchmark_data_smithi159_80260_object69638:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca04637:::benchmark_data_smithi159_80260_object76148:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca08583:::benchmark_data_smithi159_80260_object78415:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca417b2:::benchmark_data_smithi159_80260_object85777:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca5a111:::benchmark_data_smithi159_80260_object84363:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca87dfe:::benchmark_data_smithi159_80260_object72956:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca8da64:::benchmark_data_smithi159_80260_object73914:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bca9828e:::benchmark_data_smithi159_80260_object86529:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcafa1eb:::benchmark_data_smithi159_80260_object81566:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcb1a1fb:::benchmark_data_smithi159_80260_object74966:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcc685cc:::benchmark_data_smithi159_80260_object75260:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcce8972:::benchmark_data_smithi159_80260_object80082:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcd1ca32:::benchmark_data_smithi159_80260_object80918:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcd63aad:::benchmark_data_smithi159_80260_object69218:head# 2024-04-15T18:15:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcd92d1b:::benchmark_data_smithi159_80260_object87073:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcdb2219:::benchmark_data_smithi159_80260_object72138:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcdbd516:::benchmark_data_smithi159_80260_object87847:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcdd6d09:::benchmark_data_smithi159_80260_object87702:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcdead63:::benchmark_data_smithi159_80260_object69702:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcebe30c:::benchmark_data_smithi159_80260_object72460:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcec25b5:::benchmark_data_smithi159_80260_object81636:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcec31c9:::benchmark_data_smithi159_80260_object88546:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcf30cb9:::benchmark_data_smithi159_80260_object71805:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcf4979d:::benchmark_data_smithi159_80260_object77153:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcf4f760:::benchmark_data_smithi159_80260_object81293:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bcffbf36:::benchmark_data_smithi159_80260_object80256:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd09046e:::benchmark_data_smithi159_80260_object71732:head# 2024-04-15T18:15:21.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd0d375e:::benchmark_data_smithi159_80260_object85123:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd0e1aef:::benchmark_data_smithi159_80260_object68879:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd12de9c:::benchmark_data_smithi159_80260_object88941:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd145b37:::benchmark_data_smithi159_80260_object83188:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd174a59:::benchmark_data_smithi159_80260_object83743:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd1e9144:::benchmark_data_smithi159_80260_object76632:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd1eb74c:::benchmark_data_smithi159_80260_object84759:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd27da8d:::benchmark_data_smithi159_80260_object74001:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd2de512:::benchmark_data_smithi159_80260_object73301:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd30fa3a:::benchmark_data_smithi159_80260_object71693:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd358301:::benchmark_data_smithi159_80260_object70028:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd35e7fb:::benchmark_data_smithi159_80260_object76446:head# 2024-04-15T18:15:21.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd374178:::benchmark_data_smithi159_80260_object82361:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd40f475:::benchmark_data_smithi159_80260_object83611:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd48473a:::benchmark_data_smithi159_80260_object73483:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd4a10ba:::benchmark_data_smithi159_80260_object70439:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd4a3662:::benchmark_data_smithi159_80260_object84118:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd502849:::benchmark_data_smithi159_80260_object68758:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd52d844:::benchmark_data_smithi159_80260_object82396:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd552d25:::benchmark_data_smithi159_80260_object85063:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd5574c4:::benchmark_data_smithi159_80260_object80248:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd59a239:::benchmark_data_smithi159_80260_object68608:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd5d4753:::benchmark_data_smithi159_80260_object71615:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd65b372:::benchmark_data_smithi159_80260_object69670:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd694fcd:::benchmark_data_smithi159_80260_object72957:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd6c6b08:::benchmark_data_smithi159_80260_object86753:head# 2024-04-15T18:15:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd771cd4:::benchmark_data_smithi159_80260_object83632:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd809bac:::benchmark_data_smithi159_80260_object81581:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd886c5a:::benchmark_data_smithi159_80260_object84404:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd88b55f:::benchmark_data_smithi159_80260_object69220:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd89c43c:::benchmark_data_smithi159_80260_object78660:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd8eb780:::benchmark_data_smithi159_80260_object76844:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd92468f:::benchmark_data_smithi159_80260_object82570:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd928aa1:::benchmark_data_smithi159_80260_object76210:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd93a194:::benchmark_data_smithi159_80260_object77271:head# 2024-04-15T18:15:21.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd9b4885:::benchmark_data_smithi159_80260_object76137:head# 2024-04-15T18:15:22.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:15:22.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:22 smithi159 ceph-mon[45336]: pgmap v551: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 119 op/s 2024-04-15T18:15:22.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:22 smithi049 ceph-mon[40087]: pgmap v551: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 119 op/s 2024-04-15T18:15:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:22 smithi082 ceph-mon[44206]: pgmap v551: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 119 op/s 2024-04-15T18:15:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:23.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:23 smithi049 ceph-mon[40087]: osdmap e196: 12 total, 11 up, 11 in 2024-04-15T18:15:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:23 smithi082 ceph-mon[44206]: osdmap e196: 12 total, 11 up, 11 in 2024-04-15T18:15:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:23 smithi159 ceph-mon[45336]: osdmap e196: 12 total, 11 up, 11 in 2024-04-15T18:15:23.751 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:23.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:23.775 INFO:teuthology.orchestra.run.smithi049.stderr:Read #2:bd9ef735: 2024-04-15T18:15:23.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.3 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 2.d 2024-04-15T18:15:23.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:23.888 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:23.909 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:23.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:24.013 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:24.046 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:24.069 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:24.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:24.173 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:24.204 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:24.222 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.3/config 2024-04-15T18:15:24.224 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:24.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:24.329 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:24.358 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:24 smithi159 ceph-mon[45336]: pgmap v553: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 31 op/s 2024-04-15T18:15:24.361 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:24.384 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:24.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:24.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:24 smithi049 ceph-mon[40087]: pgmap v553: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 31 op/s 2024-04-15T18:15:24.487 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:24.516 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:24.535 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:24.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:24.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:24 smithi082 ceph-mon[44206]: pgmap v553: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail; 0 B/s wr, 31 op/s 2024-04-15T18:15:24.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:24.671 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:25.029 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:25.030 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:25.124 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:25.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:25.179 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' removed 2024-04-15T18:15:25.189 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:15:25.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:25.271 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:25.303 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:25.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:25.407 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:25.432 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:25.432 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:25.432 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:25 smithi159 ceph-mon[45336]: osdmap e197: 12 total, 11 up, 11 in 2024-04-15T18:15:25.432 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:15:25.432 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:15:25.433 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:25.451 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:25.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:25 smithi049 ceph-mon[40087]: osdmap e197: 12 total, 11 up, 11 in 2024-04-15T18:15:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:15:25.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:15:25.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:25 smithi082 ceph-mon[44206]: osdmap e197: 12 total, 11 up, 11 in 2024-04-15T18:15:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:15:25.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:15:25.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:25.582 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:25.601 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:25.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:25.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:25.732 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:25.750 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:25.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:25.754 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:15:25.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:25.883 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.082 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:26.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:26.215 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.254 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:26.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:26 smithi049 ceph-mon[40087]: pgmap v555: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:26.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:26 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:26.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:15:26.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:15:26.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:26 smithi049 ceph-mon[40087]: osdmap e198: 12 total, 11 up, 11 in 2024-04-15T18:15:26.358 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:26.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.406 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:26.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:26.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:26 smithi082 ceph-mon[44206]: pgmap v555: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:26.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:26 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:26.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:15:26.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:15:26.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:26 smithi082 ceph-mon[44206]: osdmap e198: 12 total, 11 up, 11 in 2024-04-15T18:15:26.536 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.555 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:26 smithi159 ceph-mon[45336]: pgmap v555: 17 pgs: 17 active+clean; 577 KiB data, 812 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:26 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2442631037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:15:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3001296131' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:15:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:26 smithi159 ceph-mon[45336]: osdmap e198: 12 total, 11 up, 11 in 2024-04-15T18:15:26.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:26.694 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.715 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:26.734 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:15:26.735 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:15:26.735 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-15T18:15:26.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-04-15T18:15:26.820 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:26.848 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:26.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:26.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:26.972 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:27.004 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.025 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:27.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:27.159 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:27.285 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:27.317 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.338 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:27.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:27.472 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.493 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:27.597 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:27.625 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.644 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:27.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:27.780 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:27.801 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:27.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:27.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:27.935 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:28.033 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:28.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:28.137 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:28.169 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:28.191 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:28.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:28.295 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:28.327 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:28.348 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:28.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:28.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:28 smithi049 ceph-mon[40087]: pgmap v557: 1 pgs: 1 active+clean; 577 KiB data, 847 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:28.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:28.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:28 smithi082 ceph-mon[44206]: pgmap v557: 1 pgs: 1 active+clean; 577 KiB data, 847 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:28.479 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:28.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:28.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:28.630 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:28.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:28 smithi159 ceph-mon[45336]: pgmap v557: 1 pgs: 1 active+clean; 577 KiB data, 847 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:29.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:15:29.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:15:29.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:15:29.810 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:29.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:29.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:29.948 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:29.970 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:29.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:30.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:30.086 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:30.109 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:30.208 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:30.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:30.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:15:30.304+0000 7ff6db59e700 0 --2- 172.21.15.49:0/2654716832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6dc113cb0 0x7ff6dc1160a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:15:30.326 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:30.354 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-04-15T18:15:30.354 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-04-15T18:15:30.354 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 2.d_head removing 2.d 2024-04-15T18:15:30.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:30.375 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-04-15T18:15:30.379 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:30.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:30 smithi159 ceph-mon[45336]: pgmap v558: 1 pgs: 1 active+clean; 577 KiB data, 865 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:15:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:15:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:15:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:30 smithi049 ceph-mon[40087]: pgmap v558: 1 pgs: 1 active+clean; 577 KiB data, 865 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:15:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:15:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:15:30.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:30.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:30 smithi082 ceph-mon[44206]: pgmap v558: 1 pgs: 1 active+clean; 577 KiB data, 865 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:30.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:15:30.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:15:30.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:15:30.514 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:30.534 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:30.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:30.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:30.669 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:30.689 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:30.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:30.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:30.825 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:30.846 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:30.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:30.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:30.982 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.003 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:31.106 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:31.138 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.160 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:31.173 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:15:31.197 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:15:31.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:15:31.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:31.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2938946005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:31.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:31.295 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.316 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:31.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:15:31.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2938946005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:31.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:31.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:31.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:15:31.452 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.469 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:15:31.469 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:15:31.473 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:31.489 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:15:31.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:15:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2938946005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:31.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:31.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:15:31.567 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:15:31.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:31.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:15:31.605 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.623 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:15:31.625 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:31.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:15:31.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:15:31.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:15:31.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:31.761 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.782 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:31.887 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:31.916 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:31.937 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:31.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:31.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.3 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.d.3 2024-04-15T18:15:32.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:32.071 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.092 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:32.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:32.224 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.242 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.3/config 2024-04-15T18:15:32.243 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:32.347 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:32.377 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.396 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: pgmap v559: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: pgmap v559: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.531 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.553 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:32.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:32.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: pgmap v559: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:32.686 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.762 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:32.867 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:32.899 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:32.920 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:32.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:33.024 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:33.054 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.074 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:33.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:33.207 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.226 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:33.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:33.357 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.376 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.584 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:33.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:33.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.743 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:33.857 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:33.889 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:33.910 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:33.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:34.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:34.045 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.065 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:34.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:34.199 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.219 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:15:34.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:15:34.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:34.351 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:34 smithi159 ceph-mon[45336]: pgmap v560: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:34.353 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.373 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:34.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:34.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:34 smithi049 ceph-mon[40087]: pgmap v560: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:34.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:34 smithi082 ceph-mon[44206]: pgmap v560: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:34.509 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.530 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:34.634 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:34.663 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.682 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:34.753 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:15:34.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:34.814 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:34.912 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:35.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:35.052 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:35.072 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:35.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:35.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:35.208 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:35.230 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:35.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:35.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:35.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:15:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.113 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:36.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:36.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:15:36.301 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:36.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:36.404 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:36.422 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:36.437 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.458 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:36.460 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:36 smithi159 ceph-mon[45336]: pgmap v561: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:36.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:36 smithi049 ceph-mon[40087]: pgmap v561: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:36.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:36 smithi082 ceph-mon[44206]: pgmap v561: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:36.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:36.591 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.612 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:36.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:36.638 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e177 pg_nums {1={19=1},2={114=16}} deleted_pools ) 2024-04-15T18:15:36.638 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 2.d 2024-04-15T18:15:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0018e1d:::benchmark_data_smithi159_80260_object85038:head# 2024-04-15T18:15:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.650 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0064ed6:::benchmark_data_smithi159_80260_object69177:head# 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b00753ad:::benchmark_data_smithi159_80260_object81077:head# 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b008fbe7:::benchmark_data_smithi159_80260_object79761:head# 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b01448d4:::benchmark_data_smithi159_80260_object68972:head# 2024-04-15T18:15:36.651 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b01b34de:::benchmark_data_smithi159_80260_object86470:head# 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b01fa662:::benchmark_data_smithi159_80260_object82254:head# 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b022bda3:::benchmark_data_smithi159_80260_object87448:head# 2024-04-15T18:15:36.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0282be4:::benchmark_data_smithi159_80260_object80448:head# 2024-04-15T18:15:36.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b02af4ee:::benchmark_data_smithi159_80260_object72237:head# 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b031ef8b:::benchmark_data_smithi159_80260_object71584:head# 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b032be7f:::benchmark_data_smithi159_80260_object70616:head# 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b035b7eb:::benchmark_data_smithi159_80260_object77595:head# 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b038c14d:::benchmark_data_smithi159_80260_object73805:head# 2024-04-15T18:15:36.662 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b039c4b4:::benchmark_data_smithi159_80260_object81154:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b03da32a:::benchmark_data_smithi159_80260_object88985:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b03f0dcb:::benchmark_data_smithi159_80260_object76660:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b04c72d8:::benchmark_data_smithi159_80260_object82833:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b04f8586:::benchmark_data_smithi159_80260_object83811:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b052c116:::benchmark_data_smithi159_80260_object83119:head# 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.663 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0543daa:::benchmark_data_smithi159_80260_object86104:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0577563:::benchmark_data_smithi159_80260_object79034:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0581559:::benchmark_data_smithi159_80260_object82149:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b05b4508:::benchmark_data_smithi159_80260_object75693:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b05f1cb4:::benchmark_data_smithi159_80260_object78161:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b06860a5:::benchmark_data_smithi159_80260_object77578:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b069a512:::benchmark_data_smithi159_80260_object83468:head# 2024-04-15T18:15:36.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b06b53d1:::benchmark_data_smithi159_80260_object77848:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b06d0469:::benchmark_data_smithi159_80260_object82646:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b06e3a72:::benchmark_data_smithi159_80260_object71610:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b072f85d:::benchmark_data_smithi159_80260_object71676:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b083f16f:::benchmark_data_smithi159_80260_object74511:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0850401:::benchmark_data_smithi159_80260_object86138:head# 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b086db25:::benchmark_data_smithi159_80260_object74523:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b089a0ee:::benchmark_data_smithi159_80260_object86090:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b08b8e44:::benchmark_data_smithi159_80260_object75016:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b08c2f70:::benchmark_data_smithi159_80260_object78858:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b08e6ec9:::benchmark_data_smithi159_80260_object72751:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0907c5a:::benchmark_data_smithi159_80260_object71758:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0950324:::benchmark_data_smithi159_80260_object82906:head# 2024-04-15T18:15:36.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b095185a:::benchmark_data_smithi159_80260_object79032:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09559e4:::benchmark_data_smithi159_80260_object85788:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0958696:::benchmark_data_smithi159_80260_object81810:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b095c739:::benchmark_data_smithi159_80260_object77584:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b097445d:::benchmark_data_smithi159_80260_object80159:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09765a9:::benchmark_data_smithi159_80260_object83388:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09b6274:::benchmark_data_smithi159_80260_object75262:head# 2024-04-15T18:15:36.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09bc8f4:::benchmark_data_smithi159_80260_object88874:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09c9bd2:::benchmark_data_smithi159_80260_object81246:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b09e4de6:::benchmark_data_smithi159_80260_object68555:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0a156af:::benchmark_data_smithi159_80260_object76768:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0a175ff:::benchmark_data_smithi159_80260_object78065:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0a27b8f:::benchmark_data_smithi159_80260_object86771:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0a62532:::benchmark_data_smithi159_80260_object87694:head# 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0a65e8f:::benchmark_data_smithi159_80260_object78522:head# 2024-04-15T18:15:36.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0aa5fa9:::benchmark_data_smithi159_80260_object83769:head# 2024-04-15T18:15:36.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0ab2f68:::benchmark_data_smithi159_80260_object80946:head# 2024-04-15T18:15:36.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0ab388e:::benchmark_data_smithi159_80260_object75809:head# 2024-04-15T18:15:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b050cf:::benchmark_data_smithi159_80260_object88479:head# 2024-04-15T18:15:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b0feb8:::benchmark_data_smithi159_80260_object84917:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b36c0c:::benchmark_data_smithi159_80260_object80135:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b385ee:::benchmark_data_smithi159_80260_object74191:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b59c24:::benchmark_data_smithi159_80260_object70802:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b6b749:::benchmark_data_smithi159_80260_object79656:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b7584c:::benchmark_data_smithi159_80260_object82103:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0b9e759:::benchmark_data_smithi159_80260_object79089:head# 2024-04-15T18:15:36.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0ba628e:::benchmark_data_smithi159_80260_object68968:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0bba882:::benchmark_data_smithi159_80260_object87999:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0be17f6:::benchmark_data_smithi159_80260_object83562:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0be4415:::benchmark_data_smithi159_80260_object88857:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0c5e708:::benchmark_data_smithi159_80260_object70010:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0ce2326:::benchmark_data_smithi159_80260_object84474:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0d01183:::benchmark_data_smithi159_80260_object70861:head# 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0d171fb:::benchmark_data_smithi159_80260_object85472:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0d4af62:::benchmark_data_smithi159_80260_object77931:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0d69d60:::benchmark_data_smithi159_80260_object81268:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0dab7ba:::benchmark_data_smithi159_80260_object82466:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0dd4f10:::benchmark_data_smithi159_80260_object70862:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0deba0b:::benchmark_data_smithi159_80260_object83977:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0dfbba6:::benchmark_data_smithi159_80260_object87014:head# 2024-04-15T18:15:36.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0e08272:::benchmark_data_smithi159_80260_object76913:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0e163b9:::benchmark_data_smithi159_80260_object76665:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0eedca5:::benchmark_data_smithi159_80260_object79025:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0ef3c5a:::benchmark_data_smithi159_80260_object77916:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0f06416:::benchmark_data_smithi159_80260_object85484:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0f3973b:::benchmark_data_smithi159_80260_object71787:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0f7bb07:::benchmark_data_smithi159_80260_object69883:head# 2024-04-15T18:15:36.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0f853a6:::benchmark_data_smithi159_80260_object70953:head# 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b0fbff9d:::benchmark_data_smithi159_80260_object72907:head# 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1055e4c:::benchmark_data_smithi159_80260_object83815:head# 2024-04-15T18:15:36.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b10872f3:::benchmark_data_smithi159_80260_object71900:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b11fb7a2:::benchmark_data_smithi159_80260_object76283:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1233d02:::benchmark_data_smithi159_80260_object85024:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1275d86:::benchmark_data_smithi159_80260_object79268:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b128645c:::benchmark_data_smithi159_80260_object84364:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b12ac9e3:::benchmark_data_smithi159_80260_object75736:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b13018ca:::benchmark_data_smithi159_80260_object81195:head# 2024-04-15T18:15:36.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b133bc13:::benchmark_data_smithi159_80260_object86804:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1389a6f:::benchmark_data_smithi159_80260_object70121:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b13b3020:::benchmark_data_smithi159_80260_object76091:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b13c9a7d:::benchmark_data_smithi159_80260_object73010:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1427f61:::benchmark_data_smithi159_80260_object75834:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b14a394e:::benchmark_data_smithi159_80260_object71619:head# 2024-04-15T18:15:36.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b14e6664:::benchmark_data_smithi159_80260_object77310:head# 2024-04-15T18:15:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b152d21c:::benchmark_data_smithi159_80260_object72511:head# 2024-04-15T18:15:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1548ffa:::benchmark_data_smithi159_80260_object87454:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b155811a:::benchmark_data_smithi159_80260_object85361:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b155eebf:::benchmark_data_smithi159_80260_object70363:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b158ad43:::benchmark_data_smithi159_80260_object70746:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b158ec27:::benchmark_data_smithi159_80260_object73050:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b15e8679:::benchmark_data_smithi159_80260_object80252:head# 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b166b510:::benchmark_data_smithi159_80260_object77841:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b169109c:::benchmark_data_smithi159_80260_object69409:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1694808:::benchmark_data_smithi159_80260_object71923:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b16d191c:::benchmark_data_smithi159_80260_object80543:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1726fe6:::benchmark_data_smithi159_80260_object73952:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b175b3fe:::benchmark_data_smithi159_80260_object75453:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1767a31:::benchmark_data_smithi159_80260_object74903:head# 2024-04-15T18:15:36.686 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1778b8c:::benchmark_data_smithi159_80260_object84751:head# 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b179af7c:::benchmark_data_smithi159_80260_object73503:head# 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b17ff248:::benchmark_data_smithi159_80260_object70367:head# 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1827dc1:::benchmark_data_smithi159_80260_object76741:head# 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b18920d6:::benchmark_data_smithi159_80260_object78854:head# 2024-04-15T18:15:36.687 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b190d645:::benchmark_data_smithi159_80260_object85771:head# 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1931181:::benchmark_data_smithi159_80260_object76707:head# 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b194e006:::benchmark_data_smithi159_80260_object80700:head# 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b19c2e57:::benchmark_data_smithi159_80260_object83961:head# 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b19e06cc:::benchmark_data_smithi159_80260_object84937:head# 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1a30e97:::benchmark_data_smithi159_80260_object72767:head# 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1a6bc61:::benchmark_data_smithi159_80260_object86607:head# 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1aac9fb:::benchmark_data_smithi159_80260_object74574:head# 2024-04-15T18:15:36.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1ad1eae:::benchmark_data_smithi159_80260_object86994:head# 2024-04-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1aecf7d:::benchmark_data_smithi159_80260_object81266:head# 2024-04-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b03c41:::benchmark_data_smithi159_80260_object70069:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b46d4a:::benchmark_data_smithi159_80260_object86995:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b517c0:::benchmark_data_smithi159_80260_object88925:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b5f2f2:::benchmark_data_smithi159_80260_object69476:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b78e71:::benchmark_data_smithi159_80260_object77822:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b8128b:::benchmark_data_smithi159_80260_object83741:head# 2024-04-15T18:15:36.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b96ed4:::benchmark_data_smithi159_80260_object69133:head# 2024-04-15T18:15:36.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1b98984:::benchmark_data_smithi159_80260_object72672:head# 2024-04-15T18:15:36.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1ba674c:::benchmark_data_smithi159_80260_object78971:head# 2024-04-15T18:15:36.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1bd1914:::benchmark_data_smithi159_80260_object73699:head# 2024-04-15T18:15:36.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1bf9ba2:::benchmark_data_smithi159_80260_object79332:head# 2024-04-15T18:15:36.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1bfc3df:::benchmark_data_smithi159_80260_object73761:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1bffa80:::benchmark_data_smithi159_80260_object83800:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1c71b5b:::benchmark_data_smithi159_80260_object76710:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1c72b5f:::benchmark_data_smithi159_80260_object69291:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1ccdbaa:::benchmark_data_smithi159_80260_object73672:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1ce2786:::benchmark_data_smithi159_80260_object87739:head# 2024-04-15T18:15:36.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1cfffeb:::benchmark_data_smithi159_80260_object78811:head# 2024-04-15T18:15:36.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1d648b8:::benchmark_data_smithi159_80260_object85411:head# 2024-04-15T18:15:36.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1d8c2e7:::benchmark_data_smithi159_80260_object78663:head# 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1d97047:::benchmark_data_smithi159_80260_object79683:head# 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1d9d9aa:::benchmark_data_smithi159_80260_object73243:head# 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1dd6cc2:::benchmark_data_smithi159_80260_object73426:head# 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1df3b52:::benchmark_data_smithi159_80260_object82318:head# 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1e0a51e:::benchmark_data_smithi159_80260_object87121:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1e3c037:::benchmark_data_smithi159_80260_object73338:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1e89113:::benchmark_data_smithi159_80260_object70575:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1ea4db0:::benchmark_data_smithi159_80260_object85883:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1eef2d2:::benchmark_data_smithi159_80260_object80081:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1f02f00:::benchmark_data_smithi159_80260_object84798:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1f52342:::benchmark_data_smithi159_80260_object86192:head# 2024-04-15T18:15:36.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1f5cc5c:::benchmark_data_smithi159_80260_object77189:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1f7939f:::benchmark_data_smithi159_80260_object83404:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b1fbf95f:::benchmark_data_smithi159_80260_object71749:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2011ac0:::benchmark_data_smithi159_80260_object79139:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b20348db:::benchmark_data_smithi159_80260_object75865:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b203a6b0:::benchmark_data_smithi159_80260_object85221:head# 2024-04-15T18:15:36.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2085f88:::benchmark_data_smithi159_80260_object77051:head# 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b20bcc0f:::benchmark_data_smithi159_80260_object70603:head# 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b20bfbf9:::benchmark_data_smithi159_80260_object80862:head# 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b20d63db:::benchmark_data_smithi159_80260_object79035:head# 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b20e0397:::benchmark_data_smithi159_80260_object84438:head# 2024-04-15T18:15:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b217319e:::benchmark_data_smithi159_80260_object75401:head# 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21b431c:::benchmark_data_smithi159_80260_object84319:head# 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21c4e13:::benchmark_data_smithi159_80260_object81364:head# 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21d21ea:::benchmark_data_smithi159_80260_object77534:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21d352d:::benchmark_data_smithi159_80260_object72070:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21e83f1:::benchmark_data_smithi159_80260_object88681:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b21e8674:::benchmark_data_smithi159_80260_object70279:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b22506f5:::benchmark_data_smithi159_80260_object88607:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b225cf52:::benchmark_data_smithi159_80260_object80243:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b22985af:::benchmark_data_smithi159_80260_object77559:head# 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b229a360:::benchmark_data_smithi159_80260_object78902:head# 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b22e3959:::benchmark_data_smithi159_80260_object72939:head# 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b22e9afd:::benchmark_data_smithi159_80260_object79462:head# 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b23044f1:::benchmark_data_smithi159_80260_object80607:head# 2024-04-15T18:15:36.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b23094b0:::benchmark_data_smithi159_80260_object84451:head# 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b239b69f:::benchmark_data_smithi159_80260_object87760:head# 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b23d3eb6:::benchmark_data_smithi159_80260_object83539:head# 2024-04-15T18:15:36.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b23f24b1:::benchmark_data_smithi159_80260_object75615:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2418b35:::benchmark_data_smithi159_80260_object73371:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b24284b3:::benchmark_data_smithi159_80260_object70309:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b24990be:::benchmark_data_smithi159_80260_object89058:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2499472:::benchmark_data_smithi159_80260_object84855:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b24b8bbb:::benchmark_data_smithi159_80260_object77232:head# 2024-04-15T18:15:36.723 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b24eb5a1:::benchmark_data_smithi159_80260_object85339:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2562a2d:::benchmark_data_smithi159_80260_object73017:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2578325:::benchmark_data_smithi159_80260_object85368:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b25c1fc9:::benchmark_data_smithi159_80260_object68583:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b25e7da1:::benchmark_data_smithi159_80260_object86580:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b25ee349:::benchmark_data_smithi159_80260_object74807:head# 2024-04-15T18:15:36.724 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b25f04cb:::benchmark_data_smithi159_80260_object86848:head# 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2625fb6:::benchmark_data_smithi159_80260_object85761:head# 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b26ab841:::benchmark_data_smithi159_80260_object78541:head# 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b26be00f:::benchmark_data_smithi159_80260_object73907:head# 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.725 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b26c2d46:::benchmark_data_smithi159_80260_object71327:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b270d4f9:::benchmark_data_smithi159_80260_object82983:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b27177ef:::benchmark_data_smithi159_80260_object71678:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b273bf48:::benchmark_data_smithi159_80260_object88235:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b275bb38:::benchmark_data_smithi159_80260_object86260:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2765a6a:::benchmark_data_smithi159_80260_object72367:head# 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.726 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b282d5e8:::benchmark_data_smithi159_80260_object88723:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b286806b:::benchmark_data_smithi159_80260_object68988:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b288940f:::benchmark_data_smithi159_80260_object78842:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2902c99:::benchmark_data_smithi159_80260_object82518:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b290893b:::benchmark_data_smithi159_80260_object82533:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b290a0a4:::benchmark_data_smithi159_80260_object87781:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2915f0c:::benchmark_data_smithi159_80260_object81358:head# 2024-04-15T18:15:36.727 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b291b5aa:::benchmark_data_smithi159_80260_object77001:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b291b897:::benchmark_data_smithi159_80260_object85213:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b294d9be:::benchmark_data_smithi159_80260_object77017:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b29823e8:::benchmark_data_smithi159_80260_object88743:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b29e55f1:::benchmark_data_smithi159_80260_object73011:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a3d259:::benchmark_data_smithi159_80260_object68716:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a45e95:::benchmark_data_smithi159_80260_object84225:head# 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.728 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a6e949:::benchmark_data_smithi159_80260_object88225:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a7122f:::benchmark_data_smithi159_80260_object80277:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a838fc:::benchmark_data_smithi159_80260_object77039:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a8e13a:::benchmark_data_smithi159_80260_object68576:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a92a1e:::benchmark_data_smithi159_80260_object87386:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2a99896:::benchmark_data_smithi159_80260_object79678:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2ad7b1f:::benchmark_data_smithi159_80260_object75812:head# 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.729 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2aeabfc:::benchmark_data_smithi159_80260_object69136:head# 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2b73eff:::benchmark_data_smithi159_80260_object79195:head# 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2bbcb1c:::benchmark_data_smithi159_80260_object83845:head# 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2be1d5e:::benchmark_data_smithi159_80260_object82395:head# 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.730 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2c8f4c9:::benchmark_data_smithi159_80260_object83876:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2cbc138:::benchmark_data_smithi159_80260_object81139:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2cc5a84:::benchmark_data_smithi159_80260_object74385:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d1ecce:::benchmark_data_smithi159_80260_object86153:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d20e03:::benchmark_data_smithi159_80260_object81353:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d3ae29:::benchmark_data_smithi159_80260_object76972:head# 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.731 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d48024:::benchmark_data_smithi159_80260_object70151:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d4899c:::benchmark_data_smithi159_80260_object87161:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2d69744:::benchmark_data_smithi159_80260_object84517:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2e58fa8:::benchmark_data_smithi159_80260_object80554:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2e77d4d:::benchmark_data_smithi159_80260_object83531:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2e9f3a5:::benchmark_data_smithi159_80260_object82820:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2eadfb7:::benchmark_data_smithi159_80260_object84398:head# 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2eec898:::benchmark_data_smithi159_80260_object68672:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2f4bfe4:::benchmark_data_smithi159_80260_object82765:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2f5de32:::benchmark_data_smithi159_80260_object68900:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2fac832:::benchmark_data_smithi159_80260_object71975:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2fbf7da:::benchmark_data_smithi159_80260_object88999:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b2fc5dbc:::benchmark_data_smithi159_80260_object74349:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b302477b:::benchmark_data_smithi159_80260_object69768:head# 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b307689d:::benchmark_data_smithi159_80260_object68857:head# 2024-04-15T18:15:36.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.770 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b307ace9:::benchmark_data_smithi159_80260_object73626:head# 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b308005c:::benchmark_data_smithi159_80260_object81876:head# 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b30a214c:::benchmark_data_smithi159_80260_object74193:head# 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b30dfcf3:::benchmark_data_smithi159_80260_object78698:head# 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b311277e:::benchmark_data_smithi159_80260_object78320:head# 2024-04-15T18:15:36.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3158ad9:::benchmark_data_smithi159_80260_object80818:head# 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3162bfa:::benchmark_data_smithi159_80260_object79749:head# 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b31f0847:::benchmark_data_smithi159_80260_object80611:head# 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.784 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3228714:::benchmark_data_smithi159_80260_object81472:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b32488a1:::benchmark_data_smithi159_80260_object72414:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3372469:::benchmark_data_smithi159_80260_object71215:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b33a00b4:::benchmark_data_smithi159_80260_object68779:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b33f7abb:::benchmark_data_smithi159_80260_object68991:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3407745:::benchmark_data_smithi159_80260_object74694:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3441301:::benchmark_data_smithi159_80260_object81459:head# 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3441ace:::benchmark_data_smithi159_80260_object89004:head# 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b345916a:::benchmark_data_smithi159_80260_object85206:head# 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b34ab825:::benchmark_data_smithi159_80260_object83178:head# 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b34abd51:::benchmark_data_smithi159_80260_object84351:head# 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b34cadad:::benchmark_data_smithi159_80260_object73418:head# 2024-04-15T18:15:36.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b34ea192:::benchmark_data_smithi159_80260_object71200:head# 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b351184b:::benchmark_data_smithi159_80260_object69234:head# 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b351f9db:::benchmark_data_smithi159_80260_object78264:head# 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3570d83:::benchmark_data_smithi159_80260_object85139:head# 2024-04-15T18:15:36.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3574f0f:::benchmark_data_smithi159_80260_object81376:head# 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b35a26b2:::benchmark_data_smithi159_80260_object87151:head# 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b35a9b0d:::benchmark_data_smithi159_80260_object85872:head# 2024-04-15T18:15:36.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b35c6c5b:::benchmark_data_smithi159_80260_object72194:head# 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b35d5fc3:::benchmark_data_smithi159_80260_object85631:head# 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b35df4b8:::benchmark_data_smithi159_80260_object80945:head# 2024-04-15T18:15:36.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b367820b:::benchmark_data_smithi159_80260_object70475:head# 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b36a0ebe:::benchmark_data_smithi159_80260_object85116:head# 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b36c9f6f:::benchmark_data_smithi159_80260_object88088:head# 2024-04-15T18:15:36.790 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b374547a:::benchmark_data_smithi159_80260_object82967:head# 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3750cd9:::benchmark_data_smithi159_80260_object76740:head# 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b378ae0f:::benchmark_data_smithi159_80260_object72917:head# 2024-04-15T18:15:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b37a7a30:::benchmark_data_smithi159_80260_object86833:head# 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b37ff229:::benchmark_data_smithi159_80260_object87188:head# 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38106f2:::benchmark_data_smithi159_80260_object88749:head# 2024-04-15T18:15:36.792 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b381ee86:::benchmark_data_smithi159_80260_object80552:head# 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38959d1:::benchmark_data_smithi159_80260_object88884:head# 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3896e83:::benchmark_data_smithi159_80260_object72115:head# 2024-04-15T18:15:36.793 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38d6fbf:::benchmark_data_smithi159_80260_object76255:head# 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38e899a:::benchmark_data_smithi159_80260_object87961:head# 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38f6c90:::benchmark_data_smithi159_80260_object70649:head# 2024-04-15T18:15:36.794 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b38f7aac:::benchmark_data_smithi159_80260_object87938:head# 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b390a6a0:::benchmark_data_smithi159_80260_object71763:head# 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b39889c8:::benchmark_data_smithi159_80260_object83412:head# 2024-04-15T18:15:36.795 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b39c0507:::benchmark_data_smithi159_80260_object80643:head# 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3a2f2ce:::benchmark_data_smithi159_80260_object76946:head# 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3a38d98:::benchmark_data_smithi159_80260_object79689:head# 2024-04-15T18:15:36.796 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3a50b0e:::benchmark_data_smithi159_80260_object81410:head# 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3a8d051:::benchmark_data_smithi159_80260_object82825:head# 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3a9d057:::benchmark_data_smithi159_80260_object85887:head# 2024-04-15T18:15:36.797 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3ac3c76:::benchmark_data_smithi159_80260_object83073:head# 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b0ed47:::benchmark_data_smithi159_80260_object80840:head# 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b1f74f:::benchmark_data_smithi159_80260_object84260:head# 2024-04-15T18:15:36.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b22861:::benchmark_data_smithi159_80260_object86339:head# 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b3dc30:::benchmark_data_smithi159_80260_object76254:head# 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b3e622:::benchmark_data_smithi159_80260_object79516:head# 2024-04-15T18:15:36.799 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3b76758:::benchmark_data_smithi159_80260_object88606:head# 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3bb751d:::benchmark_data_smithi159_80260_object78136:head# 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3be3ccb:::benchmark_data_smithi159_80260_object72242:head# 2024-04-15T18:15:36.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3c10440:::benchmark_data_smithi159_80260_object78044:head# 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3cfe65a:::benchmark_data_smithi159_80260_object83755:head# 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3d3b631:::benchmark_data_smithi159_80260_object88209:head# 2024-04-15T18:15:36.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3d4393c:::benchmark_data_smithi159_80260_object83868:head# 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3d56726:::benchmark_data_smithi159_80260_object71809:head# 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3d6245f:::benchmark_data_smithi159_80260_object79343:head# 2024-04-15T18:15:36.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3ddc8b2:::benchmark_data_smithi159_80260_object82883:head# 2024-04-15T18:15:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3df8d2c:::benchmark_data_smithi159_80260_object83898:head# 2024-04-15T18:15:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e0c714:::benchmark_data_smithi159_80260_object83932:head# 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e12367:::benchmark_data_smithi159_80260_object72748:head# 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e2f805:::benchmark_data_smithi159_80260_object73356:head# 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e35f0a:::benchmark_data_smithi159_80260_object71882:head# 2024-04-15T18:15:36.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e427c8:::benchmark_data_smithi159_80260_object71460:head# 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3e63456:::benchmark_data_smithi159_80260_object77692:head# 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3ea0614:::benchmark_data_smithi159_80260_object71211:head# 2024-04-15T18:15:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3ecb32a:::benchmark_data_smithi159_80260_object80980:head# 2024-04-15T18:15:36.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3efc054:::benchmark_data_smithi159_80260_object73120:head# 2024-04-15T18:15:36.806 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.806 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3f4ca25:::benchmark_data_smithi159_80260_object81767:head# 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3f5aac5:::benchmark_data_smithi159_80260_object84217:head# 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3f80045:::benchmark_data_smithi159_80260_object72198:head# 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b3fb458a:::benchmark_data_smithi159_80260_object72605:head# 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b40074b7:::benchmark_data_smithi159_80260_object85796:head# 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4093ebd:::benchmark_data_smithi159_80260_object81589:head# 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b409e499:::benchmark_data_smithi159_80260_object81028:head# 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b40adfd0:::benchmark_data_smithi159_80260_object68815:head# 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b416f4d4:::benchmark_data_smithi159_80260_object78218:head# 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.809 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4170b6e:::benchmark_data_smithi159_80260_object79215:head# 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4195c41:::benchmark_data_smithi159_80260_object79706:head# 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b41bc71a:::benchmark_data_smithi159_80260_object75477:head# 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b41dd147:::benchmark_data_smithi159_80260_object69035:head# 2024-04-15T18:15:36.810 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b41e5937:::benchmark_data_smithi159_80260_object75397:head# 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b421b9d5:::benchmark_data_smithi159_80260_object88297:head# 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b42580d9:::benchmark_data_smithi159_80260_object81514:head# 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4280ff5:::benchmark_data_smithi159_80260_object68637:head# 2024-04-15T18:15:36.811 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b428140e:::benchmark_data_smithi159_80260_object75621:head# 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b42acf9f:::benchmark_data_smithi159_80260_object77429:head# 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b42be092:::benchmark_data_smithi159_80260_object69865:head# 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b42f3955:::benchmark_data_smithi159_80260_object76563:head# 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.812 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4320c68:::benchmark_data_smithi159_80260_object71363:head# 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b43561b9:::benchmark_data_smithi159_80260_object79888:head# 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b437daa7:::benchmark_data_smithi159_80260_object86620:head# 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b43a25b6:::benchmark_data_smithi159_80260_object70889:head# 2024-04-15T18:15:36.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b43d91ab:::benchmark_data_smithi159_80260_object87580:head# 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b440fe05:::benchmark_data_smithi159_80260_object82678:head# 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4453328:::benchmark_data_smithi159_80260_object84733:head# 2024-04-15T18:15:36.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b447cb12:::benchmark_data_smithi159_80260_object77108:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b44efc4d:::benchmark_data_smithi159_80260_object80947:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b454c1ab:::benchmark_data_smithi159_80260_object77631:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b45a3a57:::benchmark_data_smithi159_80260_object73884:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b45bb2d0:::benchmark_data_smithi159_80260_object73901:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b460b12b:::benchmark_data_smithi159_80260_object69887:head# 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.815 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b46649ea:::benchmark_data_smithi159_80260_object76264:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b46be99e:::benchmark_data_smithi159_80260_object79372:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b470b612:::benchmark_data_smithi159_80260_object70890:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b47d5470:::benchmark_data_smithi159_80260_object71208:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b47f18f4:::benchmark_data_smithi159_80260_object73993:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b47f7845:::benchmark_data_smithi159_80260_object83750:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b47f8171:::benchmark_data_smithi159_80260_object73886:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b480a40a:::benchmark_data_smithi159_80260_object86244:head# 2024-04-15T18:15:36.816 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4864e15:::benchmark_data_smithi159_80260_object87795:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b48813a7:::benchmark_data_smithi159_80260_object83200:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4890cc8:::benchmark_data_smithi159_80260_object71498:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4964acd:::benchmark_data_smithi159_80260_object78834:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4983a3c:::benchmark_data_smithi159_80260_object78064:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b49abfaa:::benchmark_data_smithi159_80260_object72938:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b49d6711:::benchmark_data_smithi159_80260_object76057:head# 2024-04-15T18:15:36.817 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b49dd236:::benchmark_data_smithi159_80260_object70064:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b49fcf91:::benchmark_data_smithi159_80260_object82569:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a3cf13:::benchmark_data_smithi159_80260_object84181:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a5a2f2:::benchmark_data_smithi159_80260_object78980:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a69355:::benchmark_data_smithi159_80260_object76965:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a73624:::benchmark_data_smithi159_80260_object89055:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a83190:::benchmark_data_smithi159_80260_object72416:head# 2024-04-15T18:15:36.818 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a87952:::benchmark_data_smithi159_80260_object77436:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4a97a9e:::benchmark_data_smithi159_80260_object85821:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4aee3a0:::benchmark_data_smithi159_80260_object84024:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4b11527:::benchmark_data_smithi159_80260_object83929:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4b232a6:::benchmark_data_smithi159_80260_object85049:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4c48c1c:::benchmark_data_smithi159_80260_object83514:head# 2024-04-15T18:15:36.819 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4c67142:::benchmark_data_smithi159_80260_object77995:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4c72405:::benchmark_data_smithi159_80260_object82895:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4cc87c6:::benchmark_data_smithi159_80260_object86755:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4d71731:::benchmark_data_smithi159_80260_object71786:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4e30680:::benchmark_data_smithi159_80260_object82289:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4e7e7e9:::benchmark_data_smithi159_80260_object74679:head# 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.820 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4ee2c45:::benchmark_data_smithi159_80260_object74879:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f04957:::benchmark_data_smithi159_80260_object84546:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f38030:::benchmark_data_smithi159_80260_object76389:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f40fe7:::benchmark_data_smithi159_80260_object75464:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f448c2:::benchmark_data_smithi159_80260_object82363:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f5eb7a:::benchmark_data_smithi159_80260_object83756:head# 2024-04-15T18:15:36.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f6cfa9:::benchmark_data_smithi159_80260_object72587:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f804e1:::benchmark_data_smithi159_80260_object71821:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4f9d86d:::benchmark_data_smithi159_80260_object72326:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4fdde49:::benchmark_data_smithi159_80260_object78421:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b4fed900:::benchmark_data_smithi159_80260_object68763:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b502a161:::benchmark_data_smithi159_80260_object82667:head# 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b50ac17f:::benchmark_data_smithi159_80260_object68711:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b50e0189:::benchmark_data_smithi159_80260_object80376:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b50e5490:::benchmark_data_smithi159_80260_object74150:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b51108d7:::benchmark_data_smithi159_80260_object81612:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b51218e2:::benchmark_data_smithi159_80260_object69437:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b51c3807:::benchmark_data_smithi159_80260_object84627:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b51c6405:::benchmark_data_smithi159_80260_object86807:head# 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.823 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5207e8f:::benchmark_data_smithi159_80260_object87157:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b523772b:::benchmark_data_smithi159_80260_object75449:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b52692ff:::benchmark_data_smithi159_80260_object68642:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b528b1a8:::benchmark_data_smithi159_80260_object74873:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b530cfe8:::benchmark_data_smithi159_80260_object71939:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b531c148:::benchmark_data_smithi159_80260_object85602:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b53635c5:::benchmark_data_smithi159_80260_object76090:head# 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.824 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b536d05c:::benchmark_data_smithi159_80260_object84775:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5385a9c:::benchmark_data_smithi159_80260_object75202:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b53947b2:::benchmark_data_smithi159_80260_object77582:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b53bdaf5:::benchmark_data_smithi159_80260_object81937:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b53f591a:::benchmark_data_smithi159_80260_object78723:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b54003f1:::benchmark_data_smithi159_80260_object78695:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5452191:::benchmark_data_smithi159_80260_object85534:head# 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b546c36c:::benchmark_data_smithi159_80260_object87936:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b548ca29:::benchmark_data_smithi159_80260_object86510:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b54bd427:::benchmark_data_smithi159_80260_object84464:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b553bdad:::benchmark_data_smithi159_80260_object87099:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5607902:::benchmark_data_smithi159_80260_object86435:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5608e10:::benchmark_data_smithi159_80260_object74956:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b560f9aa:::benchmark_data_smithi159_80260_object72456:head# 2024-04-15T18:15:36.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b564cfce:::benchmark_data_smithi159_80260_object69673:head# 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b568eca1:::benchmark_data_smithi159_80260_object82675:head# 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5696f1c:::benchmark_data_smithi159_80260_object85650:head# 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b575ad28:::benchmark_data_smithi159_80260_object70305:head# 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.827 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b579834c:::benchmark_data_smithi159_80260_object74489:head# 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b585d0ae:::benchmark_data_smithi159_80260_object74984:head# 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b58d060d:::benchmark_data_smithi159_80260_object70245:head# 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b58e55c5:::benchmark_data_smithi159_80260_object73257:head# 2024-04-15T18:15:36.828 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b591ae46:::benchmark_data_smithi159_80260_object73840:head# 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b592d7de:::benchmark_data_smithi159_80260_object72891:head# 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b595d08c:::benchmark_data_smithi159_80260_object85714:head# 2024-04-15T18:15:36.829 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b597be13:::benchmark_data_smithi159_80260_object84762:head# 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b59a3856:::benchmark_data_smithi159_80260_object81390:head# 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b59fca2e:::benchmark_data_smithi159_80260_object85165:head# 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5a2b56c:::benchmark_data_smithi159_80260_object74269:head# 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5a32bb3:::benchmark_data_smithi159_80260_object84257:head# 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5a5de6b:::benchmark_data_smithi159_80260_object84708:head# 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5a9a553:::benchmark_data_smithi159_80260_object81714:head# 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5aba595:::benchmark_data_smithi159_80260_object76854:head# 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5b1f7eb:::benchmark_data_smithi159_80260_object84595:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5b618cb:::benchmark_data_smithi159_80260_object88179:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5bae9e2:::benchmark_data_smithi159_80260_object68985:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5bc31a4:::benchmark_data_smithi159_80260_object87654:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5bcdb53:::benchmark_data_smithi159_80260_object69583:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5cfcd93:::benchmark_data_smithi159_80260_object79397:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5d154f3:::benchmark_data_smithi159_80260_object71083:head# 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.832 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5d6ea91:::benchmark_data_smithi159_80260_object74911:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5dbafb1:::benchmark_data_smithi159_80260_object75917:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5e36686:::benchmark_data_smithi159_80260_object69990:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5e43bd0:::benchmark_data_smithi159_80260_object73352:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5e4dfaf:::benchmark_data_smithi159_80260_object83415:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5eab3b4:::benchmark_data_smithi159_80260_object82634:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5ee6553:::benchmark_data_smithi159_80260_object71994:head# 2024-04-15T18:15:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5f386fb:::benchmark_data_smithi159_80260_object86539:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b5ff4954:::benchmark_data_smithi159_80260_object86931:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b603469e:::benchmark_data_smithi159_80260_object88595:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6049561:::benchmark_data_smithi159_80260_object84528:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6062e9b:::benchmark_data_smithi159_80260_object89041:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b606444a:::benchmark_data_smithi159_80260_object77370:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6085634:::benchmark_data_smithi159_80260_object79428:head# 2024-04-15T18:15:36.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6086ae5:::benchmark_data_smithi159_80260_object79875:head# 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b60972f1:::benchmark_data_smithi159_80260_object78573:head# 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b60e4424:::benchmark_data_smithi159_80260_object79299:head# 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b60ef1b1:::benchmark_data_smithi159_80260_object80860:head# 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6120f42:::benchmark_data_smithi159_80260_object81179:head# 2024-04-15T18:15:36.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b614e5ba:::benchmark_data_smithi159_80260_object69028:head# 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6150c6d:::benchmark_data_smithi159_80260_object70218:head# 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b618de02:::benchmark_data_smithi159_80260_object78643:head# 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b61aea4c:::benchmark_data_smithi159_80260_object70887:head# 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b61d093b:::benchmark_data_smithi159_80260_object86574:head# 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b61d1744:::benchmark_data_smithi159_80260_object88875:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b61d8030:::benchmark_data_smithi159_80260_object81601:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b61dbeac:::benchmark_data_smithi159_80260_object70932:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6208c04:::benchmark_data_smithi159_80260_object85692:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6214ced:::benchmark_data_smithi159_80260_object75842:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b62216e0:::benchmark_data_smithi159_80260_object86271:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b624e45a:::benchmark_data_smithi159_80260_object79961:head# 2024-04-15T18:15:36.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b62615e7:::benchmark_data_smithi159_80260_object85051:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b62a236a:::benchmark_data_smithi159_80260_object71378:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b62ed19d:::benchmark_data_smithi159_80260_object88190:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b62f0787:::benchmark_data_smithi159_80260_object81726:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6306172:::benchmark_data_smithi159_80260_object78253:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b632620e:::benchmark_data_smithi159_80260_object73428:head# 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b63352c3:::benchmark_data_smithi159_80260_object71561:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6344319:::benchmark_data_smithi159_80260_object79012:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b634dc3a:::benchmark_data_smithi159_80260_object82110:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b639b73b:::benchmark_data_smithi159_80260_object73373:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b63e8380:::benchmark_data_smithi159_80260_object70195:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6426778:::benchmark_data_smithi159_80260_object69739:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b645ffe5:::benchmark_data_smithi159_80260_object89043:head# 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6472c10:::benchmark_data_smithi159_80260_object88193:head# 2024-04-15T18:15:36.840 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:36.842 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:36.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:36.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:36.988 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:36.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6472e26:::benchmark_data_smithi159_80260_object71152:head# 2024-04-15T18:15:36.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b647ed1c:::benchmark_data_smithi159_80260_object73555:head# 2024-04-15T18:15:36.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b64975f6:::benchmark_data_smithi159_80260_object87197:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b64f3de5:::benchmark_data_smithi159_80260_object72754:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b64f904b:::benchmark_data_smithi159_80260_object87763:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6531752:::benchmark_data_smithi159_80260_object78563:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b656305d:::benchmark_data_smithi159_80260_object77119:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b658014c:::benchmark_data_smithi159_80260_object87775:head# 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b65b1d80:::benchmark_data_smithi159_80260_object83278:head# 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b660a2a8:::benchmark_data_smithi159_80260_object73833:head# 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6621e92:::benchmark_data_smithi159_80260_object86203:head# 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6623f8d:::benchmark_data_smithi159_80260_object73711:head# 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6667ad1:::benchmark_data_smithi159_80260_object81083:head# 2024-04-15T18:15:36.994 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b66b9986:::benchmark_data_smithi159_80260_object70247:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b66f1491:::benchmark_data_smithi159_80260_object85670:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6709d78:::benchmark_data_smithi159_80260_object78277:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6732203:::benchmark_data_smithi159_80260_object71132:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b67375ef:::benchmark_data_smithi159_80260_object76879:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b675f9d6:::benchmark_data_smithi159_80260_object77976:head# 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b67a3fbf:::benchmark_data_smithi159_80260_object80704:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b67a7e5f:::benchmark_data_smithi159_80260_object71446:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b67dcc9f:::benchmark_data_smithi159_80260_object85346:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b67e0ff1:::benchmark_data_smithi159_80260_object88320:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b682e62d:::benchmark_data_smithi159_80260_object85960:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6872022:::benchmark_data_smithi159_80260_object89063:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b687532a:::benchmark_data_smithi159_80260_object80499:head# 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b68ea4bf:::benchmark_data_smithi159_80260_object88071:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b691822f:::benchmark_data_smithi159_80260_object87138:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6919121:::benchmark_data_smithi159_80260_object73866:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b696efba:::benchmark_data_smithi159_80260_object73463:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6980615:::benchmark_data_smithi159_80260_object85375:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b69a5825:::benchmark_data_smithi159_80260_object85617:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b69dc834:::benchmark_data_smithi159_80260_object80646:head# 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.997 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b69e0948:::benchmark_data_smithi159_80260_object82162:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6a2a1ac:::benchmark_data_smithi159_80260_object80155:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6a8114a:::benchmark_data_smithi159_80260_object78095:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6b1aed6:::benchmark_data_smithi159_80260_object85431:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6ba3ed7:::benchmark_data_smithi159_80260_object86294:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6be3b4c:::benchmark_data_smithi159_80260_object86919:head# 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6c298f9:::benchmark_data_smithi159_80260_object72604:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6c688b1:::benchmark_data_smithi159_80260_object68845:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6c7ef8a:::benchmark_data_smithi159_80260_object76904:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6ccbbc1:::benchmark_data_smithi159_80260_object88781:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6d1ed25:::benchmark_data_smithi159_80260_object80702:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6d681a3:::benchmark_data_smithi159_80260_object73877:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6d7b022:::benchmark_data_smithi159_80260_object85804:head# 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:36.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6d873fc:::benchmark_data_smithi159_80260_object84299:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6d87acf:::benchmark_data_smithi159_80260_object87423:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6da3978:::benchmark_data_smithi159_80260_object72354:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6db77eb:::benchmark_data_smithi159_80260_object81911:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6dd04fd:::benchmark_data_smithi159_80260_object85817:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6df3a21:::benchmark_data_smithi159_80260_object73112:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6e1b45a:::benchmark_data_smithi159_80260_object75575:head# 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6ec065f:::benchmark_data_smithi159_80260_object70961:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6f0abf1:::benchmark_data_smithi159_80260_object81086:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6f9ed59:::benchmark_data_smithi159_80260_object73456:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6fb660b:::benchmark_data_smithi159_80260_object83241:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b6fdf2b5:::benchmark_data_smithi159_80260_object75683:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b707c4bb:::benchmark_data_smithi159_80260_object77167:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b70ce5f6:::benchmark_data_smithi159_80260_object78452:head# 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b70d40d7:::benchmark_data_smithi159_80260_object80726:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7109e14:::benchmark_data_smithi159_80260_object68540:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b716c6d8:::benchmark_data_smithi159_80260_object82219:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b71d36f4:::benchmark_data_smithi159_80260_object82637:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b723d6a1:::benchmark_data_smithi159_80260_object70222:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b728a228:::benchmark_data_smithi159_80260_object84857:head# 2024-04-15T18:15:37.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b72b4a80:::benchmark_data_smithi159_80260_object88842:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b72e2d18:::benchmark_data_smithi159_80260_object86736:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7304100:::benchmark_data_smithi159_80260_object75541:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b731e344:::benchmark_data_smithi159_80260_object73486:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b734f0be:::benchmark_data_smithi159_80260_object85818:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b73573cc:::benchmark_data_smithi159_80260_object77643:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7375314:::benchmark_data_smithi159_80260_object84370:head# 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b73cc004:::benchmark_data_smithi159_80260_object71528:head# 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b73f256b:::benchmark_data_smithi159_80260_object82258:head# 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7437496:::benchmark_data_smithi159_80260_object85539:head# 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7448d86:::benchmark_data_smithi159_80260_object73590:head# 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b745e0e3:::benchmark_data_smithi159_80260_object74204:head# 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b74934bd:::benchmark_data_smithi159_80260_object73955:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b74aa096:::benchmark_data_smithi159_80260_object73975:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b74bf7f8:::benchmark_data_smithi159_80260_object76799:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7561d0b:::benchmark_data_smithi159_80260_object78254:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b75671ff:::benchmark_data_smithi159_80260_object80365:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7568d81:::benchmark_data_smithi159_80260_object77518:head# 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.005 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b75ce383:::benchmark_data_smithi159_80260_object70982:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b75f9070:::benchmark_data_smithi159_80260_object84280:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7617bf1:::benchmark_data_smithi159_80260_object78619:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b76237f7:::benchmark_data_smithi159_80260_object71076:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b764215c:::benchmark_data_smithi159_80260_object78088:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7644690:::benchmark_data_smithi159_80260_object69420:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b76476bf:::benchmark_data_smithi159_80260_object73034:head# 2024-04-15T18:15:37.006 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7676f1c:::benchmark_data_smithi159_80260_object70512:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b768c365:::benchmark_data_smithi159_80260_object80494:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b76f2a2e:::benchmark_data_smithi159_80260_object78762:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b770ca89:::benchmark_data_smithi159_80260_object81327:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b772b856:::benchmark_data_smithi159_80260_object70132:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b77430ef:::benchmark_data_smithi159_80260_object68868:head# 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.007 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7743cbd:::benchmark_data_smithi159_80260_object88978:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7745f48:::benchmark_data_smithi159_80260_object78804:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7792583:::benchmark_data_smithi159_80260_object87817:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b77aae13:::benchmark_data_smithi159_80260_object78149:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b77b7eb0:::benchmark_data_smithi159_80260_object77181:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b77fe83a:::benchmark_data_smithi159_80260_object84779:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b780a870:::benchmark_data_smithi159_80260_object69745:head# 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b782d107:::benchmark_data_smithi159_80260_object85829:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b783f73d:::benchmark_data_smithi159_80260_object76388:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78468bf:::benchmark_data_smithi159_80260_object71236:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7860447:::benchmark_data_smithi159_80260_object80203:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b787bb0a:::benchmark_data_smithi159_80260_object83970:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78ab854:::benchmark_data_smithi159_80260_object86928:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78ade07:::benchmark_data_smithi159_80260_object72200:head# 2024-04-15T18:15:37.009 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78b14e0:::benchmark_data_smithi159_80260_object69164:head# 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78bc9e1:::benchmark_data_smithi159_80260_object77836:head# 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78d06c9:::benchmark_data_smithi159_80260_object72911:head# 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78e3b6f:::benchmark_data_smithi159_80260_object74989:head# 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b78f8a90:::benchmark_data_smithi159_80260_object82961:head# 2024-04-15T18:15:37.010 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b79462a0:::benchmark_data_smithi159_80260_object84602:head# 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7983770:::benchmark_data_smithi159_80260_object83121:head# 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b79e137b:::benchmark_data_smithi159_80260_object71655:head# 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7a028c7:::benchmark_data_smithi159_80260_object88378:head# 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7a5d4a5:::benchmark_data_smithi159_80260_object75836:head# 2024-04-15T18:15:37.011 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7a683e8:::benchmark_data_smithi159_80260_object83366:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7a7017f:::benchmark_data_smithi159_80260_object88713:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7a9150a:::benchmark_data_smithi159_80260_object88420:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7af6bca:::benchmark_data_smithi159_80260_object75356:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7b2b8e1:::benchmark_data_smithi159_80260_object78749:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7bc1f69:::benchmark_data_smithi159_80260_object86776:head# 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7bcddce:::benchmark_data_smithi159_80260_object82458:head# 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7c02fef:::benchmark_data_smithi159_80260_object76779:head# 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7c03cd6:::benchmark_data_smithi159_80260_object82061:head# 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7c64794:::benchmark_data_smithi159_80260_object72294:head# 2024-04-15T18:15:37.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7c800bd:::benchmark_data_smithi159_80260_object85257:head# 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7c831bf:::benchmark_data_smithi159_80260_object84731:head# 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7ca3761:::benchmark_data_smithi159_80260_object69805:head# 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7ca4fa8:::benchmark_data_smithi159_80260_object85832:head# 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7cc1727:::benchmark_data_smithi159_80260_object68740:head# 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.014 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7cf436f:::benchmark_data_smithi159_80260_object73623:head# 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7d259d4:::benchmark_data_smithi159_80260_object80298:head# 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7d33602:::benchmark_data_smithi159_80260_object83003:head# 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7d637f4:::benchmark_data_smithi159_80260_object80257:head# 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7dc29d9:::benchmark_data_smithi159_80260_object69733:head# 2024-04-15T18:15:37.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7df47c7:::benchmark_data_smithi159_80260_object70296:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e10141:::benchmark_data_smithi159_80260_object83805:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e50b8d:::benchmark_data_smithi159_80260_object80384:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e64b53:::benchmark_data_smithi159_80260_object69966:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e6c990:::benchmark_data_smithi159_80260_object84417:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e740ba:::benchmark_data_smithi159_80260_object88754:head# 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7e7ee7f:::benchmark_data_smithi159_80260_object69332:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7eaff47:::benchmark_data_smithi159_80260_object74999:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7ee4878:::benchmark_data_smithi159_80260_object86557:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7f29612:::benchmark_data_smithi159_80260_object78712:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7f32bf0:::benchmark_data_smithi159_80260_object84549:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7f55952:::benchmark_data_smithi159_80260_object83126:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7f5dbb0:::benchmark_data_smithi159_80260_object87394:head# 2024-04-15T18:15:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7fc0f4a:::benchmark_data_smithi159_80260_object72603:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b7ff0c15:::benchmark_data_smithi159_80260_object73720:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b80049a5:::benchmark_data_smithi159_80260_object69558:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8009e3a:::benchmark_data_smithi159_80260_object73090:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8016844:::benchmark_data_smithi159_80260_object70803:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b801c7a7:::benchmark_data_smithi159_80260_object86732:head# 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8052cbb:::benchmark_data_smithi159_80260_object87003:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b80d978c:::benchmark_data_smithi159_80260_object68931:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8112f37:::benchmark_data_smithi159_80260_object75725:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8154869:::benchmark_data_smithi159_80260_object73973:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b81871d0:::benchmark_data_smithi159_80260_object76638:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8189f58:::benchmark_data_smithi159_80260_object74594:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b819fdee:::benchmark_data_smithi159_80260_object69470:head# 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.019 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b81e70c2:::benchmark_data_smithi159_80260_object86572:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b823edc7:::benchmark_data_smithi159_80260_object79070:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b82727d4:::benchmark_data_smithi159_80260_object69522:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8292a79:::benchmark_data_smithi159_80260_object78327:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b829931d:::benchmark_data_smithi159_80260_object87094:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b82d7425:::benchmark_data_smithi159_80260_object70031:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b82ea432:::benchmark_data_smithi159_80260_object76051:head# 2024-04-15T18:15:37.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b831bd05:::benchmark_data_smithi159_80260_object69622:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b834b151:::benchmark_data_smithi159_80260_object76943:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b83d7b92:::benchmark_data_smithi159_80260_object83914:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b83f6f7a:::benchmark_data_smithi159_80260_object85697:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b841d9c0:::benchmark_data_smithi159_80260_object78084:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b847cd38:::benchmark_data_smithi159_80260_object78486:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b848d4dd:::benchmark_data_smithi159_80260_object77346:head# 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b849cebc:::benchmark_data_smithi159_80260_object79224:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b84fc5b3:::benchmark_data_smithi159_80260_object80363:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b850e21c:::benchmark_data_smithi159_80260_object77361:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b85a556b:::benchmark_data_smithi159_80260_object71535:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b85c8fa5:::benchmark_data_smithi159_80260_object86111:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b86079d6:::benchmark_data_smithi159_80260_object89022:head# 2024-04-15T18:15:37.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8615e71:::benchmark_last_metadata:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b861bc82:::benchmark_data_smithi159_80260_object84683:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b862817d:::benchmark_data_smithi159_80260_object73507:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b862e0a8:::benchmark_data_smithi159_80260_object69528:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8669500:::benchmark_data_smithi159_80260_object81300:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b867b9a3:::benchmark_data_smithi159_80260_object78649:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b867c4d8:::benchmark_data_smithi159_80260_object81132:head# 2024-04-15T18:15:37.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b86ba2df:::benchmark_data_smithi159_80260_object88801:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b871c890:::benchmark_data_smithi159_80260_object86706:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b873a5f5:::benchmark_data_smithi159_80260_object75264:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b873d38b:::benchmark_data_smithi159_80260_object88989:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b874fa6c:::benchmark_data_smithi159_80260_object86746:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b877de02:::benchmark_data_smithi159_80260_object82158:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b87a3e1c:::benchmark_data_smithi159_80260_object73996:head# 2024-04-15T18:15:37.024 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b87e9a7c:::benchmark_data_smithi159_80260_object77142:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b88b41ea:::benchmark_data_smithi159_80260_object80044:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8978b23:::benchmark_data_smithi159_80260_object73062:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b899c15d:::benchmark_data_smithi159_80260_object74730:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b89c1549:::benchmark_data_smithi159_80260_object73890:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8a3cc9f:::benchmark_data_smithi159_80260_object72896:head# 2024-04-15T18:15:37.025 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8a50ba5:::benchmark_data_smithi159_80260_object73174:head# 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8a8e63d:::benchmark_data_smithi159_80260_object82730:head# 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8aa2f11:::benchmark_data_smithi159_80260_object84168:head# 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8ad7be0:::benchmark_data_smithi159_80260_object87530:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8b0cefb:::benchmark_data_smithi159_80260_object84385:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8ba9dbf:::benchmark_data_smithi159_80260_object80843:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8bb758d:::benchmark_data_smithi159_80260_object86521:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8bc705f:::benchmark_data_smithi159_80260_object85503:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8be59d1:::benchmark_data_smithi159_80260_object68862:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8c05df4:::benchmark_data_smithi159_80260_object79231:head# 2024-04-15T18:15:37.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8c4b800:::benchmark_data_smithi159_80260_object70917:head# 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8c516cd:::benchmark_data_smithi159_80260_object86293:head# 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8c86e3a:::benchmark_data_smithi159_80260_object74878:head# 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8c915e2:::benchmark_data_smithi159_80260_object73974:head# 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d27650:::benchmark_data_smithi159_80260_object75766:head# 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d5a1e9:::benchmark_data_smithi159_80260_object75087:head# 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d73ffa:::benchmark_data_smithi159_80260_object79786:head# 2024-04-15T18:15:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d785a0:::benchmark_data_smithi159_80260_object83001:head# 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d8098b:::benchmark_data_smithi159_80260_object71161:head# 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8d814b6:::benchmark_data_smithi159_80260_object71588:head# 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8dad803:::benchmark_data_smithi159_80260_object71419:head# 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8dc6814:::benchmark_data_smithi159_80260_object73989:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8dd29d4:::benchmark_data_smithi159_80260_object69595:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8e2e392:::benchmark_data_smithi159_80260_object79024:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8e857d7:::benchmark_data_smithi159_80260_object78656:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8eb47fd:::benchmark_data_smithi159_80260_object88330:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8ef6661:::benchmark_data_smithi159_80260_object80148:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8ef9bf0:::benchmark_data_smithi159_80260_object74723:head# 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f00f80:::benchmark_data_smithi159_80260_object84891:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f0f44e:::benchmark_data_smithi159_80260_object69085:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f283b7:::benchmark_data_smithi159_80260_object86105:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f2b6fd:::benchmark_data_smithi159_80260_object85831:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f3000b:::benchmark_data_smithi159_80260_object72108:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f84f26:::benchmark_data_smithi159_80260_object84717:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f8b782:::benchmark_data_smithi159_80260_object81509:head# 2024-04-15T18:15:37.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f90fc6:::benchmark_data_smithi159_80260_object72249:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f9166a:::benchmark_data_smithi159_80260_object79942:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8f9bcb0:::benchmark_data_smithi159_80260_object71794:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8fdbddd:::benchmark_data_smithi159_80260_object72052:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8fe2fca:::benchmark_data_smithi159_80260_object79290:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b8ffc387:::benchmark_data_smithi159_80260_object81460:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b90953c7:::benchmark_data_smithi159_80260_object79396:head# 2024-04-15T18:15:37.033 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b90cd23d:::benchmark_data_smithi159_80260_object69037:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b90df283:::benchmark_data_smithi159_80260_object83461:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b91289a6:::benchmark_data_smithi159_80260_object79769:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b91958b3:::benchmark_data_smithi159_80260_object83463:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b919ccdd:::benchmark_data_smithi159_80260_object74737:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b91ba3fc:::benchmark_data_smithi159_80260_object69438:head# 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.034 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b91f40f1:::benchmark_data_smithi159_80260_object83250:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9219bb6:::benchmark_data_smithi159_80260_object86612:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b926d346:::benchmark_data_smithi159_80260_object68768:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9332f7b:::benchmark_data_smithi159_80260_object70182:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b933525b:::benchmark_data_smithi159_80260_object81497:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b93dcbef:::benchmark_data_smithi159_80260_object88917:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b93f0b43:::benchmark_data_smithi159_80260_object82914:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9438126:::benchmark_data_smithi159_80260_object86713:head# 2024-04-15T18:15:37.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b943aaaf:::benchmark_data_smithi159_80260_object83080:head# 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b944bb40:::benchmark_data_smithi159_80260_object72175:head# 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b94514b9:::benchmark_data_smithi159_80260_object78116:head# 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b94c3e75:::benchmark_data_smithi159_80260_object85971:head# 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9532218:::benchmark_data_smithi159_80260_object78091:head# 2024-04-15T18:15:37.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b953900e:::benchmark_data_smithi159_80260_object84556:head# 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9588af0:::benchmark_data_smithi159_80260_object75217:head# 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b958e866:::benchmark_data_smithi159_80260_object73598:head# 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b95bb733:::benchmark_data_smithi159_80260_object75980:head# 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b95d74d8:::benchmark_data_smithi159_80260_object86624:head# 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.037 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b966e5ad:::benchmark_data_smithi159_80260_object71017:head# 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b968f7a1:::benchmark_data_smithi159_80260_object76207:head# 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b96b3ddc:::benchmark_data_smithi159_80260_object84104:head# 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b96c958e:::benchmark_data_smithi159_80260_object72100:head# 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b96d084e:::benchmark_data_smithi159_80260_object69690:head# 2024-04-15T18:15:37.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b96ef195:::benchmark_data_smithi159_80260_object71589:head# 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b96f947e:::benchmark_data_smithi159_80260_object84713:head# 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9704ded:::benchmark_data_smithi159_80260_object79859:head# 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.039 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9724ae5:::benchmark_data_smithi159_80260_object85527:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b97ee1ff:::benchmark_data_smithi159_80260_object81013:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9802563:::benchmark_data_smithi159_80260_object81791:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98466eb:::benchmark_data_smithi159_80260_object80439:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b984e24d:::benchmark_data_smithi159_80260_object68853:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b985d00f:::benchmark_data_smithi159_80260_object74446:head# 2024-04-15T18:15:37.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98905ad:::benchmark_data_smithi159_80260_object85014:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98affd3:::benchmark_data_smithi159_80260_object70465:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98e8929:::benchmark_data_smithi159_80260_object88389:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98ea6b7:::benchmark_data_smithi159_80260_object70822:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b98f04ec:::benchmark_data_smithi159_80260_object85202:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b99d0a88:::benchmark_data_smithi159_80260_object76071:head# 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b99f291d:::benchmark_data_smithi159_80260_object78419:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9a285d4:::benchmark_data_smithi159_80260_object87504:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9a7f611:::benchmark_data_smithi159_80260_object79715:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9a8c009:::benchmark_data_smithi159_80260_object76274:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9ade017:::benchmark_data_smithi159_80260_object75080:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9afd176:::benchmark_data_smithi159_80260_object80307:head# 2024-04-15T18:15:37.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9b56e27:::benchmark_data_smithi159_80260_object76236:head# 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9b68424:::benchmark_data_smithi159_80260_object76835:head# 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9b912f4:::benchmark_data_smithi159_80260_object83257:head# 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9be45a8:::benchmark_data_smithi159_80260_object72904:head# 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9c626b0:::benchmark_data_smithi159_80260_object78503:head# 2024-04-15T18:15:37.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9c63f2c:::benchmark_data_smithi159_80260_object81632:head# 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9c70ecc:::benchmark_data_smithi159_80260_object80485:head# 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9c9230f:::benchmark_data_smithi159_80260_object81317:head# 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9cc891d:::benchmark_data_smithi159_80260_object86044:head# 2024-04-15T18:15:37.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9ccc506:::benchmark_data_smithi159_80260_object76455:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9cd876a:::benchmark_data_smithi159_80260_object68591:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9cdaba0:::benchmark_data_smithi159_80260_object84642:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9d160a2:::benchmark_data_smithi159_80260_object79340:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9d1a79a:::benchmark_data_smithi159_80260_object78546:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9dc5dd2:::benchmark_data_smithi159_80260_object86663:head# 2024-04-15T18:15:37.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9dca4df:::benchmark_data_smithi159_80260_object87228:head# 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9e46b8c:::benchmark_data_smithi159_80260_object75196:head# 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9e73c3d:::benchmark_data_smithi159_80260_object84928:head# 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9e8374f:::benchmark_data_smithi159_80260_object82099:head# 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9ed729f:::benchmark_data_smithi159_80260_object84107:head# 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.046 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9f38bab:::benchmark_data_smithi159_80260_object77345:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9fb20a7:::benchmark_data_smithi159_80260_object68747:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9fb89a6:::benchmark_data_smithi159_80260_object70344:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9fd7e5a:::benchmark_data_smithi159_80260_object75279:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:b9ff6c20:::benchmark_data_smithi159_80260_object73113:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba059fae:::benchmark_data_smithi159_80260_object80654:head# 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba0df2ef:::benchmark_data_smithi159_80260_object82668:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba156314:::benchmark_data_smithi159_80260_object88982:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba219d9f:::benchmark_data_smithi159_80260_object75782:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba2787cb:::benchmark_data_smithi159_80260_object86575:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba2f36b4:::benchmark_data_smithi159_80260_object84141:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba304a2e:::benchmark_data_smithi159_80260_object71274:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba32db12:::benchmark_data_smithi159_80260_object84230:head# 2024-04-15T18:15:37.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba351ecc:::benchmark_data_smithi159_80260_object78774:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba377daa:::benchmark_data_smithi159_80260_object72418:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba3807a9:::benchmark_data_smithi159_80260_object69336:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba38e0ac:::benchmark_data_smithi159_80260_object88153:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba3ad026:::benchmark_data_smithi159_80260_object74259:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba3b14a4:::benchmark_data_smithi159_80260_object76443:head# 2024-04-15T18:15:37.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba457244:::benchmark_data_smithi159_80260_object71310:head# 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba4b9993:::benchmark_data_smithi159_80260_object75103:head# 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba4fb327:::benchmark_data_smithi159_80260_object83197:head# 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba4fc8f9:::benchmark_data_smithi159_80260_object70013:head# 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba51151a:::benchmark_data_smithi159_80260_object78673:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba51fb03:::benchmark_data_smithi159_80260_object82218:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba55c2d1:::benchmark_data_smithi159_80260_object86132:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba566499:::benchmark_data_smithi159_80260_object87472:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba58b342:::benchmark_data_smithi159_80260_object74232:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba590cd5:::benchmark_data_smithi159_80260_object84951:head# 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba5a2b6b:::benchmark_data_smithi159_80260_object86937:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba601434:::benchmark_data_smithi159_80260_object87034:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba6420b2:::benchmark_data_smithi159_80260_object80151:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba67e8d4:::benchmark_data_smithi159_80260_object86941:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba68c0e7:::benchmark_data_smithi159_80260_object76481:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba6acd7d:::benchmark_data_smithi159_80260_object86973:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba6d4664:::benchmark_data_smithi159_80260_object86179:head# 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba74ecae:::benchmark_data_smithi159_80260_object82107:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba79ecf8:::benchmark_data_smithi159_80260_object85885:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba7a398e:::benchmark_data_smithi159_80260_object82375:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba7e7cd1:::benchmark_data_smithi159_80260_object76923:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba7ef82a:::benchmark_data_smithi159_80260_object76049:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba7f63cf:::benchmark_data_smithi159_80260_object82629:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba7f9e18:::benchmark_data_smithi159_80260_object71594:head# 2024-04-15T18:15:37.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba8754aa:::benchmark_data_smithi159_80260_object82457:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba895351:::benchmark_data_smithi159_80260_object78989:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba8c3d67:::benchmark_data_smithi159_80260_object80793:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba94c056:::benchmark_data_smithi159_80260_object69157:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba980c4e:::benchmark_data_smithi159_80260_object75432:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba9b3355:::benchmark_data_smithi159_80260_object70443:head# 2024-04-15T18:15:37.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:ba9f04ae:::benchmark_data_smithi159_80260_object75748:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa051ec:::benchmark_data_smithi159_80260_object82250:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa0646d:::benchmark_data_smithi159_80260_object82127:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa192b9:::benchmark_data_smithi159_80260_object71666:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa33656:::benchmark_data_smithi159_80260_object81828:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa3a8ba:::benchmark_data_smithi159_80260_object81445:head# 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baa6bc5e:::benchmark_data_smithi159_80260_object79724:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baad60a6:::benchmark_data_smithi159_80260_object76945:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bab4db6c:::benchmark_data_smithi159_80260_object73397:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bab5d921:::benchmark_data_smithi159_80260_object74637:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bab837f4:::benchmark_data_smithi159_80260_object77426:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:babcff57:::benchmark_data_smithi159_80260_object81956:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:babd1127:::benchmark_data_smithi159_80260_object81284:head# 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:babfb117:::benchmark_data_smithi159_80260_object68881:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bac12417:::benchmark_data_smithi159_80260_object81770:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bac5e6bf:::benchmark_data_smithi159_80260_object78760:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bac8fdd7:::benchmark_data_smithi159_80260_object76605:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baca5bba:::benchmark_data_smithi159_80260_object72958:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bacbb841:::benchmark_data_smithi159_80260_object75434:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bacc5a26:::benchmark_data_smithi159_80260_object70112:head# 2024-04-15T18:15:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bace419c:::benchmark_data_smithi159_80260_object80236:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bad6622c:::benchmark_data_smithi159_80260_object72454:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bad95daa:::benchmark_data_smithi159_80260_object86060:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:badaba57:::benchmark_data_smithi159_80260_object70213:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:badcdcc8:::benchmark_data_smithi159_80260_object73553:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:badcf7f1:::benchmark_data_smithi159_80260_object87301:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bae18762:::benchmark_data_smithi159_80260_object75461:head# 2024-04-15T18:15:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baeba2b2:::benchmark_data_smithi159_80260_object85703:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baf3f90b:::benchmark_data_smithi159_80260_object88524:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:baf96371:::benchmark_data_smithi159_80260_object81286:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bafb47f6:::benchmark_data_smithi159_80260_object73052:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bafd5026:::benchmark_data_smithi159_80260_object81788:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb028b06:::benchmark_data_smithi159_80260_object87791:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb0389b3:::benchmark_data_smithi159_80260_object79997:head# 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb0390d0:::benchmark_data_smithi159_80260_object84266:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb074b80:::benchmark_data_smithi159_80260_object69192:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb0aa2a8:::benchmark_data_smithi159_80260_object68984:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb0bd15b:::benchmark_data_smithi159_80260_object85404:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1143b2:::benchmark_data_smithi159_80260_object85179:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb162d4a:::benchmark_data_smithi159_80260_object83292:head# 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1783f6:::benchmark_data_smithi159_80260_object75120:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1beb1a:::benchmark_data_smithi159_80260_object71756:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1caf82:::benchmark_data_smithi159_80260_object80065:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1cd39f:::benchmark_data_smithi159_80260_object77247:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1d93a6:::benchmark_data_smithi159_80260_object70436:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb1f3511:::benchmark_data_smithi159_80260_object83916:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb2b56ce:::benchmark_data_smithi159_80260_object75499:head# 2024-04-15T18:15:37.061 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb2dc771:::benchmark_data_smithi159_80260_object86131:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb33fe22:::benchmark_data_smithi159_80260_object87762:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb3491a8:::benchmark_data_smithi159_80260_object74086:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb34e4a7:::benchmark_data_smithi159_80260_object70095:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb35a816:::benchmark_data_smithi159_80260_object77652:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb374520:::benchmark_data_smithi159_80260_object87700:head# 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.062 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb379763:::benchmark_data_smithi159_80260_object78958:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb4142f0:::benchmark_data_smithi159_80260_object73253:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb4209a6:::benchmark_data_smithi159_80260_object76676:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb4c026a:::benchmark_data_smithi159_80260_object86999:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb4cf232:::benchmark_data_smithi159_80260_object88248:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb5348f5:::benchmark_data_smithi159_80260_object85911:head# 2024-04-15T18:15:37.063 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb58eeac:::benchmark_data_smithi159_80260_object83434:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb5b1f40:::benchmark_data_smithi159_80260_object82563:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb5fa63f:::benchmark_data_smithi159_80260_object84433:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb6368d1:::benchmark_data_smithi159_80260_object68838:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb66fae9:::benchmark_data_smithi159_80260_object76513:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb72e3dd:::benchmark_data_smithi159_80260_object73027:head# 2024-04-15T18:15:37.064 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb73ff45:::benchmark_data_smithi159_80260_object83334:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb7442aa:::benchmark_data_smithi159_80260_object73850:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb7e2ac0:::benchmark_data_smithi159_80260_object76824:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb803626:::benchmark_data_smithi159_80260_object73706:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb81945e:::benchmark_data_smithi159_80260_object69589:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb81b0d6:::benchmark_data_smithi159_80260_object69078:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb84c1f6:::benchmark_data_smithi159_80260_object69757:head# 2024-04-15T18:15:37.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb858f7a:::benchmark_data_smithi159_80260_object68830:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb89ee80:::benchmark_data_smithi159_80260_object78947:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb8c585b:::benchmark_data_smithi159_80260_object71741:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb8ce154:::benchmark_data_smithi159_80260_object77105:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb8ed53c:::benchmark_data_smithi159_80260_object70628:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb913c8e:::benchmark_data_smithi159_80260_object71876:head# 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb93333a:::benchmark_data_smithi159_80260_object88814:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb978b6b:::benchmark_data_smithi159_80260_object69044:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb9b8a1c:::benchmark_data_smithi159_80260_object69018:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb9cfc23:::benchmark_data_smithi159_80260_object84719:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bb9f1acf:::benchmark_data_smithi159_80260_object88714:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bba4a332:::benchmark_data_smithi159_80260_object87295:head# 2024-04-15T18:15:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bba5d18a:::benchmark_data_smithi159_80260_object74610:head# 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbaa7714:::benchmark_data_smithi159_80260_object78115:head# 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbabda09:::benchmark_data_smithi159_80260_object75975:head# 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbb34e33:::benchmark_data_smithi159_80260_object87774:head# 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbbb8eb9:::benchmark_data_smithi159_80260_object81326:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbbd485b:::benchmark_data_smithi159_80260_object79524:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbc6f446:::benchmark_data_smithi159_80260_object75204:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbc9c83c:::benchmark_data_smithi159_80260_object88655:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbd52326:::benchmark_data_smithi159_80260_object73446:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbd77c78:::benchmark_data_smithi159_80260_object83477:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbd92ff4:::benchmark_data_smithi159_80260_object77298:head# 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbde07cb:::benchmark_data_smithi159_80260_object70740:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbe55c5c:::benchmark_data_smithi159_80260_object72147:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbe85188:::benchmark_data_smithi159_80260_object74212:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbef86ef:::benchmark_data_smithi159_80260_object76576:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbf0b88d:::benchmark_data_smithi159_80260_object84183:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbfb7b4f:::benchmark_data_smithi159_80260_object83405:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbfbd0d9:::benchmark_data_smithi159_80260_object69736:head# 2024-04-15T18:15:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bbff97f1:::benchmark_data_smithi159_80260_object72770:head# 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc0203d6:::benchmark_data_smithi159_80260_object74629:head# 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc02d3a6:::benchmark_data_smithi159_80260_object69145:head# 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc03fe4f:::benchmark_data_smithi159_80260_object78739:head# 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc053aa1:::benchmark_data_smithi159_80260_object81567:head# 2024-04-15T18:15:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc0b398f:::benchmark_data_smithi159_80260_object70679:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc0d1d63:::benchmark_data_smithi159_80260_object87668:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc1077a6:::benchmark_data_smithi159_80260_object81403:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc10f2bd:::benchmark_data_smithi159_80260_object71908:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc13df20:::benchmark_data_smithi159_80260_object83247:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc142963:::benchmark_data_smithi159_80260_object81481:head# 2024-04-15T18:15:37.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc16603b:::benchmark_data_smithi159_80260_object79642:head# 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc1874b0:::benchmark_data_smithi159_80260_object69608:head# 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc21fa53:::benchmark_data_smithi159_80260_object82612:head# 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc2273d0:::benchmark_data_smithi159_80260_object82735:head# 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc246960:::benchmark_data_smithi159_80260_object87542:head# 2024-04-15T18:15:37.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc246ba3:::benchmark_data_smithi159_80260_object83240:head# 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc252ef0:::benchmark_data_smithi159_80260_object70078:head# 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc271577:::benchmark_data_smithi159_80260_object74490:head# 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc2d7fb4:::benchmark_data_smithi159_80260_object81490:head# 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc323226:::benchmark_data_smithi159_80260_object86544:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc34e5f8:::benchmark_data_smithi159_80260_object74197:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc365b31:::benchmark_data_smithi159_80260_object87044:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc3766cb:::benchmark_data_smithi159_80260_object73467:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc38d04a:::benchmark_data_smithi159_80260_object78812:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc4627d7:::benchmark_data_smithi159_80260_object72502:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc4702f6:::benchmark_data_smithi159_80260_object75027:head# 2024-04-15T18:15:37.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc4b42e3:::benchmark_data_smithi159_80260_object73144:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc6203d4:::benchmark_data_smithi159_80260_object86951:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc65a41d:::benchmark_data_smithi159_80260_object82297:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc6a78b4:::benchmark_data_smithi159_80260_object81114:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc6b347e:::benchmark_data_smithi159_80260_object82707:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc6be3a0:::benchmark_data_smithi159_80260_object83983:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc6c23ae:::benchmark_data_smithi159_80260_object86623:head# 2024-04-15T18:15:37.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc7360c6:::benchmark_data_smithi159_80260_object74618:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc757243:::benchmark_data_smithi159_80260_object84324:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc76f4fe:::benchmark_data_smithi159_80260_object73965:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc7b5536:::benchmark_data_smithi159_80260_object78040:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc7ccaa0:::benchmark_data_smithi159_80260_object72997:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc7fc5ce:::benchmark_data_smithi159_80260_object78123:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc862b6c:::benchmark_data_smithi159_80260_object84359:head# 2024-04-15T18:15:37.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc8a46b7:::benchmark_data_smithi159_80260_object72618:head# 2024-04-15T18:15:37.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc8d22f9:::benchmark_data_smithi159_80260_object85052:head# 2024-04-15T18:15:37.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc93e914:::benchmark_data_smithi159_80260_object80763:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc94833c:::benchmark_data_smithi159_80260_object84622:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc96eec0:::benchmark_data_smithi159_80260_object80046:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bc9c8c84:::benchmark_data_smithi159_80260_object83172:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca0156a:::benchmark_data_smithi159_80260_object69638:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca04637:::benchmark_data_smithi159_80260_object76148:head# 2024-04-15T18:15:37.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca08583:::benchmark_data_smithi159_80260_object78415:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca417b2:::benchmark_data_smithi159_80260_object85777:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca5a111:::benchmark_data_smithi159_80260_object84363:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca87dfe:::benchmark_data_smithi159_80260_object72956:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca8da64:::benchmark_data_smithi159_80260_object73914:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bca9828e:::benchmark_data_smithi159_80260_object86529:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcafa1eb:::benchmark_data_smithi159_80260_object81566:head# 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcb1a1fb:::benchmark_data_smithi159_80260_object74966:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcc685cc:::benchmark_data_smithi159_80260_object75260:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcce8972:::benchmark_data_smithi159_80260_object80082:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcd1ca32:::benchmark_data_smithi159_80260_object80918:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcd63aad:::benchmark_data_smithi159_80260_object69218:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcd92d1b:::benchmark_data_smithi159_80260_object87073:head# 2024-04-15T18:15:37.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcdb2219:::benchmark_data_smithi159_80260_object72138:head# 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcdbd516:::benchmark_data_smithi159_80260_object87847:head# 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcdd6d09:::benchmark_data_smithi159_80260_object87702:head# 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcdead63:::benchmark_data_smithi159_80260_object69702:head# 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcebe30c:::benchmark_data_smithi159_80260_object72460:head# 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcec25b5:::benchmark_data_smithi159_80260_object81636:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcec31c9:::benchmark_data_smithi159_80260_object88546:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcf30cb9:::benchmark_data_smithi159_80260_object71805:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcf4979d:::benchmark_data_smithi159_80260_object77153:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcf4f760:::benchmark_data_smithi159_80260_object81293:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bcffbf36:::benchmark_data_smithi159_80260_object80256:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd09046e:::benchmark_data_smithi159_80260_object71732:head# 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd0d375e:::benchmark_data_smithi159_80260_object85123:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd0e1aef:::benchmark_data_smithi159_80260_object68879:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd12de9c:::benchmark_data_smithi159_80260_object88941:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd145b37:::benchmark_data_smithi159_80260_object83188:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd174a59:::benchmark_data_smithi159_80260_object83743:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd1e9144:::benchmark_data_smithi159_80260_object76632:head# 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd1eb74c:::benchmark_data_smithi159_80260_object84759:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd27da8d:::benchmark_data_smithi159_80260_object74001:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd2de512:::benchmark_data_smithi159_80260_object73301:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd30fa3a:::benchmark_data_smithi159_80260_object71693:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd358301:::benchmark_data_smithi159_80260_object70028:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd35e7fb:::benchmark_data_smithi159_80260_object76446:head# 2024-04-15T18:15:37.085 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd374178:::benchmark_data_smithi159_80260_object82361:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd40f475:::benchmark_data_smithi159_80260_object83611:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd48473a:::benchmark_data_smithi159_80260_object73483:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd4a10ba:::benchmark_data_smithi159_80260_object70439:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd4a3662:::benchmark_data_smithi159_80260_object84118:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd502849:::benchmark_data_smithi159_80260_object68758:head# 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.086 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd52d844:::benchmark_data_smithi159_80260_object82396:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd552d25:::benchmark_data_smithi159_80260_object85063:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd5574c4:::benchmark_data_smithi159_80260_object80248:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd59a239:::benchmark_data_smithi159_80260_object68608:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd5d4753:::benchmark_data_smithi159_80260_object71615:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd65b372:::benchmark_data_smithi159_80260_object69670:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd694fcd:::benchmark_data_smithi159_80260_object72957:head# 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd6c6b08:::benchmark_data_smithi159_80260_object86753:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd771cd4:::benchmark_data_smithi159_80260_object83632:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd809bac:::benchmark_data_smithi159_80260_object81581:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd886c5a:::benchmark_data_smithi159_80260_object84404:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd88b55f:::benchmark_data_smithi159_80260_object69220:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd89c43c:::benchmark_data_smithi159_80260_object78660:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd8eb780:::benchmark_data_smithi159_80260_object76844:head# 2024-04-15T18:15:37.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd92468f:::benchmark_data_smithi159_80260_object82570:head# 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd928aa1:::benchmark_data_smithi159_80260_object76210:head# 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd93a194:::benchmark_data_smithi159_80260_object77271:head# 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd9b4885:::benchmark_data_smithi159_80260_object76137:head# 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bd9ef735:::benchmark_data_smithi159_80260_object85169:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda0ae9d:::benchmark_data_smithi159_80260_object86029:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda13717:::benchmark_data_smithi159_80260_object70310:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda1a217:::benchmark_data_smithi159_80260_object81518:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda3b620:::benchmark_data_smithi159_80260_object84378:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda5d47e:::benchmark_data_smithi159_80260_object79898:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda6ec8a:::benchmark_data_smithi159_80260_object82412:head# 2024-04-15T18:15:37.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda7123c:::benchmark_data_smithi159_80260_object73704:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bda7a12c:::benchmark_data_smithi159_80260_object83272:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdac5e7c:::benchmark_data_smithi159_80260_object78023:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdaf46f7:::benchmark_data_smithi159_80260_object86749:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdb1da65:::benchmark_data_smithi159_80260_object80005:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdb3992b:::benchmark_data_smithi159_80260_object80603:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdb4a1b6:::benchmark_data_smithi159_80260_object79116:head# 2024-04-15T18:15:37.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdb4d51d:::benchmark_data_smithi159_80260_object87146:head# 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdb66e1f:::benchmark_data_smithi159_80260_object83493:head# 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdbcc894:::benchmark_data_smithi159_80260_object82736:head# 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdbeaa2b:::benchmark_data_smithi159_80260_object87356:head# 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdbef404:::benchmark_data_smithi159_80260_object74652:head# 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdc3f65c:::benchmark_data_smithi159_80260_object79011:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdc40f39:::benchmark_data_smithi159_80260_object76448:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdc5d05f:::benchmark_data_smithi159_80260_object72945:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdca9684:::benchmark_data_smithi159_80260_object72332:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdcaad5f:::benchmark_data_smithi159_80260_object79803:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdccfd18:::benchmark_data_smithi159_80260_object84372:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdd4db4e:::benchmark_data_smithi159_80260_object85234:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdd5ee68:::benchmark_data_smithi159_80260_object70879:head# 2024-04-15T18:15:37.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdd90d01:::benchmark_data_smithi159_80260_object85389:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bddc1488:::benchmark_data_smithi159_80260_object85682:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bde4634e:::benchmark_data_smithi159_80260_object85760:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bde769f0:::benchmark_data_smithi159_80260_object74743:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdeca39b:::benchmark_data_smithi159_80260_object69504:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdf176cc:::benchmark_data_smithi159_80260_object73534:head# 2024-04-15T18:15:37.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdf1777a:::benchmark_data_smithi159_80260_object73481:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdf210ca:::benchmark_data_smithi159_80260_object70595:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bdf4fff3:::benchmark_data_smithi159_80260_object84596:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be000e9c:::benchmark_data_smithi159_80260_object76239:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be01852d:::benchmark_data_smithi159_80260_object79068:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be035e04:::benchmark_data_smithi159_80260_object76432:head# 2024-04-15T18:15:37.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be04bc4b:::benchmark_data_smithi159_80260_object76819:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be08555c:::benchmark_data_smithi159_80260_object87880:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be086b7a:::benchmark_data_smithi159_80260_object84082:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be0ad699:::benchmark_data_smithi159_80260_object73319:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be14967f:::benchmark_data_smithi159_80260_object72582:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be162656:::benchmark_data_smithi159_80260_object84738:head# 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be170814:::benchmark_data_smithi159_80260_object72728:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be18e86b:::benchmark_data_smithi159_80260_object79475:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be19abfe:::benchmark_data_smithi159_80260_object89081:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be1a5bd5:::benchmark_data_smithi159_80260_object74651:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be1bc97f:::benchmark_data_smithi159_80260_object80647:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be20f835:::benchmark_data_smithi159_80260_object79464:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be227687:::benchmark_data_smithi159_80260_object85098:head# 2024-04-15T18:15:37.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be22fe8c:::benchmark_data_smithi159_80260_object81665:head# 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be230a50:::benchmark_data_smithi159_80260_object71853:head# 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be230d52:::benchmark_data_smithi159_80260_object85441:head# 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be26c12c:::benchmark_data_smithi159_80260_object78624:head# 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be296ffc:::benchmark_data_smithi159_80260_object80644:head# 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be2f2645:::benchmark_data_smithi159_80260_object82367:head# 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3156da:::benchmark_data_smithi159_80260_object74360:head# 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3196c3:::benchmark_data_smithi159_80260_object79330:head# 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3216ae:::benchmark_data_smithi159_80260_object78571:head# 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be38f92a:::benchmark_data_smithi159_80260_object71133:head# 2024-04-15T18:15:37.099 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be39c334:::benchmark_data_smithi159_80260_object75329:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3a12af:::benchmark_data_smithi159_80260_object73584:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3b5085:::benchmark_data_smithi159_80260_object85533:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be3b768c:::benchmark_data_smithi159_80260_object72315:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be402984:::benchmark_data_smithi159_80260_object76407:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be434199:::benchmark_data_smithi159_80260_object87866:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be49e00d:::benchmark_data_smithi159_80260_object78499:head# 2024-04-15T18:15:37.100 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be4b6216:::benchmark_data_smithi159_80260_object74276:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be4d0078:::benchmark_data_smithi159_80260_object77252:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be510253:::benchmark_data_smithi159_80260_object88529:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be5416d3:::benchmark_data_smithi159_80260_object76007:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be5a946f:::benchmark_data_smithi159_80260_object84659:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be62859b:::benchmark_data_smithi159_80260_object79094:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be62e221:::benchmark_data_smithi159_80260_object87713:head# 2024-04-15T18:15:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be63f091:::benchmark_data_smithi159_80260_object74439:head# 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be65cc65:::benchmark_data_smithi159_80260_object80916:head# 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be6a8ccc:::benchmark_data_smithi159_80260_object87981:head# 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be6c4f86:::benchmark_data_smithi159_80260_object78791:head# 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be70b998:::benchmark_data_smithi159_80260_object71304:head# 2024-04-15T18:15:37.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be7137e4:::benchmark_data_smithi159_80260_object84699:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be76322f:::benchmark_data_smithi159_80260_object80516:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be77a126:::benchmark_data_smithi159_80260_object78444:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be79d309:::benchmark_data_smithi159_80260_object87782:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be7d907e:::benchmark_data_smithi159_80260_object84128:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be7fe7cd:::benchmark_data_smithi159_80260_object68976:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be802d94:::benchmark_data_smithi159_80260_object72203:head# 2024-04-15T18:15:37.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be85787f:::benchmark_data_smithi159_80260_object73046:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be858b4e:::benchmark_data_smithi159_80260_object75346:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be86c57a:::benchmark_data_smithi159_80260_object71815:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be8b6d13:::benchmark_data_smithi159_80260_object84272:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be8daefb:::benchmark_data_smithi159_80260_object82082:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be939a04:::benchmark_data_smithi159_80260_object71100:head# 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be969384:::benchmark_data_smithi159_80260_object81900:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be97b07e:::benchmark_data_smithi159_80260_object68995:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be9a3855:::benchmark_data_smithi159_80260_object80888:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be9c4fbb:::benchmark_data_smithi159_80260_object88413:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:be9fa210:::benchmark_data_smithi159_80260_object74130:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bea21840:::benchmark_data_smithi159_80260_object82553:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bea344fd:::benchmark_data_smithi159_80260_object87400:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bea7c2f1:::benchmark_data_smithi159_80260_object75376:head# 2024-04-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:beaab121:::benchmark_data_smithi159_80260_object70708:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bebd98d9:::benchmark_data_smithi159_80260_object76559:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec1fde9:::benchmark_data_smithi159_80260_object81285:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec33f90:::benchmark_data_smithi159_80260_object83360:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec52165:::benchmark_data_smithi159_80260_object85706:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec6161a:::benchmark_data_smithi159_80260_object79435:head# 2024-04-15T18:15:37.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec657e3:::benchmark_data_smithi159_80260_object82220:head# 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec789a9:::benchmark_data_smithi159_80260_object69816:head# 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bec90b0e:::benchmark_data_smithi159_80260_object84769:head# 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:becb4fa0:::benchmark_data_smithi159_80260_object75323:head# 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:becdd90e:::benchmark_data_smithi159_80260_object85505:head# 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:becebde5:::benchmark_data_smithi159_80260_object69464:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:becf8746:::benchmark_data_smithi159_80260_object73115:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bed31b92:::benchmark_data_smithi159_80260_object74311:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bed65db0:::benchmark_data_smithi159_80260_object68707:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bed935bc:::benchmark_data_smithi159_80260_object85141:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bedd1f53:::benchmark_data_smithi159_80260_object85166:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bee02f65:::benchmark_data_smithi159_80260_object71116:head# 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bee0c3f2:::benchmark_data_smithi159_80260_object80467:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bee7ef55:::benchmark_data_smithi159_80260_object83625:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bef413dd:::benchmark_data_smithi159_80260_object71737:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bef415cd:::benchmark_data_smithi159_80260_object73036:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:beff756b:::benchmark_data_smithi159_80260_object68925:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:beff8974:::benchmark_data_smithi159_80260_object88072:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf0135f6:::benchmark_data_smithi159_80260_object78475:head# 2024-04-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf09f503:::benchmark_data_smithi159_80260_object81374:head# 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf0af0fd:::benchmark_data_smithi159_80260_object77554:head# 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf0caa97:::benchmark_data_smithi159_80260_object88213:head# 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf0d0dd8:::benchmark_data_smithi159_80260_object85173:head# 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf100166:::benchmark_data_smithi159_80260_object79903:head# 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.110 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf125a95:::benchmark_data_smithi159_80260_object88887:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf163d8f:::benchmark_data_smithi159_80260_object81857:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf18aec0:::benchmark_data_smithi159_80260_object69110:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf1b5c23:::benchmark_data_smithi159_80260_object77882:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf1c79bb:::benchmark_data_smithi159_80260_object78387:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf1c9c85:::benchmark_data_smithi159_80260_object80076:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf1e1346:::benchmark_data_smithi159_80260_object75082:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf26d345:::benchmark_data_smithi159_80260_object69154:head# 2024-04-15T18:15:37.111 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf290c58:::benchmark_data_smithi159_80260_object74454:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf293fa1:::benchmark_data_smithi159_80260_object75591:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf2d8d14:::benchmark_data_smithi159_80260_object85824:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf313f14:::benchmark_data_smithi159_80260_object78298:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf325315:::benchmark_data_smithi159_80260_object76327:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf37dc9c:::benchmark_data_smithi159_80260_object77134:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf41c463:::benchmark_data_smithi159_80260_object74388:head# 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf441f3c:::benchmark_data_smithi159_80260_object86546:head# 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf478f19:::benchmark_data_smithi159_80260_object76033:head# 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf486751:::benchmark_data_smithi159_80260_object81055:head# 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf4b0ca8:::benchmark_data_smithi159_80260_object84462:head# 2024-04-15T18:15:37.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf4c1847:::benchmark_data_smithi159_80260_object80893:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf507e82:::benchmark_data_smithi159_80260_object75469:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf587de6:::benchmark_data_smithi159_80260_object80715:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf58a55b:::benchmark_data_smithi159_80260_object81070:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf59334e:::benchmark_data_smithi159_80260_object87462:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf5e5297:::benchmark_data_smithi159_80260_object82298:head# 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf5ebf5b:::benchmark_data_smithi159_80260_object68665:head# 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf5f7d05:::benchmark_data_smithi159_80260_object87555:head# 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf5fcb2d:::benchmark_data_smithi159_80260_object83790:head# 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf602e16:::benchmark_data_smithi159_80260_object75435:head# 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf618fdd:::benchmark_data_smithi159_80260_object88600:head# 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf6450ba:::benchmark_data_smithi159_80260_object86109:head# 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf68a897:::benchmark_data_smithi159_80260_object83985:head# 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf6e289f:::benchmark_data_smithi159_80260_object75623:head# 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf721416:::benchmark_data_smithi159_80260_object73782:head# 2024-04-15T18:15:37.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf73adfb:::benchmark_data_smithi159_80260_object86114:head# 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf73c552:::benchmark_data_smithi159_80260_object77378:head# 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf77b35b:::benchmark_data_smithi159_80260_object82625:head# 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf78553f:::benchmark_data_smithi159_80260_object79146:head# 2024-04-15T18:15:37.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf78ebd6:::benchmark_data_smithi159_80260_object83291:head# 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf7a5091:::benchmark_data_smithi159_80260_object89011:head# 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf829497:::benchmark_data_smithi159_80260_object88218:head# 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf85d7cb:::benchmark_data_smithi159_80260_object85276:head# 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf894138:::benchmark_data_smithi159_80260_object82062:head# 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf8c3470:::benchmark_data_smithi159_80260_object75947:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf8f2c6f:::benchmark_data_smithi159_80260_object87606:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf90ce22:::benchmark_data_smithi159_80260_object78322:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf93f5ae:::benchmark_data_smithi159_80260_object71201:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf959e74:::benchmark_data_smithi159_80260_object85288:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf95c3c1:::benchmark_data_smithi159_80260_object84812:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf991e4f:::benchmark_data_smithi159_80260_object73653:head# 2024-04-15T18:15:37.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf9b6791:::benchmark_data_smithi159_80260_object75074:head# 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf9c89f8:::benchmark_data_smithi159_80260_object74747:head# 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf9c9d7b:::benchmark_data_smithi159_80260_object73995:head# 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf9f34f3:::benchmark_data_smithi159_80260_object81446:head# 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.120 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bf9f8e27:::benchmark_data_smithi159_80260_object81698:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfa49c00:::benchmark_data_smithi159_80260_object88924:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfa59f23:::benchmark_data_smithi159_80260_object88575:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfb309c5:::benchmark_data_smithi159_80260_object82005:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfb9a8ff:::benchmark_data_smithi159_80260_object78468:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfbb8cbd:::benchmark_data_smithi159_80260_object70625:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfbcaaa1:::benchmark_data_smithi159_80260_object71369:head# 2024-04-15T18:15:37.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfbd4ffa:::benchmark_data_smithi159_80260_object87873:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfbeaa03:::benchmark_data_smithi159_80260_object78700:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfc084f7:::benchmark_data_smithi159_80260_object70944:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfc14e58:::benchmark_data_smithi159_80260_object69001:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfc20119:::benchmark_data_smithi159_80260_object76285:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfc3c4dd:::benchmark_data_smithi159_80260_object86809:head# 2024-04-15T18:15:37.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfcc1c12:::benchmark_data_smithi159_80260_object70724:head# 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfcc4e76:::benchmark_data_smithi159_80260_object83739:head# 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfd1ead8:::benchmark_data_smithi159_80260_object75025:head# 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfd1f85f:::benchmark_data_smithi159_80260_object80391:head# 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfd461d8:::benchmark_data_smithi159_80260_object77185:head# 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfd982e3:::benchmark_data_smithi159_80260_object69786:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfdc9f08:::benchmark_data_smithi159_80260_object69167:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfe30a16:::benchmark_data_smithi159_80260_object88369:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfe324fb:::benchmark_data_smithi159_80260_object80390:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfe4d0f5:::benchmark_data_smithi159_80260_object79723:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfe97036:::benchmark_data_smithi159_80260_object86695:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bfedbe6e:::benchmark_data_smithi159_80260_object69325:head# 2024-04-15T18:15:37.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bff525b4:::benchmark_data_smithi159_80260_object86747:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bff72b06:::benchmark_data_smithi159_80260_object82921:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bff7b5b7:::benchmark_data_smithi159_80260_object74764:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bff8a53e:::benchmark_data_smithi159_80260_object78103:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bff8c069:::benchmark_data_smithi159_80260_object83979:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #2:bffffe29:::benchmark_data_smithi159_80260_object85466:head# 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:15:37.125 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 177 info 2.d( v 177'10080 (0'0,177'10080] local-lis/les=114/115 n=1316 ec=114/114 lis/c=114/114 les/c/f=115/115/0 sis=114) 2024-04-15T18:15:37.126 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-04-15T18:15:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.127 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:37.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:37.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:37.260 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:37.281 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:37.385 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:37.416 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:37.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:37.436 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:37.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:37.570 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:37.590 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:37.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:37.723 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:37.742 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:37.846 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:37.878 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:37.900 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:37.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:38.004 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:38.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:38.055 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:38.057 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' created 2024-04-15T18:15:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:38.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:38.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:38 smithi159 ceph-mon[45336]: pgmap v562: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:38.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-15T18:15:38.159 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:38.190 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:38.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:38.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:38.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:38 smithi082 ceph-mon[44206]: pgmap v562: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-15T18:15:38.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:38.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:38 smithi049 ceph-mon[40087]: pgmap v562: 1 pgs: 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:38.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-15T18:15:38.348 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:38.599 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:38.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:38.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:38.735 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:38.757 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:38.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:38.852 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:15:38.853 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:15:38.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:38.891 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:38.910 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:38.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:39.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:39.043 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.066 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:39.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:39.202 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.223 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:39.327 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:39.354 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-15T18:15:39.354 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:39 smithi082 ceph-mon[44206]: osdmap e199: 12 total, 11 up, 11 in 2024-04-15T18:15:39.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:39 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:39.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:39 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:39.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.376 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-15T18:15:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:39 smithi159 ceph-mon[45336]: osdmap e199: 12 total, 11 up, 11 in 2024-04-15T18:15:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:39 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:39.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:39 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:39.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3280981926' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-15T18:15:39.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:39 smithi049 ceph-mon[40087]: osdmap e199: 12 total, 11 up, 11 in 2024-04-15T18:15:39.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:39 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:39.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:39 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:39.480 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:39.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-15T18:15:39.510 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.530 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:39.634 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:39.664 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.685 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:39.790 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:39.822 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.844 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:39.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:39.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:39.978 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:39.998 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:40.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:40 smithi159 ceph-mon[45336]: osdmap e200: 12 total, 11 up, 11 in 2024-04-15T18:15:40.070 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:40 smithi159 ceph-mon[45336]: pgmap v565: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:40.102 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:40.134 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.155 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:40.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/exp.2.d.3 2024-04-15T18:15:40.209 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-15T18:15:40.209 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:15:40.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:15:40.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:40.288 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:40.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:40 smithi049 ceph-mon[40087]: osdmap e200: 12 total, 11 up, 11 in 2024-04-15T18:15:40.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:40 smithi049 ceph-mon[40087]: pgmap v565: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:40.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:40.412 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:40.445 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.466 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:40.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:40 smithi082 ceph-mon[44206]: osdmap e200: 12 total, 11 up, 11 in 2024-04-15T18:15:40.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:40 smithi082 ceph-mon[44206]: pgmap v565: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 883 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:40.570 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:40.599 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.619 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:40.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:40.755 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.776 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:40.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:15:40.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:40.907 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:40.929 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:40.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:41.034 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:41.066 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:41.088 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:41.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:41.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:41.224 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:41.244 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:41.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:41.282 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:41.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:41.375 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:41 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:15:41.375 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:41 smithi082 ceph-mon[44206]: osdmap e201: 12 total, 11 up, 11 in 2024-04-15T18:15:41.377 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:41.397 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:41.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:41.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:41 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:15:41.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:41 smithi159 ceph-mon[45336]: osdmap e201: 12 total, 11 up, 11 in 2024-04-15T18:15:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:41 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:15:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:41 smithi049 ceph-mon[40087]: osdmap e201: 12 total, 11 up, 11 in 2024-04-15T18:15:41.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:42.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:42 smithi159 ceph-mon[45336]: pgmap v567: 17 pgs: 2 creating+activating, 3 unknown, 12 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:42.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:42 smithi049 ceph-mon[40087]: pgmap v567: 17 pgs: 2 creating+activating, 3 unknown, 12 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:42.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:42 smithi082 ceph-mon[44206]: pgmap v567: 17 pgs: 2 creating+activating, 3 unknown, 12 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:44.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:44 smithi159 ceph-mon[45336]: pgmap v568: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:44.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:44 smithi049 ceph-mon[40087]: pgmap v568: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:44.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:44 smithi082 ceph-mon[44206]: pgmap v568: 17 pgs: 2 creating+activating, 15 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:44.819 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:44.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:44.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:44.957 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:44.979 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:44.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:45.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:45.117 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:45.217 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:45.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:45.322 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:45.354 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:45.375 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:45.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:45.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:45.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:45.562 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:45.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:45.666 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:15:45.667 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:15:45.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:45.712 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:46.151 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-15T18:15:46.240 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:46.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:46.345 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:46.377 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:46.398 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:46.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:46 smithi159 ceph-mon[45336]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:46.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:15:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:46.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:46 smithi049 ceph-mon[40087]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:15:46.503 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:46.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:46 smithi082 ceph-mon[44206]: pgmap v569: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:46.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:15:46.534 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:46.556 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:46.661 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:46.691 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:46.710 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:46.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:46.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:46.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:46.971 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:47.001 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.021 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:47.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:47.158 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.180 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:47.285 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:47.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:15:47.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:47 smithi082 ceph-mon[44206]: osdmap e202: 12 total, 11 up, 11 in 2024-04-15T18:15:47.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:47.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:47.315 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.336 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:15:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:47 smithi159 ceph-mon[45336]: osdmap e202: 12 total, 11 up, 11 in 2024-04-15T18:15:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:47.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:47.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:47.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1118696657' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:15:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:47 smithi049 ceph-mon[40087]: osdmap e202: 12 total, 11 up, 11 in 2024-04-15T18:15:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:47.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:47.469 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.522 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:47.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:47.662 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.683 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:47.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:47.819 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:47.840 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:47.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:47.874 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-15T18:15:47.886 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:15:47.886 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:15:47.940 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:15:47.940 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:15:47.940 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_91621 2024-04-15T18:15:47.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:15:47.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:15:47.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:47.987 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.047 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:48.162 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:48.193 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.217 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:48.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:48.349 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:48.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:48 smithi049 ceph-mon[40087]: pgmap v571: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:48.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:48 smithi049 ceph-mon[40087]: osdmap e203: 12 total, 11 up, 11 in 2024-04-15T18:15:48.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:48 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:15:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:48 smithi159 ceph-mon[45336]: pgmap v571: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:48 smithi159 ceph-mon[45336]: osdmap e203: 12 total, 11 up, 11 in 2024-04-15T18:15:48.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:48 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:15:48.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:15:48.475 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:48.502 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.521 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:48.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:48 smithi082 ceph-mon[44206]: pgmap v571: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:48 smithi082 ceph-mon[44206]: osdmap e203: 12 total, 11 up, 11 in 2024-04-15T18:15:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:48 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:15:48.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:48.645 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:48.654 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.676 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:48.733 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:15:48.761 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:15:48.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:48.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:15:48.813 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.820 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:15:48.837 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:48.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:48.852 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:15:48.882 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:15:48.914 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:15:48.940 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 1711 1695 105.932 105.938 0.00398111 0.00938453 2024-04-15T18:15:48.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:15:48.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:15:48.976 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:48.980 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:15:49.001 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:15:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:49.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:15:49.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:15:49.041 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:15:49.042 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:15:49.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:49.142 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/734480011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:49.170 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:15:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/734480011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:49.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:49.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:15:49.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/734480011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:15:49.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:49.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 3181 3165 98.9001 91.875 0.00303247 0.0100347 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: pgmap v573: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:50.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:50.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:50.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:50 smithi159 ceph-mon[45336]: osdmap e204: 12 total, 11 up, 11 in 2024-04-15T18:15:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: pgmap v573: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:50.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:50 smithi049 ceph-mon[40087]: osdmap e204: 12 total, 11 up, 11 in 2024-04-15T18:15:50.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: pgmap v573: 17 pgs: 17 active+clean; 577 KiB data, 882 MiB used, 983 GiB / 983 GiB avail 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1216369195' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:15:50.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:50 smithi082 ceph-mon[44206]: osdmap e204: 12 total, 11 up, 11 in 2024-04-15T18:15:50.598 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:50.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:50.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:50.739 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:50.762 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:50.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:50.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:15:50.896 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:15:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:50.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:15:50.940 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 5277 5261 109.597 131 0.00656949 0.00903346 2024-04-15T18:15:51.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:51.048 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:51.066 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:51.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:15:51.097 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:15:51.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:51.197 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:51.698 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:51.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:51.705 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:51.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:51.839 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:51.864 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:51.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:51.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 6949 6933 108.321 104.5 0.00669772 0.00921336 2024-04-15T18:15:51.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:52.002 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.154 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:52 smithi049 ceph-mon[40087]: pgmap v575: 17 pgs: 17 active+clean; 182 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-04-15T18:15:52.260 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:52.289 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.310 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:52.414 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:52.441 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:52 smithi082 ceph-mon[44206]: pgmap v575: 17 pgs: 17 active+clean; 182 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-04-15T18:15:52.444 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.465 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:52.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:15:52.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:52.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:52 smithi159 ceph-mon[45336]: pgmap v575: 17 pgs: 17 active+clean; 182 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-04-15T18:15:52.595 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.615 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:52.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:52.748 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.771 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:52.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:52.908 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:52.933 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:52.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:52.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 8335 8319 103.979 86.625 0.00832481 0.00958643 2024-04-15T18:15:53.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:53.064 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:53.087 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:53.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:53.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:15:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:15:53.150 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:15:53.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:53.223 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:15:53.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:15:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:15:53.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:15:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:15:53.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:15:53.839 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:53.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 10087 10071 104.898 109.5 0.0477884 0.00951341 2024-04-15T18:15:54.176 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:54.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:54.279 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:15:54.306 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:54.329 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:15:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:54.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:15:54.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:54.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:54 smithi049 ceph-mon[40087]: pgmap v576: 17 pgs: 17 active+clean; 265 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 38 MiB/s wr, 610 op/s 2024-04-15T18:15:54.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:54 smithi049 ceph-mon[40087]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:15:54.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:54 smithi049 ceph-mon[40087]: osdmap e205: 12 total, 11 up, 11 in 2024-04-15T18:15:54.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:54 smithi082 ceph-mon[44206]: pgmap v576: 17 pgs: 17 active+clean; 265 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 38 MiB/s wr, 610 op/s 2024-04-15T18:15:54.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:54 smithi082 ceph-mon[44206]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:15:54.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:54 smithi082 ceph-mon[44206]: osdmap e205: 12 total, 11 up, 11 in 2024-04-15T18:15:54.515 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:54.538 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:54.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:54.631 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:15:54.631 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:15:54.631 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:15:54.632 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:15:54.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:54 smithi159 ceph-mon[45336]: pgmap v576: 17 pgs: 17 active+clean; 265 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 38 MiB/s wr, 610 op/s 2024-04-15T18:15:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:54 smithi159 ceph-mon[45336]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:15:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:54 smithi159 ceph-mon[45336]: osdmap e205: 12 total, 11 up, 11 in 2024-04-15T18:15:54.684 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:54.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 11759 11743 104.839 104.5 0.0134781 0.0093946 2024-04-15T18:15:55.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:55.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:55.268 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:55.288 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:55.393 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:55.424 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:55.446 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:55.549 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:15:55.576 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:55.593 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:15:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:15:55.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:55.726 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:55.750 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:55.855 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:15:55.894 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:55.918 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:15:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:55.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:15:55.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:15:55.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 13029 13013 101.655 79.375 0.00871387 0.0098258 2024-04-15T18:15:56.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:56.054 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:56.403 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:56.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:56.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:56 smithi049 ceph-mon[40087]: pgmap v578: 17 pgs: 17 active+clean; 500 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:15:56.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:56.539 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:56 smithi082 ceph-mon[44206]: pgmap v578: 17 pgs: 17 active+clean; 500 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:15:56.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:56 smithi159 ceph-mon[45336]: pgmap v578: 17 pgs: 17 active+clean; 500 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:15:56.894 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:56.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:56.931 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-04-15T18:15:53.086653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278331936,"num_objects":4240,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":4419,"num_write_kb":274170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4375,"ondisk_log_size":4375,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1738720,"kb_used_data":819700,"kb_used_omap":17,"kb_used_meta":918894,"kb_avail":1029457952,"statfs":{"total":1055945392128,"available":1054164942848,"internally_reserved":0,"allocated":839372800,"data_stored":836253223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17550,"internal_metadata":940948338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":277741568,"num_objects":4238,"num_object_clones":0,"num_object_copies":12714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":271232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.941110"},"pg_stats":[{"pgid":"3.f","version":"204'348","reported_seq":365,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.990066+0000","last_change":"2024-04-15T18:15:39.057796+0000","last_active":"2024-04-15T18:15:50.990066+0000","last_peered":"2024-04-15T18:15:50.990066+0000","last_clean":"2024-04-15T18:15:50.990066+0000","last_became_active":"2024-04-15T18:15:39.057408+0000","last_became_peered":"2024-04-15T18:15:39.057408+0000","last_unstale":"2024-04-15T18:15:50.990066+0000","last_undegraded":"2024-04-15T18:15:50.990066+0000","last_fullsized":"2024-04-15T18:15:50.990066+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:02.741971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"204'168","reported_seq":188,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.611592+0000","last_change":"2024-04-15T18:15:39.061855+0000","last_active":"2024-04-15T18:15:49.611592+0000","last_peered":"2024-04-15T18:15:49.611592+0000","last_clean":"2024-04-15T18:15:49.611592+0000","last_became_active":"2024-04-15T18:15:39.061483+0000","last_became_peered":"2024-04-15T18:15:39.061483+0000","last_unstale":"2024-04-15T18:15:49.611592+0000","last_undegraded":"2024-04-15T18:15:49.611592+0000","last_fullsized":"2024-04-15T18:15:49.611592+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:00.740160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11010048,"num_objects":168,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":10752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"204'339","reported_seq":359,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.988428+0000","last_change":"2024-04-15T18:15:39.059260+0000","last_active":"2024-04-15T18:15:50.988428+0000","last_peered":"2024-04-15T18:15:50.988428+0000","last_clean":"2024-04-15T18:15:50.988428+0000","last_became_active":"2024-04-15T18:15:39.058838+0000","last_became_peered":"2024-04-15T18:15:39.058838+0000","last_unstale":"2024-04-15T18:15:50.988428+0000","last_undegraded":"2024-04-15T18:15:50.988428+0000","last_fullsized":"2024-04-15T18:15:50.988428+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:40.502923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":339,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"204'152","reported_seq":172,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.395687+0000","last_change":"2024-04-15T18:15:39.436047+0000","last_active":"2024-04-15T18:15:49.395687+0000","last_peered":"2024-04-15T18:15:49.395687+0000","last_clean":"2024-04-15T18:15:49.395687+0000","last_became_active":"2024-04-15T18:15:39.435717+0000","last_became_peered":"2024-04-15T18:15:39.435717+0000","last_unstale":"2024-04-15T18:15:49.395687+0000","last_undegraded":"2024-04-15T18:15:49.395687+0000","last_fullsized":"2024-04-15T18:15:49.395687+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:58.275182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":9728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,10],"acting":[5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"204'250","reported_seq":269,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.382332+0000","last_change":"2024-04-15T18:15:39.059576+0000","last_active":"2024-04-15T18:15:50.382332+0000","last_peered":"2024-04-15T18:15:50.382332+0000","last_clean":"2024-04-15T18:15:50.382332+0000","last_became_active":"2024-04-15T18:15:39.059365+0000","last_became_peered":"2024-04-15T18:15:39.059365+0000","last_unstale":"2024-04-15T18:15:50.382332+0000","last_undegraded":"2024-04-15T18:15:50.382332+0000","last_fullsized":"2024-04-15T18:15:50.382332+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.660175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":250,"num_write_kb":16000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"204'205","reported_seq":225,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.877850+0000","last_change":"2024-04-15T18:15:39.058448+0000","last_active":"2024-04-15T18:15:49.877850+0000","last_peered":"2024-04-15T18:15:49.877850+0000","last_clean":"2024-04-15T18:15:49.877850+0000","last_became_active":"2024-04-15T18:15:39.057839+0000","last_became_peered":"2024-04-15T18:15:39.057839+0000","last_unstale":"2024-04-15T18:15:49.877850+0000","last_undegraded":"2024-04-15T18:15:49.877850+0000","last_fullsized":"2024-04-15T18:15:49.877850+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.597994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":13120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"204'240","reported_seq":260,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.340645+0000","last_change":"2024-04-15T18:15:39.059016+0000","last_active":"2024-04-15T18:15:50.340645+0000","last_peered":"2024-04-15T18:15:50.340645+0000","last_clean":"2024-04-15T18:15:50.340645+0000","last_became_active":"2024-04-15T18:15:39.058704+0000","last_became_peered":"2024-04-15T18:15:39.058704+0000","last_unstale":"2024-04-15T18:15:50.340645+0000","last_undegraded":"2024-04-15T18:15:50.340645+0000","last_fullsized":"2024-04-15T18:15:50.340645+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.497471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":15360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"204'183","reported_seq":203,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.605526+0000","last_change":"2024-04-15T18:15:39.058616+0000","last_active":"2024-04-15T18:15:49.605526+0000","last_peered":"2024-04-15T18:15:49.605526+0000","last_clean":"2024-04-15T18:15:49.605526+0000","last_became_active":"2024-04-15T18:15:39.058285+0000","last_became_peered":"2024-04-15T18:15:39.058285+0000","last_unstale":"2024-04-15T18:15:49.605526+0000","last_undegraded":"2024-04-15T18:15:49.605526+0000","last_fullsized":"2024-04-15T18:15:49.605526+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:38.332807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":11712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"204'189","reported_seq":209,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.640037+0000","last_change":"2024-04-15T18:15:39.062153+0000","last_active":"2024-04-15T18:15:49.640037+0000","last_peered":"2024-04-15T18:15:49.640037+0000","last_clean":"2024-04-15T18:15:49.640037+0000","last_became_active":"2024-04-15T18:15:39.061741+0000","last_became_peered":"2024-04-15T18:15:39.061741+0000","last_unstale":"2024-04-15T18:15:49.640037+0000","last_undegraded":"2024-04-15T18:15:49.640037+0000","last_fullsized":"2024-04-15T18:15:49.640037+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:03.899612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12386304,"num_objects":189,"num_object_clones":0,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":189,"num_write_kb":12096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"204'256","reported_seq":275,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.381664+0000","last_change":"2024-04-15T18:15:39.058580+0000","last_active":"2024-04-15T18:15:50.381664+0000","last_peered":"2024-04-15T18:15:50.381664+0000","last_clean":"2024-04-15T18:15:50.381664+0000","last_became_active":"2024-04-15T18:15:39.058216+0000","last_became_peered":"2024-04-15T18:15:39.058216+0000","last_unstale":"2024-04-15T18:15:50.381664+0000","last_undegraded":"2024-04-15T18:15:50.381664+0000","last_fullsized":"2024-04-15T18:15:50.381664+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:57.648006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":256,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":256,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"204'167","reported_seq":187,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.398178+0000","last_change":"2024-04-15T18:15:39.670253+0000","last_active":"2024-04-15T18:15:49.398178+0000","last_peered":"2024-04-15T18:15:49.398178+0000","last_clean":"2024-04-15T18:15:49.398178+0000","last_became_active":"2024-04-15T18:15:39.669933+0000","last_became_peered":"2024-04-15T18:15:39.669933+0000","last_unstale":"2024-04-15T18:15:49.398178+0000","last_undegraded":"2024-04-15T18:15:49.398178+0000","last_fullsized":"2024-04-15T18:15:49.398178+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:51.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"204'395","reported_seq":415,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:51.888901+0000","last_change":"2024-04-15T18:15:39.057945+0000","last_active":"2024-04-15T18:15:51.888901+0000","last_peered":"2024-04-15T18:15:51.888901+0000","last_clean":"2024-04-15T18:15:51.888901+0000","last_became_active":"2024-04-15T18:15:39.057159+0000","last_became_peered":"2024-04-15T18:15:39.057159+0000","last_unstale":"2024-04-15T18:15:51.888901+0000","last_undegraded":"2024-04-15T18:15:51.888901+0000","last_fullsized":"2024-04-15T18:15:51.888901+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:38.065390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":395,"num_object_clones":0,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":395,"num_write_kb":25280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"204'481","reported_seq":501,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:52.561920+0000","last_change":"2024-04-15T18:15:39.060876+0000","last_active":"2024-04-15T18:15:52.561920+0000","last_peered":"2024-04-15T18:15:52.561920+0000","last_clean":"2024-04-15T18:15:52.561920+0000","last_became_active":"2024-04-15T18:15:39.060320+0000","last_became_peered":"2024-04-15T18:15:39.060320+0000","last_unstale":"2024-04-15T18:15:52.561920+0000","last_undegraded":"2024-04-15T18:15:52.561920+0000","last_fullsized":"2024-04-15T18:15:52.561920+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:52.602066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522816,"num_objects":481,"num_object_clones":0,"num_object_copies":1443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":481,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"204'228","reported_seq":247,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:50.374716+0000","last_change":"2024-04-15T18:15:39.435414+0000","last_active":"2024-04-15T18:15:50.374716+0000","last_peered":"2024-04-15T18:15:50.374716+0000","last_clean":"2024-04-15T18:15:50.374716+0000","last_became_active":"2024-04-15T18:15:39.435165+0000","last_became_peered":"2024-04-15T18:15:39.435165+0000","last_unstale":"2024-04-15T18:15:50.374716+0000","last_undegraded":"2024-04-15T18:15:50.374716+0000","last_fullsized":"2024-04-15T18:15:50.374716+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:06.373700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":228,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"204'184","reported_seq":203,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.629890+0000","last_change":"2024-04-15T18:15:39.061919+0000","last_active":"2024-04-15T18:15:49.629890+0000","last_peered":"2024-04-15T18:15:49.629890+0000","last_clean":"2024-04-15T18:15:49.629890+0000","last_became_active":"2024-04-15T18:15:39.061499+0000","last_became_peered":"2024-04-15T18:15:39.061499+0000","last_unstale":"2024-04-15T18:15:49.629890+0000","last_undegraded":"2024-04-15T18:15:49.629890+0000","last_fullsized":"2024-04-15T18:15:49.629890+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:05.557790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":184,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"204'453","reported_seq":473,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:52.312297+0000","last_change":"2024-04-15T18:15:39.059346+0000","last_active":"2024-04-15T18:15:52.312297+0000","last_peered":"2024-04-15T18:15:52.312297+0000","last_clean":"2024-04-15T18:15:52.312297+0000","last_became_active":"2024-04-15T18:15:39.059050+0000","last_became_peered":"2024-04-15T18:15:39.059050+0000","last_unstale":"2024-04-15T18:15:52.312297+0000","last_undegraded":"2024-04-15T18:15:52.312297+0000","last_fullsized":"2024-04-15T18:15:52.312297+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:59.305776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":453,"num_write_kb":28992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":593,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-04-15T18:15:49.163334+0000","last_change":"2024-04-15T18:15:04.226208+0000","last_active":"2024-04-15T18:15:49.163334+0000","last_peered":"2024-04-15T18:15:49.163334+0000","last_clean":"2024-04-15T18:15:49.163334+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:15:49.163334+0000","last_undegraded":"2024-04-15T18:15:49.163334+0000","last_fullsized":"2024-04-15T18:15:49.163334+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:15:04.226084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:15:04.226084+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:29.514876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":277741568,"num_objects":4238,"num_object_clones":0,"num_object_copies":12714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":271232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991625216,"data_stored":991625216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4238,"ondisk_log_size":4238,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145380,"kb_used_data":73944,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93599772,"statfs":{"total":95995035648,"available":95846166528,"internally_reserved":0,"allocated":75718656,"data_stored":75436217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187541,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126116,"kb_used_data":54684,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93619036,"statfs":{"total":95995035648,"available":95865892864,"internally_reserved":0,"allocated":55996416,"data_stored":55712953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514586,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182372,"kb_used_data":74844,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93562780,"statfs":{"total":95995035648,"available":95808286720,"internally_reserved":0,"allocated":76640256,"data_stored":76356793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118696,"kb_used_data":25052,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93626456,"statfs":{"total":95995035648,"available":95873490944,"internally_reserved":0,"allocated":25653248,"data_stored":25369785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168675,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158060,"kb_used_data":60260,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93587092,"statfs":{"total":95995035648,"available":95833182208,"internally_reserved":0,"allocated":61706240,"data_stored":61419225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":100137418},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463016,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102120,"kb_used_data":48728,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93643032,"statfs":{"total":95995035648,"available":95890464768,"internally_reserved":0,"allocated":49897472,"data_stored":49615053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217828,"kb_used_data":110300,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93527324,"statfs":{"total":95995035648,"available":95771979776,"internally_reserved":0,"allocated":112947200,"data_stored":112663737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176932,"kb_used_data":79132,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93568220,"statfs":{"total":95995035648,"available":95813857280,"internally_reserved":0,"allocated":81031168,"data_stored":80747705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123044,"kb_used_data":29404,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93622108,"statfs":{"total":95995035648,"available":95869038592,"internally_reserved":0,"allocated":30109696,"data_stored":29826233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294372,"kb_used_data":222940,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93450780,"statfs":{"total":95995035648,"available":95693598720,"internally_reserved":0,"allocated":228290560,"data_stored":228007097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724652,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93800,"kb_used_data":40412,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93651352,"statfs":{"total":95995035648,"available":95898984448,"internally_reserved":0,"allocated":41381888,"data_stored":41098425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":96141312,"data_stored":96141312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113508352,"data_stored":113508352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34078720,"data_stored":34078720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":256507904,"data_stored":256507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":42074112,"data_stored":42074112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":77332480,"data_stored":77332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":89063424,"data_stored":89063424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32178176,"data_stored":32178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":93585408,"data_stored":93585408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":64946176,"data_stored":64946176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":92209152,"data_stored":92209152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:15:56.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:15:56.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 14596 14580 101.241 97.9375 0.00632865 0.00985862 2024-04-15T18:15:56.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:57.032 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:57.057 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:57.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:57.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:57.195 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:57.220 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:57.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:57.323 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:15:57.357 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:57.382 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:15:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:57.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:15:57.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:57.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:57.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:57.528 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:57.731 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:15:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:57.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:15:57.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:57.871 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:57.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 16062 16046 100.278 91.625 0.00378997 0.00996492 2024-04-15T18:15:58.065 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:15:58.076 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:58.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:58.120 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:58 smithi049 ceph-mon[40087]: pgmap v579: 17 pgs: 17 active+clean; 719 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-15T18:15:58.120 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2314147034' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:15:58.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:58.229 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:15:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:58.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:15:58.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:58.366 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:58.398 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:58.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:58 smithi159 ceph-mon[45336]: pgmap v579: 17 pgs: 17 active+clean; 719 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-15T18:15:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2314147034' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:15:58.533 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:58 smithi082 ceph-mon[44206]: pgmap v579: 17 pgs: 17 active+clean; 719 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-04-15T18:15:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2314147034' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:15:58.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 15 17163 17148 97.4223 68.875 0.054082 0.0102468 2024-04-15T18:15:59.111 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:15:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:59.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:15:59.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:15:59.246 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:59.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:59 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:59.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:15:59.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:15:59 smithi159 ceph-mon[45336]: osdmap e206: 12 total, 11 up, 11 in 2024-04-15T18:15:59.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:59 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:59.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:15:59.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:15:59 smithi049 ceph-mon[40087]: osdmap e206: 12 total, 11 up, 11 in 2024-04-15T18:15:59.490 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:15:59.500 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:15:59.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:59.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:15:59.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:59 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:15:59.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:15:59.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:15:59 smithi082 ceph-mon[44206]: osdmap e206: 12 total, 11 up, 11 in 2024-04-15T18:15:59.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:15:59.633 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:59.652 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:15:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:15:59.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:15:59.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:15:59.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:15:59.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 18098 18082 94.168 58.375 0.0258845 0.0105612 2024-04-15T18:16:00.134 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:00.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:00.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:00.271 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:00.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:00 smithi159 ceph-mon[45336]: pgmap v581: 17 pgs: 17 active+clean; 781 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:00 smithi049 ceph-mon[40087]: pgmap v581: 17 pgs: 17 active+clean; 781 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:00.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:00 smithi082 ceph-mon[44206]: pgmap v581: 17 pgs: 17 active+clean; 781 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:00.622 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:00.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:00.727 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:00.757 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:00.777 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:00.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:00.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:00.909 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:00.941 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:00.941 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 19595 19579 94.1209 93.5625 0.0021242 0.0106121 2024-04-15T18:16:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:00.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:01.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:01.077 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:01.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:16:01.279 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:01.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:01.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:01.418 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:01.442 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:01.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:01.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:01.579 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:01.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:01.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:01.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:01.746 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:01.770 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:01.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:01.874 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:01.910 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:01.939 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:01.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 21178 21162 94.4644 98.9375 0.00319999 0.0105799 2024-04-15T18:16:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:01.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:02.044 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:02.075 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:02.099 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:02.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:02.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:02 smithi159 ceph-mon[45336]: pgmap v582: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-15T18:16:02.204 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:02.247 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:02.272 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:02.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:02.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:02.409 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:02 smithi082 ceph-mon[44206]: pgmap v582: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-15T18:16:02.412 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:02.436 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:02.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:02.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:02 smithi049 ceph-mon[40087]: pgmap v582: 17 pgs: 17 active+clean; 1.1 GiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-04-15T18:16:02.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:02.577 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:02.657 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:02.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:02.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:02.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:02.858 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:02.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:02.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 23112 23096 96.2244 120.875 0.118711 0.0103645 2024-04-15T18:16:02.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:02.992 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:03.016 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:03.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:03.120 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:03.150 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:03.172 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:03.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:03.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:03.307 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:03.327 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:03.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:03.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:03.468 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:03.495 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:03.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:03.599 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:16:03.599 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:16:03.599 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:16:03.600 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:16:03.600 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:16:03.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:03.634 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:03.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 24675 24659 96.3154 97.6875 0.00442991 0.0103726 2024-04-15T18:16:04.146 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:16:04.149 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:16:04.207 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:04.240 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:04.269 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:04.302 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:04.332 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:04.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:04.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:04.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:04 smithi159 ceph-mon[45336]: pgmap v583: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-15T18:16:04.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4106262877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:04.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:04.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:04.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:04 smithi049 ceph-mon[40087]: pgmap v583: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-15T18:16:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4106262877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:04.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:04.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:04.523 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:04.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:04.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:04.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:04 smithi082 ceph-mon[44206]: pgmap v583: 17 pgs: 17 active+clean; 1.2 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-04-15T18:16:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4106262877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:04.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:04.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:16:04.730 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:04.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 26431 26415 97.1051 109.75 0.0052499 0.0102919 2024-04-15T18:16:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:16:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:16:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:05.588 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:05.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:05.698 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:05.734 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:05.757 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:05.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:05.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:05.895 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:05.920 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:05.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:05.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 28380 28364 98.4772 121.812 0.00693983 0.0101477 2024-04-15T18:16:06.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:06.052 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.072 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:06.176 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:06.214 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.240 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:06.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:06.383 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.407 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:06.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:06 smithi159 ceph-mon[45336]: pgmap v584: 17 pgs: 17 active+clean; 1.4 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:16:06.410 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.410 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.410 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.410 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.410 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:06 smithi082 ceph-mon[44206]: pgmap v584: 17 pgs: 17 active+clean; 1.4 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:16:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:06.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:06 smithi049 ceph-mon[40087]: pgmap v584: 17 pgs: 17 active+clean; 1.4 GiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:16:06.512 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:06.547 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.575 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:06.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:06.712 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.742 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:06.765 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-04-15T18:16:03.088522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1251082784,"num_objects":19083,"num_object_clones":0,"num_object_copies":57249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19083,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":19262,"num_write_kb":1224122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19218,"ondisk_log_size":19218,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":4545460,"kb_used_data":3652756,"kb_used_omap":17,"kb_used_meta":892590,"kb_avail":1026651212,"statfs":{"total":1055945392128,"available":1051290841088,"internally_reserved":0,"allocated":3740422144,"data_stored":3737192813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17550,"internal_metadata":914013042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":972750848,"num_objects":14843,"num_object_clones":0,"num_object_copies":44529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14843,"num_write_kb":949952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.925863"},"pg_stats":[{"pgid":"3.f","version":"206'1210","reported_seq":1234,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.959893+0000","last_change":"2024-04-15T18:15:39.057796+0000","last_active":"2024-04-15T18:16:00.959893+0000","last_peered":"2024-04-15T18:16:00.959893+0000","last_clean":"2024-04-15T18:16:00.959893+0000","last_became_active":"2024-04-15T18:15:39.057408+0000","last_became_peered":"2024-04-15T18:15:39.057408+0000","last_unstale":"2024-04-15T18:16:00.959893+0000","last_undegraded":"2024-04-15T18:16:00.959893+0000","last_fullsized":"2024-04-15T18:16:00.959893+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1210,"log_dups_size":0,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:02.741971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79298560,"num_objects":1210,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":77440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"206'1110","reported_seq":1133,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.587181+0000","last_change":"2024-04-15T18:15:39.061855+0000","last_active":"2024-04-15T18:15:59.587181+0000","last_peered":"2024-04-15T18:15:59.587181+0000","last_clean":"2024-04-15T18:15:59.587181+0000","last_became_active":"2024-04-15T18:15:39.061483+0000","last_became_peered":"2024-04-15T18:15:39.061483+0000","last_unstale":"2024-04-15T18:15:59.587181+0000","last_undegraded":"2024-04-15T18:15:59.587181+0000","last_fullsized":"2024-04-15T18:15:59.587181+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:00.740160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":71040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"206'1250","reported_seq":1274,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.977398+0000","last_change":"2024-04-15T18:15:39.059260+0000","last_active":"2024-04-15T18:16:00.977398+0000","last_peered":"2024-04-15T18:16:00.977398+0000","last_clean":"2024-04-15T18:16:00.977398+0000","last_became_active":"2024-04-15T18:15:39.058838+0000","last_became_peered":"2024-04-15T18:15:39.058838+0000","last_unstale":"2024-04-15T18:16:00.977398+0000","last_undegraded":"2024-04-15T18:16:00.977398+0000","last_fullsized":"2024-04-15T18:16:00.977398+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:40.502923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"206'1118","reported_seq":1141,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.381907+0000","last_change":"2024-04-15T18:15:39.436047+0000","last_active":"2024-04-15T18:15:59.381907+0000","last_peered":"2024-04-15T18:15:59.381907+0000","last_clean":"2024-04-15T18:15:59.381907+0000","last_became_active":"2024-04-15T18:15:39.435717+0000","last_became_peered":"2024-04-15T18:15:39.435717+0000","last_unstale":"2024-04-15T18:15:59.381907+0000","last_undegraded":"2024-04-15T18:15:59.381907+0000","last_fullsized":"2024-04-15T18:15:59.381907+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:58.275182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73269248,"num_objects":1118,"num_object_clones":0,"num_object_copies":3354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1118,"num_write_kb":71552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,10],"acting":[5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"206'1247","reported_seq":1269,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.383315+0000","last_change":"2024-04-15T18:15:39.059576+0000","last_active":"2024-04-15T18:16:00.383315+0000","last_peered":"2024-04-15T18:16:00.383315+0000","last_clean":"2024-04-15T18:16:00.383315+0000","last_became_active":"2024-04-15T18:15:39.059365+0000","last_became_peered":"2024-04-15T18:15:39.059365+0000","last_unstale":"2024-04-15T18:16:00.383315+0000","last_undegraded":"2024-04-15T18:16:00.383315+0000","last_fullsized":"2024-04-15T18:16:00.383315+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.660175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81723392,"num_objects":1247,"num_object_clones":0,"num_object_copies":3741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":79808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"206'1135","reported_seq":1159,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.907891+0000","last_change":"2024-04-15T18:15:39.058448+0000","last_active":"2024-04-15T18:15:59.907891+0000","last_peered":"2024-04-15T18:15:59.907891+0000","last_clean":"2024-04-15T18:15:59.907891+0000","last_became_active":"2024-04-15T18:15:39.057839+0000","last_became_peered":"2024-04-15T18:15:39.057839+0000","last_unstale":"2024-04-15T18:15:59.907891+0000","last_undegraded":"2024-04-15T18:15:59.907891+0000","last_fullsized":"2024-04-15T18:15:59.907891+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1135,"log_dups_size":0,"ondisk_log_size":1135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.597994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74383360,"num_objects":1135,"num_object_clones":0,"num_object_copies":3405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1135,"num_write_kb":72640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"206'1114","reported_seq":1138,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.344845+0000","last_change":"2024-04-15T18:15:39.059016+0000","last_active":"2024-04-15T18:16:00.344845+0000","last_peered":"2024-04-15T18:16:00.344845+0000","last_clean":"2024-04-15T18:16:00.344845+0000","last_became_active":"2024-04-15T18:15:39.058704+0000","last_became_peered":"2024-04-15T18:15:39.058704+0000","last_unstale":"2024-04-15T18:16:00.344845+0000","last_undegraded":"2024-04-15T18:16:00.344845+0000","last_fullsized":"2024-04-15T18:16:00.344845+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1114,"log_dups_size":0,"ondisk_log_size":1114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:43.497471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73007104,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1114,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"206'1157","reported_seq":1180,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.581449+0000","last_change":"2024-04-15T18:15:39.058616+0000","last_active":"2024-04-15T18:15:59.581449+0000","last_peered":"2024-04-15T18:15:59.581449+0000","last_clean":"2024-04-15T18:15:59.581449+0000","last_became_active":"2024-04-15T18:15:39.058285+0000","last_became_peered":"2024-04-15T18:15:39.058285+0000","last_unstale":"2024-04-15T18:15:59.581449+0000","last_undegraded":"2024-04-15T18:15:59.581449+0000","last_fullsized":"2024-04-15T18:15:59.581449+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:38.332807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75825152,"num_objects":1157,"num_object_clones":0,"num_object_copies":3471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1157,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"206'1113","reported_seq":1137,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.552909+0000","last_change":"2024-04-15T18:15:39.062153+0000","last_active":"2024-04-15T18:15:59.552909+0000","last_peered":"2024-04-15T18:15:59.552909+0000","last_clean":"2024-04-15T18:15:59.552909+0000","last_became_active":"2024-04-15T18:15:39.061741+0000","last_became_peered":"2024-04-15T18:15:39.061741+0000","last_unstale":"2024-04-15T18:15:59.552909+0000","last_undegraded":"2024-04-15T18:15:59.552909+0000","last_fullsized":"2024-04-15T18:15:59.552909+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1113,"log_dups_size":0,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:03.899612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72941568,"num_objects":1113,"num_object_clones":0,"num_object_copies":3339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1113,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"206'1137","reported_seq":1160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.337465+0000","last_change":"2024-04-15T18:15:39.058580+0000","last_active":"2024-04-15T18:16:00.337465+0000","last_peered":"2024-04-15T18:16:00.337465+0000","last_clean":"2024-04-15T18:16:00.337465+0000","last_became_active":"2024-04-15T18:15:39.058216+0000","last_became_peered":"2024-04-15T18:15:39.058216+0000","last_unstale":"2024-04-15T18:16:00.337465+0000","last_undegraded":"2024-04-15T18:16:00.337465+0000","last_fullsized":"2024-04-15T18:16:00.337465+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:57.648006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74514432,"num_objects":1137,"num_object_clones":0,"num_object_copies":3411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":72768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"206'1164","reported_seq":1186,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.401016+0000","last_change":"2024-04-15T18:15:39.670253+0000","last_active":"2024-04-15T18:15:59.401016+0000","last_peered":"2024-04-15T18:15:59.401016+0000","last_clean":"2024-04-15T18:15:59.401016+0000","last_became_active":"2024-04-15T18:15:39.669933+0000","last_became_peered":"2024-04-15T18:15:39.669933+0000","last_unstale":"2024-04-15T18:15:59.401016+0000","last_undegraded":"2024-04-15T18:15:59.401016+0000","last_fullsized":"2024-04-15T18:15:59.401016+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:51.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":74496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"206'1261","reported_seq":1283,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:01.898078+0000","last_change":"2024-04-15T18:15:39.057945+0000","last_active":"2024-04-15T18:16:01.898078+0000","last_peered":"2024-04-15T18:16:01.898078+0000","last_clean":"2024-04-15T18:16:01.898078+0000","last_became_active":"2024-04-15T18:15:39.057159+0000","last_became_peered":"2024-04-15T18:15:39.057159+0000","last_unstale":"2024-04-15T18:16:01.898078+0000","last_undegraded":"2024-04-15T18:16:01.898078+0000","last_fullsized":"2024-04-15T18:16:01.898078+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:38.065390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"206'1401","reported_seq":1425,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:02.599071+0000","last_change":"2024-04-15T18:15:39.060876+0000","last_active":"2024-04-15T18:16:02.599071+0000","last_peered":"2024-04-15T18:16:02.599071+0000","last_clean":"2024-04-15T18:16:02.599071+0000","last_became_active":"2024-04-15T18:15:39.060320+0000","last_became_peered":"2024-04-15T18:15:39.060320+0000","last_unstale":"2024-04-15T18:16:02.599071+0000","last_undegraded":"2024-04-15T18:16:02.599071+0000","last_fullsized":"2024-04-15T18:16:02.599071+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1401,"log_dups_size":0,"ondisk_log_size":1401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:52.602066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":4203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1401,"num_write_kb":89664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"206'1151","reported_seq":1173,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:00.384506+0000","last_change":"2024-04-15T18:15:39.435414+0000","last_active":"2024-04-15T18:16:00.384506+0000","last_peered":"2024-04-15T18:16:00.384506+0000","last_clean":"2024-04-15T18:16:00.384506+0000","last_became_active":"2024-04-15T18:15:39.435165+0000","last_became_peered":"2024-04-15T18:15:39.435165+0000","last_unstale":"2024-04-15T18:16:00.384506+0000","last_undegraded":"2024-04-15T18:16:00.384506+0000","last_fullsized":"2024-04-15T18:16:00.384506+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1151,"log_dups_size":0,"ondisk_log_size":1151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:06.373700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":73664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"206'1132","reported_seq":1156,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:59.575079+0000","last_change":"2024-04-15T18:15:39.061919+0000","last_active":"2024-04-15T18:15:59.575079+0000","last_peered":"2024-04-15T18:15:59.575079+0000","last_clean":"2024-04-15T18:15:59.575079+0000","last_became_active":"2024-04-15T18:15:39.061499+0000","last_became_peered":"2024-04-15T18:15:39.061499+0000","last_unstale":"2024-04-15T18:15:59.575079+0000","last_undegraded":"2024-04-15T18:15:59.575079+0000","last_fullsized":"2024-04-15T18:15:59.575079+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1132,"log_dups_size":0,"ondisk_log_size":1132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:05.557790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74186752,"num_objects":1132,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1132,"num_write_kb":72448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"206'1381","reported_seq":1404,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:16:02.307516+0000","last_change":"2024-04-15T18:15:39.059346+0000","last_active":"2024-04-15T18:16:02.307516+0000","last_peered":"2024-04-15T18:16:02.307516+0000","last_clean":"2024-04-15T18:16:02.307516+0000","last_became_active":"2024-04-15T18:15:39.059050+0000","last_became_peered":"2024-04-15T18:15:39.059050+0000","last_unstale":"2024-04-15T18:16:02.307516+0000","last_undegraded":"2024-04-15T18:16:02.307516+0000","last_fullsized":"2024-04-15T18:16:02.307516+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":1381,"log_dups_size":0,"ondisk_log_size":1381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:59.305776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1381,"num_write_kb":88384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":598,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-04-15T18:15:58.056693+0000","last_change":"2024-04-15T18:15:04.226208+0000","last_active":"2024-04-15T18:15:58.056693+0000","last_peered":"2024-04-15T18:15:58.056693+0000","last_clean":"2024-04-15T18:15:58.056693+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:15:58.056693+0000","last_undegraded":"2024-04-15T18:15:58.056693+0000","last_fullsized":"2024-04-15T18:15:58.056693+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:15:04.226084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:15:04.226084+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:16:29.514876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1250492416,"num_objects":19081,"num_object_clones":0,"num_object_copies":57243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19081,"num_write_kb":1221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3904700416,"data_stored":3904700416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19081,"ondisk_log_size":19081,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925905,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621892,"kb_used_data":554620,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93123260,"statfs":{"total":95995035648,"available":95358218240,"internally_reserved":0,"allocated":567930880,"data_stored":567634703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":246588,"kb_used_data":170996,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93498564,"statfs":{"total":95995035648,"available":95742529536,"internally_reserved":0,"allocated":175099904,"data_stored":174807299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514588,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":352764,"kb_used_data":253556,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93392388,"statfs":{"total":95995035648,"available":95633805312,"internally_reserved":0,"allocated":259641344,"data_stored":259348739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225664,"kb_used_data":140340,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93519488,"statfs":{"total":95995035648,"available":95763955712,"internally_reserved":0,"allocated":143708160,"data_stored":143415555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168677,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":391236,"kb_used_data":297596,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93353916,"statfs":{"total":95995035648,"available":95594409984,"internally_reserved":0,"allocated":304738304,"data_stored":304442147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463018,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":485252,"kb_used_data":422140,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93259900,"statfs":{"total":95995035648,"available":95498137600,"internally_reserved":0,"allocated":432271360,"data_stored":431975203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":413564,"kb_used_data":310196,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93331588,"statfs":{"total":95995035648,"available":95571546112,"internally_reserved":0,"allocated":317640704,"data_stored":317348099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526780,"kb_used_data":441460,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93218372,"statfs":{"total":95995035648,"available":95455612928,"internally_reserved":0,"allocated":452055040,"data_stored":451762435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171004,"kb_used_data":89844,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93574148,"statfs":{"total":95995035648,"available":95819927552,"internally_reserved":0,"allocated":92000256,"data_stored":91707651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770428,"kb_used_data":685108,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92974724,"statfs":{"total":95995035648,"available":95206117376,"internally_reserved":0,"allocated":701550592,"data_stored":701257987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724654,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":340288,"kb_used_data":286900,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93404864,"statfs":{"total":95995035648,"available":95646580736,"internally_reserved":0,"allocated":293785600,"data_stored":293492995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463208448,"data_stored":463208448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":343212032,"data_stored":343212032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97517568,"data_stored":97517568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":748290048,"data_stored":748290048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":293076992,"data_stored":293076992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":449511424,"data_stored":449511424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":317325312,"data_stored":317325312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":149487616,"data_stored":149487616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":272826368,"data_stored":272826368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":186580992,"data_stored":186580992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":583663616,"data_stored":583663616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:16:06.767 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:16:06.767 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-15T18:16:06.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:06.876 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:06.900 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:06.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:06.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 30213 30197 99.3233 114.562 0.0087726 0.0100526 2024-04-15T18:16:07.004 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:07.040 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:07.054 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:07.062 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:07.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:07.196 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.220 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:07.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:07.353 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.372 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:07.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:07.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.567 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:07.632 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:07.674 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:07.707 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.731 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:07.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:07.867 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:07.894 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:07.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:07.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:16:07.940167+0000 min lat: 0.00112542 max lat: 0.279966 avg lat: 0.0101199 2024-04-15T18:16:07.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:16:07.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 31617 31601 98.7443 87.75 0.00644441 0.0101199 2024-04-15T18:16:08.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:08.033 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:08.061 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:08.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:08.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:08.207 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:08.234 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:08.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:08.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:08 smithi082 ceph-mon[44206]: pgmap v585: 17 pgs: 17 active+clean; 1.6 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:08.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:08.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:08 smithi049 ceph-mon[40087]: pgmap v585: 17 pgs: 17 active+clean; 1.6 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:08.377 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:08.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:08 smithi159 ceph-mon[45336]: pgmap v585: 17 pgs: 17 active+clean; 1.6 GiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:08.831 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:08.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:08.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:08.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 33078 33062 98.3899 91.3125 0.112435 0.0101107 2024-04-15T18:16:08.965 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:09.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:16:09.173 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:16:09.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3468715417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:09 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:09.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:09.523 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:09.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3468715417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:09 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.556 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3468715417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.557 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:09 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:09.565 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:09.589 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:09.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:09.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:09.727 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:09.753 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:09.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:09.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:09.901 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:09.920 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:09.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:09.942 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 34525 34509 98.0281 90.4375 0.00900411 0.0101914 2024-04-15T18:16:10.025 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:10.057 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:10.078 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:10.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:10.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:10.219 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:10.295 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:10.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:10.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:10 smithi049 ceph-mon[40087]: pgmap v586: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:16:10.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:10 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:10.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:10 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:16:10.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:10 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:10.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:10 smithi049 ceph-mon[40087]: osdmap e207: 12 total, 11 up, 11 in 2024-04-15T18:16:10.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:10.437 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:10.539 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:10 smithi082 ceph-mon[44206]: pgmap v586: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:16:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:10 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:10 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:16:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:10 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:10.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:10 smithi082 ceph-mon[44206]: osdmap e207: 12 total, 11 up, 11 in 2024-04-15T18:16:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:10.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:10.607 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:10.607 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:10.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:10.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:10 smithi159 ceph-mon[45336]: pgmap v586: 17 pgs: 17 active+clean; 1.7 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:16:10.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:10 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:10.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:10 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:16:10.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:10 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:10.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:10 smithi159 ceph-mon[45336]: osdmap e207: 12 total, 11 up, 11 in 2024-04-15T18:16:10.682 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:10.707 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:10.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:10.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:10.839 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:10.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 35827 35811 97.3038 81.375 0.00821344 0.0102561 2024-04-15T18:16:11.487 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:11.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:11.591 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:11.618 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:11.649 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:11.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:11.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:11.789 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:11.814 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:11.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:11.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:11.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 37258 37242 96.9758 89.4375 0.00217734 0.0103041 2024-04-15T18:16:11.957 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:11.977 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:11.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:12.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:12.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:16:12.124 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:12.152 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:12.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:12.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:12.292 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:12.315 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:12.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:12.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:16:12.396 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:12 smithi082 ceph-mon[44206]: pgmap v588: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:16:12.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:12 smithi159 ceph-mon[45336]: pgmap v588: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:16:12.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:12.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:12 smithi049 ceph-mon[40087]: pgmap v588: 17 pgs: 17 active+clean; 2.1 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:16:12.460 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:12.488 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:12.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:12.593 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:12.626 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:12.652 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:12.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:12.751 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:12.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:12.789 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:12.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:12.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:12.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 39008 38992 97.4714 109.375 0.00813207 0.0102532 2024-04-15T18:16:12.971 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:13.011 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:13.043 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:13.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:13.147 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:13.186 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:13.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:13.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:13.643 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:13.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:13.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:13.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:13.776 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:13.797 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:13.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:13.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:13.929 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:13.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 40341 40325 96.9265 83.3125 0.00231385 0.0103054 2024-04-15T18:16:13.950 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:13.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:14.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:14.087 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:14.111 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:14.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:14.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:14.247 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:14.287 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:14.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:14.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:14.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:14 smithi159 ceph-mon[45336]: pgmap v589: 17 pgs: 17 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:16:14.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:14 smithi159 ceph-mon[45336]: osdmap e208: 12 total, 11 up, 11 in 2024-04-15T18:16:14.426 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:14 smithi082 ceph-mon[44206]: pgmap v589: 17 pgs: 17 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:16:14.426 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:14 smithi082 ceph-mon[44206]: osdmap e208: 12 total, 11 up, 11 in 2024-04-15T18:16:14.428 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:14 smithi049 ceph-mon[40087]: pgmap v589: 17 pgs: 17 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:16:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:14 smithi049 ceph-mon[40087]: osdmap e208: 12 total, 11 up, 11 in 2024-04-15T18:16:14.630 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:14.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:14.742 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:14.785 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:14.896 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:14.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:14.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 41364 41348 95.7045 63.9375 0.00619261 0.0104438 2024-04-15T18:16:14.979 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:14.980 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:15.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:15.032 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:15.233 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:15.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:15.342 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:15.376 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:15.403 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:15.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:15.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2761894526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:15.508 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:15.539 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2761894526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:15.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2761894526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:15.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:15.565 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:15.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:15.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:15.705 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:15.729 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:15.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:15.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:16:15.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:15.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:15.889 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:15.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:15.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:15.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 42728 42712 95.3309 85.25 0.00621027 0.0104836 2024-04-15T18:16:15.956 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:15.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:15.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:16.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:16.029 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:16.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:16.051 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:16.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:16.074 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:16.094 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:16.099 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:16.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:16.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:16.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:16.187 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:16:16.189 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:16.212 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:16.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:16.316 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:16.343 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:16.363 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:16.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:16.383 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:16.410 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:16.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: pgmap v591: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: osdmap e209: 12 total, 11 up, 11 in 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1998422149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:16.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:16.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:16.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:16.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: pgmap v591: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:16:16.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: osdmap e209: 12 total, 11 up, 11 in 2024-04-15T18:16:16.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1998422149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:16.505 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:16.583 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:16.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:16.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:16.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:16.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: pgmap v591: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:16:16.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: osdmap e209: 12 total, 11 up, 11 in 2024-04-15T18:16:16.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1998422149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:16.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:16.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:16.722 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:16.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 43996 43980 94.7762 79.25 0.00822713 0.0105443 2024-04-15T18:16:16.976 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:16.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:17.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:17.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:17.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:17.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osdmap e210: 12 total, 11 up, 11 in 2024-04-15T18:16:17.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:17 smithi082 ceph-mon[44206]: osdmap e211: 12 total, 11 up, 11 in 2024-04-15T18:16:17.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:17.347 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osdmap e210: 12 total, 11 up, 11 in 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:17 smithi049 ceph-mon[40087]: osdmap e211: 12 total, 11 up, 11 in 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:17.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osdmap e210: 12 total, 11 up, 11 in 2024-04-15T18:16:17.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:17.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:17 smithi159 ceph-mon[45336]: osdmap e211: 12 total, 11 up, 11 in 2024-04-15T18:16:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:17.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:17.904 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:17.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:17.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 45268 45252 94.2667 79.5 0.0127188 0.0105999 2024-04-15T18:16:17.962 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:17.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:18.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:18.101 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:18.125 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:18.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:18.149 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:16:18.149 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:16:18.150 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:16:18.150 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:16:18.150 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:16:18.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:18.257 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:18 smithi159 ceph-mon[45336]: pgmap v595: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-15T18:16:18.260 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:18.281 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:18.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:18.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:18.417 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:18.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:18 smithi049 ceph-mon[40087]: pgmap v595: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-15T18:16:18.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:18 smithi082 ceph-mon[44206]: pgmap v595: 17 pgs: 17 active+clean; 2.5 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-04-15T18:16:18.543 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:18.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:18.649 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:18.694 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:18.724 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:18.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:18.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:18.869 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:18.903 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:18.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:18.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 46678 46662 94.0684 88.125 0.00593829 0.0106256 2024-04-15T18:16:18.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:16:19.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:19.042 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:19.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.224 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:19.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:16:19.327 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:19.334 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:19.354 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.372 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:19.477 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:19.507 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.526 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:19.557 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:19.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:19.659 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:16:19.808 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:19.914 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:19.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 15 47672 47657 93.072 62.1875 0.0165707 0.0107367 2024-04-15T18:16:19.946 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:19.968 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:19.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:20.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:20.111 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:20.153 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:20.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:20.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:20.294 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:20.318 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:20.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:20 smithi049 ceph-mon[40087]: pgmap v596: 17 pgs: 17 active+clean; 2.6 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:20.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:20.397 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:20 smithi082 ceph-mon[44206]: pgmap v596: 17 pgs: 17 active+clean; 2.6 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:20.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:20 smithi159 ceph-mon[45336]: pgmap v596: 17 pgs: 17 active+clean; 2.6 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:20.421 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:20.449 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:20.799 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:20.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:20.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:20.936 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:20.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 48787 48771 92.3613 69.625 0.00286175 0.0108187 2024-04-15T18:16:21.089 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:21.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:21.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:21.226 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:21.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:21.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:21.353 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:21.356 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:21.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:21.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:21.414 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:21.418 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:21.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:21.448 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4206 2.d e124/114 2:b7208875:::benchmark_data_smithi159_80260_object4205:head v 124'256, mlcod=124'253)", 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 271.80392396500002, 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27413553299999999, 2024-04-15T18:16:21.451 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638677+0000", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7599999999999997e-07 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638790+0000", 2024-04-15T18:16:21.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113306 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912318+0000", 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27352749300000001 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912322+0000", 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5759999999999999e-06 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912371+0000", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8529999999999998e-05 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912386+0000", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5089000000000001e-05 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912779+0000", 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039285399999999999 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912812+0000", 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3209e-05 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4226 2.5 e124/114 2:a74ff3b8:::benchmark_data_smithi159_80260_object4225:head v 124'266, mlcod=124'263)", 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.655875+0000", 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 271.78672529200003, 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14261616199999999, 2024-04-15T18:16:21.456 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.655875+0000", 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.655875+0000", 2024-04-15T18:16:21.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.655877+0000", 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6330000000000001e-06 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.655927+0000", 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0599000000000001e-05 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.797948+0000", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14202076599999999 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.797953+0000", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4500000000000003e-06 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.798002+0000", 2024-04-15T18:16:21.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.808e-05 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.798017+0000", 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5696e-05 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.798440+0000", 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042302699999999997 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.460 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.798491+0000", 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0911000000000001e-05 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4648 2.5 e124/114 2:aea343ca:::benchmark_data_smithi159_80260_object4647:head v 124'291, mlcod=124'288)", 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:50.154389+0000", 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 271.28821177200001, 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.142223078, 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.154389+0000", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.154389+0000", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.154389+0000", 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3e-07 2024-04-15T18:16:21.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.154458+0000", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9231000000000002e-05 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296065+0000", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.141606858 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296088+0000", 2024-04-15T18:16:21.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2628000000000001e-05 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296138+0000", 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9706999999999997e-05 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296153+0000", 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5007999999999999e-05 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296584+0000", 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043166599999999998 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:50.296612+0000", 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7350000000000001e-05 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62467 2.d e143/114 2:b6055ada:::benchmark_data_smithi159_80260_object62466:head v 143'3982, mlcod=143'3979)", 2024-04-15T18:16:21.465 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.518181989, 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.364600912, 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924419+0000", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7699999999999992e-07 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924487+0000", 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7405999999999998e-05 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287406+0000", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.362919453 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287411+0000", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1189999999999996e-06 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287463+0000", 2024-04-15T18:16:21.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1357999999999999e-05 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287483+0000", 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0254999999999998e-05 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.288986+0000", 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001503576 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.289019+0000", 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2768000000000002e-05 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62487 2.f e143/114 2:f21ee866:::benchmark_data_smithi159_80260_object62486:head v 143'3880, mlcod=143'3878)", 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.34450178099999, 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17900353999999999, 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098100+0000", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.905e-06 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098160+0000", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0068999999999997e-05 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275509+0000", 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17734833 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275515+0000", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0360000000000003e-06 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275556+0000", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0874999999999997e-05 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275572+0000", 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6164e-05 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.277075+0000", 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015032750000000001 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.277102+0000", 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6886e-05 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62513 2.d e143/114 2:b0d4bfc1:::benchmark_data_smithi159_80260_object62512:head v 143'3986, mlcod=143'3983)", 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.12140299999999, 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.162785016, 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321199+0000", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.593e-06 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321261+0000", 2024-04-15T18:16:21.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2166000000000001e-05 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482466+0000", 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.161205454 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482471+0000", 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6090000000000003e-06 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482528+0000", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6551000000000001e-05 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482541+0000", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.305e-05 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.483953+0000", 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001412632 2024-04-15T18:16:21.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.483982+0000", 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8960999999999999e-05 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:153095 2.d e176/114 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head v 176'9796, mlcod=176'9793)", 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 110.29233985499999, 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25507247500000002, 2024-04-15T18:16:21.479 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:16:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150262+0000", 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6479999999999999e-06 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150272+0000", 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6250000000000002e-06 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400544+0000", 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25027195200000002 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400549+0000", 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2229999999999998e-06 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400606+0000", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7339000000000001e-05 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400629+0000", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3059000000000001e-05 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405295+0000", 2024-04-15T18:16:21.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046661419999999999 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405333+0000", 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7487000000000002e-05 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:3201 3.2 e204/199 3:431af5b7:::benchmark_data_smithi159_91621_object3200:head v 204'198, mlcod=204'193)", 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:16:21.484 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 31.489631648, 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18664436700000001, 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:16:21.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9200000000000001e-07 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.953106+0000", 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013703200000000001 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139138+0000", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18603144299999999 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139145+0000", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8299999999999998e-06 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139195+0000", 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0207999999999998e-05 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139218+0000", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3274000000000001e-05 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139574+0000", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035602399999999998 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139613+0000", 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8964000000000002e-05 2024-04-15T18:16:21.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:5443 3.3 3:ca948811:::benchmark_data_smithi159_91621_object5442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.144369+0000", 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.298231592, 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14147457499999999, 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5443 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144369+0000", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144369+0000", 2024-04-15T18:16:21.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144370+0000", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.218e-06 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144423+0000", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2779999999999999e-05 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144437+0000", 2024-04-15T18:16:21.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4048e-05 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144445+0000", 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5760000000000006e-06 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144508+0000", 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2340000000000003e-05 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144586+0000", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8221999999999997e-05 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.144690+0000", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104298 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.145146+0000", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00045579399999999998 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147393+0000", 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002246741 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285738+0000", 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13834523700000001 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285797+0000", 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9276e-05 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285843+0000", 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6044999999999997e-05 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24793.0:5443 3.3 e204/199 ondisk, result = 0)", 2024-04-15T18:16:21.495 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.145354+0000", 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.297245846999999, 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14049562400000001, 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.145354+0000", 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.496 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.145354+0000", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.145355+0000", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2900000000000003e-07 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.145359+0000", 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.687e-06 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.497 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285665+0000", 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14030595300000001 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285676+0000", 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1527e-05 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.498 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285723+0000", 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6625999999999997e-05 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285731+0000", 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1599999999999998e-06 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.285850+0000", 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118942 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:5448 3.3 3:ce24e703:::benchmark_data_smithi159_91621_object5447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.147437+0000", 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.295162842, 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13973881399999999, 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.500 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 5448 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147437+0000", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147437+0000", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147438+0000", 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8000000000000005e-07 2024-04-15T18:16:21.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147504+0000", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5371999999999997e-05 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147512+0000", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9189999999999999e-06 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147516+0000", 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9819999999999999e-06 2024-04-15T18:16:21.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147544+0000", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7844999999999999e-05 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147622+0000", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7523999999999998e-05 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.147710+0000", 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7713000000000005e-05 2024-04-15T18:16:21.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148095+0000", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038523799999999999 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.149519+0000", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014244920000000001 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287110+0000", 2024-04-15T18:16:21.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13759058299999999 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287146+0000", 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6257999999999997e-05 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287176+0000", 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0108e-05 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24793.0:5448 3.3 e204/199 ondisk, result = 0)", 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.148304+0000", 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.294296274000001, 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.138880376, 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148304+0000", 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148304+0000", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148305+0000", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2200000000000004e-07 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148308+0000", 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6799999999999999e-06 2024-04-15T18:16:21.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287050+0000", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13874187399999999 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287057+0000", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7549999999999997e-06 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287101+0000", 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4336999999999999e-05 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287107+0000", 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3260000000000003e-06 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.287184+0000", 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7781999999999995e-05 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:5450 3.e e204/199 3:73156652:::benchmark_data_smithi159_91621_object5449:head v 204'326, mlcod=204'324)", 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.148929+0000", 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.293671144000001, 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.135516465, 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148929+0000", 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148929+0000", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.148930+0000", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6499999999999999e-07 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.149083+0000", 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000153628 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283765+0000", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13468206599999999 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283772+0000", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8589999999999999e-06 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283880+0000", 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010752 2024-04-15T18:16:21.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283904+0000", 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3866000000000002e-05 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284418+0000", 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051392600000000005 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284446+0000", 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7934999999999999e-05 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:11743 3.e e205/199 3:77c685dd:::benchmark_data_smithi159_91621_object11742:head v 205'746, mlcod=205'743)", 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:54.834232+0000", 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 26.608368548000001, 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.132810068, 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.515 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.834232+0000", 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.834232+0000", 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.834232+0000", 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.99e-07 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.834284+0000", 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1452999999999997e-05 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.966484+0000", 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13220056199999999 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.966490+0000", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8150000000000002e-06 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.966525+0000", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5154000000000003e-05 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.966541+0000", 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5500000000000001e-05 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.967017+0000", 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00047603899999999999 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:54.967042+0000", 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4946000000000001e-05 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.519 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:12345 3.e e205/199 3:7c899c39:::benchmark_data_smithi159_91621_object12344:head v 205'778, mlcod=205'775)", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:55.458514+0000", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 25.984086204, 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14164327199999999, 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.458514+0000", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.458514+0000", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.458515+0000", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.09e-07 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.458583+0000", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8281999999999994e-05 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.599570+0000", 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14098665299999999 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.599574+0000", 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8279999999999996e-06 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.599609+0000", 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.536e-05 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.599623+0000", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3583e-05 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.600112+0000", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048957899999999997 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:55.600157+0000", 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5178e-05 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.523 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:15728 3.e e205/199 3:7aaffa93:::benchmark_data_smithi159_91621_object15727:head v 205'970, mlcod=205'967)", 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.834837381, 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21313194799999999, 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4899999999999995e-07 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607879+0000", 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011526700000000001 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819651+0000", 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21177270200000001 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819657+0000", 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819701+0000", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3424999999999998e-05 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819714+0000", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3763e-05 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.820843+0000", 2024-04-15T18:16:21.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011291389999999999 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.820895+0000", 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1411e-05 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:17319 3.8 e206/199 3:1eee15a2:::benchmark_data_smithi159_91621_object17318:head v 206'1124, mlcod=206'1121)", 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:59.088994+0000", 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.353605975000001, 2024-04-15T18:16:21.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14026693700000001, 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.088994+0000", 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.088994+0000", 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.088995+0000", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4999999999999998e-07 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.089103+0000", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010805499999999999 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.228275+0000", 2024-04-15T18:16:21.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.139172564 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.228284+0000", 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4579999999999993e-06 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.228336+0000", 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1749000000000002e-05 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.228358+0000", 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2697e-05 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.229217+0000", 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085913600000000004 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.229261+0000", 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3828000000000001e-05 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:17327 3.8 e206/199 3:16cbb8de:::benchmark_data_smithi159_91621_object17326:head v 206'1125, mlcod=206'1121)", 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:59.094937+0000", 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.347662786000001, 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13703378899999999, 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.094937+0000", 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.094937+0000", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.094938+0000", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2099999999999996e-07 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.095022+0000", 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3454000000000004e-05 2024-04-15T18:16:21.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231184+0000", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.136162166 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231189+0000", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4530000000000001e-06 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231245+0000", 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5429000000000001e-05 2024-04-15T18:16:21.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231267+0000", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1970000000000001e-05 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231936+0000", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066943899999999997 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:59.231971+0000", 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5256999999999998e-05 2024-04-15T18:16:21.536 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:19430 3.e e206/199 3:79f21c41:::benchmark_data_smithi159_91621_object19429:head v 206'1219, mlcod=206'1216)", 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:00.703985+0000", 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.738615282000001, 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.144417722, 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.703985+0000", 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.703985+0000", 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.703986+0000", 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6599999999999996e-07 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.704071+0000", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5445999999999995e-05 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.847639+0000", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14356780699999999 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.847644+0000", 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1329999999999998e-06 2024-04-15T18:16:21.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.847686+0000", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2454999999999999e-05 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.847700+0000", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.393e-05 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.848374+0000", 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067327100000000003 2024-04-15T18:16:21.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:00.848403+0000", 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9113999999999999e-05 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:45769 3.3 3:c748487c:::benchmark_data_smithi159_91621_object45768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 3.1617551430000002, 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15225887799999999, 2024-04-15T18:16:21.541 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45769 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280846+0000", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.268e-06 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280922+0000", 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5308999999999993e-05 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280935+0000", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2863e-05 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280943+0000", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1249999999999993e-06 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280984+0000", 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1341000000000003e-05 2024-04-15T18:16:21.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.281071+0000", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6646999999999999e-05 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.281156+0000", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5002999999999999e-05 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.282260+0000", 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001104516 2024-04-15T18:16:21.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.282297+0000", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6772e-05 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433007+0000", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15070974100000001 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433069+0000", 2024-04-15T18:16:21.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2612000000000003e-05 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433104+0000", 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4681000000000002e-05 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:21.547 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:16:21.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:21.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:21.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:21.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:21.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:21.580 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:21.602 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:21.603 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:21.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:21.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:21.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:21.706 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:21.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:21.737 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:21.758 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:21.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:21.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:21.900 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:21.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 50117 50101 92.0894 83.125 0.00484822 0.0108524 2024-04-15T18:16:22.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: pgmap v597: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:22.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3110156008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:22.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.183 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:16:22.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: pgmap v597: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3110156008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:22 smithi049 ceph-mon[40087]: osdmap e212: 12 total, 11 up, 11 in 2024-04-15T18:16:22.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:22 smithi082 ceph-mon[44206]: osdmap e212: 12 total, 11 up, 11 in 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: pgmap v597: 17 pgs: 17 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3110156008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:22.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:22 smithi159 ceph-mon[45336]: osdmap e212: 12 total, 11 up, 11 in 2024-04-15T18:16:22.674 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:22.859 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:22.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:22.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:22.943 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 51674 51658 92.2386 97.3125 0.0240232 0.0108351 2024-04-15T18:16:22.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:22.998 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.452 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:23.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:23.591 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:23.743 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:23.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:23.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:23.886 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:23.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 52910 52894 91.822 77.25 0.0164399 0.0108784 2024-04-15T18:16:24.012 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:24.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:24.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:24.148 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:24.171 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:24.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:24.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:24 smithi082 ceph-mon[44206]: pgmap v599: 17 pgs: 17 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:16:24.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:24.305 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:24.330 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:24.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:24.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:24 smithi159 ceph-mon[45336]: pgmap v599: 17 pgs: 17 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:16:24.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:24 smithi049 ceph-mon[40087]: pgmap v599: 17 pgs: 17 active+clean; 2.9 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:16:24.471 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:24.491 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:24.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:24.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:24.630 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:24.630 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:24.648 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:24.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:16:24.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 54284 54268 91.6611 85.875 0.021306 0.0109039 2024-04-15T18:16:25.200 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:25.214 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:16:25.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/188743700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:25 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.406 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:16:25.441 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:25.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:25.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/188743700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:25 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:25.578 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/188743700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:25 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:25.705 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:25.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:25.800 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:25.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:25.852 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:25.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 55124 55108 90.6304 52.5 0.0045601 0.0110265 2024-04-15T18:16:26.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:26 smithi082 ceph-mon[44206]: pgmap v600: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:16:26.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:26 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:26.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:26 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:26.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:26 smithi082 ceph-mon[44206]: osdmap e213: 12 total, 11 up, 11 in 2024-04-15T18:16:26.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:26.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:26.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:26.448 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:26.473 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:26 smithi049 ceph-mon[40087]: pgmap v600: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:16:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:26 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:26 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:26.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:26 smithi049 ceph-mon[40087]: osdmap e213: 12 total, 11 up, 11 in 2024-04-15T18:16:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:26.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:26.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:26.617 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:26 smithi159 ceph-mon[45336]: pgmap v600: 17 pgs: 17 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:16:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:26 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:26 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:26.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:26 smithi159 ceph-mon[45336]: osdmap e213: 12 total, 11 up, 11 in 2024-04-15T18:16:26.873 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:26.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:26.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 56349 56333 90.2695 76.5625 0.00790279 0.011069 2024-04-15T18:16:26.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:27.008 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.033 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:27.138 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:27.171 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.197 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:27.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:27.334 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.390 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:27.494 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:27.524 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.550 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:27.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:27.690 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.719 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:27.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:27.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:16:27.941785+0000 min lat: 0.00112542 max lat: 0.279966 avg lat: 0.0111256 2024-04-15T18:16:27.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:16:27.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 57513 57497 89.8314 72.75 0.00465762 0.0111256 2024-04-15T18:16:27.959 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:27.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:27.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:28.016 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:28.017 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:28.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:28.049 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:28.066 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:28.083 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:28.103 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:28.111 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:28.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:28.163 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:28.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:28.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:28.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:28.282 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:28.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:28.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:28.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:28.282 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:28.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:28 smithi082 ceph-mon[44206]: osdmap e214: 12 total, 11 up, 11 in 2024-04-15T18:16:28.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:28 smithi082 ceph-mon[44206]: pgmap v603: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:28.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:28.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:28.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1160624488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:28.306 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:28.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:28.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:28 smithi159 ceph-mon[45336]: osdmap e214: 12 total, 11 up, 11 in 2024-04-15T18:16:28.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:28 smithi159 ceph-mon[45336]: pgmap v603: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:28.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:28.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:28.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1160624488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:28.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:28.453 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:28 smithi049 ceph-mon[40087]: osdmap e214: 12 total, 11 up, 11 in 2024-04-15T18:16:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:28 smithi049 ceph-mon[40087]: pgmap v603: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:28.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1160624488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:28.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:16:28.806 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:16:28.820 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:28.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:28.924 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:28.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 58865 58849 89.7012 84.5 0.0159087 0.0111425 2024-04-15T18:16:28.954 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:28.976 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:28.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:29.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:29.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:29.124 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:29.134 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:29.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:29.203 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:29.257 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:29.288 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osdmap e215: 12 total, 11 up, 11 in 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osdmap e215: 12 total, 11 up, 11 in 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osdmap e215: 12 total, 11 up, 11 in 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:29.489 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:29.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:29.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:29.632 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:29.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 59910 59894 89.1204 65.3125 0.00397788 0.0112156 2024-04-15T18:16:29.986 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:29.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:30.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:30.142 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:30 smithi159 ceph-mon[45336]: pgmap v605: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:30.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:30 smithi049 ceph-mon[40087]: pgmap v605: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:30.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:30 smithi082 ceph-mon[44206]: pgmap v605: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:30.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:30.599 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:30.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:30.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:30.732 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:30.758 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:30.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:30.862 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:30.899 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:30.922 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:30.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:30.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 60882 60866 88.4605 60.75 0.00486403 0.0112981 2024-04-15T18:16:31.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:31.067 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:31.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:16:31.269 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:31.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:31.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:31.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:31.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:31.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:31 smithi159 ceph-mon[45336]: osdmap e216: 12 total, 11 up, 11 in 2024-04-15T18:16:31.411 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:31.411 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:31.412 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:31 smithi082 ceph-mon[44206]: osdmap e216: 12 total, 11 up, 11 in 2024-04-15T18:16:31.414 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:31.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:31.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:31.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:31 smithi049 ceph-mon[40087]: osdmap e216: 12 total, 11 up, 11 in 2024-04-15T18:16:31.523 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:31.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:31.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:31.671 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:31.801 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:31.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:31.862 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:31.906 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:31.935 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:31.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 62155 62139 88.2581 79.5625 0.018476 0.0113233 2024-04-15T18:16:31.955 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:31.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:31.983 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:4193 2.3 2:c3a89f7e:::benchmark_data_smithi159_80260_object4192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 282.34625987999999, 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29915336799999998, 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:31.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 4193 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631635+0000", 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631636+0000", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631699+0000", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3040000000000006e-05 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631709+0000", 2024-04-15T18:16:31.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8169999999999994e-06 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631714+0000", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4399999999999996e-06 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631749+0000", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4291000000000001e-05 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631839+0000", 2024-04-15T18:16:31.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0038999999999996e-05 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.631923+0000", 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3978000000000001e-05 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.632245+0000", 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00032240399999999998 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:31.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.835583+0000", 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203337675 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930731+0000", 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.095148014000000003 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930764+0000", 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3596999999999997e-05 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.930789+0000", 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4340000000000001e-05 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:65717 2.2 2:4ab79f1f:::benchmark_data_smithi159_80260_object65716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 218.49351977200001, 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27045140699999998, 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:31.990 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 65717 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484375+0000", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484376+0000", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9599999999999998e-07 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484427+0000", 2024-04-15T18:16:31.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1204e-05 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484439+0000", 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1138000000000001e-05 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484444+0000", 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8250000000000001e-06 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:31.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484492+0000", 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7579e-05 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484623+0000", 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013083499999999999 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.484784+0000", 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000161008 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486198+0000", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014138919999999999 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519765+0000", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.033567630000000001 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754699+0000", 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23493340300000001 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754759+0000", 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0655000000000001e-05 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754827+0000", 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7341999999999993e-05 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65717 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 218.49159289599999, 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.268538165, 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486302+0000", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486303+0000", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3799999999999997e-07 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.486306+0000", 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5760000000000002e-06 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753379+0000", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26707288800000001 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753389+0000", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4569999999999994e-06 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754686+0000", 2024-04-15T18:16:31.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001297749 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754693+0000", 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6730000000000003e-06 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754840+0000", 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014728399999999999 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:65740 2.2 2:41578887:::benchmark_data_smithi159_80260_object65739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 218.460520433, 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23761864199999999, 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 65740 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517375+0000", 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517376+0000", 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.288e-06 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517435+0000", 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9138000000000002e-05 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517458+0000", 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2416e-05 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517465+0000", 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6739999999999997e-06 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517511+0000", 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5875999999999999e-05 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517608+0000", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6705000000000003e-05 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.517706+0000", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7916000000000002e-05 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519639+0000", 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019331750000000001 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754588+0000", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.234948666 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754886+0000", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000298001 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754939+0000", 2024-04-15T18:16:32.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3599e-05 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754993+0000", 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4187999999999997e-05 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65740 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 218.45820502399999, 2024-04-15T18:16:32.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23491751399999999, 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519690+0000", 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519691+0000", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0999999999999998e-07 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519695+0000", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8770000000000003e-06 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.751599+0000", 2024-04-15T18:16:32.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23190482500000001 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.751605+0000", 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4070000000000001e-06 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754573+0000", 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002967819 2024-04-15T18:16:32.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754581+0000", 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1349999999999992e-06 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754608+0000", 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6741e-05 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:65740 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 218.45798152099999, 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.235089255, 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519914+0000", 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9999999999999997e-07 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.519918+0000", 2024-04-15T18:16:32.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.388e-06 2024-04-15T18:16:32.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753511+0000", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23359376100000001 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.753519+0000", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3590000000000002e-06 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754874+0000", 2024-04-15T18:16:32.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013550649999999999 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.754880+0000", 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1859999999999997e-06 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:53.755003+0000", 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122896 2024-04-15T18:16:32.016 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:68954 2.2 2:4d9d74ec:::benchmark_data_smithi159_80260_object68953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:16:32.017 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 215.59952280900001, 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23112603400000001, 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.018 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 68954 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378372+0000", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378373+0000", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378440+0000", 2024-04-15T18:16:32.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6950000000000001e-05 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378456+0000", 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6177000000000001e-05 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378463+0000", 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3100000000000003e-06 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378509+0000", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5544999999999998e-05 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378587+0000", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8337999999999994e-05 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.378701+0000", 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113859 2024-04-15T18:16:32.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.380314+0000", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016132430000000001 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.380987+0000", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067303000000000003 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609420+0000", 2024-04-15T18:16:32.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.228432627 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609456+0000", 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6075000000000003e-05 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609498+0000", 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2230000000000001e-05 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.024 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:68954 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 215.596722406, 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22833046300000001, 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.025 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381173+0000", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8999999999999998e-07 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381175+0000", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3520000000000001e-06 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609375+0000", 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22819928 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609380+0000", 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9200000000000003e-06 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609412+0000", 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2533999999999997e-05 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609417+0000", 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6260000000000003e-06 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.609503+0000", 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6460999999999998e-05 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.029 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.030 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:68957 2.2 2:4af888cc:::benchmark_data_smithi159_80260_object68956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:16:32.030 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:16:32.030 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 215.59660719600001, 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23726344099999999, 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.031 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 68957 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381288+0000", 2024-04-15T18:16:32.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381289+0000", 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699999999999997e-07 2024-04-15T18:16:32.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381393+0000", 2024-04-15T18:16:32.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010438300000000001 2024-04-15T18:16:32.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381402+0000", 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4200000000000007e-06 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381406+0000", 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6569999999999996e-06 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381434+0000", 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7229999999999998e-05 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381520+0000", 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6051999999999995e-05 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.381604+0000", 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4414999999999997e-05 2024-04-15T18:16:32.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.383060+0000", 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014556459999999999 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.383296+0000", 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000236689 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618491+0000", 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23519432200000001 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618525+0000", 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4384000000000001e-05 2024-04-15T18:16:32.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618551+0000", 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6325999999999998e-05 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.043 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:68957 2.2 e150/114 ondisk, result = 0)", 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 215.59307597, 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23373741200000001, 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.044 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:16:32.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384819+0000", 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384820+0000", 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.37e-07 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.384823+0000", 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.393e-06 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618437+0000", 2024-04-15T18:16:32.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23361385500000001 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618442+0000", 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2449999999999998e-06 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618483+0000", 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0392999999999998e-05 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618487+0000", 2024-04-15T18:16:32.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7809999999999996e-06 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:56.618557+0000", 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9208000000000006e-05 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.051 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:89108 2.3 e156/114 ondisk, result = 0)", 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.83926329299999, 2024-04-15T18:16:32.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40252157700000002, 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6599999999999996e-07 2024-04-15T18:16:32.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138636+0000", 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.348e-06 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540769+0000", 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40213289499999999 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540782+0000", 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3692999999999999e-05 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540945+0000", 2024-04-15T18:16:32.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162227 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540962+0000", 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7819999999999999e-05 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.541153+0000", 2024-04-15T18:16:32.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019102799999999999 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:97764 2.2 2:452aef37:::benchmark_data_smithi159_80260_object8651:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 185.37111773800001, 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22587837799999999, 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.060 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 97764 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:16:32.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606777+0000", 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606778+0000", 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7499999999999998e-07 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606784+0000", 2024-04-15T18:16:32.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1930000000000001e-06 2024-04-15T18:16:32.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606796+0000", 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2588999999999999e-05 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606802+0000", 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2940000000000002e-06 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.606876+0000", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3916000000000002e-05 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.607010+0000", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013359799999999999 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.607143+0000", 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013261800000000001 2024-04-15T18:16:32.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.609507+0000", 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023643539999999999 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.610155+0000", 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00064824200000000005 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832542+0000", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22238657100000001 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832605+0000", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3528999999999994e-05 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:26.832656+0000", 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0498999999999999e-05 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:129248 2.3 2:c342c84b:::benchmark_data_smithi159_80260_object40135:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:00.074205+0000", 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 151.903690442, 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22284021200000001, 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 129248 2024-04-15T18:16:32.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074205+0000", 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074205+0000", 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074206+0000", 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2440000000000001e-06 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074211+0000", 2024-04-15T18:16:32.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1510000000000003e-06 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074224+0000", 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3353000000000001e-05 2024-04-15T18:16:32.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074232+0000", 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.644e-06 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074295+0000", 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3231999999999994e-05 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.074 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074435+0000", 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000139247 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:16:32.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.074557+0000", 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012239 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.078218+0000", 2024-04-15T18:16:32.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036608180000000001 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.078548+0000", 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033059400000000003 2024-04-15T18:16:32.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.296974+0000", 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21842547600000001 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.297011+0000", 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7316e-05 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:00.297045+0000", 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3747000000000001e-05 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157649 2.3 2:c89d4810:::benchmark_data_smithi159_80260_object68536:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.93828784599999, 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26908227299999998, 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.081 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157649 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.082 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.083 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039609+0000", 2024-04-15T18:16:32.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6050000000000001e-06 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039617+0000", 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4190000000000002e-06 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039636+0000", 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8454e-05 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039645+0000", 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2110000000000004e-06 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039704+0000", 2024-04-15T18:16:32.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8634999999999997e-05 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039708+0000", 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5199999999999999e-06 2024-04-15T18:16:32.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043065+0000", 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033569559999999999 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043169+0000", 2024-04-15T18:16:32.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103738 2024-04-15T18:16:32.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043236+0000", 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6959999999999996e-05 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.047179+0000", 2024-04-15T18:16:32.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039432829999999997 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.048394+0000", 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00121502 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308578+0000", 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26018443400000002 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308639+0000", 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0284000000000001e-05 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308690+0000", 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0754000000000002e-05 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157667 2.3 2:ce409a58:::benchmark_data_smithi159_80260_object68554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.895196657, 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25352618900000001, 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.095 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157667 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082699+0000", 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.64e-07 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082704+0000", 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9840000000000001e-06 2024-04-15T18:16:32.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082715+0000", 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0516999999999999e-05 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082721+0000", 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6300000000000003e-06 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082759+0000", 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8463000000000001e-05 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082852+0000", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3418999999999994e-05 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082928+0000", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5815999999999994e-05 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086226+0000", 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032979839999999999 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086901+0000", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067478099999999995 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336152+0000", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.249250834 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336190+0000", 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7737999999999997e-05 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336225+0000", 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5058999999999998e-05 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:157649 2.3 e178/114 ondisk, result = 0)", 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.086737+0000", 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.891157924, 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22195849100000001, 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.104 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086737+0000", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086737+0000", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086738+0000", 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1800000000000004e-07 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086743+0000", 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5970000000000002e-06 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308484+0000", 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22174100999999999 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308507+0000", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2929999999999999e-05 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308562+0000", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5114999999999997e-05 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308572+0000", 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0149999999999999e-05 2024-04-15T18:16:32.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308696+0000", 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000123771 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157688 2.3 2:c5c2b454:::benchmark_data_smithi159_80260_object68575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 112.631412158, 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.316335582, 2024-04-15T18:16:32.108 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157688 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346485+0000", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.097e-06 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346496+0000", 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1211e-05 2024-04-15T18:16:32.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346520+0000", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4124000000000001e-05 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346531+0000", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0691e-05 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346590+0000", 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9417999999999999e-05 2024-04-15T18:16:32.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346693+0000", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010207700000000001 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346776+0000", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3354000000000002e-05 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.350571+0000", 2024-04-15T18:16:32.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037947829999999999 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.351115+0000", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054438000000000002 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662733+0000", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.311618171 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662772+0000", 2024-04-15T18:16:32.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9242000000000001e-05 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662819+0000", 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6034e-05 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:158069 2.2 2:4af888cc:::benchmark_data_smithi159_80260_object68956:head [delete] snapc 0=[] ondisk+write+known_if_redirected e179)", 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.40757963599999, 2024-04-15T18:16:32.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27351397799999999, 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 158069 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570316+0000", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7300000000000004e-07 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570321+0000", 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5569999999999997e-06 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570330+0000", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3389999999999999e-06 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570337+0000", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5250000000000002e-06 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570370+0000", 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2932999999999998e-05 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570459+0000", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8788000000000004e-05 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570531+0000", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2027999999999995e-05 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573921+0000", 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033902889999999999 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.601514+0000", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027593414 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843733+0000", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24221832400000001 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843785+0000", 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2438999999999997e-05 2024-04-15T18:16:32.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843829+0000", 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4369000000000002e-05 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:158069 2.2 e179/114 ondisk, result = 0)", 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.40400666799999, 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26994574500000001, 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.120 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573889+0000", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4499999999999996e-07 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573895+0000", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843667+0000", 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26977213900000002 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843673+0000", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5960000000000003e-06 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843719+0000", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6405e-05 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843726+0000", 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.323e-06 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843834+0000", 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000107898 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:17054 3.5 3:ab47d494:::benchmark_data_smithi159_91621_object17053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.374102313999998, 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26525695500000002, 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:32.124 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 17054 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1200000000000003e-07 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603877+0000", 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3918999999999994e-05 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603888+0000", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1080999999999999e-05 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603894+0000", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5860000000000004e-06 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603940+0000", 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5918000000000003e-05 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604029+0000", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9003000000000001e-05 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,6", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604111+0000", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2146999999999998e-05 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604861+0000", 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075021899999999999 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.606482+0000", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001620687 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.868983+0000", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.262501182 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.869017+0000", 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4103999999999997e-05 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.869050+0000", 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2496999999999997e-05 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:32.130 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:16:32.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:32.131 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:16:32.162 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:32.185 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:32.187 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:32 smithi159 ceph-mon[45336]: pgmap v607: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:16:32.187 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3755620885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.187 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:32.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:32.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:32 smithi082 ceph-mon[44206]: pgmap v607: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:16:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3755620885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.325 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:32.351 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:32.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:32.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:32 smithi049 ceph-mon[40087]: pgmap v607: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:16:32.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3755620885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:16:32.494 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:32.706 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:32.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:32.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:16:32.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:32.864 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:32.888 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:32.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:32.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 63398 63382 88.0229 77.6875 0.00446361 0.0113507 2024-04-15T18:16:32.993 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:33.024 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:33.046 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:33.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:33.138 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:33.150 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:33.180 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:33.210 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:33.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:33.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: osdmap e217: 12 total, 11 up, 11 in 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:16:33.358 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:33.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:33.395 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:33.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: osdmap e217: 12 total, 11 up, 11 in 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:16:33.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: osdmap e217: 12 total, 11 up, 11 in 2024-04-15T18:16:33.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:16:33.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:16:33.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:16:33.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:33.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:16:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:33.944 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 64689 64673 87.8633 80.6875 0.00528158 0.0113765 2024-04-15T18:16:34.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:34 smithi159 ceph-mon[45336]: osdmap e218: 12 total, 11 up, 11 in 2024-04-15T18:16:34.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:34 smithi159 ceph-mon[45336]: pgmap v610: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:34.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:34 smithi082 ceph-mon[44206]: osdmap e218: 12 total, 11 up, 11 in 2024-04-15T18:16:34.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:34 smithi082 ceph-mon[44206]: pgmap v610: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:34.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:34 smithi049 ceph-mon[40087]: osdmap e218: 12 total, 11 up, 11 in 2024-04-15T18:16:34.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:34 smithi049 ceph-mon[40087]: pgmap v610: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:16:34.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 15 65861 65846 87.5536 73.3125 0.00527516 0.0114153 2024-04-15T18:16:35.099 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:35 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:35.099 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:35 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:35.100 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:35 smithi082 ceph-mon[44206]: osdmap e219: 12 total, 11 up, 11 in 2024-04-15T18:16:35.138 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:16:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:16:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.138 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:66018 3.a 3:5db39921:::benchmark_data_smithi159_91621_object66017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:35.128357+0000", 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.006045234, 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0060708840000000003, 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66018 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128357+0000", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128357+0000", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128358+0000", 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1969999999999999e-06 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128418+0000", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9806999999999998e-05 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128431+0000", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3552e-05 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128438+0000", 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9090000000000003e-06 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128499+0000", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0476999999999997e-05 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128593+0000", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4860999999999999e-05 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.128699+0000", 2024-04-15T18:16:35.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105265 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.130990+0000", 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022913790000000001 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:35.132055+0000", 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001064951 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:16:35.143 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:16:35.144 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 1 2024-04-15T18:16:35.144 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:16:35.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:35 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:35.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:35 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:35.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:35 smithi159 ceph-mon[45336]: osdmap e219: 12 total, 11 up, 11 in 2024-04-15T18:16:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:35 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:35 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:35.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:35 smithi049 ceph-mon[40087]: osdmap e219: 12 total, 11 up, 11 in 2024-04-15T18:16:35.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:16:35.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 67094 67078 87.3336 77 0.00718473 0.0114447 2024-04-15T18:16:36.268 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:36.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:36 smithi082 ceph-mon[44206]: pgmap v612: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:16:36.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:36 smithi159 ceph-mon[45336]: pgmap v612: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:16:36.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:36 smithi049 ceph-mon[40087]: pgmap v612: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:16:36.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 68202 68186 86.9645 69.25 0.0193331 0.0114934 2024-04-15T18:16:37.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 69260 69244 86.5476 66.125 0.011395 0.0115494 2024-04-15T18:16:38.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:16:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:38.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:16:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:38 smithi082 ceph-mon[44206]: osdmap e220: 12 total, 11 up, 11 in 2024-04-15T18:16:38.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:38 smithi082 ceph-mon[44206]: pgmap v614: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:38.302 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:16:38.302 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:16:38.303 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:16:38.303 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:16:38.303 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:16:38.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:38.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:38.410 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:38.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:38.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:38 smithi159 ceph-mon[45336]: osdmap e220: 12 total, 11 up, 11 in 2024-04-15T18:16:38.411 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:38 smithi159 ceph-mon[45336]: pgmap v614: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:38 smithi049 ceph-mon[40087]: osdmap e220: 12 total, 11 up, 11 in 2024-04-15T18:16:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:38 smithi049 ceph-mon[40087]: pgmap v614: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:16:38.586 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:38.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:38.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:38.734 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:38.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 70625 70609 86.5232 85.3125 0.00653428 0.0115529 2024-04-15T18:16:39.150 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:16:39.195 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:39.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:39.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:39.338 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:39.365 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:39.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:39.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:39.509 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:39.532 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:39.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:39.558 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:39.638 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:39.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:39.826 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:39.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:39.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:39.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 71623 71607 86.0587 62.375 0.00525148 0.0116139 2024-04-15T18:16:39.965 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:39.988 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:39.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:40.093 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:40.134 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:40.213 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: pgmap v615: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1k op/s 2024-04-15T18:16:40.213 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:40.308 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:40.341 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:40.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:40.398 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:40.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:40.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: pgmap v615: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1k op/s 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3728718121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:40.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:40.569 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: pgmap v615: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1k op/s 2024-04-15T18:16:40.569 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3728718121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.570 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:40.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:40.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:40.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:40.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:40.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:40.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:40.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:16:40.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3728718121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:16:40.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:16:40.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:40.703 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:40.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 72600 72584 85.587 61.0625 0.00488425 0.0116789 2024-04-15T18:16:41.053 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:41.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:41.159 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:41.189 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:41.221 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:41.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:41 smithi082 ceph-mon[44206]: 3.8 scrub starts 2024-04-15T18:16:41.325 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:41.359 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:41.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:41.384 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:41.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:41 smithi049 ceph-mon[40087]: 3.8 scrub starts 2024-04-15T18:16:41.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:41.518 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:41.534 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:41.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:41 smithi159 ceph-mon[45336]: 3.8 scrub starts 2024-04-15T18:16:41.689 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:41.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:41.758 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:41.758 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:41.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:41.843 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:41.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 73693 73677 85.267 68.3125 0.00428298 0.0117224 2024-04-15T18:16:42.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: 3.4 scrub starts 2024-04-15T18:16:42.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:16:42.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:42 smithi049 ceph-mon[40087]: 3.8 scrub ok 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: 3.4 scrub starts 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:16:42.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:42 smithi082 ceph-mon[44206]: 3.8 scrub ok 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: 3.4 scrub starts 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: pgmap v616: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:16:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:42 smithi159 ceph-mon[45336]: 3.8 scrub ok 2024-04-15T18:16:42.807 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:42.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:42.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:16:42.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:42.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 74752 74736 84.92 66.1875 0.00584146 0.0117701 2024-04-15T18:16:42.955 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:43.035 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:43.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:43.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:43.167 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:43.194 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:43.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:43.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:43.339 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:43.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:16:43.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:43 smithi049 ceph-mon[40087]: 3.4 scrub ok 2024-04-15T18:16:43.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:43 smithi049 ceph-mon[40087]: 3.d scrub starts 2024-04-15T18:16:43.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:43 smithi082 ceph-mon[44206]: 3.4 scrub ok 2024-04-15T18:16:43.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:43 smithi082 ceph-mon[44206]: 3.d scrub starts 2024-04-15T18:16:43.502 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:43.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:43.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:43.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:43 smithi159 ceph-mon[45336]: 3.4 scrub ok 2024-04-15T18:16:43.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:43 smithi159 ceph-mon[45336]: 3.d scrub starts 2024-04-15T18:16:43.661 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:43.698 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:43.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:43.743 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:43.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:43.841 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:43.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:43.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:43.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 75716 75700 84.4794 60.25 0.0196365 0.0118321 2024-04-15T18:16:43.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:44.007 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:44.035 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:44.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:44.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:44.188 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:44.250 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:16:44.515 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:44 smithi082 ceph-mon[44206]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:16:44.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4261907663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.546 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:44.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:44.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:44 smithi159 ceph-mon[45336]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:16:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4261907663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.677 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:44 smithi049 ceph-mon[40087]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:16:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4261907663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:44.828 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:44.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:44.933 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:44.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 76548 76532 83.9095 52 0.00839095 0.0119124 2024-04-15T18:16:44.975 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:45.115 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:45.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:45.170 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:45.170 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:45.222 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:45.259 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:45.283 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:45.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:45.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:45.417 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: 3.9 scrub starts 2024-04-15T18:16:45.417 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: 3.d scrub ok 2024-04-15T18:16:45.417 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: 3.a scrub starts 2024-04-15T18:16:45.417 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:45.418 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:16:45.418 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:45.418 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:45 smithi082 ceph-mon[44206]: osdmap e221: 12 total, 11 up, 11 in 2024-04-15T18:16:45.420 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:45.447 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:45.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:45.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: 3.9 scrub starts 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: 3.d scrub ok 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: 3.a scrub starts 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:45.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:45 smithi049 ceph-mon[40087]: osdmap e221: 12 total, 11 up, 11 in 2024-04-15T18:16:45.591 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:45.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: 3.9 scrub starts 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: 3.d scrub ok 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: 3.a scrub starts 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:16:45.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:45 smithi159 ceph-mon[45336]: osdmap e221: 12 total, 11 up, 11 in 2024-04-15T18:16:45.843 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:16:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:45.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:16:45.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:16:45.945 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 77573 77557 83.5672 64.0625 0.0134043 0.0119608 2024-04-15T18:16:45.950 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:45.979 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.003 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:46.009 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:46.110 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:46.168 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.194 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:46.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:46.287 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:46.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:46.331 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.355 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:46.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:46.461 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:16:46.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:46 smithi049 ceph-mon[40087]: 3.9 scrub ok 2024-04-15T18:16:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:46 smithi049 ceph-mon[40087]: pgmap v619: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:46.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:46 smithi049 ceph-mon[40087]: 3.a scrub ok 2024-04-15T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:46 smithi159 ceph-mon[45336]: 3.9 scrub ok 2024-04-15T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:46 smithi159 ceph-mon[45336]: pgmap v619: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:46.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:46 smithi159 ceph-mon[45336]: 3.a scrub ok 2024-04-15T18:16:46.494 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:46.518 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:16:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:46.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:16:46.534 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:46 smithi082 ceph-mon[44206]: 3.9 scrub ok 2024-04-15T18:16:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:46 smithi082 ceph-mon[44206]: pgmap v619: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:16:46.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:46 smithi082 ceph-mon[44206]: 3.a scrub ok 2024-04-15T18:16:46.566 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:46.594 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:46.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:46.624 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:46.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:46.661 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.681 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:46.694 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:46.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:46.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:46.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:46.768 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-15T18:16:46.768 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-04-15T18:16:46.768 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T18:16:46.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3 2024-04-15T18:16:46.774 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:46.776 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:46.776 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:46.776 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:46.776 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:46.776 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:46.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:46.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:46.812 INFO:journalctl@ceph.osd.3.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:16:46.835 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:46.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 78620 78604 83.2598 65.4375 0.00713545 0.0120048 2024-04-15T18:16:47.120 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:46 smithi049 systemd[1]: Starting Ceph osd.3 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:16:47.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:16:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osdmap e222: 12 total, 11 up, 11 in 2024-04-15T18:16:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3722208605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:47.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osdmap e222: 12 total, 11 up, 11 in 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3722208605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:47.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:47.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:47.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:47.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osdmap e222: 12 total, 11 up, 11 in 2024-04-15T18:16:47.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3722208605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:47.685 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:47.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:16:47.943516+0000 min lat: 0.00112542 max lat: 0.279966 avg lat: 0.0120752 2024-04-15T18:16:47.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:16:47.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 79482 79466 82.77 53.875 0.00581327 0.0120752 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:16:48.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:48 smithi049 ceph-mon[40087]: 3.b deep-scrub starts 2024-04-15T18:16:48.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:16:48.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:48 smithi082 ceph-mon[44206]: 3.b deep-scrub starts 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:16:48.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:48 smithi159 ceph-mon[45336]: 3.b deep-scrub starts 2024-04-15T18:16:48.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 80270 80254 82.2204 49.25 0.00646611 0.012155 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T18:16:49.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:48 smithi049 bash[131517]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-15T18:16:49.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:16:49.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 81291 81275 81.9234 63.8125 0.00861034 0.0122012 2024-04-15T18:16:50.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:50 smithi082 ceph-mon[44206]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s 2024-04-15T18:16:50.614 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:50.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:50 smithi159 ceph-mon[45336]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s 2024-04-15T18:16:50.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:50 smithi049 ceph-mon[40087]: pgmap v622: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 46 MiB/s wr, 743 op/s 2024-04-15T18:16:50.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 82187 82171 81.5119 56 0.0144819 0.0122631 2024-04-15T18:16:51.262 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:16:51.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:16:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2782642450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:51 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.478 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:51 smithi049 bash[131936]: ad2fe222b9e5136b86abc02335051dbd6c675fd4768b4842a2fc6152241c8822 2024-04-15T18:16:51.478 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:51 smithi049 systemd[1]: Started Ceph osd.3 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:16:51.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2782642450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:51 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2782642450' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:51 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:16:51.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 82964 82948 80.997 48.5625 0.00648932 0.0123389 2024-04-15T18:16:52.227 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:51 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:16:51.842+0000 7f6c758027c0 -1 Falling back to public interface 2024-04-15T18:16:52.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:52 smithi082 ceph-mon[44206]: pgmap v623: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:16:52.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:52 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:52.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:52.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:52 smithi082 ceph-mon[44206]: osdmap e223: 12 total, 11 up, 11 in 2024-04-15T18:16:52.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:52 smithi159 ceph-mon[45336]: pgmap v623: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:16:52.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:52 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:52.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:52.660 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:52 smithi159 ceph-mon[45336]: osdmap e223: 12 total, 11 up, 11 in 2024-04-15T18:16:52.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:52 smithi049 ceph-mon[40087]: pgmap v623: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:16:52.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:52 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:16:52.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:16:52.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:52 smithi049 ceph-mon[40087]: osdmap e223: 12 total, 11 up, 11 in 2024-04-15T18:16:52.884 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:52.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:52.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 83949 83933 80.6979 61.5625 0.0420337 0.0123853 2024-04-15T18:16:52.990 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:53.025 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:53.332 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:53.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:16:53.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:16:53.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:16:53.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:16:53.757 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:16:53.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:16:53.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:16:53.849 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:16:53.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:16:53.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:16:53.939 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:16:53.939 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:16:53.939 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:16:53.939 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:16:53.939 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:16:53.940 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:16:53.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 84791 84775 80.2725 52.625 0.0120922 0.0124526 2024-04-15T18:16:54.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: pgmap v625: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-04-15T18:16:54.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2612282130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: pgmap v625: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-04-15T18:16:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2612282130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: pgmap v625: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2612282130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:54.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 15 85640 85625 79.8673 53.125 0.00561136 0.0125145 2024-04-15T18:16:55.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:16:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:16:55.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:16:55.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:55.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:55.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:55.316 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:55.345 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:55.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:55.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:55.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-mon[40087]: osdmap e224: 12 total, 11 up, 11 in 2024-04-15T18:16:55.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:55.416 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:55 smithi082 ceph-mon[44206]: osdmap e224: 12 total, 11 up, 11 in 2024-04-15T18:16:55.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:16:55.484 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:55.509 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:16:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:55.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:16:55.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:55.657 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:55.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:55.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:55 smithi159 ceph-mon[45336]: osdmap e224: 12 total, 11 up, 11 in 2024-04-15T18:16:55.686 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:55.690 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:55 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:16:55.406+0000 7f6c758027c0 -1 osd.3 177 log_to_monitors true 2024-04-15T18:16:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:55.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:55.729 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:55.787 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:55.787 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:55.792 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:55.837 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:55.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 86540 86524 79.519 56.1875 0.0628494 0.0125707 2024-04-15T18:16:56.197 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:56.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:56.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:16:56.353 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:56.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:56 smithi049 ceph-mon[40087]: pgmap v627: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 56 MiB/s wr, 892 op/s 2024-04-15T18:16:56.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:56 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:56 smithi049 ceph-mon[40087]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:56 smithi082 ceph-mon[44206]: pgmap v627: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 56 MiB/s wr, 892 op/s 2024-04-15T18:16:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:56 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:56 smithi082 ceph-mon[44206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.594 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:16:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:56.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:16:56.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:16:56.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:56 smithi159 ceph-mon[45336]: pgmap v627: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 56 MiB/s wr, 892 op/s 2024-04-15T18:16:56.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:56 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:56 smithi159 ceph-mon[45336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T18:16:56.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:56.738 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:56.782 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:56.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:56.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:16:56.919 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:56.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 87409 87393 79.1536 54.3125 0.0186676 0.0126284 2024-04-15T18:16:57.067 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:16:57.088 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:16:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:16:57.089 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:16:57.089 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:16:57.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:16:57.283 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:16:57.001+0000 7f6c66c5a700 -1 osd.3 225 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:16:57.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:57 smithi082 ceph-mon[44206]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:16:57.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:57 smithi082 ceph-mon[44206]: osdmap e225: 12 total, 11 up, 11 in 2024-04-15T18:16:57.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:57 smithi082 ceph-mon[44206]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:57 smithi082 ceph-mon[44206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:57.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:57 smithi159 ceph-mon[45336]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:16:57.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:57 smithi159 ceph-mon[45336]: osdmap e225: 12 total, 11 up, 11 in 2024-04-15T18:16:57.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:57 smithi159 ceph-mon[45336]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:57 smithi159 ceph-mon[45336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:57.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-mon[40087]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T18:16:57.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-mon[40087]: osdmap e225: 12 total, 11 up, 11 in 2024-04-15T18:16:57.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-mon[40087]: from='osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-mon[40087]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:16:57.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:57.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 88175 88159 78.7067 47.875 0.00289603 0.0126943 2024-04-15T18:16:58.229 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:16:58.229 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-04-15T18:16:58.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:16:58.426 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:58 smithi082 ceph-mon[44206]: pgmap v629: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s 2024-04-15T18:16:58.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:58 smithi082 ceph-mon[44206]: osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877] boot 2024-04-15T18:16:58.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:58 smithi082 ceph-mon[44206]: osdmap e226: 12 total, 12 up, 11 in 2024-04-15T18:16:58.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:58.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:58 smithi049 ceph-mon[40087]: pgmap v629: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s 2024-04-15T18:16:58.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:58 smithi049 ceph-mon[40087]: osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877] boot 2024-04-15T18:16:58.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:58 smithi049 ceph-mon[40087]: osdmap e226: 12 total, 12 up, 11 in 2024-04-15T18:16:58.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:58.473 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:16:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:58.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:16:58.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:58.607 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:58 smithi159 ceph-mon[45336]: pgmap v629: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s 2024-04-15T18:16:58.607 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:58 smithi159 ceph-mon[45336]: osd.3 [v2:172.21.15.49:6826/1036848877,v1:172.21.15.49:6827/1036848877] boot 2024-04-15T18:16:58.608 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:58 smithi159 ceph-mon[45336]: osdmap e226: 12 total, 12 up, 11 in 2024-04-15T18:16:58.608 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T18:16:58.610 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:58.634 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:58.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:58.738 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:58.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:16:58.771 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:58.788 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:16:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:16:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:58.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:16:58.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:16:58.934 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:58.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 88943 88927 78.2742 48 0.024 0.0127686 2024-04-15T18:16:58.955 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:16:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:58.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:16:59.062 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:16:59.110 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:59.116 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:16:59.116 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:16:59.135 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:16:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:59.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:16:59.242 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:16:59.303 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:59.319 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:16:59.418 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: osdmap e227: 12 total, 12 up, 11 in 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/783464637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.422 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:16:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:59.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:16:59.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:59.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:16:59.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:16:59.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: osdmap e227: 12 total, 12 up, 11 in 2024-04-15T18:16:59.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/783464637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.523 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:16:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:16:59.565 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:16:59.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:16:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:16:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:16:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: osdmap e227: 12 total, 12 up, 11 in 2024-04-15T18:16:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/783464637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:16:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:16:59.824 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:16:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:16:59.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:16:59.876 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:16:59.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:16:59.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 89922 89906 78.0368 61.1875 0.00812567 0.0128085 2024-04-15T18:16:59.966 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:00.245 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:00.304 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:00 smithi159 ceph-mon[45336]: pgmap v632: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-04-15T18:17:00.304 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:00 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:00.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:00.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:00 smithi159 ceph-mon[45336]: osdmap e228: 12 total, 12 up, 11 in 2024-04-15T18:17:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:00 smithi049 ceph-mon[40087]: pgmap v632: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-04-15T18:17:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:00 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:00 smithi049 ceph-mon[40087]: osdmap e228: 12 total, 12 up, 11 in 2024-04-15T18:17:00.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:00 smithi082 ceph-mon[44206]: pgmap v632: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-04-15T18:17:00.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:00 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:00.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:00.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:00 smithi082 ceph-mon[44206]: osdmap e228: 12 total, 12 up, 11 in 2024-04-15T18:17:00.821 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:00.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:00.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:00.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 90996 90980 77.8873 67.125 0.00472755 0.0128337 2024-04-15T18:17:00.966 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:01.123 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:01.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:01.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:01.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:01.432 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:01.463 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:01.483 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:01.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:01.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:01.628 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:01.661 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:01.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:01.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:01.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:01.802 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:01.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:01.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 15 92070 92055 77.7426 67.1875 0.022639 0.0128583 2024-04-15T18:17:01.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:01.994 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:02.021 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:02.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:02.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:02.174 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:02.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:02 smithi159 ceph-mon[45336]: pgmap v634: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-04-15T18:17:02.384 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:02.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:02.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:02 smithi049 ceph-mon[40087]: pgmap v634: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-04-15T18:17:02.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:17:02.402 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:17:02.402 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:17:02.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:02.526 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:02.679 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:02.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:02.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:02.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:02 smithi082 ceph-mon[44206]: pgmap v634: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-04-15T18:17:02.825 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:02.946 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 93076 93060 77.5435 62.8125 0.00508376 0.0128907 2024-04-15T18:17:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:03.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:03.121 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:03.151 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:03.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:03.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:03.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:17:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:17:03.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:03.229 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:17:03.231 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:03.259 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:03.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:03.319 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:03.349 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:03.380 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:03.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:03.436 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:03.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3678754554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:03.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:03.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:03.483 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:03.540 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:03.581 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3678754554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:03.587 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:03.614 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:03.617 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:03.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:03.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:03.766 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:03.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3678754554' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:03.898 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:03.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:03.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 94085 94069 77.3529 63.0625 0.015607 0.0129223 2024-04-15T18:17:04.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:04.037 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:04.111 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:04.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:04.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:04.252 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:04.457 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: pgmap v635: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 51 MiB/s wr, 811 op/s 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.459 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:04.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:04.561 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:04.586 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: pgmap v635: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 51 MiB/s wr, 811 op/s 2024-04-15T18:17:04.586 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.587 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.588 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:04.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: pgmap v635: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 51 MiB/s wr, 811 op/s 2024-04-15T18:17:04.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:04.606 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:04.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:04.710 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:04.749 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:04.779 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:04.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:04.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:04.919 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:04.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 95039 95023 77.1226 59.625 0.00607621 0.0129617 2024-04-15T18:17:04.962 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:04.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:04.994 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-04-15T18:17:03.103122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5931205152,"num_objects":90496,"num_object_clones":0,"num_object_copies":271488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90496,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":90675,"num_write_kb":5794554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92845,"ondisk_log_size":92845,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18540732,"kb_used_data":17368944,"kb_used_omap":17,"kb_used_meta":1171630,"kb_avail":1012655940,"statfs":{"total":1055945392128,"available":1036959682560,"internally_reserved":0,"allocated":17785798656,"data_stored":17782746443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17550,"internal_metadata":1199750002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":311872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004586"},"pg_stats":[{"pgid":"3.f","version":"228'5662","reported_seq":5730,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.961913+0000","last_change":"2024-04-15T18:15:39.057796+0000","last_active":"2024-04-15T18:17:00.961913+0000","last_peered":"2024-04-15T18:17:00.961913+0000","last_clean":"2024-04-15T18:17:00.961913+0000","last_became_active":"2024-04-15T18:15:39.057408+0000","last_became_peered":"2024-04-15T18:15:39.057408+0000","last_unstale":"2024-04-15T18:17:00.961913+0000","last_undegraded":"2024-04-15T18:17:00.961913+0000","last_fullsized":"2024-04-15T18:17:00.961913+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5662,"log_dups_size":0,"ondisk_log_size":5662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:02.741971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371064832,"num_objects":5662,"num_object_clones":0,"num_object_copies":16986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":362368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"228'5581","reported_seq":5648,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.655252+0000","last_change":"2024-04-15T18:15:39.061855+0000","last_active":"2024-04-15T18:16:59.655252+0000","last_peered":"2024-04-15T18:16:59.655252+0000","last_clean":"2024-04-15T18:16:59.655252+0000","last_became_active":"2024-04-15T18:15:39.061483+0000","last_became_peered":"2024-04-15T18:15:39.061483+0000","last_unstale":"2024-04-15T18:16:59.655252+0000","last_undegraded":"2024-04-15T18:16:59.655252+0000","last_fullsized":"2024-04-15T18:16:59.655252+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:00.740160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365756416,"num_objects":5581,"num_object_clones":0,"num_object_copies":16743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":357184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"228'5605","reported_seq":5728,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.985051+0000","last_change":"2024-04-15T18:16:43.894657+0000","last_active":"2024-04-15T18:17:00.985051+0000","last_peered":"2024-04-15T18:17:00.985051+0000","last_clean":"2024-04-15T18:17:00.985051+0000","last_became_active":"2024-04-15T18:15:39.058838+0000","last_became_peered":"2024-04-15T18:15:39.058838+0000","last_unstale":"2024-04-15T18:17:00.985051+0000","last_undegraded":"2024-04-15T18:17:00.985051+0000","last_fullsized":"2024-04-15T18:17:00.985051+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'4696","last_scrub_stamp":"2024-04-15T18:16:43.894471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:16:43.894471+0000","objects_scrubbed":4664,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:56.968332+0000","scrub_duration":1629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367329280,"num_objects":5605,"num_object_clones":0,"num_object_copies":16815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":358720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"228'5483","reported_seq":5550,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.425067+0000","last_change":"2024-04-15T18:15:39.436047+0000","last_active":"2024-04-15T18:16:59.425067+0000","last_peered":"2024-04-15T18:16:59.425067+0000","last_clean":"2024-04-15T18:16:59.425067+0000","last_became_active":"2024-04-15T18:15:39.435717+0000","last_became_peered":"2024-04-15T18:15:39.435717+0000","last_unstale":"2024-04-15T18:16:59.425067+0000","last_undegraded":"2024-04-15T18:16:59.425067+0000","last_fullsized":"2024-04-15T18:16:59.425067+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359333888,"num_objects":5483,"num_object_clones":0,"num_object_copies":16449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":350912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,10],"acting":[5,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"228'7912","reported_seq":8130,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.391678+0000","last_change":"2024-04-15T18:16:59.402871+0000","last_active":"2024-04-15T18:17:00.391678+0000","last_peered":"2024-04-15T18:17:00.391678+0000","last_clean":"2024-04-15T18:17:00.391678+0000","last_became_active":"2024-04-15T18:15:39.059365+0000","last_became_peered":"2024-04-15T18:15:39.059365+0000","last_unstale":"2024-04-15T18:17:00.391678+0000","last_undegraded":"2024-04-15T18:17:00.391678+0000","last_fullsized":"2024-04-15T18:17:00.391678+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":3667,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:04.396791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373424128,"num_objects":5698,"num_object_clones":0,"num_object_copies":17094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5698,"num_write_kb":364672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"228'5685","reported_seq":5810,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.920656+0000","last_change":"2024-04-15T18:16:45.808089+0000","last_active":"2024-04-15T18:16:59.920656+0000","last_peered":"2024-04-15T18:16:59.920656+0000","last_clean":"2024-04-15T18:16:59.920656+0000","last_became_active":"2024-04-15T18:15:39.057839+0000","last_became_peered":"2024-04-15T18:15:39.057839+0000","last_unstale":"2024-04-15T18:16:59.920656+0000","last_undegraded":"2024-04-15T18:16:59.920656+0000","last_fullsized":"2024-04-15T18:16:59.920656+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'4867","last_scrub_stamp":"2024-04-15T18:16:45.807839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:16:45.807839+0000","objects_scrubbed":4816,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:46.626042+0000","scrub_duration":1774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":17055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":363840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"228'5534","reported_seq":5656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.356962+0000","last_change":"2024-04-15T18:16:44.596956+0000","last_active":"2024-04-15T18:17:00.356962+0000","last_peered":"2024-04-15T18:17:00.356962+0000","last_clean":"2024-04-15T18:17:00.356962+0000","last_became_active":"2024-04-15T18:15:39.058704+0000","last_became_peered":"2024-04-15T18:15:39.058704+0000","last_unstale":"2024-04-15T18:17:00.356962+0000","last_undegraded":"2024-04-15T18:17:00.356962+0000","last_fullsized":"2024-04-15T18:17:00.356962+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'4682","last_scrub_stamp":"2024-04-15T18:16:44.596835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:16:44.596835+0000","objects_scrubbed":4659,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:56.530426+0000","scrub_duration":842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362676224,"num_objects":5534,"num_object_clones":0,"num_object_copies":16602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5534,"num_write_kb":354176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"228'5634","reported_seq":5756,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.640532+0000","last_change":"2024-04-15T18:16:41.805148+0000","last_active":"2024-04-15T18:16:59.640532+0000","last_peered":"2024-04-15T18:16:59.640532+0000","last_clean":"2024-04-15T18:16:59.640532+0000","last_became_active":"2024-04-15T18:15:39.058285+0000","last_became_peered":"2024-04-15T18:15:39.058285+0000","last_unstale":"2024-04-15T18:16:59.640532+0000","last_undegraded":"2024-04-15T18:16:59.640532+0000","last_fullsized":"2024-04-15T18:16:59.640532+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'4614","last_scrub_stamp":"2024-04-15T18:16:41.804937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:16:41.804937+0000","objects_scrubbed":4590,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:55.721948+0000","scrub_duration":928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369229824,"num_objects":5634,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5634,"num_write_kb":360576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"228'5563","reported_seq":5631,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.623341+0000","last_change":"2024-04-15T18:15:39.062153+0000","last_active":"2024-04-15T18:16:59.623341+0000","last_peered":"2024-04-15T18:16:59.623341+0000","last_clean":"2024-04-15T18:16:59.623341+0000","last_became_active":"2024-04-15T18:15:39.061741+0000","last_became_peered":"2024-04-15T18:15:39.061741+0000","last_unstale":"2024-04-15T18:16:59.623341+0000","last_undegraded":"2024-04-15T18:16:59.623341+0000","last_fullsized":"2024-04-15T18:16:59.623341+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:03.899612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":356032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"228'5683","reported_seq":5749,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.349761+0000","last_change":"2024-04-15T18:15:39.058580+0000","last_active":"2024-04-15T18:17:00.349761+0000","last_peered":"2024-04-15T18:17:00.349761+0000","last_clean":"2024-04-15T18:17:00.349761+0000","last_became_active":"2024-04-15T18:15:39.058216+0000","last_became_peered":"2024-04-15T18:15:39.058216+0000","last_unstale":"2024-04-15T18:17:00.349761+0000","last_undegraded":"2024-04-15T18:17:00.349761+0000","last_fullsized":"2024-04-15T18:17:00.349761+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:05.297329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372441088,"num_objects":5683,"num_object_clones":0,"num_object_copies":17049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":363712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"228'5670","reported_seq":5738,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.380056+0000","last_change":"2024-04-15T18:15:39.670253+0000","last_active":"2024-04-15T18:16:59.380056+0000","last_peered":"2024-04-15T18:16:59.380056+0000","last_clean":"2024-04-15T18:16:59.380056+0000","last_became_active":"2024-04-15T18:15:39.669933+0000","last_became_peered":"2024-04-15T18:15:39.669933+0000","last_unstale":"2024-04-15T18:16:59.380056+0000","last_undegraded":"2024-04-15T18:16:59.380056+0000","last_fullsized":"2024-04-15T18:16:59.380056+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371589120,"num_objects":5670,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":362880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"228'5746","reported_seq":5869,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:01.890005+0000","last_change":"2024-04-15T18:16:42.000134+0000","last_active":"2024-04-15T18:17:01.890005+0000","last_peered":"2024-04-15T18:17:01.890005+0000","last_clean":"2024-04-15T18:17:01.890005+0000","last_became_active":"2024-04-15T18:15:39.057159+0000","last_became_peered":"2024-04-15T18:15:39.057159+0000","last_unstale":"2024-04-15T18:17:01.890005+0000","last_undegraded":"2024-04-15T18:17:01.890005+0000","last_fullsized":"2024-04-15T18:17:01.890005+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'4537","last_scrub_stamp":"2024-04-15T18:16:41.999966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:16:41.999966+0000","objects_scrubbed":4513,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:44.632689+0000","scrub_duration":957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"228'5791","reported_seq":5859,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:02.665665+0000","last_change":"2024-04-15T18:15:39.060876+0000","last_active":"2024-04-15T18:17:02.665665+0000","last_peered":"2024-04-15T18:17:02.665665+0000","last_clean":"2024-04-15T18:17:02.665665+0000","last_became_active":"2024-04-15T18:15:39.060320+0000","last_became_peered":"2024-04-15T18:15:39.060320+0000","last_unstale":"2024-04-15T18:17:02.665665+0000","last_undegraded":"2024-04-15T18:17:02.665665+0000","last_fullsized":"2024-04-15T18:17:02.665665+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":17373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5791,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"228'5705","reported_seq":5772,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:00.384803+0000","last_change":"2024-04-15T18:15:39.435414+0000","last_active":"2024-04-15T18:17:00.384803+0000","last_peered":"2024-04-15T18:17:00.384803+0000","last_clean":"2024-04-15T18:17:00.384803+0000","last_became_active":"2024-04-15T18:15:39.435165+0000","last_became_peered":"2024-04-15T18:15:39.435165+0000","last_unstale":"2024-04-15T18:17:00.384803+0000","last_undegraded":"2024-04-15T18:17:00.384803+0000","last_fullsized":"2024-04-15T18:17:00.384803+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:06.373700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373882880,"num_objects":5705,"num_object_clones":0,"num_object_copies":17115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5705,"num_write_kb":365120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"228'5678","reported_seq":5746,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.636450+0000","last_change":"2024-04-15T18:15:39.061919+0000","last_active":"2024-04-15T18:16:59.636450+0000","last_peered":"2024-04-15T18:16:59.636450+0000","last_clean":"2024-04-15T18:16:59.636450+0000","last_became_active":"2024-04-15T18:15:39.061499+0000","last_became_peered":"2024-04-15T18:15:39.061499+0000","last_unstale":"2024-04-15T18:16:59.636450+0000","last_undegraded":"2024-04-15T18:16:59.636450+0000","last_fullsized":"2024-04-15T18:16:59.636450+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:05.557790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372113408,"num_objects":5678,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5678,"num_write_kb":363392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"228'5776","reported_seq":5843,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:17:02.312557+0000","last_change":"2024-04-15T18:15:39.059346+0000","last_active":"2024-04-15T18:17:02.312557+0000","last_peered":"2024-04-15T18:17:02.312557+0000","last_clean":"2024-04-15T18:17:02.312557+0000","last_became_active":"2024-04-15T18:15:39.059050+0000","last_became_peered":"2024-04-15T18:15:39.059050+0000","last_unstale":"2024-04-15T18:17:02.312557+0000","last_undegraded":"2024-04-15T18:17:02.312557+0000","last_fullsized":"2024-04-15T18:17:02.312557+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:15:38.044723+0000","last_clean_scrub_stamp":"2024-04-15T18:15:38.044723+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:04.541135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378535936,"num_objects":5776,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5776,"num_write_kb":369664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":650,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-15T18:16:59.304039+0000","last_change":"2024-04-15T18:16:39.236420+0000","last_active":"2024-04-15T18:16:59.304039+0000","last_peered":"2024-04-15T18:16:59.304039+0000","last_clean":"2024-04-15T18:16:59.304039+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:16:59.304039+0000","last_undegraded":"2024-04-15T18:16:59.304039+0000","last_fullsized":"2024-04-15T18:16:59.304039+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:16:39.236285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:16:39.236285+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:17:42.432104+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5930614784,"num_objects":90494,"num_object_clones":0,"num_object_copies":271482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90494,"num_write_kb":5791616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17892376576,"data_stored":17892376576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92708,"ondisk_log_size":92708,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925917,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2961312,"kb_used_data":2839884,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":90783840,"statfs":{"total":95995035648,"available":92962652160,"internally_reserved":0,"allocated":2908041216,"data_stored":2907764901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":124320196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798168,"kb_used_data":735056,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92946984,"statfs":{"total":95995035648,"available":95177711616,"internally_reserved":0,"allocated":752697344,"data_stored":752420005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514600,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239832,"kb_used_data":1100368,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92505320,"statfs":{"total":95995035648,"available":94725447680,"internally_reserved":0,"allocated":1126776832,"data_stored":1126499493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799968,"kb_used_data":710480,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92945184,"statfs":{"total":95995035648,"available":95175868416,"internally_reserved":0,"allocated":727531520,"data_stored":727254181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168689,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1544292,"kb_used_data":1427032,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92200860,"statfs":{"total":95995035648,"available":94413680640,"internally_reserved":0,"allocated":1461280768,"data_stored":1460999877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463030,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2225120,"kb_used_data":2143948,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91520032,"statfs":{"total":95995035648,"available":93716512768,"internally_reserved":0,"allocated":2195402752,"data_stored":2195126457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83098054},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601180,"kb_used_data":1471440,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92143972,"statfs":{"total":95995035648,"available":94355427328,"internally_reserved":0,"allocated":1506754560,"data_stored":1506477221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2294108,"kb_used_data":2172688,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91451044,"statfs":{"total":95995035648,"available":93645869056,"internally_reserved":0,"allocated":2224832512,"data_stored":2224555173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411469,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461848,"kb_used_data":376528,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93283304,"statfs":{"total":95995035648,"available":95522103296,"internally_reserved":0,"allocated":385564672,"data_stored":385287333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":970662608898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3115096,"kb_used_data":2967312,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90630056,"statfs":{"total":95995035648,"available":92805177344,"internally_reserved":0,"allocated":3038527488,"data_stored":3038250149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724666,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499808,"kb_used_data":1424208,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92245344,"statfs":{"total":95995035648,"available":94459232256,"internally_reserved":0,"allocated":1458388992,"data_stored":1458111653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2227109888,"data_stored":2227109888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1516896256,"data_stored":1516896256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385417216,"data_stored":385417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3054108672,"data_stored":3054108672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1457913856,"data_stored":1457913856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2216558592,"data_stored":2216558592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1475346432,"data_stored":1475346432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":731054080,"data_stored":731054080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1134952448,"data_stored":1134952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":759365632,"data_stored":759365632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2933653504,"data_stored":2933653504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:17:04.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-04-15T18:17:05.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:05.106 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:05.139 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:05.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:05.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:05.274 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:05.474 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:05.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:05.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:05.578 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:05.625 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:05.634 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:05.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:05.658 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:17:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:17:05.658 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:17:05.658 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:05.669 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:05.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:05.776 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:05.811 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:05.894 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:05.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:05.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 95935 95919 76.8517 56 0.00896453 0.0130075 2024-04-15T18:17:05.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:06.020 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:06.046 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:06.074 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:06.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:06.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:06.208 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:06.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:06.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:06.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:17:06.432 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:06.466 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:06.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:06.553 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:06 smithi049 ceph-mon[40087]: pgmap v636: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-15T18:17:06.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:06.615 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:06.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:06 smithi159 ceph-mon[45336]: pgmap v636: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-15T18:17:06.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:17:06.641 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:06.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:06.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:06.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:06 smithi082 ceph-mon[44206]: pgmap v636: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 57 MiB/s wr, 911 op/s 2024-04-15T18:17:06.794 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:06.804 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:06.908 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:06.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:06.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 15 97009 96994 76.7293 67.1875 0.0336413 0.0130282 2024-04-15T18:17:07.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:07.046 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:07.077 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:07.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:07.182 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:07.215 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:07.238 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:07.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:07.338 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:17:07.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:07.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:07.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:07.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:07.728 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:07.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:07.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:07.873 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:07.903 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:07.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:07.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:17:07.945128+0000 min lat: 0.00112542 max lat: 0.279966 avg lat: 0.0130596 2024-04-15T18:17:07.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:17:07.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 97999 97983 76.5428 61.8125 0.00777687 0.0130596 2024-04-15T18:17:08.009 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:08.052 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:08.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:08.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:08.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:08.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:08 smithi049 ceph-mon[40087]: pgmap v637: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-04-15T18:17:08.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:08 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:08.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:08.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:08 smithi049 ceph-mon[40087]: osdmap e229: 12 total, 12 up, 11 in 2024-04-15T18:17:08.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:08 smithi049 ceph-mon[40087]: 3.e scrub starts 2024-04-15T18:17:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:08.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:08 smithi159 ceph-mon[45336]: pgmap v637: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-04-15T18:17:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:08 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:08.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:08 smithi159 ceph-mon[45336]: osdmap e229: 12 total, 12 up, 11 in 2024-04-15T18:17:08.409 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:08 smithi159 ceph-mon[45336]: 3.e scrub starts 2024-04-15T18:17:08.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:08.556 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:08.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:08.594 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:08 smithi082 ceph-mon[44206]: pgmap v637: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 992 op/s 2024-04-15T18:17:08.594 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:08 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:08.594 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3092276747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:08.594 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:08 smithi082 ceph-mon[44206]: osdmap e229: 12 total, 12 up, 11 in 2024-04-15T18:17:08.594 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:08 smithi082 ceph-mon[44206]: 3.e scrub starts 2024-04-15T18:17:08.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:08.731 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:08.758 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:08.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:08.835 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:08.835 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:17:08.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33618 2.5 2:a7d6052a:::benchmark_data_smithi159_80260_object33617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.23308544299999, 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21175838599999999, 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.836 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33618 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596818+0000", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596842+0000", 2024-04-15T18:17:08.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3603000000000001e-05 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596852+0000", 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0225e-05 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596857+0000", 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7039999999999999e-06 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596906+0000", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8968999999999998e-05 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596985+0000", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8145999999999993e-05 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.597072+0000", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7070999999999994e-05 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.598140+0000", 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010683349999999999 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.599600+0000", 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00145977 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808504+0000", 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20890451900000001 2024-04-15T18:17:08.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808542+0000", 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7718e-05 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808575+0000", 2024-04-15T18:17:08.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3439999999999998e-05 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33625 2.5 2:a2b92d8f:::benchmark_data_smithi159_80260_object33624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.22803445599999, 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206864881, 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.843 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33625 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601869+0000", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4e-07 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601889+0000", 2024-04-15T18:17:08.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9627999999999999e-05 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601899+0000", 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0721e-05 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601905+0000", 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0580000000000003e-06 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601955+0000", 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9453000000000002e-05 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602050+0000", 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4864999999999994e-05 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602141+0000", 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1125999999999995e-05 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603155+0000", 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010146059999999999 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.604470+0000", 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013143440000000001 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808630+0000", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20416066499999999 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808689+0000", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8570999999999999e-05 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808733+0000", 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3903999999999998e-05 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.849 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33628 2.f 2:f7e4c22b:::benchmark_data_smithi159_80260_object33627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.227047685, 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206082972, 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.850 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33628 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602856+0000", 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7800000000000001e-07 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602877+0000", 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1021000000000001e-05 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602890+0000", 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3206e-05 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602898+0000", 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7230000000000004e-06 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602951+0000", 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3508000000000003e-05 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603046+0000", 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4559e-05 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:17:08.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603185+0000", 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000139169 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.604347+0000", 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011623709999999999 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613605+0000", 2024-04-15T18:17:08.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092581290000000004 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808843+0000", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19523797700000001 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808898+0000", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5179999999999997e-05 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808938+0000", 2024-04-15T18:17:08.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9351000000000003e-05 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33641 2.5 2:a8326c7a:::benchmark_data_smithi159_80260_object33640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.21848492800001, 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19743492300000001, 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.857 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33641 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2900000000000003e-07 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611435+0000", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6427e-05 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611444+0000", 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5840000000000001e-06 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611450+0000", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5450000000000003e-06 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611497+0000", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7401999999999998e-05 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611581+0000", 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4124999999999998e-05 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611682+0000", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100686 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612756+0000", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001073861 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.615264+0000", 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025077150000000002 2024-04-15T18:17:08.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808777+0000", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19351336399999999 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808825+0000", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8340000000000001e-05 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808853+0000", 2024-04-15T18:17:08.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7245000000000001e-05 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33643 2.f 2:f6b9a2f3:::benchmark_data_smithi159_80260_object33642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.21834566199999, 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.197552964, 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.863 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33643 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611558+0000", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.018e-06 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611580+0000", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2050999999999999e-05 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611593+0000", 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2695e-05 2024-04-15T18:17:08.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611601+0000", 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9659999999999996e-06 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611655+0000", 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4150000000000002e-05 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611758+0000", 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103466 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611865+0000", 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106616 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613582+0000", 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017170480000000001 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613869+0000", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000287597 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808999+0000", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.1951292 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809054+0000", 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5578999999999998e-05 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809110+0000", 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5578000000000002e-05 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33644 2.f 2:f5bf3720:::benchmark_data_smithi159_80260_object33643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.21774627500002, 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19708304400000001, 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.869 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33644 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612157+0000", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8700000000000005e-07 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612174+0000", 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7422e-05 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612184+0000", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2019999999999993e-06 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612189+0000", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3890000000000004e-06 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612752+0000", 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056267000000000003 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612824+0000", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2075000000000004e-05 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612956+0000", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013199899999999999 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613697+0000", 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074157499999999996 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.635785+0000", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022088013 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809156+0000", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17337069299999999 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809203+0000", 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6782e-05 2024-04-15T18:17:08.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809239+0000", 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6536999999999999e-05 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70703 2.5 2:a863a10a:::benchmark_data_smithi159_80260_object70702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 250.77659982099999, 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21353573100000001, 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.875 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70703 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053304+0000", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3900000000000003e-07 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053321+0000", 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7367000000000002e-05 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053330+0000", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3910000000000004e-06 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053336+0000", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6339999999999998e-06 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053393+0000", 2024-04-15T18:17:08.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6716999999999999e-05 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053508+0000", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114868 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053600+0000", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2114999999999995e-05 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056695+0000", 2024-04-15T18:17:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003095155 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056727+0000", 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1698e-05 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266759+0000", 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21003202100000001 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266803+0000", 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.405e-05 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266838+0000", 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5775999999999998e-05 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70705 2.5 2:adfd5fa3:::benchmark_data_smithi159_80260_object70704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:17:08.881 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 250.77526261599999, 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21421320799999999, 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70705 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:17:08.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054641+0000", 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9299999999999996e-07 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054657+0000", 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5908000000000001e-05 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054666+0000", 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0550000000000005e-06 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054671+0000", 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3460000000000001e-06 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055114+0000", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044297 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055238+0000", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000124256 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055390+0000", 2024-04-15T18:17:08.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015158000000000001 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056820+0000", 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001430539 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056957+0000", 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000136766 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268718+0000", 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21176034299999999 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268806+0000", 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8208999999999995e-05 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268853+0000", 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7342999999999997e-05 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70708 2.5 2:acd4c471:::benchmark_data_smithi159_80260_object70707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 250.77079486100001, 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209906643, 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70708 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3399999999999998e-07 2024-04-15T18:17:08.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059125+0000", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6895999999999999e-05 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059134+0000", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9639999999999992e-06 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059140+0000", 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2639999999999999e-06 2024-04-15T18:17:08.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059178+0000", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.824e-05 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059277+0000", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8788000000000004e-05 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059390+0000", 2024-04-15T18:17:08.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011296800000000001 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.061117+0000", 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017270600000000001 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.061151+0000", 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4467000000000001e-05 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268913+0000", 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20776150500000001 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268971+0000", 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8091000000000003e-05 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.269014+0000", 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3566000000000003e-05 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.893 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97473 2.5 2:a924ac74:::benchmark_data_smithi159_80260_object8360:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 222.71892444299999, 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25442546900000002, 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97473 2024-04-15T18:17:08.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110979+0000", 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8400000000000003e-07 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110983+0000", 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9589999999999997e-06 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110993+0000", 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9019999999999994e-06 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110998+0000", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5350000000000004e-06 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111039+0000", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0444000000000001e-05 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111132+0000", 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3224000000000001e-05 2024-04-15T18:17:08.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111217+0000", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5006999999999994e-05 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115286+0000", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040691770000000002 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115341+0000", 2024-04-15T18:17:08.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4459999999999997e-05 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365290+0000", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.249949328 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365351+0000", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0683000000000002e-05 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365404+0000", 2024-04-15T18:17:08.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2865999999999998e-05 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97475 2.5 2:ad8d4387:::benchmark_data_smithi159_80260_object8362:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 222.718764894, 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25436893199999999, 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.900 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97475 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6799999999999999e-07 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111143+0000", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8960000000000002e-06 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111153+0000", 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1940000000000004e-06 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111160+0000", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.131e-06 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.112894+0000", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017338830000000001 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.112968+0000", 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4627000000000002e-05 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.113052+0000", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3727000000000007e-05 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115400+0000", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023476539999999998 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.116209+0000", 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00080913800000000004 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365435+0000", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24922602899999999 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365469+0000", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4177000000000002e-05 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365507+0000", 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7608000000000003e-05 2024-04-15T18:17:08.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97538 2.5 2:aaf52e94:::benchmark_data_smithi159_80260_object8425:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 222.653902843, 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.245409091, 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.906 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97538 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176001+0000", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.009e-06 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176006+0000", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176015+0000", 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7340000000000003e-06 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176022+0000", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8299999999999998e-06 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176079+0000", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6844e-05 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176206+0000", 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012738800000000001 2024-04-15T18:17:08.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176297+0000", 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0346000000000005e-05 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.178508+0000", 2024-04-15T18:17:08.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022112130000000001 2024-04-15T18:17:08.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.178718+0000", 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00020958500000000001 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421343+0000", 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.242625327 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421379+0000", 2024-04-15T18:17:08.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6176000000000001e-05 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421409+0000", 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9726999999999998e-05 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97786 2.b 2:d8f085c3:::benchmark_data_smithi159_80260_object8673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.646852+0000", 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 222.18305079999999, 2024-04-15T18:17:08.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19296175900000001, 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97786 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646852+0000", 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646852+0000", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646852+0000", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6000000000000003e-07 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646856+0000", 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5310000000000002e-06 2024-04-15T18:17:08.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646866+0000", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6039999999999999e-06 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646872+0000", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8979999999999999e-06 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.646925+0000", 2024-04-15T18:17:08.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2930999999999999e-05 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.647046+0000", 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120987 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.647166+0000", 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011917999999999999 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.649433+0000", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022676229999999999 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.650165+0000", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00073174500000000001 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.839736+0000", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18957107200000001 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.839772+0000", 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5935000000000001e-05 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.839813+0000", 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1593e-05 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "rep_scrubmap(2.5 e114 from shard 0)", 2024-04-15T18:17:08.922 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.659705+0000", 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.170197925, 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19131925299999999, 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.659705+0000", 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.659705+0000", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.659705+0000", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.659716+0000", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1308000000000001e-05 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.850648+0000", 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19093178599999999 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.850653+0000", 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0170000000000004e-06 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.850905+0000", 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025202299999999998 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.850921+0000", 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5781e-05 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.851024+0000", 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102747 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.926 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:157471 2.5 2:a0abce61:::benchmark_data_smithi159_80260_object68358:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.169047318, 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.297938853, 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157471 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660856+0000", 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.274e-06 2024-04-15T18:17:08.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660862+0000", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0260000000000004e-06 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660876+0000", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3675999999999999e-05 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660885+0000", 2024-04-15T18:17:08.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6049999999999998e-06 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664182+0000", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003297407 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664311+0000", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012873600000000001 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664466+0000", 2024-04-15T18:17:08.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015463299999999999 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672212+0000", 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0077465149999999998 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.688747+0000", 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016535421000000002 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958676+0000", 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26992857199999998 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958733+0000", 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6824999999999999e-05 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958794+0000", 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1162999999999997e-05 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:157472 2.5 2:a71b28ad:::benchmark_data_smithi159_80260_object68359:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.16421764399999, 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29329156200000001, 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157472 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665686+0000", 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6299999999999993e-07 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665691+0000", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9509999999999997e-06 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665702+0000", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0906e-05 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665708+0000", 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5769999999999999e-06 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668460+0000", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027518379999999999 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668574+0000", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113497 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668647+0000", 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3139000000000006e-05 2024-04-15T18:17:08.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672383+0000", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037358439999999999 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.695755+0000", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023372826999999999 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958854+0000", 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26309830699999998 2024-04-15T18:17:08.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958912+0000", 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8384000000000002e-05 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958976+0000", 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4329e-05 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24511.0:157471 2.5 e177/114 ondisk, result = 0)", 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.16165254500001, 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29055099000000001, 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668251+0000", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9299999999999997e-07 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668258+0000", 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1400000000000002e-06 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.859551+0000", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19129326699999999 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.859564+0000", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2393e-05 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958661+0000", 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.099096975000000004 2024-04-15T18:17:08.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958669+0000", 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0730000000000005e-06 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958801+0000", 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000132349 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24511.0:157472 2.5 e177/114 ondisk, result = 0)", 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.15783125499999, 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28708425799999998, 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672072+0000", 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7199999999999998e-07 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672080+0000", 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6590000000000007e-06 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.862774+0000", 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19069477500000001 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.862783+0000", 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2789999999999998e-06 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958838+0000", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.096054971000000003 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958846+0000", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8540000000000003e-06 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.959156+0000", 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00030904800000000003 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.946 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:7739 3.9 3:942aaec8:::benchmark_data_smithi159_91621_object7738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 76.442559063000004, 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20308042000000001, 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:08.947 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 7739 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387345+0000", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0690000000000001e-06 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387371+0000", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6942000000000001e-05 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387383+0000", 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1226000000000001e-05 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387388+0000", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7209999999999999e-06 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387433+0000", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4249000000000003e-05 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387519+0000", 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6410999999999996e-05 2024-04-15T18:17:08.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387652+0000", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000133034 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.388380+0000", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00072825199999999998 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.388510+0000", 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012965200000000001 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590327+0000", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20181707700000001 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590381+0000", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4212999999999997e-05 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590424+0000", 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2574000000000003e-05 2024-04-15T18:17:08.952 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:15742 3.9 3:98ecea81:::benchmark_data_smithi159_91621_object15741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.217868487999993, 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.210797182, 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:08.953 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15742 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612035+0000", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.299e-06 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612057+0000", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1297e-05 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612067+0000", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0159999999999999e-05 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:08.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612073+0000", 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8769999999999998e-06 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612115+0000", 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2574999999999998e-05 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612199+0000", 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4005999999999995e-05 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612290+0000", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0951000000000004e-05 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612955+0000", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00066523400000000001 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.613378+0000", 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042226999999999998 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822703+0000", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20932568600000001 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822768+0000", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4869000000000005e-05 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822831+0000", 2024-04-15T18:17:08.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2958000000000003e-05 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:08.959 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:08.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:08.960 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 98936 98920 76.3207 58.5625 0.007168 0.0130979 2024-04-15T18:17:08.991 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:08.992 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:09.018 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:09.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:09.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:09.160 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:09.356 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 3 pgp_num to 16 2024-04-15T18:17:09.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:09 smithi159 ceph-mon[45336]: 3.b scrub starts 2024-04-15T18:17:09.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:17:09.518 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:09.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:09.553 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:17:09.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:09.656 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:09 smithi082 ceph-mon[44206]: 3.b scrub starts 2024-04-15T18:17:09.656 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:17:09.659 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:09.688 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:09.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:09.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:09 smithi049 ceph-mon[40087]: 3.b scrub starts 2024-04-15T18:17:09.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:17:09.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:09.834 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:09.866 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:09.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:09.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 99840 99824 76.0789 56.5 0.00477131 0.0131384 2024-04-15T18:17:09.962 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:09.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:10.005 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:10.032 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:10.138 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:10.164 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:10.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:10.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:10.286 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:10.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:10.351 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:10.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:10.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:17:10.457 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:10.490 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:10 smithi159 ceph-mon[45336]: pgmap v639: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-15T18:17:10.490 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:17:10.490 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:10 smithi159 ceph-mon[45336]: 3.e scrub ok 2024-04-15T18:17:10.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:10 smithi159 ceph-mon[45336]: osdmap e230: 12 total, 12 up, 11 in 2024-04-15T18:17:10.498 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:10.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:10 smithi049 ceph-mon[40087]: pgmap v639: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-15T18:17:10.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:17:10.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:10 smithi049 ceph-mon[40087]: 3.e scrub ok 2024-04-15T18:17:10.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:10 smithi049 ceph-mon[40087]: osdmap e230: 12 total, 12 up, 11 in 2024-04-15T18:17:10.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:10 smithi082 ceph-mon[44206]: pgmap v639: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 60 MiB/s wr, 956 op/s 2024-04-15T18:17:10.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1036966246' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:17:10.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:10 smithi082 ceph-mon[44206]: 3.e scrub ok 2024-04-15T18:17:10.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:10 smithi082 ceph-mon[44206]: osdmap e230: 12 total, 12 up, 11 in 2024-04-15T18:17:10.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 100534 100518 75.6849 43.375 0.00355275 0.0131968 2024-04-15T18:17:11.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:11.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:11.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:11.407 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:11.444 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:11.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:11.549 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:11.606 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:11.631 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:11.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:11 smithi082 ceph-mon[44206]: 3.3 scrub starts 2024-04-15T18:17:11.633 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:11 smithi082 ceph-mon[44206]: 3.6 scrub starts 2024-04-15T18:17:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:11.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:11.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:11 smithi159 ceph-mon[45336]: 3.3 scrub starts 2024-04-15T18:17:11.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:11 smithi159 ceph-mon[45336]: 3.6 scrub starts 2024-04-15T18:17:11.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:11 smithi049 ceph-mon[40087]: 3.3 scrub starts 2024-04-15T18:17:11.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:11 smithi049 ceph-mon[40087]: 3.6 scrub starts 2024-04-15T18:17:11.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:11.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:11.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 101247 101231 75.3143 44.5625 0.0193594 0.0132721 2024-04-15T18:17:12.076 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:101318 3.2 3:411c664b:::benchmark_data_smithi159_91621_object101317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:12.066434+0000", 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0078478720000000005, 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078734619999999995, 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:12.077 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 101318 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066434+0000", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066434+0000", 2024-04-15T18:17:12.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066435+0000", 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.491e-06 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066490+0000", 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4843999999999999e-05 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:12.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066504+0000", 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.393e-05 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066514+0000", 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5999999999999996e-06 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066634+0000", 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120592 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066768+0000", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000133721 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.066923+0000", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000154928 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.069450+0000", 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025272789999999999 2024-04-15T18:17:12.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:12.073700+0000", 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042501400000000003 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 1 2024-04-15T18:17:12.082 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:12.363 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:12 smithi159 ceph-mon[45336]: pgmap v641: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:17:12.363 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:12 smithi159 ceph-mon[45336]: 3.1 scrub starts 2024-04-15T18:17:12.363 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:12 smithi159 ceph-mon[45336]: 3.f scrub starts 2024-04-15T18:17:12.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:12 smithi049 ceph-mon[40087]: pgmap v641: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:17:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:12 smithi049 ceph-mon[40087]: 3.1 scrub starts 2024-04-15T18:17:12.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:12 smithi049 ceph-mon[40087]: 3.f scrub starts 2024-04-15T18:17:12.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:12 smithi082 ceph-mon[44206]: pgmap v641: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:17:12.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:12 smithi082 ceph-mon[44206]: 3.1 scrub starts 2024-04-15T18:17:12.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:12 smithi082 ceph-mon[44206]: 3.f scrub starts 2024-04-15T18:17:12.853 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:17:12.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 102002 101986 74.9834 47.1875 0.0141633 0.0133311 2024-04-15T18:17:13.237 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:13 smithi159 ceph-mon[45336]: 3.3 scrub ok 2024-04-15T18:17:13.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:13 smithi159 ceph-mon[45336]: 3.6 scrub ok 2024-04-15T18:17:13.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:13 smithi049 ceph-mon[40087]: 3.3 scrub ok 2024-04-15T18:17:13.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:13 smithi049 ceph-mon[40087]: 3.6 scrub ok 2024-04-15T18:17:13.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:13 smithi082 ceph-mon[44206]: 3.3 scrub ok 2024-04-15T18:17:13.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:13 smithi082 ceph-mon[44206]: 3.6 scrub ok 2024-04-15T18:17:13.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 102931 102915 74.7866 58.0625 0.0100211 0.0133659 2024-04-15T18:17:14.129 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:14.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:14.234 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:14.283 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:14.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:14 smithi049 ceph-mon[40087]: 3.1 scrub ok 2024-04-15T18:17:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:14 smithi049 ceph-mon[40087]: pgmap v642: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-04-15T18:17:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:14 smithi049 ceph-mon[40087]: 3.5 scrub starts 2024-04-15T18:17:14.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:14 smithi049 ceph-mon[40087]: 3.f scrub ok 2024-04-15T18:17:14.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:14 smithi159 ceph-mon[45336]: 3.1 scrub ok 2024-04-15T18:17:14.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:14 smithi159 ceph-mon[45336]: pgmap v642: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-04-15T18:17:14.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:14 smithi159 ceph-mon[45336]: 3.5 scrub starts 2024-04-15T18:17:14.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:14 smithi159 ceph-mon[45336]: 3.f scrub ok 2024-04-15T18:17:14.739 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:14.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:14.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:14 smithi082 ceph-mon[44206]: 3.1 scrub ok 2024-04-15T18:17:14.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:14 smithi082 ceph-mon[44206]: pgmap v642: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 59 MiB/s wr, 936 op/s 2024-04-15T18:17:14.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:14 smithi082 ceph-mon[44206]: 3.5 scrub starts 2024-04-15T18:17:14.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:14 smithi082 ceph-mon[44206]: 3.f scrub ok 2024-04-15T18:17:14.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:14.879 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:14.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 103793 103777 74.5461 53.875 0.0102656 0.0134091 2024-04-15T18:17:15.232 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:15.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:15.281 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:15.303 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:15.303 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:17:15.303 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:17:15.303 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:17:15.303 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:15.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:15.388 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:15.423 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:15.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:15.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:15.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:15.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: 3.2 scrub starts 2024-04-15T18:17:15.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: osdmap e231: 12 total, 12 up, 11 in 2024-04-15T18:17:15.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: 3.0 scrub starts 2024-04-15T18:17:15.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:15 smithi049 ceph-mon[40087]: 3.5 scrub ok 2024-04-15T18:17:15.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:15.576 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:15.602 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:15.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: 3.2 scrub starts 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: osdmap e231: 12 total, 12 up, 11 in 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: 3.0 scrub starts 2024-04-15T18:17:15.662 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:15 smithi159 ceph-mon[45336]: 3.5 scrub ok 2024-04-15T18:17:15.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:17:15.740 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: 3.2 scrub starts 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: osdmap e231: 12 total, 12 up, 11 in 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: 3.0 scrub starts 2024-04-15T18:17:15.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:15 smithi082 ceph-mon[44206]: 3.5 scrub ok 2024-04-15T18:17:15.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 104704 104688 74.346 56.9375 0.0327587 0.0134458 2024-04-15T18:17:16.075 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:17:16.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:16 smithi159 ceph-mon[45336]: pgmap v644: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:17:16.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:16 smithi159 ceph-mon[45336]: 3.c scrub starts 2024-04-15T18:17:16.462 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:16.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:16 smithi049 ceph-mon[40087]: pgmap v644: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:17:16.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:16 smithi049 ceph-mon[40087]: 3.c scrub starts 2024-04-15T18:17:16.494 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:17:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:16.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:17:16.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:16.633 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:16.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:16 smithi082 ceph-mon[44206]: pgmap v644: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:17:16.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:16 smithi082 ceph-mon[44206]: 3.c scrub starts 2024-04-15T18:17:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:16.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:16.896 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:16.931 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:16.947 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 15 105610 105595 74.1475 56.6875 0.0202217 0.0134811 2024-04-15T18:17:17.612 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:17.639 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:17 smithi159 ceph-mon[45336]: 3.2 scrub ok 2024-04-15T18:17:17.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:17 smithi159 ceph-mon[45336]: 3.c scrub ok 2024-04-15T18:17:17.668 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:17.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:17.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:17 smithi049 ceph-mon[40087]: 3.2 scrub ok 2024-04-15T18:17:17.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:17 smithi049 ceph-mon[40087]: 3.c scrub ok 2024-04-15T18:17:17.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:17.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:17.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:17 smithi082 ceph-mon[44206]: 3.2 scrub ok 2024-04-15T18:17:17.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:17 smithi082 ceph-mon[44206]: 3.c scrub ok 2024-04-15T18:17:17.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:17.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:17.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:17.886 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:17.918 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:17.947 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:17.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 9 106473 106464 73.9271 54.3125 0.0181197 0.0135215 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:18.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: 3.7 scrub starts 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: 3.0 scrub ok 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: pgmap v645: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/375679557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: 3.7 scrub starts 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: 3.0 scrub ok 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: pgmap v645: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/375679557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: 3.7 scrub starts 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: 3.0 scrub ok 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: pgmap v645: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/375679557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:18.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.9473 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 106473 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 73.1694 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 19.9314 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 131 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 43.375 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 1170 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 318.903 2024-04-15T18:17:18.948 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 2096 2024-04-15T18:17:18.949 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 694 2024-04-15T18:17:18.949 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0135566 2024-04-15T18:17:18.949 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.0172183 2024-04-15T18:17:18.949 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 0.954701 2024-04-15T18:17:18.949 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00112542 2024-04-15T18:17:18.954 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:17:19.080 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:17:19.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:19.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:19 smithi159 ceph-mon[45336]: osdmap e232: 12 total, 12 up, 11 in 2024-04-15T18:17:19.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:19.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:19 smithi049 ceph-mon[40087]: osdmap e232: 12 total, 12 up, 11 in 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:19.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:17:19.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:17:19.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2022287320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:19.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:19 smithi082 ceph-mon[44206]: osdmap e232: 12 total, 12 up, 11 in 2024-04-15T18:17:20.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:20 smithi159 ceph-mon[45336]: 3.7 scrub ok 2024-04-15T18:17:20.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:20 smithi159 ceph-mon[45336]: pgmap v647: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-04-15T18:17:20.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:20 smithi049 ceph-mon[40087]: 3.7 scrub ok 2024-04-15T18:17:20.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:20 smithi049 ceph-mon[40087]: pgmap v647: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-04-15T18:17:20.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:20 smithi082 ceph-mon[44206]: 3.7 scrub ok 2024-04-15T18:17:20.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:20 smithi082 ceph-mon[44206]: pgmap v647: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 38 MiB/s wr, 608 op/s 2024-04-15T18:17:20.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:17:21.136 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:21.231 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:17:21.231 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:17:21.232 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-15T18:17:21.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd in 3 2024-04-15T18:17:21.691 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:21.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:21.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:17:21.797 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:21.831 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:22.515 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:22.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:22 smithi049 ceph-mon[40087]: pgmap v648: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 884 op/s 2024-04-15T18:17:22.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:22.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:22.734 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:22 smithi159 ceph-mon[45336]: pgmap v648: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 884 op/s 2024-04-15T18:17:22.734 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:22.734 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:22.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:22 smithi082 ceph-mon[44206]: pgmap v648: 17 pgs: 17 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 48 MiB/s wr, 884 op/s 2024-04-15T18:17:22.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:22.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:23.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:23 smithi049 ceph-mon[40087]: osdmap e233: 12 total, 12 up, 11 in 2024-04-15T18:17:23.761 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:23.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:23.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:23 smithi082 ceph-mon[44206]: osdmap e233: 12 total, 12 up, 11 in 2024-04-15T18:17:23.800 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:23.830 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:23.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:23.886 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:23.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:23 smithi159 ceph-mon[45336]: osdmap e233: 12 total, 12 up, 11 in 2024-04-15T18:17:23.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:23.946 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:23.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:24.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:24.032 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:24.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:24.095 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:24.096 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:24.097 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:24.193 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:24.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:24.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:24.330 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:24.363 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:24.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:24.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:24.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: pgmap v650: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 35 MiB/s wr, 752 op/s 2024-04-15T18:17:24.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/90583122' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: pgmap v650: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 35 MiB/s wr, 752 op/s 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/90583122' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: pgmap v650: 17 pgs: 17 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 35 MiB/s wr, 752 op/s 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/90583122' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:24.821 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:17:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:17:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:4206 2.d 2:b7208875:::benchmark_data_smithi159_80260_object4205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 335.17787332699999, 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.274696569, 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4206 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638348+0000", 2024-04-15T18:17:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638371+0000", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3159e-05 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638381+0000", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9860000000000006e-06 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638387+0000", 2024-04-15T18:17:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3789999999999997e-06 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638432+0000", 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5135000000000001e-05 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638507+0000", 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4697000000000007e-05 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638591+0000", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4327999999999996e-05 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638937+0000", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034585099999999999 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.639436+0000", 2024-04-15T18:17:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049941999999999999 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.912941+0000", 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27350442800000002 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.912997+0000", 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6215000000000003e-05 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.913044+0000", 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7156e-05 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:6437 2.e e124/114 2:78458e41:::benchmark_data_smithi159_80260_object6436:head v 124'403, mlcod=124'399)", 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:51.504400+0000", 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 333.31182040200002, 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19738763400000001, 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.504400+0000", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.504400+0000", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.504401+0000", 2024-04-15T18:17:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2600000000000002e-07 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.504467+0000", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6233000000000001e-05 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701303+0000", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19683639999999999 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701307+0000", 2024-04-15T18:17:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0810000000000004e-06 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701354+0000", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6396000000000001e-05 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701377+0000", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2697999999999999e-05 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701759+0000", 2024-04-15T18:17:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00038217600000000002 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:51.701788+0000", 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9124000000000001e-05 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62307 2.d 2:bf30746f:::benchmark_data_smithi159_80260_object62306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 289.145574737, 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.226374886, 2024-04-15T18:17:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62307 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670647+0000", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0410000000000001e-06 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670664+0000", 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6765e-05 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670675+0000", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0801999999999999e-05 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670682+0000", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2230000000000001e-06 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670742+0000", 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9865999999999999e-05 2024-04-15T18:17:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670865+0000", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000123356 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.671000+0000", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000135389 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.672916+0000", 2024-04-15T18:17:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001916077 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673626+0000", 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000709813 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.896888+0000", 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22326158300000001 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.896964+0000", 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5976999999999994e-05 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.897021+0000", 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62312 2.d 2:b08159eb:::benchmark_data_smithi159_80260_object62311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:17:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 289.14271694899998, 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22816421200000001, 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62312 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673505+0000", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.66e-06 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673527+0000", 2024-04-15T18:17:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1418000000000001e-05 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673537+0000", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0336999999999999e-05 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673545+0000", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0330000000000008e-06 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673654+0000", 2024-04-15T18:17:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109175 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673778+0000", 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001237 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673897+0000", 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118857 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.675533+0000", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00163652 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.676184+0000", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00065051900000000001 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901586+0000", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22540185099999999 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901621+0000", 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4997999999999999e-05 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901668+0000", 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7144000000000001e-05 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62467 2.d 2:b6055ada:::benchmark_data_smithi159_80260_object62466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 288.89228421199999, 2024-04-15T18:17:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36534789600000001, 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62467 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923938+0000", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.238e-06 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923957+0000", 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9143999999999998e-05 2024-04-15T18:17:24.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923969+0000", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1749000000000001e-05 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923975+0000", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7279999999999998e-06 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924029+0000", 2024-04-15T18:17:24.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3616000000000003e-05 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924148+0000", 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118961 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924248+0000", 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100073 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.926022+0000", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017739539999999999 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.926202+0000", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000180416 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289203+0000", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36300033999999998 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289241+0000", 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7866e-05 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289284+0000", 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3810999999999998e-05 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:127639 2.a e169/114 2:50636722:::benchmark_data_smithi159_80260_object38526:head v 169'8108, mlcod=169'8102)", 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.139628+0000", 2024-04-15T18:17:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 206.67659283699999, 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209027989, 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139628+0000", 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139628+0000", 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139629+0000", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3500000000000005e-07 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139635+0000", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2899999999999999e-06 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.345713+0000", 2024-04-15T18:17:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206078028 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.345718+0000", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3909999999999997e-06 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.345772+0000", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4107000000000002e-05 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.345789+0000", 2024-04-15T18:17:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6639000000000001e-05 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.348619+0000", 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028298210000000002 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.348656+0000", 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6878000000000002e-05 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:127653 2.a e169/114 2:53274d6c:::benchmark_data_smithi159_80260_object38540:head v 169'8109, mlcod=169'8103)", 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.162159+0000", 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 206.654061971, 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21014295099999999, 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162159+0000", 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162159+0000", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162159+0000", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3200000000000004e-07 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162166+0000", 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8109999999999997e-06 2024-04-15T18:17:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.368652+0000", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206485683 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.368658+0000", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.46e-06 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.368712+0000", 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3470999999999997e-05 2024-04-15T18:17:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.368732+0000", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0072e-05 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372258+0000", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035265750000000001 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372302+0000", 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3146999999999999e-05 2024-04-15T18:17:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:153095 2.d 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e176)", 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 173.66653062200001, 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25579912100000002, 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 153095 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149692+0000", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3999999999999999e-06 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149703+0000", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0703000000000001e-05 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149731+0000", 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8107e-05 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149747+0000", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6135999999999999e-05 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149885+0000", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013756900000000001 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149982+0000", 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6799000000000006e-05 2024-04-15T18:17:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.150060+0000", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7757999999999997e-05 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.153583+0000", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035231580000000002 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.154591+0000", 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010079780000000001 2024-04-15T18:17:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405390+0000", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25079957800000002 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405442+0000", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2023e-05 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405489+0000", 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6912000000000001e-05 2024-04-15T18:17:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:156890 2.d 2:b8159b5c:::benchmark_data_smithi159_80260_object67777:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 167.26079150999999, 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23222245999999999, 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 156890 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555431+0000", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.838e-06 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555437+0000", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555453+0000", 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6079e-05 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555463+0000", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7210000000000006e-06 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555538+0000", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4938999999999995e-05 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555631+0000", 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2980999999999996e-05 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555714+0000", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2655000000000001e-05 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.559420+0000", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037066360000000001 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.583952+0000", 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024531976 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787555+0000", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.203603273 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787605+0000", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9555000000000002e-05 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787652+0000", 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6606000000000001e-05 2024-04-15T18:17:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:18021 3.2 3:4eb6a1c5:::benchmark_data_smithi159_91621_object18020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:59.759233+0000", 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.056987909, 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209770816, 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 18021 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759233+0000", 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759233+0000", 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759234+0000", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.531e-06 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759273+0000", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8770999999999999e-05 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759284+0000", 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161999999999999e-05 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759290+0000", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9789999999999998e-06 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759336+0000", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6270000000000003e-05 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759451+0000", 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114681 2024-04-15T18:17:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.759595+0000", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014348400000000001 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.760412+0000", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081747899999999997 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.761520+0000", 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011079390000000001 2024-04-15T18:17:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968928+0000", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20740824999999999 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968967+0000", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8658000000000002e-05 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.969004+0000", 2024-04-15T18:17:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6612000000000001e-05 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:18021 3.2 e206/199 ondisk, result = 0)", 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:59.760801+0000", 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.055419560000004, 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20821952399999999, 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.760801+0000", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.760801+0000", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.760802+0000", 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4600000000000004e-07 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.760806+0000", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5750000000000002e-06 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968556+0000", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20775000199999999 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968561+0000", 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6450000000000004e-06 2024-04-15T18:17:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968919+0000", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035808900000000002 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.968924+0000", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9699999999999998e-06 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.969021+0000", 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6496999999999994e-05 2024-04-15T18:17:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95270 3.2 3:489b241f:::benchmark_data_smithi159_91621_object95269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.641748228000001, 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.224337434, 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95270 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174475+0000", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1100000000000001e-06 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174500+0000", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5009999999999999e-05 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174518+0000", 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8921000000000001e-05 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174529+0000", 2024-04-15T18:17:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0518e-05 2024-04-15T18:17:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174601+0000", 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2020000000000005e-05 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174741+0000", 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014028700000000001 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174933+0000", 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019125199999999999 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.177734+0000", 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028019450000000001 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.193915+0000", 2024-04-15T18:17:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016180439000000001 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398737+0000", 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20482228699999999 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398777+0000", 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9320999999999998e-05 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398810+0000", 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3324000000000001e-05 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95279 3.2 3:40c87ed6:::benchmark_data_smithi159_91621_object95278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.630841816, 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.221975389, 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95279 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185380+0000", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2720000000000001e-06 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185411+0000", 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0752e-05 2024-04-15T18:17:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185423+0000", 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1749999999999999e-05 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185429+0000", 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1539999999999999e-06 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185484+0000", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4839999999999997e-05 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185634+0000", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015063399999999999 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185796+0000", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016220500000000001 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.190169+0000", 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043726950000000002 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214259+0000", 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024090305999999999 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407247+0000", 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19298747799999999 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407307+0000", 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9840999999999998e-05 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407354+0000", 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7462000000000001e-05 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95280 3.2 3:4dc79632:::benchmark_data_smithi159_91621_object95279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:17:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.630481271000001, 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22356709, 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95280 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185740+0000", 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5499999999999998e-07 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185758+0000", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7938e-05 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185767+0000", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0690000000000008e-06 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185772+0000", 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3059999999999997e-06 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188088+0000", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00231525 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188222+0000", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013395799999999999 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188380+0000", 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158768 2024-04-15T18:17:24.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.190273+0000", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018926030000000001 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214298+0000", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024025517 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409241+0000", 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19494304000000001 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409277+0000", 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5970000000000003e-05 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409306+0000", 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9016000000000001e-05 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95297 3.2 3:491a683a:::benchmark_data_smithi159_91621_object95296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:17:24.901 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.209796+0000", 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.606425146999999, 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.199632899, 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95297 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209796+0000", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209796+0000", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209797+0000", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2020000000000001e-06 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209818+0000", 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1758000000000001e-05 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209829+0000", 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0808000000000001e-05 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209835+0000", 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0560000000000001e-06 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209883+0000", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7608999999999998e-05 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.209988+0000", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000105179 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.210084+0000", 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5712999999999997e-05 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214306+0000", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042221289999999998 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.235526+0000", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021219730999999999 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409344+0000", 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.173817949 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409390+0000", 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6813000000000001e-05 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409428+0000", 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7951999999999998e-05 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.907 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95301 3.2 3:4af59164:::benchmark_data_smithi159_91621_object95300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.232680+0000", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.583540488000001, 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.202668871, 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95301 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232680+0000", 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232680+0000", 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232681+0000", 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6899999999999996e-07 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232700+0000", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9083999999999999e-05 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232711+0000", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.082e-05 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232717+0000", 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7350000000000001e-06 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232765+0000", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8678999999999999e-05 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232866+0000", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010088400000000001 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.232953+0000", 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6632999999999995e-05 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.234916+0000", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019632859999999999 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.412797+0000", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17788095900000001 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.435246+0000", 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022448610000000001 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.435305+0000", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8752999999999998e-05 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.435349+0000", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4459000000000002e-05 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100487 3.2 3:4fca02bf:::benchmark_data_smithi159_91621_object100486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:17:24.913 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.002914384, 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22316820500000001, 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100487 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:17:24.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813307+0000", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0070000000000001e-06 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813334+0000", 2024-04-15T18:17:24.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6743000000000001e-05 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813342+0000", 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7200000000000006e-06 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813347+0000", 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3349999999999997e-06 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813410+0000", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3004999999999997e-05 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813540+0000", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000129791 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813693+0000", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015294199999999999 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816455+0000", 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027618970000000001 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817014+0000", 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055896799999999996 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036401+0000", 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21938772400000001 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036439+0000", 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7888e-05 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036474+0000", 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5185000000000003e-05 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100490 3.2 3:40fe6861:::benchmark_data_smithi159_91621_object100489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:17:24.919 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.999290548999999, 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22209844400000001, 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100490 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:17:24.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816931+0000", 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.029e-06 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816973+0000", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1925999999999998e-05 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816986+0000", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2619000000000001e-05 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816994+0000", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2439999999999993e-06 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817044+0000", 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9595000000000002e-05 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817196+0000", 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015271999999999999 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817296+0000", 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9607999999999999e-05 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.819483+0000", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021871479999999999 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.820743+0000", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012595880000000001 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.038932+0000", 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21818938299999999 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.038987+0000", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5034999999999997e-05 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.039029+0000", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1548999999999998e-05 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100493 3.2 3:4a5dce73:::benchmark_data_smithi159_91621_object100492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:17:24.925 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.993623823, 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22059732400000001, 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100493 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:17:24.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822598+0000", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.612e-06 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822654+0000", 2024-04-15T18:17:24.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5065000000000002e-05 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822667+0000", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3429e-05 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822674+0000", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4440000000000002e-06 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822718+0000", 2024-04-15T18:17:24.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4092999999999999e-05 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822797+0000", 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8799000000000002e-05 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822886+0000", 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8824000000000002e-05 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.826954+0000", 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040678049999999999 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.828433+0000", 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001478704 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043135+0000", 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21470222999999999 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043169+0000", 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4236000000000002e-05 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043194+0000", 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5083e-05 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100494 3.2 3:49a7f23a:::benchmark_data_smithi159_91621_object100493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:17:24.931 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.993529078, 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22061130900000001, 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100494 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:17:24.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:24.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822734+0000", 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1838000000000002e-05 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822742+0000", 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0220000000000004e-06 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822748+0000", 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2330000000000003e-06 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.824941+0000", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002193188 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.825023+0000", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1878000000000004e-05 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.825117+0000", 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3385000000000001e-05 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.827131+0000", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020148190000000002 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.830010+0000", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028787750000000001 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043234+0000", 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21322421999999999 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043270+0000", 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6010000000000003e-05 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043303+0000", 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2441000000000003e-05 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:24.937 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:25.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.560 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:17:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:25.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:25.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:25.705 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:25.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:25.909 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:25.911 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:25.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:26.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:26.054 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:26.213 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:26.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:26.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:26.354 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:26.382 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:26.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:26.431 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.3. 2024-04-15T18:17:26.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:26 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:17:26.213+0000 7f8015e7e700 -1 mon.a@0(leader).osd e233 definitely_dead 0 2024-04-15T18:17:26.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:26 smithi049 ceph-mon[40087]: pgmap v651: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 787 op/s 2024-04-15T18:17:26.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-15T18:17:26.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:26.519 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:26.768 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:26.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:26.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:26 smithi082 ceph-mon[44206]: pgmap v651: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 787 op/s 2024-04-15T18:17:26.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-15T18:17:26.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:26.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:26 smithi159 ceph-mon[45336]: pgmap v651: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 787 op/s 2024-04-15T18:17:26.883 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-04-15T18:17:26.900 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:26.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:17:27.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:27.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:27.352 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:27.380 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:27.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-15T18:17:27.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:27 smithi049 ceph-mon[40087]: osdmap e234: 12 total, 12 up, 12 in 2024-04-15T18:17:27.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:27 smithi049 ceph-mon[40087]: osdmap e235: 12 total, 12 up, 12 in 2024-04-15T18:17:27.512 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-15T18:17:27.512 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:27 smithi159 ceph-mon[45336]: osdmap e234: 12 total, 12 up, 12 in 2024-04-15T18:17:27.512 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:27 smithi159 ceph-mon[45336]: osdmap e235: 12 total, 12 up, 12 in 2024-04-15T18:17:27.527 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:27.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:27.631 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:27.663 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:27.684 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:27.685 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3253364603' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-04-15T18:17:27.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:27 smithi082 ceph-mon[44206]: osdmap e234: 12 total, 12 up, 12 in 2024-04-15T18:17:27.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:27 smithi082 ceph-mon[44206]: osdmap e235: 12 total, 12 up, 12 in 2024-04-15T18:17:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:27.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:27.786 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:27.787 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:17:27.787 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:17:27.787 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:27.787 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:27.817 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:27.882 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-15T18:17:27.904 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:28.016 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:28.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:28.081 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:17:28.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:17:28.175 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:17:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:28.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:17:28.437 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:17:28.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:28.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:28 smithi049 ceph-mon[40087]: pgmap v654: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.2 MiB/s wr, 789 op/s 2024-04-15T18:17:28.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:17:28.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:28 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:17:28.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:28 smithi049 ceph-mon[40087]: osdmap e236: 12 total, 12 up, 12 in 2024-04-15T18:17:28.742 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:28.755 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:28.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:28.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:28 smithi082 ceph-mon[44206]: pgmap v654: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.2 MiB/s wr, 789 op/s 2024-04-15T18:17:28.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:17:28.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:28 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:28.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:17:28.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:28 smithi082 ceph-mon[44206]: osdmap e236: 12 total, 12 up, 12 in 2024-04-15T18:17:28.860 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:28.887 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:28 smithi159 ceph-mon[45336]: pgmap v654: 17 pgs: 17 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.2 MiB/s wr, 789 op/s 2024-04-15T18:17:28.887 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:17:28.888 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:28 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:28.888 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1315526428' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:17:28.888 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:28 smithi159 ceph-mon[45336]: osdmap e236: 12 total, 12 up, 12 in 2024-04-15T18:17:28.889 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:29.022 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:29.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:29.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:29.173 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:29.192 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:29.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:29.297 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:29.325 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:29.579 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:29.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:29.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:29.711 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:30.293 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:30.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:30.341 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:30.380 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:30.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:30.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:30.431 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:30.455 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:30.457 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:30.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:30.508 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:30.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:30.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:30.585 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:30.599 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:30.599 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:17:30.599 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:17:30.600 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:17:30.600 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:30.620 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:30 smithi159 ceph-mon[45336]: pgmap v656: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-04-15T18:17:30.620 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/17644387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:30.620 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:30.625 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:30.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:30.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:30.693 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:30.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:30.732 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:30 smithi049 ceph-mon[40087]: pgmap v656: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-04-15T18:17:30.732 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/17644387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:30.735 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:30.761 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:30.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:30.780 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:30.780 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:30.780 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:30.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:30.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:30 smithi082 ceph-mon[44206]: pgmap v656: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-04-15T18:17:30.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/17644387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:30.887 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:30.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:30.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:31.067 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:31.166 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:31.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:31.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:31.337 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:31.349 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:17:31.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.598 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:31.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:31.643 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:31.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:17:31.754 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:31.781 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:31.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:31.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:31.920 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:31.942 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:31.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:32.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:32.076 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:32.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:32.528 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:32.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:32.632 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:32.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:32.717 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:32.720 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:32.720 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:32 smithi082 ceph-mon[44206]: pgmap v657: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 79/295506 objects degraded (0.027%); 30731/295506 objects misplaced (10.399%); 2.1 MiB/s, 35 objects/s recovering 2024-04-15T18:17:32.720 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:32 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 79/295506 objects degraded (0.027%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:32.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:32.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:32.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:32 smithi049 ceph-mon[40087]: pgmap v657: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 79/295506 objects degraded (0.027%); 30731/295506 objects misplaced (10.399%); 2.1 MiB/s, 35 objects/s recovering 2024-04-15T18:17:32.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:32 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 79/295506 objects degraded (0.027%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:32.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:32.854 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:32.882 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:17:32.883 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:17:32.883 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-15T18:17:32.883 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:17:32.883 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:17:32.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-15T18:17:32.889 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:32.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:32.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:32.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:32 smithi159 ceph-mon[45336]: pgmap v657: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 79/295506 objects degraded (0.027%); 30731/295506 objects misplaced (10.399%); 2.1 MiB/s, 35 objects/s recovering 2024-04-15T18:17:32.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:32 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 79/295506 objects degraded (0.027%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:32.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:33.057 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:33.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:17:33.649 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:17:33.649 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:17:33.653 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:33.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:33.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:33.798 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:33.970 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:33.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:34.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:34.113 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:34.281 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:34.364 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:34.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:34.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:17:34.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:34.520 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:34.554 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:34.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:34.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:34 smithi159 ceph-mon[45336]: pgmap v658: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 79/294696 objects degraded (0.027%); 30731/294696 objects misplaced (10.428%); 1.9 MiB/s, 31 objects/s recovering 2024-04-15T18:17:34.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:34.697 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:34.725 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:34.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:34 smithi049 ceph-mon[40087]: pgmap v658: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 79/294696 objects degraded (0.027%); 30731/294696 objects misplaced (10.428%); 1.9 MiB/s, 31 objects/s recovering 2024-04-15T18:17:34.727 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:34.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:34.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:34 smithi082 ceph-mon[44206]: pgmap v658: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 79/294696 objects degraded (0.027%); 30731/294696 objects misplaced (10.428%); 1.9 MiB/s, 31 objects/s recovering 2024-04-15T18:17:34.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:34.905 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:35.106 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:35.130 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:35.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:35.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:35.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:35.313 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:35.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:35.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:35.450 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:35.513 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:17:35.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/510897477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:35 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.576 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:35.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:35.680 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:35.710 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:35.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/510897477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:35 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/510897477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:35 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:17:35.763 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:35.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:35.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:35.918 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:36.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:36.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:36.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:17:36.341 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:36.441 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:17:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:36.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:17:36.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:36.586 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:36.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:36.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 205/290709 objects degraded (0.071%); 30164/290709 objects misplaced (10.376%); 5.0 MiB/s, 81 objects/s recovering 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: osdmap e237: 12 total, 12 up, 12 in 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:36.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:36.728 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:17:36.728 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:17:36.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 205/290709 objects degraded (0.071%); 30164/290709 objects misplaced (10.376%); 5.0 MiB/s, 81 objects/s recovering 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: osdmap e237: 12 total, 12 up, 12 in 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:36.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:36.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 205/290709 objects degraded (0.071%); 30164/290709 objects misplaced (10.376%); 5.0 MiB/s, 81 objects/s recovering 2024-04-15T18:17:36.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:36.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:17:36.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: osdmap e237: 12 total, 12 up, 12 in 2024-04-15T18:17:36.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:36.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:36.791 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:37.009 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:37.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:37.057 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:17:37.058 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:17:37.058 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:17:37.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:17:37.146 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:37.186 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:17:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:37.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:17:37.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:37.344 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:37.433 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:37.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:37.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:17:37.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:37.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:37 smithi049 ceph-mon[40087]: osdmap e238: 12 total, 12 up, 12 in 2024-04-15T18:17:37.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 205/290709 objects degraded (0.071%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:37.598 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:37.784 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:37.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:37 smithi082 ceph-mon[44206]: osdmap e238: 12 total, 12 up, 12 in 2024-04-15T18:17:37.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 205/290709 objects degraded (0.071%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:37 smithi159 ceph-mon[45336]: osdmap e238: 12 total, 12 up, 12 in 2024-04-15T18:17:37.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 205/290709 objects degraded (0.071%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:38.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:38.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:38.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:38.565 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:38.592 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:38.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:38.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:38.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:38 smithi049 ceph-mon[40087]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 326/286677 objects degraded (0.114%); 29778/286677 objects misplaced (10.387%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:17:38.753 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:38.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:38 smithi082 ceph-mon[44206]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 326/286677 objects degraded (0.114%); 29778/286677 objects misplaced (10.387%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:17:38.822 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:38.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:38.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:38 smithi159 ceph-mon[45336]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 326/286677 objects degraded (0.114%); 29778/286677 objects misplaced (10.387%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:17:38.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:39.005 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:39.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:39.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:39.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:39.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:39.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:39.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:39.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:39.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:39.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:39.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:39.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:39.590 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:39.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:39.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:39.642 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:39.643 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:39.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1411943262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:39.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1411943262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:39.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:39.661 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:39.668 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:39.668 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:17:39.668 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:17:39.669 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:17:39.669 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:39.697 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:39.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:39.802 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:39.835 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:39.935 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:39.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:40.039 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1411943262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.068 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:40.418 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:40.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:40.447 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:17:40.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:17:40.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:40.610 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:40.670 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: pgmap v663: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 326/286164 objects degraded (0.114%); 29778/286164 objects misplaced (10.406%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osdmap e239: 12 total, 12 up, 12 in 2024-04-15T18:17:40.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:17:40.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:40.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:40.781 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:17:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:40.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: pgmap v663: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 326/286164 objects degraded (0.114%); 29778/286164 objects misplaced (10.406%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:17:40.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osdmap e239: 12 total, 12 up, 12 in 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:40.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:17:40.947 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:40.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:41.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: pgmap v663: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 326/286164 objects degraded (0.114%); 29778/286164 objects misplaced (10.406%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:17:41.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osdmap e239: 12 total, 12 up, 12 in 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:41.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:17:41.048 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:41.314 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:41.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:41.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:17:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:41.508 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:17:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:41.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:17:41.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:17:41.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:17:41.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:17:41.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:41.648 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:41.689 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:41.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:41.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:41.842 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:41.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:17:41.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:17:41.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:17:42.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:17:42.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:17:42.049 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:17:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:42.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:42.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:17:42.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:42.568 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:42.596 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:42.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:42.694 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:17:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:17:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:17:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:17:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:4605 2.c 2:3286844e:::benchmark_data_smithi159_80260_object4604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:50.115532+0000", 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 352.57430320399999, 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21071807400000001, 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4605 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115532+0000", 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115532+0000", 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115533+0000", 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0800000000000003e-07 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115554+0000", 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1792000000000001e-05 2024-04-15T18:17:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115567+0000", 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2126e-05 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115574+0000", 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.272e-06 2024-04-15T18:17:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115615+0000", 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1684999999999998e-05 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115689+0000", 2024-04-15T18:17:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3098999999999999e-05 2024-04-15T18:17:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115782+0000", 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3231000000000002e-05 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116759+0000", 2024-04-15T18:17:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00097698300000000006 2024-04-15T18:17:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116772+0000", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3198e-05 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326176+0000", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209404373 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326219+0000", 2024-04-15T18:17:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2410999999999998e-05 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326250+0000", 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0995999999999999e-05 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:4606 2.c 2:3058ba51:::benchmark_data_smithi159_80260_object4605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:50.115590+0000", 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 352.57424503300001, 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21079308899999999, 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4606 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115590+0000", 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115590+0000", 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115590+0000", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1799999999999995e-07 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115608+0000", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7657999999999999e-05 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115614+0000", 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.421e-06 2024-04-15T18:17:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.115618+0000", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5489999999999998e-06 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116078+0000", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046045100000000001 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116193+0000", 2024-04-15T18:17:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114659 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116328+0000", 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000134628 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.116781+0000", 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000453244 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.117055+0000", 2024-04-15T18:17:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00027399100000000001 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326288+0000", 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20923340100000001 2024-04-15T18:17:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326344+0000", 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5653999999999999e-05 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:50.326383+0000", 2024-04-15T18:17:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8915000000000003e-05 2024-04-15T18:17:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33658 2.a 2:57a3c98d:::benchmark_data_smithi159_80260_object33657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:17:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.625682+0000", 2024-04-15T18:17:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 331.06415283400003, 2024-04-15T18:17:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19099132899999999, 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33658 2024-04-15T18:17:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625682+0000", 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625682+0000", 2024-04-15T18:17:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625683+0000", 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4109999999999999e-06 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625705+0000", 2024-04-15T18:17:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1182000000000001e-05 2024-04-15T18:17:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625719+0000", 2024-04-15T18:17:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.397e-05 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625726+0000", 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5290000000000003e-06 2024-04-15T18:17:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625770+0000", 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4248000000000001e-05 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625856+0000", 2024-04-15T18:17:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5298999999999998e-05 2024-04-15T18:17:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.625946+0000", 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9986000000000002e-05 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.627046+0000", 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001099975 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.628077+0000", 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010311949999999999 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.816578+0000", 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.188501535 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.816631+0000", 2024-04-15T18:17:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2574000000000002e-05 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.816673+0000", 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2425000000000001e-05 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97721 2.c 2:317c049a:::benchmark_data_smithi159_80260_object8608:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.483527+0000", 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 256.20630779099997, 2024-04-15T18:17:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19785586599999999, 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97721 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483527+0000", 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483527+0000", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483529+0000", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7889999999999999e-06 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483537+0000", 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1769999999999998e-06 2024-04-15T18:17:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483554+0000", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6566e-05 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483561+0000", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0639999999999996e-06 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483617+0000", 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6062e-05 2024-04-15T18:17:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483711+0000", 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4479000000000001e-05 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.483787+0000", 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5922999999999998e-05 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.486012+0000", 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022246269999999999 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.486550+0000", 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00053822899999999999 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.681303+0000", 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.194753237 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.681341+0000", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7455e-05 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.681383+0000", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2258000000000001e-05 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127622 2.a 2:5c0e539e:::benchmark_data_smithi159_80260_object38509:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:17:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 224.58304800299999, 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24240418699999999, 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127622 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106788+0000", 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.065e-06 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106793+0000", 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4149999999999998e-06 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106807+0000", 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3335e-05 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106815+0000", 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7059999999999993e-06 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107273+0000", 2024-04-15T18:17:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00045806500000000001 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107377+0000", 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103539 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107453+0000", 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6493000000000001e-05 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.110356+0000", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029024250000000001 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.115580+0000", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052237230000000004 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349093+0000", 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233513357 2024-04-15T18:17:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349144+0000", 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0858e-05 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349191+0000", 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7206000000000002e-05 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127639 2.a 2:50636722:::benchmark_data_smithi159_80260_object38526:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 224.55052673899999, 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233206792, 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127639 2024-04-15T18:17:42.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139310+0000", 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.657e-06 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139315+0000", 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.305e-06 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139330+0000", 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4800000000000001e-05 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139337+0000", 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.703e-06 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139389+0000", 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2540000000000002e-05 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139474+0000", 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4976999999999996e-05 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139542+0000", 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8164000000000007e-05 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.142496+0000", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029532320000000001 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.142583+0000", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7182000000000007e-05 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372433+0000", 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22984985399999999 2024-04-15T18:17:42.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372478+0000", 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5942000000000002e-05 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372515+0000", 2024-04-15T18:17:42.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6436000000000001e-05 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127653 2.a 2:53274d6c:::benchmark_data_smithi159_80260_object38540:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 224.528006635, 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23201696699999999, 2024-04-15T18:17:42.751 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127653 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161830+0000", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.451e-06 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161836+0000", 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4060000000000002e-06 2024-04-15T18:17:42.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161850+0000", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.419e-05 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161857+0000", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1250000000000004e-06 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161915+0000", 2024-04-15T18:17:42.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8025000000000001e-05 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162004+0000", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8350000000000006e-05 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162072+0000", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8047999999999996e-05 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.165030+0000", 2024-04-15T18:17:42.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029578320000000001 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.165619+0000", 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058962999999999995 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393773+0000", 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22815326699999999 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393811+0000", 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8380000000000002e-05 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393845+0000", 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4263e-05 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:133604 2.c 2:313518bc:::benchmark_data_smithi159_80260_object44491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 217.43669321300001, 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23184059300000001, 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.758 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 133604 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253143+0000", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6300000000000001e-06 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253149+0000", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.293e-06 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253162+0000", 2024-04-15T18:17:42.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3256e-05 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253170+0000", 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8539999999999997e-06 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253224+0000", 2024-04-15T18:17:42.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4737999999999997e-05 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253344+0000", 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119697 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253453+0000", 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109275 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.256826+0000", 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033722219999999998 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.257862+0000", 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010360670000000001 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484901+0000", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.227039084 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484942+0000", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1544000000000001e-05 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484982+0000", 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9932999999999999e-05 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.764 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:145720 2.a 2:51f97572:::benchmark_data_smithi159_80260_object56607:head [delete] snapc 0=[] ondisk+write+known_if_redirected e175)", 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:20.410247+0000", 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 202.27958738999999, 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19484631199999999, 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:17:42.765 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 145720 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410247+0000", 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410247+0000", 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410249+0000", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410256+0000", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7519999999999999e-06 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410270+0000", 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4154e-05 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410279+0000", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9940000000000006e-06 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410343+0000", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4127000000000005e-05 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410486+0000", 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014308800000000001 2024-04-15T18:17:42.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.410600+0000", 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011408000000000001 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.414478+0000", 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003877602 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.414796+0000", 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031850599999999998 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.604978+0000", 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19018147899999999 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.605040+0000", 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2067000000000001e-05 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:20.605094+0000", 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3918000000000001e-05 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:5358 3.0 3:093e027e:::benchmark_data_smithi159_91621_object5357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 111.68474722099999, 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27962654100000001, 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.771 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 5358 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:17:42.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005104+0000", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6022999999999999e-05 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005109+0000", 2024-04-15T18:17:42.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9779999999999996e-06 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005113+0000", 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9790000000000004e-06 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005140+0000", 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.654e-05 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005216+0000", 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6112000000000006e-05 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005297+0000", 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0896999999999995e-05 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005719+0000", 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042199799999999998 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.011523+0000", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058042659999999998 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284648+0000", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.273124747 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284691+0000", 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3671000000000003e-05 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284714+0000", 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2809000000000001e-05 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:7739 3.9 e204/199 3:942aaec8:::benchmark_data_smithi159_91621_object7738:head v 204'476, mlcod=204'472)", 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:52.387760+0000", 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.302075282, 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20239336699999999, 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.777 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387760+0000", 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387760+0000", 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387760+0000", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3600000000000003e-07 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387781+0000", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0982999999999999e-05 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.589407+0000", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.201625364 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.589417+0000", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0686999999999999e-05 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.589477+0000", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0173999999999997e-05 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.589499+0000", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.124e-05 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590119+0000", 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00062011800000000001 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590153+0000", 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4165000000000002e-05 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:15728 3.e 3:7aaffa93:::benchmark_data_smithi159_91621_object15727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:57.607223+0000", 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 105.082612252, 2024-04-15T18:17:42.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.214668251, 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15728 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607223+0000", 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607223+0000", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607223+0000", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2099999999999996e-07 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607241+0000", 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7549000000000001e-05 2024-04-15T18:17:42.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607254+0000", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2784e-05 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607262+0000", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8660000000000006e-06 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607300+0000", 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8272999999999998e-05 2024-04-15T18:17:42.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607406+0000", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106464 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.607504+0000", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8107000000000001e-05 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.608202+0000", 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069711399999999998 2024-04-15T18:17:42.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.609296+0000", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010940209999999999 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.821802+0000", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21250597600000001 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.821832+0000", 2024-04-15T18:17:42.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0827000000000001e-05 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.821891+0000", 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8548999999999998e-05 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:47308 3.0 3:09f1b554:::benchmark_data_smithi159_91621_object47307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:16:19.501859+0000", 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 83.187975678000001, 2024-04-15T18:17:42.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21478139299999999, 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 47308 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501859+0000", 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501859+0000", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501860+0000", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6199999999999997e-07 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501878+0000", 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7688000000000001e-05 2024-04-15T18:17:42.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501892+0000", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4242e-05 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.501901+0000", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6330000000000002e-06 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.503228+0000", 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001327915 2024-04-15T18:17:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.503354+0000", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012556399999999999 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.503495+0000", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000141292 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.505127+0000", 2024-04-15T18:17:42.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016312150000000001 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.505505+0000", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000377995 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716534+0000", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.211029882 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716595+0000", 2024-04-15T18:17:42.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0535000000000002e-05 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716641+0000", 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5670000000000002e-05 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:47317 3.0 3:0e5e286f:::benchmark_data_smithi159_91621_object47316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:16:19.513095+0000", 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 83.176739612999995, 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.203653269, 2024-04-15T18:17:42.793 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 47317 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513095+0000", 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513095+0000", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513097+0000", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6390000000000001e-06 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513125+0000", 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8014e-05 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513138+0000", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3083e-05 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513147+0000", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4050000000000006e-06 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513199+0000", 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1918e-05 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513298+0000", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8488999999999998e-05 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.513395+0000", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7584999999999995e-05 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.515027+0000", 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016314019999999999 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.515488+0000", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046160899999999998 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716686+0000", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.201197185 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716721+0000", 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-04-15T18:17:42.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.716749+0000", 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7630000000000001e-05 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:47320 3.0 3:063babe4:::benchmark_data_smithi159_91621_object47319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:16:19.520374+0000", 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 83.169460639999997, 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19709242199999999, 2024-04-15T18:17:42.799 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 47320 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520374+0000", 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520374+0000", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520376+0000", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.556e-06 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520397+0000", 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1353999999999999e-05 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520413+0000", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6089000000000001e-05 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520424+0000", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0736e-05 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520478+0000", 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4218000000000002e-05 2024-04-15T18:17:42.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520591+0000", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011244899999999999 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.520700+0000", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000108903 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.522543+0000", 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018436539999999999 2024-04-15T18:17:42.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.522823+0000", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028009600000000002 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.717374+0000", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.194550208 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.717426+0000", 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2955999999999999e-05 2024-04-15T18:17:42.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:19.717467+0000", 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0203000000000001e-05 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:113491 3.0 3:04647499:::benchmark_data_smithi159_91621_object7015:head [delete] snapc 0=[] ondisk+write+known_if_redirected e233)", 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.328305588999999, 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51810229799999996, 2024-04-15T18:17:42.805 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113491 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361530+0000", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.78e-07 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361535+0000", 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8110000000000002e-06 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361547+0000", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1824999999999999e-05 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361555+0000", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4740000000000005e-06 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361610+0000", 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4533000000000001e-05 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361681+0000", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1631000000000006e-05 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361742+0000", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0473999999999997e-05 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.364473+0000", 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027306689999999998 2024-04-15T18:17:42.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.367892+0000", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034197569999999998 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879504+0000", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.511612132 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879569+0000", 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4632000000000001e-05 2024-04-15T18:17:42.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879632+0000", 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2482000000000002e-05 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:113491 3.0 e233/199 ondisk, result = 0)", 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.321395386999999, 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51119831599999999, 2024-04-15T18:17:42.811 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368440+0000", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4600000000000005e-07 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368443+0000", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1939999999999998e-06 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879387+0000", 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51094377400000002 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879398+0000", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1049999999999999e-05 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879485+0000", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6513000000000003e-05 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879497+0000", 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.221e-05 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879638+0000", 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014102900000000001 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:113549 3.0 3:0fbc0d3a:::benchmark_data_smithi159_91621_object7073:head [delete] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.252965482, 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64867516599999997, 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.815 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113549 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436871+0000", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436877+0000", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4999999999999999e-06 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436888+0000", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1647e-05 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436896+0000", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1410000000000005e-06 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436946+0000", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9110999999999998e-05 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.437042+0000", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6496000000000006e-05 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.437121+0000", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8616999999999996e-05 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439983+0000", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028622880000000002 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440106+0000", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012289899999999999 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085464+0000", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64535768299999996 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085502+0000", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8847999999999998e-05 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085545+0000", 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2194999999999998e-05 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:113549 3.0 e234/199 ondisk, result = 0)", 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.250020490000001, 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64573638099999997, 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439815+0000", 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.44e-07 2024-04-15T18:17:42.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439820+0000", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9359999999999998e-06 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085391+0000", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64557072699999996 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085398+0000", 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9e-06 2024-04-15T18:17:42.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085452+0000", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4095000000000003e-05 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085460+0000", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8630000000000008e-06 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085551+0000", 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1216000000000002e-05 2024-04-15T18:17:42.824 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:113551 3.e 3:7e2db53f:::benchmark_data_smithi159_91621_object7075:head [delete] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.440243+0000", 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.249591761000001, 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.204228354, 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:17:42.825 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113551 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440243+0000", 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440243+0000", 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440244+0000", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440248+0000", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9539999999999998e-06 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440256+0000", 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440262+0000", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7630000000000003e-06 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440304+0000", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1372000000000003e-05 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440400+0000", 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5723999999999994e-05 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440467+0000", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7862999999999997e-05 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.443021+0000", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025536640000000002 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.443569+0000", 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000547498 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.644406+0000", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20083773199999999 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.644440+0000", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3864e-05 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.644471+0000", 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1310999999999999e-05 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.830 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:17:42.831 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:17:42.831 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:17:42.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:42.872 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:42 smithi082 ceph-mon[44206]: pgmap v665: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 514/279006 objects degraded (0.184%); 28847/279006 objects misplaced (10.339%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:42.873 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 514/279006 objects degraded (0.184%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:42.883 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:42.927 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:42.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:43.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:17:43.068 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:43.089 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:17:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:43.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:17:43.119 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:43.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:42 smithi159 ceph-mon[45336]: pgmap v665: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 514/279006 objects degraded (0.184%); 28847/279006 objects misplaced (10.339%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:43.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 514/279006 objects degraded (0.184%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:43.197 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:43.228 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:43.232 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:42 smithi049 ceph-mon[40087]: pgmap v665: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 514/279006 objects degraded (0.184%); 28847/279006 objects misplaced (10.339%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:43.232 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 514/279006 objects degraded (0.184%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:43.483 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:43.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:43.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:17:43.587 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:43.617 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:43.637 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:43.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:43.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:43.819 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:43.891 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:17:43.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1950578458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:43.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:43 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:44.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1950578458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:44.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:43 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:44.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1950578458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:44.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:43 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:17:44.408 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:17:44.408 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:17:44.408 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-15T18:17:44.408 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:17:44.408 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:17:44.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-15T18:17:44.793 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: pgmap v666: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 514/278055 objects degraded (0.185%); 28847/278055 objects misplaced (10.375%); 8.2 MiB/s, 132 objects/s recovering 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:45.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:44 smithi082 ceph-mon[44206]: osdmap e240: 12 total, 12 up, 12 in 2024-04-15T18:17:45.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: pgmap v666: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 514/278055 objects degraded (0.185%); 28847/278055 objects misplaced (10.375%); 8.2 MiB/s, 132 objects/s recovering 2024-04-15T18:17:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:45.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:44 smithi159 ceph-mon[45336]: osdmap e240: 12 total, 12 up, 12 in 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: pgmap v666: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 514/278055 objects degraded (0.185%); 28847/278055 objects misplaced (10.375%); 8.2 MiB/s, 132 objects/s recovering 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:17:45.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:44 smithi049 ceph-mon[40087]: osdmap e240: 12 total, 12 up, 12 in 2024-04-15T18:17:45.202 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:45.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:45.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:45.322 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:45.350 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:45.562 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:45.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:45.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:17:45.723 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:45.806 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:17:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:17:45.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:45.959 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:46.329 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:46.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:46.438 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:17:46.480 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:46.613 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:17:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:46.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:17:46.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:46.745 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:46.767 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:46.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:46.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:17:46.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:17:46.961 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:17:46.961 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:17:46.962 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: pgmap v668: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 545/274668 objects degraded (0.198%); 28336/274668 objects misplaced (10.316%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3955072304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:46.975 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:47.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: pgmap v668: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 545/274668 objects degraded (0.198%); 28336/274668 objects misplaced (10.316%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3955072304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: pgmap v668: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 545/274668 objects degraded (0.198%); 28336/274668 objects misplaced (10.316%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:17:47.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3955072304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:17:47.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.298 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.299 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.299 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.299 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.299 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:47.350 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:17:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:47.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:17:47.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:17:47.507 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:47.605 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:17:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:47.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:17:47.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:47.755 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:48.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 545/274668 objects degraded (0.198%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 545/274668 objects degraded (0.198%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:17:48.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 545/274668 objects degraded (0.198%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:49.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:48 smithi159 ceph-mon[45336]: pgmap v669: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 673/270915 objects degraded (0.248%); 27929/270915 objects misplaced (10.309%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:49.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:48 smithi049 ceph-mon[40087]: pgmap v669: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 673/270915 objects degraded (0.248%); 27929/270915 objects misplaced (10.309%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:49.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:48 smithi082 ceph-mon[44206]: pgmap v669: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 673/270915 objects degraded (0.248%); 27929/270915 objects misplaced (10.309%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:51.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:50 smithi049 ceph-mon[40087]: pgmap v670: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 673/270387 objects degraded (0.249%); 27929/270387 objects misplaced (10.329%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:17:51.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:51.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:51.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:50 smithi049 ceph-mon[40087]: osdmap e241: 12 total, 12 up, 12 in 2024-04-15T18:17:51.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:50 smithi082 ceph-mon[44206]: pgmap v670: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 673/270387 objects degraded (0.249%); 27929/270387 objects misplaced (10.329%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:17:51.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:51.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:51.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:50 smithi082 ceph-mon[44206]: osdmap e241: 12 total, 12 up, 12 in 2024-04-15T18:17:51.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:50 smithi159 ceph-mon[45336]: pgmap v670: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 673/270387 objects degraded (0.249%); 27929/270387 objects misplaced (10.329%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:17:51.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:51.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:51.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:50 smithi159 ceph-mon[45336]: osdmap e241: 12 total, 12 up, 12 in 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:17:51.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:17:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:17:51.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:17:51.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:17:51.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:17:51.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:17:52.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:52.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:52.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:52.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:52.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:52.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:52.796 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:17:52.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 0 1 2024-04-15T18:17:53.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:52 smithi049 ceph-mon[40087]: pgmap v672: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 886/263427 objects degraded (0.336%); 26973/263427 objects misplaced (10.239%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:53.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:53.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:53.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:53 smithi049 ceph-mon[40087]: osdmap e242: 12 total, 12 up, 12 in 2024-04-15T18:17:53.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 886/263427 objects degraded (0.336%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:53.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:52 smithi082 ceph-mon[44206]: pgmap v672: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 886/263427 objects degraded (0.336%); 26973/263427 objects misplaced (10.239%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:53.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:53.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:53.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:52 smithi082 ceph-mon[44206]: osdmap e242: 12 total, 12 up, 12 in 2024-04-15T18:17:53.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 886/263427 objects degraded (0.336%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:53.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:52 smithi159 ceph-mon[45336]: pgmap v672: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 886/263427 objects degraded (0.336%); 26973/263427 objects misplaced (10.239%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:17:53.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:53.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:53.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:52 smithi159 ceph-mon[45336]: osdmap e242: 12 total, 12 up, 12 in 2024-04-15T18:17:53.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 886/263427 objects degraded (0.336%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:55.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:54 smithi049 ceph-mon[40087]: pgmap v674: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 886/262362 objects degraded (0.338%); 26973/262362 objects misplaced (10.281%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:17:55.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:55.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:55.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:54 smithi049 ceph-mon[40087]: osdmap e243: 12 total, 12 up, 12 in 2024-04-15T18:17:55.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:54 smithi082 ceph-mon[44206]: pgmap v674: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 886/262362 objects degraded (0.338%); 26973/262362 objects misplaced (10.281%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:17:55.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:55.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:55.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:54 smithi082 ceph-mon[44206]: osdmap e243: 12 total, 12 up, 12 in 2024-04-15T18:17:55.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:54 smithi159 ceph-mon[45336]: pgmap v674: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 886/262362 objects degraded (0.338%); 26973/262362 objects misplaced (10.281%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:17:55.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:55.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:55.408 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:54 smithi159 ceph-mon[45336]: osdmap e243: 12 total, 12 up, 12 in 2024-04-15T18:17:57.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:56 smithi082 ceph-mon[44206]: pgmap v676: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 922/259323 objects degraded (0.356%); 26773/259323 objects misplaced (10.324%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:17:57.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:56 smithi159 ceph-mon[45336]: pgmap v676: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 922/259323 objects degraded (0.356%); 26773/259323 objects misplaced (10.324%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:17:57.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:56 smithi049 ceph-mon[40087]: pgmap v676: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 922/259323 objects degraded (0.356%); 26773/259323 objects misplaced (10.324%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:17:57.985 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:57.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:58.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:17:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:58.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 922/259323 objects degraded (0.356%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:58.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:58 smithi049 ceph-mon[40087]: pgmap v677: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1120/253422 objects degraded (0.442%); 25980/253422 objects misplaced (10.252%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:58.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 922/259323 objects degraded (0.356%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:58.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:58 smithi159 ceph-mon[45336]: pgmap v677: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1120/253422 objects degraded (0.442%); 25980/253422 objects misplaced (10.252%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:58.521 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:17:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:58.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:17:58.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 922/259323 objects degraded (0.356%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:17:58.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:58 smithi082 ceph-mon[44206]: pgmap v677: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1120/253422 objects degraded (0.442%); 25980/253422 objects misplaced (10.252%); 10 MiB/s, 167 objects/s recovering 2024-04-15T18:17:58.566 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:58.574 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:17:58.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:17:58.655 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:58.675 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:17:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:58.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:17:58.779 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:58.806 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:58.826 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:58.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:58.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:17:58.958 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:58.980 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:17:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:58.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:17:59.084 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:17:59.113 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:59.134 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:17:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:59.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:17:59.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:17:59.308 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:17:59.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:17:59.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:17:59.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:17:59 smithi159 ceph-mon[45336]: osdmap e244: 12 total, 12 up, 12 in 2024-04-15T18:17:59.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:17:59.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:17:59.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:17:59 smithi049 ceph-mon[40087]: osdmap e244: 12 total, 12 up, 12 in 2024-04-15T18:17:59.495 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:17:59.495 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:17:59.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:17:59.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:17:59.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:17:59 smithi082 ceph-mon[44206]: osdmap e244: 12 total, 12 up, 12 in 2024-04-15T18:17:59.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:17:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:17:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:17:59.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:00.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:00.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:00 smithi049 ceph-mon[40087]: pgmap v679: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 1120/252882 objects degraded (0.443%); 25980/252882 objects misplaced (10.274%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:18:00.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:00 smithi159 ceph-mon[45336]: pgmap v679: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 1120/252882 objects degraded (0.443%); 25980/252882 objects misplaced (10.274%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:18:00.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:00 smithi082 ceph-mon[44206]: pgmap v679: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 1120/252882 objects degraded (0.443%); 25980/252882 objects misplaced (10.274%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:18:00.717 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:00.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:00.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:00.891 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:01.124 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:01.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:01.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:01.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:18:01.816 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:01.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:01.923 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:01.958 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:01.990 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:01.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:02.093 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:02.126 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:02.151 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:02.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:02.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:02.309 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:02.454 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:02.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:02.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: 3.4 scrub starts 2024-04-15T18:18:02.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:02.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:02.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:02.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:02.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1365/244677 objects degraded (0.558%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: 3.4 scrub starts 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:02.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1365/244677 objects degraded (0.558%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:02.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:02.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:02.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: 3.4 scrub starts 2024-04-15T18:18:02.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:02.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:02.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1365/244677 objects degraded (0.558%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:02.669 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:02.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:02.779 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:02.806 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:02.895 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:02.930 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:03 smithi049 ceph-mon[40087]: 3.4 scrub ok 2024-04-15T18:18:03.236 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:18:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4091677876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:03.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:03 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:03.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:03 smithi082 ceph-mon[44206]: 3.4 scrub ok 2024-04-15T18:18:03.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4091677876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:03.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:03 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:03.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:03 smithi159 ceph-mon[45336]: 3.4 scrub ok 2024-04-15T18:18:03.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4091677876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:03.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:03 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:04.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: pgmap v681: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 1365/243768 objects degraded (0.560%); 24990/243768 objects misplaced (10.252%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: osdmap e245: 12 total, 12 up, 12 in 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:04.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: pgmap v681: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 1365/243768 objects degraded (0.560%); 24990/243768 objects misplaced (10.252%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: osdmap e245: 12 total, 12 up, 12 in 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:04.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:04.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:04.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: pgmap v681: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 1365/243768 objects degraded (0.560%); 24990/243768 objects misplaced (10.252%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:18:04.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:18:04.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:04.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: osdmap e245: 12 total, 12 up, 12 in 2024-04-15T18:18:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:18:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:18:04.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:18:04.883 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:04.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:04.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:05.056 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:06.837 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:06.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:06.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:07.042 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:07.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1469/240135 objects degraded (0.612%); 24444/240135 objects misplaced (10.179%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:18:07.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:06 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1469/240135 objects degraded (0.612%); 24444/240135 objects misplaced (10.179%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:06 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1469/240135 objects degraded (0.612%); 24444/240135 objects misplaced (10.179%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:18:07.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:18:07.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:18:07.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:06 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:18:07.320 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:07.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:07.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:07.481 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1469/240135 objects degraded (0.612%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: osdmap e246: 12 total, 12 up, 12 in 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3912772299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:08.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:07 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:08.189 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-04-15T18:18:08.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:08.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:08.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1469/240135 objects degraded (0.612%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:08.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: osdmap e246: 12 total, 12 up, 12 in 2024-04-15T18:18:08.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3912772299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:08.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:07 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1469/240135 objects degraded (0.612%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: osdmap e246: 12 total, 12 up, 12 in 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3912772299' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:08.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:07 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:18:09.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:08 smithi159 ceph-mon[45336]: pgmap v684: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 1586/235701 objects degraded (0.673%); 24049/235701 objects misplaced (10.203%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:09.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:08 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:18:09.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:08 smithi159 ceph-mon[45336]: osdmap e247: 12 total, 12 up, 12 in 2024-04-15T18:18:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:08 smithi049 ceph-mon[40087]: pgmap v684: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 1586/235701 objects degraded (0.673%); 24049/235701 objects misplaced (10.203%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:08 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:18:09.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:08 smithi049 ceph-mon[40087]: osdmap e247: 12 total, 12 up, 12 in 2024-04-15T18:18:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:08 smithi082 ceph-mon[44206]: pgmap v684: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 1586/235701 objects degraded (0.673%); 24049/235701 objects misplaced (10.203%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:08 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:18:09.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:08 smithi082 ceph-mon[44206]: osdmap e247: 12 total, 12 up, 12 in 2024-04-15T18:18:09.599 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:18:09.599 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:18:10.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:18:10.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 1 1 2024-04-15T18:18:10.777 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:10.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:10.806 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:10.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:10.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:10 smithi049 ceph-mon[40087]: pgmap v687: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 1586/235188 objects degraded (0.674%); 24049/235188 objects misplaced (10.225%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:18:10.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:11.047 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:11.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:11.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:11.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:10 smithi159 ceph-mon[45336]: pgmap v687: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 1586/235188 objects degraded (0.674%); 24049/235188 objects misplaced (10.225%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:18:11.182 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:11.202 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:11.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:11.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:10 smithi082 ceph-mon[44206]: pgmap v687: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 1586/235188 objects degraded (0.674%); 24049/235188 objects misplaced (10.225%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:18:11.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:11.336 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:11.357 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:11.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:11.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:11.524 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:11.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:18:11.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:18:11.566 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:11.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:11.630 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:18:11.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:11.694 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:18:11.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:11.755 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:18:11.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:18:11.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:18:11.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:18:11.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:18:12.014 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:18:12.049 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:18:12.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:18:12.122 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:18:12.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:18:12.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2117137860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:12.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2117137860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:12.296 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2117137860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:12.621 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:12.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:12.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:12.802 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:12.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: pgmap v688: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 1821/227301 objects degraded (0.801%); 23064/227301 objects misplaced (10.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:18:12.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:12.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1821/227301 objects degraded (0.801%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: pgmap v688: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 1821/227301 objects degraded (0.801%); 23064/227301 objects misplaced (10.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1821/227301 objects degraded (0.801%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: pgmap v688: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 1821/227301 objects degraded (0.801%); 23064/227301 objects misplaced (10.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1821/227301 objects degraded (0.801%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:13.395 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:13.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:13.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:14.136 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:14.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:14.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:14.207 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:18:14.207 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:18:14.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:18:14.207 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:18:14.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 239/235 [PushOp(3:4ad20f5a:::benchmark_data_smithi159_91621_object13797:head, version: 205'836, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ad20f5a:::benchmark_data_smithi159_91621_object13797:head@205'836, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:40.057024+0000", 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 34.145277481999997, 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.065665578000000002, 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.057024+0000", 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.057024+0000", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.057024+0000", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8999999999999996e-07 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.057171+0000", 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014643800000000001 2024-04-15T18:18:14.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.120285+0000", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.063114359999999994 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.120302+0000", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7130999999999999e-05 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.120408+0000", 2024-04-15T18:18:14.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105741 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.120422+0000", 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3557000000000001e-05 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.122689+0000", 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022676609999999998 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:120264 3.b e239/235 3:d3c346b0:::benchmark_data_smithi159_91621_object13788:head v 239'9740, mlcod=206'1517)", 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:40.384401+0000", 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.817900082999998, 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.059576049999999998, 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384401+0000", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384401+0000", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384402+0000", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1620000000000001e-06 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384412+0000", 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3870000000000001e-06 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384452+0000", 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0271000000000001e-05 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.384461+0000", 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9160000000000007e-06 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.393216+0000", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0087552810000000002 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.393249+0000", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3064000000000001e-05 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.443943+0000", 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.050693571 2024-04-15T18:18:14.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.443977+0000", 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4397999999999998e-05 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:120291 3.2 e239/235 3:40bbdec9:::benchmark_data_smithi159_91621_object13815:head v 239'7582, mlcod=205'846)", 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:40.445025+0000", 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.757276267999998, 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.057313900000000001, 2024-04-15T18:18:14.216 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445025+0000", 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445025+0000", 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445026+0000", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445033+0000", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8070000000000002e-06 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445078+0000", 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5637999999999999e-05 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445094+0000", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5869e-05 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445424+0000", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00033028499999999999 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.445454+0000", 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9981e-05 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.502268+0000", 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.056814157999999997 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:40.502339+0000", 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0463999999999995e-05 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.220 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 239/235 [PushOp(3:47c7e8b8:::benchmark_data_smithi159_91621_object14155:head, version: 205'868, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:47c7e8b8:::benchmark_data_smithi159_91621_object14155:head@205'868, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:41.116968+0000", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.085333120999998, 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.056424747999999997, 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.116968+0000", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.116968+0000", 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.116969+0000", 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.066e-06 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.117007+0000", 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7465000000000001e-05 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.170693+0000", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053685846000000002 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.170709+0000", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6943999999999999e-05 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.170809+0000", 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9283999999999993e-05 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.170829+0000", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0446999999999999e-05 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.173393+0000", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025636959999999999 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:120748 3.2 e239/235 3:471ec45d:::benchmark_data_smithi159_91621_object14272:head v 239'7628, mlcod=205'879)", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:41.452810+0000", 2024-04-15T18:18:14.224 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 32.749491466000002, 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.067857065999999994, 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.452810+0000", 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.452810+0000", 2024-04-15T18:18:14.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.452811+0000", 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2500000000000001e-06 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.452822+0000", 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0521e-05 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.479080+0000", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.026258422999999999 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.479087+0000", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5690000000000001e-06 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.479173+0000", 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6081999999999993e-05 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.479200+0000", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6855999999999998e-05 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.520622+0000", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.041422355000000001 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:41.520667+0000", 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5009999999999998e-05 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.228 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:121543 3.c e239/235 3:3fe2010a:::benchmark_data_smithi159_91621_object15067:head v 239'7457, mlcod=205'953)", 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:43.100250+0000", 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.102051380999999, 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.055063011000000002, 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100250+0000", 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100250+0000", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100250+0000", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9399999999999995e-07 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100254+0000", 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7620000000000001e-06 2024-04-15T18:18:14.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114148+0000", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013893407 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114160+0000", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2638e-05 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114421+0000", 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000260449 2024-04-15T18:18:14.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114446+0000", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5704999999999999e-05 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.155265+0000", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.040818552000000001 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.155313+0000", 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8004000000000003e-05 2024-04-15T18:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:121545 3.c e239/235 3:326b54b8:::benchmark_data_smithi159_91621_object15069:head v 239'7458, mlcod=205'953)", 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:43.100256+0000", 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.102045729, 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.055084830000000001, 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100256+0000", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100256+0000", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100256+0000", 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4000000000000001e-07 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.100260+0000", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0629999999999999e-06 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114183+0000", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013923413000000001 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.114187+0000", 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2440000000000004e-06 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.122115+0000", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079274340000000006 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.122141+0000", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5961000000000001e-05 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.155316+0000", 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033174985999999997 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.155340+0000", 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4589000000000001e-05 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:121802 3.5 e239/235 3:a53f4c2f:::benchmark_data_smithi159_91621_object15326:head v 239'7718, mlcod=205'1012)", 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:43.676441+0000", 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 30.525860101999999, 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.054663541000000003, 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.237 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.676441+0000", 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.676441+0000", 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.676442+0000", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8000000000000004e-07 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.676454+0000", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.181e-05 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.704075+0000", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.027620709 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.704083+0000", 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5040000000000002e-06 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.704157+0000", 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3951999999999999e-05 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.704187+0000", 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0406e-05 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.731052+0000", 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.026864656000000001 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:43.731105+0000", 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2624000000000003e-05 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 240/235 [PushOp(3:4a9184f3:::benchmark_data_smithi159_91621_object16178:head, version: 205'1003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4a9184f3:::benchmark_data_smithi159_91621_object16178:head@205'1003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:44.654504+0000", 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 29.547797194000001, 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.089680331000000002, 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.654504+0000", 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.654504+0000", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.654505+0000", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9699999999999994e-07 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.654685+0000", 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000180182 2024-04-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.742179+0000", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.087493786000000004 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.742192+0000", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3111e-05 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.742266+0000", 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3905999999999993e-05 2024-04-15T18:18:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.742277+0000", 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0540000000000001e-05 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.744184+0000", 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019078089999999999 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 240/235 [PushOp(3:465939d0:::benchmark_data_smithi159_91621_object16237:head, version: 205'1009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:465939d0:::benchmark_data_smithi159_91621_object16237:head@205'1009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:44.909153+0000", 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 29.293148640999998, 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.093278723999999993, 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.909153+0000", 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.909153+0000", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.909154+0000", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.15e-06 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:44.909241+0000", 2024-04-15T18:18:14.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7126999999999994e-05 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:45.000419+0000", 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.091178190000000006 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:45.000433+0000", 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3638000000000001e-05 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:45.000508+0000", 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4882000000000006e-05 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:45.000516+0000", 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1100000000000003e-06 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:45.002431+0000", 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001915627 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.249 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:4e49654d:::benchmark_data_smithi159_91621_object31778:head, version: 206'2008, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4e49654d:::benchmark_data_smithi159_91621_object31778:head@206'2008, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:11.714669+0000", 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.4876324690000002, 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.090748230999999999, 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.714669+0000", 2024-04-15T18:18:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.714669+0000", 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.714669+0000", 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2500000000000001e-07 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.715059+0000", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038944000000000001 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.801705+0000", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.086646170999999994 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.801722+0000", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7084000000000001e-05 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.801851+0000", 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012959399999999999 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.801870+0000", 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8938999999999999e-05 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.805417+0000", 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035466780000000002 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:4785a230:::benchmark_data_smithi159_91621_object31891:head, version: 206'2012, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4785a230:::benchmark_data_smithi159_91621_object31891:head@206'2012, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.316707547, 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.202080965, 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885595+0000", 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.116e-06 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885822+0000", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022720600000000001 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085100+0000", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19927798299999999 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085112+0000", 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2310000000000001e-05 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085208+0000", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5311999999999998e-05 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085222+0000", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4099e-05 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.087675+0000", 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024529389999999999 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.257 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:4b26f16d:::benchmark_data_smithi159_91621_object31968:head, version: 206'2015, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4b26f16d:::benchmark_data_smithi159_91621_object31968:head@206'2015, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:12.126330+0000", 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.0759710340000002, 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.067619243999999995, 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.126330+0000", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.126330+0000", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.126331+0000", 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.96e-07 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.126381+0000", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0417000000000002e-05 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.178697+0000", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.052316224000000001 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.178719+0000", 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1557999999999999e-05 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.191453+0000", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012733863999999999 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.191468+0000", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5437000000000001e-05 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.193949+0000", 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002481348 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.261 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:45af419c:::benchmark_data_smithi159_91621_object32052:head, version: 206'2020, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:45af419c:::benchmark_data_smithi159_91621_object32052:head@206'2020, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:12.221317+0000", 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.980984619, 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.065321295000000001, 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.221317+0000", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.221317+0000", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.221318+0000", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5300000000000003e-07 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.221394+0000", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6548e-05 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.283182+0000", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.061788360000000001 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.283200+0000", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7873999999999999e-05 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.283299+0000", 2024-04-15T18:18:14.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8697999999999996e-05 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.283314+0000", 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4748e-05 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.286638+0000", 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033242139999999998 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:48c49d5e:::benchmark_data_smithi159_91621_object32070:head, version: 206'2024, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:48c49d5e:::benchmark_data_smithi159_91621_object32070:head@206'2024, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:12.318459+0000", 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.8838419870000001, 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.064089066, 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.318459+0000", 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.318459+0000", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.318460+0000", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2900000000000002e-07 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.318516+0000", 2024-04-15T18:18:14.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5702999999999998e-05 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.379140+0000", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.060624487999999997 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.379156+0000", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5413999999999998e-05 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.379239+0000", 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3251999999999995e-05 2024-04-15T18:18:14.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.379253+0000", 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3779e-05 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.382548+0000", 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003295601 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:139561 3.b e247/235 3:da63beec:::benchmark_data_smithi159_91621_object33085:head v 247'10949, mlcod=214'3593)", 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:14.014344+0000", 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.187957177, 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.057200446000000002, 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014344+0000", 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014344+0000", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014345+0000", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2950000000000001e-06 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014355+0000", 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4050000000000006e-06 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014375+0000", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0415e-05 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.014384+0000", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4679999999999992e-06 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.015395+0000", 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010113259999999999 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.015426+0000", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0750999999999997e-05 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.071497+0000", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.056071309 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.071545+0000", 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7477e-05 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.273 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:139562 3.b e247/235 3:d7c56f94:::benchmark_data_smithi159_91621_object33086:head v 247'10950, mlcod=214'3593)", 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:14.018316+0000", 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.18398524599999999, 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053256879999999999, 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018316+0000", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018316+0000", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018316+0000", 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.72e-07 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018320+0000", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8519999999999997e-06 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018400+0000", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9845999999999994e-05 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.018405+0000", 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.101e-06 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.023892+0000", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005486711 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.023922+0000", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0090000000000002e-05 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.071549+0000", 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.047626934000000003 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.071573+0000", 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4074e-05 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.5 247/235 [PushOp(3:a4924864:::benchmark_data_smithi159_91621_object33161:head, version: 206'2165, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a4924864:::benchmark_data_smithi159_91621_object33161:head@206'2165, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:14.081292+0000", 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.12100881099999999, 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.054541398999999997, 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.278 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.081292+0000", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.081292+0000", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.081293+0000", 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0070000000000001e-06 2024-04-15T18:18:14.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.081348+0000", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4401000000000003e-05 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132675+0000", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.051327418 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132696+0000", 2024-04-15T18:18:14.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0554e-05 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132809+0000", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011351800000000001 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132822+0000", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3074000000000001e-05 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.135834+0000", 2024-04-15T18:18:14.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030114270000000001 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.c 247/235 [PushOp(3:3383464b:::benchmark_data_smithi159_91621_object33181:head, version: 206'2072, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:3383464b:::benchmark_data_smithi159_91621_object33181:head@206'2072, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:14.089471+0000", 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.112830623, 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.069346249999999998, 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:14.282 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.089471+0000", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.089471+0000", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.089480+0000", 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9439999999999994e-06 2024-04-15T18:18:14.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.089545+0000", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5848999999999999e-05 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.156001+0000", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.066455705000000004 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.156019+0000", 2024-04-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7914000000000002e-05 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.156276+0000", 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025648100000000003 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.156291+0000", 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5131e-05 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.158817+0000", 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025262259999999999 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:139653 3.c e247/235 3:3768a391:::benchmark_data_smithi159_91621_object33177:head v 247'8597, mlcod=206'2069)", 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:14.132622+0000", 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.069679218000000001, 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.057667936000000003, 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132622+0000", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132622+0000", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132623+0000", 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.122e-06 2024-04-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.132633+0000", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9359999999999994e-06 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.157591+0000", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.024957938999999998 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.157600+0000", 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-04-15T18:18:14.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.158862+0000", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012612000000000001 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.158887+0000", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5316e-05 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.190244+0000", 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031357161000000001 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:14.190290+0000", 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5892e-05 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:18:14.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:14.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:14.576 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:14.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:14.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:14.724 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:14.759 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:14.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:14.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:14.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:14 smithi049 ceph-mon[40087]: pgmap v689: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1821/226239 objects degraded (0.805%); 23064/226239 objects misplaced (10.195%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:14.894 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:14.913 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:14.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:15.016 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:18:15.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:14 smithi082 ceph-mon[44206]: pgmap v689: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1821/226239 objects degraded (0.805%); 23064/226239 objects misplaced (10.195%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:15.043 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:15.062 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:18:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:15.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:18:15.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:14 smithi159 ceph-mon[45336]: pgmap v689: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1821/226239 objects degraded (0.805%); 23064/226239 objects misplaced (10.195%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:15.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:15.204 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:15.232 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:15.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:15.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:15.395 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:15.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:18:15.619 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:15.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:15.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:15.791 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:16.898 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-04-15T18:18:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:16 smithi159 ceph-mon[45336]: pgmap v690: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1956/222258 objects degraded (0.880%); 22507/222258 objects misplaced (10.127%); 9.1 MiB/s, 146 objects/s recovering 2024-04-15T18:18:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/140364555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.158 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:16 smithi049 ceph-mon[40087]: pgmap v690: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1956/222258 objects degraded (0.880%); 22507/222258 objects misplaced (10.127%); 9.1 MiB/s, 146 objects/s recovering 2024-04-15T18:18:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/140364555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:16 smithi082 ceph-mon[44206]: pgmap v690: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1956/222258 objects degraded (0.880%); 22507/222258 objects misplaced (10.127%); 9.1 MiB/s, 146 objects/s recovering 2024-04-15T18:18:17.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/140364555' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:18:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:17.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:17.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:18:17.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:17.727 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:18.004 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:18.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:18.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 2 1 2024-04-15T18:18:18.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:18.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:17 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:18:18.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:17 smithi049 ceph-mon[40087]: osdmap e248: 12 total, 12 up, 12 in 2024-04-15T18:18:18.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1956/222258 objects degraded (0.880%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:18.221 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:18.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:17 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:18:18.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:17 smithi082 ceph-mon[44206]: osdmap e248: 12 total, 12 up, 12 in 2024-04-15T18:18:18.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1956/222258 objects degraded (0.880%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:18.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:17 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:18:18.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:17 smithi159 ceph-mon[45336]: osdmap e248: 12 total, 12 up, 12 in 2024-04-15T18:18:18.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1956/222258 objects degraded (0.880%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:18.586 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:18.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:18.666 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:18.695 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:18.736 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:18.745 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:18.767 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:18.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:18.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:18.921 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:18.961 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:18.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: pgmap v692: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 2068/218148 objects degraded (0.948%); 22106/218148 objects misplaced (10.133%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: 3.9 scrub starts 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: 3.9 scrub ok 2024-04-15T18:18:19.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:18 smithi082 ceph-mon[44206]: osdmap e249: 12 total, 12 up, 12 in 2024-04-15T18:18:19.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:19.150 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:19.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: pgmap v692: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 2068/218148 objects degraded (0.948%); 22106/218148 objects misplaced (10.133%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:18:19.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: 3.9 scrub starts 2024-04-15T18:18:19.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:19.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:19.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: 3.9 scrub ok 2024-04-15T18:18:19.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:18 smithi049 ceph-mon[40087]: osdmap e249: 12 total, 12 up, 12 in 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: pgmap v692: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 2068/218148 objects degraded (0.948%); 22106/218148 objects misplaced (10.133%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: 3.9 scrub starts 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: 3.9 scrub ok 2024-04-15T18:18:19.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:18 smithi159 ceph-mon[45336]: osdmap e249: 12 total, 12 up, 12 in 2024-04-15T18:18:19.934 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:20.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:20.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:20.289 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:20.309 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:20.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:20.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:20.482 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:18:21.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:20 smithi049 ceph-mon[40087]: pgmap v694: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2068/217572 objects degraded (0.950%); 22106/217572 objects misplaced (10.160%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:18:21.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2703780662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:21.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:21.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:20 smithi082 ceph-mon[44206]: pgmap v694: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2068/217572 objects degraded (0.950%); 22106/217572 objects misplaced (10.160%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:18:21.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2703780662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:21.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:21.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:20 smithi159 ceph-mon[45336]: pgmap v694: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 405 op/s; 2068/217572 objects degraded (0.950%); 22106/217572 objects misplaced (10.160%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:18:21.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2703780662' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:21.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:18:22.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:21 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:22.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:18:22.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:21 smithi049 ceph-mon[40087]: osdmap e250: 12 total, 12 up, 12 in 2024-04-15T18:18:22.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:21 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:22.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:18:22.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:21 smithi082 ceph-mon[44206]: osdmap e250: 12 total, 12 up, 12 in 2024-04-15T18:18:22.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:21 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:22.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:18:22.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:21 smithi159 ceph-mon[45336]: osdmap e250: 12 total, 12 up, 12 in 2024-04-15T18:18:23.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:22 smithi049 ceph-mon[40087]: pgmap v696: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2269/210084 objects degraded (1.080%); 21147/210084 objects misplaced (10.066%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:18:23.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2269/210084 objects degraded (1.080%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:23.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:22 smithi082 ceph-mon[44206]: pgmap v696: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2269/210084 objects degraded (1.080%); 21147/210084 objects misplaced (10.066%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:18:23.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2269/210084 objects degraded (1.080%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:23.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:22 smithi159 ceph-mon[45336]: pgmap v696: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 2269/210084 objects degraded (1.080%); 21147/210084 objects misplaced (10.066%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:18:23.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2269/210084 objects degraded (1.080%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:24.653 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:24.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:24.762 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:24.793 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:24.815 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:24.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:24.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:24.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:24 smithi049 ceph-mon[40087]: pgmap v697: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2269/208851 objects degraded (1.086%); 21147/208851 objects misplaced (10.125%); 7.5 MiB/s, 121 objects/s recovering 2024-04-15T18:18:24.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2925693965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:24.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:18:24.994 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.043 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:18:25.081 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:18:25.081 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:18:25.083 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:25.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:18:25.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:18:25.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:24 smithi159 ceph-mon[45336]: pgmap v697: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2269/208851 objects degraded (1.086%); 21147/208851 objects misplaced (10.125%); 7.5 MiB/s, 121 objects/s recovering 2024-04-15T18:18:25.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2925693965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:25.186 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:18:25.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:25.224 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:18:25.226 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:24 smithi082 ceph-mon[44206]: pgmap v697: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2269/208851 objects degraded (1.086%); 21147/208851 objects misplaced (10.125%); 7.5 MiB/s, 121 objects/s recovering 2024-04-15T18:18:25.226 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2925693965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:25.226 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.251 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:25.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:18:25.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:18:25.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:25.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:18:25.390 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.414 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:25.422 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:18:25.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:18:25.508 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:18:25.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:18:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:18:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:18:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:18:25.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:18:25.519 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:18:25.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:18:25.554 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.577 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:25.686 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:25.717 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.735 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:25.844 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:25.882 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:25.914 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:25.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:26.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:26.073 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:26.104 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:26.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:26.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:18:26.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:26.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.272 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.272 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.272 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.272 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.272 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.276 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:26.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.290 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.406 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:26.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:26.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:26.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:26.543 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:26.562 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:26.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:26.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:26.716 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:27.184 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:27.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:26 smithi082 ceph-mon[44206]: pgmap v698: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 2332/205071 objects degraded (1.137%); 20597/205071 objects misplaced (10.044%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:18:27.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:27.379 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:27.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:27.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:26 smithi159 ceph-mon[45336]: pgmap v698: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 2332/205071 objects degraded (1.137%); 20597/205071 objects misplaced (10.044%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:18:27.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:26 smithi049 ceph-mon[40087]: pgmap v698: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 544 op/s; 2332/205071 objects degraded (1.137%); 20597/205071 objects misplaced (10.044%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:18:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:27.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:27.552 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:27.587 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:27.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:27.663 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:27.674 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:27.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:27.720 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:27.739 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:27.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:27.846 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:27.895 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:27.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:27.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:28.055 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:28.252 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2332/205071 objects degraded (1.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:28.252 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:28.252 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:28.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2332/205071 objects degraded (1.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:28.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:28.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:28.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2332/205071 objects degraded (1.137%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:18:28.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:28.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:29.136 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:29.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:29.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:18:29.252 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:29.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:28 smithi082 ceph-mon[44206]: pgmap v699: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2452/200766 objects degraded (1.221%); 20164/200766 objects misplaced (10.044%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:29.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:28 smithi082 ceph-mon[44206]: osdmap e251: 12 total, 12 up, 12 in 2024-04-15T18:18:29.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:28 smithi049 ceph-mon[40087]: pgmap v699: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2452/200766 objects degraded (1.221%); 20164/200766 objects misplaced (10.044%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:29.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:28 smithi049 ceph-mon[40087]: osdmap e251: 12 total, 12 up, 12 in 2024-04-15T18:18:29.297 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:29.327 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:29.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:29.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:28 smithi159 ceph-mon[45336]: pgmap v699: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2452/200766 objects degraded (1.221%); 20164/200766 objects misplaced (10.044%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:29.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:28 smithi159 ceph-mon[45336]: osdmap e251: 12 total, 12 up, 12 in 2024-04-15T18:18:29.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:29.499 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:31.139 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:18:31.140 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:18:31.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:31 smithi159 ceph-mon[45336]: pgmap v701: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2452/200166 objects degraded (1.225%); 20164/200166 objects misplaced (10.074%); 6.1 MiB/s, 97 objects/s recovering 2024-04-15T18:18:31.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:31.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:31.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:31 smithi049 ceph-mon[40087]: pgmap v701: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2452/200166 objects degraded (1.225%); 20164/200166 objects misplaced (10.074%); 6.1 MiB/s, 97 objects/s recovering 2024-04-15T18:18:31.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:31.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:31 smithi082 ceph-mon[44206]: pgmap v701: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2452/200166 objects degraded (1.225%); 20164/200166 objects misplaced (10.074%); 6.1 MiB/s, 97 objects/s recovering 2024-04-15T18:18:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:31.612 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:31.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:31.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:31.818 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:31.885 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:31.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:31.993 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:32.028 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.051 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:32.105 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-04-15T18:18:32.112 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:18:32.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:32 smithi049 ceph-mon[40087]: osdmap e252: 12 total, 12 up, 12 in 2024-04-15T18:18:32.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:32 smithi049 ceph-mon[40087]: pgmap v703: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2649/192558 objects degraded (1.376%); 19193/192558 objects misplaced (9.967%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:18:32.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1429409497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:32.157 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:32.183 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.207 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:32.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:32.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:32 smithi159 ceph-mon[45336]: osdmap e252: 12 total, 12 up, 12 in 2024-04-15T18:18:32.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:32 smithi159 ceph-mon[45336]: pgmap v703: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2649/192558 objects degraded (1.376%); 19193/192558 objects misplaced (9.967%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:18:32.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1429409497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:32.341 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.363 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:32.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:32.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:32 smithi082 ceph-mon[44206]: osdmap e252: 12 total, 12 up, 12 in 2024-04-15T18:18:32.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:32 smithi082 ceph-mon[44206]: pgmap v703: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 2649/192558 objects degraded (1.376%); 19193/192558 objects misplaced (9.967%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:18:32.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1429409497' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:18:32.506 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:32.508 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:32.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:32.697 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.726 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:32.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:32.856 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:32.879 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:32.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:32.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:33.023 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:33.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:33.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:33.156 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:33 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:33.156 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:18:33.156 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:33.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:33 smithi159 ceph-mon[45336]: osdmap e253: 12 total, 12 up, 12 in 2024-04-15T18:18:33.157 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:33 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2649/192558 objects degraded (1.376%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:33.180 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:33.201 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:33.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:33.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:33 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:33.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:33 smithi082 ceph-mon[44206]: osdmap e253: 12 total, 12 up, 12 in 2024-04-15T18:18:33.289 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:33 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2649/192558 objects degraded (1.376%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:33.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:33.349 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:33.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:33.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:33 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:33.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:18:33.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4065579033' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:18:33.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:33 smithi049 ceph-mon[40087]: osdmap e253: 12 total, 12 up, 12 in 2024-04-15T18:18:33.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:33 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2649/192558 objects degraded (1.376%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:33.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:33.572 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:34.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:34 smithi049 ceph-mon[40087]: pgmap v705: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 2649/191424 objects degraded (1.384%); 19193/191424 objects misplaced (10.026%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:34.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:34 smithi159 ceph-mon[45336]: pgmap v705: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 2649/191424 objects degraded (1.384%); 19193/191424 objects misplaced (10.026%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:34.544 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:34 smithi082 ceph-mon[44206]: pgmap v705: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 2649/191424 objects degraded (1.384%); 19193/191424 objects misplaced (10.026%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:18:34.875 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:34.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 4 1 2024-04-15T18:18:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:34.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:34.981 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:35.015 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.043 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:35.154 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:35.182 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.204 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:35.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:35.374 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.465 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:35.571 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:18:35.599 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.624 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:18:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:18:35.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:35.755 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.778 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:35.861 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:35.867 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:35.884 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:35.894 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:18:35.913 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:35.932 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:35.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:35.939 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:18:35.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:18:36.025 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:18:36.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:36.062 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:36.062 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:18:36.080 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:36.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:36.100 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:18:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:18:36.177 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:18:36.184 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:36.217 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:36.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:18:36.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:36.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:36.264 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:18:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:18:36.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:36.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:18:36.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:18:36.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:18:36.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:36 smithi049 ceph-mon[40087]: pgmap v706: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2709/187614 objects degraded (1.444%); 18671/187614 objects misplaced (9.952%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:18:36.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/161795093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:36.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:36.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:36.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:36.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:36 smithi082 ceph-mon[44206]: pgmap v706: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2709/187614 objects degraded (1.444%); 18671/187614 objects misplaced (9.952%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:18:36.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/161795093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:36.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:36.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:36.645 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:36.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:36 smithi159 ceph-mon[45336]: pgmap v706: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2709/187614 objects degraded (1.444%); 18671/187614 objects misplaced (9.952%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:18:36.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/161795093' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:36.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:36.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:36.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:18:36.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:36.863 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:18:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:36.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:18:36.967 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:36.995 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:37.015 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:37.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:37.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:18:37.154 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:37.173 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:18:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:37.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:18:37.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:18:37.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:37.347 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:37.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.407 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2709/187614 objects degraded (1.444%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2709/187614 objects degraded (1.444%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:37.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2709/187614 objects degraded (1.444%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:38.135 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:38.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:38.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:38.318 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:38.459 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:38.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:38 smithi049 ceph-mon[40087]: pgmap v707: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 2820/183486 objects degraded (1.537%); 18265/183486 objects misplaced (9.954%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:18:38.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:38.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:38 smithi082 ceph-mon[44206]: pgmap v707: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 2820/183486 objects degraded (1.537%); 18265/183486 objects misplaced (9.954%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:18:38.567 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:38.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:38 smithi159 ceph-mon[45336]: pgmap v707: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 694 op/s; 2820/183486 objects degraded (1.537%); 18265/183486 objects misplaced (9.954%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:18:38.595 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:38.618 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:38.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:38.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:38.779 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:38.819 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:38.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:38.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:38.957 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:38.978 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:38.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:39.082 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:39.123 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:39.162 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:39.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:39.271 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:39.339 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:39.363 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:39.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:39.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:39.502 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:39.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:39.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:39.677 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:40.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:18:40.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:40 smithi082 ceph-mon[44206]: pgmap v708: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 441 op/s; 2820/182940 objects degraded (1.541%); 18265/182940 objects misplaced (9.984%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:18:40.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:40 smithi159 ceph-mon[45336]: pgmap v708: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 441 op/s; 2820/182940 objects degraded (1.541%); 18265/182940 objects misplaced (9.984%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:18:40.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:40 smithi049 ceph-mon[40087]: pgmap v708: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 441 op/s; 2820/182940 objects degraded (1.541%); 18265/182940 objects misplaced (9.984%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:18:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:42 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:42 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:42 smithi082 ceph-mon[44206]: pgmap v709: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 3016/175263 objects degraded (1.721%); 17313/175263 objects misplaced (9.878%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:42 smithi082 ceph-mon[44206]: osdmap e254: 12 total, 12 up, 12 in 2024-04-15T18:18:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3016/175263 objects degraded (1.721%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:42.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:42 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:42.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:42 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:42.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:42 smithi159 ceph-mon[45336]: pgmap v709: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 3016/175263 objects degraded (1.721%); 17313/175263 objects misplaced (9.878%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:42.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:42 smithi159 ceph-mon[45336]: osdmap e254: 12 total, 12 up, 12 in 2024-04-15T18:18:42.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3016/175263 objects degraded (1.721%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:42.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:42 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:42.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:42 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:42.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:42 smithi049 ceph-mon[40087]: pgmap v709: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 3016/175263 objects degraded (1.721%); 17313/175263 objects misplaced (9.878%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:42.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:42 smithi049 ceph-mon[40087]: osdmap e254: 12 total, 12 up, 12 in 2024-04-15T18:18:42.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3016/175263 objects degraded (1.721%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:43.359 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:43.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:43.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:43.571 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:43.923 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:62472 2.1 e143/114 2:81d12c07:::benchmark_data_smithi159_80260_object62471:head v 143'3896, mlcod=143'3893)", 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 368.00914412600002, 2024-04-15T18:18:43.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38137517700000001, 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929921+0000", 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929922+0000", 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3099999999999997e-07 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:35.929945+0000", 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3850999999999999e-05 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308461+0000", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37851550900000003 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308470+0000", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7199999999999995e-06 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308524+0000", 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4682999999999998e-05 2024-04-15T18:18:43.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.308547+0000", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447999999999999e-05 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.311264+0000", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027170699999999998 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:12:36.311296+0000", 2024-04-15T18:18:43.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2165000000000001e-05 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:97475 2.5 e159/114 2:ad8d4387:::benchmark_data_smithi159_80260_object8362:head v 159'6011, mlcod=159'6005)", 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 317.82587092099999, 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25188648699999999, 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.948 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113194+0000", 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113195+0000", 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7100000000000002e-07 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.113200+0000", 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9060000000000001e-06 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357485+0000", 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24428497299999999 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357492+0000", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8350000000000004e-06 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357561+0000", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8725000000000004e-05 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.357580+0000", 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9048000000000001e-05 2024-04-15T18:18:43.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365036+0000", 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0074560140000000004 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:13:26.365081+0000", 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4415000000000001e-05 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.952 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24511.0:133604 2.c e172/114 2:313518bc:::benchmark_data_smithi159_80260_object44491:head v 172'8414, mlcod=172'8411)", 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 278.68544906300002, 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23120621699999999, 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253616+0000", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253617+0000", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.253623+0000", 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5470000000000002e-06 2024-04-15T18:18:43.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477897+0000", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.224274311 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477906+0000", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3059999999999998e-06 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477964+0000", 2024-04-15T18:18:43.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7972e-05 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.477989+0000", 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5466e-05 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.484780+0000", 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067909049999999999 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:14:05.484822+0000", 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2033999999999998e-05 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:82171 3.b 3:dbe4dd0b:::benchmark_data_smithi159_91621_object82170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:16:50.926191+0000", 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 113.012873911, 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27445134799999998, 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.957 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82171 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926191+0000", 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926191+0000", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926192+0000", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5300000000000002e-07 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926273+0000", 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1321000000000003e-05 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926282+0000", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7690000000000002e-06 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.926287+0000", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2159999999999997e-06 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.963576+0000", 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.037288494999999998 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.963656+0000", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9876999999999994e-05 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.963740+0000", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4332000000000005e-05 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.965541+0000", 2024-04-15T18:18:43.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018010789999999999 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.977287+0000", 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011746097 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200519+0000", 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223231293 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200584+0000", 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5256e-05 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200642+0000", 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8659999999999997e-05 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:82198 3.b 3:d5dafa1b:::benchmark_data_smithi159_91621_object82197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:16:50.955571+0000", 2024-04-15T18:18:43.963 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.983494377, 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.245181132, 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82198 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955571+0000", 2024-04-15T18:18:43.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955571+0000", 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955572+0000", 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1969999999999999e-06 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955623+0000", 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1347999999999997e-05 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955634+0000", 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.076e-05 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.955639+0000", 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1019999999999996e-06 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969665+0000", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014025908 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969811+0000", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014575499999999999 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969983+0000", 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017196600000000001 2024-04-15T18:18:43.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.977296+0000", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073130269999999997 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.979712+0000", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024166169999999998 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200682+0000", 2024-04-15T18:18:43.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22096992300000001 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200718+0000", 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5589000000000001e-05 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200752+0000", 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3939999999999997e-05 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:82212 3.b 3:da1dccbd:::benchmark_data_smithi159_91621_object82211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:16:50.969317+0000", 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.969747701, 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.231607644, 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82212 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.970 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969317+0000", 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969317+0000", 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969318+0000", 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899999999999999e-06 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969397+0000", 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8584999999999994e-05 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969405+0000", 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1999999999999994e-06 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969410+0000", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8810000000000004e-06 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.972539+0000", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031286069999999998 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.972614+0000", 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4950000000000006e-05 2024-04-15T18:18:43.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.972703+0000", 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9458999999999998e-05 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.977430+0000", 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004726788 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.981580+0000", 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041500929999999997 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200810+0000", 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.219230016 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200870+0000", 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9908000000000003e-05 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200925+0000", 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5067e-05 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.976 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:82213 3.b 3:dfc2af0b:::benchmark_data_smithi159_91621_object82212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:16:50.969579+0000", 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.96948622799999, 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23145453299999999, 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.977 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82213 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969579+0000", 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969579+0000", 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969579+0000", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3e-07 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969683+0000", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104113 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969689+0000", 2024-04-15T18:18:43.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2660000000000001e-06 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.969693+0000", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8229999999999997e-06 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.974897+0000", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052042989999999999 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.975022+0000", 2024-04-15T18:18:43.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012513699999999999 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.975152+0000", 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129531 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.977434+0000", 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022824379999999999 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988857+0000", 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011422748999999999 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.200972+0000", 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.212115674 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.201002+0000", 2024-04-15T18:18:43.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0018999999999998e-05 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.201033+0000", 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0854e-05 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.983 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:82235 3.6 3:6d7e39e7:::benchmark_data_smithi159_91621_object82234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:16:50.988034+0000", 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 112.951030972, 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22471576300000001, 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82235 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988034+0000", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988034+0000", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988035+0000", 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8700000000000005e-07 2024-04-15T18:18:43.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988109+0000", 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4196000000000006e-05 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988118+0000", 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4250000000000004e-06 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988124+0000", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3820000000000003e-06 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988165+0000", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1752999999999998e-05 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988248+0000", 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2906999999999998e-05 2024-04-15T18:18:43.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.988331+0000", 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2327e-05 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:50.990375+0000", 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020448060000000001 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.062111+0000", 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.071735807999999998 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.212648+0000", 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15053687499999999 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.212706+0000", 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7522999999999998e-05 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:16:51.212750+0000", 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4073999999999998e-05 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113542 3.7 3:e62b2763:::benchmark_data_smithi159_91621_object7066:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127571+0000", 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811493713999994, 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31608156700000001, 2024-04-15T18:18:43.990 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113542 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127571+0000", 2024-04-15T18:18:43.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127571+0000", 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127574+0000", 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.351e-06 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127586+0000", 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2303999999999999e-05 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127616+0000", 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0328000000000002e-05 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127631+0000", 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5089000000000001e-05 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128200+0000", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056883400000000005 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128212+0000", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2131e-05 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.443479+0000", 2024-04-15T18:18:43.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31526685599999998 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.443653+0000", 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017367399999999999 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113555 3.7 3:e70e233f:::benchmark_data_smithi159_91621_object7079:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127660+0000", 2024-04-15T18:18:43.995 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811405162, 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.352602215, 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113555 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127660+0000", 2024-04-15T18:18:43.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127660+0000", 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127660+0000", 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8400000000000005e-07 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:43.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127664+0000", 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2119999999999999e-06 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127671+0000", 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7479999999999993e-06 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127678+0000", 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.579e-06 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128239+0000", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056160599999999998 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128247+0000", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3150000000000003e-06 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.447813+0000", 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.319565871 2024-04-15T18:18:43.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.449615+0000", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018018870000000001 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7,11", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.449799+0000", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000184866 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.464186+0000", 2024-04-15T18:18:44.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014386265000000001 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.464354+0000", 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000168429 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.466781+0000", 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002427022 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.477354+0000", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010572972999999999 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.477403+0000", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9128999999999997e-05 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.480262+0000", 2024-04-15T18:18:44.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028588289999999998 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113563 3.7 3:e28b2fcb:::benchmark_data_smithi159_91621_object7087:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127686+0000", 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811378934999993, 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.355913745, 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113563 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127686+0000", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127686+0000", 2024-04-15T18:18:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127686+0000", 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0900000000000002e-07 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127690+0000", 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.05e-06 2024-04-15T18:18:44.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127696+0000", 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8730000000000001e-06 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127702+0000", 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6699999999999999e-06 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128271+0000", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056876599999999995 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128278+0000", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4529999999999996e-06 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.453232+0000", 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32495357800000002 2024-04-15T18:18:44.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.453378+0000", 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014600600000000001 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7,11", 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.453558+0000", 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000180302 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.464306+0000", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010747447 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.466840+0000", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025344159999999998 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.466858+0000", 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8430000000000001e-05 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.480338+0000", 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013479391 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.480408+0000", 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0136999999999996e-05 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.483600+0000", 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003191817 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113544 3.b 3:d3047f12:::benchmark_data_smithi159_91621_object7068:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127713+0000", 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811351783000006, 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38852929600000002, 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:18:44.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113544 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127713+0000", 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127713+0000", 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127714+0000", 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1800000000000001e-07 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127716+0000", 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8480000000000001e-06 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127723+0000", 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5579999999999997e-06 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127729+0000", 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8640000000000001e-06 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128298+0000", 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056954200000000003 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128308+0000", 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.859e-06 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.516069+0000", 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38776102800000001 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.516242+0000", 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173179 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113577 3.b 3:d38c20dd:::benchmark_data_smithi159_91621_object7101:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127739+0000", 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811325529000001, 2024-04-15T18:18:44.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42167964299999999, 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113577 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127739+0000", 2024-04-15T18:18:44.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127739+0000", 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127740+0000", 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6300000000000001e-07 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127742+0000", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4930000000000001e-06 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127749+0000", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2369999999999997e-06 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127754+0000", 2024-04-15T18:18:44.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9179999999999999e-06 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128332+0000", 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057743599999999996 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128340+0000", 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.977e-06 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.519172+0000", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39083242200000001 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.519286+0000", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011418599999999999 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.519388+0000", 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101957 2024-04-15T18:18:44.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.532462+0000", 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013073408999999999 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.540709+0000", 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0082468469999999999 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.545705+0000", 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004996304 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.545786+0000", 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0881e-05 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.549419+0000", 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036332130000000002 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.023 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113579 3.b 3:da936e3a:::benchmark_data_smithi159_91621_object7103:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127761+0000", 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811304047999997, 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.437943217, 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.024 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113579 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127761+0000", 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127761+0000", 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127761+0000", 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6300000000000001e-07 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127763+0000", 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1339999999999998e-06 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127785+0000", 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2025e-05 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127791+0000", 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5940000000000001e-06 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128361+0000", 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057043999999999997 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128372+0000", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0387999999999999e-05 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.523784+0000", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39541216200000001 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.523974+0000", 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019021199999999999 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.524092+0000", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117664 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.532511+0000", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084195380000000007 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.540723+0000", 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.008211309 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.561927+0000", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.02120414 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.561994+0000", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7539999999999994e-05 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.565704+0000", 2024-04-15T18:18:44.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037097079999999999 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:113583 3.b 3:d145b71a:::benchmark_data_smithi159_91621_object7107:head [delete] snapc 0=[] ondisk+write+known_if_redirected e236)", 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:28.127799+0000", 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 75.811265664999993, 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.44113616700000002, 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.031 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113583 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127799+0000", 2024-04-15T18:18:44.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127799+0000", 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127800+0000", 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5400000000000002e-07 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127803+0000", 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8890000000000002e-06 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127809+0000", 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.742e-06 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.127815+0000", 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128395+0000", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057961999999999998 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.128401+0000", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1850000000000001e-06 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.527963+0000", 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.399561746 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.528152+0000", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018911199999999999 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.528343+0000", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019124800000000001 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.540796+0000", 2024-04-15T18:18:44.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012452758 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.540817+0000", 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1605999999999999e-05 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.565795+0000", 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.024977840000000001 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.565864+0000", 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9000999999999999e-05 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:28.568935+0000", 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030712740000000001 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:121154 3.1 3:822bad97:::benchmark_data_smithi159_91621_object14678:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:18:44.039 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:42.132251+0000", 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 61.806814197000001, 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227204562, 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121154 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132251+0000", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132251+0000", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132252+0000", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.499e-06 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132258+0000", 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132785+0000", 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052673399999999995 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132796+0000", 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0706999999999999e-05 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132857+0000", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0863000000000003e-05 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.132973+0000", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116611 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,10", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133082+0000", 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109079 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.140286+0000", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072034880000000001 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.140946+0000", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065968699999999995 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.359369+0000", 2024-04-15T18:18:44.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.218423067 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.359417+0000", 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8644999999999999e-05 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.359455+0000", 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8002e-05 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:121157 3.6 3:6ce65175:::benchmark_data_smithi159_91621_object14681:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:17:42.133533+0000", 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 61.805532231999997, 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22477074599999999, 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.046 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 121157 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133533+0000", 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133533+0000", 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133533+0000", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9500000000000002e-07 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133537+0000", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1149999999999998e-06 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133546+0000", 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1230000000000006e-06 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133552+0000", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1059999999999996e-06 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133609+0000", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7145000000000003e-05 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133812+0000", 2024-04-15T18:18:44.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000203418 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.133930+0000", 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117287 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.140450+0000", 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0065201749999999996 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.141680+0000", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012299240000000001 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.358171+0000", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21649170000000001 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.358242+0000", 2024-04-15T18:18:44.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0871000000000006e-05 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:17:42.358303+0000", 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1186999999999995e-05 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:152882 3.7 3:e845a6da:::benchmark_data_smithi159_91621_object46406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e253)", 2024-04-15T18:18:44.052 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:37.146589+0000", 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.7924756080000002, 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27446953200000002, 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 152882 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146589+0000", 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146589+0000", 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146591+0000", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7439999999999999e-06 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146600+0000", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9940000000000006e-06 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146613+0000", 2024-04-15T18:18:44.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3030000000000001e-05 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.146624+0000", 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0823e-05 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.152040+0000", 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054160470000000002 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for rw locks", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.152138+0000", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7730000000000001e-05 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.181926+0000", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.029788041000000001 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.182030+0000", 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104095 2024-04-15T18:18:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7,11", 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.182167+0000", 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000137479 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.196765+0000", 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014597924 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.196782+0000", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6759999999999999e-05 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.200050+0000", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032679649999999998 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.418174+0000", 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21812383499999999 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.418241+0000", 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7398999999999996e-05 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.421059+0000", 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028176659999999999 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.060 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.24793.0:152882 3.7 e253/235 ondisk, result = 0)", 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:37.189575+0000", 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.7494900619999996, 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23149403800000001, 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.189575+0000", 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.189575+0000", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.189576+0000", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.189584+0000", 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2279999999999998e-06 2024-04-15T18:18:44.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.412671+0000", 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223087391 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.412679+0000", 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2409999999999998e-06 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.418160+0000", 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054814089999999996 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.418167+0000", 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2439999999999996e-06 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:37.421069+0000", 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029016189999999998 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24793.0:155004 3.7 3:e8db7b87:::benchmark_data_smithi159_91621_object48528:head [delete] snapc 0=[] ondisk+write+known_if_redirected e253)", 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:40.782256+0000", 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.1568085539999999, 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24682638400000001, 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:18:44.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24793", 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 155004 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782256+0000", 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:18:44.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782256+0000", 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782258+0000", 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.671e-06 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782265+0000", 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0709999999999997e-06 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782280+0000", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4524e-05 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.782288+0000", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4330000000000004e-06 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.793515+0000", 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011226806000000001 2024-04-15T18:18:44.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for rw locks", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.793630+0000", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115365 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.856711+0000", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.063080910000000004 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.856800+0000", 2024-04-15T18:18:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8962999999999995e-05 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7,11", 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.856892+0000", 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2093000000000001e-05 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.869789+0000", 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012896253999999999 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.874388+0000", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045996789999999997 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:40.890596+0000", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016207433 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:41.026157+0000", 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13556146599999999 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:41.026229+0000", 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2317000000000006e-05 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:41.029083+0000", 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002853399 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:18:44.072 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:18:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:44.074 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:44.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:44.075 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:44.175 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:18:44.206 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:44.232 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:18:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:44.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:18:44.273 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-04-15T18:18:44.338 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:18:44.366 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:44.390 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:18:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:44.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:18:44.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:44.521 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:44 smithi049 ceph-mon[40087]: pgmap v711: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3016/174261 objects degraded (1.731%); 17313/174261 objects misplaced (9.935%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:44.521 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:18:44.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:44 smithi082 ceph-mon[44206]: pgmap v711: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3016/174261 objects degraded (1.731%); 17313/174261 objects misplaced (9.935%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:44.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:18:44.540 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:44 smithi159 ceph-mon[45336]: pgmap v711: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3016/174261 objects degraded (1.731%); 17313/174261 objects misplaced (9.935%); 9.4 MiB/s, 151 objects/s recovering 2024-04-15T18:18:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:18:44.802 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:44.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:44.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:44.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:44.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:44.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:45.065 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:45.099 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:45.131 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:45.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:45.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:45.284 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:45.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:45.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 5 1 2024-04-15T18:18:45.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:45.508 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:45.509 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:18:45.509 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:45 smithi082 ceph-mon[44206]: osdmap e255: 12 total, 12 up, 12 in 2024-04-15T18:18:45.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:45.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:45.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:45.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:18:45.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:45 smithi159 ceph-mon[45336]: osdmap e255: 12 total, 12 up, 12 in 2024-04-15T18:18:45.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2302614896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:18:45.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:45 smithi049 ceph-mon[40087]: osdmap e255: 12 total, 12 up, 12 in 2024-04-15T18:18:45.710 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:46.123 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:46.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:46.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:46.256 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:46.277 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:46.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:46.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:46 smithi049 ceph-mon[40087]: pgmap v713: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3042/170610 objects degraded (1.783%); 16741/170610 objects misplaced (9.812%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:46.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:46.431 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:46.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:46 smithi082 ceph-mon[44206]: pgmap v713: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3042/170610 objects degraded (1.783%); 16741/170610 objects misplaced (9.812%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:46.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:46 smithi159 ceph-mon[45336]: pgmap v713: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3042/170610 objects degraded (1.783%); 16741/170610 objects misplaced (9.812%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:18:47.542 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3042/170610 objects degraded (1.783%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:47.590 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:47.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3042/170610 objects degraded (1.783%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:47.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:47.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3042/170610 objects degraded (1.783%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:47.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:47.732 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:47.758 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:47.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:47.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:47.900 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:47.933 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:47.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:48.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:48.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:18:48.123 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:48.141 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:48.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:48.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:48.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:48 smithi049 ceph-mon[40087]: pgmap v714: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 3105/166374 objects degraded (1.866%); 16398/166374 objects misplaced (9.856%); 12 MiB/s, 197 objects/s recovering 2024-04-15T18:18:48.293 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:48.355 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:48.437 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:48.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:48.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:48.630 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:48.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:48 smithi159 ceph-mon[45336]: pgmap v714: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 3105/166374 objects degraded (1.866%); 16398/166374 objects misplaced (9.856%); 12 MiB/s, 197 objects/s recovering 2024-04-15T18:18:48.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:48 smithi082 ceph-mon[44206]: pgmap v714: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 3105/166374 objects degraded (1.866%); 16398/166374 objects misplaced (9.856%); 12 MiB/s, 197 objects/s recovering 2024-04-15T18:18:49.001 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:49.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:49.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:49.172 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:49.202 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:49.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:49.218 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:18:49.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:18:49.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:18:49.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:49.334 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:49.344 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:18:49.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:18:49.356 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:49.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:49.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:18:49.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:18:49.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:49.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:18:49.523 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:49.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:18:49.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3072296506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:49.554 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:18:49.597 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:18:49.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:18:49.634 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:49.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:49.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:18:49.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3072296506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:49.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:18:49.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3072296506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:49.677 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:18:49.704 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:18:49.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:18:49.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:18:49.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:49.886 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:49.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:49.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:50.095 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:50.095 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:50.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:50.207 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:18:50.238 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:50.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:18:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:50.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:18:50.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:50.421 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: pgmap v715: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 3105/165903 objects degraded (1.872%); 16398/165903 objects misplaced (9.884%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osdmap e256: 12 total, 12 up, 12 in 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:50.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:50.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: pgmap v715: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 3105/165903 objects degraded (1.872%); 16398/165903 objects misplaced (9.884%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osdmap e256: 12 total, 12 up, 12 in 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: pgmap v715: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 389 op/s; 3105/165903 objects degraded (1.872%); 16398/165903 objects misplaced (9.884%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:18:50.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2454222665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osdmap e256: 12 total, 12 up, 12 in 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:50.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.301 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:51.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:51.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:51.463 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:51.504 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:51.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:18:51.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.677 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:51.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:51.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:52.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:52 smithi049 ceph-mon[40087]: pgmap v717: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 3151/158583 objects degraded (1.987%); 15480/158583 objects misplaced (9.761%); 13 MiB/s, 212 objects/s recovering 2024-04-15T18:18:52.310 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3151/158583 objects degraded (1.987%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:52.649 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:18:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:52.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:18:52.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:52 smithi159 ceph-mon[45336]: pgmap v717: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 3151/158583 objects degraded (1.987%); 15480/158583 objects misplaced (9.761%); 13 MiB/s, 212 objects/s recovering 2024-04-15T18:18:52.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3151/158583 objects degraded (1.987%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:52.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:18:52.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:52.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:52 smithi082 ceph-mon[44206]: pgmap v717: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 3151/158583 objects degraded (1.987%); 15480/158583 objects misplaced (9.761%); 13 MiB/s, 212 objects/s recovering 2024-04-15T18:18:52.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3151/158583 objects degraded (1.987%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:52.825 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:52.893 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:52.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:53.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:18:53.054 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:53.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:18:53.528 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:18:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:53.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:18:53.611 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:53.633 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:53.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:53.676 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:53.697 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:53.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:53.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:53.857 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:54.364 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-04-15T18:18:54.483 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:18:54.483 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:18:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:54 smithi159 ceph-mon[45336]: pgmap v718: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 3151/157569 objects degraded (2.000%); 15480/157569 objects misplaced (9.824%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:18:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/540948347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:54 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:54.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:54 smithi049 ceph-mon[40087]: pgmap v718: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 3151/157569 objects degraded (2.000%); 15480/157569 objects misplaced (9.824%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:18:54.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/540948347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:54.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:54 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:54.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:54 smithi082 ceph-mon[44206]: pgmap v718: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 3151/157569 objects degraded (2.000%); 15480/157569 objects misplaced (9.824%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:18:54.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/540948347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:54.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:54 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:18:55.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:55 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:18:55.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:55 smithi159 ceph-mon[45336]: osdmap e257: 12 total, 12 up, 12 in 2024-04-15T18:18:55.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:55 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:18:55.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:55 smithi049 ceph-mon[40087]: osdmap e257: 12 total, 12 up, 12 in 2024-04-15T18:18:55.794 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:55 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:18:55.795 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:55 smithi082 ceph-mon[44206]: osdmap e257: 12 total, 12 up, 12 in 2024-04-15T18:18:55.813 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:55.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:55.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:55.966 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:56.337 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:18:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:56.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:18:56.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:18:56.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:56.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:56.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:56 smithi159 ceph-mon[45336]: pgmap v720: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 3154/153831 objects degraded (2.050%); 14965/153831 objects misplaced (9.728%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:56.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:56 smithi049 ceph-mon[40087]: pgmap v720: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 3154/153831 objects degraded (2.050%); 14965/153831 objects misplaced (9.728%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:56.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:56 smithi082 ceph-mon[44206]: pgmap v720: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 3154/153831 objects degraded (2.050%); 14965/153831 objects misplaced (9.728%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:18:57.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3154/153831 objects degraded (2.050%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:57.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:57.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:57.653 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3154/153831 objects degraded (2.050%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:57.653 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:57.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3154/153831 objects degraded (2.050%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:18:57.680 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:57.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:57.789 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:57.850 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:57.884 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:57.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:57.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 6 1 2024-04-15T18:18:57.988 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:58.038 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:58.080 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:58.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:58.185 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:58.223 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:58.250 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:58.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:58.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:58.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:58 smithi049 ceph-mon[40087]: pgmap v721: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 3175/149574 objects degraded (2.123%); 14604/149574 objects misplaced (9.764%); 13 MiB/s, 215 objects/s recovering 2024-04-15T18:18:58.405 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:58.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:58.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:18:58.606 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:58.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:58 smithi159 ceph-mon[45336]: pgmap v721: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 3175/149574 objects degraded (2.123%); 14604/149574 objects misplaced (9.764%); 13 MiB/s, 215 objects/s recovering 2024-04-15T18:18:58.730 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:18:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:58.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:18:58.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:58 smithi082 ceph-mon[44206]: pgmap v721: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 3175/149574 objects degraded (2.123%); 14604/149574 objects misplaced (9.764%); 13 MiB/s, 215 objects/s recovering 2024-04-15T18:18:58.794 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:18:58.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:18:58.863 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:58.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:18:58.882 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:18:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:58.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:18:58.932 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:18:58.940 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:58.940 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:18:58.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:18:58.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:18:59.014 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:59.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:18:59.040 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:18:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:59.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:18:59.070 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:18:59.117 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:18:59.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:18:59.162 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:18:59.188 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:59.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:18:59.230 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:18:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:59.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:18:59.236 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:18:59.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:18:59.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:18:59.336 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:18:59.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:59.412 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:18:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:18:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:59.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:18:59.534 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:18:59.570 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:59.597 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:18:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:18:59.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:18:59.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2413367690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:59.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:59.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1283180253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:18:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:59.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:18:59.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2413367690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:59.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:59.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1283180253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:18:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:59.751 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:18:59.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2413367690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:18:59.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:18:59.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1283180253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:18:59.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:18:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:00.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:00.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:00.422 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: pgmap v722: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3175/148995 objects degraded (2.131%); 14604/148995 objects misplaced (9.802%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:00.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:00 smithi049 ceph-mon[40087]: osdmap e258: 12 total, 12 up, 12 in 2024-04-15T18:19:00.564 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:00.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:00.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: pgmap v722: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3175/148995 objects degraded (2.131%); 14604/148995 objects misplaced (9.802%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:19:00.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:00.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:00 smithi159 ceph-mon[45336]: osdmap e258: 12 total, 12 up, 12 in 2024-04-15T18:19:00.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:00.725 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: pgmap v722: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 3175/148995 objects degraded (2.131%); 14604/148995 objects misplaced (9.802%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:00.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:19:00.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:00.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:00 smithi082 ceph-mon[44206]: osdmap e258: 12 total, 12 up, 12 in 2024-04-15T18:19:00.990 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:00.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:01.096 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:01.146 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:01.176 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:01.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:01.285 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:01.314 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:01.335 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:01.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:01.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:01.511 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:02.109 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:02.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:02.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:02.249 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:02.270 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:02.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:02.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:02.445 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:02 smithi159 ceph-mon[45336]: pgmap v724: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 3213/141519 objects degraded (2.270%); 13722/141519 objects misplaced (9.696%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:19:02.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3213/141519 objects degraded (2.270%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:02.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:02 smithi049 ceph-mon[40087]: pgmap v724: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 3213/141519 objects degraded (2.270%); 13722/141519 objects misplaced (9.696%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:19:02.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3213/141519 objects degraded (2.270%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:02.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:02 smithi082 ceph-mon[44206]: pgmap v724: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 3213/141519 objects degraded (2.270%); 13722/141519 objects misplaced (9.696%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:19:02.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3213/141519 objects degraded (2.270%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:03.418 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-04-15T18:19:03.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:19:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:19:03.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:19:04.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:19:04.479 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:04.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:04.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:04.628 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:04.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:04 smithi049 ceph-mon[40087]: pgmap v725: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 3213/140322 objects degraded (2.290%); 13722/140322 objects misplaced (9.779%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:19:04.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:19:04.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:04 smithi049 ceph-mon[40087]: osdmap e259: 12 total, 12 up, 12 in 2024-04-15T18:19:04.647 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:04.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:04.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:04 smithi159 ceph-mon[45336]: pgmap v725: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 3213/140322 objects degraded (2.290%); 13722/140322 objects misplaced (9.779%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:19:04.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:19:04.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:04 smithi159 ceph-mon[45336]: osdmap e259: 12 total, 12 up, 12 in 2024-04-15T18:19:04.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:04.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:04 smithi082 ceph-mon[44206]: pgmap v725: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 3213/140322 objects degraded (2.290%); 13722/140322 objects misplaced (9.779%); 12 MiB/s, 191 objects/s recovering 2024-04-15T18:19:04.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/205874175' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:19:04.793 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:04 smithi082 ceph-mon[44206]: osdmap e259: 12 total, 12 up, 12 in 2024-04-15T18:19:04.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:04.972 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:04.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:05.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 7 1 2024-04-15T18:19:05.088 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:05.116 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.135 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:05.243 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:05.280 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.303 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:05.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:05.435 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.459 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:05.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:05.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:05.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:05 smithi082 ceph-mon[44206]: osdmap e260: 12 total, 12 up, 12 in 2024-04-15T18:19:05.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:05.623 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.647 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:05.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:05.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:05.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:05 smithi049 ceph-mon[40087]: osdmap e260: 12 total, 12 up, 12 in 2024-04-15T18:19:05.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:05.784 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:05.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:05.785 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:05 smithi159 ceph-mon[45336]: osdmap e260: 12 total, 12 up, 12 in 2024-04-15T18:19:05.785 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.806 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:05.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:05.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:05.963 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:05.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:06.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:06.097 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:06.121 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:06.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:06.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:06.258 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:06.279 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:06.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:06.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:06.456 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:06.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:19:06.688 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:06.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:06 smithi049 ceph-mon[40087]: pgmap v728: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 3223/136401 objects degraded (2.363%); 13159/136401 objects misplaced (9.647%); 15 MiB/s, 232 objects/s recovering 2024-04-15T18:19:06.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:06.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:06.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:06 smithi082 ceph-mon[44206]: pgmap v728: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 3223/136401 objects degraded (2.363%); 13159/136401 objects misplaced (9.647%); 15 MiB/s, 232 objects/s recovering 2024-04-15T18:19:06.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:06.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:06.862 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:06.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:06 smithi159 ceph-mon[45336]: pgmap v728: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 3223/136401 objects degraded (2.363%); 13159/136401 objects misplaced (9.647%); 15 MiB/s, 232 objects/s recovering 2024-04-15T18:19:06.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:06.943 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:19:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:19:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:19:06.943 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:19:06.943 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:19:06.967 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:07.135 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:07.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:07.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:07.328 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:07.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3223/136401 objects degraded (2.363%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:07.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:07.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:07.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:07 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:19:07.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3223/136401 objects degraded (2.363%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:07.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:07.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:07.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:07 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:19:07.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3223/136401 objects degraded (2.363%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:07.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:07.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:07.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:07 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:19:08.439 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:08.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:08 smithi049 ceph-mon[40087]: pgmap v729: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 3223/131844 objects degraded (2.445%); 12822/131844 objects misplaced (9.725%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:19:08.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:08 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:08.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:08 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:08.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:08.545 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:08.574 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:08.597 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:08.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:08.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:08.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:08 smithi082 ceph-mon[44206]: pgmap v729: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 3223/131844 objects degraded (2.445%); 12822/131844 objects misplaced (9.725%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:19:08.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:08 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:08.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:08 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:08.830 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:08.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:08.845 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:08.850 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:08.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:19:08.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:08 smithi159 ceph-mon[45336]: pgmap v729: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 3223/131844 objects degraded (2.445%); 12822/131844 objects misplaced (9.725%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:19:08.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:08 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:08.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:08 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:08.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:08.960 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:08.982 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:08.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:09.087 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:09.115 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:09.135 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:09.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:09.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:09.291 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:09.387 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:09.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:09.494 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:09.520 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:09.544 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:09.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:09.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:09.699 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:09.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:09 smithi049 ceph-mon[40087]: osdmap e261: 12 total, 12 up, 12 in 2024-04-15T18:19:09.726 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:09.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:09.729 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:09 smithi082 ceph-mon[44206]: osdmap e261: 12 total, 12 up, 12 in 2024-04-15T18:19:09.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:09.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:09 smithi159 ceph-mon[45336]: osdmap e261: 12 total, 12 up, 12 in 2024-04-15T18:19:09.916 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:10.373 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:19:10.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:19:10.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:19:10.411 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:19:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: pgmap v731: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 3223/131229 objects degraded (2.456%); 12822/131229 objects misplaced (9.771%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3990175374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: pgmap v731: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 3223/131229 objects degraded (2.456%); 12822/131229 objects misplaced (9.771%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3990175374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: pgmap v731: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 3223/131229 objects degraded (2.456%); 12822/131229 objects misplaced (9.771%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3990175374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:10.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.695 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:11.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:11.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.863 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:11.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:12.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:12 smithi082 ceph-mon[44206]: pgmap v732: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 3238/123765 objects degraded (2.616%); 11907/123765 objects misplaced (9.621%); 14 MiB/s, 230 objects/s recovering 2024-04-15T18:19:12.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3238/123765 objects degraded (2.616%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:12.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:12 smithi159 ceph-mon[45336]: pgmap v732: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 3238/123765 objects degraded (2.616%); 11907/123765 objects misplaced (9.621%); 14 MiB/s, 230 objects/s recovering 2024-04-15T18:19:12.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3238/123765 objects degraded (2.616%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:12.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:12 smithi049 ceph-mon[40087]: pgmap v732: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 3238/123765 objects degraded (2.616%); 11907/123765 objects misplaced (9.621%); 14 MiB/s, 230 objects/s recovering 2024-04-15T18:19:12.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3238/123765 objects degraded (2.616%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:13.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:19:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=1937119606,config=1981513354,health=2874323957,kv=1398228884,logm=2628446573}) 2024-04-15T18:19:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220948347}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218697657}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482345244}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557785538}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636893240}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054420667}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097022897}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166951179}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689942070}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825615464}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221585525}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990732253}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985980557}) 2024-04-15T18:19:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714114312}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361818226}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412925487}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310980959}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815693995}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761837329}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2818886905}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804134678}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979529002}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1686993928}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198142149}) 2024-04-15T18:19:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155530085}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377283139}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911155048}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800505511}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986034832}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715911113}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=789808909}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084383892}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652928255}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746587494}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018140717}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=3} crc {logm=1074784433,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2759055767}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=13,mon_config_key=87} crc {mgrstat=2975125078,mon_config_key=2225753844}) 2024-04-15T18:19:13.791 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2878027746}) 2024-04-15T18:19:13.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=1161553257,monmap=3123577571}) 2024-04-15T18:19:13.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=2796332942,osd_pg_creating=3314912197,osdmap=1556835747}) 2024-04-15T18:19:13.792 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2430246486}) 2024-04-15T18:19:13.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=1937119606,config=1981513354,health=2874323957,kv=1398228884,logm=2628446573}) 2024-04-15T18:19:13.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220948347}) 2024-04-15T18:19:13.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218697657}) 2024-04-15T18:19:13.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482345244}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557785538}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636893240}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054420667}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097022897}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166951179}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689942070}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825615464}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221585525}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990732253}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985980557}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714114312}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361818226}) 2024-04-15T18:19:13.907 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412925487}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310980959}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815693995}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761837329}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2818886905}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804134678}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979529002}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1686993928}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198142149}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155530085}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377283139}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911155048}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800505511}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986034832}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715911113}) 2024-04-15T18:19:13.908 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=789808909}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084383892}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652928255}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746587494}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018140717}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=3} crc {logm=1074784433,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2759055767}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=13,mon_config_key=87} crc {mgrstat=2975125078,mon_config_key=2225753844}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2878027746}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=1161553257,monmap=3123577571}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=2796332942,osd_pg_creating=3314912197,osdmap=1556835747}) 2024-04-15T18:19:13.909 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2430246486}) 2024-04-15T18:19:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=1937119606,config=1981513354,health=2874323957,kv=1398228884,logm=2628446573}) 2024-04-15T18:19:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220948347}) 2024-04-15T18:19:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218697657}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1482345244}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557785538}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1636893240}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3054420667}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097022897}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166951179}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689942070}) 2024-04-15T18:19:13.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825615464}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221585525}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1990732253}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985980557}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714114312}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361818226}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412925487}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310980959}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2815693995}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761837329}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2818886905}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=804134678}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979529002}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1686993928}) 2024-04-15T18:19:13.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198142149}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155530085}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=377283139}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911155048}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800505511}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986034832}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715911113}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=789808909}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084383892}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652928255}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=746587494}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018140717}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=73,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=3} crc {logm=1074784433,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2759055767}) 2024-04-15T18:19:13.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=13,mon_config_key=87} crc {mgrstat=2975125078,mon_config_key=2225753844}) 2024-04-15T18:19:13.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2878027746}) 2024-04-15T18:19:13.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=1161553257,monmap=3123577571}) 2024-04-15T18:19:13.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=2796332942,osd_pg_creating=3314912197,osdmap=1556835747}) 2024-04-15T18:19:13.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=22} crc {osdmap=2430246486}) 2024-04-15T18:19:14.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:14 smithi082 ceph-mon[44206]: pgmap v733: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 3238/122649 objects degraded (2.640%); 11907/122649 objects misplaced (9.708%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:19:14.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:14 smithi159 ceph-mon[45336]: pgmap v733: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 3238/122649 objects degraded (2.640%); 11907/122649 objects misplaced (9.708%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:19:14.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:14 smithi049 ceph-mon[40087]: pgmap v733: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 3238/122649 objects degraded (2.640%); 11907/122649 objects misplaced (9.708%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:19:16.667 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:16.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:16.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:16.841 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:16 smithi082 ceph-mon[44206]: pgmap v734: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/118707 objects degraded (2.737%); 11301/118707 objects misplaced (9.520%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:19:16.841 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:16.871 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:16.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:16.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:16 smithi159 ceph-mon[45336]: pgmap v734: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/118707 objects degraded (2.737%); 11301/118707 objects misplaced (9.520%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:19:16.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:16 smithi049 ceph-mon[40087]: pgmap v734: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/118707 objects degraded (2.737%); 11301/118707 objects misplaced (9.520%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:19:16.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:17.030 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:17.906 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3249/118707 objects degraded (2.737%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:17.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3249/118707 objects degraded (2.737%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:18.041 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3249/118707 objects degraded (2.737%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:18.897 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:18.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:19.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:19.176 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:19 smithi049 ceph-mon[40087]: pgmap v735: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/114147 objects degraded (2.846%); 10896/114147 objects misplaced (9.546%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:19.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:19 smithi159 ceph-mon[45336]: pgmap v735: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/114147 objects degraded (2.846%); 10896/114147 objects misplaced (9.546%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:19.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:19 smithi082 ceph-mon[44206]: pgmap v735: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 3249/114147 objects degraded (2.846%); 10896/114147 objects misplaced (9.546%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:19.613 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:19.852 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:19.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:19.862 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:19.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:19.994 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:20.025 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:20.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:20.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:20.170 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-04-15T18:19:20.208 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:20.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:20.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:20.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: osdmap e262: 12 total, 12 up, 12 in 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: pgmap v737: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 3249/113568 objects degraded (2.861%); 10896/113568 objects misplaced (9.594%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3834511276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:20.350 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:20.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:20.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:20.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:20.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:20.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: osdmap e262: 12 total, 12 up, 12 in 2024-04-15T18:19:20.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: pgmap v737: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 3249/113568 objects degraded (2.861%); 10896/113568 objects misplaced (9.594%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:20.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:20.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:20.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3834511276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:20.554 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: osdmap e262: 12 total, 12 up, 12 in 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: pgmap v737: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 3249/113568 objects degraded (2.861%); 10896/113568 objects misplaced (9.594%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3834511276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:19:20.843 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:20.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:20.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:21.014 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:21.039 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:21.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:21.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:19:21.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:21 smithi049 ceph-mon[40087]: osdmap e263: 12 total, 12 up, 12 in 2024-04-15T18:19:21.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2924135586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.180 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.186 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:21.219 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:19:21.223 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:21.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:21.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:19:21.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:21 smithi082 ceph-mon[44206]: osdmap e263: 12 total, 12 up, 12 in 2024-04-15T18:19:21.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2924135586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.288 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.326 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:21.352 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:21.375 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:21.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:21.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:21.525 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:21.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:21.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:19:21.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:21 smithi159 ceph-mon[45336]: osdmap e263: 12 total, 12 up, 12 in 2024-04-15T18:19:21.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2924135586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:19:21.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:21.723 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: pgmap v739: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 3249/105735 objects degraded (3.073%); 9885/105735 objects misplaced (9.349%); 16 MiB/s, 251 objects/s recovering 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: osdmap e264: 12 total, 12 up, 12 in 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:22.189 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3249/113568 objects degraded (2.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: pgmap v739: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 3249/105735 objects degraded (3.073%); 9885/105735 objects misplaced (9.349%); 16 MiB/s, 251 objects/s recovering 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: osdmap e264: 12 total, 12 up, 12 in 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:22.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3249/113568 objects degraded (2.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:22.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:22.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: pgmap v739: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 3249/105735 objects degraded (3.073%); 9885/105735 objects misplaced (9.349%); 16 MiB/s, 251 objects/s recovering 2024-04-15T18:19:22.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:19:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: osdmap e264: 12 total, 12 up, 12 in 2024-04-15T18:19:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:22.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3249/113568 objects degraded (2.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:22.891 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:19:22.892 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:19:23.098 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:23.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:23.211 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:23.240 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:23.266 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:23.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:23.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:23.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:23.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:23.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:23 smithi049 ceph-mon[40087]: osdmap e265: 12 total, 12 up, 12 in 2024-04-15T18:19:23.425 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:23.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:23.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:23.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:23 smithi082 ceph-mon[44206]: osdmap e265: 12 total, 12 up, 12 in 2024-04-15T18:19:23.561 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:23.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:23.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:19:23.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:23.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:23.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:23 smithi159 ceph-mon[45336]: osdmap e265: 12 total, 12 up, 12 in 2024-04-15T18:19:23.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:19:23.721 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:23.759 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:19:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:23.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:19:23.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:23.901 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:23.924 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:23.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:24.030 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:24.063 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.085 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:24.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:24.240 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.352 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:24.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:19:24.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:24.490 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.514 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:24.516 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:24.516 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:24.516 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:24 smithi159 ceph-mon[45336]: pgmap v742: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3249/104622 objects degraded (3.105%); 9885/104622 objects misplaced (9.448%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:24.516 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:24 smithi159 ceph-mon[45336]: osdmap e266: 12 total, 12 up, 12 in 2024-04-15T18:19:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:24.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:24.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:24.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:24 smithi082 ceph-mon[44206]: pgmap v742: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3249/104622 objects degraded (3.105%); 9885/104622 objects misplaced (9.448%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:24.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:24 smithi082 ceph-mon[44206]: osdmap e266: 12 total, 12 up, 12 in 2024-04-15T18:19:24.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:24.645 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.668 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:24.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:24.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:24.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:24 smithi049 ceph-mon[40087]: pgmap v742: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3249/104622 objects degraded (3.105%); 9885/104622 objects misplaced (9.448%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:24.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:24 smithi049 ceph-mon[40087]: osdmap e266: 12 total, 12 up, 12 in 2024-04-15T18:19:24.771 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:24.797 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 8 1 2024-04-15T18:19:24.822 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:24.832 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:24.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:24.965 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:24.993 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:24.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:25.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:25.162 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:25.529 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:25.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:19:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:25.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:25.613 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:19:25.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:25.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:19:25.701 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:19:25.763 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:19:25.811 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:19:25.834 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:25.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:25.850 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:19:25.895 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:19:25.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:25.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:19:25.966 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:25.979 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:19:25.989 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:25.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:26.013 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:19:26.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:19:26.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:19:26.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:19:26.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:26.140 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:26.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: pgmap v744: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3255/100476 objects degraded (3.240%); 9297/100476 objects misplaced (9.253%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:26.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4285498736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:26.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.259 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: pgmap v744: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3255/100476 objects degraded (3.240%); 9297/100476 objects misplaced (9.253%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4285498736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: pgmap v744: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3255/100476 objects degraded (3.240%); 9297/100476 objects misplaced (9.253%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4285498736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:26.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3255/100476 objects degraded (3.240%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:27.570 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3255/100476 objects degraded (3.240%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:27.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3255/100476 objects degraded (3.240%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:28.054 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:28.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:28.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:28.189 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:28.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:28.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:28.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:28.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:28.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:28 smithi049 ceph-mon[40087]: pgmap v745: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 3255/95952 objects degraded (3.392%); 8942/95952 objects misplaced (9.319%); 9.7 MiB/s, 156 objects/s recovering 2024-04-15T18:19:28.352 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:28.383 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:28.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:28.493 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:28.522 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:28.522 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:28 smithi159 ceph-mon[45336]: pgmap v745: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 3255/95952 objects degraded (3.392%); 8942/95952 objects misplaced (9.319%); 9.7 MiB/s, 156 objects/s recovering 2024-04-15T18:19:28.523 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:28.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:28.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:28 smithi082 ceph-mon[44206]: pgmap v745: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 3255/95952 objects degraded (3.392%); 8942/95952 objects misplaced (9.319%); 9.7 MiB/s, 156 objects/s recovering 2024-04-15T18:19:28.546 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:28.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:28.650 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:28.677 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:28.701 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:28.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:28.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:28.883 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:28.990 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:29.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:29.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:29.191 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:29.209 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:19:29.209 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:19:29.210 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:19:29.210 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:19:30.295 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:19:30.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:30 smithi082 ceph-mon[44206]: pgmap v746: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 3255/95415 objects degraded (3.411%); 8942/95415 objects misplaced (9.372%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:19:30.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:19:30.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:30 smithi159 ceph-mon[45336]: pgmap v746: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 3255/95415 objects degraded (3.411%); 8942/95415 objects misplaced (9.372%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:19:30.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:19:30.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:30 smithi049 ceph-mon[40087]: pgmap v746: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 3255/95415 objects degraded (3.411%); 8942/95415 objects misplaced (9.372%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:19:30.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:19:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:31 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:31.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:31 smithi049 ceph-mon[40087]: osdmap e267: 12 total, 12 up, 12 in 2024-04-15T18:19:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:31 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:31.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:31 smithi082 ceph-mon[44206]: osdmap e267: 12 total, 12 up, 12 in 2024-04-15T18:19:31.580 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:31.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:31.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:31 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:31.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1204227051' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:19:31.657 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:31 smithi159 ceph-mon[45336]: osdmap e267: 12 total, 12 up, 12 in 2024-04-15T18:19:31.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:31.722 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:31.759 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:31.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:31.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:31.906 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:31.936 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:31.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:32.044 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:19:32.084 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:32.120 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:19:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:32.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:19:32.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:19:32.226 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:32.254 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:32.277 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:32.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:32.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:32.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:32 smithi049 ceph-mon[40087]: pgmap v748: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 3301/87222 objects degraded (3.785%); 7948/87222 objects misplaced (9.112%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:19:32.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3301/87222 objects degraded (3.785%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:32.450 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:32.507 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:32.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:32.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:32 smithi082 ceph-mon[44206]: pgmap v748: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 3301/87222 objects degraded (3.785%); 7948/87222 objects misplaced (9.112%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:19:32.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3301/87222 objects degraded (3.785%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:32.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:32 smithi159 ceph-mon[45336]: pgmap v748: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 3301/87222 objects degraded (3.785%); 7948/87222 objects misplaced (9.112%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:19:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3301/87222 objects degraded (3.785%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:32.658 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:32.687 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:32.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:32.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:32.826 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:32.847 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:32.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:32.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:32.984 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:33.009 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:33.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:33.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:33.157 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:33.424 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:33.433 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:33.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:33.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:19:33.578 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:33.603 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:19:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:33.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:19:33.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:33.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:19:33.794 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:33.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:33.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:33.981 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:34.130 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:34.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:34.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:19:34.288 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:34.317 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:19:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:34.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:19:34.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:34.457 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:34.476 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:34.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:34.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:19:34.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:34 smithi082 ceph-mon[44206]: pgmap v749: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 3301/86106 objects degraded (3.834%); 7948/86106 objects misplaced (9.230%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:34.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:34.590 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:34 smithi049 ceph-mon[40087]: pgmap v749: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 3301/86106 objects degraded (3.834%); 7948/86106 objects misplaced (9.230%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:34.607 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:34 smithi159 ceph-mon[45336]: pgmap v749: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 3301/86106 objects degraded (3.834%); 7948/86106 objects misplaced (9.230%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:19:34.609 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:34.630 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:34.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:34.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:34.792 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:35.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:35.153 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:35.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:35.220 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:35.310 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-04-15T18:19:35.339 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:35.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:35.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:19:35.447 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:19:35.490 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:35.522 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:19:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:35.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:19:35.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:35.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:19:35.666 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:35.669 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:19:35.700 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:35.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:35.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:35.865 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:36.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:36 smithi159 ceph-mon[45336]: pgmap v750: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 3301/82176 objects degraded (4.017%); 7371/82176 objects misplaced (8.970%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:19:36.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:19:36.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:36 smithi159 ceph-mon[45336]: osdmap e268: 12 total, 12 up, 12 in 2024-04-15T18:19:36.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:36 smithi049 ceph-mon[40087]: pgmap v750: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 3301/82176 objects degraded (4.017%); 7371/82176 objects misplaced (8.970%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:19:36.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:19:36.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:36 smithi049 ceph-mon[40087]: osdmap e268: 12 total, 12 up, 12 in 2024-04-15T18:19:36.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:36 smithi082 ceph-mon[44206]: pgmap v750: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 3301/82176 objects degraded (4.017%); 7371/82176 objects misplaced (8.970%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:19:36.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2651479504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:19:36.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:36 smithi082 ceph-mon[44206]: osdmap e268: 12 total, 12 up, 12 in 2024-04-15T18:19:37.234 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:37.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:37.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:37.380 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:37.408 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:37.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:37.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3301/82176 objects degraded (4.017%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:37.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:19:37.564 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:37.590 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:19:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:37.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:19:37.597 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3301/82176 objects degraded (4.017%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:37.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3301/82176 objects degraded (4.017%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:37.698 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:19:37.731 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:37.754 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:19:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:37.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:19:37.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:37.905 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:37.945 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:37.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:38.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:38.080 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:38.101 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:38.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:38.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:19:38.243 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:19:38.243 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:19:38.243 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:19:38.243 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:19:38.243 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:19:38.256 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:38.286 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:19:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:38.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:19:38.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:19:38.447 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:39.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:39 smithi159 ceph-mon[45336]: pgmap v752: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3301/77664 objects degraded (4.250%); 6950/77664 objects misplaced (8.949%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:19:39.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:39 smithi049 ceph-mon[40087]: pgmap v752: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3301/77664 objects degraded (4.250%); 6950/77664 objects misplaced (8.949%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:19:39.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:39 smithi082 ceph-mon[44206]: pgmap v752: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 3301/77664 objects degraded (4.250%); 6950/77664 objects misplaced (8.949%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:19:39.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:19:40.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:40.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:19:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:40.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:19:40.219 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:40.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 9 1 2024-04-15T18:19:40.316 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:40.345 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:40.365 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:40.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:40.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:40.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:40.447 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:40.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:19:40.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:40.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:40.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:19:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:40.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:19:40.677 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:40.708 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:40.731 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:40.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:40.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:19:40.868 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:40.889 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:19:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:40.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:19:40.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:41.027 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:41.047 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:41.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:41.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:41.220 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:41.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:41 smithi159 ceph-mon[45336]: pgmap v753: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3301/77055 objects degraded (4.284%); 6950/77055 objects misplaced (9.020%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:19:41.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:41 smithi159 ceph-mon[45336]: osdmap e269: 12 total, 12 up, 12 in 2024-04-15T18:19:41.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:41 smithi049 ceph-mon[40087]: pgmap v753: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3301/77055 objects degraded (4.284%); 6950/77055 objects misplaced (9.020%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:19:41.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:41 smithi049 ceph-mon[40087]: osdmap e269: 12 total, 12 up, 12 in 2024-04-15T18:19:41.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:41 smithi082 ceph-mon[44206]: pgmap v753: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3301/77055 objects degraded (4.284%); 6950/77055 objects misplaced (9.020%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:19:41.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:41 smithi082 ceph-mon[44206]: osdmap e269: 12 total, 12 up, 12 in 2024-04-15T18:19:41.589 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:19:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:41.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:19:41.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:41.723 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:41.746 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:41.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:41.854 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:41.882 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:41.901 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:41.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:42.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:42.038 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:42.058 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:42.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:42.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:19:42.215 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:42.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:42 smithi049 ceph-mon[40087]: pgmap v755: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 3301/68388 objects degraded (4.827%); 5945/68388 objects misplaced (8.693%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:42.349 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3301/77055 objects degraded (4.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:42.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:19:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:19:42.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:42 smithi159 ceph-mon[45336]: pgmap v755: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 3301/68388 objects degraded (4.827%); 5945/68388 objects misplaced (8.693%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:42.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3301/77055 objects degraded (4.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:42.461 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:19:42.490 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:42.510 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:19:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:42.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:19:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:42 smithi082 ceph-mon[44206]: pgmap v755: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 3301/68388 objects degraded (4.827%); 5945/68388 objects misplaced (8.693%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:19:42.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3301/77055 objects degraded (4.284%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:42.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:42.655 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:43.112 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:43.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:43.151 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:19:43.203 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:19:43.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:43.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:19:43.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:43.275 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:19:43.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:19:43.302 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:43.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:43.327 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:19:43.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:19:43.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:19:43.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:19:43.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1579459437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:43.480 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:43.481 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:19:43.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1579459437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:43.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:19:43.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:19:43.642 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:19:43.644 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:19:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:43.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:19:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1579459437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:19:43.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:19:43.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:19:43.800 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:43.845 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:19:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:19:43.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:19:43.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:19:43.995 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:19:44.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: pgmap v756: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 3301/67209 objects degraded (4.912%); 5945/67209 objects misplaced (8.846%); 11 MiB/s, 178 objects/s recovering 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/285752770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:19:44.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: pgmap v756: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 3301/67209 objects degraded (4.912%); 5945/67209 objects misplaced (8.846%); 11 MiB/s, 178 objects/s recovering 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/285752770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: pgmap v756: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 3301/67209 objects degraded (4.912%); 5945/67209 objects misplaced (8.846%); 11 MiB/s, 178 objects/s recovering 2024-04-15T18:19:44.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/285752770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:44.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:45.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:19:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:45 smithi049 ceph-mon[40087]: osdmap e270: 12 total, 12 up, 12 in 2024-04-15T18:19:45.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:45.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:19:45.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:45 smithi082 ceph-mon[44206]: osdmap e270: 12 total, 12 up, 12 in 2024-04-15T18:19:45.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:19:45.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:19:45.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:19:45.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:45 smithi159 ceph-mon[45336]: osdmap e270: 12 total, 12 up, 12 in 2024-04-15T18:19:46.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:46 smithi049 ceph-mon[40087]: pgmap v758: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3301/63087 objects degraded (5.232%); 5299/63087 objects misplaced (8.400%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:19:46.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:46 smithi082 ceph-mon[44206]: pgmap v758: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3301/63087 objects degraded (5.232%); 5299/63087 objects misplaced (8.400%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:19:46.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:46 smithi159 ceph-mon[45336]: pgmap v758: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3301/63087 objects degraded (5.232%); 5299/63087 objects misplaced (8.400%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:19:47.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: 3.4 scrub starts 2024-04-15T18:19:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: osdmap e271: 12 total, 12 up, 12 in 2024-04-15T18:19:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: 3.4 scrub ok 2024-04-15T18:19:47.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3301/63087 objects degraded (5.232%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: 3.4 scrub starts 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: osdmap e271: 12 total, 12 up, 12 in 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: 3.4 scrub ok 2024-04-15T18:19:47.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3301/63087 objects degraded (5.232%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:47.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: 3.4 scrub starts 2024-04-15T18:19:47.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:47.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:47.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: osdmap e271: 12 total, 12 up, 12 in 2024-04-15T18:19:47.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: 3.4 scrub ok 2024-04-15T18:19:47.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3301/63087 objects degraded (5.232%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:48.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:48 smithi049 ceph-mon[40087]: pgmap v760: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 3301/58404 objects degraded (5.652%); 5065/58404 objects misplaced (8.672%); 13 MiB/s, 245 objects/s recovering 2024-04-15T18:19:48.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:48 smithi082 ceph-mon[44206]: pgmap v760: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 3301/58404 objects degraded (5.652%); 5065/58404 objects misplaced (8.672%); 13 MiB/s, 245 objects/s recovering 2024-04-15T18:19:48.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:48 smithi159 ceph-mon[45336]: pgmap v760: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 3301/58404 objects degraded (5.652%); 5065/58404 objects misplaced (8.672%); 13 MiB/s, 245 objects/s recovering 2024-04-15T18:19:50.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:50 smithi049 ceph-mon[40087]: pgmap v761: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 442 op/s; 3301/57774 objects degraded (5.714%); 5065/57774 objects misplaced (8.767%); 6.9 MiB/s, 137 objects/s recovering 2024-04-15T18:19:50.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:50 smithi082 ceph-mon[44206]: pgmap v761: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 442 op/s; 3301/57774 objects degraded (5.714%); 5065/57774 objects misplaced (8.767%); 6.9 MiB/s, 137 objects/s recovering 2024-04-15T18:19:50.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:50 smithi159 ceph-mon[45336]: pgmap v761: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 442 op/s; 3301/57774 objects degraded (5.714%); 5065/57774 objects misplaced (8.767%); 6.9 MiB/s, 137 objects/s recovering 2024-04-15T18:19:52.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:52 smithi049 ceph-mon[40087]: 3.9 scrub starts 2024-04-15T18:19:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:52 smithi049 ceph-mon[40087]: pgmap v762: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 3301/51888 objects degraded (6.362%); 4249/51888 objects misplaced (8.189%); 13 MiB/s, 239 objects/s recovering 2024-04-15T18:19:52.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3301/51888 objects degraded (6.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:52.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:52 smithi082 ceph-mon[44206]: 3.9 scrub starts 2024-04-15T18:19:52.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:52 smithi082 ceph-mon[44206]: pgmap v762: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 3301/51888 objects degraded (6.362%); 4249/51888 objects misplaced (8.189%); 13 MiB/s, 239 objects/s recovering 2024-04-15T18:19:52.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3301/51888 objects degraded (6.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:52.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:52 smithi159 ceph-mon[45336]: 3.9 scrub starts 2024-04-15T18:19:52.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:52 smithi159 ceph-mon[45336]: pgmap v762: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 3301/51888 objects degraded (6.362%); 4249/51888 objects misplaced (8.189%); 13 MiB/s, 239 objects/s recovering 2024-04-15T18:19:52.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3301/51888 objects degraded (6.362%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:53.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:19:53.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:19:53.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:19:53.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:53 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:19:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:19:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:19:53.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:53 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:53.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:19:53.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:19:53.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:19:53.543 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:53 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:19:54.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:54.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:54.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: pgmap v763: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1006 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3282/48282 objects degraded (6.798%); 3992/48282 objects misplaced (8.268%); 14 MiB/s, 270 objects/s recovering 2024-04-15T18:19:54.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:54.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:54.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:54.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:54.539 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: pgmap v763: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1006 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3282/48282 objects degraded (6.798%); 3992/48282 objects misplaced (8.268%); 14 MiB/s, 270 objects/s recovering 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:54.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: pgmap v763: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 1006 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 3282/48282 objects degraded (6.798%); 3992/48282 objects misplaced (8.268%); 14 MiB/s, 270 objects/s recovering 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:19:54.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:19:55.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:55 smithi082 ceph-mon[44206]: osdmap e272: 12 total, 12 up, 12 in 2024-04-15T18:19:55.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:55 smithi159 ceph-mon[45336]: osdmap e272: 12 total, 12 up, 12 in 2024-04-15T18:19:55.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:55 smithi049 ceph-mon[40087]: osdmap e272: 12 total, 12 up, 12 in 2024-04-15T18:19:56.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:56 smithi082 ceph-mon[44206]: pgmap v765: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 946 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 3282/45405 objects degraded (7.228%); 3474/45405 objects misplaced (7.651%); 11 MiB/s, 205 objects/s recovering 2024-04-15T18:19:56.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:56 smithi159 ceph-mon[45336]: pgmap v765: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 946 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 3282/45405 objects degraded (7.228%); 3474/45405 objects misplaced (7.651%); 11 MiB/s, 205 objects/s recovering 2024-04-15T18:19:56.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:56 smithi049 ceph-mon[40087]: pgmap v765: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 946 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 506 op/s; 3282/45405 objects degraded (7.228%); 3474/45405 objects misplaced (7.651%); 11 MiB/s, 205 objects/s recovering 2024-04-15T18:19:57.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3282/45405 objects degraded (7.228%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:57.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3282/45405 objects degraded (7.228%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:57.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3282/45405 objects degraded (7.228%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:19:58.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:58 smithi049 ceph-mon[40087]: pgmap v766: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 825 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3001/39597 objects degraded (7.579%); 2954/39597 objects misplaced (7.460%); 13 MiB/s, 265 objects/s recovering 2024-04-15T18:19:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:19:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:19:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:19:58 smithi049 ceph-mon[40087]: osdmap e273: 12 total, 12 up, 12 in 2024-04-15T18:19:58.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:58 smithi082 ceph-mon[44206]: pgmap v766: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 825 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3001/39597 objects degraded (7.579%); 2954/39597 objects misplaced (7.460%); 13 MiB/s, 265 objects/s recovering 2024-04-15T18:19:58.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:19:58.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:19:58.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:19:58 smithi082 ceph-mon[44206]: osdmap e273: 12 total, 12 up, 12 in 2024-04-15T18:19:58.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:58 smithi159 ceph-mon[45336]: pgmap v766: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 825 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 3001/39597 objects degraded (7.579%); 2954/39597 objects misplaced (7.460%); 13 MiB/s, 265 objects/s recovering 2024-04-15T18:19:58.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:19:58.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:19:58.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:19:58 smithi159 ceph-mon[45336]: osdmap e273: 12 total, 12 up, 12 in 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: pgmap v768: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 813 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3001/39009 objects degraded (7.693%); 2954/39009 objects misplaced (7.573%); 10 MiB/s, 229 objects/s recovering 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: [WRN] PG_DEGRADED: Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: pg 3.2 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [6,9] 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,2] 2024-04-15T18:20:00.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-04-15T18:20:00.538 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:00 smithi082 ceph-mon[44206]: pg 3.c is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,6] 2024-04-15T18:20:00.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:20:00.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: pgmap v768: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 813 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3001/39009 objects degraded (7.693%); 2954/39009 objects misplaced (7.573%); 10 MiB/s, 229 objects/s recovering 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: [WRN] PG_DEGRADED: Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: pg 3.2 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [6,9] 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,2] 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-04-15T18:20:00.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:00 smithi049 ceph-mon[40087]: pg 3.c is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,6] 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: pgmap v768: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 813 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 3001/39009 objects degraded (7.693%); 2954/39009 objects misplaced (7.573%); 10 MiB/s, 229 objects/s recovering 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: [WRN] PG_DEGRADED: Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: pg 3.2 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [6,9] 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,2] 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,4] 2024-04-15T18:20:00.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:00 smithi159 ceph-mon[45336]: pg 3.c is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,6] 2024-04-15T18:20:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:02 smithi082 ceph-mon[44206]: pgmap v769: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 685 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 3001/32871 objects degraded (9.130%); 2056/32871 objects misplaced (6.255%); 15 MiB/s, 277 objects/s recovering 2024-04-15T18:20:02.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3001/32871 objects degraded (9.130%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:02 smithi159 ceph-mon[45336]: pgmap v769: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 685 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 3001/32871 objects degraded (9.130%); 2056/32871 objects misplaced (6.255%); 15 MiB/s, 277 objects/s recovering 2024-04-15T18:20:02.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3001/32871 objects degraded (9.130%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:02 smithi049 ceph-mon[40087]: pgmap v769: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 685 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 3001/32871 objects degraded (9.130%); 2056/32871 objects misplaced (6.255%); 15 MiB/s, 277 objects/s recovering 2024-04-15T18:20:02.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3001/32871 objects degraded (9.130%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:03.640 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:03.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:03.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:04.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:04 smithi082 ceph-mon[44206]: pgmap v770: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 612 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 2729/29334 objects degraded (9.303%); 1782/29334 objects misplaced (6.075%); 15 MiB/s, 308 objects/s recovering 2024-04-15T18:20:04.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:04 smithi159 ceph-mon[45336]: pgmap v770: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 612 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 2729/29334 objects degraded (9.303%); 1782/29334 objects misplaced (6.075%); 15 MiB/s, 308 objects/s recovering 2024-04-15T18:20:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:04 smithi049 ceph-mon[40087]: pgmap v770: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 612 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 2729/29334 objects degraded (9.303%); 1782/29334 objects misplaced (6.075%); 15 MiB/s, 308 objects/s recovering 2024-04-15T18:20:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:06.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:06.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:06 smithi082 ceph-mon[44206]: pgmap v771: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 546 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 2729/26172 objects degraded (10.427%); 1216/26172 objects misplaced (4.646%); 14 MiB/s, 291 objects/s recovering 2024-04-15T18:20:06.574 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:06.622 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:06.654 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:06.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:06 smithi159 ceph-mon[45336]: pgmap v771: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 546 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 2729/26172 objects degraded (10.427%); 1216/26172 objects misplaced (4.646%); 14 MiB/s, 291 objects/s recovering 2024-04-15T18:20:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:06.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:06.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:06 smithi049 ceph-mon[40087]: pgmap v771: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 546 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 640 op/s; 2729/26172 objects degraded (10.427%); 1216/26172 objects misplaced (4.646%); 14 MiB/s, 291 objects/s recovering 2024-04-15T18:20:06.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:07.436 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:07.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:07.543 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:07.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2729/26172 objects degraded (10.427%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:07.575 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:07.576 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:07.576 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:07.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:07.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:07.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:07.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2729/26172 objects degraded (10.427%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:07.747 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:07.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:07.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2729/26172 objects degraded (10.427%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:07.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:07.919 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:08.185 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:08.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:08.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:20:08.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:08.331 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:08.382 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:08.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:08.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:08 smithi049 ceph-mon[40087]: pgmap v772: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 400 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 2491/19179 objects degraded (12.988%); 698/19179 objects misplaced (3.639%); 14 MiB/s, 287 objects/s recovering 2024-04-15T18:20:08.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:08.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:20:08.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:08 smithi082 ceph-mon[44206]: pgmap v772: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 400 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 2491/19179 objects degraded (12.988%); 698/19179 objects misplaced (3.639%); 14 MiB/s, 287 objects/s recovering 2024-04-15T18:20:08.544 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:08.578 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:08.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:08.645 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:08.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:08 smithi159 ceph-mon[45336]: pgmap v772: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 400 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 2491/19179 objects degraded (12.988%); 698/19179 objects misplaced (3.639%); 14 MiB/s, 287 objects/s recovering 2024-04-15T18:20:08.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:08.733 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:08.957 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:08.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:09.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:09.129 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:09.172 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:09.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:09.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:09.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2097475781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:09 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.326 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:09.351 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-04-15T18:20:09.365 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:09.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:09.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:09.534 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:09.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2097475781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:09 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.740 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:09.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:09.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2097475781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:09 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:20:09.844 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:09.874 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:09.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:09.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:10.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:10.014 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:10.069 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:10.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:10.243 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:10.297 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.325 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:10.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:10.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:10.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:10 smithi049 ceph-mon[40087]: pgmap v773: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 385 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2491/18480 objects degraded (13.479%); 698/18480 objects misplaced (3.777%); 13 MiB/s, 260 objects/s recovering 2024-04-15T18:20:10.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:10 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:20:10.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:10 smithi049 ceph-mon[40087]: osdmap e274: 12 total, 12 up, 12 in 2024-04-15T18:20:10.469 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.512 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:10.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:10.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:10 smithi082 ceph-mon[44206]: pgmap v773: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 385 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2491/18480 objects degraded (13.479%); 698/18480 objects misplaced (3.777%); 13 MiB/s, 260 objects/s recovering 2024-04-15T18:20:10.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:10 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:20:10.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:10 smithi082 ceph-mon[44206]: osdmap e274: 12 total, 12 up, 12 in 2024-04-15T18:20:10.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 10 1 2024-04-15T18:20:10.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:10.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:10 smithi159 ceph-mon[45336]: pgmap v773: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 385 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2491/18480 objects degraded (13.479%); 698/18480 objects misplaced (3.777%); 13 MiB/s, 260 objects/s recovering 2024-04-15T18:20:10.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:10 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:20:10.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:10 smithi159 ceph-mon[45336]: osdmap e274: 12 total, 12 up, 12 in 2024-04-15T18:20:10.658 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.680 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:10.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:10.784 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:10.821 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.846 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:10.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:10.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:10.997 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:10.998 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:10.998 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:11.019 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:11.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:11.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:11.174 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:11.299 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:11.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:11.408 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:11.438 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:11.790 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:11.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:11.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:20:11.896 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:11.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:11.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:11.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:12.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:12.107 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:12.224 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:12.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:12 smithi049 ceph-mon[40087]: pgmap v775: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 255 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 2491/12222 objects degraded (20.381%); 586/12222 objects misplaced (4.795%); 9.2 MiB/s, 284 objects/s recovering 2024-04-15T18:20:12.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2491/12222 objects degraded (20.381%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:12.575 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:12.576 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:12 smithi082 ceph-mon[44206]: pgmap v775: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 255 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 2491/12222 objects degraded (20.381%); 586/12222 objects misplaced (4.795%); 9.2 MiB/s, 284 objects/s recovering 2024-04-15T18:20:12.576 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2491/12222 objects degraded (20.381%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:12.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:12.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:12 smithi159 ceph-mon[45336]: pgmap v775: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 255 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 2491/12222 objects degraded (20.381%); 586/12222 objects misplaced (4.795%); 9.2 MiB/s, 284 objects/s recovering 2024-04-15T18:20:12.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2491/12222 objects degraded (20.381%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:12.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:12.738 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:12.883 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:12.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:12.917 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:12.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:13.037 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:13.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:13.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:13.207 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:13.236 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:13.257 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:13.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:13.343 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:20:13.361 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:13.395 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:13.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2881521059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.511 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:13 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2881521059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:13 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.649 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2881521059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:13 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:13.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:13.757 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:13.797 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:14.009 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:14.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:14.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:14.167 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:14.238 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:14.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:14.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:14.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:14.378 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:14.393 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:14.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:14.419 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:14.419 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:14.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:14.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:14.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:14.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: pgmap v776: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 2244/8571 objects degraded (26.181%); 582/8571 objects misplaced (6.790%); 7.5 MiB/s, 280 objects/s recovering 2024-04-15T18:20:14.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:14.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:14.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: osdmap e275: 12 total, 12 up, 12 in 2024-04-15T18:20:14.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3970182708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:14.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:14.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:14.562 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:14.589 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:14.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: pgmap v776: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 2244/8571 objects degraded (26.181%); 582/8571 objects misplaced (6.790%); 7.5 MiB/s, 280 objects/s recovering 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: osdmap e275: 12 total, 12 up, 12 in 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3970182708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:14.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:14.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: pgmap v776: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 2244/8571 objects degraded (26.181%); 582/8571 objects misplaced (6.790%); 7.5 MiB/s, 280 objects/s recovering 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: osdmap e275: 12 total, 12 up, 12 in 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3970182708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:14.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:14.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:15.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:20:15.237 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:15.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:15.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:15.377 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:15.467 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.568 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 106473 objects 2024-04-15T18:20:15.568 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :176.617 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:15.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:16.235 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:16.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:16.340 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:16.343 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-15T18:20:16.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-15T18:20:16.368 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:16.390 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:16.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:16.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:16.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:16 smithi049 ceph-mon[40087]: pgmap v778: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 115 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1903/5511 objects degraded (34.531%); 582/5511 objects misplaced (10.561%); 927 KiB/s, 249 objects/s recovering 2024-04-15T18:20:16.532 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:16.566 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:16.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:16.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:16 smithi159 ceph-mon[45336]: pgmap v778: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 115 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1903/5511 objects degraded (34.531%); 582/5511 objects misplaced (10.561%); 927 KiB/s, 249 objects/s recovering 2024-04-15T18:20:16.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:16.700 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:16.707 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:16.729 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:16.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:16.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:16 smithi082 ceph-mon[44206]: pgmap v778: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 115 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1903/5511 objects degraded (34.531%); 582/5511 objects misplaced (10.561%); 927 KiB/s, 249 objects/s recovering 2024-04-15T18:20:16.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:16.859 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:17.108 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:17.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-04-15T18:20:17.213 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:17.240 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:17 smithi049 ceph-mon[40087]: osdmap e276: 12 total, 12 up, 12 in 2024-04-15T18:20:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:20:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1903/5511 objects degraded (34.531%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:20:17.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:17 smithi049 ceph-mon[40087]: osdmap e277: 12 total, 12 up, 12 in 2024-04-15T18:20:17.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:17 smithi082 ceph-mon[44206]: osdmap e276: 12 total, 12 up, 12 in 2024-04-15T18:20:17.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:20:17.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1903/5511 objects degraded (34.531%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:17.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:20:17.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:17 smithi082 ceph-mon[44206]: osdmap e277: 12 total, 12 up, 12 in 2024-04-15T18:20:17.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:20:17.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:17 smithi159 ceph-mon[45336]: osdmap e276: 12 total, 12 up, 12 in 2024-04-15T18:20:17.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:20:17.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1903/5511 objects degraded (34.531%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:17.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1588473081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:20:17.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:17 smithi159 ceph-mon[45336]: osdmap e277: 12 total, 12 up, 12 in 2024-04-15T18:20:17.789 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:17.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:17.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:17.921 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:17.940 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:17.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:18.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:20:18.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:4206 2.d e124/114 2:b7208875:::benchmark_data_smithi159_80260_object4205:head v 124'256, mlcod=124'253)", 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:20:18.220 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 508.580174774, 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27413553299999999, 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638676+0000", 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638677+0000", 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7599999999999997e-07 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.638790+0000", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113306 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912318+0000", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27352749300000001 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912322+0000", 2024-04-15T18:20:18.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5759999999999999e-06 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912371+0000", 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8529999999999998e-05 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912386+0000", 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5089000000000001e-05 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912779+0000", 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039285399999999999 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:49.912812+0000", 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3209e-05 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.225 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62467 2.d e143/114 2:b6055ada:::benchmark_data_smithi159_80260_object62466:head v 143'3982, mlcod=143'3979)", 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 462.294432798, 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.364600912, 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924418+0000", 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924419+0000", 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7699999999999992e-07 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:35.924487+0000", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7405999999999998e-05 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287406+0000", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.362919453 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287411+0000", 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1189999999999996e-06 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287463+0000", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1357999999999999e-05 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.287483+0000", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0254999999999998e-05 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.288986+0000", 2024-04-15T18:20:18.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001503576 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.289019+0000", 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2768000000000002e-05 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62487 2.f e143/114 2:f21ee866:::benchmark_data_smithi159_80260_object62486:head v 143'3880, mlcod=143'3878)", 2024-04-15T18:20:18.230 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 462.12075259, 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17900353999999999, 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098098+0000", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098100+0000", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.905e-06 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.098160+0000", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0068999999999997e-05 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275509+0000", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17734833 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275515+0000", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0360000000000003e-06 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275556+0000", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0874999999999997e-05 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.275572+0000", 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6164e-05 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.277075+0000", 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015032750000000001 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.277102+0000", 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6886e-05 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:62513 2.d e143/114 2:b0d4bfc1:::benchmark_data_smithi159_80260_object62512:head v 143'3986, mlcod=143'3983)", 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 461.89765380900002, 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.162785016, 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:20:18.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321197+0000", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321199+0000", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.593e-06 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.321261+0000", 2024-04-15T18:20:18.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2166000000000001e-05 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482466+0000", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.161205454 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482471+0000", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6090000000000003e-06 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482528+0000", 2024-04-15T18:20:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6551000000000001e-05 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.482541+0000", 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.305e-05 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.483953+0000", 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001412632 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:36.483982+0000", 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8960999999999999e-05 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:153095 2.d e176/114 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head v 176'9796, mlcod=176'9793)", 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 347.068590664, 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25507247500000002, 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.239 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150262+0000", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6479999999999999e-06 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150272+0000", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6250000000000002e-06 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400544+0000", 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25027195200000002 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400549+0000", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2229999999999998e-06 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400606+0000", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7339000000000001e-05 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400629+0000", 2024-04-15T18:20:18.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3059000000000001e-05 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405295+0000", 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046661419999999999 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405333+0000", 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7487000000000002e-05 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:3201 3.2 e204/199 3:431af5b7:::benchmark_data_smithi159_91621_object3200:head v 204'198, mlcod=204'193)", 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 268.26588245699998, 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18664436700000001, 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:20:18.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.952969+0000", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9200000000000001e-07 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:49.953106+0000", 2024-04-15T18:20:18.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013703200000000001 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139138+0000", 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18603144299999999 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139145+0000", 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8299999999999998e-06 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139195+0000", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0207999999999998e-05 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139218+0000", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3274000000000001e-05 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139574+0000", 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035602399999999998 2024-04-15T18:20:18.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:50.139613+0000", 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8964000000000002e-05 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:15728 3.e e205/199 3:7aaffa93:::benchmark_data_smithi159_91621_object15727:head v 205'970, mlcod=205'967)", 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 260.61108818999998, 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21313194799999999, 2024-04-15T18:20:18.248 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607763+0000", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4899999999999995e-07 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.607879+0000", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011526700000000001 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819651+0000", 2024-04-15T18:20:18.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21177270200000001 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819657+0000", 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819701+0000", 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3424999999999998e-05 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.819714+0000", 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3763e-05 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.820843+0000", 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011291389999999999 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.820895+0000", 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1411e-05 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.252 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:45769 3.3 3:c748487c:::benchmark_data_smithi159_91621_object45768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 239.938005952, 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15225887799999999, 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.253 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 45769 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280845+0000", 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280846+0000", 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.268e-06 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280922+0000", 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5308999999999993e-05 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280935+0000", 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2863e-05 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280943+0000", 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1249999999999993e-06 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.280984+0000", 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1341000000000003e-05 2024-04-15T18:20:18.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.281071+0000", 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6646999999999999e-05 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.281156+0000", 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5002999999999999e-05 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.282260+0000", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001104516 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.282297+0000", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6772e-05 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433007+0000", 2024-04-15T18:20:18.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15070974100000001 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433069+0000", 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2612000000000003e-05 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:18.433104+0000", 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4681000000000002e-05 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.259 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:82192 3.3 3:cf0d204e:::benchmark_data_smithi159_91621_object82191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 207.27184618300001, 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26314599599999999, 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82192 2024-04-15T18:20:18.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947006+0000", 2024-04-15T18:20:18.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4890000000000001e-06 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947096+0000", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9630999999999996e-05 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947112+0000", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5571000000000001e-05 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947119+0000", 2024-04-15T18:20:18.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6040000000000004e-06 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986537+0000", 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.039417671000000001 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986667+0000", 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013041100000000001 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986813+0000", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014619599999999999 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991475+0000", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046618220000000004 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991584+0000", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010877799999999999 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210072+0000", 2024-04-15T18:20:18.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21848774500000001 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210107+0000", 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5611999999999997e-05 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210151+0000", 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3466e-05 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:82205 3.3 3:cead4c8e:::benchmark_data_smithi159_91621_object82204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 207.25566696300001, 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24726364100000001, 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82205 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.266 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963186+0000", 2024-04-15T18:20:18.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7859999999999999e-06 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963315+0000", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012882900000000001 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963331+0000", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6065e-05 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963339+0000", 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5350000000000004e-06 2024-04-15T18:20:18.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989027+0000", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.025687326 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989114+0000", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7418999999999998e-05 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989211+0000", 2024-04-15T18:20:18.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6518e-05 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991499+0000", 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022884369999999999 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991589+0000", 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9748999999999997e-05 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210351+0000", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21876195400000001 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210406+0000", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5059999999999998e-05 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210448+0000", 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1962999999999998e-05 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.271 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:113539 3.2 e233/199)", 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:26.419795+0000", 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 171.79905652900001, 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.155801456, 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.419795+0000", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.419795+0000", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.419795+0000", 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0999999999999999e-07 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.419800+0000", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0250000000000002e-06 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.575470+0000", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.155670216 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.575476+0000", 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8980000000000001e-06 2024-04-15T18:20:18.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.575585+0000", 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108396 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.575596+0000", 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1411e-05 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:113551 3.e e234/199 3:7e2db53f:::benchmark_data_smithi159_91621_object7075:head v 234'7047, mlcod=233'7045)", 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:26.440708+0000", 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 171.778143387, 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203608383, 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.440708+0000", 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.440708+0000", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.440708+0000", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.97e-07 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.440711+0000", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2200000000000001e-06 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.641465+0000", 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20075426499999999 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.641471+0000", 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7960000000000001e-06 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.641542+0000", 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0599000000000006e-05 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.641557+0000", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.52e-05 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.644281+0000", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027239759999999999 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.644316+0000", 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5129999999999997e-05 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:113573 3.a e234/199 3:55511c97:::benchmark_data_smithi159_91621_object7097:head v 234'7161, mlcod=234'7158)", 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 171.55513185300001, 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41958572, 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663720+0000", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8800000000000002e-07 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663727+0000", 2024-04-15T18:20:18.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6490000000000002e-06 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079390+0000", 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41566319699999998 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079401+0000", 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1106e-05 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079486+0000", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5329999999999998e-05 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079506+0000", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9528e-05 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083269+0000", 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037634790000000001 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083305+0000", 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5642999999999997e-05 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113580 3.3 3:c4e755da:::benchmark_data_smithi159_91621_object7104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e235)", 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 171.135062041, 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27359631099999998, 2024-04-15T18:20:18.284 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113580 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083791+0000", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6479999999999999e-06 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083796+0000", 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.485e-06 2024-04-15T18:20:18.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083810+0000", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3864e-05 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083818+0000", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5460000000000003e-06 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083866+0000", 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8829999999999998e-05 2024-04-15T18:20:18.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083982+0000", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0001153 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.084071+0000", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9450000000000006e-05 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087618+0000", 2024-04-15T18:20:18.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035471399999999998 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087772+0000", 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015348900000000001 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357160+0000", 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26938846300000002 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357259+0000", 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8851000000000006e-05 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357385+0000", 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012624500000000001 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24793.0:113580 3.3 e235/199 ondisk, result = 0)", 2024-04-15T18:20:18.290 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 171.13109134699999, 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26963310400000001, 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087761+0000", 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.122e-06 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087768+0000", 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.356968+0000", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26919998699999997 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.356985+0000", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6609999999999999e-05 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357132+0000", 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147217 2024-04-15T18:20:18.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357149+0000", 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7127000000000001e-05 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357393+0000", 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000243553 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121162 3.3 3:c1f10df4:::benchmark_data_smithi159_91621_object14686:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.142855+0000", 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 156.07599589099999, 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.221065182, 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121162 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142855+0000", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142855+0000", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142857+0000", 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.699e-06 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142864+0000", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4909999999999999e-06 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142880+0000", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5641000000000002e-05 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142890+0000", 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6229999999999992e-06 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.142966+0000", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6675999999999996e-05 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.143062+0000", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5838e-05 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.143131+0000", 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8885000000000002e-05 2024-04-15T18:20:18.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.146847+0000", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037160520000000001 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.148026+0000", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00117891 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.363815+0000", 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.215789285 2024-04-15T18:20:18.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.363872+0000", 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7154999999999997e-05 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.363920+0000", 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7926999999999997e-05 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121169 3.3 3:caa2620f:::benchmark_data_smithi159_91621_object14693:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.154580+0000", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 156.06427104400001, 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20953934099999999, 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121169 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154580+0000", 2024-04-15T18:20:18.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154580+0000", 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154582+0000", 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.43e-06 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154590+0000", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.481e-06 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154610+0000", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9766e-05 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154620+0000", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0332999999999999e-05 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154680+0000", 2024-04-15T18:20:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9687e-05 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154806+0000", 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012624899999999999 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.154916+0000", 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110103 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.158633+0000", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037172860000000002 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.158855+0000", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00022143299999999999 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364024+0000", 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20516968599999999 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364081+0000", 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6595000000000003e-05 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364119+0000", 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8291999999999999e-05 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121174 3.3 3:c06d44fa:::benchmark_data_smithi159_91621_object14698:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.172983+0000", 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 156.04586855700001, 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19126834700000001, 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121174 2024-04-15T18:20:18.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.172983+0000", 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.172983+0000", 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.172985+0000", 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0829999999999998e-06 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.172994+0000", 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2769999999999994e-06 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.173015+0000", 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0630999999999999e-05 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.173024+0000", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5850000000000006e-06 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.173086+0000", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1514999999999996e-05 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.173216+0000", 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013027700000000001 2024-04-15T18:20:18.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.173323+0000", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010733 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.177093+0000", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037698200000000001 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.179519+0000", 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002426194 2024-04-15T18:20:18.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364166+0000", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.184647221 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364213+0000", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6304999999999998e-05 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.364251+0000", 2024-04-15T18:20:18.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8108999999999997e-05 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:121649 3.e e239/199 3:70d1154d:::benchmark_data_smithi159_91621_object15173:head v 239'7559, mlcod=239'7556)", 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:43.299147+0000", 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 154.919704169, 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.218373809, 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.299147+0000", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.299147+0000", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.299148+0000", 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9299999999999997e-07 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.299154+0000", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9190000000000004e-06 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.507472+0000", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20831849599999999 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.507481+0000", 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4810000000000006e-06 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.507536+0000", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5556000000000001e-05 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.507565+0000", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8872000000000001e-05 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.517476+0000", 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0099110190000000001 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.517521+0000", 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4773e-05 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:153508 3.3 3:c411a006:::benchmark_data_smithi159_91621_object47032:head [delete] snapc 0=[] ondisk+write+known_if_redirected e253)", 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:38.224797+0000", 2024-04-15T18:20:18.317 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 99.994054388999999, 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15147872300000001, 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 153508 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224797+0000", 2024-04-15T18:20:18.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224797+0000", 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224799+0000", 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.832e-06 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:18.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224805+0000", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1879999999999999e-06 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224818+0000", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3647e-05 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224826+0000", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9559999999999997e-06 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:18.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224885+0000", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8486000000000002e-05 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.224996+0000", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011114 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.225111+0000", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000114782 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:18.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.229730+0000", 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004619756 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.242388+0000", 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012657501999999999 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.376197+0000", 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13380903599999999 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.376241+0000", 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4388000000000003e-05 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:38.376275+0000", 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4010000000000001e-05 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:18.323 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:18.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:18 smithi082 ceph-mon[44206]: pgmap v781: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 9.6 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1410/441 objects degraded (319.728%); 352/441 objects misplaced (79.819%); 43 KiB/s, 367 objects/s recovering 2024-04-15T18:20:18.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:18 smithi082 ceph-mon[44206]: 3.a scrub starts 2024-04-15T18:20:18.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:18 smithi082 ceph-mon[44206]: 3.a scrub ok 2024-04-15T18:20:18.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:18 smithi082 ceph-mon[44206]: osdmap e278: 12 total, 12 up, 12 in 2024-04-15T18:20:18.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:18 smithi159 ceph-mon[45336]: pgmap v781: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 9.6 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1410/441 objects degraded (319.728%); 352/441 objects misplaced (79.819%); 43 KiB/s, 367 objects/s recovering 2024-04-15T18:20:18.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:18 smithi159 ceph-mon[45336]: 3.a scrub starts 2024-04-15T18:20:18.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:18 smithi159 ceph-mon[45336]: 3.a scrub ok 2024-04-15T18:20:18.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:18 smithi159 ceph-mon[45336]: osdmap e278: 12 total, 12 up, 12 in 2024-04-15T18:20:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:18 smithi049 ceph-mon[40087]: pgmap v781: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 9.6 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 1410/441 objects degraded (319.728%); 352/441 objects misplaced (79.819%); 43 KiB/s, 367 objects/s recovering 2024-04-15T18:20:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:18 smithi049 ceph-mon[40087]: 3.a scrub starts 2024-04-15T18:20:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:18 smithi049 ceph-mon[40087]: 3.a scrub ok 2024-04-15T18:20:18.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:18 smithi049 ceph-mon[40087]: osdmap e278: 12 total, 12 up, 12 in 2024-04-15T18:20:18.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:20:19.260 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:19.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 11 1 2024-04-15T18:20:19.434 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:19.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:19.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:19.570 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:19.591 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:19.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:19.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:19.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:19.733 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:19.750 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:20.154 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' removed 2024-04-15T18:20:20.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:20 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:20:20.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:20 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:20:20.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:20 smithi049 ceph-mon[40087]: osdmap e279: 12 total, 12 up, 12 in 2024-04-15T18:20:20.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:20:20.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:20 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:20:20.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:20 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:20:20.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:20 smithi159 ceph-mon[45336]: osdmap e279: 12 total, 12 up, 12 in 2024-04-15T18:20:20.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:20:20.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:20 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:20:20.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:20 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:20:20.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:20 smithi082 ceph-mon[44206]: osdmap e279: 12 total, 12 up, 12 in 2024-04-15T18:20:20.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:20:21.090 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:21.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:21.174 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:21.175 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:20:21.175 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:20:21.175 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:21.193 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:21.225 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:21.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:21 smithi159 ceph-mon[45336]: pgmap v784: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 1410/141 objects degraded (1000.000%); 352/141 objects misplaced (249.645%); 0 B/s, 270 objects/s recovering 2024-04-15T18:20:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:21 smithi159 ceph-mon[45336]: 3.d deep-scrub starts 2024-04-15T18:20:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:21 smithi159 ceph-mon[45336]: 3.d deep-scrub ok 2024-04-15T18:20:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:20:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:21 smithi159 ceph-mon[45336]: osdmap e280: 12 total, 12 up, 12 in 2024-04-15T18:20:21.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:21 smithi049 ceph-mon[40087]: pgmap v784: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 1410/141 objects degraded (1000.000%); 352/141 objects misplaced (249.645%); 0 B/s, 270 objects/s recovering 2024-04-15T18:20:21.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:21 smithi049 ceph-mon[40087]: 3.d deep-scrub starts 2024-04-15T18:20:21.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:21 smithi049 ceph-mon[40087]: 3.d deep-scrub ok 2024-04-15T18:20:21.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:20:21.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:21 smithi049 ceph-mon[40087]: osdmap e280: 12 total, 12 up, 12 in 2024-04-15T18:20:21.475 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:21.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:21 smithi082 ceph-mon[44206]: pgmap v784: 17 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 1410/141 objects degraded (1000.000%); 352/141 objects misplaced (249.645%); 0 B/s, 270 objects/s recovering 2024-04-15T18:20:21.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:21 smithi082 ceph-mon[44206]: 3.d deep-scrub starts 2024-04-15T18:20:21.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:21 smithi082 ceph-mon[44206]: 3.d deep-scrub ok 2024-04-15T18:20:21.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/66496068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:20:21.477 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:21 smithi082 ceph-mon[44206]: osdmap e280: 12 total, 12 up, 12 in 2024-04-15T18:20:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:21.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:21.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:21.610 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:21.631 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:21.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:21.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:21.766 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:21.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:20:22.015 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:22.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:22.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:22.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:22.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:22.118 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:22.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:22.150 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:22.158 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:22.164 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:22.173 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:22.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:22.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:22.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:22.236 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:22.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:22.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:22.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:22.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3640096374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:22.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:22.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:22 smithi049 ceph-mon[40087]: osdmap e281: 12 total, 12 up, 12 in 2024-04-15T18:20:22.305 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:22.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:22.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:22.338 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:22.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:22.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:22.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3640096374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:22.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:22.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:22 smithi159 ceph-mon[45336]: osdmap e281: 12 total, 12 up, 12 in 2024-04-15T18:20:22.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3640096374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:22.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:20:22.427 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:22 smithi082 ceph-mon[44206]: osdmap e281: 12 total, 12 up, 12 in 2024-04-15T18:20:22.539 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:20:22.540 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:20:22.540 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-15T18:20:22.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-04-15T18:20:23.154 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:23.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:23.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:23.287 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:23.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized) 2024-04-15T18:20:23.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: pgmap v786: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1410/141 objects degraded (1000.000%), 4 pgs degraded, 4 pgs undersized) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:23.946 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:23.946 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:24.639 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:20:24.905 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:25.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:20:25.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:25 smithi082 ceph-mon[44206]: pgmap v788: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:25.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:25.419 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:25.461 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:25.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:25 smithi049 ceph-mon[40087]: pgmap v788: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:25.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:25.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:25.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:25 smithi159 ceph-mon[45336]: pgmap v788: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:25.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:25.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:25.838 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:25.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:25.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:25.971 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:26.168 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:20:26.169 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-04-15T18:20:26.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:26 smithi082 ceph-mon[44206]: osdmap e282: 12 total, 12 up, 12 in 2024-04-15T18:20:26.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:20:26.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:20:26.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:26 smithi049 ceph-mon[40087]: osdmap e282: 12 total, 12 up, 12 in 2024-04-15T18:20:26.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:20:26.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:20:26.621 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:26.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:26.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:26 smithi159 ceph-mon[45336]: osdmap e282: 12 total, 12 up, 12 in 2024-04-15T18:20:26.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:20:26.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:20:26.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:26.757 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:26.779 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:26.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:26.824 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:26.824 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:26.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:26.914 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:27.164 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:27.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:27.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:27.298 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: pgmap v790: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:20:27.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:20:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:27 smithi049 ceph-mon[40087]: osdmap e283: 12 total, 12 up, 12 in 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: pgmap v790: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:20:27.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:27 smithi082 ceph-mon[44206]: osdmap e283: 12 total, 12 up, 12 in 2024-04-15T18:20:27.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: pgmap v790: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2651387521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3332501817' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:20:27.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:27 smithi159 ceph-mon[45336]: osdmap e283: 12 total, 12 up, 12 in 2024-04-15T18:20:27.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 3 1 2024-04-15T18:20:27.795 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:28.091 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:28.348 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:28.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:28.452 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:28.484 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:28.505 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:28.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:28.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:28.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:28.659 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:28.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:28.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:28.791 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:28.810 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:28.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:28.914 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:28.942 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:28.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:28.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:28.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:29.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:20:29.064 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:29.091 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:29.109 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:29.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:29.213 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:29.239 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:29.257 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:29.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:29.361 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:29.368 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:29 smithi049 ceph-mon[40087]: pgmap v792: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:29.385 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:29 smithi159 ceph-mon[45336]: pgmap v792: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:29.387 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:29.405 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:29.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:29.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:29.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:29 smithi082 ceph-mon[44206]: pgmap v792: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:29.545 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:30.177 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' created 2024-04-15T18:20:30.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-15T18:20:30.476 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-15T18:20:30.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-15T18:20:30.896 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:30.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:31.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:31.031 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:31.131 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:31.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:31.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:31.264 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:31.463 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:31.465 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:31 smithi082 ceph-mon[44206]: pgmap v793: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:31.465 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-15T18:20:31.465 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:31 smithi082 ceph-mon[44206]: osdmap e284: 12 total, 12 up, 12 in 2024-04-15T18:20:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:31.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:31 smithi049 ceph-mon[40087]: pgmap v793: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-15T18:20:31.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:31 smithi049 ceph-mon[40087]: osdmap e284: 12 total, 12 up, 12 in 2024-04-15T18:20:31.474 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:31.474 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:20:31.474 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:20:31.475 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:31.567 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:31.597 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:31 smithi159 ceph-mon[45336]: pgmap v793: 1 pgs: 1 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:31.597 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1061601848' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-15T18:20:31.597 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:31 smithi159 ceph-mon[45336]: osdmap e284: 12 total, 12 up, 12 in 2024-04-15T18:20:31.599 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:31.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-15T18:20:31.620 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:31.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:31.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:31.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:31.693 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:31.694 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:31.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:31.752 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:31.771 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:31.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:31.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:31.875 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:31.907 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:31.927 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:31.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:32.031 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:32.061 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:32.081 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:32.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:32.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:32.213 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:32.233 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:32.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:32.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:20:32.326 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:32.326 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osdmap e285: 12 total, 12 up, 12 in 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: pgmap v796: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/132060057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:32.368 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:32.386 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:32.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:32.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osdmap e285: 12 total, 12 up, 12 in 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: pgmap v796: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/132060057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.517 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.517 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.517 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.517 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.517 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.518 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:32.537 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:32.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:32.545 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:32.545 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:32.566 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:32.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osdmap e285: 12 total, 12 up, 12 in 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: pgmap v796: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/132060057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:32.668 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:32.867 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:32.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:32.971 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:33.003 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:33.025 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:33.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:33.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:33.160 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:20:33.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:33 smithi159 ceph-mon[45336]: osdmap e286: 12 total, 12 up, 12 in 2024-04-15T18:20:33.457 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.457 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.457 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.457 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.457 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:20:33.458 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:33 smithi082 ceph-mon[44206]: osdmap e286: 12 total, 12 up, 12 in 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:20:33.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:33 smithi049 ceph-mon[40087]: osdmap e286: 12 total, 12 up, 12 in 2024-04-15T18:20:34.194 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-04-15T18:20:34.210 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:34.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:34.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:34 smithi082 ceph-mon[44206]: pgmap v798: 17 pgs: 2 creating+peering, 11 active+clean, 4 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:34.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:20:34.301 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:20:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:20:34.301 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:20:34.301 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:34.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:34.346 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:34.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:20:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:34 smithi049 ceph-mon[40087]: pgmap v798: 17 pgs: 2 creating+peering, 11 active+clean, 4 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:20:34.649 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:34.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:34.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:34 smithi159 ceph-mon[45336]: pgmap v798: 17 pgs: 2 creating+peering, 11 active+clean, 4 unknown; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:34.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:20:34.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:34.782 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:34.980 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:34.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:35.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:20:35.096 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:35.124 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:35.148 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:35.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:35.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:35.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:20:35.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:35 smithi049 ceph-mon[40087]: osdmap e287: 12 total, 12 up, 12 in 2024-04-15T18:20:35.281 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:35.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:20:35.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:35 smithi082 ceph-mon[44206]: osdmap e287: 12 total, 12 up, 12 in 2024-04-15T18:20:35.298 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:35.319 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:35.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:35.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:35.450 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:35.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1765399021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:20:35.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:35 smithi159 ceph-mon[45336]: osdmap e287: 12 total, 12 up, 12 in 2024-04-15T18:20:35.666 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:35.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:35.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:35.800 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:36.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:36.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:36.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:36 smithi049 ceph-mon[40087]: osdmap e288: 12 total, 12 up, 12 in 2024-04-15T18:20:36.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:36 smithi049 ceph-mon[40087]: pgmap v801: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:36.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:36.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:36.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:36 smithi082 ceph-mon[44206]: osdmap e288: 12 total, 12 up, 12 in 2024-04-15T18:20:36.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:36 smithi082 ceph-mon[44206]: pgmap v801: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:36.647 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:36.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:36.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:36.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:36.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:36 smithi159 ceph-mon[45336]: osdmap e288: 12 total, 12 up, 12 in 2024-04-15T18:20:36.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:36 smithi159 ceph-mon[45336]: pgmap v801: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:36.751 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:36.766 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-15T18:20:36.766 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-15T18:20:36.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:20:36.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:20:36.783 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:36.806 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:36.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:36.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:36.940 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:36.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:36.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:37.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:37.093 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:37.191 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:37.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:37.199 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6347 2.7 2:e016f06f:::benchmark_data_smithi159_80260_object6346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.449125+0000", 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.74984873999995, 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28303931999999998, 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6347 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449125+0000", 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449125+0000", 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449127+0000", 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1260000000000001e-06 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449162+0000", 2024-04-15T18:20:37.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5355000000000003e-05 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449173+0000", 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1538e-05 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449179+0000", 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8270000000000003e-06 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449229+0000", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9898000000000002e-05 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449353+0000", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012428299999999999 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449511+0000", 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000157417 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.449973+0000", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046223299999999999 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.450388+0000", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000415072 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732049+0000", 2024-04-15T18:20:37.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28166097000000001 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732110+0000", 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1097000000000001e-05 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732165+0000", 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4503999999999999e-05 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6362 2.7 2:e94222d5:::benchmark_data_smithi159_80260_object6361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.459772+0000", 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.73920248100001, 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27284787399999999, 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6362 2024-04-15T18:20:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459772+0000", 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459772+0000", 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459773+0000", 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.33e-07 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459828+0000", 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5748000000000002e-05 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459840+0000", 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1090000000000001e-05 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459845+0000", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6180000000000003e-06 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459883+0000", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7477e-05 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.459967+0000", 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4491000000000001e-05 2024-04-15T18:20:37.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.460054+0000", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7047999999999997e-05 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.460461+0000", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00040648099999999999 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.460852+0000", 2024-04-15T18:20:37.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039101000000000001 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732560+0000", 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27170790700000003 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732594+0000", 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4768000000000003e-05 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732620+0000", 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5303000000000002e-05 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6392 2.7 2:ec4452f0:::benchmark_data_smithi159_80260_object6391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.478784+0000", 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.72019017800005, 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25926159300000001, 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6392 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478784+0000", 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478784+0000", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478785+0000", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1230000000000001e-06 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478839+0000", 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3579000000000003e-05 2024-04-15T18:20:37.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478850+0000", 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1306e-05 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478856+0000", 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6799999999999998e-06 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.478903+0000", 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7341e-05 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.479037+0000", 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133767 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.479199+0000", 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016214499999999999 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.479683+0000", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048447399999999998 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.480234+0000", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00055024999999999996 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.737982+0000", 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25774844499999999 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.738018+0000", 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5615999999999999e-05 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.738046+0000", 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7866999999999999e-05 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:6347 2.7 e124/114 ondisk, result = 0)", 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.480014+0000", 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.71896035400005, 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25216543200000002, 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.480014+0000", 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.480014+0000", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.480014+0000", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.06e-07 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.480018+0000", 2024-04-15T18:20:37.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7979999999999999e-06 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.731998+0000", 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25197978300000001 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732003+0000", 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8969999999999999e-06 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732036+0000", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3154000000000001e-05 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732044+0000", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6590000000000007e-06 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732179+0000", 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135535 2024-04-15T18:20:37.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:6398 2.7 2:e7f6a0d6:::benchmark_data_smithi159_80260_object6397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.481879+0000", 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.71709513500002, 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28169209899999997, 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 6398 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481879+0000", 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481879+0000", 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481880+0000", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0100000000000004e-07 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481966+0000", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5945000000000005e-05 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481976+0000", 2024-04-15T18:20:37.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9299999999999998e-06 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.481981+0000", 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3290000000000001e-06 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.482028+0000", 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6799999999999999e-05 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.482149+0000", 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000121455 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.482292+0000", 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143237 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.482718+0000", 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042549900000000003 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.483073+0000", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00035533899999999998 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.763499+0000", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28042560700000002 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.763531+0000", 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2341000000000001e-05 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.763571+0000", 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9916000000000002e-05 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:6362 2.7 e124/114 ondisk, result = 0)", 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:11:51.483056+0000", 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 525.71591787099999, 2024-04-15T18:20:37.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.249568969, 2024-04-15T18:20:37.233 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.233 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.233 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.483056+0000", 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.483056+0000", 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.483057+0000", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8999999999999998e-07 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.483058+0000", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6899999999999999e-06 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732495+0000", 2024-04-15T18:20:37.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24943632299999999 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732501+0000", 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.793e-06 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732547+0000", 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5266000000000003e-05 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732554+0000", 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.401e-06 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:11:51.732625+0000", 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1205999999999996e-05 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:64501 2.7 2:e87c3425:::benchmark_data_smithi159_80260_object64500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e150)", 2024-04-15T18:20:37.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:12:52.508126+0000", 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 464.69084865999997, 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.211648906, 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 64501 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508126+0000", 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508126+0000", 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508127+0000", 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7600000000000006e-07 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508160+0000", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3324999999999997e-05 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508171+0000", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0778000000000001e-05 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508176+0000", 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5629999999999999e-06 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508214+0000", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7883999999999998e-05 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508310+0000", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5853000000000005e-05 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.508395+0000", 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5228000000000005e-05 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.510206+0000", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001811266 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.512067+0000", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0018602130000000001 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.719698+0000", 2024-04-15T18:20:37.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.207630957 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.719739+0000", 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1065000000000001e-05 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:12:52.719774+0000", 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5797999999999998e-05 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157250 2.7 2:e680b69a:::benchmark_data_smithi159_80260_object68137:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:38.242534+0000", 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 358.95644047600001, 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21686218700000001, 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157250 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242534+0000", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242534+0000", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242535+0000", 2024-04-15T18:20:37.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.049e-06 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242542+0000", 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5560000000000002e-06 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242554+0000", 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1846e-05 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242561+0000", 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.64e-06 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242613+0000", 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2113000000000001e-05 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242748+0000", 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013515399999999999 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.242869+0000", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000121396 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.246024+0000", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031543629999999999 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.362141+0000", 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.11611761 2024-04-15T18:20:37.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459290+0000", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.097148294999999996 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459347+0000", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7584999999999998e-05 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459396+0000", 2024-04-15T18:20:37.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8579999999999999e-05 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:157250 2.7 e177/114 ondisk, result = 0)", 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:38.248837+0000", 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 358.95013755500003, 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21056571700000001, 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:20:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.248837+0000", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.248837+0000", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.248837+0000", 2024-04-15T18:20:37.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4099999999999998e-07 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.248842+0000", 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6539999999999998e-06 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459109+0000", 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21026682799999999 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459114+0000", 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5539999999999997e-06 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459277+0000", 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016296400000000001 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459284+0000", 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9700000000000002e-06 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:38.459402+0000", 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118106 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:5358 3.0 e204/199 3:093e027e:::benchmark_data_smithi159_91621_object5357:head v 204'329, mlcod=204'326)", 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:51.005512+0000", 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.19346177599999, 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27906233600000002, 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.254 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.005512+0000", 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.005512+0000", 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.005513+0000", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5900000000000002e-07 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.005611+0000", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8082e-05 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283984+0000", 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27837336200000001 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.283991+0000", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0929999999999997e-06 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284043+0000", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1360999999999998e-05 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284064+0000", 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1430999999999999e-05 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284531+0000", 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046694299999999999 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:51.284575+0000", 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3605e-05 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:48079 3.a 3:5b62cbe5:::benchmark_data_smithi159_91621_object48078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e211)", 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:20.266504+0000", 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 256.93247034199999, 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216506009, 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 48079 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.259 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266504+0000", 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266504+0000", 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266505+0000", 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.268e-06 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266528+0000", 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2447e-05 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266541+0000", 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3423e-05 2024-04-15T18:20:37.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266549+0000", 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5939999999999996e-06 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266597+0000", 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8031999999999997e-05 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266688+0000", 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1231999999999997e-05 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.266774+0000", 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6531000000000002e-05 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.268309+0000", 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015349070000000001 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.272287+0000", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039772849999999997 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.482901+0000", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21061484499999999 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.482952+0000", 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0893999999999998e-05 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:20.483010+0000", 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7550999999999998e-05 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:95301 3.2 e228/199 3:4af59164:::benchmark_data_smithi159_91621_object95300:head v 228'5992, mlcod=228'5985)", 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:05.233168+0000", 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 211.965806418, 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.201982774, 2024-04-15T18:20:37.265 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.233168+0000", 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.233168+0000", 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.233169+0000", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.233245+0000", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6915999999999993e-05 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.432369+0000", 2024-04-15T18:20:37.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19912313800000001 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.432375+0000", 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6200000000000001e-06 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.432439+0000", 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4244000000000004e-05 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.432476+0000", 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6961000000000001e-05 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.435128+0000", 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026515639999999999 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:05.435151+0000", 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2597999999999999e-05 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113573 3.a 3:55511c97:::benchmark_data_smithi159_91621_object7097:head [delete] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:26.663406+0000", 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 190.53556809700001, 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.420137396, 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113573 2024-04-15T18:20:37.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663406+0000", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663406+0000", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663407+0000", 2024-04-15T18:20:37.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3200000000000003e-07 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663411+0000", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1810000000000003e-06 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663422+0000", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0784e-05 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663428+0000", 2024-04-15T18:20:37.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7019999999999998e-06 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663480+0000", 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2330000000000002e-05 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663570+0000", 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9586000000000006e-05 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:20:37.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663642+0000", 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2321000000000001e-05 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.666174+0000", 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002531742 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.666332+0000", 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000157851 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083466+0000", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41713425999999998 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083509+0000", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3362999999999998e-05 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083544+0000", 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4344000000000002e-05 2024-04-15T18:20:37.275 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113522 3.2 3:448c0e61:::benchmark_data_smithi159_91621_object7046:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.108983+0000", 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 189.08999088300001, 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37065005499999998, 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.276 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113522 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.108983+0000", 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.108983+0000", 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.108986+0000", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7470000000000001e-06 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.108997+0000", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1042e-05 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109018+0000", 2024-04-15T18:20:37.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0656999999999999e-05 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109036+0000", 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8700999999999999e-05 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.110403+0000", 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001366824 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.110416+0000", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2989999999999999e-05 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.479441+0000", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.369024776 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.479633+0000", 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000192318 2024-04-15T18:20:37.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113539 3.2 3:4d15d8a0:::benchmark_data_smithi159_91621_object7063:head [delete] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.109065+0000", 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 189.089909097, 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37274885800000002, 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113539 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109065+0000", 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109065+0000", 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109066+0000", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8500000000000002e-07 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109069+0000", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2959999999999999e-06 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109078+0000", 2024-04-15T18:20:37.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6770000000000001e-06 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.109084+0000", 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2310000000000001e-06 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.110441+0000", 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001357577 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.110448+0000", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0330000000000003e-06 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481674+0000", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37122534000000001 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481814+0000", 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014021899999999999 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:157704 3.a 3:57d2c4ca:::benchmark_data_smithi159_91621_object51228:head [delete] snapc 0=[] ondisk+write+known_if_redirected e255)", 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:45.602083+0000", 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.596891149, 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22132805999999999, 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157704 2024-04-15T18:20:37.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602083+0000", 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602083+0000", 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602084+0000", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3399999999999997e-07 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602088+0000", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4259999999999996e-06 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602098+0000", 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5189999999999998e-06 2024-04-15T18:20:37.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602104+0000", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8579999999999996e-06 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602144+0000", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0550000000000003e-05 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602229+0000", 2024-04-15T18:20:37.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5104000000000004e-05 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.602319+0000", 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9876999999999993e-05 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.608488+0000", 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0061685569999999999 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.616273+0000", 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0077852720000000002 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823303+0000", 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20702947799999999 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823363+0000", 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0180000000000003e-05 2024-04-15T18:20:37.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823411+0000", 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8304999999999999e-05 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24793.0:157704 3.a e255/199 ondisk, result = 0)", 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:45.611149+0000", 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.587825083, 2024-04-15T18:20:37.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21226789800000001, 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.611149+0000", 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.611149+0000", 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.611150+0000", 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.78e-07 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.611157+0000", 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6420000000000001e-06 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823222+0000", 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21206550699999999 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823230+0000", 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0700000000000007e-06 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823288+0000", 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7238999999999999e-05 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823297+0000", 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1740000000000006e-06 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:45.823417+0000", 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120288 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:184093 3.2 3:454f188a:::benchmark_data_smithi159_91621_object77617:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-04-15T18:20:37.296 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:19:30.378748+0000", 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 66.820225906999994, 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21139005799999999, 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 184093 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378748+0000", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378748+0000", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378750+0000", 2024-04-15T18:20:37.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.666e-06 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378757+0000", 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3429999999999998e-06 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378775+0000", 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7961e-05 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378785+0000", 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.251e-06 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.378849+0000", 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4530999999999996e-05 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.379015+0000", 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165995 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.379192+0000", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017666599999999999 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.383377+0000", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041851070000000004 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.383414+0000", 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7224000000000001e-05 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.588723+0000", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20530857899999999 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.588794+0000", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0975000000000004e-05 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.590138+0000", 2024-04-15T18:20:37.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00134476 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:184111 3.2 3:438df7bf:::benchmark_data_smithi159_91621_object77635:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:19:30.408106+0000", 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 66.790868161000006, 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20590092300000001, 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.303 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 184111 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408106+0000", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408106+0000", 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408107+0000", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2860000000000001e-06 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408114+0000", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3509999999999998e-06 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408124+0000", 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0501e-05 2024-04-15T18:20:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408130+0000", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1070000000000001e-06 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408179+0000", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8653999999999999e-05 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408331+0000", 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015245600000000001 2024-04-15T18:20:37.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.408507+0000", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017531400000000001 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.413386+0000", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0048797520000000002 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.413528+0000", 2024-04-15T18:20:37.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014179700000000001 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.612909+0000", 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19938051600000001 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.612950+0000", 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0967999999999998e-05 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:30.614007+0000", 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010572209999999999 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:212948 3.2 3:409c861b:::benchmark_data_smithi159_91621_object106472:head [delete] snapc 0=[] ondisk+write+known_if_redirected e275)", 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:15.005483+0000", 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 22.193491380000001, 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.56468193700000002, 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:20:37.309 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 212948 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005483+0000", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:20:37.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005483+0000", 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005485+0000", 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9989999999999998e-06 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005494+0000", 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7299999999999994e-06 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005510+0000", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6444999999999999e-05 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005520+0000", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0144e-05 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005587+0000", 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7162000000000005e-05 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005727+0000", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013946499999999999 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.005865+0000", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013861599999999999 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.010123+0000", 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004257444 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.113356+0000", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.10323277 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.569916+0000", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45656052800000002 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.569960+0000", 2024-04-15T18:20:37.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4132999999999999e-05 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.570165+0000", 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000204501 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:20:37.315 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:37.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:20:37.343 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:37.415 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:37.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:37.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:37.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:37.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:37.548 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:37.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:37.804 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:37.892 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:20:37.896 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:37.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:37.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:38.032 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:38.054 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:38.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:38.154 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:38.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:38.188 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:38.209 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:38.212 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-15T18:20:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:38.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:38.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:38 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:38.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:38.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:38 smithi049 ceph-mon[40087]: osdmap e289: 12 total, 12 up, 12 in 2024-04-15T18:20:38.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:38 smithi049 ceph-mon[40087]: pgmap v803: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:38.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:20:38.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:38.344 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:38.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:38 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:38.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:38.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:38 smithi159 ceph-mon[45336]: osdmap e289: 12 total, 12 up, 12 in 2024-04-15T18:20:38.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:38 smithi159 ceph-mon[45336]: pgmap v803: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:38.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:20:38.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:38 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:38.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2431331574' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:38.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:38 smithi082 ceph-mon[44206]: osdmap e289: 12 total, 12 up, 12 in 2024-04-15T18:20:38.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:38 smithi082 ceph-mon[44206]: pgmap v803: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:38.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:20:39.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:20:39.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:39 smithi049 ceph-mon[40087]: osdmap e290: 12 total, 12 up, 12 in 2024-04-15T18:20:39.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:20:39.499 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:39 smithi082 ceph-mon[44206]: osdmap e290: 12 total, 12 up, 12 in 2024-04-15T18:20:39.597 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:39.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:39.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/458341539' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:20:39.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:39 smithi159 ceph-mon[45336]: osdmap e290: 12 total, 12 up, 12 in 2024-04-15T18:20:39.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:39.730 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:39.854 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:39.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:39.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:39.989 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:40.111 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:20:40.111 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:20:40.111 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:20:40.111 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:20:40.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:40 smithi049 ceph-mon[40087]: pgmap v805: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:40.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:40 smithi082 ceph-mon[44206]: pgmap v805: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:40.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:40 smithi159 ceph-mon[45336]: pgmap v805: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:40.791 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:20:40.791 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":290,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:20:38.207343+0000","last_up_change":"2024-04-15T18:16:57.279980+0000","last_in_change":"2024-04-15T18:17:26.424473+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_2","create_time":"2024-04-15T18:20:29.399522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"290","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.92000001668930054,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6819","nonce":1108239618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6821","nonce":1108239618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6825","nonce":1108239618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1108239618},{"type":"v1","addr":"172.21.15.49:6823","nonce":1108239618}]},"public_addr":"172.21.15.49:6819/1108239618","cluster_addr":"172.21.15.49:6821/1108239618","heartbeat_back_addr":"172.21.15.49:6825/1108239618","heartbeat_front_addr":"172.21.15.49:6823/1108239618","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":177,"up_from":226,"up_thru":284,"down_at":178,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6827","nonce":1036848877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6829","nonce":1036848877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6833","nonce":1036848877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6831","nonce":1036848877}]},"public_addr":"172.21.15.49:6827/1036848877","cluster_addr":"172.21.15.49:6829/1036848877","heartbeat_back_addr":"172.21.15.49:6833/1036848877","heartbeat_front_addr":"172.21.15.49:6831/1036848877","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":279,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6809","nonce":1943742992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6811","nonce":1943742992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6815","nonce":1943742992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1943742992},{"type":"v1","addr":"172.21.15.82:6813","nonce":1943742992}]},"public_addr":"172.21.15.82:6809/1943742992","cluster_addr":"172.21.15.82:6811/1943742992","heartbeat_back_addr":"172.21.15.82:6815/1943742992","heartbeat_front_addr":"172.21.15.82:6813/1943742992","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":284,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:53.106434+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:53.378363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:20:24.243269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:20:34.805806+0000","dead_epoch":177},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:57.242810+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:20:30.321740+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:20:25.425415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:21.207683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:45.769707+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:19.268009+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:20:26.003093+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:19:39.271020+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:20:40.938 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:40.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:41.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:41.074 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:41.095 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:41.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:41.099 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:41.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:41.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:41.199 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:41.229 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:41.250 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:41.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:41.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:41.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:41.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:41.372 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-15T18:20:41.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:41 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:20:41.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3438964236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:20:41.382 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/58665562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:41.383 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:20:41.383 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:20:41.384 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:41.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:41.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:20:41.404 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:41.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:41.423 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:41.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:20:41.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:20:41.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_102124 2024-04-15T18:20:41.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:20:41.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:20:41.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:41.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:41.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:41.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:41.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:41.472 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:41.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:41 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:20:41.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3438964236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:20:41.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/58665562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:41.508 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:41.518 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:41 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:20:41.518 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3438964236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:20:41.518 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/58665562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:41.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:41.556 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:20:41.564 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:41.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:41.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:41.714 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:41.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:41.837 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:20:41.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:41.917 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:42.069 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:42.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:42.091 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:42.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:42.208 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:42.232 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:42.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:42.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:42.363 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:42.381 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:42.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:42.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 2394 2378 148.606 148.625 0.0537108 0.00598396 2024-04-15T18:20:42.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: pgmap v806: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.485 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:42.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: pgmap v806: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.523 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: pgmap v806: 17 pgs: 17 active+clean; 577 KiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:42.778 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:42.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:42.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:42.926 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:42.953 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:42.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:43.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:43.089 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:43.224 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:43.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:43.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:43.364 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:43.369 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:43.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 4420 4404 137.61 126.625 0.00167861 0.00724685 2024-04-15T18:20:43.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:43.574 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:43.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:43.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:43.679 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:43.711 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:43.753 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:43.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:43.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:43.889 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:43.893 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:43.893 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:44.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 6007 5991 124.798 99.1875 0.0170139 0.007683 2024-04-15T18:20:44.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:44 smithi082 ceph-mon[44206]: pgmap v807: 17 pgs: 17 active+clean; 83 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 165 op/s 2024-04-15T18:20:44.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:44 smithi159 ceph-mon[45336]: pgmap v807: 17 pgs: 17 active+clean; 83 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 165 op/s 2024-04-15T18:20:44.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:20:44.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:44 smithi049 ceph-mon[40087]: pgmap v807: 17 pgs: 17 active+clean; 83 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s wr, 165 op/s 2024-04-15T18:20:44.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:20:45.106 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:20:45.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 7562 7546 117.892 97.1875 0.0013584 0.00836392 2024-04-15T18:20:46.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 8956 8940 111.738 87.125 0.0094162 0.00893888 2024-04-15T18:20:46.474 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:46 smithi082 ceph-mon[44206]: pgmap v808: 17 pgs: 17 active+clean; 243 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-04-15T18:20:46.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:46 smithi159 ceph-mon[45336]: pgmap v808: 17 pgs: 17 active+clean; 243 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-04-15T18:20:46.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:46 smithi049 ceph-mon[40087]: pgmap v808: 17 pgs: 17 active+clean; 243 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 483 op/s 2024-04-15T18:20:46.945 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:46.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:47.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:47.086 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:47.147 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:20:47.147 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:20:47.189 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:47.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:47.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:47.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:47.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:47.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 10748 10732 111.78 112 0.00187024 0.00892251 2024-04-15T18:20:47.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:47.465 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:47.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:47.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:47.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:47 smithi049 ceph-mon[40087]: osdmap e291: 12 total, 12 up, 12 in 2024-04-15T18:20:47.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:47.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:47.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:47 smithi082 ceph-mon[44206]: osdmap e291: 12 total, 12 up, 12 in 2024-04-15T18:20:47.495 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:47.517 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:47.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:47.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:47.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:47.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:47.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:47 smithi159 ceph-mon[45336]: osdmap e291: 12 total, 12 up, 12 in 2024-04-15T18:20:47.661 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:47.860 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:47.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:47.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:20:47.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:48.009 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:48.214 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:48.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:48.228 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:20:48.245 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:48 smithi159 ceph-mon[45336]: pgmap v810: 17 pgs: 17 active+clean; 375 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s 2024-04-15T18:20:48.245 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:20:48.252 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:20:48.273 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:48 smithi049 ceph-mon[40087]: pgmap v810: 17 pgs: 17 active+clean; 375 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s 2024-04-15T18:20:48.273 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:20:48.317 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:48.346 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:48.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 12208 12192 108.846 91.25 0.0432494 0.00916849 2024-04-15T18:20:48.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:48 smithi082 ceph-mon[44206]: pgmap v810: 17 pgs: 17 active+clean; 375 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 666 op/s 2024-04-15T18:20:48.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:20:48.548 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:48.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:48.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:48.691 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:48.723 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:48.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:48.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:48.861 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:48.880 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:48.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:48.984 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:49.013 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:49.138 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:49.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:49.242 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:49.248 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:49.248 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:49.248 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:49 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:49.248 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:20:49.249 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:49 smithi082 ceph-mon[44206]: osdmap e292: 12 total, 12 up, 12 in 2024-04-15T18:20:49.277 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:49.302 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:49.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:49.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:49.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:49.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:49 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:49.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:20:49.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:49 smithi159 ceph-mon[45336]: osdmap e292: 12 total, 12 up, 12 in 2024-04-15T18:20:49.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:49.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:49.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:49.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:49 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:20:49.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2096819654' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:20:49.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:49 smithi049 ceph-mon[40087]: osdmap e292: 12 total, 12 up, 12 in 2024-04-15T18:20:49.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 15 13887 13872 108.364 105 0.00208047 0.00921378 2024-04-15T18:20:49.439 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:49.793 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:49.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:49.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:49.927 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:50.026 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:50.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:50.062 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:50.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:50.119 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:50.134 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:20:50.145 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:50.161 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:50.172 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:50.180 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:20:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:50.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:20:50.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:50.228 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:50.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:50.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:50.300 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:50.316 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:20:50.316 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:20:50.316 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:20:50.316 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:20:50.320 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:50 smithi159 ceph-mon[45336]: pgmap v812: 17 pgs: 17 active+clean; 565 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:20:50.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2435271843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:50.321 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:50.323 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:50.338 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:50.350 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:50.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:50.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:50.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:50.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:50.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:50.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:50.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:50.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:50.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:50.438 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 15521 15505 107.663 102.062 0.0319528 0.00924873 2024-04-15T18:20:50.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:50.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:50 smithi082 ceph-mon[44206]: pgmap v812: 17 pgs: 17 active+clean; 565 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:20:50.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2435271843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:50.478 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:50.480 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:50.498 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:50.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:50.601 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:50.632 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:50.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:50 smithi049 ceph-mon[40087]: pgmap v812: 17 pgs: 17 active+clean; 565 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:20:50.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2435271843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:50.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.084 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:51.086 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:20:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:51.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:51.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:51.220 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:51.249 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:51.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:51.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:51.388 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.411 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:51.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:51.439 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 16600 16584 103.63 67.4375 0.00490816 0.00963457 2024-04-15T18:20:51.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:20:51.467 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:51.546 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:51.713 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:51.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:51.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:51.834 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:51.853 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:52.304 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:52.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:52.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:52.437 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:52.439 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 18500 18484 105.003 118.75 0.00910618 0.00947759 2024-04-15T18:20:52.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:52.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:52 smithi082 ceph-mon[44206]: pgmap v813: 17 pgs: 17 active+clean; 760 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:20:52.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:52.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:52 smithi159 ceph-mon[45336]: pgmap v813: 17 pgs: 17 active+clean; 760 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:20:52.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:52.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:52 smithi049 ceph-mon[40087]: pgmap v813: 17 pgs: 17 active+clean; 760 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-04-15T18:20:53.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:20:53.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 20328 20312 105.772 114.25 0.00254736 0.00944193 2024-04-15T18:20:53.453 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:20:53.453 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:20:53.453 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:20:53.453 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:20:53.453 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:20:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:54.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:54.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:20:54.328 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:54.349 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:20:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:54.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:20:54.360 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:20:54.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 21705 21689 104.256 86.0625 0.00240625 0.00957886 2024-04-15T18:20:54.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:54.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:54 smithi049 ceph-mon[40087]: pgmap v814: 17 pgs: 17 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:20:54.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:20:54.486 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:54.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:54.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:54.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:54 smithi082 ceph-mon[44206]: pgmap v814: 17 pgs: 17 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:20:54.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:20:54.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:54 smithi159 ceph-mon[45336]: pgmap v814: 17 pgs: 17 active+clean; 1.0 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:20:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:20:54.649 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:54.675 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:54.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:54.714 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:20:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:20:54.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:54.808 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:54.861 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:54.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:54.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:55.009 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:55.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:55.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:55.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:20:55.168 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:55.192 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:20:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:55.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:20:55.298 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:55.329 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:55.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 22811 22795 101.746 69.125 0.0683066 0.0098084 2024-04-15T18:20:55.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:20:55.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:20:55.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: osdmap e293: 12 total, 12 up, 12 in 2024-04-15T18:20:55.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:20:55.522 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:20:55.523 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: osdmap e293: 12 total, 12 up, 12 in 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:20:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:20:55.578 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: osdmap e293: 12 total, 12 up, 12 in 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:20:55.580 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:20:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:55.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:55.686 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:20:55.723 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:55.747 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:20:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:55.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:20:55.851 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:20:55.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:56.138 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:20:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:56.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:20:56.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:56.284 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:56.333 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:56.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:56.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 24610 24594 102.458 112.438 0.0131398 0.00974656 2024-04-15T18:20:56.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:56.485 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:56.698 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:20:56.698 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:20:56.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:20:56.698 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:20:56.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33618 2.5 2:a7d6052a:::benchmark_data_smithi159_80260_object33617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.10058670700005, 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21175838599999999, 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33618 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596817+0000", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596818+0000", 2024-04-15T18:20:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596842+0000", 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3603000000000001e-05 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596852+0000", 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0225e-05 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596857+0000", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7039999999999999e-06 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596906+0000", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8968999999999998e-05 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.596985+0000", 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8145999999999993e-05 2024-04-15T18:20:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.597072+0000", 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7070999999999994e-05 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.598140+0000", 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010683349999999999 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.599600+0000", 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00145977 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808504+0000", 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20890451900000001 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808542+0000", 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7718e-05 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808575+0000", 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3439999999999998e-05 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33625 2.5 2:a2b92d8f:::benchmark_data_smithi159_80260_object33624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.09553572000004, 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206864881, 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33625 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601868+0000", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601869+0000", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4e-07 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601889+0000", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9627999999999999e-05 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601899+0000", 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0721e-05 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601905+0000", 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0580000000000003e-06 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.601955+0000", 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9453000000000002e-05 2024-04-15T18:20:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602050+0000", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4864999999999994e-05 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602141+0000", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1125999999999995e-05 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603155+0000", 2024-04-15T18:20:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010146059999999999 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.604470+0000", 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013143440000000001 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808630+0000", 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20416066499999999 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808689+0000", 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8570999999999999e-05 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808733+0000", 2024-04-15T18:20:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3903999999999998e-05 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33628 2.f 2:f7e4c22b:::benchmark_data_smithi159_80260_object33627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:20:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.094548949, 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206082972, 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33628 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602855+0000", 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602856+0000", 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7800000000000001e-07 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602877+0000", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1021000000000001e-05 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602890+0000", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3206e-05 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602898+0000", 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7230000000000004e-06 2024-04-15T18:20:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.602951+0000", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3508000000000003e-05 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603046+0000", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4559e-05 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.603185+0000", 2024-04-15T18:20:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000139169 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.604347+0000", 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011623709999999999 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613605+0000", 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092581290000000004 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808843+0000", 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19523797700000001 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808898+0000", 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5179999999999997e-05 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808938+0000", 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9351000000000003e-05 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33641 2.5 2:a8326c7a:::benchmark_data_smithi159_80260_object33640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.08598619199995, 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19743492300000001, 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33641 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611418+0000", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2900000000000003e-07 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611435+0000", 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6427e-05 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611444+0000", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5840000000000001e-06 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611450+0000", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5450000000000003e-06 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611497+0000", 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7401999999999998e-05 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611581+0000", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4124999999999998e-05 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611682+0000", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100686 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612756+0000", 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001073861 2024-04-15T18:20:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.615264+0000", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025077150000000002 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808777+0000", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19351336399999999 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808825+0000", 2024-04-15T18:20:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8340000000000001e-05 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808853+0000", 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7245000000000001e-05 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33643 2.f 2:f6b9a2f3:::benchmark_data_smithi159_80260_object33642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.08584692600004, 2024-04-15T18:20:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.197552964, 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33643 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611557+0000", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611558+0000", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.018e-06 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611580+0000", 2024-04-15T18:20:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2050999999999999e-05 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611593+0000", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2695e-05 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611601+0000", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9659999999999996e-06 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611655+0000", 2024-04-15T18:20:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4150000000000002e-05 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611758+0000", 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103466 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.611865+0000", 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106616 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613582+0000", 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017170480000000001 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613869+0000", 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000287597 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.808999+0000", 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.1951292 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809054+0000", 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5578999999999998e-05 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809110+0000", 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5578000000000002e-05 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:33644 2.f 2:f5bf3720:::benchmark_data_smithi159_80260_object33643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-04-15T18:20:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 525.08524753899997, 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19708304400000001, 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 33644 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:20:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612156+0000", 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612157+0000", 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8700000000000005e-07 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612174+0000", 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7422e-05 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612184+0000", 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2019999999999993e-06 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612189+0000", 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3890000000000004e-06 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612752+0000", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056267000000000003 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612824+0000", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2075000000000004e-05 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,9", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.612956+0000", 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013199899999999999 2024-04-15T18:20:56.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.613697+0000", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074157499999999996 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.635785+0000", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022088013 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809156+0000", 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17337069299999999 2024-04-15T18:20:56.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809203+0000", 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6782e-05 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:11.809239+0000", 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6536999999999999e-05 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70703 2.5 2:a863a10a:::benchmark_data_smithi159_80260_object70702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 478.64410108499999, 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21353573100000001, 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70703 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.738 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053303+0000", 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053304+0000", 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3900000000000003e-07 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053321+0000", 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7367000000000002e-05 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053330+0000", 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3910000000000004e-06 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053336+0000", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6339999999999998e-06 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053393+0000", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6716999999999999e-05 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053508+0000", 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114868 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.053600+0000", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2114999999999995e-05 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056695+0000", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003095155 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056727+0000", 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1698e-05 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266759+0000", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21003202100000001 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266803+0000", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.405e-05 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.266838+0000", 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5775999999999998e-05 2024-04-15T18:20:56.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70705 2.5 2:adfd5fa3:::benchmark_data_smithi159_80260_object70704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 478.64276388000002, 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21421320799999999, 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.744 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70705 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054640+0000", 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054641+0000", 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9299999999999996e-07 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054657+0000", 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5908000000000001e-05 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054666+0000", 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0550000000000005e-06 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.054671+0000", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3460000000000001e-06 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055114+0000", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044297 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055238+0000", 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000124256 2024-04-15T18:20:56.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.055390+0000", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015158000000000001 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056820+0000", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001430539 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.056957+0000", 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000136766 2024-04-15T18:20:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268718+0000", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21176034299999999 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268806+0000", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8208999999999995e-05 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268853+0000", 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7342999999999997e-05 2024-04-15T18:20:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:70708 2.5 2:acd4c471:::benchmark_data_smithi159_80260_object70707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e151)", 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 478.63829612500001, 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209906643, 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 70708 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059108+0000", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3399999999999998e-07 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059125+0000", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6895999999999999e-05 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059134+0000", 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9639999999999992e-06 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059140+0000", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2639999999999999e-06 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059178+0000", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.824e-05 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059277+0000", 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8788000000000004e-05 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.059390+0000", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011296800000000001 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.061117+0000", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017270600000000001 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.061151+0000", 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4467000000000001e-05 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268913+0000", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20776150500000001 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.268971+0000", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8091000000000003e-05 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:58.269014+0000", 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3566000000000003e-05 2024-04-15T18:20:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97473 2.5 2:a924ac74:::benchmark_data_smithi159_80260_object8360:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 450.58642570699999, 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25442546900000002, 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97473 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110978+0000", 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110979+0000", 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8400000000000003e-07 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110983+0000", 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9589999999999997e-06 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110993+0000", 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9019999999999994e-06 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.110998+0000", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5350000000000004e-06 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111039+0000", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0444000000000001e-05 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111132+0000", 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3224000000000001e-05 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111217+0000", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5006999999999994e-05 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115286+0000", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040691770000000002 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115341+0000", 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4459999999999997e-05 2024-04-15T18:20:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365290+0000", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.249949328 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365351+0000", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0683000000000002e-05 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365404+0000", 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2865999999999998e-05 2024-04-15T18:20:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97475 2.5 2:ad8d4387:::benchmark_data_smithi159_80260_object8362:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 450.586266158, 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25436893199999999, 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97475 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111138+0000", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6799999999999999e-07 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111143+0000", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8960000000000002e-06 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111153+0000", 2024-04-15T18:20:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1940000000000004e-06 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.111160+0000", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.131e-06 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.112894+0000", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017338830000000001 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.112968+0000", 2024-04-15T18:20:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4627000000000002e-05 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.113052+0000", 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3727000000000007e-05 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.115400+0000", 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023476539999999998 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.116209+0000", 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00080913800000000004 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365435+0000", 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24922602899999999 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365469+0000", 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4177000000000002e-05 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.365507+0000", 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7608000000000003e-05 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:97538 2.5 2:aaf52e94:::benchmark_data_smithi159_80260_object8425:head [delete] snapc 0=[] ondisk+write+known_if_redirected e159)", 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 450.52140410700002, 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.245409091, 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 97538 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176000+0000", 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176001+0000", 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.009e-06 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176006+0000", 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176015+0000", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7340000000000003e-06 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176022+0000", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8299999999999998e-06 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176079+0000", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6844e-05 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176206+0000", 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012738800000000001 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.176297+0000", 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0346000000000005e-05 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.178508+0000", 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022112130000000001 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.178718+0000", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00020958500000000001 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421343+0000", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.242625327 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421379+0000", 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6176000000000001e-05 2024-04-15T18:20:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:26.421409+0000", 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9726999999999998e-05 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:157471 2.5 2:a0abce61:::benchmark_data_smithi159_80260_object68358:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 378.03654858200002, 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.297938853, 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157471 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660855+0000", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660856+0000", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.274e-06 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660862+0000", 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0260000000000004e-06 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660876+0000", 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3675999999999999e-05 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.660885+0000", 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6049999999999998e-06 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664182+0000", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003297407 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664311+0000", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012873600000000001 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.664466+0000", 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015463299999999999 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672212+0000", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0077465149999999998 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.688747+0000", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016535421000000002 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958676+0000", 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26992857199999998 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958733+0000", 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6824999999999999e-05 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958794+0000", 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1162999999999997e-05 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:157472 2.5 2:a71b28ad:::benchmark_data_smithi159_80260_object68359:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:20:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 378.03171890800002, 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29329156200000001, 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157472 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665685+0000", 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665686+0000", 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6299999999999993e-07 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665691+0000", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9509999999999997e-06 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665702+0000", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0906e-05 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.665708+0000", 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5769999999999999e-06 2024-04-15T18:20:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668460+0000", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027518379999999999 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668574+0000", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113497 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668647+0000", 2024-04-15T18:20:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3139000000000006e-05 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672383+0000", 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037358439999999999 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.695755+0000", 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023372826999999999 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958854+0000", 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26309830699999998 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958912+0000", 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8384000000000002e-05 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958976+0000", 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4329e-05 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24511.0:157471 2.5 e177/114 ondisk, result = 0)", 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 378.02915380899998, 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29055099000000001, 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:20:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668250+0000", 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668251+0000", 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9299999999999997e-07 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.668258+0000", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1400000000000002e-06 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.859551+0000", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19129326699999999 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.859564+0000", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2393e-05 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958661+0000", 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.099096975000000004 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958669+0000", 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0730000000000005e-06 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958801+0000", 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000132349 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24511.0:157472 2.5 e177/114 ondisk, result = 0)", 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 378.02533251900002, 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28708425799999998, 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672071+0000", 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672072+0000", 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7199999999999998e-07 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.672080+0000", 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6590000000000007e-06 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.862774+0000", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19069477500000001 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.862783+0000", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2789999999999998e-06 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958838+0000", 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.096054971000000003 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.958846+0000", 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8540000000000003e-06 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.959156+0000", 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00030904800000000003 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:7739 3.9 3:942aaec8:::benchmark_data_smithi159_91621_object7738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 304.31006032699997, 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20308042000000001, 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 7739 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387343+0000", 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387345+0000", 2024-04-15T18:20:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0690000000000001e-06 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387371+0000", 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6942000000000001e-05 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387383+0000", 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1226000000000001e-05 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387388+0000", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7209999999999999e-06 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387433+0000", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4249000000000003e-05 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387519+0000", 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6410999999999996e-05 2024-04-15T18:20:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.387652+0000", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000133034 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.388380+0000", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00072825199999999998 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.388510+0000", 2024-04-15T18:20:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012965200000000001 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590327+0000", 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20181707700000001 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590381+0000", 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4212999999999997e-05 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:52.590424+0000", 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2574000000000003e-05 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:15742 3.9 3:98ecea81:::benchmark_data_smithi159_91621_object15741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e205)", 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 299.08536975200002, 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.210797182, 2024-04-15T18:20:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15742 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612034+0000", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612035+0000", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.299e-06 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612057+0000", 2024-04-15T18:20:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1297e-05 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612067+0000", 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0159999999999999e-05 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612073+0000", 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8769999999999998e-06 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612115+0000", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2574999999999998e-05 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612199+0000", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4005999999999995e-05 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612290+0000", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0951000000000004e-05 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.612955+0000", 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00066523400000000001 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.613378+0000", 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042226999999999998 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822703+0000", 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20932568600000001 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822768+0000", 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4869000000000005e-05 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:57.822831+0000", 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2958000000000003e-05 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:121470 3.9 3:94978a8e:::benchmark_data_smithi159_91621_object14994:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:42.900613+0000", 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 193.79679083100001, 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19723163199999999, 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 121470 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900613+0000", 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900613+0000", 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900614+0000", 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4479999999999999e-06 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900620+0000", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6609999999999997e-06 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900633+0000", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2811e-05 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900641+0000", 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0080000000000002e-06 2024-04-15T18:20:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900699+0000", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7760000000000003e-05 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900826+0000", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012727800000000001 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.900945+0000", 2024-04-15T18:20:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119562 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.903741+0000", 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027959080000000002 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:42.904515+0000", 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00077390200000000003 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.097768+0000", 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19325234299999999 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.097804+0000", 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6780999999999999e-05 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.097845+0000", 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0170000000000003e-05 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:143115 3.9 3:9757d508:::benchmark_data_smithi159_91621_object36639:head [delete] snapc 0=[] ondisk+write+known_if_redirected e249)", 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:20.211805+0000", 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 156.48559874700001, 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19915411399999999, 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:20:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 143115 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211805+0000", 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211805+0000", 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211807+0000", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.592e-06 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211814+0000", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7139999999999993e-06 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211832+0000", 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7492000000000001e-05 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.211842+0000", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7599999999999997e-06 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.212426+0000", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058400100000000001 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.212523+0000", 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7204e-05 2024-04-15T18:20:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7,11", 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.212593+0000", 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0102000000000001e-05 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.215686+0000", 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003093284 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.220042+0000", 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043553730000000001 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.410879+0000", 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19083697699999999 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.410917+0000", 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.862e-05 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:20:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:20.410959+0000", 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1995e-05 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:20:56.823 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:20:56.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:56 smithi159 ceph-mon[45336]: pgmap v816: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-15T18:20:56.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:56.825 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:56.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:56.864 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:20:56.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:20:56.965 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:56.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:56 smithi049 ceph-mon[40087]: pgmap v816: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-15T18:20:56.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:56.992 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:20:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:56.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:20:57.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:56 smithi082 ceph-mon[44206]: pgmap v816: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-04-15T18:20:57.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:20:57.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:57.133 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:57.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 25875 25859 100.996 79.0625 0.0335811 0.00984454 2024-04-15T18:20:57.489 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:20:57.490 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:57.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:57.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:57.598 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:57 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:57.621 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:57.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:20:57.642 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:57.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:57.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:20:57.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:20:57.716 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:20:57.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:20:57.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:20:57.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:20:57.773 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:57.779 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:20:57.791 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:20:57.792 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:20:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:57.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:20:57.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:20:57.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:20:57.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:20:57.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:20:57.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:57.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:57 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:57.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:57 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:20:57.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:57.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:57 smithi159 ceph-mon[45336]: osdmap e294: 12 total, 12 up, 12 in 2024-04-15T18:20:57.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:57 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:20:57.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:57.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:57 smithi049 ceph-mon[40087]: osdmap e294: 12 total, 12 up, 12 in 2024-04-15T18:20:57.930 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:20:57.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:20:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:20:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:20:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:20:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:20:57.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:20:58.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:57 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:20:58.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:57 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:20:58.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3130371314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:20:58.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:57 smithi082 ceph-mon[44206]: osdmap e294: 12 total, 12 up, 12 in 2024-04-15T18:20:58.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:58.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:58.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:58.434 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:58.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 26775 26759 98.3636 56.25 0.00993269 0.0101578 2024-04-15T18:20:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:58.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:58.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:20:58.600 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2303091262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.624 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2303091262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:58.979 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:20:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:58.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: pgmap v818: 17 pgs: 17 active+clean; 1.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2303091262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:59.112 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:59.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:59.278 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:20:59.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:20:59.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 28354 28338 98.3811 98.6875 0.00553829 0.0101546 2024-04-15T18:20:59.445 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.466 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:20:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:20:59.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:20:59.603 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.630 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:20:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:20:59.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:20:59.762 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.783 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:20:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:20:59.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:20:59.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:20:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:20:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:20:59.915 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:20:59.934 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:20:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:20:59.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:00.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:00.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:20:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:00.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:00.070 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:00.097 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:00.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:00.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:21:00.206 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:00.240 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:00.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 29725 29709 97.7128 85.6875 0.00950026 0.0102258 2024-04-15T18:21:00.524 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:00.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:00 smithi049 ceph-mon[40087]: pgmap v819: 17 pgs: 17 active+clean; 1.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-15T18:21:00.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:00 smithi159 ceph-mon[45336]: pgmap v819: 17 pgs: 17 active+clean; 1.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-15T18:21:00.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:21:01.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:00 smithi082 ceph-mon[44206]: pgmap v819: 17 pgs: 17 active+clean; 1.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-04-15T18:21:01.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:21:01.442935+0000 min lat: 0.00105766 max lat: 0.296547 avg lat: 0.0102465 2024-04-15T18:21:01.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:21:01.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 31235 31219 97.5454 94.375 0.00622973 0.0102465 2024-04-15T18:21:01.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:01.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:01.842 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:21:01.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:21:01.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:01.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:02.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:02.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:02.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 32693 32677 97.2392 91.125 0.0141924 0.0102781 2024-04-15T18:21:02.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:02.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:02.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:02 smithi159 ceph-mon[45336]: pgmap v820: 17 pgs: 17 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:21:02.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:02 smithi159 ceph-mon[45336]: osdmap e295: 12 total, 12 up, 12 in 2024-04-15T18:21:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:02 smithi049 ceph-mon[40087]: pgmap v820: 17 pgs: 17 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:21:02.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:02 smithi049 ceph-mon[40087]: osdmap e295: 12 total, 12 up, 12 in 2024-04-15T18:21:02.996 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:03.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:03.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:03.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:03.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:02 smithi082 ceph-mon[44206]: pgmap v820: 17 pgs: 17 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:21:03.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:02 smithi082 ceph-mon[44206]: osdmap e295: 12 total, 12 up, 12 in 2024-04-15T18:21:03.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:03.138 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:03.160 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:03.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:03.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:03.298 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:03.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 33946 33930 96.3787 78.3125 0.0109459 0.0103623 2024-04-15T18:21:03.449 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:03.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:03.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:03.576 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:21:03.576 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:21:03.580 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:03.598 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:03.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:03.620 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:21:03.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:03.730 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:03.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:03.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:04.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:04.265 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:21:04.280 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:04.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:04.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:04.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:04.377 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:21:04.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:04.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:04.410 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:04.426 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:04.435 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:04.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:04.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 34096 34080 92.5961 9.375 0.00251739 0.0104681 2024-04-15T18:21:04.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:04.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:04.499 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:04.523 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:04.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:04.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:04.554 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:04.571 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:04.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:04.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:04.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: pgmap v822: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:21:04.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:04.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: osdmap e296: 12 total, 12 up, 12 in 2024-04-15T18:21:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2931808860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:04.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:04.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: pgmap v822: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:21:04.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: osdmap e296: 12 total, 12 up, 12 in 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2931808860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:04.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: pgmap v822: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1899274310' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: osdmap e296: 12 total, 12 up, 12 in 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2931808860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.060 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:05.321 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:05.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:05.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:05.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 34100 34084 88.7483 0.25 0.00221014 0.0104671 2024-04-15T18:21:05.459 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:05.481 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:05.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:05.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:05.614 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.856 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.932 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.932 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:05.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:06.399 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:06.399 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:21:06.399 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:21:06.399 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:21:06.399 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:06.440 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 34100 34084 85.1985 0 - 0.0104671 2024-04-15T18:21:06.665 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:06.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:06.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:06.800 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:06 smithi159 ceph-mon[45336]: pgmap v824: 17 pgs: 17 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-04-15T18:21:06.803 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:06.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:21:06.887 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:06 smithi049 ceph-mon[40087]: pgmap v824: 17 pgs: 17 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-04-15T18:21:06.927 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:06.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:07.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:07.035 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:06 smithi082 ceph-mon[44206]: pgmap v824: 17 pgs: 17 active+clean; 2.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-04-15T18:21:07.060 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:07.162 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:07.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:07.239 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:21:07.246 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:07.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:07.299 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:07.320 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:07.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:21:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:21:07.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:07.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 34100 34084 81.9218 0 - 0.0104671 2024-04-15T18:21:07.453 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:07.473 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:07.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:07.501 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:07.577 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:07.609 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:07.632 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:07.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:07.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:07 smithi049 ceph-mon[40087]: osdmap e297: 12 total, 12 up, 12 in 2024-04-15T18:21:07.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:07.754 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:07.754 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:07.754 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:07 smithi159 ceph-mon[45336]: osdmap e297: 12 total, 12 up, 12 in 2024-04-15T18:21:07.764 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:07.818 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:07.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:07.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:21:07.922 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:07.950 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:08.035 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:08.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:08.036 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:07 smithi082 ceph-mon[44206]: osdmap e297: 12 total, 12 up, 12 in 2024-04-15T18:21:08.298 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:08.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:08.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:08.431 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:08.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 34100 34084 78.8878 0 - 0.0104671 2024-04-15T18:21:08.630 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:08.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:08.734 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:08.762 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:09.012 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:09.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:09.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:09.145 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:09.243 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:09.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:4206 2.d 2:b7208875:::benchmark_data_smithi159_80260_object4205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e124)", 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:21:09.269 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 559.63024355000005, 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.274696569, 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4206 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638347+0000", 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638348+0000", 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638371+0000", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3159e-05 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638381+0000", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9860000000000006e-06 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638387+0000", 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3789999999999997e-06 2024-04-15T18:21:09.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638432+0000", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5135000000000001e-05 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638507+0000", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4697000000000007e-05 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638591+0000", 2024-04-15T18:21:09.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4327999999999996e-05 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.638937+0000", 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034585099999999999 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.639436+0000", 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049941999999999999 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.912941+0000", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27350442800000002 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.912997+0000", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6215000000000003e-05 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:11:49.913044+0000", 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7156e-05 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.275 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62307 2.d 2:bf30746f:::benchmark_data_smithi159_80260_object62306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 513.59794495999995, 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.226374886, 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62307 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670646+0000", 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670647+0000", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0410000000000001e-06 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670664+0000", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6765e-05 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670675+0000", 2024-04-15T18:21:09.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0801999999999999e-05 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670682+0000", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2230000000000001e-06 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670742+0000", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9865999999999999e-05 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.670865+0000", 2024-04-15T18:21:09.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000123356 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.671000+0000", 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000135389 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.672916+0000", 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001916077 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673626+0000", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000709813 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.896888+0000", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22326158300000001 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.896964+0000", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5976999999999994e-05 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.897021+0000", 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62312 2.d 2:b08159eb:::benchmark_data_smithi159_80260_object62311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 513.59508717200004, 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22816421200000001, 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.282 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62312 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:21:09.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673504+0000", 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673505+0000", 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.66e-06 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673527+0000", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1418000000000001e-05 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673537+0000", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0336999999999999e-05 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673545+0000", 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0330000000000008e-06 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673654+0000", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109175 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673778+0000", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001237 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.673897+0000", 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118857 2024-04-15T18:21:09.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.675533+0000", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00163652 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.676184+0000", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00065051900000000001 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901586+0000", 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22540185099999999 2024-04-15T18:21:09.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901621+0000", 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4997999999999999e-05 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.901668+0000", 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7144000000000001e-05 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:62467 2.d 2:b6055ada:::benchmark_data_smithi159_80260_object62466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e143)", 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 513.34465443500005, 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36534789600000001, 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 62467 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923936+0000", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923938+0000", 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.238e-06 2024-04-15T18:21:09.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923957+0000", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9143999999999998e-05 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923969+0000", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1749000000000001e-05 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.923975+0000", 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7279999999999998e-06 2024-04-15T18:21:09.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924029+0000", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3616000000000003e-05 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924148+0000", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118961 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.924248+0000", 2024-04-15T18:21:09.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100073 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.926022+0000", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017739539999999999 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:35.926202+0000", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000180416 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289203+0000", 2024-04-15T18:21:09.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36300033999999998 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289241+0000", 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7866e-05 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:12:36.289284+0000", 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3810999999999998e-05 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:153095 2.d 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e176)", 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 398.11890084499998, 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25579912100000002, 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 153095 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.295 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149692+0000", 2024-04-15T18:21:09.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3999999999999999e-06 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149703+0000", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0703000000000001e-05 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149731+0000", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8107e-05 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149747+0000", 2024-04-15T18:21:09.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6135999999999999e-05 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149885+0000", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013756900000000001 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149982+0000", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6799000000000006e-05 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.150060+0000", 2024-04-15T18:21:09.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7757999999999997e-05 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.153583+0000", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035231580000000002 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.154591+0000", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010079780000000001 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405390+0000", 2024-04-15T18:21:09.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25079957800000002 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405442+0000", 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2023e-05 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405489+0000", 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6912000000000001e-05 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:156890 2.d 2:b8159b5c:::benchmark_data_smithi159_80260_object67777:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 391.71316173299999, 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23222245999999999, 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.301 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 156890 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555431+0000", 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.838e-06 2024-04-15T18:21:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555437+0000", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555453+0000", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6079e-05 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555463+0000", 2024-04-15T18:21:09.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7210000000000006e-06 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555538+0000", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4938999999999995e-05 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555631+0000", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2980999999999996e-05 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555714+0000", 2024-04-15T18:21:09.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2655000000000001e-05 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.559420+0000", 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037066360000000001 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.583952+0000", 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024531976 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787555+0000", 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.203603273 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787605+0000", 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9555000000000002e-05 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787652+0000", 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6606000000000001e-05 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95270 3.2 3:489b241f:::benchmark_data_smithi159_91621_object95269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 244.09411845100001, 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.224337434, 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95270 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174475+0000", 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1100000000000001e-06 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174500+0000", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5009999999999999e-05 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174518+0000", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8921000000000001e-05 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174529+0000", 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0518e-05 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174601+0000", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2020000000000005e-05 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174741+0000", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014028700000000001 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174933+0000", 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019125199999999999 2024-04-15T18:21:09.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.177734+0000", 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028019450000000001 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.193915+0000", 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016180439000000001 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398737+0000", 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20482228699999999 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398777+0000", 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9320999999999998e-05 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398810+0000", 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3324000000000001e-05 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95279 3.2 3:40c87ed6:::benchmark_data_smithi159_91621_object95278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 244.08321203899999, 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.221975389, 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95279 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.314 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185379+0000", 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185380+0000", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2720000000000001e-06 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185411+0000", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0752e-05 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185423+0000", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1749999999999999e-05 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185429+0000", 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1539999999999999e-06 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185484+0000", 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4839999999999997e-05 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185634+0000", 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015063399999999999 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185796+0000", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016220500000000001 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.190169+0000", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043726950000000002 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214259+0000", 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024090305999999999 2024-04-15T18:21:09.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407247+0000", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19298747799999999 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407307+0000", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9840999999999998e-05 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.407354+0000", 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7462000000000001e-05 2024-04-15T18:21:09.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95280 3.2 3:4dc79632:::benchmark_data_smithi159_91621_object95279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 244.08285149400001, 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22356709, 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.320 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95280 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185740+0000", 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5499999999999998e-07 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185758+0000", 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7938e-05 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185767+0000", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0690000000000008e-06 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185772+0000", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3059999999999997e-06 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188088+0000", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00231525 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188222+0000", 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013395799999999999 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188380+0000", 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158768 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.190273+0000", 2024-04-15T18:21:09.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018926030000000001 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214298+0000", 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024025517 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409241+0000", 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19494304000000001 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409277+0000", 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5970000000000003e-05 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409306+0000", 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9016000000000001e-05 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.326 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100487 3.2 3:4fca02bf:::benchmark_data_smithi159_91621_object100486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 238.45528460700001, 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22316820500000001, 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100487 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813306+0000", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813307+0000", 2024-04-15T18:21:09.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0070000000000001e-06 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813334+0000", 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6743000000000001e-05 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813342+0000", 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7200000000000006e-06 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813347+0000", 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3349999999999997e-06 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813410+0000", 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3004999999999997e-05 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813540+0000", 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000129791 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.813693+0000", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015294199999999999 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816455+0000", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027618970000000001 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817014+0000", 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055896799999999996 2024-04-15T18:21:09.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036401+0000", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21938772400000001 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036439+0000", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7888e-05 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.036474+0000", 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5185000000000003e-05 2024-04-15T18:21:09.332 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100490 3.2 3:40fe6861:::benchmark_data_smithi159_91621_object100489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 238.451660772, 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22209844400000001, 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.333 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100490 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816930+0000", 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816931+0000", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.029e-06 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816973+0000", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1925999999999998e-05 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816986+0000", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2619000000000001e-05 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.816994+0000", 2024-04-15T18:21:09.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2439999999999993e-06 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817044+0000", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9595000000000002e-05 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817196+0000", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015271999999999999 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.817296+0000", 2024-04-15T18:21:09.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9607999999999999e-05 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.819483+0000", 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021871479999999999 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.820743+0000", 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012595880000000001 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.038932+0000", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21818938299999999 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.038987+0000", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5034999999999997e-05 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.039029+0000", 2024-04-15T18:21:09.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1548999999999998e-05 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100493 3.2 3:4a5dce73:::benchmark_data_smithi159_91621_object100492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 238.44599404600001, 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22059732400000001, 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.339 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100493 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822597+0000", 2024-04-15T18:21:09.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822598+0000", 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.612e-06 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822654+0000", 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5065000000000002e-05 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822667+0000", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3429e-05 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822674+0000", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4440000000000002e-06 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822718+0000", 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4092999999999999e-05 2024-04-15T18:21:09.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822797+0000", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8799000000000002e-05 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822886+0000", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8824000000000002e-05 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.826954+0000", 2024-04-15T18:21:09.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040678049999999999 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.828433+0000", 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001478704 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043135+0000", 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21470222999999999 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043169+0000", 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4236000000000002e-05 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043194+0000", 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5083e-05 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:100494 3.2 3:49a7f23a:::benchmark_data_smithi159_91621_object100493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e230)", 2024-04-15T18:21:09.345 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 238.445899301, 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22061130900000001, 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 100494 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822692+0000", 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822734+0000", 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1838000000000002e-05 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822742+0000", 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0220000000000004e-06 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.822748+0000", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2330000000000003e-06 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.824941+0000", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002193188 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.825023+0000", 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1878000000000004e-05 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.825117+0000", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3385000000000001e-05 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.827131+0000", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020148190000000002 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:10.830010+0000", 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028787750000000001 2024-04-15T18:21:09.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043234+0000", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21322421999999999 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043270+0000", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6010000000000003e-05 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:11.043303+0000", 2024-04-15T18:21:09.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2441000000000003e-05 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:158593 3.0 e255/199 3:066955e5:::benchmark_data_smithi159_91621_object52117:head v 255'10025, mlcod=255'10022)", 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 141.972027631, 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.239038166, 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296564+0000", 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2600000000000001e-07 2024-04-15T18:21:09.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296571+0000", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7870000000000004e-06 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530850+0000", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.234278768 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530857+0000", 2024-04-15T18:21:09.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530914+0000", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7123000000000002e-05 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530935+0000", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1597e-05 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535555+0000", 2024-04-15T18:21:09.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046197399999999998 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535601+0000", 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6391999999999999e-05 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:185262 3.d e267/199 3:b4a17326:::benchmark_data_smithi159_91621_object78786:head v 267'11416, mlcod=267'11412)", 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 96.970251176000005, 2024-04-15T18:21:09.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23697396100000001, 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.06e-07 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298345+0000", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1680000000000003e-06 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531088+0000", 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.232742949 2024-04-15T18:21:09.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531098+0000", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5899999999999997e-06 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531159+0000", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0865000000000001e-05 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531176+0000", 2024-04-15T18:21:09.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7589e-05 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.535265+0000", 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040884110000000001 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.535314+0000", 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8782999999999997e-05 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:212948 3.2 e275/235 3:409c861b:::benchmark_data_smithi159_91621_object106472:head v 275'13490, mlcod=242'8024)", 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 54.262592797000003, 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56367495999999995, 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005999+0000", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3680000000000001e-06 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.006010+0000", 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0786e-05 2024-04-15T18:21:09.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565792+0000", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55978132000000003 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565799+0000", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.255e-06 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565863+0000", 2024-04-15T18:21:09.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4552000000000001e-05 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565893+0000", 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9329999999999999e-05 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569633+0000", 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037404080000000002 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569673+0000", 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9941000000000003e-05 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15515 4.2 4:4aec9481:::benchmark_data_smithi159_102124_object15514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.851265572999999, 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233061504, 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.365 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15515 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:21:09.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417327+0000", 2024-04-15T18:21:09.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3090000000000001e-06 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417347+0000", 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0038999999999999e-05 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417367+0000", 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0679e-05 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417375+0000", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4080000000000001e-06 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417423+0000", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8251000000000003e-05 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417512+0000", 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8677999999999994e-05 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417602+0000", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0295000000000002e-05 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.418326+0000", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00072434099999999998 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.418645+0000", 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000318762 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650261+0000", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.231616023 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650324+0000", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2487e-05 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650387+0000", 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3231999999999994e-05 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.371 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15517 4.2 4:4c161e0b:::benchmark_data_smithi159_102124_object15516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.848184552999999, 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26252755999999999, 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:09.372 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15517 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420407+0000", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3099999999999996e-07 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420426+0000", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8371000000000001e-05 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420435+0000", 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.448e-06 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420440+0000", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5149999999999997e-06 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420481+0000", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0311000000000001e-05 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420569+0000", 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8140999999999995e-05 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420654+0000", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5535e-05 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421174+0000", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00051933100000000005 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421526+0000", 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035268700000000002 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682869+0000", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26134240600000003 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682904+0000", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682934+0000", 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9674000000000001e-05 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.377 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15520 4.2 4:4779e579:::benchmark_data_smithi159_102124_object15519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.843530409, 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27309090499999999, 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15520 2024-04-15T18:21:09.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425061+0000", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8800000000000001e-07 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425080+0000", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8600000000000001e-05 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425090+0000", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9219999999999992e-06 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425096+0000", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.84e-06 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425134+0000", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8637000000000003e-05 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425212+0000", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7958000000000001e-05 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:21:09.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425297+0000", 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4568999999999996e-05 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425802+0000", 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050520399999999998 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426334+0000", 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00053202999999999996 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698085+0000", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27175072900000002 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698115+0000", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0343000000000001e-05 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698151+0000", 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6185e-05 2024-04-15T18:21:09.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25126.0:15515 4.2 e292/284 ondisk, result = 0)", 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.84258496, 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22438891599999999, 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:09.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8299999999999997e-07 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426009+0000", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.543e-06 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650191+0000", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22418169800000001 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650198+0000", 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.046e-06 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650248+0000", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9849000000000003e-05 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650255+0000", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4850000000000003e-06 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650395+0000", 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013981200000000001 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.387 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:09.388 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:09.388 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:09.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:09.389 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:09 smithi159 ceph-mon[45336]: pgmap v826: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:09.389 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:09 smithi159 ceph-mon[45336]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:09.390 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":826,"stamp":"2024-04-15T18:21:07.217174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2207908384,"num_objects":33683,"num_object_clones":0,"num_object_copies":101049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33683,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":33862,"num_write_kb":2158522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33818,"ondisk_log_size":33818,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":9493076,"kb_used_data":7371856,"kb_used_omap":61,"kb_used_meta":2121026,"kb_avail":1115448748,"statfs":{"total":1151940427776,"available":1142219517952,"internally_reserved":0,"allocated":7548780544,"data_stored":7545190552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63076,"internal_metadata":2171931036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":449511424,"num_objects":6859,"num_object_clones":0,"num_object_copies":20577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6859,"num_write_kb":438976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001259"},"pg_stats":[{"pgid":"4.f","version":"296'2116","reported_seq":2149,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.135181+0000","last_change":"2024-04-15T18:20:31.187350+0000","last_active":"2024-04-15T18:21:04.135181+0000","last_peered":"2024-04-15T18:21:04.135181+0000","last_clean":"2024-04-15T18:21:04.135181+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:21:04.135181+0000","last_undegraded":"2024-04-15T18:21:04.135181+0000","last_fullsized":"2024-04-15T18:21:04.135181+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2116,"log_dups_size":0,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:30.858721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'1985","reported_seq":2015,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.772085+0000","last_change":"2024-04-15T18:20:31.649801+0000","last_active":"2024-04-15T18:21:02.772085+0000","last_peered":"2024-04-15T18:21:02.772085+0000","last_clean":"2024-04-15T18:21:02.772085+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:21:02.772085+0000","last_undegraded":"2024-04-15T18:21:02.772085+0000","last_fullsized":"2024-04-15T18:21:02.772085+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":1985,"log_dups_size":0,"ondisk_log_size":1985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:50.981853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130088960,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1985,"num_write_kb":127040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"295'2004","reported_seq":2035,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.776403+0000","last_change":"2024-04-15T18:20:31.190942+0000","last_active":"2024-04-15T18:21:02.776403+0000","last_peered":"2024-04-15T18:21:02.776403+0000","last_clean":"2024-04-15T18:21:02.776403+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:21:02.776403+0000","last_undegraded":"2024-04-15T18:21:02.776403+0000","last_fullsized":"2024-04-15T18:21:02.776403+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2004,"log_dups_size":0,"ondisk_log_size":2004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:53.941924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"295'2059","reported_seq":2090,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.761874+0000","last_change":"2024-04-15T18:20:31.194126+0000","last_active":"2024-04-15T18:21:02.761874+0000","last_peered":"2024-04-15T18:21:02.761874+0000","last_clean":"2024-04-15T18:21:02.761874+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:21:02.761874+0000","last_undegraded":"2024-04-15T18:21:02.761874+0000","last_fullsized":"2024-04-15T18:21:02.761874+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:34.929680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134938624,"num_objects":2059,"num_object_clones":0,"num_object_copies":6177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":131776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"296'2189","reported_seq":2223,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.146371+0000","last_change":"2024-04-15T18:20:31.188255+0000","last_active":"2024-04-15T18:21:04.146371+0000","last_peered":"2024-04-15T18:21:04.146371+0000","last_clean":"2024-04-15T18:21:04.146371+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:21:04.146371+0000","last_undegraded":"2024-04-15T18:21:04.146371+0000","last_fullsized":"2024-04-15T18:21:04.146371+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2189,"log_dups_size":0,"ondisk_log_size":2189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:44.908871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143458304,"num_objects":2189,"num_object_clones":0,"num_object_copies":6567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":140096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"296'2121","reported_seq":2155,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.174823+0000","last_change":"2024-04-15T18:20:31.649834+0000","last_active":"2024-04-15T18:21:04.174823+0000","last_peered":"2024-04-15T18:21:04.174823+0000","last_clean":"2024-04-15T18:21:04.174823+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:21:04.174823+0000","last_undegraded":"2024-04-15T18:21:04.174823+0000","last_fullsized":"2024-04-15T18:21:04.174823+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2121,"log_dups_size":0,"ondisk_log_size":2121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:34.639676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139001856,"num_objects":2121,"num_object_clones":0,"num_object_copies":6363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":135744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"296'2102","reported_seq":2136,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.318966+0000","last_change":"2024-04-15T18:20:31.187477+0000","last_active":"2024-04-15T18:21:04.318966+0000","last_peered":"2024-04-15T18:21:04.318966+0000","last_clean":"2024-04-15T18:21:04.318966+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:21:04.318966+0000","last_undegraded":"2024-04-15T18:21:04.318966+0000","last_fullsized":"2024-04-15T18:21:04.318966+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:58.148518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137756672,"num_objects":2102,"num_object_clones":0,"num_object_copies":6306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2102,"num_write_kb":134528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"296'2094","reported_seq":2128,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.171230+0000","last_change":"2024-04-15T18:20:31.188527+0000","last_active":"2024-04-15T18:21:04.171230+0000","last_peered":"2024-04-15T18:21:04.171230+0000","last_clean":"2024-04-15T18:21:04.171230+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:21:04.171230+0000","last_undegraded":"2024-04-15T18:21:04.171230+0000","last_fullsized":"2024-04-15T18:21:04.171230+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:47.168660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137232384,"num_objects":2094,"num_object_clones":0,"num_object_copies":6282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2094,"num_write_kb":134016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"296'2095","reported_seq":2128,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.321444+0000","last_change":"2024-04-15T18:20:31.649944+0000","last_active":"2024-04-15T18:21:04.321444+0000","last_peered":"2024-04-15T18:21:04.321444+0000","last_clean":"2024-04-15T18:21:04.321444+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:21:04.321444+0000","last_undegraded":"2024-04-15T18:21:04.321444+0000","last_fullsized":"2024-04-15T18:21:04.321444+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:56.271634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"295'2004","reported_seq":2034,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.315687+0000","last_change":"2024-04-15T18:20:31.476084+0000","last_active":"2024-04-15T18:21:02.315687+0000","last_peered":"2024-04-15T18:21:02.315687+0000","last_clean":"2024-04-15T18:21:02.315687+0000","last_became_active":"2024-04-15T18:20:31.475717+0000","last_became_peered":"2024-04-15T18:20:31.475717+0000","last_unstale":"2024-04-15T18:21:02.315687+0000","last_undegraded":"2024-04-15T18:21:02.315687+0000","last_fullsized":"2024-04-15T18:21:02.315687+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2004,"log_dups_size":0,"ondisk_log_size":2004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:46.821671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":128256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"296'2222","reported_seq":2256,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.143610+0000","last_change":"2024-04-15T18:20:31.476020+0000","last_active":"2024-04-15T18:21:04.143610+0000","last_peered":"2024-04-15T18:21:04.143610+0000","last_clean":"2024-04-15T18:21:04.143610+0000","last_became_active":"2024-04-15T18:20:31.475694+0000","last_became_peered":"2024-04-15T18:20:31.475694+0000","last_unstale":"2024-04-15T18:21:04.143610+0000","last_undegraded":"2024-04-15T18:21:04.143610+0000","last_fullsized":"2024-04-15T18:21:04.143610+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:59.332806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145620992,"num_objects":2222,"num_object_clones":0,"num_object_copies":6666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":142208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"296'2139","reported_seq":2173,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.442503+0000","last_change":"2024-04-15T18:20:31.189251+0000","last_active":"2024-04-15T18:21:04.442503+0000","last_peered":"2024-04-15T18:21:04.442503+0000","last_clean":"2024-04-15T18:21:04.442503+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:21:04.442503+0000","last_undegraded":"2024-04-15T18:21:04.442503+0000","last_fullsized":"2024-04-15T18:21:04.442503+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:43.857517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140181504,"num_objects":2139,"num_object_clones":0,"num_object_copies":6417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2139,"num_write_kb":136896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"295'2093","reported_seq":2124,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.307460+0000","last_change":"2024-04-15T18:20:31.187024+0000","last_active":"2024-04-15T18:21:02.307460+0000","last_peered":"2024-04-15T18:21:02.307460+0000","last_clean":"2024-04-15T18:21:02.307460+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:21:02.307460+0000","last_undegraded":"2024-04-15T18:21:02.307460+0000","last_fullsized":"2024-04-15T18:21:02.307460+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2093,"log_dups_size":0,"ondisk_log_size":2093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:50.963993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137166848,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2093,"num_write_kb":133952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"296'2259","reported_seq":2293,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.163757+0000","last_change":"2024-04-15T18:20:31.188565+0000","last_active":"2024-04-15T18:21:04.163757+0000","last_peered":"2024-04-15T18:21:04.163757+0000","last_clean":"2024-04-15T18:21:04.163757+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:21:04.163757+0000","last_undegraded":"2024-04-15T18:21:04.163757+0000","last_fullsized":"2024-04-15T18:21:04.163757+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:56.716039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148045824,"num_objects":2259,"num_object_clones":0,"num_object_copies":6777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2259,"num_write_kb":144576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"295'2024","reported_seq":2053,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-04-15T18:21:02.542745+0000","last_change":"2024-04-15T18:20:31.186380+0000","last_active":"2024-04-15T18:21:02.542745+0000","last_peered":"2024-04-15T18:21:02.542745+0000","last_clean":"2024-04-15T18:21:02.542745+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:21:02.542745+0000","last_undegraded":"2024-04-15T18:21:02.542745+0000","last_fullsized":"2024-04-15T18:21:02.542745+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2024,"log_dups_size":0,"ondisk_log_size":2024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:55.494529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132644864,"num_objects":2024,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2024,"num_write_kb":129536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"296'2175","reported_seq":2209,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:04.177131+0000","last_change":"2024-04-15T18:20:31.649923+0000","last_active":"2024-04-15T18:21:04.177131+0000","last_peered":"2024-04-15T18:21:04.177131+0000","last_clean":"2024-04-15T18:21:04.177131+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:21:04.177131+0000","last_undegraded":"2024-04-15T18:21:04.177131+0000","last_fullsized":"2024-04-15T18:21:04.177131+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:33.831733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142540800,"num_objects":2175,"num_object_clones":0,"num_object_copies":6525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":139200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":796,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-04-15T18:21:03.652267+0000","last_change":"2024-04-15T18:20:19.017864+0000","last_active":"2024-04-15T18:21:03.652267+0000","last_peered":"2024-04-15T18:21:03.652267+0000","last_clean":"2024-04-15T18:21:03.652267+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:21:03.652267+0000","last_undegraded":"2024-04-15T18:21:03.652267+0000","last_fullsized":"2024-04-15T18:21:03.652267+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:20:19.017756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:20:19.017756+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:25.679635+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2207318016,"num_objects":33681,"num_object_clones":0,"num_object_copies":101043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33681,"num_write_kb":2155584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6628311040,"data_stored":6628311040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33681,"ondisk_log_size":33681,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925968,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1523300,"kb_used_data":1275544,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92221852,"statfs":{"total":95995035648,"available":94435176448,"internally_reserved":0,"allocated":1306157056,"data_stored":1305852918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":253688260},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187603,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623116,"kb_used_data":519748,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93122036,"statfs":{"total":95995035648,"available":95356964864,"internally_reserved":0,"allocated":532221952,"data_stored":531925982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":105839040},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514648,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550592,"kb_used_data":375032,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93194560,"statfs":{"total":95995035648,"available":95431229440,"internally_reserved":0,"allocated":384032768,"data_stored":383741294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874397,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122208,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622944,"statfs":{"total":95995035648,"available":95869894656,"internally_reserved":0,"allocated":802816,"data_stored":502250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168738,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571684,"kb_used_data":405848,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93173468,"statfs":{"total":95995035648,"available":95409631232,"internally_reserved":0,"allocated":415588352,"data_stored":415284234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463079,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1296544,"kb_used_data":1134872,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92448608,"statfs":{"total":95995035648,"available":94667374592,"internally_reserved":0,"allocated":1162108928,"data_stored":1161804810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418844,"kb_used_data":1225232,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92326308,"statfs":{"total":95995035648,"available":94542139392,"internally_reserved":0,"allocated":1254637568,"data_stored":1254337002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638300,"kb_used_data":408592,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93106852,"statfs":{"total":95995035648,"available":95341416448,"internally_reserved":0,"allocated":418398208,"data_stored":418097642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384012,"kb_used_data":266756,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93361140,"statfs":{"total":95995035648,"available":95601807360,"internally_reserved":0,"allocated":273158144,"data_stored":272862174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608946,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900944,"kb_used_data":637316,"kb_used_omap":44,"kb_used_meta":263571,"kb_avail":92844208,"statfs":{"total":95995035648,"available":95072468992,"internally_reserved":0,"allocated":652611584,"data_stored":652315614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":269897258},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986352,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043856,"kb_used_data":836356,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92701296,"statfs":{"total":95995035648,"available":94926127104,"internally_reserved":0,"allocated":856428544,"data_stored":856132574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724715,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419676,"kb_used_data":285776,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93325476,"statfs":{"total":95995035648,"available":95565287424,"internally_reserved":0,"allocated":292634624,"data_stored":292334058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":417595392,"data_stored":417595392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1253834752,"data_stored":1253834752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276037632,"data_stored":276037632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":674562048,"data_stored":674562048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":534511616,"data_stored":534511616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":291831808,"data_stored":291831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":839319552,"data_stored":839319552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":414253056,"data_stored":414253056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397803520,"data_stored":397803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":540344320,"data_stored":540344320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":988217344,"data_stored":988217344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:21:09.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:21:09.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 34100 34084 76.0703 0 - 0.0104671 2024-04-15T18:21:09.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:09 smithi049 ceph-mon[40087]: pgmap v826: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:09.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:09 smithi049 ceph-mon[40087]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:09.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:09 smithi082 ceph-mon[44206]: pgmap v826: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:09.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:09 smithi082 ceph-mon[44206]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:09.603 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:09.605 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:09.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:09.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:09.736 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:09.862 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:09.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:09.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:09.986 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:21:09.995 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:10.092 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:10.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:10.147 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:21:10.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:10.226 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:10.290 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:10.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:10.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 34100 34084 73.4473 0 - 0.0104671 2024-04-15T18:21:10.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:10.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:10.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:10.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:10.992 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:11.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:11 smithi049 ceph-mon[40087]: pgmap v827: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 903 op/s 2024-04-15T18:21:11.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:11 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:11.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:11.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:11 smithi049 ceph-mon[40087]: osdmap e298: 12 total, 12 up, 12 in 2024-04-15T18:21:11.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:11 smithi159 ceph-mon[45336]: pgmap v827: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 903 op/s 2024-04-15T18:21:11.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:11 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:11.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:11.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:11 smithi159 ceph-mon[45336]: osdmap e298: 12 total, 12 up, 12 in 2024-04-15T18:21:11.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 34100 34084 70.999 0 - 0.0104671 2024-04-15T18:21:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:11 smithi082 ceph-mon[44206]: pgmap v827: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 903 op/s 2024-04-15T18:21:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:11 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/687826190' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:11 smithi082 ceph-mon[44206]: osdmap e298: 12 total, 12 up, 12 in 2024-04-15T18:21:12.110 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:12.110 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:21:12.110 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:21:12.111 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:12.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 34100 34084 68.7087 0 - 0.0104671 2024-04-15T18:21:12.558 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:12.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:12.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:12.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:12.716 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:12.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:12.811 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:21:12.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:12.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:21:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:13.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:13.071 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:13.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:13.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:13.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:13.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:13 smithi049 ceph-mon[40087]: pgmap v829: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 394 op/s 2024-04-15T18:21:13.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:13 smithi049 ceph-mon[40087]: osdmap e299: 12 total, 12 up, 12 in 2024-04-15T18:21:13.284 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:13.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:13.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:13.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:21:13.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:13.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:13.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:13 smithi159 ceph-mon[45336]: pgmap v829: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 394 op/s 2024-04-15T18:21:13.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:13 smithi159 ceph-mon[45336]: osdmap e299: 12 total, 12 up, 12 in 2024-04-15T18:21:13.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:13.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 34100 34084 66.5615 0 - 0.0104671 2024-04-15T18:21:13.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:13.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:13.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:13.501 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:13.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:13.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:13 smithi082 ceph-mon[44206]: pgmap v829: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 394 op/s 2024-04-15T18:21:13.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:13 smithi082 ceph-mon[44206]: osdmap e299: 12 total, 12 up, 12 in 2024-04-15T18:21:13.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:13.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:13.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:13.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:13.588 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:13.591 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:13.623 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:13.645 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:13.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:13.679 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:21:13.681 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:21:13.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:13.778 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:14.224 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/980788142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:14.224 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.225 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.441 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 34100 34084 64.5445 0 - 0.0104671 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/980788142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.532 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2024-04-15T18:21:13.218218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2235367968,"num_objects":34102,"num_object_clones":0,"num_object_copies":102306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34102,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":34281,"num_write_kb":2185338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34237,"ondisk_log_size":34237,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":9598592,"kb_used_data":7464876,"kb_used_omap":61,"kb_used_meta":2133506,"kb_avail":1115343232,"statfs":{"total":1151940427776,"available":1142111469568,"internally_reserved":0,"allocated":7644033024,"data_stored":7640310448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63076,"internal_metadata":2184710556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":55246848,"num_objects":843,"num_object_clones":0,"num_object_copies":2529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":843,"num_write_kb":53952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.589181"},"pg_stats":[{"pgid":"4.f","version":"296'2116","reported_seq":2153,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-15T18:21:10.148072+0000","last_change":"2024-04-15T18:20:31.187350+0000","last_active":"2024-04-15T18:21:10.148072+0000","last_peered":"2024-04-15T18:21:10.148072+0000","last_clean":"2024-04-15T18:21:10.148072+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:21:10.148072+0000","last_undegraded":"2024-04-15T18:21:10.148072+0000","last_fullsized":"2024-04-15T18:21:10.148072+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2116,"log_dups_size":0,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:30.858721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"296'2043","reported_seq":2082,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.310026+0000","last_change":"2024-04-15T18:20:31.649801+0000","last_active":"2024-04-15T18:21:12.310026+0000","last_peered":"2024-04-15T18:21:12.310026+0000","last_clean":"2024-04-15T18:21:12.310026+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:21:12.310026+0000","last_undegraded":"2024-04-15T18:21:12.310026+0000","last_fullsized":"2024-04-15T18:21:12.310026+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2043,"log_dups_size":0,"ondisk_log_size":2043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:50.981853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133890048,"num_objects":2043,"num_object_clones":0,"num_object_copies":6129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":130752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"296'2050","reported_seq":2089,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.309946+0000","last_change":"2024-04-15T18:20:31.190942+0000","last_active":"2024-04-15T18:21:12.309946+0000","last_peered":"2024-04-15T18:21:12.309946+0000","last_clean":"2024-04-15T18:21:12.309946+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:21:12.309946+0000","last_undegraded":"2024-04-15T18:21:12.309946+0000","last_fullsized":"2024-04-15T18:21:12.309946+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:53.941924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"296'2115","reported_seq":2154,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.309930+0000","last_change":"2024-04-15T18:20:31.194126+0000","last_active":"2024-04-15T18:21:12.309930+0000","last_peered":"2024-04-15T18:21:12.309930+0000","last_clean":"2024-04-15T18:21:12.309930+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:21:12.309930+0000","last_undegraded":"2024-04-15T18:21:12.309930+0000","last_fullsized":"2024-04-15T18:21:12.309930+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:34.929680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":135360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"296'2189","reported_seq":2227,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-15T18:21:10.869084+0000","last_change":"2024-04-15T18:20:31.188255+0000","last_active":"2024-04-15T18:21:10.869084+0000","last_peered":"2024-04-15T18:21:10.869084+0000","last_clean":"2024-04-15T18:21:10.869084+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:21:10.869084+0000","last_undegraded":"2024-04-15T18:21:10.869084+0000","last_fullsized":"2024-04-15T18:21:10.869084+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2189,"log_dups_size":0,"ondisk_log_size":2189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:44.908871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143458304,"num_objects":2189,"num_object_clones":0,"num_object_copies":6567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":140096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"296'2121","reported_seq":2157,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:06.707816+0000","last_change":"2024-04-15T18:20:31.649834+0000","last_active":"2024-04-15T18:21:06.707816+0000","last_peered":"2024-04-15T18:21:06.707816+0000","last_clean":"2024-04-15T18:21:06.707816+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:21:06.707816+0000","last_undegraded":"2024-04-15T18:21:06.707816+0000","last_fullsized":"2024-04-15T18:21:06.707816+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2121,"log_dups_size":0,"ondisk_log_size":2121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:34.639676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139001856,"num_objects":2121,"num_object_clones":0,"num_object_copies":6363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2121,"num_write_kb":135744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"296'2102","reported_seq":2138,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:07.808959+0000","last_change":"2024-04-15T18:20:31.187477+0000","last_active":"2024-04-15T18:21:07.808959+0000","last_peered":"2024-04-15T18:21:07.808959+0000","last_clean":"2024-04-15T18:21:07.808959+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:21:07.808959+0000","last_undegraded":"2024-04-15T18:21:07.808959+0000","last_fullsized":"2024-04-15T18:21:07.808959+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:58.148518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137756672,"num_objects":2102,"num_object_clones":0,"num_object_copies":6306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2102,"num_write_kb":134528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"296'2094","reported_seq":2130,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:07.808814+0000","last_change":"2024-04-15T18:20:31.188527+0000","last_active":"2024-04-15T18:21:07.808814+0000","last_peered":"2024-04-15T18:21:07.808814+0000","last_clean":"2024-04-15T18:21:07.808814+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:21:07.808814+0000","last_undegraded":"2024-04-15T18:21:07.808814+0000","last_fullsized":"2024-04-15T18:21:07.808814+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2094,"log_dups_size":0,"ondisk_log_size":2094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:47.168660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137232384,"num_objects":2094,"num_object_clones":0,"num_object_copies":6282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2094,"num_write_kb":134016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"296'2095","reported_seq":2132,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-15T18:21:10.148159+0000","last_change":"2024-04-15T18:20:31.649944+0000","last_active":"2024-04-15T18:21:10.148159+0000","last_peered":"2024-04-15T18:21:10.148159+0000","last_clean":"2024-04-15T18:21:10.148159+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:21:10.148159+0000","last_undegraded":"2024-04-15T18:21:10.148159+0000","last_fullsized":"2024-04-15T18:21:10.148159+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:56.271634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"296'2098","reported_seq":2121,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.309523+0000","last_change":"2024-04-15T18:20:31.476084+0000","last_active":"2024-04-15T18:21:12.309523+0000","last_peered":"2024-04-15T18:21:12.309523+0000","last_clean":"2024-04-15T18:21:12.309523+0000","last_became_active":"2024-04-15T18:20:31.475717+0000","last_became_peered":"2024-04-15T18:20:31.475717+0000","last_unstale":"2024-04-15T18:21:12.309523+0000","last_undegraded":"2024-04-15T18:21:12.309523+0000","last_fullsized":"2024-04-15T18:21:12.309523+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:46.821671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137494528,"num_objects":2098,"num_object_clones":0,"num_object_copies":6294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2098,"num_write_kb":134272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"296'2222","reported_seq":2260,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-15T18:21:10.147898+0000","last_change":"2024-04-15T18:20:31.476020+0000","last_active":"2024-04-15T18:21:10.147898+0000","last_peered":"2024-04-15T18:21:10.147898+0000","last_clean":"2024-04-15T18:21:10.147898+0000","last_became_active":"2024-04-15T18:20:31.475694+0000","last_became_peered":"2024-04-15T18:20:31.475694+0000","last_unstale":"2024-04-15T18:21:10.147898+0000","last_undegraded":"2024-04-15T18:21:10.147898+0000","last_fullsized":"2024-04-15T18:21:10.147898+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:59.332806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145620992,"num_objects":2222,"num_object_clones":0,"num_object_copies":6666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":142208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"296'2139","reported_seq":2175,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:07.809025+0000","last_change":"2024-04-15T18:20:31.189251+0000","last_active":"2024-04-15T18:21:07.809025+0000","last_peered":"2024-04-15T18:21:07.809025+0000","last_clean":"2024-04-15T18:21:07.809025+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:21:07.809025+0000","last_undegraded":"2024-04-15T18:21:07.809025+0000","last_fullsized":"2024-04-15T18:21:07.809025+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:43.857517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140181504,"num_objects":2139,"num_object_clones":0,"num_object_copies":6417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2139,"num_write_kb":136896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"296'2194","reported_seq":2233,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.309523+0000","last_change":"2024-04-15T18:20:31.187024+0000","last_active":"2024-04-15T18:21:12.309523+0000","last_peered":"2024-04-15T18:21:12.309523+0000","last_clean":"2024-04-15T18:21:12.309523+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:21:12.309523+0000","last_undegraded":"2024-04-15T18:21:12.309523+0000","last_fullsized":"2024-04-15T18:21:12.309523+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:50.963993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143785984,"num_objects":2194,"num_object_clones":0,"num_object_copies":6582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2194,"num_write_kb":140416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"296'2259","reported_seq":2295,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:07.809012+0000","last_change":"2024-04-15T18:20:31.188565+0000","last_active":"2024-04-15T18:21:07.809012+0000","last_peered":"2024-04-15T18:21:07.809012+0000","last_clean":"2024-04-15T18:21:07.809012+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:21:07.809012+0000","last_undegraded":"2024-04-15T18:21:07.809012+0000","last_fullsized":"2024-04-15T18:21:07.809012+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2259,"log_dups_size":0,"ondisk_log_size":2259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:56.716039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148045824,"num_objects":2259,"num_object_clones":0,"num_object_copies":6777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2259,"num_write_kb":144576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"296'2088","reported_seq":2127,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-04-15T18:21:12.309542+0000","last_change":"2024-04-15T18:20:31.186380+0000","last_active":"2024-04-15T18:21:12.309542+0000","last_peered":"2024-04-15T18:21:12.309542+0000","last_clean":"2024-04-15T18:21:12.309542+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:21:12.309542+0000","last_undegraded":"2024-04-15T18:21:12.309542+0000","last_fullsized":"2024-04-15T18:21:12.309542+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:55.494529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136839168,"num_objects":2088,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":133632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"296'2175","reported_seq":2211,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-04-15T18:21:06.707822+0000","last_change":"2024-04-15T18:20:31.649923+0000","last_active":"2024-04-15T18:21:06.707822+0000","last_peered":"2024-04-15T18:21:06.707822+0000","last_clean":"2024-04-15T18:21:06.707822+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:21:06.707822+0000","last_undegraded":"2024-04-15T18:21:06.707822+0000","last_fullsized":"2024-04-15T18:21:06.707822+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:33.831733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142540800,"num_objects":2175,"num_object_clones":0,"num_object_copies":6525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":139200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":800,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-04-15T18:21:10.150509+0000","last_change":"2024-04-15T18:20:19.017864+0000","last_active":"2024-04-15T18:21:10.150509+0000","last_peered":"2024-04-15T18:21:10.150509+0000","last_clean":"2024-04-15T18:21:10.150509+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:21:10.150509+0000","last_undegraded":"2024-04-15T18:21:10.150509+0000","last_fullsized":"2024-04-15T18:21:10.150509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:20:19.017756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:20:19.017756+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:21:25.679635+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2234777600,"num_objects":34100,"num_object_clones":0,"num_object_copies":102300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34100,"num_write_kb":2182400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6703284224,"data_stored":6703284224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34100,"ondisk_log_size":34100,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925969,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513968,"kb_used_data":1266212,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92231184,"statfs":{"total":95995035648,"available":94444732416,"internally_reserved":0,"allocated":1296601088,"data_stored":1296292404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":253688260},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187605,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648128,"kb_used_data":540596,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93097024,"statfs":{"total":95995035648,"available":95331352576,"internally_reserved":0,"allocated":553570304,"data_stored":553256000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514650,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586420,"kb_used_data":406696,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93158732,"statfs":{"total":95995035648,"available":95394541568,"internally_reserved":0,"allocated":416456704,"data_stored":416146996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874398,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122216,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622936,"statfs":{"total":95995035648,"available":95869886464,"internally_reserved":0,"allocated":815104,"data_stored":509992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168739,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571708,"kb_used_data":405872,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93173444,"statfs":{"total":95995035648,"available":95409606656,"internally_reserved":0,"allocated":415612928,"data_stored":415299668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463080,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286956,"kb_used_data":1125284,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92458196,"statfs":{"total":95995035648,"available":94677192704,"internally_reserved":0,"allocated":1152290816,"data_stored":1151982152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575046,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418872,"kb_used_data":1225256,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92326280,"statfs":{"total":95995035648,"available":94542110720,"internally_reserved":0,"allocated":1254662144,"data_stored":1254352436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738567,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638324,"kb_used_data":408616,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93106828,"statfs":{"total":95995035648,"available":95341391872,"internally_reserved":0,"allocated":418422784,"data_stored":418113076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393536,"kb_used_data":276276,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93351616,"statfs":{"total":95995035648,"available":95592054784,"internally_reserved":0,"allocated":282906624,"data_stored":282592320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608948,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945588,"kb_used_data":681960,"kb_used_omap":44,"kb_used_meta":263571,"kb_avail":92799564,"statfs":{"total":95995035648,"available":95026753536,"internally_reserved":0,"allocated":698327040,"data_stored":698017332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":269897258},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986354,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053188,"kb_used_data":841524,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92691964,"statfs":{"total":95995035648,"available":94916571136,"internally_reserved":0,"allocated":861720576,"data_stored":861406272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724716,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419688,"kb_used_data":285788,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93325464,"statfs":{"total":95995035648,"available":95565275136,"internally_reserved":0,"allocated":292646912,"data_stored":292341800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":417595392,"data_stored":417595392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1253834752,"data_stored":1253834752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282066944,"data_stored":282066944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":697499648,"data_stored":697499648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":550305792,"data_stored":550305792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":291831808,"data_stored":291831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":839319552,"data_stored":839319552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":414253056,"data_stored":414253056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":415629312,"data_stored":415629312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":552730624,"data_stored":552730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":988217344,"data_stored":988217344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:21:14.533 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:21:14.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/980788142' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:14.626 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:14.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:14.730 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:14.761 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:14.780 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:14.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:21:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:21:14.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:14.914 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:15.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: pgmap v831: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-04-15T18:21:15.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:15 smithi159 ceph-mon[45336]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: pgmap v831: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-04-15T18:21:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:15 smithi049 ceph-mon[40087]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:15.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 34100 34084 62.6462 0 - 0.0104671 2024-04-15T18:21:15.469 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:15.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: pgmap v831: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:15.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:15 smithi082 ceph-mon[44206]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:21:15.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:21:15.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:15.602 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:15.846 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:15.850 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:15.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:15.958 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:15.990 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.011 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:16.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:16.145 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.166 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:16.270 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:16.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:16.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:16.298 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.318 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:16.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:16.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:16.421 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:16.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 34100 34084 60.8563 0 - 0.0104671 2024-04-15T18:21:16.449 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.467 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:16.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:16.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:16.571 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:16.601 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.620 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:16.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:16.751 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.777 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:16.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:16.919 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:16.938 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:16.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:17.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:17.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:21:17.200 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:17.227 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:17.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: pgmap v832: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-04-15T18:21:17.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: osdmap e300: 12 total, 12 up, 12 in 2024-04-15T18:21:17.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:17.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:17.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:21:17.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:17 smithi082 ceph-mon[44206]: osdmap e301: 12 total, 12 up, 12 in 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: pgmap v832: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: osdmap e300: 12 total, 12 up, 12 in 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:21:17.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:17 smithi159 ceph-mon[45336]: osdmap e301: 12 total, 12 up, 12 in 2024-04-15T18:21:17.425 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:17.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:17.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 34100 34084 59.1659 0 - 0.0104671 2024-04-15T18:21:17.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: pgmap v832: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: osdmap e300: 12 total, 12 up, 12 in 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2588164190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:21:17.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:17 smithi049 ceph-mon[40087]: osdmap e301: 12 total, 12 up, 12 in 2024-04-15T18:21:17.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:17.561 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:17.582 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:17.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:17.649 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:21:17.649 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:21:17.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:17.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:17.901 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:17.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:17.971 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:18.419 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:18.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:18.434 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:21:18.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 34287 34271 57.8828 0.899038 0.00265919 0.0172697 2024-04-15T18:21:18.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:18.569 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:18.630 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:18.734 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:18.765 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:18.788 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:18.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:18.797 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:18.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:18.931 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:19.065 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:19.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:19.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:19.205 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:19.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:19 smithi049 ceph-mon[40087]: pgmap v835: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:19.229 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:19.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:19.333 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:19.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:19 smithi159 ceph-mon[45336]: pgmap v835: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:19.370 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:19.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 36128 36112 59.3872 115.062 0.0126645 0.0168329 2024-04-15T18:21:19.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:19 smithi082 ceph-mon[44206]: pgmap v835: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:19.730 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:19.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:19.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:19.782 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:19.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:21:19.829 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:19.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:19.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:21:19.853 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:19.864 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:19.885 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:19.888 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:19.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:19.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:19.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:19.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:20.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:20.028 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:20.052 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:20.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:20.126 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:21:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:20.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:20.157 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:20.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:20.207 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:20.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:20.295 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:20.350 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:20.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:20.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1887966615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:20.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 37278 37262 59.7072 71.875 0.0070749 0.0167435 2024-04-15T18:21:20.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:20.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1887966615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.485 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1887966615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:20.485 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.485 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:20.489 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:20.510 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:20.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:20.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:20.645 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:20.669 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:20.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:20.773 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:20.801 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:20.824 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:37753 4.3 4:cc9180ae:::benchmark_data_smithi159_102124_object37752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:20.814024+0000", 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.011909436000000001, 2024-04-15T18:21:20.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011947638999999999, 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 37753 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814024+0000", 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814024+0000", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814025+0000", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0500000000000002e-07 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814053+0000", 2024-04-15T18:21:20.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7514000000000001e-05 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814060+0000", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5870000000000004e-06 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.814065+0000", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9239999999999998e-06 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.815583+0000", 2024-04-15T18:21:20.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015179670000000001 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.815696+0000", 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011295599999999999 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.815817+0000", 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120708 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:20.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.817058+0000", 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012411309999999999 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.817445+0000", 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00038653199999999998 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:37757 4.9 4:97241396:::benchmark_data_smithi159_102124_object37756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:20.831 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:20.819216+0000", 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0067176839999999998, 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068617330000000001, 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 37757 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:20.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819216+0000", 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819216+0000", 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819217+0000", 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0839999999999999e-06 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819240+0000", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2351e-05 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819254+0000", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4327e-05 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819263+0000", 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9660000000000002e-06 2024-04-15T18:21:20.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819314+0000", 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1405e-05 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819457+0000", 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000142695 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,3", 2024-04-15T18:21:20.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.819606+0000", 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014872299999999999 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821401+0000", 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001795584 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:37759 4.1 4:8c62c3d4:::benchmark_data_smithi159_102124_object37758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:20.821478+0000", 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0044561590000000003, 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046980629999999997, 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:20.837 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 37759 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821478+0000", 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821478+0000", 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821479+0000", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-06 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821528+0000", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9496000000000002e-05 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821543+0000", 2024-04-15T18:21:20.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4616e-05 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821552+0000", 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3540000000000006e-06 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821611+0000", 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8723e-05 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:20.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821760+0000", 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000148517 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,6", 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.821930+0000", 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017018400000000001 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823741+0000", 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001811542 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:37761 4.8 4:1fa36458:::benchmark_data_smithi159_102124_object37760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:20.823442+0000", 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.002491989, 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028249780000000001, 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:20.842 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 37761 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823442+0000", 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823442+0000", 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823443+0000", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.142e-06 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823498+0000", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4947000000000001e-05 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823513+0000", 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4927e-05 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823522+0000", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8689999999999993e-06 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823583+0000", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1464999999999995e-05 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823736+0000", 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015271100000000001 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,7", 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.823894+0000", 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015832500000000001 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:37762 4.1 4:85dc354c:::benchmark_data_smithi159_102124_object37761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:20.825309+0000", 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.00062506699999999998, 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010415730000000001, 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:20.846 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 37762 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825309+0000", 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825309+0000", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825310+0000", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9099999999999991e-07 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825363+0000", 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3170000000000001e-05 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825373+0000", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0360000000000001e-05 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825381+0000", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8439999999999998e-06 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825421+0000", 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0250999999999998e-05 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825507+0000", 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5226e-05 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,6", 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:20.825591+0000", 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4484e-05 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:20.850 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:21:20.851 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 5 2024-04-15T18:21:20.851 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:20.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:20.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:20.990 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:21.141 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:21.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:21.246 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:21.278 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:21.302 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:21.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: pgmap v836: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.405 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.406 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:21.438 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:21.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:21:21.445065+0000 min lat: 0.00105766 max lat: 15.0119 avg lat: 0.0164876 2024-04-15T18:21:21.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:21:21.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 38824 38808 60.6299 96.625 0.0137954 0.0164876 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: pgmap v836: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: pgmap v836: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:21.691 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:21.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:21.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:21:21.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:21.829 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:21.850 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:21.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:21.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:21.990 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:22.016 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:22.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:22.091 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:22.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:22.147 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:22.166 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:22.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:22.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:22.296 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:22.320 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:22.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:22.426 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:22.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 39898 39882 60.7882 67.125 0.0471527 0.0164161 2024-04-15T18:21:22.453 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:22.902 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:22.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:23.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:23.038 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:23.330 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:23 smithi159 ceph-mon[45336]: pgmap v837: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s 2024-04-15T18:21:23.330 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:23 smithi159 ceph-mon[45336]: osdmap e302: 12 total, 12 up, 12 in 2024-04-15T18:21:23.343 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:23.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:23.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 41233 41217 61.3273 83.4375 0.00779275 0.0163001 2024-04-15T18:21:23.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:23.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:23 smithi049 ceph-mon[40087]: pgmap v837: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s 2024-04-15T18:21:23.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:23 smithi049 ceph-mon[40087]: osdmap e302: 12 total, 12 up, 12 in 2024-04-15T18:21:23.489 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:23.514 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:23.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:23.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:23 smithi082 ceph-mon[44206]: pgmap v837: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s 2024-04-15T18:21:23.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:23 smithi082 ceph-mon[44206]: osdmap e302: 12 total, 12 up, 12 in 2024-04-15T18:21:23.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:23.654 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:23.680 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:23.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:23.785 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:23.812 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:23.833 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:23.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:23.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:23.938 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:21:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:21:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:21:23.938 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:23.966 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:23.988 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:23.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:24.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:21:24.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:24.132 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:24.172 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:24.172 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:21:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:24.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:24.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:24.324 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:24.329 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3683673618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.329 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:24 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.432 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:24.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3683673618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:24 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:24.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:24.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 42521 42505 61.773 80.5 0.0171141 0.0161817 2024-04-15T18:21:24.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3683673618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:24 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:24.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:24.574 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:24.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:21:24.918 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:24.930 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:24.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:25.020 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:21:25.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:25.085 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:25.176 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:25.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:25.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:25.328 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:25.394 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:25.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:25.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:25 smithi159 ceph-mon[45336]: pgmap v839: 17 pgs: 17 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 641 op/s 2024-04-15T18:21:25.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:25 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:21:25.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:25 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:21:25.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:25 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:25.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:25 smithi159 ceph-mon[45336]: osdmap e303: 12 total, 12 up, 12 in 2024-04-15T18:21:25.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:25 smithi049 ceph-mon[40087]: pgmap v839: 17 pgs: 17 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 641 op/s 2024-04-15T18:21:25.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:25 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:21:25.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:25 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:21:25.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:25 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:25.419 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:25 smithi049 ceph-mon[40087]: osdmap e303: 12 total, 12 up, 12 in 2024-04-15T18:21:25.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 15 44025 44010 62.5067 94.0625 0.00231502 0.0159924 2024-04-15T18:21:25.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:25.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:25 smithi082 ceph-mon[44206]: pgmap v839: 17 pgs: 17 active+clean; 2.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 641 op/s 2024-04-15T18:21:25.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:25 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:21:25.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:25 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:21:25.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:25 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:25.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:25 smithi082 ceph-mon[44206]: osdmap e303: 12 total, 12 up, 12 in 2024-04-15T18:21:25.540 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:25.900 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:25.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:26.006 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:26.040 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:26.069 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:26.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:26.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:26.205 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:26.358 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:26.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:26.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 45568 45552 63.2591 96.375 0.00323053 0.0158003 2024-04-15T18:21:26.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:26.495 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:26.700 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:26.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:26.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:26.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:26.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:21:26.953 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:26.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127622 2.a 2:5c0e539e:::benchmark_data_smithi159_80260_object38509:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 448.91548575600001, 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24240418699999999, 2024-04-15T18:21:27.024 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127622 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106787+0000", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106788+0000", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.065e-06 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106793+0000", 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4149999999999998e-06 2024-04-15T18:21:27.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106807+0000", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3335e-05 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.106815+0000", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7059999999999993e-06 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107273+0000", 2024-04-15T18:21:27.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00045806500000000001 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107377+0000", 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103539 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.107453+0000", 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6493000000000001e-05 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.110356+0000", 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029024250000000001 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.115580+0000", 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052237230000000004 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349093+0000", 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233513357 2024-04-15T18:21:27.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349144+0000", 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0858e-05 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.349191+0000", 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7206000000000002e-05 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127639 2.a 2:50636722:::benchmark_data_smithi159_80260_object38526:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 448.88296449199999, 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233206792, 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127639 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.031 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139308+0000", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139310+0000", 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.657e-06 2024-04-15T18:21:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139315+0000", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.305e-06 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139330+0000", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4800000000000001e-05 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139337+0000", 2024-04-15T18:21:27.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.703e-06 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139389+0000", 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2540000000000002e-05 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139474+0000", 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4976999999999996e-05 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.139542+0000", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8164000000000007e-05 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.142496+0000", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029532320000000001 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.142583+0000", 2024-04-15T18:21:27.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7182000000000007e-05 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372433+0000", 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22984985399999999 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372478+0000", 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5942000000000002e-05 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.372515+0000", 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6436000000000001e-05 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:127653 2.a 2:53274d6c:::benchmark_data_smithi159_80260_object38540:head [delete] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 448.86044438800002, 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23201696699999999, 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.037 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 127653 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161828+0000", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161830+0000", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.451e-06 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161836+0000", 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4060000000000002e-06 2024-04-15T18:21:27.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161850+0000", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.419e-05 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161857+0000", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1250000000000004e-06 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.161915+0000", 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8025000000000001e-05 2024-04-15T18:21:27.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162004+0000", 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8350000000000006e-05 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.162072+0000", 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8047999999999996e-05 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.165030+0000", 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029578320000000001 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.165619+0000", 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058962999999999995 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393773+0000", 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22815326699999999 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393811+0000", 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8380000000000002e-05 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:13:58.393845+0000", 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4263e-05 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.043 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:133604 2.c 2:313518bc:::benchmark_data_smithi159_80260_object44491:head [delete] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 441.76913096599998, 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23184059300000001, 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 133604 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253142+0000", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253143+0000", 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6300000000000001e-06 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253149+0000", 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.293e-06 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253162+0000", 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3256e-05 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253170+0000", 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8539999999999997e-06 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253224+0000", 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4737999999999997e-05 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253344+0000", 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119697 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.253453+0000", 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109275 2024-04-15T18:21:27.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.256826+0000", 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033722219999999998 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.257862+0000", 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010360670000000001 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484901+0000", 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.227039084 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484942+0000", 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1544000000000001e-05 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.484982+0000", 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9932999999999999e-05 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:5358 3.0 3:093e027e:::benchmark_data_smithi159_91621_object5357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e204)", 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 336.01718497399997, 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27962654100000001, 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.051 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 5358 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005088+0000", 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005104+0000", 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6022999999999999e-05 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005109+0000", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9779999999999996e-06 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005113+0000", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9790000000000004e-06 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005140+0000", 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.654e-05 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005216+0000", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6112000000000006e-05 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005297+0000", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0896999999999995e-05 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.005719+0000", 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042199799999999998 2024-04-15T18:21:27.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.011523+0000", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058042659999999998 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284648+0000", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.273124747 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284691+0000", 2024-04-15T18:21:27.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3671000000000003e-05 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:51.284714+0000", 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2809000000000001e-05 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:113491 3.0 3:04647499:::benchmark_data_smithi159_91621_object7015:head [delete] snapc 0=[] ondisk+write+known_if_redirected e233)", 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:21:27.057 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.66074334199999, 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51810229799999996, 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113491 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:21:27.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361529+0000", 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361530+0000", 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.78e-07 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361535+0000", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8110000000000002e-06 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361547+0000", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1824999999999999e-05 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361555+0000", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4740000000000005e-06 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361610+0000", 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4533000000000001e-05 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361681+0000", 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1631000000000006e-05 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.361742+0000", 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0473999999999997e-05 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.364473+0000", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027306689999999998 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.367892+0000", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034197569999999998 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879504+0000", 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.511612132 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879569+0000", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4632000000000001e-05 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879632+0000", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2482000000000002e-05 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:113491 3.0 e233/199 ondisk, result = 0)", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:21:27.063 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.65383313999999, 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51119831599999999, 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368439+0000", 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368440+0000", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4600000000000005e-07 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.368443+0000", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1939999999999998e-06 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879387+0000", 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51094377400000002 2024-04-15T18:21:27.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879398+0000", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1049999999999999e-05 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879485+0000", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6513000000000003e-05 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879497+0000", 2024-04-15T18:21:27.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.221e-05 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.879638+0000", 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014102900000000001 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:113549 3.0 3:0fbc0d3a:::benchmark_data_smithi159_91621_object7073:head [delete] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.585403235, 2024-04-15T18:21:27.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64867516599999997, 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113549 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436869+0000", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436871+0000", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436877+0000", 2024-04-15T18:21:27.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4999999999999999e-06 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436888+0000", 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1647e-05 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436896+0000", 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1410000000000005e-06 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.436946+0000", 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9110999999999998e-05 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.437042+0000", 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6496000000000006e-05 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.437121+0000", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8616999999999996e-05 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439983+0000", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028622880000000002 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.440106+0000", 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012289899999999999 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085464+0000", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64535768299999996 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085502+0000", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8847999999999998e-05 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085545+0000", 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2194999999999998e-05 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.073 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:113549 3.0 e234/199 ondisk, result = 0)", 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.58245824299999, 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64573638099999997, 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439814+0000", 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439815+0000", 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.44e-07 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:26.439820+0000", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9359999999999998e-06 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085391+0000", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64557072699999996 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085398+0000", 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9e-06 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085452+0000", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4095000000000003e-05 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085460+0000", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8630000000000008e-06 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:27.085551+0000", 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1216000000000002e-05 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:121649 3.e 3:70d1154d:::benchmark_data_smithi159_91621_object15173:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:43.298659+0000", 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 223.72361338900001, 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21892636600000001, 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 121649 2024-04-15T18:21:27.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298659+0000", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298659+0000", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298660+0000", 2024-04-15T18:21:27.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6500000000000008e-07 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298665+0000", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6330000000000004e-06 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298676+0000", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0701e-05 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298684+0000", 2024-04-15T18:21:27.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0099999999999995e-06 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298735+0000", 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1600000000000001e-05 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298855+0000", 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119403 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.298936+0000", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1129000000000002e-05 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.301818+0000", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028819169999999999 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.302639+0000", 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00082123899999999998 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.517507+0000", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21486840099999999 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.517538+0000", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0632000000000001e-05 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:43.517586+0000", 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7735999999999998e-05 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.083 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:158589 3.0 3:058de406:::benchmark_data_smithi159_91621_object52113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e255)", 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.290371+0000", 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 159.731901849, 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24544333300000001, 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158589 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290371+0000", 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290371+0000", 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290371+0000", 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4199999999999995e-07 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290375+0000", 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0069999999999999e-06 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290384+0000", 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.952e-06 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290392+0000", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8280000000000003e-06 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290429+0000", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6602e-05 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290508+0000", 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8845999999999997e-05 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.290579+0000", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1100999999999996e-05 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.294609+0000", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00402995 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300914+0000", 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063051720000000004 2024-04-15T18:21:27.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535713+0000", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.234798756 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535767+0000", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4262999999999999e-05 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535814+0000", 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7213999999999999e-05 2024-04-15T18:21:27.089 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:158589 3.0 e255/199 ondisk, result = 0)", 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.295312+0000", 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 159.72696043100001, 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.240508057, 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.295312+0000", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.295312+0000", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.295313+0000", 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5499999999999998e-07 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.295316+0000", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5870000000000001e-06 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535653+0000", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24033697900000001 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535659+0000", 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4299999999999997e-06 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535698+0000", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9042999999999999e-05 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535706+0000", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4479999999999994e-06 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535820+0000", 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114015 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.093 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:158593 3.0 3:066955e5:::benchmark_data_smithi159_91621_object52117:head [delete] snapc 0=[] ondisk+write+known_if_redirected e255)", 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.296256+0000", 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 159.726016792, 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.245647645, 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:21:27.094 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158593 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296256+0000", 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296256+0000", 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296257+0000", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0729999999999999e-06 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296261+0000", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5619999999999997e-06 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296271+0000", 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296276+0000", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2499999999999997e-06 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296312+0000", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6464999999999997e-05 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296413+0000", 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100784 2024-04-15T18:21:27.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296494+0000", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0632999999999999e-05 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300293+0000", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037990649999999999 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541616+0000", 2024-04-15T18:21:27.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24132332500000001 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541833+0000", 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00021652999999999999 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541866+0000", 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3643000000000003e-05 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541903+0000", 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7169999999999998e-05 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.24793.0:158593 3.0 e255/199 ondisk, result = 0)", 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.300650+0000", 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 159.72162273500001, 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.241259165, 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300650+0000", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300650+0000", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300656+0000", 2024-04-15T18:21:27.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4629999999999998e-06 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.300667+0000", 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0543999999999999e-05 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541793+0000", 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24112639199999999 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541797+0000", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4539999999999999e-06 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541827+0000", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9978e-05 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541830+0000", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.428e-06 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.541909+0000", 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8906000000000007e-05 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15513 4.8 4:15de59ea:::benchmark_data_smithi159_102124_object15512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.414139+0000", 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.608133314, 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22293015299999999, 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.104 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15513 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414139+0000", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414139+0000", 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414140+0000", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8100000000000002e-07 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414158+0000", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7909000000000001e-05 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414167+0000", 2024-04-15T18:21:27.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2310000000000002e-06 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414172+0000", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1370000000000001e-06 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414210+0000", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8083000000000001e-05 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414287+0000", 2024-04-15T18:21:27.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6558999999999997e-05 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,7", 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414383+0000", 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6137999999999993e-05 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.414970+0000", 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058682999999999999 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.542374+0000", 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.127404304 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.636987+0000", 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.094613194999999997 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.637025+0000", 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7675000000000001e-05 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.637069+0000", 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4410999999999999e-05 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:15517 4.2 e292/284 4:4c161e0b:::benchmark_data_smithi159_102124_object15516:head v 292'1005, mlcod=292'1000)", 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.420739+0000", 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.601533299000003, 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229377042, 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.110 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420739+0000", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420739+0000", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420740+0000", 2024-04-15T18:21:27.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9200000000000001e-07 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420760+0000", 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0378999999999999e-05 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.649285+0000", 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22852428399999999 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.649292+0000", 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5429999999999996e-06 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.649346+0000", 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4284999999999999e-05 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.649380+0000", 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3654e-05 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650067+0000", 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000686876 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650116+0000", 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9428999999999997e-05 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15519 4.3 4:cd27a75b:::benchmark_data_smithi159_102124_object15518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:21:27.114 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.423562+0000", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.598710273999998, 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23134848099999999, 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15519 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423562+0000", 2024-04-15T18:21:27.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423562+0000", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423563+0000", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1999999999999998e-07 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423582+0000", 2024-04-15T18:21:27.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8502999999999999e-05 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423592+0000", 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559e-05 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423601+0000", 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0499999999999997e-06 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423644+0000", 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3090000000000002e-05 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423735+0000", 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0381000000000001e-05 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.423822+0000", 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7465000000000004e-05 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.424454+0000", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00063164099999999995 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.424897+0000", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044302099999999999 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.654838+0000", 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22994076099999999 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.654879+0000", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0849000000000001e-05 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.654911+0000", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2240999999999998e-05 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:15520 4.2 e292/284 4:4779e579:::benchmark_data_smithi159_102124_object15519:head v 292'1006, mlcod=292'1001)", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.425380+0000", 2024-04-15T18:21:27.120 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.596892826000001, 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25736304700000001, 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425380+0000", 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425380+0000", 2024-04-15T18:21:27.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425381+0000", 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.06e-07 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425399+0000", 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8780999999999999e-05 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.681922+0000", 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25652229999999998 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.681929+0000", 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7880000000000007e-06 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.681984+0000", 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4577999999999998e-05 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682006+0000", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2537000000000001e-05 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682705+0000", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069873199999999998 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682743+0000", 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7625e-05 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.124 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:26053 4.8 4:1ab7d31f:::benchmark_data_smithi159_102124_object26052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e294)", 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:57.676979+0000", 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.345293855000001, 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23910674600000001, 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 26053 2024-04-15T18:21:27.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.676979+0000", 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.676979+0000", 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.676979+0000", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4799999999999998e-07 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677013+0000", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3973999999999997e-05 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677021+0000", 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2960000000000001e-06 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677025+0000", 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2679999999999996e-06 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677066+0000", 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1072000000000002e-05 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677140+0000", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3548999999999997e-05 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 6,7", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677224+0000", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4189000000000003e-05 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.677906+0000", 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068191900000000001 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.705948+0000", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.028041894000000001 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.915995+0000", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21004761199999999 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.916048+0000", 2024-04-15T18:21:27.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2621000000000003e-05 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.916086+0000", 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7703999999999997e-05 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:39780 4.1 4:86de28a1:::benchmark_data_smithi159_102124_object39779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:22.078213+0000", 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.9440596079999999, 2024-04-15T18:21:27.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24104384600000001, 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 39780 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078213+0000", 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078213+0000", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078215+0000", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.531e-06 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078244+0000", 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9079000000000001e-05 2024-04-15T18:21:27.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078255+0000", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1771e-05 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078261+0000", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0800000000000002e-06 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078317+0000", 2024-04-15T18:21:27.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5024e-05 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078422+0000", 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010581000000000001 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,6", 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078519+0000", 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6972000000000006e-05 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080236+0000", 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017164579999999999 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080434+0000", 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019849399999999999 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.319149+0000", 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23871477799999999 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.319206+0000", 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6728000000000003e-05 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.319257+0000", 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1121000000000001e-05 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:21:27.137 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:21:27.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:27.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:27.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:27 smithi049 ceph-mon[40087]: pgmap v841: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 768 op/s 2024-04-15T18:21:27.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 47111 47095 63.9802 96.4375 0.0252786 0.0156237 2024-04-15T18:21:27.453 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:27 smithi159 ceph-mon[45336]: pgmap v841: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 768 op/s 2024-04-15T18:21:27.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:27 smithi082 ceph-mon[44206]: pgmap v841: 17 pgs: 17 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 768 op/s 2024-04-15T18:21:27.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:21:28.028 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:28.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:28.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:28.161 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:28.181 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:28.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:28.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:28.316 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:28.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 48347 48331 64.2624 77.25 0.014802 0.0155527 2024-04-15T18:21:28.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:28 smithi049 ceph-mon[40087]: osdmap e304: 12 total, 12 up, 12 in 2024-04-15T18:21:28.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:28 smithi082 ceph-mon[44206]: osdmap e304: 12 total, 12 up, 12 in 2024-04-15T18:21:28.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:28 smithi159 ceph-mon[45336]: osdmap e304: 12 total, 12 up, 12 in 2024-04-15T18:21:29.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 49570 49554 64.5159 76.4375 0.00468192 0.0154949 2024-04-15T18:21:29.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:29 smithi049 ceph-mon[40087]: pgmap v843: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:21:29.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:29 smithi082 ceph-mon[44206]: pgmap v843: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:21:29.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:29 smithi159 ceph-mon[45336]: pgmap v843: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:21:30.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 50762 50746 64.7195 74.5 0.00282072 0.0154431 2024-04-15T18:21:30.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:30.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:30 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:30.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:30 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:30.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:30 smithi049 ceph-mon[40087]: osdmap e305: 12 total, 12 up, 12 in 2024-04-15T18:21:30.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:30 smithi049 ceph-mon[40087]: pgmap v845: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:30.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:30.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:30 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:30.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:30 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:30.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:30 smithi082 ceph-mon[44206]: osdmap e305: 12 total, 12 up, 12 in 2024-04-15T18:21:30.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:30 smithi082 ceph-mon[44206]: pgmap v845: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:30.546 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:30.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:21:30.617 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:30.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:30 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:30.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:30 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:30.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:30 smithi159 ceph-mon[45336]: osdmap e305: 12 total, 12 up, 12 in 2024-04-15T18:21:30.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:30 smithi159 ceph-mon[45336]: pgmap v845: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:21:31.169 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:21:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:21:31.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:31.308 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:31.331 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:31.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:31.393 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:31.432 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:21:31.436 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:31.442 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 51925 51909 64.8788 72.6875 0.00501769 0.0154072 2024-04-15T18:21:31.463 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:31.468 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:31.491 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:31.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:31.526 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:31.562 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:31.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:31.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:31.626 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:31.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:31.652 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:31.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:31.663 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:31.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:31.736 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:31.757 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:31.788 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:31.808 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:31.914 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:31.942 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:31.962 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:31.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:32.066 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:32.093 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:32.114 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:32.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:32.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:32.253 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:32.413 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:32.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:32.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 52939 52923 64.8492 63.375 0.0121995 0.0154146 2024-04-15T18:21:32.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:32.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: pgmap v846: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:21:32.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2909561710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:32.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:32 smithi082 ceph-mon[44206]: osdmap e306: 12 total, 12 up, 12 in 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: pgmap v846: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2909561710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.548 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:32 smithi159 ceph-mon[45336]: osdmap e306: 12 total, 12 up, 12 in 2024-04-15T18:21:32.549 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:32.575 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:32.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:32.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: pgmap v846: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:21:32.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2909561710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:32.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:32.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:32 smithi049 ceph-mon[40087]: osdmap e306: 12 total, 12 up, 12 in 2024-04-15T18:21:32.678 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:32.709 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:32.731 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:32.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:32.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:32.877 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.010 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:33.115 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:33.141 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.160 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:33.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:33.296 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.320 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:33.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: 4.f scrub starts 2024-04-15T18:21:33.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:33 smithi159 ceph-mon[45336]: 4.f scrub ok 2024-04-15T18:21:33.426 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:33.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 54198 54182 65.1152 78.6875 0.00485437 0.0153504 2024-04-15T18:21:33.458 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.482 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:21:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:21:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: 4.f scrub starts 2024-04-15T18:21:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:33 smithi082 ceph-mon[44206]: 4.f scrub ok 2024-04-15T18:21:33.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:21:33.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:21:33.586 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:33.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: 4.f scrub starts 2024-04-15T18:21:33.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:33 smithi049 ceph-mon[40087]: 4.f scrub ok 2024-04-15T18:21:33.587 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:33.616 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.638 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:33.742 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:33.773 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.797 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:33.903 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:33.936 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:33.958 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:33.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:34.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:34.100 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.124 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:34.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:34.264 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.289 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:34.310 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:21:34.393 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:34.424 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 55514 55498 65.4382 82.25 0.0224142 0.0152744 2024-04-15T18:21:34.449 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:34.534 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-15T18:21:34.535 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:21:34.535 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-15T18:21:34.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2 2024-04-15T18:21:34.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:34 smithi082 ceph-mon[44206]: pgmap v848: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:21:34.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2811724683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:34 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.553 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:34.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:34 smithi049 ceph-mon[40087]: pgmap v848: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:21:34.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2811724683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:34 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.581 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:34 smithi159 ceph-mon[45336]: pgmap v848: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-04-15T18:21:34.581 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2811724683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.581 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:34 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:21:34.583 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.607 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:21:34.699 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:21:34.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:34.748 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.775 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:34.880 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:34.917 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:34.945 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:34.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:34.972 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:21:34 smithi049 systemd[1]: Stopping Ceph osd.2 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:21:35.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:35.090 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:35.139 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:35.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:35.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:35.282 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:35.310 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:35.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:35.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:35.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 56838 56822 65.7586 82.75 0.0296799 0.0152018 2024-04-15T18:21:35.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:35 smithi049 ceph-mon[40087]: 4.5 deep-scrub starts 2024-04-15T18:21:35.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:35 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:35.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:35 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:35.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:35 smithi049 ceph-mon[40087]: osdmap e307: 12 total, 12 up, 12 in 2024-04-15T18:21:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:35.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:35 smithi082 ceph-mon[44206]: 4.5 deep-scrub starts 2024-04-15T18:21:35.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:35 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:35.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:35 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:35.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:35 smithi082 ceph-mon[44206]: osdmap e307: 12 total, 12 up, 12 in 2024-04-15T18:21:35.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:35 smithi159 ceph-mon[45336]: 4.5 deep-scrub starts 2024-04-15T18:21:35.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:35 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:35.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:35 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:21:35.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:35 smithi159 ceph-mon[45336]: osdmap e307: 12 total, 12 up, 12 in 2024-04-15T18:21:35.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:21:36.307 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:36.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:36.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:36.442 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:36.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 57720 57704 65.5652 55.125 0.111748 0.0152453 2024-04-15T18:21:36.465 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:36.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:36.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:36.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:36 smithi049 ceph-mon[40087]: pgmap v850: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-15T18:21:36.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:36.609 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:36.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:36 smithi159 ceph-mon[45336]: pgmap v850: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-15T18:21:36.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:36 smithi082 ceph-mon[44206]: pgmap v850: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-04-15T18:21:36.865 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:21:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[74744]: 2024-04-15T18:21:36.571+0000 7f0cff72c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:21:36.865 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:21:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[74744]: 2024-04-15T18:21:36.571+0000 7f0cff72c700 -1 osd.2 307 *** Got signal Terminated *** 2024-04-15T18:21:36.865 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:21:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[74744]: 2024-04-15T18:21:36.571+0000 7f0cff72c700 -1 osd.2 307 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:21:36.870 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:36.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:36.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:37.010 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:37.031 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:37.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:37.138 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:37.173 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:37.195 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:37.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:37.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:37.325 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:37.343 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:37.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:37.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 58512 58496 65.2782 49.5 0.0294805 0.0152567 2024-04-15T18:21:37.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:37.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:37 smithi049 ceph-mon[40087]: osd.2 marked itself down and dead 2024-04-15T18:21:37.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:37 smithi049 ceph-mon[40087]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:21:37.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:37 smithi049 ceph-mon[40087]: osdmap e308: 12 total, 11 up, 12 in 2024-04-15T18:21:37.478 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:37.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:37 smithi159 ceph-mon[45336]: osd.2 marked itself down and dead 2024-04-15T18:21:37.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:37 smithi159 ceph-mon[45336]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:21:37.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:37 smithi159 ceph-mon[45336]: osdmap e308: 12 total, 11 up, 12 in 2024-04-15T18:21:37.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:37 smithi082 ceph-mon[44206]: osd.2 marked itself down and dead 2024-04-15T18:21:37.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:37 smithi082 ceph-mon[44206]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:21:37.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:37 smithi082 ceph-mon[44206]: osdmap e308: 12 total, 11 up, 12 in 2024-04-15T18:21:37.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:21:38.228 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:38.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:38.341 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:38.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:21:38.375 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:38.377 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:21:38 smithi049 bash[158115]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2 2024-04-15T18:21:38.401 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:38.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:38.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 58906 58890 64.565 24.625 0.00308305 0.0154784 2024-04-15T18:21:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:38.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:38.532 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:38.553 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:38.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:38.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:38.692 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:38.717 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:38.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:38.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:38.860 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:38.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:39.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:39.108 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:39.187 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:39.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:39.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:39 smithi049 ceph-mon[40087]: pgmap v852: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:21:39.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:39 smithi049 ceph-mon[40087]: osdmap e309: 12 total, 11 up, 12 in 2024-04-15T18:21:39.247 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:21:39.247 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:21:39.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:39.322 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:39.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:39.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 59638 59622 64.2405 45.75 0.00241719 0.0155529 2024-04-15T18:21:39.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:39.484 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:39.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:39.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:39 smithi082 ceph-mon[44206]: pgmap v852: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:21:39.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:39 smithi082 ceph-mon[44206]: osdmap e309: 12 total, 11 up, 12 in 2024-04-15T18:21:39.629 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:39 smithi159 ceph-mon[45336]: pgmap v852: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:21:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:39 smithi159 ceph-mon[45336]: osdmap e309: 12 total, 11 up, 12 in 2024-04-15T18:21:39.662 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:39.691 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:39.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:39.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:39.837 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:39.863 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:39.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:39.873 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:39.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:40.019 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:40.374 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:40.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:40.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 60788 60772 64.3698 71.875 0.0169227 0.0155294 2024-04-15T18:21:40.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:40.508 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:40.531 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:40.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:40.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:40.669 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:40.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:40.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:40.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:21:40.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:40.990 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.016 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:41.122 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:41.154 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.176 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:41.221 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:41 smithi049 ceph-mon[40087]: pgmap v854: 17 pgs: 1 peering, 1 active+clean+scrubbing+deep, 15 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-04-15T18:21:41.282 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:41.316 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.340 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:41.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:41 smithi159 ceph-mon[45336]: pgmap v854: 17 pgs: 1 peering, 1 active+clean+scrubbing+deep, 15 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-04-15T18:21:41.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:41.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:21:41.446845+0000 min lat: 0.00105766 max lat: 15.0119 avg lat: 0.0154975 2024-04-15T18:21:41.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:21:41.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 61946 61930 64.5031 72.375 0.00617029 0.0154975 2024-04-15T18:21:41.479 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:41 smithi082 ceph-mon[44206]: pgmap v854: 17 pgs: 1 peering, 1 active+clean+scrubbing+deep, 15 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-04-15T18:21:41.480 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.504 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:41.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:41.636 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.660 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:41.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:41.790 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.812 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:41.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:41.946 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:41.968 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:41.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:41.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:42.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:42.105 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:42.126 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:42.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:42.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:42.258 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:42.279 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:42.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:42.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:42.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2874000212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:42.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:42 smithi049 ceph-mon[40087]: osdmap e310: 12 total, 11 up, 12 in 2024-04-15T18:21:42.422 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:42.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 62712 62696 64.2305 47.875 0.00324156 0.0155613 2024-04-15T18:21:42.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2874000212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:42.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:42 smithi082 ceph-mon[44206]: osdmap e310: 12 total, 11 up, 12 in 2024-04-15T18:21:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2874000212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:42.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:42 smithi159 ceph-mon[45336]: osdmap e310: 12 total, 11 up, 12 in 2024-04-15T18:21:43.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:43.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:43.092 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:43.093 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:43.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: pgmap v855: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:21:43.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.443 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 63937 63921 64.4293 76.5625 0.00673621 0.0155144 2024-04-15T18:21:43.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: pgmap v855: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:21:43.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: pgmap v855: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:43.762 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:43.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-15T18:21:43.780 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:43.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:43.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:21:43.843 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-15T18:21:43.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd down 2 2024-04-15T18:21:43.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:43.938 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:44.027 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:44.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:44.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:44.170 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:44.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:44 smithi049 ceph-mon[40087]: pgmap v857: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 7861/184998 objects degraded (4.249%) 2024-04-15T18:21:44.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:44 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 7861/184998 objects degraded (4.249%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:44.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:44 smithi049 ceph-mon[40087]: osdmap e311: 12 total, 11 up, 12 in 2024-04-15T18:21:44.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 65204 65188 64.6634 79.1875 0.0106321 0.0154584 2024-04-15T18:21:44.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:44 smithi082 ceph-mon[44206]: pgmap v857: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 7861/184998 objects degraded (4.249%) 2024-04-15T18:21:44.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:44 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 7861/184998 objects degraded (4.249%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:44.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:44 smithi082 ceph-mon[44206]: osdmap e311: 12 total, 11 up, 12 in 2024-04-15T18:21:44.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:44 smithi159 ceph-mon[45336]: pgmap v857: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 7861/184998 objects degraded (4.249%) 2024-04-15T18:21:44.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:44 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 7861/184998 objects degraded (4.249%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:44.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:44 smithi159 ceph-mon[45336]: osdmap e311: 12 total, 11 up, 12 in 2024-04-15T18:21:44.935 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:44.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:45.041 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:45.070 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:45.093 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:45.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:45.197 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:45.233 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:45.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:45.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:45.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:45.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:21:45.395 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:21:45.397 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:45.421 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:45.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:45.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 66306 66290 64.7291 68.875 0.0198314 0.0154438 2024-04-15T18:21:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:45 smithi049 ceph-mon[40087]: 4.5 deep-scrub ok 2024-04-15T18:21:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:45.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:45.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:45 smithi082 ceph-mon[44206]: 4.5 deep-scrub ok 2024-04-15T18:21:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:45.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:45.586 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:45.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:45.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:45.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:45.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:45 smithi159 ceph-mon[45336]: 4.5 deep-scrub ok 2024-04-15T18:21:45.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:45.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:45.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:45.870 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:45.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:45.914 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:45.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:46.017 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:46.180 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:46.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:46.246 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:21:46.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:46.322 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:46.341 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:46.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:46.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 67439 67423 64.8226 70.8125 0.00577916 0.0154217 2024-04-15T18:21:46.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:46.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:46 smithi049 ceph-mon[40087]: pgmap v860: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s; 7861/189198 objects degraded (4.155%) 2024-04-15T18:21:46.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:46 smithi049 ceph-mon[40087]: osdmap e312: 12 total, 11 up, 12 in 2024-04-15T18:21:46.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:46.486 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:46.510 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:46.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:46.514 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:46 smithi159 ceph-mon[45336]: pgmap v860: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s; 7861/189198 objects degraded (4.155%) 2024-04-15T18:21:46.514 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:46 smithi159 ceph-mon[45336]: osdmap e312: 12 total, 11 up, 12 in 2024-04-15T18:21:46.514 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:46.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:46 smithi082 ceph-mon[44206]: pgmap v860: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+undersized+degraded, 14 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 928 op/s; 7861/189198 objects degraded (4.155%) 2024-04-15T18:21:46.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:46 smithi082 ceph-mon[44206]: osdmap e312: 12 total, 11 up, 12 in 2024-04-15T18:21:46.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:21:46.614 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:46.647 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:46.675 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:46.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:46.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:21:46.819 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:46.921 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:21:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:46.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:21:47.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:47.063 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:47.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:21:47.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 68754 68738 65.0856 82.1875 0.0191811 0.0153588 2024-04-15T18:21:47.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:47 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:47.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:47.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:47 smithi082 ceph-mon[44206]: osdmap e313: 12 total, 11 up, 12 in 2024-04-15T18:21:47.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:47 smithi082 ceph-mon[44206]: osdmap e314: 12 total, 11 up, 12 in 2024-04-15T18:21:47.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:47 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:47.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:47.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:47 smithi049 ceph-mon[40087]: osdmap e313: 12 total, 11 up, 12 in 2024-04-15T18:21:47.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:47 smithi049 ceph-mon[40087]: osdmap e314: 12 total, 11 up, 12 in 2024-04-15T18:21:47.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:47 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:21:47.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2970860071' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:21:47.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:47 smithi159 ceph-mon[45336]: osdmap e313: 12 total, 11 up, 12 in 2024-04-15T18:21:47.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:47 smithi159 ceph-mon[45336]: osdmap e314: 12 total, 11 up, 12 in 2024-04-15T18:21:48.009 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 247/235 [PushOp(3:4785a230:::benchmark_data_smithi159_91621_object31891:head, version: 206'2012, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4785a230:::benchmark_data_smithi159_91621_object31891:head@206'2012, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 216.12365670899999, 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.202080965, 2024-04-15T18:21:48.010 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885594+0000", 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885595+0000", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.116e-06 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:11.885822+0000", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022720600000000001 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085100+0000", 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19927798299999999 2024-04-15T18:21:48.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085112+0000", 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2310000000000001e-05 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085208+0000", 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5311999999999998e-05 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.085222+0000", 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4099e-05 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:12.087675+0000", 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024529389999999999 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 253/235 [PushOp(3:43924633:::benchmark_data_smithi159_91621_object45721:head, version: 211'2883, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:43924633:::benchmark_data_smithi159_91621_object45721:head@211'2883, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:18:35.913334+0000", 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 192.09591594599999, 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17515568000000001, 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:35.913334+0000", 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:35.913334+0000", 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:35.913336+0000", 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2720000000000001e-06 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:35.918517+0000", 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051813529999999997 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:36.084815+0000", 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16629787300000001 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:36.084835+0000", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0438999999999999e-05 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:36.084930+0000", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4975000000000005e-05 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:36.084945+0000", 2024-04-15T18:21:48.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4871999999999999e-05 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:18:36.088490+0000", 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035448960000000001 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(3.2 258/235 [PushOp(3:4972e341:::benchmark_data_smithi159_91621_object61667:head, version: 216'3902, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4972e341:::benchmark_data_smithi159_91621_object61667:head@216'3902, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:19:02.367256+0000", 2024-04-15T18:21:48.019 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 165.64199455100001, 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18471985699999999, 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.367256+0000", 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.367256+0000", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.367257+0000", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0999999999999997e-07 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.367327+0000", 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9887000000000004e-05 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.547496+0000", 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.180169616 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.547516+0000", 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9538000000000002e-05 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.548126+0000", 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00060988399999999999 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.548141+0000", 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5308e-05 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:02.551976+0000", 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038347139999999999 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.24793.0:185715 3.2 e267/235 3:442b176b:::benchmark_data_smithi159_91621_object79239:head v 267'11716, mlcod=222'5164)", 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:19:33.088272+0000", 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 134.92097870399999, 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.158021414, 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.088272+0000", 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.088272+0000", 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.088273+0000", 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.78e-07 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.088280+0000", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.61e-06 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.240655+0000", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152374603 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.240663+0000", 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0029999999999994e-06 2024-04-15T18:21:48.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.240723+0000", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0365999999999998e-05 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.240755+0000", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2118e-05 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.246244+0000", 2024-04-15T18:21:48.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00548899 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:19:33.246293+0000", 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8846e-05 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.2 e275,235 3:4c80762f:::benchmark_data_smithi159_91621_object20580:head,242'8026)", 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:21:48.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 92.997779988999994, 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37528735600000002, 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011471+0000", 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011475+0000", 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6940000000000001e-06 2024-04-15T18:21:48.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386618+0000", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37514348800000002 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386630+0000", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1559999999999999e-05 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386699+0000", 2024-04-15T18:21:48.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9227000000000007e-05 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386705+0000", 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4790000000000001e-06 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386758+0000", 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2429000000000002e-05 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.2 e275,235 3:4aa76ad9:::benchmark_data_smithi159_91621_object21018:head,242'8068)", 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 92.539876609999993, 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59152146100000003, 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:21:48.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469375+0000", 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8899999999999999e-07 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469380+0000", 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9849999999999997e-06 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060797+0000", 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59141723800000001 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060803+0000", 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5479999999999998e-06 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060850+0000", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6220999999999997e-05 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060855+0000", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8109999999999999e-06 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060895+0000", 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9969000000000003e-05 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.036 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e275,235 3:af3b238d:::benchmark_data_smithi159_91621_object23467:head,243'8240)", 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 92.223784972000004, 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29041672899999998, 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785466+0000", 2024-04-15T18:21:48.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2799999999999996e-07 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785470+0000", 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2039999999999999e-06 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075781+0000", 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29031098599999999 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075787+0000", 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3229999999999997e-06 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075835+0000", 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8937999999999998e-05 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075842+0000", 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9310000000000002e-06 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075882+0000", 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9818999999999999e-05 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e276,235 3:a6b57d98:::benchmark_data_smithi159_91621_object30030:head,247'8679)", 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 91.222432128999998, 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33609018000000002, 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786819+0000", 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3599999999999999e-07 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786822+0000", 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3239999999999999e-06 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122780+0000", 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.335958219 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122786+0000", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4299999999999997e-06 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122844+0000", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7929000000000001e-05 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122853+0000", 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3780000000000007e-06 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122908+0000", 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5464000000000003e-05 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e277,235 3:ad41a5e8:::benchmark_data_smithi159_91621_object45130:head,253'9622)", 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 90.111582517000002, 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24596616600000001, 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.045 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2800000000000002e-07 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897672+0000", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3890000000000001e-06 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143032+0000", 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24536002900000001 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143038+0000", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0299999999999999e-06 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143562+0000", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052384700000000003 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143572+0000", 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0637e-05 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143634+0000", 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1805999999999997e-05 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e278,235 3:a6cd20bc:::benchmark_data_smithi159_91621_object76133:head,266'11576)", 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 89.192494142000001, 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38370323899999997, 2024-04-15T18:21:48.049 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816757+0000", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4000000000000002e-07 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816760+0000", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3629999999999998e-06 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200349+0000", 2024-04-15T18:21:48.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38358931899999998 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200355+0000", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5310000000000001e-06 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200415+0000", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9953e-05 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200422+0000", 2024-04-15T18:21:48.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0849999999999999e-06 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200460+0000", 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7548e-05 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e279,235 3:ac24a022:::benchmark_data_smithi159_91621_object76992:head,266'11636)", 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.764579603000001, 2024-04-15T18:21:48.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227295887, 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244672+0000", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244681+0000", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4129999999999995e-06 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471801+0000", 2024-04-15T18:21:48.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227119823 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471816+0000", 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4686e-05 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471897+0000", 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1538000000000004e-05 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471905+0000", 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8220000000000007e-06 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471967+0000", 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1418e-05 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:22145 4.f e293/284 4:fa800ec6:::benchmark_data_smithi159_102124_object22144:head v 293'1361, mlcod=293'1357)", 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:54.756874+0000", 2024-04-15T18:21:48.057 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 53.252376445000003, 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14126976899999999, 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.756874+0000", 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.756874+0000", 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.756875+0000", 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9400000000000004e-07 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.756902+0000", 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7654999999999999e-05 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.897322+0000", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14041941399999999 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.897329+0000", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.233e-06 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.897382+0000", 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2877000000000002e-05 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.897396+0000", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4486e-05 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.898110+0000", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00071310500000000001 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.898144+0000", 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4205000000000002e-05 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.061 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:22146 4.0 4:0e9ff0c6:::benchmark_data_smithi159_102124_object22145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:54.759265+0000", 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 53.249985268000003, 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.150457852, 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:21:48.062 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 22146 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759265+0000", 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759265+0000", 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759266+0000", 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.065e-06 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759321+0000", 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4438999999999998e-05 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759331+0000", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0627e-05 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759337+0000", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6810000000000003e-06 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759387+0000", 2024-04-15T18:21:48.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0157000000000001e-05 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759477+0000", 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9436000000000002e-05 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.759575+0000", 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8796999999999996e-05 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:21:48.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.760372+0000", 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079623200000000002 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.760460+0000", 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8609000000000005e-05 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.909626+0000", 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149166253 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.909686+0000", 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9505e-05 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:54.909723+0000", 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7051000000000002e-05 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:39801 4.c e301/284 4:3ea5c1e4:::benchmark_data_smithi159_102124_object39800:head v 301'2488, mlcod=301'2484)", 2024-04-15T18:21:48.068 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 25.912324737999999, 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21772696599999999, 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8100000000000003e-07 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096981+0000", 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4803999999999999e-05 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313237+0000", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21625582600000001 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313245+0000", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1610000000000003e-06 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313366+0000", 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120587 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313392+0000", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6222000000000001e-05 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.314621+0000", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001229401 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.314653+0000", 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1483999999999999e-05 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.072 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:39884 4.c e302/284 4:3dc4d635:::benchmark_data_smithi159_102124_object39883:head v 302'2493, mlcod=302'2489)", 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:22.371343+0000", 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 25.637907108, 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19814072999999999, 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.371343+0000", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.371343+0000", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.371344+0000", 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8400000000000002e-07 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.371399+0000", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4945999999999999e-05 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.567584+0000", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.196185005 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.567591+0000", 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9179999999999997e-06 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.567671+0000", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9939999999999997e-05 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.567685+0000", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4116e-05 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.569430+0000", 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001745173 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.569484+0000", 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3647999999999999e-05 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:49931 4.9 e305/284 4:9fdb0bab:::benchmark_data_smithi159_102124_object49930:head v 305'3050, mlcod=305'3047)", 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:29.719201+0000", 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 18.290049143000001, 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.145517128, 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.077 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.719201+0000", 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.719201+0000", 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.719202+0000", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3099999999999997e-07 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.719261+0000", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9063e-05 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.863304+0000", 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14404307399999999 2024-04-15T18:21:48.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.863311+0000", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7009999999999999e-06 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.863372+0000", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1119999999999998e-05 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.863395+0000", 2024-04-15T18:21:48.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2969e-05 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.864667+0000", 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001272511 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:29.864718+0000", 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0958999999999998e-05 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:66501 4.0 4:0505fca5:::benchmark_data_smithi159_102124_object66500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e312)", 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:45.585252+0000", 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.4239983199999999, 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14672617499999999, 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:21:48.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 66501 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585252+0000", 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585252+0000", 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585253+0000", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.345e-06 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585293+0000", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0022999999999999e-05 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585308+0000", 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4245e-05 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585317+0000", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9209999999999998e-06 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585368+0000", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1703000000000003e-05 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585471+0000", 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010312800000000001 2024-04-15T18:21:48.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.585564+0000", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2521000000000004e-05 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.587860+0000", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022960490000000001 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.588708+0000", 2024-04-15T18:21:48.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00084797099999999999 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731890+0000", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.143182055 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731940+0000", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0034000000000002e-05 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731978+0000", 2024-04-15T18:21:48.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8179999999999997e-05 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25126.0:66501 4.0 e312/284 ondisk, result = 0)", 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:45.589158+0000", 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.4200926150000002, 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14282876799999999, 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.589158+0000", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.589158+0000", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.589159+0000", 2024-04-15T18:21:48.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.589164+0000", 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1610000000000002e-06 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731798+0000", 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14263457700000001 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731812+0000", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3584000000000001e-05 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731876+0000", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4338999999999995e-05 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731886+0000", 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2909999999999996e-06 2024-04-15T18:21:48.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.731987+0000", 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101075 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:66564 4.c e312/284 4:347b82c9:::benchmark_data_smithi159_102124_object66563:head v 312'4206, mlcod=312'4203)", 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:45.662311+0000", 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.3469399019999999, 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155078828, 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.092 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.662311+0000", 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.662311+0000", 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.662312+0000", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.903e-06 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.662364+0000", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1452000000000001e-05 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.812820+0000", 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15045652000000001 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.812827+0000", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6800000000000004e-06 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.812875+0000", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8279000000000003e-05 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.812903+0000", 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7591000000000001e-05 2024-04-15T18:21:48.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.817341+0000", 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044376399999999996 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.817389+0000", 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8763000000000001e-05 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.096 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:66567 4.9 e312/284 4:9daffb57:::benchmark_data_smithi159_102124_object66566:head v 312'4070, mlcod=312'4065)", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:45.667747+0000", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.3415031829999999, 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15215541899999999, 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.667747+0000", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:21:48.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.667747+0000", 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.667749+0000", 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.53e-06 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.667814+0000", 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4899000000000003e-05 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.818115+0000", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.150301146 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.818122+0000", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.922e-06 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.818169+0000", 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931000000000002e-05 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.818184+0000", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4987000000000001e-05 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.819865+0000", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001681592 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:45.819903+0000", 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7412e-05 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:21:48.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:21:48.101 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:21:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.102 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:48.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:48.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:48.236 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:48.264 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:48.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:48.402 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:48.428 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:48.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 69963 69947 65.2419 75.5625 0.00343961 0.0153209 2024-04-15T18:21:48.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:48.566 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:48.589 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:48.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2 is already down. 2024-04-15T18:21:48.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:48.739 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:48.770 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:48.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:48.913 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:48.969 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:48.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:48 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:21:48.665+0000 7f8015e7e700 -1 mon.a@0(leader).osd e314 definitely_dead 0 2024-04-15T18:21:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:48.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:49.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:49.104 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.125 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:49.229 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:49.258 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.279 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:49.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:49.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:49 smithi049 ceph-mon[40087]: pgmap v863: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 902 op/s; 8232/195831 objects degraded (4.204%) 2024-04-15T18:21:49.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3173884494' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.394 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.409 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:49 smithi082 ceph-mon[44206]: pgmap v863: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 902 op/s; 8232/195831 objects degraded (4.204%) 2024-04-15T18:21:49.409 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3173884494' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.409 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.411 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.431 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-04-15T18:21:49.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd out 2 2024-04-15T18:21:49.432 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:49.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 70992 70976 65.2282 64.3125 0.00619512 0.0153245 2024-04-15T18:21:49.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.590 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:49 smithi159 ceph-mon[45336]: pgmap v863: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 902 op/s; 8232/195831 objects degraded (4.204%) 2024-04-15T18:21:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3173884494' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["2"]}]: dispatch 2024-04-15T18:21:49.694 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:49.727 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.751 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:49.857 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:49.883 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:49.902 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:49.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:50.005 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.050 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:50.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:21:50.153 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:50.162 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:50.180 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.198 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:50.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:50.334 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.368 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:50.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:50.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:50.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:50 smithi049 ceph-mon[40087]: osdmap e315: 12 total, 11 up, 12 in 2024-04-15T18:21:50.443 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:50.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 71822 71806 65.0346 51.875 0.0151577 0.0153714 2024-04-15T18:21:50.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:50.501 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:50.501 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:50.501 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:50 smithi082 ceph-mon[44206]: osdmap e315: 12 total, 11 up, 12 in 2024-04-15T18:21:50.502 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.528 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:50.633 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:50.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:50.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:50.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:50 smithi159 ceph-mon[45336]: osdmap e315: 12 total, 11 up, 12 in 2024-04-15T18:21:50.667 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:50.798 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:50.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:21:50.827 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.849 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:50.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:50.953 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:50.981 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:50.999 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:51.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:51.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:51.133 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:51.231 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:51.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:51.337 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:51.364 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:51.386 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:51.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:51.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 72679 72663 64.8707 53.5625 0.0485125 0.0154088 2024-04-15T18:21:51.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: pgmap v865: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 8584/202524 objects degraded (4.239%) 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8584/202524 objects degraded (4.239%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:51.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:51 smithi049 ceph-mon[40087]: osdmap e316: 12 total, 11 up, 12 in 2024-04-15T18:21:51.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: pgmap v865: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 8584/202524 objects degraded (4.239%) 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8584/202524 objects degraded (4.239%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:51.517 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:51 smithi159 ceph-mon[45336]: osdmap e316: 12 total, 11 up, 12 in 2024-04-15T18:21:51.520 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: pgmap v865: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 8584/202524 objects degraded (4.239%) 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8584/202524 objects degraded (4.239%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:51.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:51 smithi082 ceph-mon[44206]: osdmap e316: 12 total, 11 up, 12 in 2024-04-15T18:21:51.544 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:51.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:51.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:51.689 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:51.896 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:51.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:52.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:52.037 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:52.424 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:52.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:52.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 73966 73950 65.0898 80.4375 0.0246597 0.0153581 2024-04-15T18:21:52.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:52.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:52.589 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:52.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:52.695 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:52.727 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:52.754 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:52.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:52.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:52.892 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:52.918 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:52.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:53.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:53.072 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:53.138 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:53.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:53.150 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:53.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:53.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:53.284 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:53.307 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:53.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:53.412 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:53.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 75130 75114 65.1961 72.75 0.0123831 0.0153337 2024-04-15T18:21:53.451 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:53 smithi159 ceph-mon[45336]: pgmap v867: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 8584/208953 objects degraded (4.108%) 2024-04-15T18:21:53.451 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3525281729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:53.453 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:53.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:53 smithi049 ceph-mon[40087]: pgmap v867: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 8584/208953 objects degraded (4.108%) 2024-04-15T18:21:53.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3525281729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:53.477 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:53.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:53.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:53 smithi082 ceph-mon[44206]: pgmap v867: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 8584/208953 objects degraded (4.108%) 2024-04-15T18:21:53.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3525281729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:21:53.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:21:53.631 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:53.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:21:53.770 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:21:53.809 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:21:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:21:53.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:21:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:53.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:21:53.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:21:53.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:21:53.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:53.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:21:53.981 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:21:54.002 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:54.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:21:54.071 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:21:54.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:21:54.103 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:21:54.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:21:54.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:54.136 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.158 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:54.264 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:54.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:54 smithi082 ceph-mon[44206]: pgmap v868: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 9215/218691 objects degraded (4.214%) 2024-04-15T18:21:54.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:54.299 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.322 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:54.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:54.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 76367 76351 65.362 77.3125 0.0118432 0.0152938 2024-04-15T18:21:54.460 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:54 smithi049 ceph-mon[40087]: pgmap v868: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 9215/218691 objects degraded (4.214%) 2024-04-15T18:21:54.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:54.482 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:54.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:54.612 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:54 smithi159 ceph-mon[45336]: pgmap v868: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 9215/218691 objects degraded (4.214%) 2024-04-15T18:21:54.613 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:54.615 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.637 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:54.741 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:54.779 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.808 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:54.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:54.947 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:54.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:54 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:21:54.666+0000 7f8015e7e700 -1 mon.a@0(leader).osd e316 definitely_dead 0 2024-04-15T18:21:54.975 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:54.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:55.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:55.116 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.145 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3303189534' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.205 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.206 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: Client client.admin marked osd.2 out, after it was down for 17 seconds 2024-04-15T18:21:55.206 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:55 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:55.220 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.2. 2024-04-15T18:21:55.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:55.275 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.294 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:55.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:55.424 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.443 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:55.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 77296 77280 65.2633 58.0625 0.192842 0.0152811 2024-04-15T18:21:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3303189534' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: Client client.admin marked osd.2 out, after it was down for 17 seconds 2024-04-15T18:21:55.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:55 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3303189534' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: Client client.admin marked osd.2 out, after it was down for 17 seconds 2024-04-15T18:21:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:55 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:55.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:21:55.578 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.597 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:21:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:21:55.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:21:55.727 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:21:55.743 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:21:55.746 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:21:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:21:55.850 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:21:55.876 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:55.894 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:21:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:55.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:21:55.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:56.024 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:56.042 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:56.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:56.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:56.172 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:56.191 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:56.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:56.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:56.324 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:56.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:21:56.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-15T18:21:56.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: osdmap e317: 12 total, 11 up, 11 in 2024-04-15T18:21:56.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: pgmap v870: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 9215/222654 objects degraded (4.139%) 2024-04-15T18:21:56.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:21:56.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:21:56.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 77350 77334 64.4381 3.375 0.00252486 0.0153104 2024-04-15T18:21:56.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:21:56.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-15T18:21:56.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: osdmap e317: 12 total, 11 up, 11 in 2024-04-15T18:21:56.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: pgmap v870: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 9215/222654 objects degraded (4.139%) 2024-04-15T18:21:56.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:21:56.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: osdmap e317: 12 total, 11 up, 11 in 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: pgmap v870: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 9215/222654 objects degraded (4.139%) 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:21:56.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:21:56.639 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-15T18:21:56.639 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi049.front.sepia.ceph.com 2024-04-15T18:21:56.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.2 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-15T18:21:56.672 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:56.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:56.776 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:56.805 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:56.824 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:56.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:56.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:56.960 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:56.981 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:56.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:57.085 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:57.115 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.116 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.2/config 2024-04-15T18:21:57.135 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:57.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:57.264 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 9215/222654 objects degraded (4.139%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: osdmap e318: 12 total, 11 up, 11 in 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:21:57.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:57.266 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.286 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:57.390 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:21:57.416 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.434 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:21:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:21:57.444 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 77354 77338 63.5935 0.25 0.00477911 0.0153188 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 9215/222654 objects degraded (4.139%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: osdmap e318: 12 total, 11 up, 11 in 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:21:57.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:57.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:57.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 9215/222654 objects degraded (4.139%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: osdmap e318: 12 total, 11 up, 11 in 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:21:57.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:57.566 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.585 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:57.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:57.725 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:21:57.789 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:57.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:57.924 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:57.950 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:57.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:58.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:58.085 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:58.109 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:58.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:58.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:58.271 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:58.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 78036 78020 63.3211 42.625 0.0664257 0.0157854 2024-04-15T18:21:58.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: osdmap e319: 12 total, 11 up, 11 in 2024-04-15T18:21:58.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: pgmap v873: 17 pgs: 1 remapped+peering, 1 active+undersized+degraded, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s; 4564/227790 objects degraded (2.004%) 2024-04-15T18:21:58.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:58.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:58.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:58.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: osdmap e319: 12 total, 11 up, 11 in 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: pgmap v873: 17 pgs: 1 remapped+peering, 1 active+undersized+degraded, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s; 4564/227790 objects degraded (2.004%) 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:58.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: osdmap e319: 12 total, 11 up, 11 in 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: pgmap v873: 17 pgs: 1 remapped+peering, 1 active+undersized+degraded, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s; 4564/227790 objects degraded (2.004%) 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:21:58.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:21:58.729 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:58.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:58.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:21:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:58.972 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:21:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:58.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:21:59.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:21:59.130 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.160 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:21:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:59.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:21:59.230 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:21:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:59.256 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:21:59.265 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:21:59.302 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.330 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:21:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:59.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:21:59.404 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:21:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:59.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:59.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 78972 78956 63.2593 58.5 0.0197267 0.0158023 2024-04-15T18:21:59.470 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.494 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:59.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:21:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:21:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:59.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:59.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:59.601 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:21:59.632 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.663 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:21:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:59.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:21:59.767 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:21:59.777 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:21:59.802 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.829 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:21:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:21:59.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:21:59.938 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:21:59.974 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:21:59.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:00.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:00.146 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:00.171 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:00.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:00.333 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:00.382 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:00.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 79813 79797 63.1238 52.5625 0.0187633 0.0158366 2024-04-15T18:22:00.466 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:22:00.466 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:22:00.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:00 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:00.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:00 smithi049 ceph-mon[40087]: pgmap v874: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 4788/231591 objects degraded (2.067%) 2024-04-15T18:22:00.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:00.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:00 smithi049 ceph-mon[40087]: osdmap e320: 12 total, 11 up, 11 in 2024-04-15T18:22:00.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:00 smithi049 ceph-mon[40087]: 4.c scrub starts 2024-04-15T18:22:00.489 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:00.518 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:00 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:00.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:00 smithi082 ceph-mon[44206]: pgmap v874: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 4788/231591 objects degraded (2.067%) 2024-04-15T18:22:00.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:00.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:00 smithi082 ceph-mon[44206]: osdmap e320: 12 total, 11 up, 11 in 2024-04-15T18:22:00.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:00 smithi082 ceph-mon[44206]: 4.c scrub starts 2024-04-15T18:22:00.522 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:00.556 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:00.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:00 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:00.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:00 smithi159 ceph-mon[45336]: pgmap v874: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 4788/231591 objects degraded (2.067%) 2024-04-15T18:22:00.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2582705505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:00.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:00 smithi159 ceph-mon[45336]: osdmap e320: 12 total, 11 up, 11 in 2024-04-15T18:22:00.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:00 smithi159 ceph-mon[45336]: 4.c scrub starts 2024-04-15T18:22:00.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:00.689 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:00.711 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:00.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:00.843 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:00.872 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:00.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:00.977 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:01.006 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:01.027 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:01.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:01.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:01.173 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:01.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:01 smithi082 ceph-mon[44206]: 4.c scrub ok 2024-04-15T18:22:01.301 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:01.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:01 smithi049 ceph-mon[40087]: 4.c scrub ok 2024-04-15T18:22:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:01.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:01.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:01.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:22:01.448535+0000 min lat: 0.00105766 max lat: 15.0119 avg lat: 0.0158526 2024-04-15T18:22:01.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:22:01.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 80739 80723 63.0582 57.875 0.00980824 0.0158526 2024-04-15T18:22:01.451 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:01.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:01 smithi159 ceph-mon[45336]: 4.c scrub ok 2024-04-15T18:22:01.669 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:01.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:01.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:01.806 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:01.829 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:01.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:01.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:02.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:22:02.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 81579 81563 62.9278 52.5 0.0658967 0.0158851 2024-04-15T18:22:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:02 smithi082 ceph-mon[44206]: pgmap v876: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 647 op/s; 4788/234312 objects degraded (2.043%) 2024-04-15T18:22:02.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 4788/234312 objects degraded (2.043%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:22:02.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:02 smithi159 ceph-mon[45336]: pgmap v876: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 647 op/s; 4788/234312 objects degraded (2.043%) 2024-04-15T18:22:02.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 4788/234312 objects degraded (2.043%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:22:02.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:02 smithi049 ceph-mon[40087]: pgmap v876: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 647 op/s; 4788/234312 objects degraded (2.043%) 2024-04-15T18:22:02.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 4788/234312 objects degraded (2.043%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:22:03.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 82478 82462 62.8455 56.1875 0.00406905 0.0159063 2024-04-15T18:22:03.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:03.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:03.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:03 smithi082 ceph-mon[44206]: osdmap e321: 12 total, 11 up, 11 in 2024-04-15T18:22:03.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:03 smithi082 ceph-mon[44206]: 4.3 scrub starts 2024-04-15T18:22:03.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:03 smithi082 ceph-mon[44206]: 4.e scrub starts 2024-04-15T18:22:03.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:03.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:03.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:03 smithi159 ceph-mon[45336]: osdmap e321: 12 total, 11 up, 11 in 2024-04-15T18:22:03.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:03 smithi159 ceph-mon[45336]: 4.3 scrub starts 2024-04-15T18:22:03.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:03 smithi159 ceph-mon[45336]: 4.e scrub starts 2024-04-15T18:22:03.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:03.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:03.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:03 smithi049 ceph-mon[40087]: osdmap e321: 12 total, 11 up, 11 in 2024-04-15T18:22:03.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:03 smithi049 ceph-mon[40087]: 4.3 scrub starts 2024-04-15T18:22:03.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:03 smithi049 ceph-mon[40087]: 4.e scrub starts 2024-04-15T18:22:04.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 83298 83282 62.7057 51.25 0.0200606 0.0159421 2024-04-15T18:22:04.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:04 smithi159 ceph-mon[45336]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 9145/242058 objects degraded (3.778%); 5.4 MiB/s, 86 objects/s recovering 2024-04-15T18:22:04.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:04 smithi159 ceph-mon[45336]: 4.3 scrub ok 2024-04-15T18:22:04.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:04 smithi049 ceph-mon[40087]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 9145/242058 objects degraded (3.778%); 5.4 MiB/s, 86 objects/s recovering 2024-04-15T18:22:04.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:04 smithi049 ceph-mon[40087]: 4.3 scrub ok 2024-04-15T18:22:04.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:04 smithi082 ceph-mon[44206]: pgmap v878: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 9145/242058 objects degraded (3.778%); 5.4 MiB/s, 86 objects/s recovering 2024-04-15T18:22:04.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:04 smithi082 ceph-mon[44206]: 4.3 scrub ok 2024-04-15T18:22:05.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 84020 84004 62.4964 45.125 0.0108658 0.0159949 2024-04-15T18:22:05.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:05 smithi159 ceph-mon[45336]: 4.e scrub ok 2024-04-15T18:22:05.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:05 smithi159 ceph-mon[45336]: 4.b scrub starts 2024-04-15T18:22:05.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:05 smithi159 ceph-mon[45336]: 4.a scrub starts 2024-04-15T18:22:05.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:05 smithi049 ceph-mon[40087]: 4.e scrub ok 2024-04-15T18:22:05.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:05 smithi049 ceph-mon[40087]: 4.b scrub starts 2024-04-15T18:22:05.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:05 smithi049 ceph-mon[40087]: 4.a scrub starts 2024-04-15T18:22:05.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:05 smithi082 ceph-mon[44206]: 4.e scrub ok 2024-04-15T18:22:05.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:05 smithi082 ceph-mon[44206]: 4.b scrub starts 2024-04-15T18:22:05.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:05 smithi082 ceph-mon[44206]: 4.a scrub starts 2024-04-15T18:22:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:06.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:06.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:06.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:06 smithi049 ceph-mon[40087]: pgmap v879: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 733 op/s; 9145/245388 objects degraded (3.727%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:22:06.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 85016 85000 62.4935 62.25 0.0266695 0.015997 2024-04-15T18:22:06.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:06 smithi159 ceph-mon[45336]: pgmap v879: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 733 op/s; 9145/245388 objects degraded (3.727%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:22:06.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:06 smithi159 ceph-mon[45336]: 4.a scrub ok 2024-04-15T18:22:06.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:06 smithi049 ceph-mon[40087]: 4.a scrub ok 2024-04-15T18:22:06.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:06 smithi082 ceph-mon[44206]: pgmap v879: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 733 op/s; 9145/245388 objects degraded (3.727%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:22:06.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:06 smithi082 ceph-mon[44206]: 4.a scrub ok 2024-04-15T18:22:07.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 85938 85922 62.4368 57.625 0.00871165 0.0160095 2024-04-15T18:22:07.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:07 smithi049 ceph-mon[40087]: 4.b scrub ok 2024-04-15T18:22:07.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 9145/245388 objects degraded (3.727%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:07.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:07 smithi159 ceph-mon[45336]: 4.b scrub ok 2024-04-15T18:22:07.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 9145/245388 objects degraded (3.727%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:07.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:07 smithi082 ceph-mon[44206]: 4.b scrub ok 2024-04-15T18:22:07.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 9145/245388 objects degraded (3.727%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:08.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:22:08.365 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:08.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:08.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:08.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 86815 86799 62.3491 54.8125 0.0770287 0.0160309 2024-04-15T18:22:08.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:08.519 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:08.805 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:08.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:08.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:08.948 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:09.010 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:09.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:09.049 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:09.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:09.153 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:09.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:09 smithi049 ceph-mon[40087]: pgmap v880: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 770 op/s; 8893/250083 objects degraded (3.556%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:22:09.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3507259710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:09.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:09.373 INFO:teuthology.orchestra.run.smithi049.stdout:4.4 2024-04-15T18:22:09.373 INFO:teuthology.orchestra.run.smithi049.stdout:4.6 2024-04-15T18:22:09.375 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:09.406 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:09.431 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:09.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:09.435 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:09 smithi159 ceph-mon[45336]: pgmap v880: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 770 op/s; 8893/250083 objects degraded (3.556%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:22:09.435 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3507259710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:09.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 87472 87456 62.1072 41.0625 0.00372822 0.0160951 2024-04-15T18:22:09.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:09 smithi082 ceph-mon[44206]: pgmap v880: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 770 op/s; 8893/250083 objects degraded (3.556%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:22:09.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3507259710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:09.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:09.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:09.588 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:09.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:09.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:09.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:22:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:09.794 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:22:09.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:22:09.882 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:22:09.915 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:22:09.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:22:09.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:22:10.017 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:22:10.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:22:10.088 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:22:10.121 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:22:10.122 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:22:10.122 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:22:10.122 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:22:10.122 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:22:10.122 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:22:10.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 88448 88432 62.0947 61 0.0165287 0.0160989 2024-04-15T18:22:10.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:10.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:10.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:10 smithi049 ceph-mon[40087]: osdmap e322: 12 total, 11 up, 11 in 2024-04-15T18:22:10.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:10.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:10.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:10.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:10 smithi082 ceph-mon[44206]: osdmap e322: 12 total, 11 up, 11 in 2024-04-15T18:22:10.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:10.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:10.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:10.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:10 smithi159 ceph-mon[45336]: osdmap e322: 12 total, 11 up, 11 in 2024-04-15T18:22:10.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:10.742 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:10.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:10.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:10.891 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:10.924 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:10.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:11.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:11.066 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.094 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:11.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: pgmap v882: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 53 MiB/s wr, 855 op/s; 8626/254853 objects degraded (3.385%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:22:11.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: 4.8 scrub starts 2024-04-15T18:22:11.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:11 smithi049 ceph-mon[40087]: 4.d scrub starts 2024-04-15T18:22:11.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:11.234 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.260 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:11.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:11.409 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.438 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:11.441 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: pgmap v882: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 53 MiB/s wr, 855 op/s; 8626/254853 objects degraded (3.385%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:22:11.441 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: 4.8 scrub starts 2024-04-15T18:22:11.442 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:11 smithi159 ceph-mon[45336]: 4.d scrub starts 2024-04-15T18:22:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:11.445 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 8 89351 89343 62.0373 56.9375 0.0311817 0.0161124 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: pgmap v882: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 53 MiB/s wr, 855 op/s; 8626/254853 objects degraded (3.385%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: 4.8 scrub starts 2024-04-15T18:22:11.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:11 smithi082 ceph-mon[44206]: 4.d scrub starts 2024-04-15T18:22:11.544 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:11.572 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.590 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:11.694 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:11.722 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.743 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:11.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:11.873 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:11.892 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:11.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:11.996 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:12.024 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:12.044 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:12.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:12.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:12.175 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:12.196 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:12.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:12.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:12.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:12 smithi049 ceph-mon[40087]: 4.8 scrub ok 2024-04-15T18:22:12.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8626/254853 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:12 smithi159 ceph-mon[45336]: 4.8 scrub ok 2024-04-15T18:22:12.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8626/254853 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:12.328 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:12.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.2 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.4 --file /var/log/ceph/exp.4.4.2 2024-04-15T18:22:12.348 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:12.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.7016 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 89351 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 61.5693 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 35.9592 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 148.625 2024-04-15T18:22:12.446 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 985 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 575.355 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 2378 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 0 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0161464 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.198856 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 15.0119 2024-04-15T18:22:12.447 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00105766 2024-04-15T18:22:12.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:12.481 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:12.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:12 smithi082 ceph-mon[44206]: 4.8 scrub ok 2024-04-15T18:22:12.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8626/254853 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:12.789 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.2/config 2024-04-15T18:22:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:22:12.811 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:22:13.299 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:22:13.437 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:13 smithi049 ceph-mon[40087]: pgmap v883: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 8626/259710 objects degraded (3.321%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:22:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:13 smithi049 ceph-mon[40087]: 4.d scrub ok 2024-04-15T18:22:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:13.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:13.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:13 smithi082 ceph-mon[44206]: pgmap v883: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 8626/259710 objects degraded (3.321%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:22:13.537 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:13 smithi082 ceph-mon[44206]: 4.d scrub ok 2024-04-15T18:22:13.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:13.586 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:13 smithi159 ceph-mon[45336]: pgmap v883: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 8626/259710 objects degraded (3.321%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:22:13.586 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:13 smithi159 ceph-mon[45336]: 4.d scrub ok 2024-04-15T18:22:13.586 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:13.623 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:13.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:13.729 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:13.763 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:13.794 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:13.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:13.882 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:22:13.882 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:22:13.882 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:22:13.882 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:22:13.882 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:22:13.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:13.944 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:13.976 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:13.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:14.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:14.112 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.133 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:14.193 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:14 smithi159 ceph-mon[45336]: pgmap v884: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 810 op/s; 8026/266382 objects degraded (3.013%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:14.193 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:14.193 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:14.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:14.269 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.293 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:14.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:14.433 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.456 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:14.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:14 smithi049 ceph-mon[40087]: pgmap v884: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 810 op/s; 8026/266382 objects degraded (3.013%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:14.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:14.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:14.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:14 smithi082 ceph-mon[44206]: pgmap v884: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 51 MiB/s wr, 810 op/s; 8026/266382 objects degraded (3.013%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:14.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:14.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:14.561 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:14.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.617 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:14.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:14.755 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.780 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:14.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:22:14.885 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:14.914 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:14.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:22:14.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:14.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:15.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:15.086 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.112 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:15.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:15 smithi049 ceph-mon[40087]: osdmap e323: 12 total, 11 up, 11 in 2024-04-15T18:22:15.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:15.193 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:15 smithi159 ceph-mon[45336]: osdmap e323: 12 total, 11 up, 11 in 2024-04-15T18:22:15.193 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:15.208 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:22:15.217 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:15.247 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.274 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:15.379 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:15.412 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.433 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:15.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:15 smithi082 ceph-mon[44206]: osdmap e323: 12 total, 11 up, 11 in 2024-04-15T18:22:15.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:15.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:15.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.599 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:22:15.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:15.734 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.756 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:15.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:15.900 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:15.930 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:15.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:16.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:16.068 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:16.087 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:16.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:16.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:16.287 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:16.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:16.295 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:16 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:16.295 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:16.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:16 smithi049 ceph-mon[40087]: osdmap e324: 12 total, 11 up, 11 in 2024-04-15T18:22:16.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:16 smithi049 ceph-mon[40087]: pgmap v887: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 44 MiB/s wr, 760 op/s; 8026/266535 objects degraded (3.011%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:22:16.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:16.431 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:16 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:16.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:16 smithi082 ceph-mon[44206]: osdmap e324: 12 total, 11 up, 11 in 2024-04-15T18:22:16.541 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:16 smithi082 ceph-mon[44206]: pgmap v887: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 44 MiB/s wr, 760 op/s; 8026/266535 objects degraded (3.011%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:22:16.553 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:16.558 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:16.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:16.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:16 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:16.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3564845950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:16.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:16 smithi159 ceph-mon[45336]: osdmap e324: 12 total, 11 up, 11 in 2024-04-15T18:22:16.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:16 smithi159 ceph-mon[45336]: pgmap v887: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 44 MiB/s wr, 760 op/s; 8026/266535 objects degraded (3.011%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:22:16.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:16.710 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:16.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:16.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:16.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:16.980 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:17.004 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:17.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:17.063 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 4.4 info 4.4( v 307'3739 (0'0,307'3739] local-lis/les=284/285 n=3739 ec=284/284 lis/c=284/284 les/c/f=285/285/0 sis=284) 2024-04-15T18:22:17.081 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2002b423:::benchmark_data_smithi159_102124_object14057:head# 2024-04-15T18:22:17.081 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20034ce2:::benchmark_data_smithi159_102124_object30971:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200485e2:::benchmark_data_smithi159_102124_object21128:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2005116f:::benchmark_data_smithi159_102124_object27139:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2005c650:::benchmark_data_smithi159_102124_object12512:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20069ff3:::benchmark_data_smithi159_102124_object17449:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2007ebb4:::benchmark_data_smithi159_102124_object35755:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2008198e:::benchmark_data_smithi159_102124_object46986:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2008c0bd:::benchmark_data_smithi159_102124_object12874:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200c01d8:::benchmark_data_smithi159_102124_object50121:head# 2024-04-15T18:22:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200d2aa7:::benchmark_data_smithi159_102124_object39058:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200e2b33:::benchmark_data_smithi159_102124_object43455:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200e2b35:::benchmark_data_smithi159_102124_object32979:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200e92a8:::benchmark_data_smithi159_102124_object24538:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:200ead45:::benchmark_data_smithi159_102124_object15740:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20124c03:::benchmark_data_smithi159_102124_object27553:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20128afa:::benchmark_data_smithi159_102124_object42690:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2012c4cf:::benchmark_data_smithi159_102124_object15287:head# 2024-04-15T18:22:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2012ca35:::benchmark_data_smithi159_102124_object36486:head# 2024-04-15T18:22:17.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2015a747:::benchmark_data_smithi159_102124_object51617:head# 2024-04-15T18:22:17.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2015c385:::benchmark_data_smithi159_102124_object27400:head# 2024-04-15T18:22:17.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201758c7:::benchmark_data_smithi159_102124_object21390:head# 2024-04-15T18:22:17.084 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201827db:::benchmark_data_smithi159_102124_object36460:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201945a3:::benchmark_data_smithi159_102124_object27824:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20198f2a:::benchmark_data_smithi159_102124_object25295:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201c5c6e:::benchmark_data_smithi159_102124_object5739:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201eea46:::benchmark_data_smithi159_102124_object28513:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:201ef650:::benchmark_data_smithi159_102124_object41882:head# 2024-04-15T18:22:17.085 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2020e8ed:::benchmark_data_smithi159_102124_object25858:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2023c2e5:::benchmark_data_smithi159_102124_object57657:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20255032:::benchmark_data_smithi159_102124_object50145:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20258d40:::benchmark_data_smithi159_102124_object45326:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2025df64:::benchmark_data_smithi159_102124_object52586:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2026ea31:::benchmark_data_smithi159_102124_object5326:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202a2903:::benchmark_data_smithi159_102124_object30610:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202acfc7:::benchmark_data_smithi159_102124_object38797:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202c814a:::benchmark_data_smithi159_102124_object50573:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202d3099:::benchmark_data_smithi159_102124_object27081:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202e5c09:::benchmark_data_smithi159_102124_object8480:head# 2024-04-15T18:22:17.087 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202eb7d5:::benchmark_data_smithi159_102124_object17694:head# 2024-04-15T18:22:17.088 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:202f5b20:::benchmark_data_smithi159_102124_object23565:head# 2024-04-15T18:22:17.090 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20300349:::benchmark_data_smithi159_102124_object26069:head# 2024-04-15T18:22:17.091 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2030ab34:::benchmark_data_smithi159_102124_object4913:head# 2024-04-15T18:22:17.091 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2031a3a7:::benchmark_data_smithi159_102124_object8840:head# 2024-04-15T18:22:17.091 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2031bc86:::benchmark_data_smithi159_102124_object12033:head# 2024-04-15T18:22:17.091 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20336a88:::benchmark_data_smithi159_102124_object17633:head# 2024-04-15T18:22:17.091 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20344fee:::benchmark_data_smithi159_102124_object29997:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20345a40:::benchmark_data_smithi159_102124_object4838:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2034b69a:::benchmark_data_smithi159_102124_object14981:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2035824c:::benchmark_data_smithi159_102124_object53453:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203608eb:::benchmark_data_smithi159_102124_object2093:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2036aa85:::benchmark_data_smithi159_102124_object25617:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20390c6e:::benchmark_data_smithi159_102124_object53439:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20398480:::benchmark_data_smithi159_102124_object239:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203a6d3c:::benchmark_data_smithi159_102124_object51633:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203ae260:::benchmark_data_smithi159_102124_object27762:head# 2024-04-15T18:22:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203bfddb:::benchmark_data_smithi159_102124_object50243:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203cc020:::benchmark_data_smithi159_102124_object19707:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203d3bea:::benchmark_data_smithi159_102124_object11342:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203e6a4c:::benchmark_data_smithi159_102124_object24601:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:203fa210:::benchmark_data_smithi159_102124_object24881:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20417cf1:::benchmark_data_smithi159_102124_object13297:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2041bd7c:::benchmark_data_smithi159_102124_object2557:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2042404f:::benchmark_data_smithi159_102124_object8955:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20425048:::benchmark_data_smithi159_102124_object15991:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2042589f:::benchmark_data_smithi159_102124_object4825:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20441dec:::benchmark_data_smithi159_102124_object27774:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:204455cd:::benchmark_data_smithi159_102124_object39666:head# 2024-04-15T18:22:17.094 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2047bd30:::benchmark_data_smithi159_102124_object16977:head# 2024-04-15T18:22:17.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2048866c:::benchmark_data_smithi159_102124_object5078:head# 2024-04-15T18:22:17.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2048c8d6:::benchmark_data_smithi159_102124_object18731:head# 2024-04-15T18:22:17.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20495530:::benchmark_data_smithi159_102124_object11873:head# 2024-04-15T18:22:17.095 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:204b7191:::benchmark_data_smithi159_102124_object43021:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:204bd1c8:::benchmark_data_smithi159_102124_object15372:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:204c059f:::benchmark_data_smithi159_102124_object35483:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:204f8154:::benchmark_data_smithi159_102124_object43462:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20523036:::benchmark_data_smithi159_102124_object5239:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2052de54:::benchmark_data_smithi159_102124_object22054:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20531996:::benchmark_data_smithi159_102124_object5834:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205454da:::benchmark_data_smithi159_102124_object46974:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205565ff:::benchmark_data_smithi159_102124_object34903:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20556f64:::benchmark_data_smithi159_102124_object41820:head# 2024-04-15T18:22:17.096 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2055b78f:::benchmark_data_smithi159_102124_object40211:head# 2024-04-15T18:22:17.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2055ebf3:::benchmark_data_smithi159_102124_object36473:head# 2024-04-15T18:22:17.097 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2056d863:::benchmark_data_smithi159_102124_object5776:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2056dc94:::benchmark_data_smithi159_102124_object17662:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20577af0:::benchmark_data_smithi159_102124_object40027:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205889c1:::benchmark_data_smithi159_102124_object17858:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20590408:::benchmark_data_smithi159_102124_object1363:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205ab778:::benchmark_data_smithi159_102124_object17621:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205b40e6:::benchmark_data_smithi159_102124_object19579:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205baad0:::benchmark_data_smithi159_102124_object7601:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205cf416:::benchmark_data_smithi159_102124_object28128:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205df261:::benchmark_data_smithi159_102124_object19490:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205e6d3a:::benchmark_data_smithi159_102124_object55275:head# 2024-04-15T18:22:17.099 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205f0788:::benchmark_data_smithi159_102124_object11154:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205f2217:::benchmark_data_smithi159_102124_object17618:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:205f4f56:::benchmark_data_smithi159_102124_object3924:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20607971:::benchmark_data_smithi159_102124_object42872:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20610c77:::benchmark_data_smithi159_102124_object23614:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20615d1d:::benchmark_data_smithi159_102124_object17668:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206386ab:::benchmark_data_smithi159_102124_object49676:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206524a0:::benchmark_data_smithi159_102124_object42563:head# 2024-04-15T18:22:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206624f3:::benchmark_data_smithi159_102124_object17120:head# 2024-04-15T18:22:17.102 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20677688:::benchmark_data_smithi159_102124_object57316:head# 2024-04-15T18:22:17.102 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206974cc:::benchmark_data_smithi159_102124_object53293:head# 2024-04-15T18:22:17.102 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2069fb79:::benchmark_data_smithi159_102124_object1096:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206c12e4:::benchmark_data_smithi159_102124_object27614:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:206eb8a6:::benchmark_data_smithi159_102124_object36408:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20703b1f:::benchmark_data_smithi159_102124_object34172:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20724302:::benchmark_data_smithi159_102124_object17797:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20739c55:::benchmark_data_smithi159_102124_object50592:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2073ea2d:::benchmark_data_smithi159_102124_object2133:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207757cd:::benchmark_data_smithi159_102124_object33714:head# 2024-04-15T18:22:17.103 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20779e60:::benchmark_data_smithi159_102124_object55158:head# 2024-04-15T18:22:17.105 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207a3a2c:::benchmark_data_smithi159_102124_object6387:head# 2024-04-15T18:22:17.105 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207b11a7:::benchmark_data_smithi159_102124_object18272:head# 2024-04-15T18:22:17.105 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207d39f8:::benchmark_data_smithi159_102124_object11330:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207d757e:::benchmark_data_smithi159_102124_object32346:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207ea96d:::benchmark_data_smithi159_102124_object37092:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207ed40c:::benchmark_data_smithi159_102124_object27670:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:207ffb29:::benchmark_data_smithi159_102124_object36426:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2080807d:::benchmark_data_smithi159_102124_object7828:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2080d55c:::benchmark_data_smithi159_102124_object39213:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2081c31a:::benchmark_data_smithi159_102124_object42050:head# 2024-04-15T18:22:17.106 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2081d3b6:::benchmark_data_smithi159_102124_object5144:head# 2024-04-15T18:22:17.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2082d402:::benchmark_data_smithi159_102124_object56537:head# 2024-04-15T18:22:17.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20830ab0:::benchmark_data_smithi159_102124_object480:head# 2024-04-15T18:22:17.108 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208318fd:::benchmark_data_smithi159_102124_object11011:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20857169:::benchmark_data_smithi159_102124_object56020:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208b2b49:::benchmark_data_smithi159_102124_object57852:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208c097c:::benchmark_data_smithi159_102124_object2964:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208c1a44:::benchmark_data_smithi159_102124_object42579:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208c4dab:::benchmark_data_smithi159_102124_object6635:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208c81a4:::benchmark_data_smithi159_102124_object52432:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208caa9d:::benchmark_data_smithi159_102124_object37682:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208cbcfa:::benchmark_data_smithi159_102124_object54502:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208de8c8:::benchmark_data_smithi159_102124_object58424:head# 2024-04-15T18:22:17.109 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:208def1b:::benchmark_data_smithi159_102124_object10018:head# 2024-04-15T18:22:17.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20902f8d:::benchmark_data_smithi159_102124_object22983:head# 2024-04-15T18:22:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2091044e:::benchmark_data_smithi159_102124_object44129:head# 2024-04-15T18:22:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:209311ac:::benchmark_data_smithi159_102124_object28384:head# 2024-04-15T18:22:17.111 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20982c70:::benchmark_data_smithi159_102124_object22063:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:209926f8:::benchmark_data_smithi159_102124_object34416:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:209cd857:::benchmark_data_smithi159_102124_object48211:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a2f36d:::benchmark_data_smithi159_102124_object48966:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a2f572:::benchmark_data_smithi159_102124_object55257:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a3ae84:::benchmark_data_smithi159_102124_object44235:head# 2024-04-15T18:22:17.112 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a52623:::benchmark_data_smithi159_102124_object47631:head# 2024-04-15T18:22:17.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a538d8:::benchmark_data_smithi159_102124_object26502:head# 2024-04-15T18:22:17.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a6c7c3:::benchmark_data_smithi159_102124_object44614:head# 2024-04-15T18:22:17.113 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a729b7:::benchmark_data_smithi159_102124_object10166:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20a929d8:::benchmark_data_smithi159_102124_object30223:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20aa2640:::benchmark_data_smithi159_102124_object3212:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ae15d4:::benchmark_data_smithi159_102124_object47168:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20aff8f2:::benchmark_data_smithi159_102124_object40257:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b014f4:::benchmark_data_smithi159_102124_object17661:head# 2024-04-15T18:22:17.117 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b05ff5:::benchmark_data_smithi159_102124_object38310:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b06f52:::benchmark_data_smithi159_102124_object11630:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b2be78:::benchmark_data_smithi159_102124_object58053:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b35b4b:::benchmark_data_smithi159_102124_object25779:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b3769b:::benchmark_data_smithi159_102124_object49247:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b37caa:::benchmark_data_smithi159_102124_object41148:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b42c23:::benchmark_data_smithi159_102124_object44121:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b62b51:::benchmark_data_smithi159_102124_object19670:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b65f51:::benchmark_data_smithi159_102124_object4134:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b679ae:::benchmark_data_smithi159_102124_object35689:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b7d5be:::benchmark_data_smithi159_102124_object32836:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b854be:::benchmark_data_smithi159_102124_object20357:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b8d337:::benchmark_data_smithi159_102124_object8926:head# 2024-04-15T18:22:17.118 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b907e1:::benchmark_data_smithi159_102124_object9787:head# 2024-04-15T18:22:17.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20b9fb2b:::benchmark_data_smithi159_102124_object864:head# 2024-04-15T18:22:17.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ba8b00:::benchmark_data_smithi159_102124_object55394:head# 2024-04-15T18:22:17.119 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20bc1027:::benchmark_data_smithi159_102124_object19448:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20bc5247:::benchmark_data_smithi159_102124_object28839:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20be3603:::benchmark_data_smithi159_102124_object13024:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20bf4b43:::benchmark_data_smithi159_102124_object2999:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c18f39:::benchmark_data_smithi159_102124_object19236:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c24a78:::benchmark_data_smithi159_102124_object26655:head# 2024-04-15T18:22:17.120 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c271e4:::benchmark_data_smithi159_102124_object54991:head# 2024-04-15T18:22:17.121 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c32e97:::benchmark_data_smithi159_102124_object757:head# 2024-04-15T18:22:17.121 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c46fe1:::benchmark_data_smithi159_102124_object20860:head# 2024-04-15T18:22:17.121 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c66b8c:::benchmark_data_smithi159_102124_object12720:head# 2024-04-15T18:22:17.121 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c6b1e8:::benchmark_data_smithi159_102124_object48768:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c7b055:::benchmark_data_smithi159_102124_object49920:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c81674:::benchmark_data_smithi159_102124_object9359:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c92a65:::benchmark_data_smithi159_102124_object40223:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20c96c31:::benchmark_data_smithi159_102124_object37575:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20cab4da:::benchmark_data_smithi159_102124_object40742:head# 2024-04-15T18:22:17.122 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20cb4dfd:::benchmark_data_smithi159_102124_object27477:head# 2024-04-15T18:22:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20cb7a8a:::benchmark_data_smithi159_102124_object7882:head# 2024-04-15T18:22:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ced747:::benchmark_data_smithi159_102124_object57165:head# 2024-04-15T18:22:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20cfa261:::benchmark_data_smithi159_102124_object52555:head# 2024-04-15T18:22:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d0930d:::benchmark_data_smithi159_102124_object11383:head# 2024-04-15T18:22:17.123 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d53418:::benchmark_data_smithi159_102124_object22105:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d5355c:::benchmark_data_smithi159_102124_object36607:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d552e9:::benchmark_data_smithi159_102124_object50398:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d71de0:::benchmark_data_smithi159_102124_object23175:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d8068d:::benchmark_data_smithi159_102124_object54564:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d80fc1:::benchmark_data_smithi159_102124_object46614:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20d8a32f:::benchmark_data_smithi159_102124_object37271:head# 2024-04-15T18:22:17.125 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20dbfa1c:::benchmark_data_smithi159_102124_object51889:head# 2024-04-15T18:22:17.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e07980:::benchmark_data_smithi159_102124_object14486:head# 2024-04-15T18:22:17.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e0e09a:::benchmark_data_smithi159_102124_object52301:head# 2024-04-15T18:22:17.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e2eaec:::benchmark_data_smithi159_102124_object46289:head# 2024-04-15T18:22:17.126 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e2f91b:::benchmark_data_smithi159_102124_object46513:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e4848e:::benchmark_data_smithi159_102124_object23974:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e5587c:::benchmark_data_smithi159_102124_object40972:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e6e9e7:::benchmark_data_smithi159_102124_object19619:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e7d20c:::benchmark_data_smithi159_102124_object42257:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e95155:::benchmark_data_smithi159_102124_object37982:head# 2024-04-15T18:22:17.127 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e99436:::benchmark_data_smithi159_102124_object48198:head# 2024-04-15T18:22:17.128 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20e9b9ca:::benchmark_data_smithi159_102124_object24211:head# 2024-04-15T18:22:17.128 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ea7107:::benchmark_data_smithi159_102124_object38655:head# 2024-04-15T18:22:17.128 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ea7da1:::benchmark_data_smithi159_102124_object28980:head# 2024-04-15T18:22:17.128 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ea97ed:::benchmark_data_smithi159_102124_object45060:head# 2024-04-15T18:22:17.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20eac977:::benchmark_data_smithi159_102124_object22590:head# 2024-04-15T18:22:17.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ec04b1:::benchmark_data_smithi159_102124_object13107:head# 2024-04-15T18:22:17.129 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ecc601:::benchmark_data_smithi159_102124_object26453:head# 2024-04-15T18:22:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ed23d2:::benchmark_data_smithi159_102124_object53641:head# 2024-04-15T18:22:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ed4268:::benchmark_data_smithi159_102124_object19261:head# 2024-04-15T18:22:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f0938a:::benchmark_data_smithi159_102124_object22787:head# 2024-04-15T18:22:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f19296:::benchmark_data_smithi159_102124_object7064:head# 2024-04-15T18:22:17.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f1eff3:::benchmark_data_smithi159_102124_object44072:head# 2024-04-15T18:22:17.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f32bbb:::benchmark_data_smithi159_102124_object32698:head# 2024-04-15T18:22:17.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f34009:::benchmark_data_smithi159_102124_object3985:head# 2024-04-15T18:22:17.133 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f41d99:::benchmark_data_smithi159_102124_object46269:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f657ea:::benchmark_data_smithi159_102124_object57053:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f7446e:::benchmark_data_smithi159_102124_object23190:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f96faf:::benchmark_data_smithi159_102124_object36069:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20f9d68a:::benchmark_data_smithi159_102124_object13576:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20fcb9be:::benchmark_data_smithi159_102124_object25950:head# 2024-04-15T18:22:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20fdf52a:::benchmark_data_smithi159_102124_object22930:head# 2024-04-15T18:22:17.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:20ff45a1:::benchmark_data_smithi159_102124_object9143:head# 2024-04-15T18:22:17.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2100c3b8:::benchmark_data_smithi159_102124_object1001:head# 2024-04-15T18:22:17.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21029175:::benchmark_data_smithi159_102124_object16010:head# 2024-04-15T18:22:17.135 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21032fcb:::benchmark_data_smithi159_102124_object1278:head# 2024-04-15T18:22:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21039aaa:::benchmark_data_smithi159_102124_object18478:head# 2024-04-15T18:22:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2103ba37:::benchmark_data_smithi159_102124_object29878:head# 2024-04-15T18:22:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210500bb:::benchmark_data_smithi159_102124_object21264:head# 2024-04-15T18:22:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21056823:::benchmark_data_smithi159_102124_object4290:head# 2024-04-15T18:22:17.136 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2105e3f0:::benchmark_data_smithi159_102124_object15791:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21080ad4:::benchmark_data_smithi159_102124_object21338:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2108d6ba:::benchmark_data_smithi159_102124_object27800:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210955e4:::benchmark_data_smithi159_102124_object34321:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210abee9:::benchmark_data_smithi159_102124_object52623:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210cde2e:::benchmark_data_smithi159_102124_object40360:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210e0003:::benchmark_data_smithi159_102124_object44886:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210e9b26:::benchmark_data_smithi159_102124_object42547:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210ec994:::benchmark_data_smithi159_102124_object47362:head# 2024-04-15T18:22:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210ed748:::benchmark_data_smithi159_102124_object16425:head# 2024-04-15T18:22:17.138 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:210f5a5e:::benchmark_data_smithi159_102124_object22173:head# 2024-04-15T18:22:17.138 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21119a05:::benchmark_data_smithi159_102124_object23401:head# 2024-04-15T18:22:17.138 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21158612:::benchmark_data_smithi159_102124_object13237:head# 2024-04-15T18:22:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211727b8:::benchmark_data_smithi159_102124_object14592:head# 2024-04-15T18:22:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2117f622:::benchmark_data_smithi159_102124_object11718:head# 2024-04-15T18:22:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2117fd0b:::benchmark_data_smithi159_102124_object17956:head# 2024-04-15T18:22:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211b544b:::benchmark_data_smithi159_102124_object19908:head# 2024-04-15T18:22:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211c0d93:::benchmark_data_smithi159_102124_object29780:head# 2024-04-15T18:22:17.142 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211c5f23:::benchmark_data_smithi159_102124_object54321:head# 2024-04-15T18:22:17.142 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211d3a83:::benchmark_data_smithi159_102124_object961:head# 2024-04-15T18:22:17.142 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211db8e2:::benchmark_data_smithi159_102124_object47510:head# 2024-04-15T18:22:17.142 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:211e59ed:::benchmark_data_smithi159_102124_object387:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212063af:::benchmark_data_smithi159_102124_object8957:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212263d2:::benchmark_data_smithi159_102124_object29752:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212496c3:::benchmark_data_smithi159_102124_object26091:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21256713:::benchmark_data_smithi159_102124_object149:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2127500f:::benchmark_data_smithi159_102124_object32025:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2129a303:::benchmark_data_smithi159_102124_object30412:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212a6839:::benchmark_data_smithi159_102124_object4120:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212a7969:::benchmark_data_smithi159_102124_object23776:head# 2024-04-15T18:22:17.144 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212b1d80:::benchmark_data_smithi159_102124_object14003:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212b2837:::benchmark_data_smithi159_102124_object58315:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212b2ac9:::benchmark_data_smithi159_102124_object15676:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212ba875:::benchmark_data_smithi159_102124_object31129:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212bb32d:::benchmark_data_smithi159_102124_object18027:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212cff9a:::benchmark_data_smithi159_102124_object23718:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212d1f8d:::benchmark_data_smithi159_102124_object54215:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:212d2466:::benchmark_data_smithi159_102124_object27866:head# 2024-04-15T18:22:17.146 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21307b55:::benchmark_data_smithi159_102124_object5383:head# 2024-04-15T18:22:17.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2130a8dc:::benchmark_data_smithi159_102124_object22375:head# 2024-04-15T18:22:17.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2130aa32:::benchmark_data_smithi159_102124_object30275:head# 2024-04-15T18:22:17.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21316319:::benchmark_data_smithi159_102124_object27267:head# 2024-04-15T18:22:17.147 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2132d0da:::benchmark_data_smithi159_102124_object52001:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21355151:::benchmark_data_smithi159_102124_object52577:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2136f866:::benchmark_data_smithi159_102124_object13139:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2137a84a:::benchmark_data_smithi159_102124_object9776:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213820c4:::benchmark_data_smithi159_102124_object18121:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2139128c:::benchmark_data_smithi159_102124_object51409:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213acd08:::benchmark_data_smithi159_102124_object51205:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213b7ad4:::benchmark_data_smithi159_102124_object58282:head# 2024-04-15T18:22:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213b9087:::benchmark_data_smithi159_102124_object31877:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213cf459:::benchmark_data_smithi159_102124_object10708:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213f46f6:::benchmark_data_smithi159_102124_object7836:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:213fd50a:::benchmark_data_smithi159_102124_object12673:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:214055c6:::benchmark_data_smithi159_102124_object45632:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21409619:::benchmark_data_smithi159_102124_object25675:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2140f065:::benchmark_data_smithi159_102124_object726:head# 2024-04-15T18:22:17.151 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2142327c:::benchmark_data_smithi159_102124_object22962:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21430502:::benchmark_data_smithi159_102124_object8606:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2143cf58:::benchmark_data_smithi159_102124_object42326:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2144c5e9:::benchmark_data_smithi159_102124_object46492:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21450511:::benchmark_data_smithi159_102124_object29553:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21472c3c:::benchmark_data_smithi159_102124_object690:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2149db68:::benchmark_data_smithi159_102124_object48695:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:214a93aa:::benchmark_data_smithi159_102124_object26912:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:214b1d5f:::benchmark_data_smithi159_102124_object35460:head# 2024-04-15T18:22:17.153 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:214c1641:::benchmark_data_smithi159_102124_object5649:head# 2024-04-15T18:22:17.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215011bd:::benchmark_data_smithi159_102124_object42192:head# 2024-04-15T18:22:17.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21503753:::benchmark_data_smithi159_102124_object31352:head# 2024-04-15T18:22:17.154 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2150924b:::benchmark_data_smithi159_102124_object4490:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21511b04:::benchmark_data_smithi159_102124_object55808:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21516d3e:::benchmark_data_smithi159_102124_object15856:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215449ad:::benchmark_data_smithi159_102124_object37484:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21550dcc:::benchmark_data_smithi159_102124_object42365:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21552d7b:::benchmark_data_smithi159_102124_object33204:head# 2024-04-15T18:22:17.155 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21589327:::benchmark_data_smithi159_102124_object43601:head# 2024-04-15T18:22:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215939a8:::benchmark_data_smithi159_102124_object15382:head# 2024-04-15T18:22:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215b27a1:::benchmark_data_smithi159_102124_object56026:head# 2024-04-15T18:22:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215d0e2a:::benchmark_data_smithi159_102124_object15131:head# 2024-04-15T18:22:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215d8615:::benchmark_data_smithi159_102124_object20445:head# 2024-04-15T18:22:17.156 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215e974c:::benchmark_data_smithi159_102124_object45159:head# 2024-04-15T18:22:17.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:215faa8d:::benchmark_data_smithi159_102124_object40904:head# 2024-04-15T18:22:17.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2162a325:::benchmark_data_smithi159_102124_object15219:head# 2024-04-15T18:22:17.157 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2162fc02:::benchmark_data_smithi159_102124_object5582:head# 2024-04-15T18:22:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2163882f:::benchmark_data_smithi159_102124_object57278:head# 2024-04-15T18:22:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21641e34:::benchmark_data_smithi159_102124_object49395:head# 2024-04-15T18:22:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216424e9:::benchmark_data_smithi159_102124_object49156:head# 2024-04-15T18:22:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2164a658:::benchmark_data_smithi159_102124_object44761:head# 2024-04-15T18:22:17.158 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21650214:::benchmark_data_smithi159_102124_object116:head# 2024-04-15T18:22:17.159 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2167af4f:::benchmark_data_smithi159_102124_object58171:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2168b20e:::benchmark_data_smithi159_102124_object37719:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2169320c:::benchmark_data_smithi159_102124_object44242:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21694f72:::benchmark_data_smithi159_102124_object37819:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216a1960:::benchmark_data_smithi159_102124_object11013:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216a523c:::benchmark_data_smithi159_102124_object11307:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216ae693:::benchmark_data_smithi159_102124_object5251:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216c5ae0:::benchmark_data_smithi159_102124_object26065:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216e1101:::benchmark_data_smithi159_102124_object26576:head# 2024-04-15T18:22:17.163 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216e20f6:::benchmark_data_smithi159_102124_object3703:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216e2512:::benchmark_data_smithi159_102124_object40780:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:216fe97b:::benchmark_data_smithi159_102124_object14951:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21713d86:::benchmark_data_smithi159_102124_object38842:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2172cfd6:::benchmark_data_smithi159_102124_object46892:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2174db49:::benchmark_data_smithi159_102124_object9415:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2175c08a:::benchmark_data_smithi159_102124_object47639:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2176dae6:::benchmark_data_smithi159_102124_object41772:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:217813af:::benchmark_data_smithi159_102124_object57448:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21787f2d:::benchmark_data_smithi159_102124_object43890:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21794478:::benchmark_data_smithi159_102124_object35801:head# 2024-04-15T18:22:17.164 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:217a39d4:::benchmark_data_smithi159_102124_object20977:head# 2024-04-15T18:22:17.165 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:217a75bf:::benchmark_data_smithi159_102124_object50927:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2180526c:::benchmark_data_smithi159_102124_object21981:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218148dc:::benchmark_data_smithi159_102124_object33967:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2181b82a:::benchmark_data_smithi159_102124_object52265:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21820892:::benchmark_data_smithi159_102124_object33839:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21827e4f:::benchmark_data_smithi159_102124_object41071:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2182af08:::benchmark_data_smithi159_102124_object16885:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2182d27d:::benchmark_data_smithi159_102124_object22128:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2184bfa7:::benchmark_data_smithi159_102124_object44792:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21858c83:::benchmark_data_smithi159_102124_object10588:head# 2024-04-15T18:22:17.166 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2185b428:::benchmark_data_smithi159_102124_object53861:head# 2024-04-15T18:22:17.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218792ed:::benchmark_data_smithi159_102124_object31699:head# 2024-04-15T18:22:17.167 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2187feff:::benchmark_data_smithi159_102124_object12684:head# 2024-04-15T18:22:17.167 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2188e7d7:::benchmark_data_smithi159_102124_object21449:head# 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218c6f77:::benchmark_data_smithi159_102124_object9642:head# 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218e813a:::benchmark_data_smithi159_102124_object13699:head# 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218e9416:::benchmark_data_smithi159_102124_object11683:head# 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:218f0f5d:::benchmark_data_smithi159_102124_object1040:head# 2024-04-15T18:22:17.168 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21903bbf:::benchmark_data_smithi159_102124_object5439:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21909b02:::benchmark_data_smithi159_102124_object47834:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2190e58d:::benchmark_data_smithi159_102124_object39893:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21917caa:::benchmark_data_smithi159_102124_object37805:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219269cd:::benchmark_data_smithi159_102124_object26366:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2192ee1a:::benchmark_data_smithi159_102124_object50117:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2193ac2f:::benchmark_data_smithi159_102124_object14564:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2194fb9c:::benchmark_data_smithi159_102124_object47975:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219711c8:::benchmark_data_smithi159_102124_object30314:head# 2024-04-15T18:22:17.169 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21978e80:::benchmark_data_smithi159_102124_object37662:head# 2024-04-15T18:22:17.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2199906f:::benchmark_data_smithi159_102124_object20320:head# 2024-04-15T18:22:17.170 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2199e215:::benchmark_data_smithi159_102124_object44915:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219b1150:::benchmark_data_smithi159_102124_object32766:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219c6000:::benchmark_data_smithi159_102124_object39600:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219d3fac:::benchmark_data_smithi159_102124_object35418:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219def4c:::benchmark_data_smithi159_102124_object11194:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219e509a:::benchmark_data_smithi159_102124_object40654:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219e5594:::benchmark_data_smithi159_102124_object34058:head# 2024-04-15T18:22:17.172 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219e594d:::benchmark_data_smithi159_102124_object3269:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:219fcc22:::benchmark_data_smithi159_102124_object16745:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a03b7b:::benchmark_data_smithi159_102124_object8598:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a0c15a:::benchmark_data_smithi159_102124_object1865:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a28b3e:::benchmark_data_smithi159_102124_object1638:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a2fbee:::benchmark_data_smithi159_102124_object15896:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a3d4a3:::benchmark_data_smithi159_102124_object29788:head# 2024-04-15T18:22:17.173 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a4b1ad:::benchmark_data_smithi159_102124_object32033:head# 2024-04-15T18:22:17.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a714ab:::benchmark_data_smithi159_102124_object47735:head# 2024-04-15T18:22:17.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a740d5:::benchmark_data_smithi159_102124_object35604:head# 2024-04-15T18:22:17.174 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21a760b4:::benchmark_data_smithi159_102124_object16044:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21aa3209:::benchmark_data_smithi159_102124_object731:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ab302f:::benchmark_data_smithi159_102124_object20259:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21abd52d:::benchmark_data_smithi159_102124_object51752:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21abda6d:::benchmark_data_smithi159_102124_object23476:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ac223a:::benchmark_data_smithi159_102124_object51379:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ac6da5:::benchmark_data_smithi159_102124_object6282:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21adb0d3:::benchmark_data_smithi159_102124_object16890:head# 2024-04-15T18:22:17.175 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ae6e86:::benchmark_data_smithi159_102124_object19289:head# 2024-04-15T18:22:17.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ae884c:::benchmark_data_smithi159_102124_object95:head# 2024-04-15T18:22:17.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b1673e:::benchmark_data_smithi159_102124_object33951:head# 2024-04-15T18:22:17.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b18526:::benchmark_data_smithi159_102124_object23090:head# 2024-04-15T18:22:17.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b1dc79:::benchmark_data_smithi159_102124_object20772:head# 2024-04-15T18:22:17.176 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b1e576:::benchmark_data_smithi159_102124_object41472:head# 2024-04-15T18:22:17.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b3c028:::benchmark_data_smithi159_102124_object18025:head# 2024-04-15T18:22:17.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b4204d:::benchmark_data_smithi159_102124_object14953:head# 2024-04-15T18:22:17.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b517fe:::benchmark_data_smithi159_102124_object26620:head# 2024-04-15T18:22:17.177 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b5236e:::benchmark_data_smithi159_102124_object24012:head# 2024-04-15T18:22:17.178 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b56ed3:::benchmark_data_smithi159_102124_object29454:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b59813:::benchmark_data_smithi159_102124_object2734:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b75c47:::benchmark_data_smithi159_102124_object3102:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b77eac:::benchmark_data_smithi159_102124_object36311:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b86237:::benchmark_data_smithi159_102124_object3829:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b8ac2a:::benchmark_data_smithi159_102124_object11209:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b9012f:::benchmark_data_smithi159_102124_object17853:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21b9d344:::benchmark_data_smithi159_102124_object39400:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21bc061e:::benchmark_data_smithi159_102124_object28747:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21bcd092:::benchmark_data_smithi159_102124_object48820:head# 2024-04-15T18:22:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21bd168b:::benchmark_data_smithi159_102124_object37388:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21bd9eec:::benchmark_data_smithi159_102124_object41535:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21bde605:::benchmark_data_smithi159_102124_object37817:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21c03b1b:::benchmark_data_smithi159_102124_object57621:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21c3f8e9:::benchmark_data_smithi159_102124_object47447:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21c6eb93:::benchmark_data_smithi159_102124_object33830:head# 2024-04-15T18:22:17.180 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21c8502b:::benchmark_data_smithi159_102124_object53875:head# 2024-04-15T18:22:17.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21cb7591:::benchmark_data_smithi159_102124_object31736:head# 2024-04-15T18:22:17.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21cb795b:::benchmark_data_smithi159_102124_object845:head# 2024-04-15T18:22:17.184 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21cbec35:::benchmark_data_smithi159_102124_object11942:head# 2024-04-15T18:22:17.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21cc583c:::benchmark_data_smithi159_102124_object14336:head# 2024-04-15T18:22:17.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21cd1bc5:::benchmark_data_smithi159_102124_object31171:head# 2024-04-15T18:22:17.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d23982:::benchmark_data_smithi159_102124_object31612:head# 2024-04-15T18:22:17.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d27383:::benchmark_data_smithi159_102124_object57459:head# 2024-04-15T18:22:17.185 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d2f2d8:::benchmark_data_smithi159_102124_object40689:head# 2024-04-15T18:22:17.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d5e170:::benchmark_data_smithi159_102124_object16854:head# 2024-04-15T18:22:17.186 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d5ede0:::benchmark_data_smithi159_102124_object12764:head# 2024-04-15T18:22:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d69091:::benchmark_data_smithi159_102124_object45791:head# 2024-04-15T18:22:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d6a3cb:::benchmark_data_smithi159_102124_object52302:head# 2024-04-15T18:22:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d6abc7:::benchmark_data_smithi159_102124_object8756:head# 2024-04-15T18:22:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d7a63a:::benchmark_data_smithi159_102124_object53902:head# 2024-04-15T18:22:17.187 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d83379:::benchmark_data_smithi159_102124_object7184:head# 2024-04-15T18:22:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d927ca:::benchmark_data_smithi159_102124_object36040:head# 2024-04-15T18:22:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21d977e5:::benchmark_data_smithi159_102124_object8447:head# 2024-04-15T18:22:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21dac676:::benchmark_data_smithi159_102124_object4130:head# 2024-04-15T18:22:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21dc984d:::benchmark_data_smithi159_102124_object4419:head# 2024-04-15T18:22:17.188 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ddf846:::benchmark_data_smithi159_102124_object40998:head# 2024-04-15T18:22:17.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21dec006:::benchmark_data_smithi159_102124_object16577:head# 2024-04-15T18:22:17.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21dfc55a:::benchmark_data_smithi159_102124_object14317:head# 2024-04-15T18:22:17.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e059ec:::benchmark_data_smithi159_102124_object41638:head# 2024-04-15T18:22:17.191 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e16c18:::benchmark_data_smithi159_102124_object38544:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e16ee8:::benchmark_data_smithi159_102124_object28134:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e300b2:::benchmark_data_smithi159_102124_object41275:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e33427:::benchmark_data_smithi159_102124_object30011:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e4300e:::benchmark_data_smithi159_102124_object47164:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e4b672:::benchmark_data_smithi159_102124_object17803:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e4c11b:::benchmark_data_smithi159_102124_object32100:head# 2024-04-15T18:22:17.192 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e59bf4:::benchmark_data_smithi159_102124_object49949:head# 2024-04-15T18:22:17.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e6aaf1:::benchmark_data_smithi159_102124_object36193:head# 2024-04-15T18:22:17.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21e83f38:::benchmark_data_smithi159_102124_object35869:head# 2024-04-15T18:22:17.193 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ea96cb:::benchmark_data_smithi159_102124_object12523:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21eab6b2:::benchmark_data_smithi159_102124_object3231:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ec200a:::benchmark_data_smithi159_102124_object17475:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f08d2f:::benchmark_data_smithi159_102124_object25072:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f0f705:::benchmark_data_smithi159_102124_object5917:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f178c6:::benchmark_data_smithi159_102124_object55976:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f1e2de:::benchmark_data_smithi159_102124_object39771:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f3161d:::benchmark_data_smithi159_102124_object37151:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f37e4c:::benchmark_data_smithi159_102124_object32738:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f4826a:::benchmark_data_smithi159_102124_object16291:head# 2024-04-15T18:22:17.194 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f6a652:::benchmark_data_smithi159_102124_object51759:head# 2024-04-15T18:22:17.195 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21f75f14:::benchmark_data_smithi159_102124_object57687:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21fa5862:::benchmark_data_smithi159_102124_object7543:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21fa72bb:::benchmark_data_smithi159_102124_object19248:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21fd0ab5:::benchmark_data_smithi159_102124_object35175:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21fdc14a:::benchmark_data_smithi159_102124_object39285:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21fdcc52:::benchmark_data_smithi159_102124_object24558:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ff0600:::benchmark_data_smithi159_102124_object33382:head# 2024-04-15T18:22:17.196 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:21ff2c75:::benchmark_data_smithi159_102124_object22997:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2200293e:::benchmark_data_smithi159_102124_object16495:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22049ef5:::benchmark_data_smithi159_102124_object14780:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2205c8bb:::benchmark_data_smithi159_102124_object34963:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22060836:::benchmark_data_smithi159_102124_object28094:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2206633f:::benchmark_data_smithi159_102124_object34622:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22078d7c:::benchmark_data_smithi159_102124_object16113:head# 2024-04-15T18:22:17.197 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2207de25:::benchmark_data_smithi159_102124_object24583:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22083332:::benchmark_data_smithi159_102124_object1621:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:220aa8dc:::benchmark_data_smithi159_102124_object36207:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:220ae28c:::benchmark_data_smithi159_102124_object26731:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:220ceeee:::benchmark_data_smithi159_102124_object30446:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:220de4b0:::benchmark_data_smithi159_102124_object41168:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22101277:::benchmark_data_smithi159_102124_object1119:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22102823:::benchmark_data_smithi159_102124_object53455:head# 2024-04-15T18:22:17.199 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2210692f:::benchmark_data_smithi159_102124_object35330:head# 2024-04-15T18:22:17.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:221079dc:::benchmark_data_smithi159_102124_object49465:head# 2024-04-15T18:22:17.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22110bbf:::benchmark_data_smithi159_102124_object10761:head# 2024-04-15T18:22:17.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2212a912:::benchmark_data_smithi159_102124_object29704:head# 2024-04-15T18:22:17.200 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22134df1:::benchmark_data_smithi159_102124_object31014:head# 2024-04-15T18:22:17.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22161c69:::benchmark_data_smithi159_102124_object44946:head# 2024-04-15T18:22:17.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2216ca5e:::benchmark_data_smithi159_102124_object16671:head# 2024-04-15T18:22:17.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22176d37:::benchmark_data_smithi159_102124_object14122:head# 2024-04-15T18:22:17.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2218da03:::benchmark_data_smithi159_102124_object34467:head# 2024-04-15T18:22:17.203 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:221caa05:::benchmark_data_smithi159_102124_object12013:head# 2024-04-15T18:22:17.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:221e5e53:::benchmark_data_smithi159_102124_object5745:head# 2024-04-15T18:22:17.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22205566:::benchmark_data_smithi159_102124_object41395:head# 2024-04-15T18:22:17.204 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22215996:::benchmark_data_smithi159_102124_object45794:head# 2024-04-15T18:22:17.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222263d6:::benchmark_data_smithi159_102124_object31897:head# 2024-04-15T18:22:17.205 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2222a8fc:::benchmark_data_smithi159_102124_object24395:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222341ba:::benchmark_data_smithi159_102124_object29244:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22239e70:::benchmark_data_smithi159_102124_object7788:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2223d4ae:::benchmark_data_smithi159_102124_object2483:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2224796a:::benchmark_data_smithi159_102124_object15494:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2225014b:::benchmark_data_smithi159_102124_object11533:head# 2024-04-15T18:22:17.206 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22253667:::benchmark_data_smithi159_102124_object58410:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2225e192:::benchmark_data_smithi159_102124_object18122:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22277e59:::benchmark_data_smithi159_102124_object11534:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22281e59:::benchmark_data_smithi159_102124_object37233:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222a96e5:::benchmark_data_smithi159_102124_object9245:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222af27e:::benchmark_data_smithi159_102124_object34543:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222c3121:::benchmark_data_smithi159_102124_object10292:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222ce673:::benchmark_data_smithi159_102124_object51608:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222f61ac:::benchmark_data_smithi159_102124_object46952:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222fa31c:::benchmark_data_smithi159_102124_object24023:head# 2024-04-15T18:22:17.209 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:222ff400:::benchmark_data_smithi159_102124_object1416:head# 2024-04-15T18:22:17.210 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22304711:::benchmark_data_smithi159_102124_object43318:head# 2024-04-15T18:22:17.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22309825:::benchmark_data_smithi159_102124_object5600:head# 2024-04-15T18:22:17.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223173c0:::benchmark_data_smithi159_102124_object28069:head# 2024-04-15T18:22:17.211 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22324699:::benchmark_data_smithi159_102124_object24779:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2233ffd7:::benchmark_data_smithi159_102124_object7435:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22345c6b:::benchmark_data_smithi159_102124_object5218:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22350011:::benchmark_data_smithi159_102124_object31365:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2237cb5a:::benchmark_data_smithi159_102124_object55931:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2237dbe9:::benchmark_data_smithi159_102124_object32293:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22381958:::benchmark_data_smithi159_102124_object25391:head# 2024-04-15T18:22:17.212 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22397911:::benchmark_data_smithi159_102124_object12434:head# 2024-04-15T18:22:17.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223c04fa:::benchmark_data_smithi159_102124_object51667:head# 2024-04-15T18:22:17.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223c1fa5:::benchmark_data_smithi159_102124_object46479:head# 2024-04-15T18:22:17.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223cb702:::benchmark_data_smithi159_102124_object21465:head# 2024-04-15T18:22:17.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223cd645:::benchmark_data_smithi159_102124_object21331:head# 2024-04-15T18:22:17.213 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223d0831:::benchmark_data_smithi159_102124_object45341:head# 2024-04-15T18:22:17.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223dded3:::benchmark_data_smithi159_102124_object32526:head# 2024-04-15T18:22:17.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:223e728c:::benchmark_data_smithi159_102124_object2119:head# 2024-04-15T18:22:17.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22417712:::benchmark_data_smithi159_102124_object45852:head# 2024-04-15T18:22:17.214 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224269c5:::benchmark_data_smithi159_102124_object17177:head# 2024-04-15T18:22:17.215 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2244ae10:::benchmark_data_smithi159_102124_object47993:head# 2024-04-15T18:22:17.215 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2244be9d:::benchmark_data_smithi159_102124_object32350:head# 2024-04-15T18:22:17.215 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2244c41c:::benchmark_data_smithi159_102124_object49383:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2245bebc:::benchmark_data_smithi159_102124_object57610:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2245c90d:::benchmark_data_smithi159_102124_object55053:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224b23e3:::benchmark_data_smithi159_102124_object25146:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224c239e:::benchmark_data_smithi159_102124_object24687:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224c3a96:::benchmark_data_smithi159_102124_object17874:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224c5665:::benchmark_data_smithi159_102124_object16993:head# 2024-04-15T18:22:17.216 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224cdf50:::benchmark_data_smithi159_102124_object23290:head# 2024-04-15T18:22:17.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224def1c:::benchmark_data_smithi159_102124_object16652:head# 2024-04-15T18:22:17.217 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:224fa1ac:::benchmark_data_smithi159_102124_object23715:head# 2024-04-15T18:22:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2251c540:::benchmark_data_smithi159_102124_object32079:head# 2024-04-15T18:22:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225323d2:::benchmark_data_smithi159_102124_object37471:head# 2024-04-15T18:22:17.218 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22555ba2:::benchmark_data_smithi159_102124_object42757:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225571d5:::benchmark_data_smithi159_102124_object15507:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2255ea62:::benchmark_data_smithi159_102124_object30288:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2257027e:::benchmark_data_smithi159_102124_object2699:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22578455:::benchmark_data_smithi159_102124_object57231:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22579749:::benchmark_data_smithi159_102124_object3439:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22582b04:::benchmark_data_smithi159_102124_object27654:head# 2024-04-15T18:22:17.219 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22596a03:::benchmark_data_smithi159_102124_object18544:head# 2024-04-15T18:22:17.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225abe1a:::benchmark_data_smithi159_102124_object24630:head# 2024-04-15T18:22:17.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225b5a95:::benchmark_data_smithi159_102124_object48412:head# 2024-04-15T18:22:17.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225bb1f4:::benchmark_data_smithi159_102124_object26623:head# 2024-04-15T18:22:17.220 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:225ef7ca:::benchmark_data_smithi159_102124_object30296:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2260379b:::benchmark_data_smithi159_102124_object4694:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22610ba6:::benchmark_data_smithi159_102124_object45833:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2263d5d6:::benchmark_data_smithi159_102124_object13478:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2264d497:::benchmark_data_smithi159_102124_object52079:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22661d5b:::benchmark_data_smithi159_102124_object23097:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2266af19:::benchmark_data_smithi159_102124_object23630:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22676369:::benchmark_data_smithi159_102124_object55265:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2267b2db:::benchmark_data_smithi159_102124_object53107:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2269812d:::benchmark_data_smithi159_102124_object51291:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22699bfd:::benchmark_data_smithi159_102124_object9001:head# 2024-04-15T18:22:17.222 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:226c35ea:::benchmark_data_smithi159_102124_object24473:head# 2024-04-15T18:22:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:226db46e:::benchmark_data_smithi159_102124_object32127:head# 2024-04-15T18:22:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:226e83fe:::benchmark_data_smithi159_102124_object34677:head# 2024-04-15T18:22:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:226e9e16:::benchmark_data_smithi159_102124_object27131:head# 2024-04-15T18:22:17.223 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227060cc:::benchmark_data_smithi159_102124_object49391:head# 2024-04-15T18:22:17.224 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22709d25:::benchmark_data_smithi159_102124_object39395:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22712f50:::benchmark_data_smithi159_102124_object32015:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22728f92:::benchmark_data_smithi159_102124_object30954:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22729018:::benchmark_data_smithi159_102124_object2254:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2272cfb3:::benchmark_data_smithi159_102124_object46211:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227493b0:::benchmark_data_smithi159_102124_object45718:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22768453:::benchmark_data_smithi159_102124_object53928:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2279d5d2:::benchmark_data_smithi159_102124_object28366:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227a7588:::benchmark_data_smithi159_102124_object15252:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227ad279:::benchmark_data_smithi159_102124_object19644:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227bc86d:::benchmark_data_smithi159_102124_object33043:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227c057f:::benchmark_data_smithi159_102124_object24036:head# 2024-04-15T18:22:17.225 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227cf5d2:::benchmark_data_smithi159_102124_object44790:head# 2024-04-15T18:22:17.226 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227d6e08:::benchmark_data_smithi159_102124_object9943:head# 2024-04-15T18:22:17.227 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227f5ddd:::benchmark_data_smithi159_102124_object30079:head# 2024-04-15T18:22:17.227 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:227f8354:::benchmark_data_smithi159_102124_object37056:head# 2024-04-15T18:22:17.227 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2280b99b:::benchmark_data_smithi159_102124_object41887:head# 2024-04-15T18:22:17.227 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228196d2:::benchmark_data_smithi159_102124_object5579:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22835a60:::benchmark_data_smithi159_102124_object16415:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2283c79a:::benchmark_data_smithi159_102124_object20155:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22850a58:::benchmark_data_smithi159_102124_object84:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2286b553:::benchmark_data_smithi159_102124_object22599:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2286ea79:::benchmark_data_smithi159_102124_object31375:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22887d2c:::benchmark_data_smithi159_102124_object57712:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228a1764:::benchmark_data_smithi159_102124_object43627:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228a1e55:::benchmark_data_smithi159_102124_object25789:head# 2024-04-15T18:22:17.228 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228bee8f:::benchmark_data_smithi159_102124_object12091:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228c90fd:::benchmark_data_smithi159_102124_object1438:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228d03dc:::benchmark_data_smithi159_102124_object31374:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228d5fd2:::benchmark_data_smithi159_102124_object26574:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228d6e0e:::benchmark_data_smithi159_102124_object13176:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228da68a:::benchmark_data_smithi159_102124_object23451:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:228e20ed:::benchmark_data_smithi159_102124_object3537:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2290c412:::benchmark_data_smithi159_102124_object50697:head# 2024-04-15T18:22:17.230 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2290fc42:::benchmark_data_smithi159_102124_object53464:head# 2024-04-15T18:22:17.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2292b55d:::benchmark_data_smithi159_102124_object56731:head# 2024-04-15T18:22:17.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2293464f:::benchmark_data_smithi159_102124_object20003:head# 2024-04-15T18:22:17.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229545e1:::benchmark_data_smithi159_102124_object55293:head# 2024-04-15T18:22:17.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2295d0ba:::benchmark_data_smithi159_102124_object37218:head# 2024-04-15T18:22:17.232 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22965aad:::benchmark_data_smithi159_102124_object536:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2296982b:::benchmark_data_smithi159_102124_object41941:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229737ae:::benchmark_data_smithi159_102124_object34049:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22978fb3:::benchmark_data_smithi159_102124_object105:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2297ee9e:::benchmark_data_smithi159_102124_object16859:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229961c9:::benchmark_data_smithi159_102124_object29509:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229bf4fb:::benchmark_data_smithi159_102124_object24088:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229e0146:::benchmark_data_smithi159_102124_object33796:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229e8955:::benchmark_data_smithi159_102124_object54096:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:229fa161:::benchmark_data_smithi159_102124_object8364:head# 2024-04-15T18:22:17.233 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a09009:::benchmark_data_smithi159_102124_object19924:head# 2024-04-15T18:22:17.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a1c67c:::benchmark_data_smithi159_102124_object4714:head# 2024-04-15T18:22:17.234 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a32f6c:::benchmark_data_smithi159_102124_object38840:head# 2024-04-15T18:22:17.235 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a5f81c:::benchmark_data_smithi159_102124_object37490:head# 2024-04-15T18:22:17.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a68264:::benchmark_data_smithi159_102124_object45215:head# 2024-04-15T18:22:17.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a682b2:::benchmark_data_smithi159_102124_object37227:head# 2024-04-15T18:22:17.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a72d30:::benchmark_data_smithi159_102124_object4132:head# 2024-04-15T18:22:17.236 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a8791e:::benchmark_data_smithi159_102124_object46039:head# 2024-04-15T18:22:17.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22a8eab5:::benchmark_data_smithi159_102124_object32361:head# 2024-04-15T18:22:17.237 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ab0d26:::benchmark_data_smithi159_102124_object53949:head# 2024-04-15T18:22:17.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22acb146:::benchmark_data_smithi159_102124_object30467:head# 2024-04-15T18:22:17.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ad6836:::benchmark_data_smithi159_102124_object47681:head# 2024-04-15T18:22:17.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22adde11:::benchmark_data_smithi159_102124_object40574:head# 2024-04-15T18:22:17.238 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22aee9e7:::benchmark_data_smithi159_102124_object26154:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b18725:::benchmark_data_smithi159_102124_object33521:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b1edb0:::benchmark_data_smithi159_102124_object32757:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b34d31:::benchmark_data_smithi159_102124_object24424:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b356c9:::benchmark_data_smithi159_102124_object15843:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b35f2b:::benchmark_data_smithi159_102124_object10687:head# 2024-04-15T18:22:17.241 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b38a98:::benchmark_data_smithi159_102124_object35983:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b61b55:::benchmark_data_smithi159_102124_object41254:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b66f43:::benchmark_data_smithi159_102124_object56475:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b8948f:::benchmark_data_smithi159_102124_object29102:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22b8a796:::benchmark_data_smithi159_102124_object4224:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ba0568:::benchmark_data_smithi159_102124_object17719:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22bb31f7:::benchmark_data_smithi159_102124_object23362:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22bbf757:::benchmark_data_smithi159_102124_object32612:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22bcbce2:::benchmark_data_smithi159_102124_object11722:head# 2024-04-15T18:22:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22be6a1a:::benchmark_data_smithi159_102124_object55150:head# 2024-04-15T18:22:17.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22bfafef:::benchmark_data_smithi159_102124_object51643:head# 2024-04-15T18:22:17.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c06b61:::benchmark_data_smithi159_102124_object20579:head# 2024-04-15T18:22:17.243 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c0e96d:::benchmark_data_smithi159_102124_object22885:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c53e85:::benchmark_data_smithi159_102124_object35854:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c5a37d:::benchmark_data_smithi159_102124_object19972:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c6315a:::benchmark_data_smithi159_102124_object1882:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c6f04c:::benchmark_data_smithi159_102124_object12815:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c89c4d:::benchmark_data_smithi159_102124_object33387:head# 2024-04-15T18:22:17.246 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22c8e92f:::benchmark_data_smithi159_102124_object689:head# 2024-04-15T18:22:17.247 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22caa839:::benchmark_data_smithi159_102124_object35052:head# 2024-04-15T18:22:17.247 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22cac605:::benchmark_data_smithi159_102124_object720:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22caf849:::benchmark_data_smithi159_102124_object14487:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22cbb502:::benchmark_data_smithi159_102124_object22408:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ccd54c:::benchmark_data_smithi159_102124_object1737:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22cf6424:::benchmark_data_smithi159_102124_object50033:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22cff58e:::benchmark_data_smithi159_102124_object30973:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d3383f:::benchmark_data_smithi159_102124_object7136:head# 2024-04-15T18:22:17.250 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d4d4c8:::benchmark_data_smithi159_102124_object20682:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d5631c:::benchmark_data_smithi159_102124_object43005:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d59193:::benchmark_data_smithi159_102124_object35013:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d5964d:::benchmark_data_smithi159_102124_object8296:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d621fb:::benchmark_data_smithi159_102124_object53029:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d701b5:::benchmark_data_smithi159_102124_object51331:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d87210:::benchmark_data_smithi159_102124_object57875:head# 2024-04-15T18:22:17.251 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d8acd8:::benchmark_data_smithi159_102124_object17877:head# 2024-04-15T18:22:17.254 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22d9c6af:::benchmark_data_smithi159_102124_object8718:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22dabf3c:::benchmark_data_smithi159_102124_object20165:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22db7d19:::benchmark_data_smithi159_102124_object34782:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22dc17f4:::benchmark_data_smithi159_102124_object16201:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22dc6b51:::benchmark_data_smithi159_102124_object48701:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22dd38fb:::benchmark_data_smithi159_102124_object50620:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ddc4f6:::benchmark_data_smithi159_102124_object15170:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22decd46:::benchmark_data_smithi159_102124_object22691:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e0d06a:::benchmark_data_smithi159_102124_object19769:head# 2024-04-15T18:22:17.255 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e0fd5e:::benchmark_data_smithi159_102124_object7021:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e14720:::benchmark_data_smithi159_102124_object22170:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e3ba63:::benchmark_data_smithi159_102124_object24655:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e5a84e:::benchmark_data_smithi159_102124_object55632:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e5ae8b:::benchmark_data_smithi159_102124_object29471:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e7c0f4:::benchmark_data_smithi159_102124_object27052:head# 2024-04-15T18:22:17.256 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22e7e2ff:::benchmark_data_smithi159_102124_object6707:head# 2024-04-15T18:22:17.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ea418b:::benchmark_data_smithi159_102124_object17519:head# 2024-04-15T18:22:17.258 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ea7b7e:::benchmark_data_smithi159_102124_object1891:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22eae519:::benchmark_data_smithi159_102124_object17034:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22eb03e9:::benchmark_data_smithi159_102124_object53285:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ed7a41:::benchmark_data_smithi159_102124_object37330:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22edce7b:::benchmark_data_smithi159_102124_object35072:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f133de:::benchmark_data_smithi159_102124_object2980:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f468c0:::benchmark_data_smithi159_102124_object5691:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f478f6:::benchmark_data_smithi159_102124_object1205:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f47948:::benchmark_data_smithi159_102124_object43799:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f554b1:::benchmark_data_smithi159_102124_object34740:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f562e2:::benchmark_data_smithi159_102124_object2366:head# 2024-04-15T18:22:17.259 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f62dc8:::benchmark_data_smithi159_102124_object26813:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f71e04:::benchmark_data_smithi159_102124_object44435:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f7d010:::benchmark_data_smithi159_102124_object47321:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f7de4e:::benchmark_data_smithi159_102124_object54952:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f80ec8:::benchmark_data_smithi159_102124_object15725:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22f96599:::benchmark_data_smithi159_102124_object15257:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22fc79d7:::benchmark_data_smithi159_102124_object16656:head# 2024-04-15T18:22:17.260 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22fc88f1:::benchmark_data_smithi159_102124_object34397:head# 2024-04-15T18:22:17.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22fd9c47:::benchmark_data_smithi159_102124_object55313:head# 2024-04-15T18:22:17.265 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22fe256f:::benchmark_data_smithi159_102124_object4640:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:22ffe434:::benchmark_data_smithi159_102124_object26854:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23003721:::benchmark_data_smithi159_102124_object14831:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23010443:::benchmark_data_smithi159_102124_object6267:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2301280a:::benchmark_data_smithi159_102124_object18064:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230394dc:::benchmark_data_smithi159_102124_object42741:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23054ddb:::benchmark_data_smithi159_102124_object15493:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23068378:::benchmark_data_smithi159_102124_object44019:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2306ae60:::benchmark_data_smithi159_102124_object16251:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23081414:::benchmark_data_smithi159_102124_object31630:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2308141d:::benchmark_data_smithi159_102124_object32924:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23082383:::benchmark_data_smithi159_102124_object15783:head# 2024-04-15T18:22:17.266 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230afdfe:::benchmark_data_smithi159_102124_object10172:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230b224b:::benchmark_data_smithi159_102124_object22474:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230b88ab:::benchmark_data_smithi159_102124_object12682:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230c9a4d:::benchmark_data_smithi159_102124_object20250:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230d48cb:::benchmark_data_smithi159_102124_object8022:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230ef031:::benchmark_data_smithi159_102124_object8587:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:230faa12:::benchmark_data_smithi159_102124_object3987:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2310c803:::benchmark_data_smithi159_102124_object28759:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:231210cc:::benchmark_data_smithi159_102124_object22107:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23126c04:::benchmark_data_smithi159_102124_object10833:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2312fc4c:::benchmark_data_smithi159_102124_object45588:head# 2024-04-15T18:22:17.267 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23136eea:::benchmark_data_smithi159_102124_object8595:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2313a5b9:::benchmark_data_smithi159_102124_object55965:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2314b3eb:::benchmark_data_smithi159_102124_object4332:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2314ba57:::benchmark_data_smithi159_102124_object9247:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2314bb3c:::benchmark_data_smithi159_102124_object3553:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23155e38:::benchmark_data_smithi159_102124_object2497:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23171b7f:::benchmark_data_smithi159_102124_object17154:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:231982c1:::benchmark_data_smithi159_102124_object16924:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:231a82b1:::benchmark_data_smithi159_102124_object58279:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:231dac79:::benchmark_data_smithi159_102124_object8784:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:231e4cff:::benchmark_data_smithi159_102124_object40116:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:232035eb:::benchmark_data_smithi159_102124_object17607:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2326691a:::benchmark_data_smithi159_102124_object45050:head# 2024-04-15T18:22:17.268 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23267662:::benchmark_data_smithi159_102124_object44420:head# 2024-04-15T18:22:17.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2326bd21:::benchmark_data_smithi159_102124_object24435:head# 2024-04-15T18:22:17.269 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:232d56dd:::benchmark_data_smithi159_102124_object52562:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:232eb208:::benchmark_data_smithi159_102124_object11597:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:232fc7da:::benchmark_data_smithi159_102124_object28090:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23300135:::benchmark_data_smithi159_102124_object25919:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2331dd22:::benchmark_data_smithi159_102124_object24928:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2332535a:::benchmark_data_smithi159_102124_object36185:head# 2024-04-15T18:22:17.272 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23327e76:::benchmark_data_smithi159_102124_object3447:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:233291d9:::benchmark_data_smithi159_102124_object619:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2333a353:::benchmark_data_smithi159_102124_object20814:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2333bb5b:::benchmark_data_smithi159_102124_object53370:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23354655:::benchmark_data_smithi159_102124_object48364:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23359168:::benchmark_data_smithi159_102124_object43305:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2336dcc2:::benchmark_data_smithi159_102124_object26266:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23382c77:::benchmark_data_smithi159_102124_object44025:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2338837a:::benchmark_data_smithi159_102124_object30678:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23389e0e:::benchmark_data_smithi159_102124_object46802:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:233a07d1:::benchmark_data_smithi159_102124_object2799:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:233c2e6f:::benchmark_data_smithi159_102124_object53747:head# 2024-04-15T18:22:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:233c4a7b:::benchmark_data_smithi159_102124_object37478:head# 2024-04-15T18:22:17.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:233f8ffa:::benchmark_data_smithi159_102124_object36427:head# 2024-04-15T18:22:17.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2340c90b:::benchmark_data_smithi159_102124_object20380:head# 2024-04-15T18:22:17.274 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2341de5d:::benchmark_data_smithi159_102124_object38704:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23429c39:::benchmark_data_smithi159_102124_object51770:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2344cca7:::benchmark_data_smithi159_102124_object39271:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2346396d:::benchmark_data_smithi159_102124_object42849:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23469344:::benchmark_data_smithi159_102124_object8508:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23470439:::benchmark_data_smithi159_102124_object31167:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2347af81:::benchmark_data_smithi159_102124_object21847:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2347cf5c:::benchmark_data_smithi159_102124_object24796:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23482e80:::benchmark_data_smithi159_102124_object2461:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2348794b:::benchmark_data_smithi159_102124_object32253:head# 2024-04-15T18:22:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23498a0e:::benchmark_data_smithi159_102124_object10956:head# 2024-04-15T18:22:17.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:234a3e59:::benchmark_data_smithi159_102124_object10659:head# 2024-04-15T18:22:17.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:234a4d7d:::benchmark_data_smithi159_102124_object38422:head# 2024-04-15T18:22:17.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:234a8a35:::benchmark_data_smithi159_102124_object5184:head# 2024-04-15T18:22:17.277 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:234e9797:::benchmark_data_smithi159_102124_object44653:head# 2024-04-15T18:22:17.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:234f4a48:::benchmark_data_smithi159_102124_object45404:head# 2024-04-15T18:22:17.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23503925:::benchmark_data_smithi159_102124_object49967:head# 2024-04-15T18:22:17.278 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2350d1c3:::benchmark_data_smithi159_102124_object12529:head# 2024-04-15T18:22:17.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23522fe9:::benchmark_data_smithi159_102124_object32568:head# 2024-04-15T18:22:17.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2353d0fe:::benchmark_data_smithi159_102124_object6402:head# 2024-04-15T18:22:17.279 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2355f357:::benchmark_data_smithi159_102124_object6531:head# 2024-04-15T18:22:17.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:235604d2:::benchmark_data_smithi159_102124_object4526:head# 2024-04-15T18:22:17.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:235760ba:::benchmark_data_smithi159_102124_object18440:head# 2024-04-15T18:22:17.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23589158:::benchmark_data_smithi159_102124_object32779:head# 2024-04-15T18:22:17.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:235957b8:::benchmark_data_smithi159_102124_object53351:head# 2024-04-15T18:22:17.281 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:235eb46c:::benchmark_data_smithi159_102124_object28189:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:235ec080:::benchmark_data_smithi159_102124_object56016:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2361e1e4:::benchmark_data_smithi159_102124_object18984:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23635d00:::benchmark_data_smithi159_102124_object3433:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2363c626:::benchmark_data_smithi159_102124_object34329:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2366a09b:::benchmark_data_smithi159_102124_object6443:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236825b6:::benchmark_data_smithi159_102124_object56911:head# 2024-04-15T18:22:17.282 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23687d62:::benchmark_data_smithi159_102124_object21014:head# 2024-04-15T18:22:17.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236a28b3:::benchmark_data_smithi159_102124_object13443:head# 2024-04-15T18:22:17.284 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236a29c0:::benchmark_data_smithi159_102124_object46446:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236a380a:::benchmark_data_smithi159_102124_object55241:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236a6719:::benchmark_data_smithi159_102124_object22442:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236b4bcb:::benchmark_data_smithi159_102124_object34932:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236ee358:::benchmark_data_smithi159_102124_object51642:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236f89cd:::benchmark_data_smithi159_102124_object4638:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:236fbecd:::benchmark_data_smithi159_102124_object29193:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23711a47:::benchmark_data_smithi159_102124_object34512:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237238a6:::benchmark_data_smithi159_102124_object56241:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237506c3:::benchmark_data_smithi159_102124_object6643:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23753ead:::benchmark_data_smithi159_102124_object41513:head# 2024-04-15T18:22:17.285 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237545fa:::benchmark_data_smithi159_102124_object16545:head# 2024-04-15T18:22:17.286 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2375d4df:::benchmark_data_smithi159_102124_object25832:head# 2024-04-15T18:22:17.286 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23763849:::benchmark_data_smithi159_102124_object42576:head# 2024-04-15T18:22:17.286 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237ae13f:::benchmark_data_smithi159_102124_object55602:head# 2024-04-15T18:22:17.286 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237ae55f:::benchmark_data_smithi159_102124_object44690:head# 2024-04-15T18:22:17.287 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8026/266535 objects degraded (3.011%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237b7e0b:::benchmark_data_smithi159_102124_object7272:head# 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:237ca780:::benchmark_data_smithi159_102124_object7038:head# 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2380d63b:::benchmark_data_smithi159_102124_object50135:head# 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23814050:::benchmark_data_smithi159_102124_object36951:head# 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2381698b:::benchmark_data_smithi159_102124_object3149:head# 2024-04-15T18:22:17.290 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2382c2c6:::benchmark_data_smithi159_102124_object9040:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2382c63f:::benchmark_data_smithi159_102124_object10432:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23846830:::benchmark_data_smithi159_102124_object41028:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23876b92:::benchmark_data_smithi159_102124_object31595:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2387c2a4:::benchmark_data_smithi159_102124_object2932:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2388e6da:::benchmark_data_smithi159_102124_object45002:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238ab000:::benchmark_data_smithi159_102124_object5803:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238ab736:::benchmark_data_smithi159_102124_object20584:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238b68c5:::benchmark_data_smithi159_102124_object37333:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238df0da:::benchmark_data_smithi159_102124_object32901:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238eb1ca:::benchmark_data_smithi159_102124_object19600:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238ed87d:::benchmark_data_smithi159_102124_object16037:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:238fca02:::benchmark_data_smithi159_102124_object33595:head# 2024-04-15T18:22:17.291 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23901480:::benchmark_data_smithi159_102124_object42892:head# 2024-04-15T18:22:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2390dd68:::benchmark_data_smithi159_102124_object55696:head# 2024-04-15T18:22:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2391ad2f:::benchmark_data_smithi159_102124_object614:head# 2024-04-15T18:22:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23926efc:::benchmark_data_smithi159_102124_object34691:head# 2024-04-15T18:22:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23936205:::benchmark_data_smithi159_102124_object54401:head# 2024-04-15T18:22:17.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23936c64:::benchmark_data_smithi159_102124_object34388:head# 2024-04-15T18:22:17.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239480f3:::benchmark_data_smithi159_102124_object13687:head# 2024-04-15T18:22:17.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23960e50:::benchmark_data_smithi159_102124_object35529:head# 2024-04-15T18:22:17.293 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2398ee2e:::benchmark_data_smithi159_102124_object16168:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239960e1:::benchmark_data_smithi159_102124_object23339:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239b606f:::benchmark_data_smithi159_102124_object48370:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239b6cf8:::benchmark_data_smithi159_102124_object40519:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239bbace:::benchmark_data_smithi159_102124_object40034:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239bd7a9:::benchmark_data_smithi159_102124_object53035:head# 2024-04-15T18:22:17.294 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239cabe1:::benchmark_data_smithi159_102124_object50634:head# 2024-04-15T18:22:17.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239e0285:::benchmark_data_smithi159_102124_object19122:head# 2024-04-15T18:22:17.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239e304c:::benchmark_data_smithi159_102124_object13683:head# 2024-04-15T18:22:17.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239e5bff:::benchmark_data_smithi159_102124_object34241:head# 2024-04-15T18:22:17.295 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239f8387:::benchmark_data_smithi159_102124_object55099:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:239f99ff:::benchmark_data_smithi159_102124_object13268:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a018c7:::benchmark_data_smithi159_102124_object46536:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a09b43:::benchmark_data_smithi159_102124_object54659:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a19f46:::benchmark_data_smithi159_102124_object19152:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a237be:::benchmark_data_smithi159_102124_object45376:head# 2024-04-15T18:22:17.300 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a2e0c4:::benchmark_data_smithi159_102124_object34117:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a3c860:::benchmark_data_smithi159_102124_object20830:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a54080:::benchmark_data_smithi159_102124_object29698:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a5949c:::benchmark_data_smithi159_102124_object29521:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a5e1b6:::benchmark_data_smithi159_102124_object38302:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a775c8:::benchmark_data_smithi159_102124_object34423:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a781cc:::benchmark_data_smithi159_102124_object24534:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23a92416:::benchmark_data_smithi159_102124_object58431:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23aa19f8:::benchmark_data_smithi159_102124_object18372:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23aceba5:::benchmark_data_smithi159_102124_object51418:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23adbbff:::benchmark_data_smithi159_102124_object14362:head# 2024-04-15T18:22:17.301 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ae33a8:::benchmark_data_smithi159_102124_object18607:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ae402d:::benchmark_data_smithi159_102124_object5136:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ae68ab:::benchmark_data_smithi159_102124_object48532:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23aefca2:::benchmark_data_smithi159_102124_object17453:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23af260a:::benchmark_data_smithi159_102124_object6304:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23afc2b7:::benchmark_data_smithi159_102124_object45359:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23b2d684:::benchmark_data_smithi159_102124_object10063:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23b30297:::benchmark_data_smithi159_102124_object37757:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23b4741c:::benchmark_data_smithi159_102124_object8077:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23b51bf6:::benchmark_data_smithi159_102124_object6889:head# 2024-04-15T18:22:17.302 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23b6d454:::benchmark_data_smithi159_102124_object32761:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ba5816:::benchmark_data_smithi159_102124_object34164:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23bd0b5b:::benchmark_data_smithi159_102124_object46810:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23bd3806:::benchmark_data_smithi159_102124_object31682:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23bdc1ac:::benchmark_data_smithi159_102124_object57211:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23be5a0d:::benchmark_data_smithi159_102124_object43952:head# 2024-04-15T18:22:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23bee57b:::benchmark_data_smithi159_102124_object2295:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c04ecc:::benchmark_data_smithi159_102124_object40192:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c296c2:::benchmark_data_smithi159_102124_object37933:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c2d73d:::benchmark_data_smithi159_102124_object53323:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c3bb58:::benchmark_data_smithi159_102124_object34658:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c6e50d:::benchmark_data_smithi159_102124_object13348:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c71243:::benchmark_data_smithi159_102124_object49265:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c79350:::benchmark_data_smithi159_102124_object16640:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23c99aaa:::benchmark_data_smithi159_102124_object23943:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ca5823:::benchmark_data_smithi159_102124_object36186:head# 2024-04-15T18:22:17.311 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d0c5fe:::benchmark_data_smithi159_102124_object4713:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d19dca:::benchmark_data_smithi159_102124_object53975:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d2a51c:::benchmark_data_smithi159_102124_object19733:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d2a7ec:::benchmark_data_smithi159_102124_object44724:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d3515b:::benchmark_data_smithi159_102124_object41128:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d39ab5:::benchmark_data_smithi159_102124_object30560:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d3a905:::benchmark_data_smithi159_102124_object31617:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d742c2:::benchmark_data_smithi159_102124_object27234:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d83481:::benchmark_data_smithi159_102124_object17651:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d83a3c:::benchmark_data_smithi159_102124_object1677:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23d90a2b:::benchmark_data_smithi159_102124_object15986:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23dae957:::benchmark_data_smithi159_102124_object45155:head# 2024-04-15T18:22:17.312 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23db3857:::benchmark_data_smithi159_102124_object54076:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23dde881:::benchmark_data_smithi159_102124_object23958:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23de9d04:::benchmark_data_smithi159_102124_object32972:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23e1f3c8:::benchmark_data_smithi159_102124_object14790:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23e373aa:::benchmark_data_smithi159_102124_object37168:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23e73318:::benchmark_data_smithi159_102124_object55071:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23e843d1:::benchmark_data_smithi159_102124_object33189:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23eab097:::benchmark_data_smithi159_102124_object12903:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23eab6d0:::benchmark_data_smithi159_102124_object21705:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23eb5e2c:::benchmark_data_smithi159_102124_object52266:head# 2024-04-15T18:22:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ecde47:::benchmark_data_smithi159_102124_object14669:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ed6d4a:::benchmark_data_smithi159_102124_object29925:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23ee845b:::benchmark_data_smithi159_102124_object28959:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23eea0b8:::benchmark_data_smithi159_102124_object47377:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f0ea57:::benchmark_data_smithi159_102124_object38985:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f105b4:::benchmark_data_smithi159_102124_object23972:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f1dad2:::benchmark_data_smithi159_102124_object57848:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f78234:::benchmark_data_smithi159_102124_object11097:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f8160a:::benchmark_data_smithi159_102124_object1049:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f8e527:::benchmark_data_smithi159_102124_object5601:head# 2024-04-15T18:22:17.314 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f8fe43:::benchmark_data_smithi159_102124_object45824:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f9483d:::benchmark_data_smithi159_102124_object58088:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f97a1a:::benchmark_data_smithi159_102124_object22420:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23f9b414:::benchmark_data_smithi159_102124_object49905:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23fa3c49:::benchmark_data_smithi159_102124_object17710:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23fc3678:::benchmark_data_smithi159_102124_object20058:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23fd3371:::benchmark_data_smithi159_102124_object48104:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:23feaaa3:::benchmark_data_smithi159_102124_object48526:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2400aa4d:::benchmark_data_smithi159_102124_object32433:head# 2024-04-15T18:22:17.315 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2400b57a:::benchmark_data_smithi159_102124_object18214:head# 2024-04-15T18:22:17.317 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240111f9:::benchmark_data_smithi159_102124_object34233:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2402bf12:::benchmark_data_smithi159_102124_object1563:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24031a84:::benchmark_data_smithi159_102124_object14392:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2404112b:::benchmark_data_smithi159_102124_object30328:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2405231a:::benchmark_data_smithi159_102124_object45815:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24055895:::benchmark_data_smithi159_102124_object44368:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2405d718:::benchmark_data_smithi159_102124_object9067:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240826c8:::benchmark_data_smithi159_102124_object22618:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2409cf22:::benchmark_data_smithi159_102124_object14563:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240afd1e:::benchmark_data_smithi159_102124_object58142:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240b1522:::benchmark_data_smithi159_102124_object17577:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240bcdc4:::benchmark_data_smithi159_102124_object49436:head# 2024-04-15T18:22:17.318 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240c40be:::benchmark_data_smithi159_102124_object36118:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240c5125:::benchmark_data_smithi159_102124_object38079:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240d3124:::benchmark_data_smithi159_102124_object17347:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:240db244:::benchmark_data_smithi159_102124_object7755:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241093a1:::benchmark_data_smithi159_102124_object14906:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2410a43a:::benchmark_data_smithi159_102124_object14875:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2410a986:::benchmark_data_smithi159_102124_object49402:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241499d1:::benchmark_data_smithi159_102124_object20774:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2414dc36:::benchmark_data_smithi159_102124_object8119:head# 2024-04-15T18:22:17.319 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2416f41c:::benchmark_data_smithi159_102124_object36602:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2417bd4d:::benchmark_data_smithi159_102124_object12034:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241a235b:::benchmark_data_smithi159_102124_object40108:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241c0979:::benchmark_data_smithi159_102124_object47821:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241d6663:::benchmark_data_smithi159_102124_object1407:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241e5d14:::benchmark_data_smithi159_102124_object50416:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241ecf79:::benchmark_data_smithi159_102124_object11824:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241f4ece:::benchmark_data_smithi159_102124_object1940:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:241f7674:::benchmark_data_smithi159_102124_object48365:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2420065f:::benchmark_data_smithi159_102124_object19968:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24206164:::benchmark_data_smithi159_102124_object47546:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2420c839:::benchmark_data_smithi159_102124_object55119:head# 2024-04-15T18:22:17.322 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2420f283:::benchmark_data_smithi159_102124_object29962:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24221503:::benchmark_data_smithi159_102124_object48784:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2422343d:::benchmark_data_smithi159_102124_object24843:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24247687:::benchmark_data_smithi159_102124_object33426:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242af99a:::benchmark_data_smithi159_102124_object27425:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242bc438:::benchmark_data_smithi159_102124_object25127:head# 2024-04-15T18:22:17.323 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242cddcf:::benchmark_data_smithi159_102124_object43708:head# 2024-04-15T18:22:17.324 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242d4f6e:::benchmark_data_smithi159_102124_object10400:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242da722:::benchmark_data_smithi159_102124_object36621:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242e4c1e:::benchmark_data_smithi159_102124_object46804:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:242e6aa7:::benchmark_data_smithi159_102124_object55187:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2433961e:::benchmark_data_smithi159_102124_object26350:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2433babd:::benchmark_data_smithi159_102124_object44662:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2433bbf7:::benchmark_data_smithi159_102124_object45535:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243423ff:::benchmark_data_smithi159_102124_object38541:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24385f18:::benchmark_data_smithi159_102124_object30872:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2438cc75:::benchmark_data_smithi159_102124_object19624:head# 2024-04-15T18:22:17.325 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2438efe4:::benchmark_data_smithi159_102124_object41051:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24391320:::benchmark_data_smithi159_102124_object45612:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243976ff:::benchmark_data_smithi159_102124_object10510:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243b1916:::benchmark_data_smithi159_102124_object28945:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243c6750:::benchmark_data_smithi159_102124_object15328:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243cd8bf:::benchmark_data_smithi159_102124_object33285:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243d8876:::benchmark_data_smithi159_102124_object18596:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:243f8448:::benchmark_data_smithi159_102124_object331:head# 2024-04-15T18:22:17.327 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24408322:::benchmark_data_smithi159_102124_object26964:head# 2024-04-15T18:22:17.328 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2440e49a:::benchmark_data_smithi159_102124_object57400:head# 2024-04-15T18:22:17.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2441ed50:::benchmark_data_smithi159_102124_object13030:head# 2024-04-15T18:22:17.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24452a42:::benchmark_data_smithi159_102124_object50330:head# 2024-04-15T18:22:17.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2446bd36:::benchmark_data_smithi159_102124_object39876:head# 2024-04-15T18:22:17.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244717e2:::benchmark_data_smithi159_102124_object46405:head# 2024-04-15T18:22:17.329 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244765d6:::benchmark_data_smithi159_102124_object22296:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2447df84:::benchmark_data_smithi159_102124_object55391:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2447edee:::benchmark_data_smithi159_102124_object24548:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2447f717:::benchmark_data_smithi159_102124_object35840:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2448a514:::benchmark_data_smithi159_102124_object13181:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2449596c:::benchmark_data_smithi159_102124_object51691:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244a6176:::benchmark_data_smithi159_102124_object47789:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244b14f3:::benchmark_data_smithi159_102124_object7598:head# 2024-04-15T18:22:17.330 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244badd8:::benchmark_data_smithi159_102124_object17517:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244c479e:::benchmark_data_smithi159_102124_object35705:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:244d08ba:::benchmark_data_smithi159_102124_object13745:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24503217:::benchmark_data_smithi159_102124_object208:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2454a9c1:::benchmark_data_smithi159_102124_object11664:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24560bab:::benchmark_data_smithi159_102124_object27519:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24563ae7:::benchmark_data_smithi159_102124_object42630:head# 2024-04-15T18:22:17.332 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2456de53:::benchmark_data_smithi159_102124_object43042:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24573dfe:::benchmark_data_smithi159_102124_object9566:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245791fd:::benchmark_data_smithi159_102124_object55593:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24586a7a:::benchmark_data_smithi159_102124_object28895:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245881c7:::benchmark_data_smithi159_102124_object47716:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245a7fbf:::benchmark_data_smithi159_102124_object28168:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245b7e50:::benchmark_data_smithi159_102124_object20177:head# 2024-04-15T18:22:17.333 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245bda00:::benchmark_data_smithi159_102124_object44020:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:245f942e:::benchmark_data_smithi159_102124_object34127:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24610067:::benchmark_data_smithi159_102124_object30635:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24693dda:::benchmark_data_smithi159_102124_object3832:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:246a747e:::benchmark_data_smithi159_102124_object53495:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:246b992a:::benchmark_data_smithi159_102124_object43681:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:246c1309:::benchmark_data_smithi159_102124_object46698:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:246ea550:::benchmark_data_smithi159_102124_object23240:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:246f8254:::benchmark_data_smithi159_102124_object32338:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24706ae5:::benchmark_data_smithi159_102124_object31283:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24716843:::benchmark_data_smithi159_102124_object56938:head# 2024-04-15T18:22:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2471b541:::benchmark_data_smithi159_102124_object20203:head# 2024-04-15T18:22:17.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24724f08:::benchmark_data_smithi159_102124_object48542:head# 2024-04-15T18:22:17.336 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2472501b:::benchmark_data_smithi159_102124_object10591:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247271fe:::benchmark_data_smithi159_102124_object38612:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24750515:::benchmark_data_smithi159_102124_object9674:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247515b3:::benchmark_data_smithi159_102124_object16513:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24770182:::benchmark_data_smithi159_102124_object38634:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24775c3c:::benchmark_data_smithi159_102124_object24983:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2477aef6:::benchmark_data_smithi159_102124_object47938:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24789045:::benchmark_data_smithi159_102124_object10564:head# 2024-04-15T18:22:17.337 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247910ee:::benchmark_data_smithi159_102124_object1665:head# 2024-04-15T18:22:17.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2479d3bd:::benchmark_data_smithi159_102124_object51552:head# 2024-04-15T18:22:17.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247c61a7:::benchmark_data_smithi159_102124_object1496:head# 2024-04-15T18:22:17.340 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247d351f:::benchmark_data_smithi159_102124_object20291:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247db6c6:::benchmark_data_smithi159_102124_object1236:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247db81d:::benchmark_data_smithi159_102124_object7871:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247f02a8:::benchmark_data_smithi159_102124_object6339:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:247f2a4c:::benchmark_data_smithi159_102124_object7787:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248033c5:::benchmark_data_smithi159_102124_object51560:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2481d5ae:::benchmark_data_smithi159_102124_object676:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2481da43:::benchmark_data_smithi159_102124_object3010:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2483c0b7:::benchmark_data_smithi159_102124_object41111:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24842daf:::benchmark_data_smithi159_102124_object44454:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2484a84c:::benchmark_data_smithi159_102124_object16408:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2484e898:::benchmark_data_smithi159_102124_object14331:head# 2024-04-15T18:22:17.341 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2486dc53:::benchmark_data_smithi159_102124_object29600:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24871165:::benchmark_data_smithi159_102124_object16676:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24874cd1:::benchmark_data_smithi159_102124_object24746:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2487aef5:::benchmark_data_smithi159_102124_object44909:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2488b633:::benchmark_data_smithi159_102124_object34267:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24898609:::benchmark_data_smithi159_102124_object43623:head# 2024-04-15T18:22:17.342 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24899708:::benchmark_data_smithi159_102124_object43996:head# 2024-04-15T18:22:17.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248c1613:::benchmark_data_smithi159_102124_object23096:head# 2024-04-15T18:22:17.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248c3a5f:::benchmark_data_smithi159_102124_object21460:head# 2024-04-15T18:22:17.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248cf66e:::benchmark_data_smithi159_102124_object12638:head# 2024-04-15T18:22:17.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248d732f:::benchmark_data_smithi159_102124_object49039:head# 2024-04-15T18:22:17.344 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248e3302:::benchmark_data_smithi159_102124_object13660:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248e8b89:::benchmark_data_smithi159_102124_object51324:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:248faffa:::benchmark_data_smithi159_102124_object39474:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2490520b:::benchmark_data_smithi159_102124_object10703:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2490fc31:::benchmark_data_smithi159_102124_object5089:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2494322d:::benchmark_data_smithi159_102124_object52018:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2494a8d8:::benchmark_data_smithi159_102124_object58400:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24960890:::benchmark_data_smithi159_102124_object29794:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249664ac:::benchmark_data_smithi159_102124_object36089:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2496b6a3:::benchmark_data_smithi159_102124_object18701:head# 2024-04-15T18:22:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2496b70d:::benchmark_data_smithi159_102124_object2892:head# 2024-04-15T18:22:17.346 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2496bb0f:::benchmark_data_smithi159_102124_object3617:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2496f6d2:::benchmark_data_smithi159_102124_object11386:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249b96c2:::benchmark_data_smithi159_102124_object42562:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249c31bf:::benchmark_data_smithi159_102124_object55164:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249cc05e:::benchmark_data_smithi159_102124_object52473:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249d0b82:::benchmark_data_smithi159_102124_object29588:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249e22e6:::benchmark_data_smithi159_102124_object43407:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249ef69b:::benchmark_data_smithi159_102124_object37696:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:249f75e1:::benchmark_data_smithi159_102124_object20162:head# 2024-04-15T18:22:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a05eef:::benchmark_data_smithi159_102124_object52872:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a0e50f:::benchmark_data_smithi159_102124_object4628:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a14f45:::benchmark_data_smithi159_102124_object13654:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a4d67a:::benchmark_data_smithi159_102124_object14097:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a58fe2:::benchmark_data_smithi159_102124_object42381:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a59b2d:::benchmark_data_smithi159_102124_object54684:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24a92c5e:::benchmark_data_smithi159_102124_object45405:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ab2312:::benchmark_data_smithi159_102124_object12343:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ac0de2:::benchmark_data_smithi159_102124_object49589:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ae3fed:::benchmark_data_smithi159_102124_object24437:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24afe7b2:::benchmark_data_smithi159_102124_object50156:head# 2024-04-15T18:22:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b12eef:::benchmark_data_smithi159_102124_object17079:head# 2024-04-15T18:22:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b1729f:::benchmark_data_smithi159_102124_object42334:head# 2024-04-15T18:22:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b46801:::benchmark_data_smithi159_102124_object41348:head# 2024-04-15T18:22:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b6841f:::benchmark_data_smithi159_102124_object42220:head# 2024-04-15T18:22:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b875ef:::benchmark_data_smithi159_102124_object8394:head# 2024-04-15T18:22:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24b93f55:::benchmark_data_smithi159_102124_object38382:head# 2024-04-15T18:22:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24bcc68e:::benchmark_data_smithi159_102124_object46866:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24bdc1d0:::benchmark_data_smithi159_102124_object29983:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24bef0b7:::benchmark_data_smithi159_102124_object57040:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24bf0b82:::benchmark_data_smithi159_102124_object43530:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c04908:::benchmark_data_smithi159_102124_object33699:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c08788:::benchmark_data_smithi159_102124_object54961:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c264cf:::benchmark_data_smithi159_102124_object48912:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c3cb2b:::benchmark_data_smithi159_102124_object14026:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c4ae62:::benchmark_data_smithi159_102124_object22999:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c54f10:::benchmark_data_smithi159_102124_object47564:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c604ff:::benchmark_data_smithi159_102124_object6372:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c66413:::benchmark_data_smithi159_102124_object17497:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c71958:::benchmark_data_smithi159_102124_object39508:head# 2024-04-15T18:22:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c79eea:::benchmark_data_smithi159_102124_object8455:head# 2024-04-15T18:22:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c86666:::benchmark_data_smithi159_102124_object14718:head# 2024-04-15T18:22:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c89e1c:::benchmark_data_smithi159_102124_object41977:head# 2024-04-15T18:22:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c8adc3:::benchmark_data_smithi159_102124_object42925:head# 2024-04-15T18:22:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c91788:::benchmark_data_smithi159_102124_object20896:head# 2024-04-15T18:22:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24c9da37:::benchmark_data_smithi159_102124_object39183:head# 2024-04-15T18:22:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24cbe103:::benchmark_data_smithi159_102124_object18517:head# 2024-04-15T18:22:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24cce2e3:::benchmark_data_smithi159_102124_object52885:head# 2024-04-15T18:22:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ce21a7:::benchmark_data_smithi159_102124_object46418:head# 2024-04-15T18:22:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ceb84c:::benchmark_data_smithi159_102124_object24383:head# 2024-04-15T18:22:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24cf8142:::benchmark_data_smithi159_102124_object23654:head# 2024-04-15T18:22:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24cff4dc:::benchmark_data_smithi159_102124_object26273:head# 2024-04-15T18:22:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d2079c:::benchmark_data_smithi159_102124_object29572:head# 2024-04-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d20aaa:::benchmark_data_smithi159_102124_object7979:head# 2024-04-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d2b694:::benchmark_data_smithi159_102124_object35317:head# 2024-04-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d46cec:::benchmark_data_smithi159_102124_object5704:head# 2024-04-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d4b646:::benchmark_data_smithi159_102124_object35842:head# 2024-04-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d5d130:::benchmark_data_smithi159_102124_object12775:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d70cfc:::benchmark_data_smithi159_102124_object4467:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d818df:::benchmark_data_smithi159_102124_object14364:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24d90399:::benchmark_data_smithi159_102124_object15657:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24dc3fa2:::benchmark_data_smithi159_102124_object15077:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24dc41d9:::benchmark_data_smithi159_102124_object2174:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24df7421:::benchmark_data_smithi159_102124_object54355:head# 2024-04-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e15138:::benchmark_data_smithi159_102124_object10351:head# 2024-04-15T18:22:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e1cf99:::benchmark_data_smithi159_102124_object16367:head# 2024-04-15T18:22:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e2624e:::benchmark_data_smithi159_102124_object3048:head# 2024-04-15T18:22:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e4801f:::benchmark_data_smithi159_102124_object2561:head# 2024-04-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e50b43:::benchmark_data_smithi159_102124_object5491:head# 2024-04-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e5eb5f:::benchmark_data_smithi159_102124_object18038:head# 2024-04-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e6502f:::benchmark_data_smithi159_102124_object49175:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e66a90:::benchmark_data_smithi159_102124_object26163:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e680d0:::benchmark_data_smithi159_102124_object44595:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e69c1c:::benchmark_data_smithi159_102124_object22287:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e6a209:::benchmark_data_smithi159_102124_object20305:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e77108:::benchmark_data_smithi159_102124_object46953:head# 2024-04-15T18:22:17.364 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e77e40:::benchmark_data_smithi159_102124_object40532:head# 2024-04-15T18:22:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e79ef5:::benchmark_data_smithi159_102124_object56292:head# 2024-04-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e7a718:::benchmark_data_smithi159_102124_object6086:head# 2024-04-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24e962fa:::benchmark_data_smithi159_102124_object26558:head# 2024-04-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ea3dd8:::benchmark_data_smithi159_102124_object34672:head# 2024-04-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24eb132d:::benchmark_data_smithi159_102124_object8867:head# 2024-04-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24eb37d5:::benchmark_data_smithi159_102124_object41402:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ec3702:::benchmark_data_smithi159_102124_object23081:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ed70b5:::benchmark_data_smithi159_102124_object18488:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24eda6b0:::benchmark_data_smithi159_102124_object42004:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24eefa2c:::benchmark_data_smithi159_102124_object35781:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ef7bec:::benchmark_data_smithi159_102124_object44922:head# 2024-04-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f14b7c:::benchmark_data_smithi159_102124_object28840:head# 2024-04-15T18:22:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f283c4:::benchmark_data_smithi159_102124_object31539:head# 2024-04-15T18:22:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f2c30d:::benchmark_data_smithi159_102124_object24745:head# 2024-04-15T18:22:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f34111:::benchmark_data_smithi159_102124_object41769:head# 2024-04-15T18:22:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f4b46a:::benchmark_data_smithi159_102124_object49314:head# 2024-04-15T18:22:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f64790:::benchmark_data_smithi159_102124_object50942:head# 2024-04-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24f811e4:::benchmark_data_smithi159_102124_object27898:head# 2024-04-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24fa4906:::benchmark_data_smithi159_102124_object52174:head# 2024-04-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24faaf60:::benchmark_data_smithi159_102124_object37464:head# 2024-04-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24fc1738:::benchmark_data_smithi159_102124_object21641:head# 2024-04-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24fea26e:::benchmark_data_smithi159_102124_object20374:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ff9076:::benchmark_data_smithi159_102124_object976:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:24ffd7d5:::benchmark_data_smithi159_102124_object44721:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25020080:::benchmark_data_smithi159_102124_object26290:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25023da0:::benchmark_data_smithi159_102124_object33120:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2502accf:::benchmark_data_smithi159_102124_object14950:head# 2024-04-15T18:22:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2503f0b2:::benchmark_data_smithi159_102124_object6838:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25041349:::benchmark_data_smithi159_102124_object39472:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25060b7b:::benchmark_data_smithi159_102124_object24762:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2509d9a9:::benchmark_data_smithi159_102124_object57436:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250bfff2:::benchmark_data_smithi159_102124_object49263:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250c06ca:::benchmark_data_smithi159_102124_object56233:head# 2024-04-15T18:22:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250c65c6:::benchmark_data_smithi159_102124_object47588:head# 2024-04-15T18:22:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250d756e:::benchmark_data_smithi159_102124_object14401:head# 2024-04-15T18:22:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250f05c2:::benchmark_data_smithi159_102124_object56404:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:250fa7d5:::benchmark_data_smithi159_102124_object28499:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25134435:::benchmark_data_smithi159_102124_object18963:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25152f87:::benchmark_data_smithi159_102124_object13636:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251a620d:::benchmark_data_smithi159_102124_object3939:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251a7d19:::benchmark_data_smithi159_102124_object32090:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251a8021:::benchmark_data_smithi159_102124_object28641:head# 2024-04-15T18:22:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251a8259:::benchmark_data_smithi159_102124_object43394:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251ad9b3:::benchmark_data_smithi159_102124_object46829:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251b8a5b:::benchmark_data_smithi159_102124_object10808:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251bd5af:::benchmark_data_smithi159_102124_object2326:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251d767d:::benchmark_data_smithi159_102124_object18519:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251e338d:::benchmark_data_smithi159_102124_object14558:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:251f10f6:::benchmark_data_smithi159_102124_object32767:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25238af3:::benchmark_data_smithi159_102124_object22681:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25243d1e:::benchmark_data_smithi159_102124_object12283:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25258dd4:::benchmark_data_smithi159_102124_object49581:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25259a75:::benchmark_data_smithi159_102124_object51719:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25276d6d:::benchmark_data_smithi159_102124_object23407:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25279f66:::benchmark_data_smithi159_102124_object53090:head# 2024-04-15T18:22:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2527d8e9:::benchmark_data_smithi159_102124_object24274:head# 2024-04-15T18:22:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:252a2be4:::benchmark_data_smithi159_102124_object39079:head# 2024-04-15T18:22:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:252a7a26:::benchmark_data_smithi159_102124_object9881:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:252b179f:::benchmark_data_smithi159_102124_object17088:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:252ffc52:::benchmark_data_smithi159_102124_object50173:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25316e15:::benchmark_data_smithi159_102124_object40105:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2533ab46:::benchmark_data_smithi159_102124_object12488:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253404b8:::benchmark_data_smithi159_102124_object26516:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25342295:::benchmark_data_smithi159_102124_object10391:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2534bf5f:::benchmark_data_smithi159_102124_object1983:head# 2024-04-15T18:22:17.384 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2538f5d2:::benchmark_data_smithi159_102124_object18073:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25397205:::benchmark_data_smithi159_102124_object44854:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253a08f0:::benchmark_data_smithi159_102124_object20089:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253aeba5:::benchmark_data_smithi159_102124_object16963:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253b77fc:::benchmark_data_smithi159_102124_object28410:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253c7def:::benchmark_data_smithi159_102124_object26358:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253c9202:::benchmark_data_smithi159_102124_object27038:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253dc802:::benchmark_data_smithi159_102124_object51975:head# 2024-04-15T18:22:17.385 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253e7572:::benchmark_data_smithi159_102124_object52573:head# 2024-04-15T18:22:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:253e8918:::benchmark_data_smithi159_102124_object51092:head# 2024-04-15T18:22:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2541ad49:::benchmark_data_smithi159_102124_object16572:head# 2024-04-15T18:22:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25425b56:::benchmark_data_smithi159_102124_object43265:head# 2024-04-15T18:22:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25449c6e:::benchmark_data_smithi159_102124_object32547:head# 2024-04-15T18:22:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25456f01:::benchmark_data_smithi159_102124_object2931:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25457956:::benchmark_data_smithi159_102124_object15452:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2545b206:::benchmark_data_smithi159_102124_object27861:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2546556c:::benchmark_data_smithi159_102124_object33214:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2547340a:::benchmark_data_smithi159_102124_object2123:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25476a0d:::benchmark_data_smithi159_102124_object55959:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2547ae62:::benchmark_data_smithi159_102124_object54863:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25489681:::benchmark_data_smithi159_102124_object13157:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25490b97:::benchmark_data_smithi159_102124_object34844:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25498638:::benchmark_data_smithi159_102124_object55836:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254b9de9:::benchmark_data_smithi159_102124_object25988:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254be9ea:::benchmark_data_smithi159_102124_object5055:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254ceb1f:::benchmark_data_smithi159_102124_object40923:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254dbfcc:::benchmark_data_smithi159_102124_object11526:head# 2024-04-15T18:22:17.389 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254e62aa:::benchmark_data_smithi159_102124_object25136:head# 2024-04-15T18:22:17.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254f4565:::benchmark_data_smithi159_102124_object37776:head# 2024-04-15T18:22:17.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:254f58a2:::benchmark_data_smithi159_102124_object5854:head# 2024-04-15T18:22:17.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25502405:::benchmark_data_smithi159_102124_object10008:head# 2024-04-15T18:22:17.390 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25518dfb:::benchmark_data_smithi159_102124_object1003:head# 2024-04-15T18:22:17.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2554c3db:::benchmark_data_smithi159_102124_object7723:head# 2024-04-15T18:22:17.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25555ead:::benchmark_data_smithi159_102124_object38477:head# 2024-04-15T18:22:17.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25567a06:::benchmark_data_smithi159_102124_object16551:head# 2024-04-15T18:22:17.391 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2557d0ec:::benchmark_data_smithi159_102124_object8186:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2559fdfb:::benchmark_data_smithi159_102124_object41922:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:255cc214:::benchmark_data_smithi159_102124_object56355:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:255e3e37:::benchmark_data_smithi159_102124_object41511:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:255ffaa1:::benchmark_data_smithi159_102124_object30710:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2560aaca:::benchmark_data_smithi159_102124_object26738:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:256153dc:::benchmark_data_smithi159_102124_object22125:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25632061:::benchmark_data_smithi159_102124_object27950:head# 2024-04-15T18:22:17.392 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2563c8fe:::benchmark_data_smithi159_102124_object37642:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2563cc1c:::benchmark_data_smithi159_102124_object39987:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2567f34c:::benchmark_data_smithi159_102124_object32932:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25694f4e:::benchmark_data_smithi159_102124_object36074:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:256b34ae:::benchmark_data_smithi159_102124_object2563:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:256c8f45:::benchmark_data_smithi159_102124_object34197:head# 2024-04-15T18:22:17.394 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:256ee4bc:::benchmark_data_smithi159_102124_object45501:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:256fad91:::benchmark_data_smithi159_102124_object16175:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2572200e:::benchmark_data_smithi159_102124_object10780:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25725786:::benchmark_data_smithi159_102124_object41686:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2573cc46:::benchmark_data_smithi159_102124_object10989:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:257513a0:::benchmark_data_smithi159_102124_object32317:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2576929c:::benchmark_data_smithi159_102124_object52127:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25791d08:::benchmark_data_smithi159_102124_object5337:head# 2024-04-15T18:22:17.395 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2579d6ea:::benchmark_data_smithi159_102124_object289:head# 2024-04-15T18:22:17.398 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:257a3263:::benchmark_data_smithi159_102124_object6029:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25838eb6:::benchmark_data_smithi159_102124_object1843:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2583a2e5:::benchmark_data_smithi159_102124_object21785:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2587a1b4:::benchmark_data_smithi159_102124_object2065:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2587d1db:::benchmark_data_smithi159_102124_object12532:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25898c9b:::benchmark_data_smithi159_102124_object48858:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2589b953:::benchmark_data_smithi159_102124_object16229:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:258ab7c2:::benchmark_data_smithi159_102124_object56074:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:258bebc8:::benchmark_data_smithi159_102124_object31886:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:258d09fd:::benchmark_data_smithi159_102124_object15173:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:258d5339:::benchmark_data_smithi159_102124_object892:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259049f5:::benchmark_data_smithi159_102124_object36324:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25914139:::benchmark_data_smithi159_102124_object15337:head# 2024-04-15T18:22:17.399 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259143bc:::benchmark_data_smithi159_102124_object8375:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259257c8:::benchmark_data_smithi159_102124_object45209:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25934347:::benchmark_data_smithi159_102124_object35542:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259361bc:::benchmark_data_smithi159_102124_object56434:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259429fe:::benchmark_data_smithi159_102124_object28985:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25970dfc:::benchmark_data_smithi159_102124_object6778:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2597452a:::benchmark_data_smithi159_102124_object27342:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2597a43d:::benchmark_data_smithi159_102124_object31140:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2597b5ae:::benchmark_data_smithi159_102124_object42282:head# 2024-04-15T18:22:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259a0014:::benchmark_data_smithi159_102124_object47114:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259a7809:::benchmark_data_smithi159_102124_object10264:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259b131f:::benchmark_data_smithi159_102124_object57248:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259bcfe7:::benchmark_data_smithi159_102124_object25448:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259d253d:::benchmark_data_smithi159_102124_object43928:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259d8afc:::benchmark_data_smithi159_102124_object9262:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:259ddfa1:::benchmark_data_smithi159_102124_object616:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a0995d:::benchmark_data_smithi159_102124_object57945:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a0eaa7:::benchmark_data_smithi159_102124_object27203:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a113e1:::benchmark_data_smithi159_102124_object18986:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a1b827:::benchmark_data_smithi159_102124_object40348:head# 2024-04-15T18:22:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a471d3:::benchmark_data_smithi159_102124_object6552:head# 2024-04-15T18:22:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a4cb92:::benchmark_data_smithi159_102124_object56701:head# 2024-04-15T18:22:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a58c8f:::benchmark_data_smithi159_102124_object15366:head# 2024-04-15T18:22:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a7b8f9:::benchmark_data_smithi159_102124_object43464:head# 2024-04-15T18:22:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25a9876e:::benchmark_data_smithi159_102124_object12277:head# 2024-04-15T18:22:17.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ac892b:::benchmark_data_smithi159_102124_object38782:head# 2024-04-15T18:22:17.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ae0d25:::benchmark_data_smithi159_102124_object20908:head# 2024-04-15T18:22:17.404 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25af0b04:::benchmark_data_smithi159_102124_object38621:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25af45fe:::benchmark_data_smithi159_102124_object23402:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25af8739:::benchmark_data_smithi159_102124_object9846:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b07421:::benchmark_data_smithi159_102124_object24703:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b24461:::benchmark_data_smithi159_102124_object6093:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b37472:::benchmark_data_smithi159_102124_object33212:head# 2024-04-15T18:22:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b4469b:::benchmark_data_smithi159_102124_object7147:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b5c051:::benchmark_data_smithi159_102124_object57646:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b69594:::benchmark_data_smithi159_102124_object41802:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b6c422:::benchmark_data_smithi159_102124_object40000:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25b88842:::benchmark_data_smithi159_102124_object23442:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ba4665:::benchmark_data_smithi159_102124_object17094:head# 2024-04-15T18:22:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bb466f:::benchmark_data_smithi159_102124_object20102:head# 2024-04-15T18:22:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bb9470:::benchmark_data_smithi159_102124_object17775:head# 2024-04-15T18:22:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bc05f4:::benchmark_data_smithi159_102124_object6245:head# 2024-04-15T18:22:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bcb4d2:::benchmark_data_smithi159_102124_object45978:head# 2024-04-15T18:22:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25be5acf:::benchmark_data_smithi159_102124_object16177:head# 2024-04-15T18:22:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bf15dd:::benchmark_data_smithi159_102124_object23854:head# 2024-04-15T18:22:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bf2d36:::benchmark_data_smithi159_102124_object48075:head# 2024-04-15T18:22:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25bf2ef4:::benchmark_data_smithi159_102124_object15532:head# 2024-04-15T18:22:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c07640:::benchmark_data_smithi159_102124_object25117:head# 2024-04-15T18:22:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c0d554:::benchmark_data_smithi159_102124_object10922:head# 2024-04-15T18:22:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c0d69a:::benchmark_data_smithi159_102124_object53001:head# 2024-04-15T18:22:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c166ac:::benchmark_data_smithi159_102124_object26320:head# 2024-04-15T18:22:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c1ccfc:::benchmark_data_smithi159_102124_object55110:head# 2024-04-15T18:22:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c34a01:::benchmark_data_smithi159_102124_object36804:head# 2024-04-15T18:22:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c40fc7:::benchmark_data_smithi159_102124_object16228:head# 2024-04-15T18:22:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c4691b:::benchmark_data_smithi159_102124_object29020:head# 2024-04-15T18:22:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c5f83e:::benchmark_data_smithi159_102124_object25507:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c731f0:::benchmark_data_smithi159_102124_object16876:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c84788:::benchmark_data_smithi159_102124_object22993:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c863fa:::benchmark_data_smithi159_102124_object48637:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25c8c514:::benchmark_data_smithi159_102124_object53577:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ca7d03:::benchmark_data_smithi159_102124_object50032:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cb5629:::benchmark_data_smithi159_102124_object8214:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cc3120:::benchmark_data_smithi159_102124_object9656:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cc5c98:::benchmark_data_smithi159_102124_object53811:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cc61ac:::benchmark_data_smithi159_102124_object50875:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cc91f7:::benchmark_data_smithi159_102124_object2864:head# 2024-04-15T18:22:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cd4c5f:::benchmark_data_smithi159_102124_object24344:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cd602c:::benchmark_data_smithi159_102124_object8260:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cd6887:::benchmark_data_smithi159_102124_object26616:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ce5ece:::benchmark_data_smithi159_102124_object48213:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cedb88:::benchmark_data_smithi159_102124_object29328:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cf4945:::benchmark_data_smithi159_102124_object18437:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25cfe85d:::benchmark_data_smithi159_102124_object28378:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d0c053:::benchmark_data_smithi159_102124_object7622:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d0f225:::benchmark_data_smithi159_102124_object44872:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d335e4:::benchmark_data_smithi159_102124_object36143:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d353d3:::benchmark_data_smithi159_102124_object43037:head# 2024-04-15T18:22:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d523b6:::benchmark_data_smithi159_102124_object26022:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d5c3ba:::benchmark_data_smithi159_102124_object48366:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d659d5:::benchmark_data_smithi159_102124_object18058:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d74737:::benchmark_data_smithi159_102124_object41983:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d76d73:::benchmark_data_smithi159_102124_object37482:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d7a933:::benchmark_data_smithi159_102124_object7530:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d9c4a8:::benchmark_data_smithi159_102124_object50646:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25d9c4cc:::benchmark_data_smithi159_102124_object2298:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25daf94e:::benchmark_data_smithi159_102124_object10230:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25dd133e:::benchmark_data_smithi159_102124_object4892:head# 2024-04-15T18:22:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ddf9b5:::benchmark_data_smithi159_102124_object35435:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25df1ac1:::benchmark_data_smithi159_102124_object29923:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e15e72:::benchmark_data_smithi159_102124_object41931:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e445df:::benchmark_data_smithi159_102124_object9110:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e55333:::benchmark_data_smithi159_102124_object41398:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e65800:::benchmark_data_smithi159_102124_object31420:head# 2024-04-15T18:22:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e70ffc:::benchmark_data_smithi159_102124_object54875:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25e8cd7d:::benchmark_data_smithi159_102124_object33185:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ea01ce:::benchmark_data_smithi159_102124_object10767:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25eb0d92:::benchmark_data_smithi159_102124_object55784:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25ed154a:::benchmark_data_smithi159_102124_object54644:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25eedc3d:::benchmark_data_smithi159_102124_object44313:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f022b0:::benchmark_data_smithi159_102124_object9805:head# 2024-04-15T18:22:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f02e4c:::benchmark_data_smithi159_102124_object51340:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f0db1d:::benchmark_data_smithi159_102124_object22331:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f16f73:::benchmark_data_smithi159_102124_object49428:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f2d509:::benchmark_data_smithi159_102124_object24306:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f4ecc4:::benchmark_data_smithi159_102124_object26322:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f76982:::benchmark_data_smithi159_102124_object51594:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f7d69b:::benchmark_data_smithi159_102124_object25914:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f8f7a4:::benchmark_data_smithi159_102124_object44599:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25f9d393:::benchmark_data_smithi159_102124_object29203:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fa4f03:::benchmark_data_smithi159_102124_object15709:head# 2024-04-15T18:22:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25faa1e8:::benchmark_data_smithi159_102124_object32987:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fb255b:::benchmark_data_smithi159_102124_object4005:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fbad0c:::benchmark_data_smithi159_102124_object44504:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fbced7:::benchmark_data_smithi159_102124_object43575:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fd3d22:::benchmark_data_smithi159_102124_object40363:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fd5b69:::benchmark_data_smithi159_102124_object42474:head# 2024-04-15T18:22:17.426 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fdd9ae:::benchmark_data_smithi159_102124_object11911:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fe5e39:::benchmark_data_smithi159_102124_object342:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:25fe75a7:::benchmark_data_smithi159_102124_object9862:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2600e64a:::benchmark_data_smithi159_102124_object8502:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2601d2b1:::benchmark_data_smithi159_102124_object40103:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260353b2:::benchmark_data_smithi159_102124_object38610:head# 2024-04-15T18:22:17.428 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26037583:::benchmark_data_smithi159_102124_object42127:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2603c7c1:::benchmark_data_smithi159_102124_object47167:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2603eb2d:::benchmark_data_smithi159_102124_object44842:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2604bfe2:::benchmark_data_smithi159_102124_object52467:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26054a2a:::benchmark_data_smithi159_102124_object17812:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2605734b:::benchmark_data_smithi159_102124_object11133:head# 2024-04-15T18:22:17.429 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26057b53:::benchmark_data_smithi159_102124_object53384:head# 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26057bcb:::benchmark_data_smithi159_102124_object42416:head# 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2605c5a0:::benchmark_data_smithi159_102124_object22978:head# 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26061adf:::benchmark_data_smithi159_102124_object40323:head# 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2606a2a8:::benchmark_data_smithi159_102124_object22979:head# 2024-04-15T18:22:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26084dbc:::benchmark_data_smithi159_102124_object41796:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26086127:::benchmark_data_smithi159_102124_object53069:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2608da5e:::benchmark_data_smithi159_102124_object50668:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26090209:::benchmark_data_smithi159_102124_object46598:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26097ea4:::benchmark_data_smithi159_102124_object2402:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260980ce:::benchmark_data_smithi159_102124_object24293:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260a32e8:::benchmark_data_smithi159_102124_object58189:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260a7898:::benchmark_data_smithi159_102124_object36649:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260c44d9:::benchmark_data_smithi159_102124_object35821:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260d7b49:::benchmark_data_smithi159_102124_object2836:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260f6f96:::benchmark_data_smithi159_102124_object14419:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260f8d86:::benchmark_data_smithi159_102124_object51523:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:260fcc7e:::benchmark_data_smithi159_102124_object7617:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26109614:::benchmark_data_smithi159_102124_object52194:head# 2024-04-15T18:22:17.432 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26121853:::benchmark_data_smithi159_102124_object51933:head# 2024-04-15T18:22:17.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2612fb78:::benchmark_data_smithi159_102124_object27880:head# 2024-04-15T18:22:17.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2613b6d7:::benchmark_data_smithi159_102124_object46441:head# 2024-04-15T18:22:17.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2613e361:::benchmark_data_smithi159_102124_object3782:head# 2024-04-15T18:22:17.433 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2614d264:::benchmark_data_smithi159_102124_object18642:head# 2024-04-15T18:22:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:17.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8026/266535 objects degraded (3.011%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2615e903:::benchmark_data_smithi159_102124_object29295:head# 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26182b62:::benchmark_data_smithi159_102124_object2968:head# 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261845ab:::benchmark_data_smithi159_102124_object57037:head# 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2618ae41:::benchmark_data_smithi159_102124_object49652:head# 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2619b3c5:::benchmark_data_smithi159_102124_object4205:head# 2024-04-15T18:22:17.434 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261c1910:::benchmark_data_smithi159_102124_object17947:head# 2024-04-15T18:22:17.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:17.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261d0106:::benchmark_data_smithi159_102124_object40450:head# 2024-04-15T18:22:17.436 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261d340f:::benchmark_data_smithi159_102124_object44775:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261d8b33:::benchmark_data_smithi159_102124_object9171:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261da60a:::benchmark_data_smithi159_102124_object30360:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261dee27:::benchmark_data_smithi159_102124_object13265:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:261e4d8a:::benchmark_data_smithi159_102124_object22598:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2620190f:::benchmark_data_smithi159_102124_object36112:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262050af:::benchmark_data_smithi159_102124_object25751:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2620df56:::benchmark_data_smithi159_102124_object31734:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2620dfe6:::benchmark_data_smithi159_102124_object40168:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26216631:::benchmark_data_smithi159_102124_object20533:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26227863:::benchmark_data_smithi159_102124_object55723:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26247b22:::benchmark_data_smithi159_102124_object54919:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2624839e:::benchmark_data_smithi159_102124_object18163:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26269f72:::benchmark_data_smithi159_102124_object5920:head# 2024-04-15T18:22:17.437 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2626c1e7:::benchmark_data_smithi159_102124_object1974:head# 2024-04-15T18:22:17.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2626f15c:::benchmark_data_smithi159_102124_object38154:head# 2024-04-15T18:22:17.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2629291b:::benchmark_data_smithi159_102124_object19995:head# 2024-04-15T18:22:17.438 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262957bf:::benchmark_data_smithi159_102124_object36155:head# 2024-04-15T18:22:17.442 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2629bd5b:::benchmark_data_smithi159_102124_object15339:head# 2024-04-15T18:22:17.445 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262a2bde:::benchmark_data_smithi159_102124_object19684:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262d2b3b:::benchmark_data_smithi159_102124_object34312:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262d53a8:::benchmark_data_smithi159_102124_object26860:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262d6750:::benchmark_data_smithi159_102124_object16982:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262de795:::benchmark_data_smithi159_102124_object30055:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262e4a95:::benchmark_data_smithi159_102124_object56784:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:262f27c2:::benchmark_data_smithi159_102124_object3593:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26319d53:::benchmark_data_smithi159_102124_object27638:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26325c18:::benchmark_data_smithi159_102124_object284:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26341814:::benchmark_data_smithi159_102124_object57024:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26348456:::benchmark_data_smithi159_102124_object44576:head# 2024-04-15T18:22:17.452 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:263770e0:::benchmark_data_smithi159_102124_object15956:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2637b3dd:::benchmark_data_smithi159_102124_object18930:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2638fa82:::benchmark_data_smithi159_102124_object42659:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:263bd974:::benchmark_data_smithi159_102124_object24656:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:263c0ba7:::benchmark_data_smithi159_102124_object23528:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:263e1376:::benchmark_data_smithi159_102124_object18396:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:263f4853:::benchmark_data_smithi159_102124_object12224:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26402021:::benchmark_data_smithi159_102124_object49671:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:264298c4:::benchmark_data_smithi159_102124_object14309:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26436d16:::benchmark_data_smithi159_102124_object27826:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26440dee:::benchmark_data_smithi159_102124_object8440:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26444ef7:::benchmark_data_smithi159_102124_object20873:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2644caa5:::benchmark_data_smithi159_102124_object41596:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26455909:::benchmark_data_smithi159_102124_object50873:head# 2024-04-15T18:22:17.453 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26465148:::benchmark_data_smithi159_102124_object20364:head# 2024-04-15T18:22:17.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:264779a6:::benchmark_data_smithi159_102124_object45447:head# 2024-04-15T18:22:17.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2647e06a:::benchmark_data_smithi159_102124_object51028:head# 2024-04-15T18:22:17.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26498561:::benchmark_data_smithi159_102124_object58013:head# 2024-04-15T18:22:17.454 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:264a95f7:::benchmark_data_smithi159_102124_object24167:head# 2024-04-15T18:22:17.457 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:264e9b98:::benchmark_data_smithi159_102124_object54471:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2650064c:::benchmark_data_smithi159_102124_object58304:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26505a1f:::benchmark_data_smithi159_102124_object44375:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2652b99f:::benchmark_data_smithi159_102124_object15305:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2652f022:::benchmark_data_smithi159_102124_object46213:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2653d349:::benchmark_data_smithi159_102124_object56799:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26545915:::benchmark_data_smithi159_102124_object33611:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26561c7e:::benchmark_data_smithi159_102124_object4960:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26577c94:::benchmark_data_smithi159_102124_object19485:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26595fbb:::benchmark_data_smithi159_102124_object38816:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2659ac8e:::benchmark_data_smithi159_102124_object16925:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265a6680:::benchmark_data_smithi159_102124_object18206:head# 2024-04-15T18:22:17.458 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265b5ca0:::benchmark_data_smithi159_102124_object32125:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265c796b:::benchmark_data_smithi159_102124_object27619:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265e0307:::benchmark_data_smithi159_102124_object10101:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265e152f:::benchmark_data_smithi159_102124_object19409:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:265f683b:::benchmark_data_smithi159_102124_object33404:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2660795d:::benchmark_data_smithi159_102124_object4772:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2663037e:::benchmark_data_smithi159_102124_object15236:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2664f5c2:::benchmark_data_smithi159_102124_object20043:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26686cbb:::benchmark_data_smithi159_102124_object43165:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266921d3:::benchmark_data_smithi159_102124_object20667:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266a253c:::benchmark_data_smithi159_102124_object22241:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266a6785:::benchmark_data_smithi159_102124_object50359:head# 2024-04-15T18:22:17.459 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266bfe7d:::benchmark_data_smithi159_102124_object22747:head# 2024-04-15T18:22:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266c1ac8:::benchmark_data_smithi159_102124_object22944:head# 2024-04-15T18:22:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266ce958:::benchmark_data_smithi159_102124_object14199:head# 2024-04-15T18:22:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266d240e:::benchmark_data_smithi159_102124_object42163:head# 2024-04-15T18:22:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266e2cc6:::benchmark_data_smithi159_102124_object51321:head# 2024-04-15T18:22:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266f2273:::benchmark_data_smithi159_102124_object37082:head# 2024-04-15T18:22:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:266f2708:::benchmark_data_smithi159_102124_object30085:head# 2024-04-15T18:22:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26710614:::benchmark_data_smithi159_102124_object18819:head# 2024-04-15T18:22:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2672a879:::benchmark_data_smithi159_102124_object854:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:267772b7:::benchmark_data_smithi159_102124_object26952:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26783ced:::benchmark_data_smithi159_102124_object47226:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26789f8b:::benchmark_data_smithi159_102124_object43078:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:267a1467:::benchmark_data_smithi159_102124_object22032:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:267cd7b6:::benchmark_data_smithi159_102124_object349:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26818217:::benchmark_data_smithi159_102124_object48816:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26818584:::benchmark_data_smithi159_102124_object1819:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268368f7:::benchmark_data_smithi159_102124_object55866:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26842183:::benchmark_data_smithi159_102124_object55507:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268623f4:::benchmark_data_smithi159_102124_object49186:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26863674:::benchmark_data_smithi159_102124_object30270:head# 2024-04-15T18:22:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2686adf5:::benchmark_data_smithi159_102124_object10129:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2687851b:::benchmark_data_smithi159_102124_object12657:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26881390:::benchmark_data_smithi159_102124_object49902:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26884aad:::benchmark_data_smithi159_102124_object53010:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2688ce40:::benchmark_data_smithi159_102124_object28886:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2688fd8c:::benchmark_data_smithi159_102124_object50907:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268bc4bd:::benchmark_data_smithi159_102124_object23952:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268bd48b:::benchmark_data_smithi159_102124_object3200:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268c0b61:::benchmark_data_smithi159_102124_object50057:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268e5104:::benchmark_data_smithi159_102124_object8935:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:268f6a26:::benchmark_data_smithi159_102124_object24008:head# 2024-04-15T18:22:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26900f18:::benchmark_data_smithi159_102124_object43176:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26910caa:::benchmark_data_smithi159_102124_object19533:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269165d3:::benchmark_data_smithi159_102124_object7348:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269460e4:::benchmark_data_smithi159_102124_object11972:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2696131a:::benchmark_data_smithi159_102124_object53319:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2696d3a3:::benchmark_data_smithi159_102124_object54229:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269836bb:::benchmark_data_smithi159_102124_object33936:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26988030:::benchmark_data_smithi159_102124_object33150:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269aa7fa:::benchmark_data_smithi159_102124_object25811:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269bfefd:::benchmark_data_smithi159_102124_object31844:head# 2024-04-15T18:22:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269c1860:::benchmark_data_smithi159_102124_object45207:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269c52b1:::benchmark_data_smithi159_102124_object13019:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:269d6197:::benchmark_data_smithi159_102124_object11218:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a2129a:::benchmark_data_smithi159_102124_object12467:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a23738:::benchmark_data_smithi159_102124_object32499:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a2412f:::benchmark_data_smithi159_102124_object42732:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a2b190:::benchmark_data_smithi159_102124_object46415:head# 2024-04-15T18:22:17.472 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a2b2cf:::benchmark_data_smithi159_102124_object25353:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a39721:::benchmark_data_smithi159_102124_object35238:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a3e49f:::benchmark_data_smithi159_102124_object7134:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a441be:::benchmark_data_smithi159_102124_object47766:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26a4f6e1:::benchmark_data_smithi159_102124_object11985:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26aa3768:::benchmark_data_smithi159_102124_object5525:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26aa6bda:::benchmark_data_smithi159_102124_object35906:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26aaadf7:::benchmark_data_smithi159_102124_object42769:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26abee1b:::benchmark_data_smithi159_102124_object12257:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ae4ed7:::benchmark_data_smithi159_102124_object33319:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26aeef9a:::benchmark_data_smithi159_102124_object41571:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26af391c:::benchmark_data_smithi159_102124_object13798:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b1667e:::benchmark_data_smithi159_102124_object28409:head# 2024-04-15T18:22:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b3aeb1:::benchmark_data_smithi159_102124_object25257:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b444a4:::benchmark_data_smithi159_102124_object7240:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b51318:::benchmark_data_smithi159_102124_object30672:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b51875:::benchmark_data_smithi159_102124_object57520:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b720c8:::benchmark_data_smithi159_102124_object45245:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b76b6d:::benchmark_data_smithi159_102124_object43449:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b88e07:::benchmark_data_smithi159_102124_object41224:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b8a6a4:::benchmark_data_smithi159_102124_object19171:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26b96c26:::benchmark_data_smithi159_102124_object9405:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26bbd3ca:::benchmark_data_smithi159_102124_object57333:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26bcca55:::benchmark_data_smithi159_102124_object27571:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26be1c16:::benchmark_data_smithi159_102124_object34806:head# 2024-04-15T18:22:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26be35ec:::benchmark_data_smithi159_102124_object6085:head# 2024-04-15T18:22:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26bf35f9:::benchmark_data_smithi159_102124_object37506:head# 2024-04-15T18:22:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26bf366c:::benchmark_data_smithi159_102124_object43336:head# 2024-04-15T18:22:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26bf5943:::benchmark_data_smithi159_102124_object3163:head# 2024-04-15T18:22:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c06ed1:::benchmark_data_smithi159_102124_object56992:head# 2024-04-15T18:22:17.476 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c14039:::benchmark_data_smithi159_102124_object26849:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c18a13:::benchmark_data_smithi159_102124_object16995:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c21978:::benchmark_data_smithi159_102124_object41009:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c287cb:::benchmark_data_smithi159_102124_object31408:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c2a143:::benchmark_data_smithi159_102124_object34477:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c2d76c:::benchmark_data_smithi159_102124_object36966:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c35952:::benchmark_data_smithi159_102124_object12522:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c3664a:::benchmark_data_smithi159_102124_object52557:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c46bb3:::benchmark_data_smithi159_102124_object47513:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c4eabc:::benchmark_data_smithi159_102124_object16110:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c56358:::benchmark_data_smithi159_102124_object10500:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26c8c39f:::benchmark_data_smithi159_102124_object25487:head# 2024-04-15T18:22:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ca8fdd:::benchmark_data_smithi159_102124_object9401:head# 2024-04-15T18:22:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d01817:::benchmark_data_smithi159_102124_object35576:head# 2024-04-15T18:22:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d0d614:::benchmark_data_smithi159_102124_object38215:head# 2024-04-15T18:22:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d181e8:::benchmark_data_smithi159_102124_object53753:head# 2024-04-15T18:22:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d2bc54:::benchmark_data_smithi159_102124_object28790:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d2d019:::benchmark_data_smithi159_102124_object20071:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d42480:::benchmark_data_smithi159_102124_object50133:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d972bb:::benchmark_data_smithi159_102124_object20704:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d98fd6:::benchmark_data_smithi159_102124_object18828:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26d9edaf:::benchmark_data_smithi159_102124_object17722:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26da1c25:::benchmark_data_smithi159_102124_object1533:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26db00ea:::benchmark_data_smithi159_102124_object41035:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26db1a3e:::benchmark_data_smithi159_102124_object54365:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26db53ba:::benchmark_data_smithi159_102124_object50302:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26dc61d1:::benchmark_data_smithi159_102124_object52377:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26dc71a9:::benchmark_data_smithi159_102124_object23458:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26dc8c6d:::benchmark_data_smithi159_102124_object29214:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26de62f4:::benchmark_data_smithi159_102124_object42671:head# 2024-04-15T18:22:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e10318:::benchmark_data_smithi159_102124_object19946:head# 2024-04-15T18:22:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e18b44:::benchmark_data_smithi159_102124_object47379:head# 2024-04-15T18:22:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e33cce:::benchmark_data_smithi159_102124_object45879:head# 2024-04-15T18:22:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e3e463:::benchmark_data_smithi159_102124_object26577:head# 2024-04-15T18:22:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e51d20:::benchmark_data_smithi159_102124_object11369:head# 2024-04-15T18:22:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e5cab5:::benchmark_data_smithi159_102124_object56800:head# 2024-04-15T18:22:17.482 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e5da03:::benchmark_data_smithi159_102124_object52503:head# 2024-04-15T18:22:17.482 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e8c7a4:::benchmark_data_smithi159_102124_object53308:head# 2024-04-15T18:22:17.482 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26e94e39:::benchmark_data_smithi159_102124_object28482:head# 2024-04-15T18:22:17.482 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ea07f3:::benchmark_data_smithi159_102124_object12734:head# 2024-04-15T18:22:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26eb8caa:::benchmark_data_smithi159_102124_object44896:head# 2024-04-15T18:22:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ecf77a:::benchmark_data_smithi159_102124_object20455:head# 2024-04-15T18:22:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26eecdca:::benchmark_data_smithi159_102124_object28357:head# 2024-04-15T18:22:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f04e06:::benchmark_data_smithi159_102124_object53748:head# 2024-04-15T18:22:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f1c47f:::benchmark_data_smithi159_102124_object5763:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f34483:::benchmark_data_smithi159_102124_object3308:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f3ad31:::benchmark_data_smithi159_102124_object3745:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f55995:::benchmark_data_smithi159_102124_object9218:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f5a001:::benchmark_data_smithi159_102124_object25501:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f5a3ae:::benchmark_data_smithi159_102124_object29427:head# 2024-04-15T18:22:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f5e0e5:::benchmark_data_smithi159_102124_object57240:head# 2024-04-15T18:22:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f73cf2:::benchmark_data_smithi159_102124_object7703:head# 2024-04-15T18:22:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f7432e:::benchmark_data_smithi159_102124_object15722:head# 2024-04-15T18:22:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f7c8c9:::benchmark_data_smithi159_102124_object11600:head# 2024-04-15T18:22:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26f91708:::benchmark_data_smithi159_102124_object17115:head# 2024-04-15T18:22:17.488 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26fca5dc:::benchmark_data_smithi159_102124_object45176:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26fdd575:::benchmark_data_smithi159_102124_object52751:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26fe3855:::benchmark_data_smithi159_102124_object56264:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26fed884:::benchmark_data_smithi159_102124_object56955:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ff1e36:::benchmark_data_smithi159_102124_object418:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ff2947:::benchmark_data_smithi159_102124_object17050:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:26ff5fc6:::benchmark_data_smithi159_102124_object7489:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27007734:::benchmark_data_smithi159_102124_object53690:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2700f793:::benchmark_data_smithi159_102124_object25154:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27015e85:::benchmark_data_smithi159_102124_object8398:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27036371:::benchmark_data_smithi159_102124_object7537:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2703b112:::benchmark_data_smithi159_102124_object9503:head# 2024-04-15T18:22:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2703bdd4:::benchmark_data_smithi159_102124_object39214:head# 2024-04-15T18:22:17.490 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27053816:::benchmark_data_smithi159_102124_object32585:head# 2024-04-15T18:22:17.494 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27058b27:::benchmark_data_smithi159_102124_object52228:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:270601a2:::benchmark_data_smithi159_102124_object41416:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27067f3c:::benchmark_data_smithi159_102124_object47099:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2708e514:::benchmark_data_smithi159_102124_object36788:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27093952:::benchmark_data_smithi159_102124_object28233:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:270adfb8:::benchmark_data_smithi159_102124_object7646:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:270d78c0:::benchmark_data_smithi159_102124_object34865:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:270f04de:::benchmark_data_smithi159_102124_object48350:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:270f58f6:::benchmark_data_smithi159_102124_object56329:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271010fb:::benchmark_data_smithi159_102124_object9297:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2711c30c:::benchmark_data_smithi159_102124_object19131:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2713d3a4:::benchmark_data_smithi159_102124_object37584:head# 2024-04-15T18:22:17.495 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27161626:::benchmark_data_smithi159_102124_object31915:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27179507:::benchmark_data_smithi159_102124_object14631:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2718f3b8:::benchmark_data_smithi159_102124_object3865:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2719b95f:::benchmark_data_smithi159_102124_object58252:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2719cd29:::benchmark_data_smithi159_102124_object28024:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271af96c:::benchmark_data_smithi159_102124_object52582:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271cba92:::benchmark_data_smithi159_102124_object6131:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271e3564:::benchmark_data_smithi159_102124_object17159:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271e89a9:::benchmark_data_smithi159_102124_object9612:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271ec047:::benchmark_data_smithi159_102124_object1914:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271f3363:::benchmark_data_smithi159_102124_object22631:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271f86f7:::benchmark_data_smithi159_102124_object48004:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:271f9ef0:::benchmark_data_smithi159_102124_object40866:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272015ff:::benchmark_data_smithi159_102124_object26784:head# 2024-04-15T18:22:17.496 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27208ee1:::benchmark_data_smithi159_102124_object16102:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27212dbb:::benchmark_data_smithi159_102124_object36753:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2722c367:::benchmark_data_smithi159_102124_object36703:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272317ef:::benchmark_data_smithi159_102124_object4462:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272321df:::benchmark_data_smithi159_102124_object56132:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27233a01:::benchmark_data_smithi159_102124_object54884:head# 2024-04-15T18:22:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2724f9b2:::benchmark_data_smithi159_102124_object18131:head# 2024-04-15T18:22:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27266f42:::benchmark_data_smithi159_102124_object8180:head# 2024-04-15T18:22:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2729ee32:::benchmark_data_smithi159_102124_object57915:head# 2024-04-15T18:22:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272ab165:::benchmark_data_smithi159_102124_object3460:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272ad91e:::benchmark_data_smithi159_102124_object39345:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272b734c:::benchmark_data_smithi159_102124_object776:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272b8520:::benchmark_data_smithi159_102124_object43270:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272b8b83:::benchmark_data_smithi159_102124_object41985:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272c9755:::benchmark_data_smithi159_102124_object54986:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272df14e:::benchmark_data_smithi159_102124_object21346:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272e8009:::benchmark_data_smithi159_102124_object45120:head# 2024-04-15T18:22:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272ec561:::benchmark_data_smithi159_102124_object25913:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272faba7:::benchmark_data_smithi159_102124_object50820:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:272fbd31:::benchmark_data_smithi159_102124_object35807:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2733d69d:::benchmark_data_smithi159_102124_object30651:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27345b28:::benchmark_data_smithi159_102124_object9832:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27351528:::benchmark_data_smithi159_102124_object28650:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27362fff:::benchmark_data_smithi159_102124_object56681:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2736425b:::benchmark_data_smithi159_102124_object55962:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27373213:::benchmark_data_smithi159_102124_object21238:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:273a8947:::benchmark_data_smithi159_102124_object30131:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:273cf0ba:::benchmark_data_smithi159_102124_object43196:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:273d2e11:::benchmark_data_smithi159_102124_object32614:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:273ee8dc:::benchmark_data_smithi159_102124_object44597:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:274290f9:::benchmark_data_smithi159_102124_object58316:head# 2024-04-15T18:22:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27433633:::benchmark_data_smithi159_102124_object27030:head# 2024-04-15T18:22:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2744abec:::benchmark_data_smithi159_102124_object4987:head# 2024-04-15T18:22:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27458a8a:::benchmark_data_smithi159_102124_object1248:head# 2024-04-15T18:22:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27459e45:::benchmark_data_smithi159_102124_object49111:head# 2024-04-15T18:22:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27464a7c:::benchmark_data_smithi159_102124_object40198:head# 2024-04-15T18:22:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27467d1c:::benchmark_data_smithi159_102124_object33414:head# 2024-04-15T18:22:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2746d823:::benchmark_data_smithi159_102124_object54485:head# 2024-04-15T18:22:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2748d586:::benchmark_data_smithi159_102124_object14347:head# 2024-04-15T18:22:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2748edda:::benchmark_data_smithi159_102124_object5999:head# 2024-04-15T18:22:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2749adae:::benchmark_data_smithi159_102124_object51919:head# 2024-04-15T18:22:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:274af554:::benchmark_data_smithi159_102124_object27652:head# 2024-04-15T18:22:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:274f3a6c:::benchmark_data_smithi159_102124_object46615:head# 2024-04-15T18:22:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2750bf2b:::benchmark_data_smithi159_102124_object53279:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27510151:::benchmark_data_smithi159_102124_object53359:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275102d9:::benchmark_data_smithi159_102124_object50150:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275142c6:::benchmark_data_smithi159_102124_object3148:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2751f4a6:::benchmark_data_smithi159_102124_object1768:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2752f8cd:::benchmark_data_smithi159_102124_object1958:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275477e1:::benchmark_data_smithi159_102124_object43258:head# 2024-04-15T18:22:17.512 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2754b925:::benchmark_data_smithi159_102124_object30513:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27555853:::benchmark_data_smithi159_102124_object26603:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2755b4a5:::benchmark_data_smithi159_102124_object7621:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275642b2:::benchmark_data_smithi159_102124_object45261:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2756b8bc:::benchmark_data_smithi159_102124_object15689:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2756de7d:::benchmark_data_smithi159_102124_object20275:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2756e006:::benchmark_data_smithi159_102124_object39665:head# 2024-04-15T18:22:17.513 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2757beb8:::benchmark_data_smithi159_102124_object16128:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2758efc9:::benchmark_data_smithi159_102124_object13085:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275b8499:::benchmark_data_smithi159_102124_object38090:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275bdde7:::benchmark_data_smithi159_102124_object29251:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275c7885:::benchmark_data_smithi159_102124_object53602:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275cc951:::benchmark_data_smithi159_102124_object12649:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275d28a0:::benchmark_data_smithi159_102124_object53893:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:275edf99:::benchmark_data_smithi159_102124_object9895:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:276087ed:::benchmark_data_smithi159_102124_object32423:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2762aebf:::benchmark_data_smithi159_102124_object11460:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2767c40a:::benchmark_data_smithi159_102124_object50433:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2768f3ec:::benchmark_data_smithi159_102124_object25778:head# 2024-04-15T18:22:17.514 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:276a7dc4:::benchmark_data_smithi159_102124_object53634:head# 2024-04-15T18:22:17.515 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:276bca36:::benchmark_data_smithi159_102124_object48036:head# 2024-04-15T18:22:17.515 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:276bda7d:::benchmark_data_smithi159_102124_object16378:head# 2024-04-15T18:22:17.515 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27701011:::benchmark_data_smithi159_102124_object25514:head# 2024-04-15T18:22:17.515 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277118dd:::benchmark_data_smithi159_102124_object10503:head# 2024-04-15T18:22:17.516 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2771f51d:::benchmark_data_smithi159_102124_object44713:head# 2024-04-15T18:22:17.516 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27722e89:::benchmark_data_smithi159_102124_object14966:head# 2024-04-15T18:22:17.516 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27723b6c:::benchmark_data_smithi159_102124_object44927:head# 2024-04-15T18:22:17.516 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277573a0:::benchmark_data_smithi159_102124_object23019:head# 2024-04-15T18:22:17.516 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27772f54:::benchmark_data_smithi159_102124_object7053:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27788afd:::benchmark_data_smithi159_102124_object21103:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277919dd:::benchmark_data_smithi159_102124_object41372:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27796d94:::benchmark_data_smithi159_102124_object56219:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2779dafc:::benchmark_data_smithi159_102124_object47236:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2779e1b7:::benchmark_data_smithi159_102124_object21595:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277bea72:::benchmark_data_smithi159_102124_object48920:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277c92eb:::benchmark_data_smithi159_102124_object6770:head# 2024-04-15T18:22:17.517 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:277de83b:::benchmark_data_smithi159_102124_object38041:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27806079:::benchmark_data_smithi159_102124_object3814:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2781f504:::benchmark_data_smithi159_102124_object13365:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:278262ec:::benchmark_data_smithi159_102124_object33182:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2787268a:::benchmark_data_smithi159_102124_object41702:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2788caf6:::benchmark_data_smithi159_102124_object8030:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:278c2598:::benchmark_data_smithi159_102124_object1128:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:278d69f3:::benchmark_data_smithi159_102124_object19583:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:278d9467:::benchmark_data_smithi159_102124_object29986:head# 2024-04-15T18:22:17.518 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:278f86fc:::benchmark_data_smithi159_102124_object42075:head# 2024-04-15T18:22:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2790317a:::benchmark_data_smithi159_102124_object21130:head# 2024-04-15T18:22:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27916548:::benchmark_data_smithi159_102124_object30183:head# 2024-04-15T18:22:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27922a58:::benchmark_data_smithi159_102124_object13996:head# 2024-04-15T18:22:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27947335:::benchmark_data_smithi159_102124_object56360:head# 2024-04-15T18:22:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:279494b4:::benchmark_data_smithi159_102124_object30761:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2795a81a:::benchmark_data_smithi159_102124_object49768:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27988e45:::benchmark_data_smithi159_102124_object41835:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2798ef35:::benchmark_data_smithi159_102124_object52716:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27999a33:::benchmark_data_smithi159_102124_object3063:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:279ca42d:::benchmark_data_smithi159_102124_object28058:head# 2024-04-15T18:22:17.522 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:279e2430:::benchmark_data_smithi159_102124_object12157:head# 2024-04-15T18:22:17.523 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:279fbcc1:::benchmark_data_smithi159_102124_object3067:head# 2024-04-15T18:22:17.523 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a2324a:::benchmark_data_smithi159_102124_object3504:head# 2024-04-15T18:22:17.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a27ff3:::benchmark_data_smithi159_102124_object18539:head# 2024-04-15T18:22:17.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a2de60:::benchmark_data_smithi159_102124_object40470:head# 2024-04-15T18:22:17.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a30f81:::benchmark_data_smithi159_102124_object27630:head# 2024-04-15T18:22:17.524 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a55016:::benchmark_data_smithi159_102124_object58160:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a5aa42:::benchmark_data_smithi159_102124_object3191:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a67a42:::benchmark_data_smithi159_102124_object47233:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a7dbee:::benchmark_data_smithi159_102124_object6078:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a7fe02:::benchmark_data_smithi159_102124_object33481:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27a958e2:::benchmark_data_smithi159_102124_object54687:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27aa4c4c:::benchmark_data_smithi159_102124_object53153:head# 2024-04-15T18:22:17.525 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27acd251:::benchmark_data_smithi159_102124_object38132:head# 2024-04-15T18:22:17.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27adc61a:::benchmark_data_smithi159_102124_object50303:head# 2024-04-15T18:22:17.526 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27addbef:::benchmark_data_smithi159_102124_object26458:head# 2024-04-15T18:22:17.527 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27af5893:::benchmark_data_smithi159_102124_object16529:head# 2024-04-15T18:22:17.527 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27b58ad5:::benchmark_data_smithi159_102124_object43376:head# 2024-04-15T18:22:17.527 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27b6c306:::benchmark_data_smithi159_102124_object50353:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27b8c4d2:::benchmark_data_smithi159_102124_object12440:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27b8e2c8:::benchmark_data_smithi159_102124_object58454:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27b9e94b:::benchmark_data_smithi159_102124_object25010:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27bad7a7:::benchmark_data_smithi159_102124_object28133:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27bbf8aa:::benchmark_data_smithi159_102124_object14911:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27bd67f8:::benchmark_data_smithi159_102124_object12707:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c22a26:::benchmark_data_smithi159_102124_object18903:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c2da12:::benchmark_data_smithi159_102124_object442:head# 2024-04-15T18:22:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c2f271:::benchmark_data_smithi159_102124_object8978:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c409dc:::benchmark_data_smithi159_102124_object53278:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c49d23:::benchmark_data_smithi159_102124_object33255:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c520c7:::benchmark_data_smithi159_102124_object31738:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c54dd1:::benchmark_data_smithi159_102124_object57129:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c6b935:::benchmark_data_smithi159_102124_object148:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27c9d0d5:::benchmark_data_smithi159_102124_object10626:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ca3d31:::benchmark_data_smithi159_102124_object19413:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ca83ee:::benchmark_data_smithi159_102124_object29832:head# 2024-04-15T18:22:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27cad303:::benchmark_data_smithi159_102124_object45563:head# 2024-04-15T18:22:17.534 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27cb45d0:::benchmark_data_smithi159_102124_object53208:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27cc8d7f:::benchmark_data_smithi159_102124_object2149:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27cda5f3:::benchmark_data_smithi159_102124_object22207:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27cff870:::benchmark_data_smithi159_102124_object57876:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27d02aed:::benchmark_data_smithi159_102124_object32098:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27d305a7:::benchmark_data_smithi159_102124_object47616:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27d60059:::benchmark_data_smithi159_102124_object7286:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27d68337:::benchmark_data_smithi159_102124_object49415:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27d9e44f:::benchmark_data_smithi159_102124_object52599:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dae632:::benchmark_data_smithi159_102124_object8411:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dbfc63:::benchmark_data_smithi159_102124_object11010:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dc2ba1:::benchmark_data_smithi159_102124_object19664:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dc76d3:::benchmark_data_smithi159_102124_object1190:head# 2024-04-15T18:22:17.535 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dc9dc5:::benchmark_data_smithi159_102124_object2678:head# 2024-04-15T18:22:17.536 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27de7c14:::benchmark_data_smithi159_102124_object8789:head# 2024-04-15T18:22:17.536 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27de9289:::benchmark_data_smithi159_102124_object42090:head# 2024-04-15T18:22:17.536 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27dec9aa:::benchmark_data_smithi159_102124_object54030:head# 2024-04-15T18:22:17.536 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27df83f8:::benchmark_data_smithi159_102124_object43603:head# 2024-04-15T18:22:17.536 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27df9c4e:::benchmark_data_smithi159_102124_object23581:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e44fb3:::benchmark_data_smithi159_102124_object1270:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e4accb:::benchmark_data_smithi159_102124_object35295:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e54f60:::benchmark_data_smithi159_102124_object5139:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e59645:::benchmark_data_smithi159_102124_object13247:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e5e168:::benchmark_data_smithi159_102124_object39316:head# 2024-04-15T18:22:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e5e7b5:::benchmark_data_smithi159_102124_object41851:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e770a6:::benchmark_data_smithi159_102124_object34378:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27e81695:::benchmark_data_smithi159_102124_object33423:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ebbbad:::benchmark_data_smithi159_102124_object25307:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ebebc0:::benchmark_data_smithi159_102124_object5069:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ec6c0e:::benchmark_data_smithi159_102124_object44369:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ece549:::benchmark_data_smithi159_102124_object15348:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27eee58a:::benchmark_data_smithi159_102124_object57007:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27f005ce:::benchmark_data_smithi159_102124_object40117:head# 2024-04-15T18:22:17.538 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27f2db8c:::benchmark_data_smithi159_102124_object11804:head# 2024-04-15T18:22:17.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27f31783:::benchmark_data_smithi159_102124_object50827:head# 2024-04-15T18:22:17.539 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27f915fd:::benchmark_data_smithi159_102124_object49046:head# 2024-04-15T18:22:17.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27f958a5:::benchmark_data_smithi159_102124_object58085:head# 2024-04-15T18:22:17.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fa7ab6:::benchmark_data_smithi159_102124_object14075:head# 2024-04-15T18:22:17.540 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fa9cee:::benchmark_data_smithi159_102124_object19821:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fabbee:::benchmark_data_smithi159_102124_object18367:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fcda11:::benchmark_data_smithi159_102124_object57642:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fd6e24:::benchmark_data_smithi159_102124_object17619:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27fe7963:::benchmark_data_smithi159_102124_object36863:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27febdca:::benchmark_data_smithi159_102124_object9998:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:27ffd4a0:::benchmark_data_smithi159_102124_object46852:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2804f579:::benchmark_data_smithi159_102124_object44808:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28054bba:::benchmark_data_smithi159_102124_object48599:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2808219a:::benchmark_data_smithi159_102124_object47873:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:280c791b:::benchmark_data_smithi159_102124_object44461:head# 2024-04-15T18:22:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:280cdb01:::benchmark_data_smithi159_102124_object55640:head# 2024-04-15T18:22:17.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:280d6fac:::benchmark_data_smithi159_102124_object2765:head# 2024-04-15T18:22:17.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:280ecbb6:::benchmark_data_smithi159_102124_object10265:head# 2024-04-15T18:22:17.545 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2811b92f:::benchmark_data_smithi159_102124_object15032:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2811fc02:::benchmark_data_smithi159_102124_object50024:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2813b346:::benchmark_data_smithi159_102124_object39416:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28141a0b:::benchmark_data_smithi159_102124_object24083:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281443d3:::benchmark_data_smithi159_102124_object11363:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28146020:::benchmark_data_smithi159_102124_object33183:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281558e4:::benchmark_data_smithi159_102124_object2506:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28162f77:::benchmark_data_smithi159_102124_object29638:head# 2024-04-15T18:22:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2818ebb6:::benchmark_data_smithi159_102124_object38819:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2819053d:::benchmark_data_smithi159_102124_object45845:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2819f403:::benchmark_data_smithi159_102124_object50094:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281a1221:::benchmark_data_smithi159_102124_object44716:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281a17f7:::benchmark_data_smithi159_102124_object47058:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281c7d37:::benchmark_data_smithi159_102124_object9761:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:281e511a:::benchmark_data_smithi159_102124_object39314:head# 2024-04-15T18:22:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28211b83:::benchmark_data_smithi159_102124_object10288:head# 2024-04-15T18:22:17.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:282348d8:::benchmark_data_smithi159_102124_object33506:head# 2024-04-15T18:22:17.550 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2827a7ef:::benchmark_data_smithi159_102124_object17573:head# 2024-04-15T18:22:17.551 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:282a1525:::benchmark_data_smithi159_102124_object48883:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:282be60b:::benchmark_data_smithi159_102124_object32186:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:282c23de:::benchmark_data_smithi159_102124_object57932:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:282fa01a:::benchmark_data_smithi159_102124_object15084:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28301f47:::benchmark_data_smithi159_102124_object30257:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2831bbc8:::benchmark_data_smithi159_102124_object13396:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28321232:::benchmark_data_smithi159_102124_object42585:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283342fe:::benchmark_data_smithi159_102124_object58083:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2834eaf2:::benchmark_data_smithi159_102124_object25213:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283623d4:::benchmark_data_smithi159_102124_object24430:head# 2024-04-15T18:22:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283694c4:::benchmark_data_smithi159_102124_object47396:head# 2024-04-15T18:22:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2837153f:::benchmark_data_smithi159_102124_object10310:head# 2024-04-15T18:22:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2838479c:::benchmark_data_smithi159_102124_object52738:head# 2024-04-15T18:22:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283a03e4:::benchmark_data_smithi159_102124_object17270:head# 2024-04-15T18:22:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283b891b:::benchmark_data_smithi159_102124_object36385:head# 2024-04-15T18:22:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283c596a:::benchmark_data_smithi159_102124_object57036:head# 2024-04-15T18:22:17.556 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283c9458:::benchmark_data_smithi159_102124_object49266:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:283de3e8:::benchmark_data_smithi159_102124_object55100:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2840a396:::benchmark_data_smithi159_102124_object3861:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2840d5b6:::benchmark_data_smithi159_102124_object13701:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2841859a:::benchmark_data_smithi159_102124_object37335:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2841dbeb:::benchmark_data_smithi159_102124_object46721:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284376d9:::benchmark_data_smithi159_102124_object3409:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2843c2b0:::benchmark_data_smithi159_102124_object57542:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28461405:::benchmark_data_smithi159_102124_object33957:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284890db:::benchmark_data_smithi159_102124_object32296:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28489b12:::benchmark_data_smithi159_102124_object5647:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2849f21f:::benchmark_data_smithi159_102124_object33919:head# 2024-04-15T18:22:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284b7166:::benchmark_data_smithi159_102124_object33363:head# 2024-04-15T18:22:17.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284bf5e0:::benchmark_data_smithi159_102124_object47106:head# 2024-04-15T18:22:17.558 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284c3c7e:::benchmark_data_smithi159_102124_object46442:head# 2024-04-15T18:22:17.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284c62c3:::benchmark_data_smithi159_102124_object54528:head# 2024-04-15T18:22:17.559 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:284cb472:::benchmark_data_smithi159_102124_object21509:head# 2024-04-15T18:22:17.561 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2853c927:::benchmark_data_smithi159_102124_object7657:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285624dd:::benchmark_data_smithi159_102124_object8342:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28564de4:::benchmark_data_smithi159_102124_object27661:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2856e5ae:::benchmark_data_smithi159_102124_object49856:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28588fb5:::benchmark_data_smithi159_102124_object6690:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2858cb7c:::benchmark_data_smithi159_102124_object43939:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2858d87e:::benchmark_data_smithi159_102124_object31587:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28597123:::benchmark_data_smithi159_102124_object20169:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285c33ab:::benchmark_data_smithi159_102124_object4658:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285c3c65:::benchmark_data_smithi159_102124_object1993:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285e3e08:::benchmark_data_smithi159_102124_object55258:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285e4c9c:::benchmark_data_smithi159_102124_object44885:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:285fe546:::benchmark_data_smithi159_102124_object13138:head# 2024-04-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28600a7f:::benchmark_data_smithi159_102124_object8749:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2860a90c:::benchmark_data_smithi159_102124_object25410:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2861114a:::benchmark_data_smithi159_102124_object58344:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2861634c:::benchmark_data_smithi159_102124_object5752:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2863786e:::benchmark_data_smithi159_102124_object12069:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2863a9f3:::benchmark_data_smithi159_102124_object4766:head# 2024-04-15T18:22:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2864985a:::benchmark_data_smithi159_102124_object11963:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28649c19:::benchmark_data_smithi159_102124_object40810:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2865217d:::benchmark_data_smithi159_102124_object57866:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28656e3b:::benchmark_data_smithi159_102124_object684:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2865fa83:::benchmark_data_smithi159_102124_object34653:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2866d997:::benchmark_data_smithi159_102124_object36295:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28698946:::benchmark_data_smithi159_102124_object15471:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2869e844:::benchmark_data_smithi159_102124_object6415:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:286a2b14:::benchmark_data_smithi159_102124_object20101:head# 2024-04-15T18:22:17.566 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:286da10c:::benchmark_data_smithi159_102124_object52229:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:286fa456:::benchmark_data_smithi159_102124_object53094:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28706204:::benchmark_data_smithi159_102124_object42003:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28716e86:::benchmark_data_smithi159_102124_object49296:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28723a13:::benchmark_data_smithi159_102124_object36471:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28732440:::benchmark_data_smithi159_102124_object36096:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28740818:::benchmark_data_smithi159_102124_object35817:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2874ab16:::benchmark_data_smithi159_102124_object53409:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2874d045:::benchmark_data_smithi159_102124_object15664:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2876710d:::benchmark_data_smithi159_102124_object41480:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2876d62a:::benchmark_data_smithi159_102124_object44137:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28774680:::benchmark_data_smithi159_102124_object11148:head# 2024-04-15T18:22:17.567 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2877720a:::benchmark_data_smithi159_102124_object33949:head# 2024-04-15T18:22:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28799aa4:::benchmark_data_smithi159_102124_object3543:head# 2024-04-15T18:22:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2879c63a:::benchmark_data_smithi159_102124_object33448:head# 2024-04-15T18:22:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:287b4309:::benchmark_data_smithi159_102124_object17929:head# 2024-04-15T18:22:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:287d0e01:::benchmark_data_smithi159_102124_object281:head# 2024-04-15T18:22:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:287d8e9d:::benchmark_data_smithi159_102124_object19430:head# 2024-04-15T18:22:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:287e0dee:::benchmark_data_smithi159_102124_object13833:head# 2024-04-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28805fc6:::benchmark_data_smithi159_102124_object46041:head# 2024-04-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28811d21:::benchmark_data_smithi159_102124_object12651:head# 2024-04-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28817faf:::benchmark_data_smithi159_102124_object27422:head# 2024-04-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2881e2a4:::benchmark_data_smithi159_102124_object9793:head# 2024-04-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28835db1:::benchmark_data_smithi159_102124_object31379:head# 2024-04-15T18:22:17.570 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8026/266535 objects degraded (3.011%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:288406e2:::benchmark_data_smithi159_102124_object30029:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2884bd70:::benchmark_data_smithi159_102124_object39559:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28856995:::benchmark_data_smithi159_102124_object22453:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28869b4f:::benchmark_data_smithi159_102124_object27909:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2886c5c4:::benchmark_data_smithi159_102124_object44640:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28872f8d:::benchmark_data_smithi159_102124_object41862:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28888d65:::benchmark_data_smithi159_102124_object51578:head# 2024-04-15T18:22:17.574 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2888a057:::benchmark_data_smithi159_102124_object40255:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:288a28d2:::benchmark_data_smithi159_102124_object56593:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:288a9797:::benchmark_data_smithi159_102124_object23845:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2892342e:::benchmark_data_smithi159_102124_object47551:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28923b6b:::benchmark_data_smithi159_102124_object3746:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28930c3f:::benchmark_data_smithi159_102124_object12678:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28939a20:::benchmark_data_smithi159_102124_object27179:head# 2024-04-15T18:22:17.575 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2893d59a:::benchmark_data_smithi159_102124_object22023:head# 2024-04-15T18:22:17.576 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28940a69:::benchmark_data_smithi159_102124_object46357:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289544c1:::benchmark_data_smithi159_102124_object13148:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2896725c:::benchmark_data_smithi159_102124_object4088:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2896a64a:::benchmark_data_smithi159_102124_object41270:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2897722e:::benchmark_data_smithi159_102124_object27906:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28982beb:::benchmark_data_smithi159_102124_object39089:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2898807d:::benchmark_data_smithi159_102124_object54860:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2898a71e:::benchmark_data_smithi159_102124_object8024:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2899f4ec:::benchmark_data_smithi159_102124_object7273:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289b9891:::benchmark_data_smithi159_102124_object28776:head# 2024-04-15T18:22:17.579 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289be15b:::benchmark_data_smithi159_102124_object40062:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289ce0f7:::benchmark_data_smithi159_102124_object37139:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289e6ec6:::benchmark_data_smithi159_102124_object37321:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289ec5b5:::benchmark_data_smithi159_102124_object3509:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289f45c9:::benchmark_data_smithi159_102124_object54609:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:289f54c2:::benchmark_data_smithi159_102124_object27484:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a1ef2b:::benchmark_data_smithi159_102124_object11412:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a2dffc:::benchmark_data_smithi159_102124_object11954:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a4108d:::benchmark_data_smithi159_102124_object54602:head# 2024-04-15T18:22:17.580 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a4332a:::benchmark_data_smithi159_102124_object40476:head# 2024-04-15T18:22:17.581 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a43830:::benchmark_data_smithi159_102124_object4235:head# 2024-04-15T18:22:17.581 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a452a4:::benchmark_data_smithi159_102124_object46244:head# 2024-04-15T18:22:17.581 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a4aa21:::benchmark_data_smithi159_102124_object42209:head# 2024-04-15T18:22:17.581 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a732cc:::benchmark_data_smithi159_102124_object38670:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a7e558:::benchmark_data_smithi159_102124_object38002:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a84679:::benchmark_data_smithi159_102124_object41175:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a85bca:::benchmark_data_smithi159_102124_object21244:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a8d544:::benchmark_data_smithi159_102124_object58247:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28a9a587:::benchmark_data_smithi159_102124_object4684:head# 2024-04-15T18:22:17.582 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ad0a95:::benchmark_data_smithi159_102124_object3917:head# 2024-04-15T18:22:17.583 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28aef23c:::benchmark_data_smithi159_102124_object53994:head# 2024-04-15T18:22:17.583 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28af2e5a:::benchmark_data_smithi159_102124_object29012:head# 2024-04-15T18:22:17.592 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28af6264:::benchmark_data_smithi159_102124_object48524:head# 2024-04-15T18:22:17.592 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28af9b7f:::benchmark_data_smithi159_102124_object15115:head# 2024-04-15T18:22:17.592 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28afac13:::benchmark_data_smithi159_102124_object7746:head# 2024-04-15T18:22:17.592 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28aff461:::benchmark_data_smithi159_102124_object1440:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b2338e:::benchmark_data_smithi159_102124_object17416:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b37fea:::benchmark_data_smithi159_102124_object52222:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b51a43:::benchmark_data_smithi159_102124_object8425:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b5a1a5:::benchmark_data_smithi159_102124_object11254:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b6cb6c:::benchmark_data_smithi159_102124_object19830:head# 2024-04-15T18:22:17.593 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28b964e6:::benchmark_data_smithi159_102124_object34757:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28bb8e07:::benchmark_data_smithi159_102124_object55162:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28bc0d76:::benchmark_data_smithi159_102124_object19803:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28bc80f6:::benchmark_data_smithi159_102124_object5441:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28bcf45a:::benchmark_data_smithi159_102124_object19552:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28bd6daa:::benchmark_data_smithi159_102124_object40475:head# 2024-04-15T18:22:17.594 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28be9b66:::benchmark_data_smithi159_102124_object9713:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c1cbce:::benchmark_data_smithi159_102124_object53904:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c35a2d:::benchmark_data_smithi159_102124_object18299:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c44d4d:::benchmark_data_smithi159_102124_object1850:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c60d89:::benchmark_data_smithi159_102124_object46911:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c65422:::benchmark_data_smithi159_102124_object42219:head# 2024-04-15T18:22:17.595 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c7a6ab:::benchmark_data_smithi159_102124_object6495:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c7b479:::benchmark_data_smithi159_102124_object47927:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28c7b984:::benchmark_data_smithi159_102124_object26978:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28cb40fb:::benchmark_data_smithi159_102124_object27245:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28cc73a2:::benchmark_data_smithi159_102124_object53195:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28cde145:::benchmark_data_smithi159_102124_object24466:head# 2024-04-15T18:22:17.596 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28cf86d6:::benchmark_data_smithi159_102124_object32424:head# 2024-04-15T18:22:17.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d1957d:::benchmark_data_smithi159_102124_object10466:head# 2024-04-15T18:22:17.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d234af:::benchmark_data_smithi159_102124_object29367:head# 2024-04-15T18:22:17.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d3c058:::benchmark_data_smithi159_102124_object40039:head# 2024-04-15T18:22:17.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d40bbc:::benchmark_data_smithi159_102124_object18698:head# 2024-04-15T18:22:17.597 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d42c21:::benchmark_data_smithi159_102124_object771:head# 2024-04-15T18:22:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:17.611 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:17.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d42f98:::benchmark_data_smithi159_102124_object48948:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d51385:::benchmark_data_smithi159_102124_object12356:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d540a0:::benchmark_data_smithi159_102124_object38053:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d5a277:::benchmark_data_smithi159_102124_object7256:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d6e71c:::benchmark_data_smithi159_102124_object48107:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d7b436:::benchmark_data_smithi159_102124_object35532:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d819d1:::benchmark_data_smithi159_102124_object37666:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28d8e5b5:::benchmark_data_smithi159_102124_object23712:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28dab7d5:::benchmark_data_smithi159_102124_object16188:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28dd2d60:::benchmark_data_smithi159_102124_object51623:head# 2024-04-15T18:22:17.627 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28dd4ff8:::benchmark_data_smithi159_102124_object8189:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28de1165:::benchmark_data_smithi159_102124_object10493:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28de6012:::benchmark_data_smithi159_102124_object41299:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28df9699:::benchmark_data_smithi159_102124_object52457:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e2210a:::benchmark_data_smithi159_102124_object9424:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e3428d:::benchmark_data_smithi159_102124_object12969:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e43dbf:::benchmark_data_smithi159_102124_object46156:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e4b3c3:::benchmark_data_smithi159_102124_object55137:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e56421:::benchmark_data_smithi159_102124_object24303:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e57d53:::benchmark_data_smithi159_102124_object23923:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e5a754:::benchmark_data_smithi159_102124_object5246:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e5bfe7:::benchmark_data_smithi159_102124_object7702:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e61b7f:::benchmark_data_smithi159_102124_object4610:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e854b2:::benchmark_data_smithi159_102124_object27664:head# 2024-04-15T18:22:17.628 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e86cf7:::benchmark_data_smithi159_102124_object37397:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e91d70:::benchmark_data_smithi159_102124_object34638:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28e9e5fa:::benchmark_data_smithi159_102124_object43777:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ea5d9d:::benchmark_data_smithi159_102124_object10734:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ea7afc:::benchmark_data_smithi159_102124_object20797:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ee287e:::benchmark_data_smithi159_102124_object49959:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28eedd4b:::benchmark_data_smithi159_102124_object42774:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ef4b80:::benchmark_data_smithi159_102124_object48023:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f157a6:::benchmark_data_smithi159_102124_object23436:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f17920:::benchmark_data_smithi159_102124_object15487:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f2d23a:::benchmark_data_smithi159_102124_object55594:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f38320:::benchmark_data_smithi159_102124_object32217:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f3ee94:::benchmark_data_smithi159_102124_object5972:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f51469:::benchmark_data_smithi159_102124_object46095:head# 2024-04-15T18:22:17.629 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f577d5:::benchmark_data_smithi159_102124_object18161:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f5faa7:::benchmark_data_smithi159_102124_object5050:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28f646c0:::benchmark_data_smithi159_102124_object12426:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28fab7cd:::benchmark_data_smithi159_102124_object42024:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28fc9bcc:::benchmark_data_smithi159_102124_object48990:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:28ffc6f1:::benchmark_data_smithi159_102124_object14130:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2900d976:::benchmark_data_smithi159_102124_object7791:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2901e907:::benchmark_data_smithi159_102124_object49874:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2903f2e0:::benchmark_data_smithi159_102124_object29085:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290608dd:::benchmark_data_smithi159_102124_object13693:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2906d5b1:::benchmark_data_smithi159_102124_object31244:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29088614:::benchmark_data_smithi159_102124_object45254:head# 2024-04-15T18:22:17.630 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290a50ee:::benchmark_data_smithi159_102124_object39465:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290a817d:::benchmark_data_smithi159_102124_object46108:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290e1965:::benchmark_data_smithi159_102124_object27600:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290e81ee:::benchmark_data_smithi159_102124_object27365:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:290f624c:::benchmark_data_smithi159_102124_object38067:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29112543:::benchmark_data_smithi159_102124_object39951:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2911ebeb:::benchmark_data_smithi159_102124_object22291:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291269d8:::benchmark_data_smithi159_102124_object39219:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291473a5:::benchmark_data_smithi159_102124_object12042:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29150851:::benchmark_data_smithi159_102124_object26898:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291509ea:::benchmark_data_smithi159_102124_object6807:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291545cb:::benchmark_data_smithi159_102124_object41129:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29170503:::benchmark_data_smithi159_102124_object12971:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29176131:::benchmark_data_smithi159_102124_object31751:head# 2024-04-15T18:22:17.631 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2917ecc9:::benchmark_data_smithi159_102124_object3572:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2918aa87:::benchmark_data_smithi159_102124_object18211:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29192076:::benchmark_data_smithi159_102124_object16834:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291950f9:::benchmark_data_smithi159_102124_object51476:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291a2f72:::benchmark_data_smithi159_102124_object3564:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:291da2af:::benchmark_data_smithi159_102124_object25492:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29213861:::benchmark_data_smithi159_102124_object11681:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2921dd55:::benchmark_data_smithi159_102124_object43103:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2921ee4f:::benchmark_data_smithi159_102124_object36163:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2922921a:::benchmark_data_smithi159_102124_object2339:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2922b9d8:::benchmark_data_smithi159_102124_object45307:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29242d6e:::benchmark_data_smithi159_102124_object8582:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2924841b:::benchmark_data_smithi159_102124_object3273:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2924d3fb:::benchmark_data_smithi159_102124_object57924:head# 2024-04-15T18:22:17.632 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29258667:::benchmark_data_smithi159_102124_object52502:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292773b4:::benchmark_data_smithi159_102124_object58148:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29292d23:::benchmark_data_smithi159_102124_object40482:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292b2645:::benchmark_data_smithi159_102124_object54240:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292b480b:::benchmark_data_smithi159_102124_object15149:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292c7ae1:::benchmark_data_smithi159_102124_object51927:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292e0da3:::benchmark_data_smithi159_102124_object25231:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:292f1c8c:::benchmark_data_smithi159_102124_object47143:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29301c2b:::benchmark_data_smithi159_102124_object714:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:293092f3:::benchmark_data_smithi159_102124_object2961:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29324b67:::benchmark_data_smithi159_102124_object53019:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2934af95:::benchmark_data_smithi159_102124_object6833:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2934e750:::benchmark_data_smithi159_102124_object58362:head# 2024-04-15T18:22:17.633 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29350b09:::benchmark_data_smithi159_102124_object6502:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29378f71:::benchmark_data_smithi159_102124_object10329:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2937c043:::benchmark_data_smithi159_102124_object37101:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29389a63:::benchmark_data_smithi159_102124_object8479:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:293c476c:::benchmark_data_smithi159_102124_object3764:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:293d3bcf:::benchmark_data_smithi159_102124_object44706:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:293e9fdb:::benchmark_data_smithi159_102124_object33512:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:293eea1d:::benchmark_data_smithi159_102124_object8660:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29410e8b:::benchmark_data_smithi159_102124_object26788:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29414ceb:::benchmark_data_smithi159_102124_object38003:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29429887:::benchmark_data_smithi159_102124_object48905:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2943d102:::benchmark_data_smithi159_102124_object49489:head# 2024-04-15T18:22:17.634 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29450678:::benchmark_data_smithi159_102124_object24217:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294716e6:::benchmark_data_smithi159_102124_object3442:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29485bcc:::benchmark_data_smithi159_102124_object25253:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2948baad:::benchmark_data_smithi159_102124_object54517:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294978c8:::benchmark_data_smithi159_102124_object32588:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294b3ba4:::benchmark_data_smithi159_102124_object24097:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294b734c:::benchmark_data_smithi159_102124_object7995:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294d4844:::benchmark_data_smithi159_102124_object24961:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294de263:::benchmark_data_smithi159_102124_object54918:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294fa931:::benchmark_data_smithi159_102124_object13548:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:294fff30:::benchmark_data_smithi159_102124_object19594:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29511bae:::benchmark_data_smithi159_102124_object41215:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2953c256:::benchmark_data_smithi159_102124_object31508:head# 2024-04-15T18:22:17.635 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2953ce5b:::benchmark_data_smithi159_102124_object52726:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29550f78:::benchmark_data_smithi159_102124_object1314:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2956edd6:::benchmark_data_smithi159_102124_object30474:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:295adb9b:::benchmark_data_smithi159_102124_object19768:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:295ba06d:::benchmark_data_smithi159_102124_object50320:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:295d087e:::benchmark_data_smithi159_102124_object54737:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:295fdb5b:::benchmark_data_smithi159_102124_object55998:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29609070:::benchmark_data_smithi159_102124_object5248:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29618a54:::benchmark_data_smithi159_102124_object10866:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2962b9fa:::benchmark_data_smithi159_102124_object56998:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296645c8:::benchmark_data_smithi159_102124_object48865:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2967d8fc:::benchmark_data_smithi159_102124_object13445:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29694337:::benchmark_data_smithi159_102124_object21746:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296ad758:::benchmark_data_smithi159_102124_object18999:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296b6fc4:::benchmark_data_smithi159_102124_object5603:head# 2024-04-15T18:22:17.636 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296baf39:::benchmark_data_smithi159_102124_object4871:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296c24fc:::benchmark_data_smithi159_102124_object32598:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296ccd25:::benchmark_data_smithi159_102124_object48437:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296e5abc:::benchmark_data_smithi159_102124_object22992:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:296e83e5:::benchmark_data_smithi159_102124_object18157:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29713a28:::benchmark_data_smithi159_102124_object9831:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29717281:::benchmark_data_smithi159_102124_object51628:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2971be78:::benchmark_data_smithi159_102124_object42850:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2972d60e:::benchmark_data_smithi159_102124_object36403:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29730e7d:::benchmark_data_smithi159_102124_object19434:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29748bdd:::benchmark_data_smithi159_102124_object12774:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29757ece:::benchmark_data_smithi159_102124_object19469:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2975ba14:::benchmark_data_smithi159_102124_object46061:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2976add4:::benchmark_data_smithi159_102124_object12100:head# 2024-04-15T18:22:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297884b9:::benchmark_data_smithi159_102124_object41339:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2979a442:::benchmark_data_smithi159_102124_object13174:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297a6227:::benchmark_data_smithi159_102124_object30411:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297b99dd:::benchmark_data_smithi159_102124_object7425:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297bfa37:::benchmark_data_smithi159_102124_object1101:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297c04cd:::benchmark_data_smithi159_102124_object3727:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297c4a1e:::benchmark_data_smithi159_102124_object55373:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:297d7374:::benchmark_data_smithi159_102124_object4463:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2980289e:::benchmark_data_smithi159_102124_object56797:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2981c10b:::benchmark_data_smithi159_102124_object219:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298290db:::benchmark_data_smithi159_102124_object35079:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2982cbee:::benchmark_data_smithi159_102124_object26913:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298556bf:::benchmark_data_smithi159_102124_object28563:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2986c0c4:::benchmark_data_smithi159_102124_object13747:head# 2024-04-15T18:22:17.638 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298731c1:::benchmark_data_smithi159_102124_object9940:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298824e8:::benchmark_data_smithi159_102124_object8050:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29882f80:::benchmark_data_smithi159_102124_object49454:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2988321a:::benchmark_data_smithi159_102124_object2507:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298b7cb9:::benchmark_data_smithi159_102124_object34196:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298c57cf:::benchmark_data_smithi159_102124_object15368:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298cb614:::benchmark_data_smithi159_102124_object52951:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298e2f3c:::benchmark_data_smithi159_102124_object40961:head# 2024-04-15T18:22:17.639 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:298f66be:::benchmark_data_smithi159_102124_object40407:head# 2024-04-15T18:22:17.645 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2990e56c:::benchmark_data_smithi159_102124_object23195:head# 2024-04-15T18:22:17.645 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2992abb4:::benchmark_data_smithi159_102124_object41146:head# 2024-04-15T18:22:17.645 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2992f8f9:::benchmark_data_smithi159_102124_object26301:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:299734aa:::benchmark_data_smithi159_102124_object11130:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29973a9a:::benchmark_data_smithi159_102124_object8619:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29994a6c:::benchmark_data_smithi159_102124_object6288:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2999bfdd:::benchmark_data_smithi159_102124_object28644:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2999d446:::benchmark_data_smithi159_102124_object37324:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:299aaa4d:::benchmark_data_smithi159_102124_object11646:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:299bbe24:::benchmark_data_smithi159_102124_object6381:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:299d5207:::benchmark_data_smithi159_102124_object56192:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:299eae23:::benchmark_data_smithi159_102124_object26891:head# 2024-04-15T18:22:17.646 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a004ed:::benchmark_data_smithi159_102124_object21659:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a1030b:::benchmark_data_smithi159_102124_object17562:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a1f4a2:::benchmark_data_smithi159_102124_object47487:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a27731:::benchmark_data_smithi159_102124_object14793:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a321f4:::benchmark_data_smithi159_102124_object36448:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a3f001:::benchmark_data_smithi159_102124_object39042:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a4cf58:::benchmark_data_smithi159_102124_object35661:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a571fb:::benchmark_data_smithi159_102124_object52446:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a58c91:::benchmark_data_smithi159_102124_object33034:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a85d06:::benchmark_data_smithi159_102124_object26202:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a8b9e6:::benchmark_data_smithi159_102124_object19793:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29a99f58:::benchmark_data_smithi159_102124_object26268:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ae4fb1:::benchmark_data_smithi159_102124_object52711:head# 2024-04-15T18:22:17.647 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b35ae7:::benchmark_data_smithi159_102124_object57837:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b377f7:::benchmark_data_smithi159_102124_object55315:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b3f47d:::benchmark_data_smithi159_102124_object48943:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b45e6f:::benchmark_data_smithi159_102124_object47481:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b47a4b:::benchmark_data_smithi159_102124_object53346:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b49bd2:::benchmark_data_smithi159_102124_object36278:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b4c73d:::benchmark_data_smithi159_102124_object3423:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b52756:::benchmark_data_smithi159_102124_object55535:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b588cd:::benchmark_data_smithi159_102124_object10420:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b5e172:::benchmark_data_smithi159_102124_object27040:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b627bd:::benchmark_data_smithi159_102124_object25273:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29b90d16:::benchmark_data_smithi159_102124_object15156:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ba7521:::benchmark_data_smithi159_102124_object21201:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bc11ed:::benchmark_data_smithi159_102124_object52874:head# 2024-04-15T18:22:17.648 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bd401c:::benchmark_data_smithi159_102124_object27398:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bd40f8:::benchmark_data_smithi159_102124_object52114:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bd58a3:::benchmark_data_smithi159_102124_object123:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bd72ab:::benchmark_data_smithi159_102124_object45320:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bdadf6:::benchmark_data_smithi159_102124_object17725:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29bef1d3:::benchmark_data_smithi159_102124_object45343:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c04b7f:::benchmark_data_smithi159_102124_object21842:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c0aece:::benchmark_data_smithi159_102124_object35262:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c13e24:::benchmark_data_smithi159_102124_object25689:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c31b4f:::benchmark_data_smithi159_102124_object40435:head# 2024-04-15T18:22:17.649 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c3d5b5:::benchmark_data_smithi159_102124_object10126:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c4f012:::benchmark_data_smithi159_102124_object12172:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c5ab5c:::benchmark_data_smithi159_102124_object47640:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c6e229:::benchmark_data_smithi159_102124_object57812:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29c9243a:::benchmark_data_smithi159_102124_object11666:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ca76a5:::benchmark_data_smithi159_102124_object16489:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29cdc663:::benchmark_data_smithi159_102124_object35861:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29cdece9:::benchmark_data_smithi159_102124_object24993:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ce0471:::benchmark_data_smithi159_102124_object50255:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ce441b:::benchmark_data_smithi159_102124_object50675:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29cefe4c:::benchmark_data_smithi159_102124_object27806:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29cf2ff8:::benchmark_data_smithi159_102124_object4727:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29cfbbb6:::benchmark_data_smithi159_102124_object20960:head# 2024-04-15T18:22:17.650 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29d00218:::benchmark_data_smithi159_102124_object9190:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29d318c9:::benchmark_data_smithi159_102124_object30498:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29d4919a:::benchmark_data_smithi159_102124_object732:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29d70c1e:::benchmark_data_smithi159_102124_object25155:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29d7ee73:::benchmark_data_smithi159_102124_object37315:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29da06f4:::benchmark_data_smithi159_102124_object34651:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29dda44c:::benchmark_data_smithi159_102124_object14879:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ddc04f:::benchmark_data_smithi159_102124_object6950:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29de9d12:::benchmark_data_smithi159_102124_object20038:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29deb49d:::benchmark_data_smithi159_102124_object14017:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29defe21:::benchmark_data_smithi159_102124_object35641:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29df806a:::benchmark_data_smithi159_102124_object54120:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e1ccc1:::benchmark_data_smithi159_102124_object745:head# 2024-04-15T18:22:17.651 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e1eb78:::benchmark_data_smithi159_102124_object53642:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e25d2a:::benchmark_data_smithi159_102124_object14584:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e55a45:::benchmark_data_smithi159_102124_object20218:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e684d4:::benchmark_data_smithi159_102124_object7210:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29e9f9e4:::benchmark_data_smithi159_102124_object29230:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ea7f39:::benchmark_data_smithi159_102124_object47736:head# 2024-04-15T18:22:17.652 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ecfb77:::benchmark_data_smithi159_102124_object38629:head# 2024-04-15T18:22:17.658 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ed4db8:::benchmark_data_smithi159_102124_object50103:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29eed757:::benchmark_data_smithi159_102124_object57541:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f221cf:::benchmark_data_smithi159_102124_object19640:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f52161:::benchmark_data_smithi159_102124_object25746:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f5c898:::benchmark_data_smithi159_102124_object52412:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f5d320:::benchmark_data_smithi159_102124_object240:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f88fc3:::benchmark_data_smithi159_102124_object51391:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f8ad9b:::benchmark_data_smithi159_102124_object55803:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29f93092:::benchmark_data_smithi159_102124_object38957:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29fa58fd:::benchmark_data_smithi159_102124_object31053:head# 2024-04-15T18:22:17.659 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29fa763b:::benchmark_data_smithi159_102124_object16853:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29fab3e6:::benchmark_data_smithi159_102124_object22584:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29fd2404:::benchmark_data_smithi159_102124_object37674:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:29ff967f:::benchmark_data_smithi159_102124_object36541:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a011332:::benchmark_data_smithi159_102124_object39341:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a02ac1f:::benchmark_data_smithi159_102124_object52772:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0349d5:::benchmark_data_smithi159_102124_object36896:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0383bf:::benchmark_data_smithi159_102124_object7879:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0386b8:::benchmark_data_smithi159_102124_object7852:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a041957:::benchmark_data_smithi159_102124_object5597:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a04bdce:::benchmark_data_smithi159_102124_object36387:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a055f4b:::benchmark_data_smithi159_102124_object8530:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0cbbce:::benchmark_data_smithi159_102124_object50901:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0cfd83:::benchmark_data_smithi159_102124_object29091:head# 2024-04-15T18:22:17.660 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a0fc756:::benchmark_data_smithi159_102124_object40490:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1040d5:::benchmark_data_smithi159_102124_object48208:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a114e3f:::benchmark_data_smithi159_102124_object14104:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a12fcea:::benchmark_data_smithi159_102124_object13061:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1302a1:::benchmark_data_smithi159_102124_object11479:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1431ee:::benchmark_data_smithi159_102124_object26772:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a149915:::benchmark_data_smithi159_102124_object48652:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a14e08f:::benchmark_data_smithi159_102124_object19119:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a15a3c9:::benchmark_data_smithi159_102124_object15788:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a163f70:::benchmark_data_smithi159_102124_object39637:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a16d09b:::benchmark_data_smithi159_102124_object1903:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1711f9:::benchmark_data_smithi159_102124_object28307:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a17acd4:::benchmark_data_smithi159_102124_object36116:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a19abbf:::benchmark_data_smithi159_102124_object25073:head# 2024-04-15T18:22:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1a67c5:::benchmark_data_smithi159_102124_object13257:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1ac8b0:::benchmark_data_smithi159_102124_object29675:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1ae91a:::benchmark_data_smithi159_102124_object43587:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1b3614:::benchmark_data_smithi159_102124_object18406:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1c355d:::benchmark_data_smithi159_102124_object49598:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1c6595:::benchmark_data_smithi159_102124_object57110:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1ddaf3:::benchmark_data_smithi159_102124_object37839:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1e32ec:::benchmark_data_smithi159_102124_object39277:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a1f4124:::benchmark_data_smithi159_102124_object17758:head# 2024-04-15T18:22:17.662 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a2142c3:::benchmark_data_smithi159_102124_object49631:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a223801:::benchmark_data_smithi159_102124_object51599:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a243356:::benchmark_data_smithi159_102124_object13346:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a24dde6:::benchmark_data_smithi159_102124_object52671:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a258f49:::benchmark_data_smithi159_102124_object14085:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a259f24:::benchmark_data_smithi159_102124_object15972:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a26e0a1:::benchmark_data_smithi159_102124_object13429:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a27bc86:::benchmark_data_smithi159_102124_object5644:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a2c47b0:::benchmark_data_smithi159_102124_object34046:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a2cedf6:::benchmark_data_smithi159_102124_object382:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a2dbf57:::benchmark_data_smithi159_102124_object39904:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a318388:::benchmark_data_smithi159_102124_object32214:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a32d443:::benchmark_data_smithi159_102124_object14738:head# 2024-04-15T18:22:17.663 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a34a73b:::benchmark_data_smithi159_102124_object25896:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a35bcff:::benchmark_data_smithi159_102124_object7315:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a35e879:::benchmark_data_smithi159_102124_object46150:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a369a55:::benchmark_data_smithi159_102124_object32461:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a39acb9:::benchmark_data_smithi159_102124_object14295:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a3a0818:::benchmark_data_smithi159_102124_object15294:head# 2024-04-15T18:22:17.664 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a3ad63a:::benchmark_data_smithi159_102124_object28582:head# 2024-04-15T18:22:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a3afaae:::benchmark_data_smithi159_102124_object29964:head# 2024-04-15T18:22:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a3e0ea6:::benchmark_data_smithi159_102124_object48224:head# 2024-04-15T18:22:17.673 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a3f61e2:::benchmark_data_smithi159_102124_object6676:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a40c377:::benchmark_data_smithi159_102124_object55885:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a42c9b6:::benchmark_data_smithi159_102124_object42883:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a45d502:::benchmark_data_smithi159_102124_object33421:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a461f0d:::benchmark_data_smithi159_102124_object32944:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a462c58:::benchmark_data_smithi159_102124_object27269:head# 2024-04-15T18:22:17.674 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a46cda8:::benchmark_data_smithi159_102124_object218:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4708ba:::benchmark_data_smithi159_102124_object15247:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4a6f6e:::benchmark_data_smithi159_102124_object1334:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4b9937:::benchmark_data_smithi159_102124_object32737:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4ba8c9:::benchmark_data_smithi159_102124_object15758:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4be079:::benchmark_data_smithi159_102124_object40309:head# 2024-04-15T18:22:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4c058d:::benchmark_data_smithi159_102124_object17367:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4d1aa6:::benchmark_data_smithi159_102124_object41423:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a4fba35:::benchmark_data_smithi159_102124_object1753:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5042ae:::benchmark_data_smithi159_102124_object36940:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5057b0:::benchmark_data_smithi159_102124_object11830:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5058bd:::benchmark_data_smithi159_102124_object25032:head# 2024-04-15T18:22:17.676 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a520868:::benchmark_data_smithi159_102124_object56253:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a53b99e:::benchmark_data_smithi159_102124_object40582:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a54bef3:::benchmark_data_smithi159_102124_object878:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a564b36:::benchmark_data_smithi159_102124_object35311:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5673a4:::benchmark_data_smithi159_102124_object36056:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a578652:::benchmark_data_smithi159_102124_object54423:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a58d642:::benchmark_data_smithi159_102124_object25190:head# 2024-04-15T18:22:17.677 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5940aa:::benchmark_data_smithi159_102124_object36809:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a595d85:::benchmark_data_smithi159_102124_object52729:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a596a29:::benchmark_data_smithi159_102124_object2049:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5a0740:::benchmark_data_smithi159_102124_object39388:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5a1891:::benchmark_data_smithi159_102124_object55992:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5ae8dc:::benchmark_data_smithi159_102124_object48691:head# 2024-04-15T18:22:17.678 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5c32db:::benchmark_data_smithi159_102124_object17589:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5f2472:::benchmark_data_smithi159_102124_object29523:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5f9606:::benchmark_data_smithi159_102124_object54314:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a5fd56a:::benchmark_data_smithi159_102124_object11311:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a614e7e:::benchmark_data_smithi159_102124_object50548:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a615edf:::benchmark_data_smithi159_102124_object19555:head# 2024-04-15T18:22:17.679 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a618ee2:::benchmark_data_smithi159_102124_object42685:head# 2024-04-15T18:22:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a61bc56:::benchmark_data_smithi159_102124_object24990:head# 2024-04-15T18:22:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6426db:::benchmark_data_smithi159_102124_object55773:head# 2024-04-15T18:22:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a670722:::benchmark_data_smithi159_102124_object33457:head# 2024-04-15T18:22:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6efd1d:::benchmark_data_smithi159_102124_object9259:head# 2024-04-15T18:22:17.680 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6f4906:::benchmark_data_smithi159_102124_object2891:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6f9ee1:::benchmark_data_smithi159_102124_object31901:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6fa41d:::benchmark_data_smithi159_102124_object47298:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a6fdeab:::benchmark_data_smithi159_102124_object49499:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a70b0a4:::benchmark_data_smithi159_102124_object52908:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a714cdc:::benchmark_data_smithi159_102124_object57443:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a71f6a5:::benchmark_data_smithi159_102124_object30313:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a75b475:::benchmark_data_smithi159_102124_object5223:head# 2024-04-15T18:22:17.681 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a764c83:::benchmark_data_smithi159_102124_object32552:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a76cb9b:::benchmark_data_smithi159_102124_object12557:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a780740:::benchmark_data_smithi159_102124_object37738:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a780d3e:::benchmark_data_smithi159_102124_object51664:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a79cfa2:::benchmark_data_smithi159_102124_object23276:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a7cadc4:::benchmark_data_smithi159_102124_object20931:head# 2024-04-15T18:22:17.682 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a7d4f5d:::benchmark_data_smithi159_102124_object41079:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a7dd419:::benchmark_data_smithi159_102124_object26549:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a7eaa90:::benchmark_data_smithi159_102124_object11350:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8050e6:::benchmark_data_smithi159_102124_object47565:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a82e7ce:::benchmark_data_smithi159_102124_object5412:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8582cb:::benchmark_data_smithi159_102124_object12078:head# 2024-04-15T18:22:17.683 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a874abb:::benchmark_data_smithi159_102124_object9340:head# 2024-04-15T18:22:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8a9352:::benchmark_data_smithi159_102124_object20914:head# 2024-04-15T18:22:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8b857d:::benchmark_data_smithi159_102124_object36671:head# 2024-04-15T18:22:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8d2347:::benchmark_data_smithi159_102124_object18444:head# 2024-04-15T18:22:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8f27a8:::benchmark_data_smithi159_102124_object13513:head# 2024-04-15T18:22:17.709 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a8f7476:::benchmark_data_smithi159_102124_object32224:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a91ce0a:::benchmark_data_smithi159_102124_object9859:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a91e010:::benchmark_data_smithi159_102124_object33704:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a94ee45:::benchmark_data_smithi159_102124_object37061:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a971cac:::benchmark_data_smithi159_102124_object49470:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2a9b0702:::benchmark_data_smithi159_102124_object53541:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa2d9b5:::benchmark_data_smithi159_102124_object40233:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa3a278:::benchmark_data_smithi159_102124_object6941:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa40d40:::benchmark_data_smithi159_102124_object43369:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa4dcd2:::benchmark_data_smithi159_102124_object4995:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa5af50:::benchmark_data_smithi159_102124_object14140:head# 2024-04-15T18:22:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa61948:::benchmark_data_smithi159_102124_object55555:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa7c1bb:::benchmark_data_smithi159_102124_object9216:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aa98635:::benchmark_data_smithi159_102124_object32558:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aacdbeb:::benchmark_data_smithi159_102124_object34502:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aad8df4:::benchmark_data_smithi159_102124_object49301:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aae93de:::benchmark_data_smithi159_102124_object50869:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aaf1faa:::benchmark_data_smithi159_102124_object36345:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab192db:::benchmark_data_smithi159_102124_object24781:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab31a93:::benchmark_data_smithi159_102124_object2098:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab394ef:::benchmark_data_smithi159_102124_object34831:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab41b1b:::benchmark_data_smithi159_102124_object34407:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab4dcee:::benchmark_data_smithi159_102124_object34627:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab5a92f:::benchmark_data_smithi159_102124_object55457:head# 2024-04-15T18:22:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab80dab:::benchmark_data_smithi159_102124_object53140:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ab85767:::benchmark_data_smithi159_102124_object10900:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2abaed09:::benchmark_data_smithi159_102124_object1261:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2abd88eb:::benchmark_data_smithi159_102124_object13968:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2abf4e7f:::benchmark_data_smithi159_102124_object34887:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ac1901a:::benchmark_data_smithi159_102124_object22766:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ac1a7ab:::benchmark_data_smithi159_102124_object21959:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ac22792:::benchmark_data_smithi159_102124_object34460:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ac4d593:::benchmark_data_smithi159_102124_object45136:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aca46ec:::benchmark_data_smithi159_102124_object46495:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2acdbaea:::benchmark_data_smithi159_102124_object49725:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2acf4aca:::benchmark_data_smithi159_102124_object27054:head# 2024-04-15T18:22:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad00546:::benchmark_data_smithi159_102124_object24783:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad0b16c:::benchmark_data_smithi159_102124_object26928:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad34f04:::benchmark_data_smithi159_102124_object18276:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad54a20:::benchmark_data_smithi159_102124_object13756:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad6d616:::benchmark_data_smithi159_102124_object17863:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad8d686:::benchmark_data_smithi159_102124_object15885:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad8d830:::benchmark_data_smithi159_102124_object5499:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ad921ec:::benchmark_data_smithi159_102124_object45444:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2adade22:::benchmark_data_smithi159_102124_object11842:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2adb41f5:::benchmark_data_smithi159_102124_object43263:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2adb6e61:::benchmark_data_smithi159_102124_object54029:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2adc43de:::benchmark_data_smithi159_102124_object40069:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2add7d37:::benchmark_data_smithi159_102124_object53371:head# 2024-04-15T18:22:17.713 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ade7616:::benchmark_data_smithi159_102124_object47242:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2adfc4ca:::benchmark_data_smithi159_102124_object50700:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae1f355:::benchmark_data_smithi159_102124_object41094:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae23e6a:::benchmark_data_smithi159_102124_object18261:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae30fc3:::benchmark_data_smithi159_102124_object52050:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae5c39d:::benchmark_data_smithi159_102124_object51634:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae741be:::benchmark_data_smithi159_102124_object35551:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ae947a5:::benchmark_data_smithi159_102124_object8377:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aea876c:::benchmark_data_smithi159_102124_object40816:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aed2c26:::benchmark_data_smithi159_102124_object4778:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aeef69a:::benchmark_data_smithi159_102124_object42288:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2aef1513:::benchmark_data_smithi159_102124_object15220:head# 2024-04-15T18:22:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af03667:::benchmark_data_smithi159_102124_object13571:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af26222:::benchmark_data_smithi159_102124_object26324:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af2fda8:::benchmark_data_smithi159_102124_object47910:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af39c44:::benchmark_data_smithi159_102124_object44571:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af40b00:::benchmark_data_smithi159_102124_object25917:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af41ddf:::benchmark_data_smithi159_102124_object57327:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af461d8:::benchmark_data_smithi159_102124_object30623:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af85a53:::benchmark_data_smithi159_102124_object3157:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af96443:::benchmark_data_smithi159_102124_object23505:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af9774e:::benchmark_data_smithi159_102124_object9022:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af9ba50:::benchmark_data_smithi159_102124_object11730:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2af9d04e:::benchmark_data_smithi159_102124_object24133:head# 2024-04-15T18:22:17.715 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2afa2104:::benchmark_data_smithi159_102124_object17782:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2afbbda8:::benchmark_data_smithi159_102124_object9575:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2afc78d8:::benchmark_data_smithi159_102124_object47856:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2afec6d0:::benchmark_data_smithi159_102124_object32415:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b01d215:::benchmark_data_smithi159_102124_object44318:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0293b8:::benchmark_data_smithi159_102124_object30693:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b077384:::benchmark_data_smithi159_102124_object17972:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b07c639:::benchmark_data_smithi159_102124_object18043:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b08bcb6:::benchmark_data_smithi159_102124_object42988:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b09c394:::benchmark_data_smithi159_102124_object52410:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0b3b4f:::benchmark_data_smithi159_102124_object18735:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0cd36d:::benchmark_data_smithi159_102124_object38517:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0e9f55:::benchmark_data_smithi159_102124_object45976:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0ecd12:::benchmark_data_smithi159_102124_object19896:head# 2024-04-15T18:22:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b0fa1b3:::benchmark_data_smithi159_102124_object855:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b104eb7:::benchmark_data_smithi159_102124_object6327:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b107d97:::benchmark_data_smithi159_102124_object28252:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b140fc6:::benchmark_data_smithi159_102124_object57772:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b14ff30:::benchmark_data_smithi159_102124_object44717:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b16c5a1:::benchmark_data_smithi159_102124_object38999:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b18c370:::benchmark_data_smithi159_102124_object40631:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b18f6f4:::benchmark_data_smithi159_102124_object15661:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1a751c:::benchmark_data_smithi159_102124_object55101:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1a89e5:::benchmark_data_smithi159_102124_object16452:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1aba86:::benchmark_data_smithi159_102124_object841:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1b0911:::benchmark_data_smithi159_102124_object55920:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1c4aa5:::benchmark_data_smithi159_102124_object35902:head# 2024-04-15T18:22:17.717 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1c7b9d:::benchmark_data_smithi159_102124_object10:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1cee15:::benchmark_data_smithi159_102124_object39414:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b1e05d1:::benchmark_data_smithi159_102124_object55677:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b20542c:::benchmark_data_smithi159_102124_object6938:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b20b40f:::benchmark_data_smithi159_102124_object49789:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b21bb80:::benchmark_data_smithi159_102124_object6431:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b239e0f:::benchmark_data_smithi159_102124_object16390:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b260fde:::benchmark_data_smithi159_102124_object35116:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b266861:::benchmark_data_smithi159_102124_object5323:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b27a878:::benchmark_data_smithi159_102124_object15074:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b283ab6:::benchmark_data_smithi159_102124_object47251:head# 2024-04-15T18:22:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b292160:::benchmark_data_smithi159_102124_object25014:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b29dc3d:::benchmark_data_smithi159_102124_object11776:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b2c6c07:::benchmark_data_smithi159_102124_object9845:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b2e90eb:::benchmark_data_smithi159_102124_object25996:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b311f7b:::benchmark_data_smithi159_102124_object11336:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3176c6:::benchmark_data_smithi159_102124_object6174:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b32b234:::benchmark_data_smithi159_102124_object23809:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b32cd58:::benchmark_data_smithi159_102124_object44936:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b340ae4:::benchmark_data_smithi159_102124_object47663:head# 2024-04-15T18:22:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b34fd5e:::benchmark_data_smithi159_102124_object26383:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b352c07:::benchmark_data_smithi159_102124_object32437:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b36b00c:::benchmark_data_smithi159_102124_object52399:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b397bdd:::benchmark_data_smithi159_102124_object57823:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3aab73:::benchmark_data_smithi159_102124_object7889:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3acb80:::benchmark_data_smithi159_102124_object6868:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3b4d68:::benchmark_data_smithi159_102124_object56098:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3b9351:::benchmark_data_smithi159_102124_object18448:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3d3572:::benchmark_data_smithi159_102124_object2328:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3d49d0:::benchmark_data_smithi159_102124_object13896:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b3f6d7e:::benchmark_data_smithi159_102124_object39629:head# 2024-04-15T18:22:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b40e0e1:::benchmark_data_smithi159_102124_object49764:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b413e4c:::benchmark_data_smithi159_102124_object2913:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b46ddd6:::benchmark_data_smithi159_102124_object41050:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b46f26d:::benchmark_data_smithi159_102124_object48697:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b47b4e2:::benchmark_data_smithi159_102124_object3549:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4ba63e:::benchmark_data_smithi159_102124_object52871:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4c0bba:::benchmark_data_smithi159_102124_object47424:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4c50e2:::benchmark_data_smithi159_102124_object36098:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4c74d2:::benchmark_data_smithi159_102124_object47753:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4ecc93:::benchmark_data_smithi159_102124_object2723:head# 2024-04-15T18:22:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b4f7e10:::benchmark_data_smithi159_102124_object43222:head# 2024-04-15T18:22:17.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b504d1c:::benchmark_data_smithi159_102124_object3188:head# 2024-04-15T18:22:17.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b506f2f:::benchmark_data_smithi159_102124_object36469:head# 2024-04-15T18:22:17.722 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b51cc6a:::benchmark_data_smithi159_102124_object32244:head# 2024-04-15T18:22:17.722 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b52212a:::benchmark_data_smithi159_102124_object14492:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b54e88a:::benchmark_data_smithi159_102124_object32900:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b55d42f:::benchmark_data_smithi159_102124_object133:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b57ead3:::benchmark_data_smithi159_102124_object20648:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b584a56:::benchmark_data_smithi159_102124_object35211:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b58fa1d:::benchmark_data_smithi159_102124_object4711:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b59b37d:::benchmark_data_smithi159_102124_object9078:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5a2c15:::benchmark_data_smithi159_102124_object34387:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5ae70d:::benchmark_data_smithi159_102124_object49042:head# 2024-04-15T18:22:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5b6bcd:::benchmark_data_smithi159_102124_object49722:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5e7564:::benchmark_data_smithi159_102124_object17268:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5e8168:::benchmark_data_smithi159_102124_object32145:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b5f1c2f:::benchmark_data_smithi159_102124_object42387:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b602335:::benchmark_data_smithi159_102124_object55089:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b60ad51:::benchmark_data_smithi159_102124_object6769:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b617b83:::benchmark_data_smithi159_102124_object28443:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b62855c:::benchmark_data_smithi159_102124_object41691:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b62ef58:::benchmark_data_smithi159_102124_object16556:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b640085:::benchmark_data_smithi159_102124_object22804:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b650441:::benchmark_data_smithi159_102124_object19864:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b66eb65:::benchmark_data_smithi159_102124_object13436:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b67ae66:::benchmark_data_smithi159_102124_object13844:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6917da:::benchmark_data_smithi159_102124_object36038:head# 2024-04-15T18:22:17.753 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b69c64b:::benchmark_data_smithi159_102124_object8892:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b69cfe4:::benchmark_data_smithi159_102124_object5697:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b69e141:::benchmark_data_smithi159_102124_object46276:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6a8bce:::benchmark_data_smithi159_102124_object11563:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6b5d84:::benchmark_data_smithi159_102124_object50112:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6d8b31:::benchmark_data_smithi159_102124_object35772:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6e95ea:::benchmark_data_smithi159_102124_object10843:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b6ef8fb:::benchmark_data_smithi159_102124_object45260:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b717f53:::benchmark_data_smithi159_102124_object17236:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b71b694:::benchmark_data_smithi159_102124_object3671:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b71d8bb:::benchmark_data_smithi159_102124_object14751:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b71ea9c:::benchmark_data_smithi159_102124_object40607:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b737566:::benchmark_data_smithi159_102124_object6210:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b73e814:::benchmark_data_smithi159_102124_object35735:head# 2024-04-15T18:22:17.754 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b73ebfe:::benchmark_data_smithi159_102124_object48634:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b76225c:::benchmark_data_smithi159_102124_object42047:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b763015:::benchmark_data_smithi159_102124_object38822:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b78dff3:::benchmark_data_smithi159_102124_object21207:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b7a5347:::benchmark_data_smithi159_102124_object9111:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b7a986b:::benchmark_data_smithi159_102124_object6448:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b7d15f2:::benchmark_data_smithi159_102124_object17503:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b7e5c36:::benchmark_data_smithi159_102124_object14868:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b7f1ba9:::benchmark_data_smithi159_102124_object48635:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b801e91:::benchmark_data_smithi159_102124_object35664:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b80dd77:::benchmark_data_smithi159_102124_object47334:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b81110e:::benchmark_data_smithi159_102124_object14786:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b8459d5:::benchmark_data_smithi159_102124_object19524:head# 2024-04-15T18:22:17.755 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b880430:::benchmark_data_smithi159_102124_object45800:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b88b966:::benchmark_data_smithi159_102124_object29031:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b897b1b:::benchmark_data_smithi159_102124_object22275:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b89d914:::benchmark_data_smithi159_102124_object53234:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b8b68f4:::benchmark_data_smithi159_102124_object6064:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b8b75f3:::benchmark_data_smithi159_102124_object45780:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b8dc708:::benchmark_data_smithi159_102124_object23741:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b8fe340:::benchmark_data_smithi159_102124_object14281:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b913374:::benchmark_data_smithi159_102124_object43768:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b918a44:::benchmark_data_smithi159_102124_object10781:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b926f28:::benchmark_data_smithi159_102124_object55697:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b949390:::benchmark_data_smithi159_102124_object51068:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b969718:::benchmark_data_smithi159_102124_object55393:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b98527d:::benchmark_data_smithi159_102124_object30484:head# 2024-04-15T18:22:17.756 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9aef4c:::benchmark_data_smithi159_102124_object8857:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9bdb19:::benchmark_data_smithi159_102124_object56057:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9cb646:::benchmark_data_smithi159_102124_object10853:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9cee68:::benchmark_data_smithi159_102124_object37309:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9cfba6:::benchmark_data_smithi159_102124_object34074:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9d7770:::benchmark_data_smithi159_102124_object21138:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9e58f6:::benchmark_data_smithi159_102124_object43807:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2b9f0cbe:::benchmark_data_smithi159_102124_object11962:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba0785b:::benchmark_data_smithi159_102124_object37468:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba07996:::benchmark_data_smithi159_102124_object37042:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba07b7b:::benchmark_data_smithi159_102124_object44841:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba1d493:::benchmark_data_smithi159_102124_object27877:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba1ecbe:::benchmark_data_smithi159_102124_object54012:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba2a8be:::benchmark_data_smithi159_102124_object29184:head# 2024-04-15T18:22:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba3955c:::benchmark_data_smithi159_102124_object3037:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba3f0ad:::benchmark_data_smithi159_102124_object52500:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba48627:::benchmark_data_smithi159_102124_object11531:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba5dcce:::benchmark_data_smithi159_102124_object14489:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba65bc6:::benchmark_data_smithi159_102124_object45766:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba6f703:::benchmark_data_smithi159_102124_object30410:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba7034d:::benchmark_data_smithi159_102124_object49850:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ba883dc:::benchmark_data_smithi159_102124_object37098:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2baaef77:::benchmark_data_smithi159_102124_object39762:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2babb627:::benchmark_data_smithi159_102124_object38433:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bac54fb:::benchmark_data_smithi159_102124_object28994:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2badf088:::benchmark_data_smithi159_102124_object57625:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bae46a6:::benchmark_data_smithi159_102124_object33794:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb1130b:::benchmark_data_smithi159_102124_object40335:head# 2024-04-15T18:22:17.758 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb229e0:::benchmark_data_smithi159_102124_object41899:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb2557f:::benchmark_data_smithi159_102124_object6215:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb2edce:::benchmark_data_smithi159_102124_object18797:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb3343f:::benchmark_data_smithi159_102124_object10911:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb3ec4d:::benchmark_data_smithi159_102124_object1089:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb55c5c:::benchmark_data_smithi159_102124_object15092:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb56f47:::benchmark_data_smithi159_102124_object17650:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb7acaa:::benchmark_data_smithi159_102124_object48105:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb81552:::benchmark_data_smithi159_102124_object25576:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb87f7f:::benchmark_data_smithi159_102124_object17327:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb8fdfa:::benchmark_data_smithi159_102124_object21618:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb9478f:::benchmark_data_smithi159_102124_object18889:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bb9c265:::benchmark_data_smithi159_102124_object47841:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbb40b1:::benchmark_data_smithi159_102124_object582:head# 2024-04-15T18:22:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbb7a15:::benchmark_data_smithi159_102124_object58188:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbc0ff6:::benchmark_data_smithi159_102124_object13338:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbe09a7:::benchmark_data_smithi159_102124_object48009:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbe1149:::benchmark_data_smithi159_102124_object5085:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbe3a82:::benchmark_data_smithi159_102124_object10471:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbe7e17:::benchmark_data_smithi159_102124_object50054:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbe7ed1:::benchmark_data_smithi159_102124_object24924:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbed100:::benchmark_data_smithi159_102124_object13421:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bbfcad8:::benchmark_data_smithi159_102124_object49409:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc16117:::benchmark_data_smithi159_102124_object1004:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc165f3:::benchmark_data_smithi159_102124_object28933:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc3f45c:::benchmark_data_smithi159_102124_object44286:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc52b99:::benchmark_data_smithi159_102124_object50074:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc54062:::benchmark_data_smithi159_102124_object26850:head# 2024-04-15T18:22:17.760 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bc7a4fd:::benchmark_data_smithi159_102124_object40399:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bcab63a:::benchmark_data_smithi159_102124_object42045:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bcaf92b:::benchmark_data_smithi159_102124_object46609:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bcc556d:::benchmark_data_smithi159_102124_object1820:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bce104b:::benchmark_data_smithi159_102124_object46744:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bcee70b:::benchmark_data_smithi159_102124_object16740:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd2c80f:::benchmark_data_smithi159_102124_object53873:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd3c296:::benchmark_data_smithi159_102124_object31334:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd62d64:::benchmark_data_smithi159_102124_object19196:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd666d8:::benchmark_data_smithi159_102124_object23729:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd6c4bf:::benchmark_data_smithi159_102124_object28072:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd8091e:::benchmark_data_smithi159_102124_object37574:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd99b6e:::benchmark_data_smithi159_102124_object4114:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd9a016:::benchmark_data_smithi159_102124_object54007:head# 2024-04-15T18:22:17.761 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bd9ecb2:::benchmark_data_smithi159_102124_object57401:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bdb1722:::benchmark_data_smithi159_102124_object57386:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bdd27aa:::benchmark_data_smithi159_102124_object44637:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bddf287:::benchmark_data_smithi159_102124_object54934:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bde0188:::benchmark_data_smithi159_102124_object36433:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bdf5c41:::benchmark_data_smithi159_102124_object22392:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be07e57:::benchmark_data_smithi159_102124_object24733:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be0d078:::benchmark_data_smithi159_102124_object26270:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be1c426:::benchmark_data_smithi159_102124_object23260:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be54003:::benchmark_data_smithi159_102124_object11624:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be62614:::benchmark_data_smithi159_102124_object51422:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be66ae4:::benchmark_data_smithi159_102124_object49701:head# 2024-04-15T18:22:17.762 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be6cc22:::benchmark_data_smithi159_102124_object3552:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be73411:::benchmark_data_smithi159_102124_object335:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be8d40d:::benchmark_data_smithi159_102124_object54489:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be8e310:::benchmark_data_smithi159_102124_object1838:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be92522:::benchmark_data_smithi159_102124_object45186:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2be946a7:::benchmark_data_smithi159_102124_object26691:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2beb5cf1:::benchmark_data_smithi159_102124_object58006:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bebe79b:::benchmark_data_smithi159_102124_object23404:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bed016d:::benchmark_data_smithi159_102124_object21729:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bed54d7:::benchmark_data_smithi159_102124_object9900:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf0c5a5:::benchmark_data_smithi159_102124_object41543:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf0dffb:::benchmark_data_smithi159_102124_object12836:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf108ef:::benchmark_data_smithi159_102124_object14153:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf15f2f:::benchmark_data_smithi159_102124_object13879:head# 2024-04-15T18:22:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf1f891:::benchmark_data_smithi159_102124_object46667:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf267aa:::benchmark_data_smithi159_102124_object52652:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf4181b:::benchmark_data_smithi159_102124_object26291:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf73d6a:::benchmark_data_smithi159_102124_object20766:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf75a66:::benchmark_data_smithi159_102124_object7498:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bf9dd2b:::benchmark_data_smithi159_102124_object22835:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bfa7e4d:::benchmark_data_smithi159_102124_object18031:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bfb5f2f:::benchmark_data_smithi159_102124_object8007:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bfdb7ed:::benchmark_data_smithi159_102124_object3954:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bfdf58a:::benchmark_data_smithi159_102124_object5653:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bff1e86:::benchmark_data_smithi159_102124_object50153:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bff7033:::benchmark_data_smithi159_102124_object35732:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2bff9158:::benchmark_data_smithi159_102124_object41713:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0377fc:::benchmark_data_smithi159_102124_object7945:head# 2024-04-15T18:22:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c03dc24:::benchmark_data_smithi159_102124_object19575:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c05de86:::benchmark_data_smithi159_102124_object38661:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c05fa76:::benchmark_data_smithi159_102124_object7775:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c08a09b:::benchmark_data_smithi159_102124_object604:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0939ab:::benchmark_data_smithi159_102124_object11378:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c09797a:::benchmark_data_smithi159_102124_object42277:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0c123f:::benchmark_data_smithi159_102124_object5618:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0c5df5:::benchmark_data_smithi159_102124_object26077:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0ceb6a:::benchmark_data_smithi159_102124_object10321:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0d9d22:::benchmark_data_smithi159_102124_object6861:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0e3428:::benchmark_data_smithi159_102124_object28096:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0e5fce:::benchmark_data_smithi159_102124_object57268:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0e7496:::benchmark_data_smithi159_102124_object37854:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c0f97ab:::benchmark_data_smithi159_102124_object13067:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1021a2:::benchmark_data_smithi159_102124_object49193:head# 2024-04-15T18:22:17.765 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c10fc59:::benchmark_data_smithi159_102124_object25274:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1214c3:::benchmark_data_smithi159_102124_object14471:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c131807:::benchmark_data_smithi159_102124_object1031:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1346ba:::benchmark_data_smithi159_102124_object45021:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c136f96:::benchmark_data_smithi159_102124_object20911:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c15f559:::benchmark_data_smithi159_102124_object9600:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1838b3:::benchmark_data_smithi159_102124_object50235:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1a3c26:::benchmark_data_smithi159_102124_object17058:head# 2024-04-15T18:22:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1c980a:::benchmark_data_smithi159_102124_object26758:head# 2024-04-15T18:22:17.767 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1d27fc:::benchmark_data_smithi159_102124_object10323:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1e1333:::benchmark_data_smithi159_102124_object16969:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c1fa461:::benchmark_data_smithi159_102124_object16742:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c22675b:::benchmark_data_smithi159_102124_object3263:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2320e9:::benchmark_data_smithi159_102124_object39813:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c23be16:::benchmark_data_smithi159_102124_object34795:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c243c17:::benchmark_data_smithi159_102124_object54476:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c263f53:::benchmark_data_smithi159_102124_object46386:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c26fe39:::benchmark_data_smithi159_102124_object5708:head# 2024-04-15T18:22:17.777 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c277297:::benchmark_data_smithi159_102124_object5707:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c279fe9:::benchmark_data_smithi159_102124_object58260:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c281e5c:::benchmark_data_smithi159_102124_object46288:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c293f29:::benchmark_data_smithi159_102124_object44932:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c29921a:::benchmark_data_smithi159_102124_object44895:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2a7a12:::benchmark_data_smithi159_102124_object41630:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2a82d2:::benchmark_data_smithi159_102124_object30675:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2bbe8b:::benchmark_data_smithi159_102124_object37326:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2c1683:::benchmark_data_smithi159_102124_object7634:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2c86b7:::benchmark_data_smithi159_102124_object970:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2df90d:::benchmark_data_smithi159_102124_object55269:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2f198a:::benchmark_data_smithi159_102124_object11779:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c2ff0f7:::benchmark_data_smithi159_102124_object805:head# 2024-04-15T18:22:17.778 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c301e5c:::benchmark_data_smithi159_102124_object42315:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c316bbc:::benchmark_data_smithi159_102124_object29402:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c31c45f:::benchmark_data_smithi159_102124_object24004:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c355253:::benchmark_data_smithi159_102124_object51325:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c35dc7b:::benchmark_data_smithi159_102124_object53934:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c362912:::benchmark_data_smithi159_102124_object53933:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c36a667:::benchmark_data_smithi159_102124_object7589:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c36b3d2:::benchmark_data_smithi159_102124_object36413:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c382025:::benchmark_data_smithi159_102124_object26123:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c387770:::benchmark_data_smithi159_102124_object1869:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c3b32f2:::benchmark_data_smithi159_102124_object36942:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c3d2301:::benchmark_data_smithi159_102124_object8182:head# 2024-04-15T18:22:17.779 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c3e160b:::benchmark_data_smithi159_102124_object21850:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c3e53f0:::benchmark_data_smithi159_102124_object1577:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c3f4336:::benchmark_data_smithi159_102124_object50792:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4174b0:::benchmark_data_smithi159_102124_object55626:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4240b8:::benchmark_data_smithi159_102124_object31397:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c441656:::benchmark_data_smithi159_102124_object24804:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c463b4c:::benchmark_data_smithi159_102124_object40778:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4b3e95:::benchmark_data_smithi159_102124_object8040:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4c15be:::benchmark_data_smithi159_102124_object33523:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4d17fd:::benchmark_data_smithi159_102124_object8344:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4d3edb:::benchmark_data_smithi159_102124_object28017:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4dc74d:::benchmark_data_smithi159_102124_object49971:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4e1964:::benchmark_data_smithi159_102124_object57033:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c4fdec0:::benchmark_data_smithi159_102124_object173:head# 2024-04-15T18:22:17.780 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c518f05:::benchmark_data_smithi159_102124_object6340:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c519f62:::benchmark_data_smithi159_102124_object26436:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c544c0b:::benchmark_data_smithi159_102124_object41025:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c54d11f:::benchmark_data_smithi159_102124_object3195:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c56045b:::benchmark_data_smithi159_102124_object52944:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c585119:::benchmark_data_smithi159_102124_object1673:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c599eb4:::benchmark_data_smithi159_102124_object6047:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c5a9cfc:::benchmark_data_smithi159_102124_object25610:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c5ae149:::benchmark_data_smithi159_102124_object21010:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c5b1b3c:::benchmark_data_smithi159_102124_object27303:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c5eb338:::benchmark_data_smithi159_102124_object36681:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c603e82:::benchmark_data_smithi159_102124_object32220:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c61fe08:::benchmark_data_smithi159_102124_object31020:head# 2024-04-15T18:22:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c62171e:::benchmark_data_smithi159_102124_object36430:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c62d3e7:::benchmark_data_smithi159_102124_object26070:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c637672:::benchmark_data_smithi159_102124_object1455:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c64bf05:::benchmark_data_smithi159_102124_object22826:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6540ba:::benchmark_data_smithi159_102124_object4675:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c65f2a2:::benchmark_data_smithi159_102124_object33982:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c669c34:::benchmark_data_smithi159_102124_object49526:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c693623:::benchmark_data_smithi159_102124_object6885:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6a4e81:::benchmark_data_smithi159_102124_object17847:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6d3ae5:::benchmark_data_smithi159_102124_object22980:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6d7fc3:::benchmark_data_smithi159_102124_object3768:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6e7e03:::benchmark_data_smithi159_102124_object1492:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6ebd94:::benchmark_data_smithi159_102124_object48574:head# 2024-04-15T18:22:17.782 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c6f7e5c:::benchmark_data_smithi159_102124_object29001:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c700560:::benchmark_data_smithi159_102124_object21614:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c704f1c:::benchmark_data_smithi159_102124_object15000:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c70e0f7:::benchmark_data_smithi159_102124_object58115:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c729cf8:::benchmark_data_smithi159_102124_object21816:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c741096:::benchmark_data_smithi159_102124_object4555:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c745390:::benchmark_data_smithi159_102124_object45437:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c74e43b:::benchmark_data_smithi159_102124_object13655:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c793b30:::benchmark_data_smithi159_102124_object18762:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c79ed40:::benchmark_data_smithi159_102124_object13730:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7a2d0d:::benchmark_data_smithi159_102124_object21410:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7a9313:::benchmark_data_smithi159_102124_object26764:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7b393f:::benchmark_data_smithi159_102124_object29919:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7bca1a:::benchmark_data_smithi159_102124_object45300:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7c49ce:::benchmark_data_smithi159_102124_object12671:head# 2024-04-15T18:22:17.783 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7de5bc:::benchmark_data_smithi159_102124_object18215:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c7f3aba:::benchmark_data_smithi159_102124_object35096:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c800f01:::benchmark_data_smithi159_102124_object44920:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c806133:::benchmark_data_smithi159_102124_object2009:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8084bb:::benchmark_data_smithi159_102124_object47503:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8096b2:::benchmark_data_smithi159_102124_object17401:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c80c134:::benchmark_data_smithi159_102124_object33564:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c80f7e9:::benchmark_data_smithi159_102124_object31226:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c811f2c:::benchmark_data_smithi159_102124_object36489:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c815361:::benchmark_data_smithi159_102124_object3994:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c822898:::benchmark_data_smithi159_102124_object24400:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c826d49:::benchmark_data_smithi159_102124_object36415:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c83fcb2:::benchmark_data_smithi159_102124_object25441:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c854215:::benchmark_data_smithi159_102124_object53758:head# 2024-04-15T18:22:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c85b109:::benchmark_data_smithi159_102124_object23643:head# 2024-04-15T18:22:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:17.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:17.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c876316:::benchmark_data_smithi159_102124_object46169:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8773cc:::benchmark_data_smithi159_102124_object20164:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8862d4:::benchmark_data_smithi159_102124_object8854:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8892cd:::benchmark_data_smithi159_102124_object2297:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c88e3ad:::benchmark_data_smithi159_102124_object7699:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c898cd5:::benchmark_data_smithi159_102124_object19867:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c89d72a:::benchmark_data_smithi159_102124_object50049:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8c5513:::benchmark_data_smithi159_102124_object14998:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8c754b:::benchmark_data_smithi159_102124_object49182:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8d3787:::benchmark_data_smithi159_102124_object42648:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8fc6a1:::benchmark_data_smithi159_102124_object6881:head# 2024-04-15T18:22:17.796 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c8fdbdc:::benchmark_data_smithi159_102124_object40859:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c94168a:::benchmark_data_smithi159_102124_object23381:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c949bfc:::benchmark_data_smithi159_102124_object6050:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c951df6:::benchmark_data_smithi159_102124_object30425:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c96936e:::benchmark_data_smithi159_102124_object27696:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c96a291:::benchmark_data_smithi159_102124_object46359:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c9721e0:::benchmark_data_smithi159_102124_object10222:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c97c88b:::benchmark_data_smithi159_102124_object26160:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c983428:::benchmark_data_smithi159_102124_object43909:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c98bf45:::benchmark_data_smithi159_102124_object24657:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c99c138:::benchmark_data_smithi159_102124_object46824:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c9ae883:::benchmark_data_smithi159_102124_object17359:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c9c6b1e:::benchmark_data_smithi159_102124_object43855:head# 2024-04-15T18:22:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2c9dd683:::benchmark_data_smithi159_102124_object18833:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca12cf9:::benchmark_data_smithi159_102124_object24854:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca30a50:::benchmark_data_smithi159_102124_object49588:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca3897f:::benchmark_data_smithi159_102124_object36515:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca4bfac:::benchmark_data_smithi159_102124_object12954:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca75f27:::benchmark_data_smithi159_102124_object21147:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca8e869:::benchmark_data_smithi159_102124_object44415:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ca94455:::benchmark_data_smithi159_102124_object18351:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2caa387d:::benchmark_data_smithi159_102124_object44579:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cab598f:::benchmark_data_smithi159_102124_object40625:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cab9597:::benchmark_data_smithi159_102124_object12290:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cace85b:::benchmark_data_smithi159_102124_object41572:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb311a7:::benchmark_data_smithi159_102124_object48811:head# 2024-04-15T18:22:17.798 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb4b905:::benchmark_data_smithi159_102124_object22574:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb4c584:::benchmark_data_smithi159_102124_object40180:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb5248a:::benchmark_data_smithi159_102124_object46741:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb54347:::benchmark_data_smithi159_102124_object28743:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb5ded8:::benchmark_data_smithi159_102124_object37356:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb5e216:::benchmark_data_smithi159_102124_object27183:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb6bec9:::benchmark_data_smithi159_102124_object41487:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb77e11:::benchmark_data_smithi159_102124_object8021:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb89287:::benchmark_data_smithi159_102124_object17542:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb9982b:::benchmark_data_smithi159_102124_object32265:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cb9f8a2:::benchmark_data_smithi159_102124_object50588:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cba64b8:::benchmark_data_smithi159_102124_object56349:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cbabc1e:::benchmark_data_smithi159_102124_object45987:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cbb162e:::benchmark_data_smithi159_102124_object51774:head# 2024-04-15T18:22:17.799 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cbb21a5:::benchmark_data_smithi159_102124_object7258:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cbdc4c9:::benchmark_data_smithi159_102124_object18033:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cbea0b8:::benchmark_data_smithi159_102124_object17754:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc0456a:::benchmark_data_smithi159_102124_object17448:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc073a7:::benchmark_data_smithi159_102124_object50920:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc12789:::benchmark_data_smithi159_102124_object749:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc26535:::benchmark_data_smithi159_102124_object57484:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc28418:::benchmark_data_smithi159_102124_object56498:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc4236c:::benchmark_data_smithi159_102124_object13686:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc4d8d2:::benchmark_data_smithi159_102124_object1319:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc5b2a2:::benchmark_data_smithi159_102124_object13008:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc6d418:::benchmark_data_smithi159_102124_object329:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc70281:::benchmark_data_smithi159_102124_object50758:head# 2024-04-15T18:22:17.800 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc7453f:::benchmark_data_smithi159_102124_object5123:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc8d54e:::benchmark_data_smithi159_102124_object5013:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cc94911:::benchmark_data_smithi159_102124_object12364:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ccc0aa1:::benchmark_data_smithi159_102124_object36081:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ccd660d:::benchmark_data_smithi159_102124_object58318:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cce19a6:::benchmark_data_smithi159_102124_object11894:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ccece3a:::benchmark_data_smithi159_102124_object57707:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ccf157b:::benchmark_data_smithi159_102124_object2046:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ccf5d29:::benchmark_data_smithi159_102124_object57019:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd3a492:::benchmark_data_smithi159_102124_object41325:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd3d47e:::benchmark_data_smithi159_102124_object11670:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd43f6d:::benchmark_data_smithi159_102124_object1965:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd4f9a1:::benchmark_data_smithi159_102124_object633:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd6ee99:::benchmark_data_smithi159_102124_object8326:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd6eeb8:::benchmark_data_smithi159_102124_object48201:head# 2024-04-15T18:22:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd6f2b7:::benchmark_data_smithi159_102124_object44608:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd8151d:::benchmark_data_smithi159_102124_object46290:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd88e4c:::benchmark_data_smithi159_102124_object20801:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd8a235:::benchmark_data_smithi159_102124_object51261:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cd8a5e9:::benchmark_data_smithi159_102124_object18659:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cdc7f98:::benchmark_data_smithi159_102124_object1025:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cdd22e6:::benchmark_data_smithi159_102124_object52487:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cdd59e2:::benchmark_data_smithi159_102124_object40752:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cdd6def:::benchmark_data_smithi159_102124_object38232:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce20d9e:::benchmark_data_smithi159_102124_object36794:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce27d7b:::benchmark_data_smithi159_102124_object17903:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce3c41c:::benchmark_data_smithi159_102124_object31406:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce68792:::benchmark_data_smithi159_102124_object23346:head# 2024-04-15T18:22:17.802 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce6f03f:::benchmark_data_smithi159_102124_object35984:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce85d69:::benchmark_data_smithi159_102124_object9945:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ce9b720:::benchmark_data_smithi159_102124_object21245:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cea65f9:::benchmark_data_smithi159_102124_object53264:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ceaea08:::benchmark_data_smithi159_102124_object27578:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ceb716c:::benchmark_data_smithi159_102124_object52524:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ced5823:::benchmark_data_smithi159_102124_object7821:head# 2024-04-15T18:22:17.803 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cef3150:::benchmark_data_smithi159_102124_object57942:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cef54ae:::benchmark_data_smithi159_102124_object19179:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cefc9fc:::benchmark_data_smithi159_102124_object47504:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf29023:::benchmark_data_smithi159_102124_object27156:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf2d720:::benchmark_data_smithi159_102124_object43743:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf416fa:::benchmark_data_smithi159_102124_object26688:head# 2024-04-15T18:22:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf4fa27:::benchmark_data_smithi159_102124_object1362:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf699a7:::benchmark_data_smithi159_102124_object11485:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf6dd4c:::benchmark_data_smithi159_102124_object17063:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf7b73c:::benchmark_data_smithi159_102124_object943:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf7e4eb:::benchmark_data_smithi159_102124_object27488:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf82a07:::benchmark_data_smithi159_102124_object33187:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cf96bdc:::benchmark_data_smithi159_102124_object15338:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cfc267b:::benchmark_data_smithi159_102124_object53295:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cfd1eee:::benchmark_data_smithi159_102124_object49555:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cfd8cb7:::benchmark_data_smithi159_102124_object49218:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2cfea787:::benchmark_data_smithi159_102124_object33877:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d019969:::benchmark_data_smithi159_102124_object3329:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d01de47:::benchmark_data_smithi159_102124_object15157:head# 2024-04-15T18:22:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d02d9ad:::benchmark_data_smithi159_102124_object51064:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d02f887:::benchmark_data_smithi159_102124_object23169:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d0336e8:::benchmark_data_smithi159_102124_object7913:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d03ad5a:::benchmark_data_smithi159_102124_object31418:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d06b7c0:::benchmark_data_smithi159_102124_object1708:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d08323e:::benchmark_data_smithi159_102124_object35619:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d0900c1:::benchmark_data_smithi159_102124_object53306:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d097500:::benchmark_data_smithi159_102124_object8382:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d0ad771:::benchmark_data_smithi159_102124_object6171:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d10f8d0:::benchmark_data_smithi159_102124_object21805:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d11414a:::benchmark_data_smithi159_102124_object21900:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d11bcad:::benchmark_data_smithi159_102124_object38770:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d126ae8:::benchmark_data_smithi159_102124_object37769:head# 2024-04-15T18:22:17.824 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d12e4e3:::benchmark_data_smithi159_102124_object3030:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d137f6f:::benchmark_data_smithi159_102124_object54328:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d143580:::benchmark_data_smithi159_102124_object25329:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d147355:::benchmark_data_smithi159_102124_object12480:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d165f70:::benchmark_data_smithi159_102124_object20489:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d17afb5:::benchmark_data_smithi159_102124_object7313:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d19089b:::benchmark_data_smithi159_102124_object40411:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d1a2af4:::benchmark_data_smithi159_102124_object7762:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d1f2f0e:::benchmark_data_smithi159_102124_object31172:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2303c5:::benchmark_data_smithi159_102124_object32673:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d23463d:::benchmark_data_smithi159_102124_object18990:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2388dc:::benchmark_data_smithi159_102124_object16662:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d244655:::benchmark_data_smithi159_102124_object56965:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d287c52:::benchmark_data_smithi159_102124_object36412:head# 2024-04-15T18:22:17.825 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d298e83:::benchmark_data_smithi159_102124_object9540:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2b5615:::benchmark_data_smithi159_102124_object26636:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2caf6e:::benchmark_data_smithi159_102124_object29322:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2e7544:::benchmark_data_smithi159_102124_object9307:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d2edc92:::benchmark_data_smithi159_102124_object12245:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d304e3d:::benchmark_data_smithi159_102124_object58439:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d323713:::benchmark_data_smithi159_102124_object36938:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d33a16d:::benchmark_data_smithi159_102124_object39549:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3478ee:::benchmark_data_smithi159_102124_object45642:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3657be:::benchmark_data_smithi159_102124_object14957:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3766dc:::benchmark_data_smithi159_102124_object2271:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3b4abb:::benchmark_data_smithi159_102124_object37899:head# 2024-04-15T18:22:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3c909b:::benchmark_data_smithi159_102124_object24785:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3d0e74:::benchmark_data_smithi159_102124_object25510:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d3e999d:::benchmark_data_smithi159_102124_object44943:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d424dcb:::benchmark_data_smithi159_102124_object4167:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4262a5:::benchmark_data_smithi159_102124_object50616:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d427b73:::benchmark_data_smithi159_102124_object14528:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d439abb:::benchmark_data_smithi159_102124_object6684:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d458361:::benchmark_data_smithi159_102124_object43437:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d46c5b7:::benchmark_data_smithi159_102124_object15224:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d471174:::benchmark_data_smithi159_102124_object26240:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d498b4e:::benchmark_data_smithi159_102124_object44097:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d49f804:::benchmark_data_smithi159_102124_object4226:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4b78e2:::benchmark_data_smithi159_102124_object39464:head# 2024-04-15T18:22:17.827 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4c9304:::benchmark_data_smithi159_102124_object53666:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4c986d:::benchmark_data_smithi159_102124_object17738:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4da26a:::benchmark_data_smithi159_102124_object16247:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4e0139:::benchmark_data_smithi159_102124_object101:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4e4973:::benchmark_data_smithi159_102124_object30395:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4f9b21:::benchmark_data_smithi159_102124_object15936:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d4fda6c:::benchmark_data_smithi159_102124_object12519:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d502388:::benchmark_data_smithi159_102124_object3846:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d505b44:::benchmark_data_smithi159_102124_object4851:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d512ddb:::benchmark_data_smithi159_102124_object12820:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d52db52:::benchmark_data_smithi159_102124_object33562:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d543dc7:::benchmark_data_smithi159_102124_object6109:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d544294:::benchmark_data_smithi159_102124_object19741:head# 2024-04-15T18:22:17.828 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d54e06a:::benchmark_data_smithi159_102124_object22074:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d558533:::benchmark_data_smithi159_102124_object11395:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d56dfed:::benchmark_data_smithi159_102124_object33470:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d57b6d7:::benchmark_data_smithi159_102124_object10056:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d57dc81:::benchmark_data_smithi159_102124_object47538:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d57f4e5:::benchmark_data_smithi159_102124_object25260:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d58d7b1:::benchmark_data_smithi159_102124_object1729:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d590181:::benchmark_data_smithi159_102124_object34289:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5b9a0a:::benchmark_data_smithi159_102124_object14582:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5c1fea:::benchmark_data_smithi159_102124_object11663:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5d05fc:::benchmark_data_smithi159_102124_object15458:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5d377c:::benchmark_data_smithi159_102124_object33545:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5d7c79:::benchmark_data_smithi159_102124_object2779:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d5f1c00:::benchmark_data_smithi159_102124_object4446:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d60392e:::benchmark_data_smithi159_102124_object25725:head# 2024-04-15T18:22:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d63ef0d:::benchmark_data_smithi159_102124_object43245:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6653ce:::benchmark_data_smithi159_102124_object48242:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d671192:::benchmark_data_smithi159_102124_object23966:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d687289:::benchmark_data_smithi159_102124_object57361:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d68c404:::benchmark_data_smithi159_102124_object57357:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d69e0b6:::benchmark_data_smithi159_102124_object49645:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6a28ca:::benchmark_data_smithi159_102124_object22211:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6b603d:::benchmark_data_smithi159_102124_object5533:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6b9b7c:::benchmark_data_smithi159_102124_object35098:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6eb96c:::benchmark_data_smithi159_102124_object32768:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6f3b5b:::benchmark_data_smithi159_102124_object18242:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6fa2e2:::benchmark_data_smithi159_102124_object52325:head# 2024-04-15T18:22:17.830 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d6fce81:::benchmark_data_smithi159_102124_object28515:head# 2024-04-15T18:22:17.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7208a2:::benchmark_data_smithi159_102124_object43644:head# 2024-04-15T18:22:17.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d72841a:::benchmark_data_smithi159_102124_object25531:head# 2024-04-15T18:22:17.831 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d735676:::benchmark_data_smithi159_102124_object40791:head# 2024-04-15T18:22:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d73cc15:::benchmark_data_smithi159_102124_object48772:head# 2024-04-15T18:22:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d75d240:::benchmark_data_smithi159_102124_object8409:head# 2024-04-15T18:22:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d76b441:::benchmark_data_smithi159_102124_object51445:head# 2024-04-15T18:22:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d77697f:::benchmark_data_smithi159_102124_object1977:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7976d4:::benchmark_data_smithi159_102124_object41576:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7b8064:::benchmark_data_smithi159_102124_object23311:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7c0c6e:::benchmark_data_smithi159_102124_object12469:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7ee665:::benchmark_data_smithi159_102124_object39071:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d7f1939:::benchmark_data_smithi159_102124_object7967:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d802165:::benchmark_data_smithi159_102124_object6193:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d81c176:::benchmark_data_smithi159_102124_object47539:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d81c23a:::benchmark_data_smithi159_102124_object54186:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d825975:::benchmark_data_smithi159_102124_object51107:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d836bc0:::benchmark_data_smithi159_102124_object32903:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d859ca6:::benchmark_data_smithi159_102124_object15154:head# 2024-04-15T18:22:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d860ffe:::benchmark_data_smithi159_102124_object3087:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d872812:::benchmark_data_smithi159_102124_object26985:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8762f2:::benchmark_data_smithi159_102124_object21609:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d878dac:::benchmark_data_smithi159_102124_object50887:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d88da51:::benchmark_data_smithi159_102124_object36307:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8ab5b5:::benchmark_data_smithi159_102124_object47885:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8b56f8:::benchmark_data_smithi159_102124_object20726:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8d51d8:::benchmark_data_smithi159_102124_object12560:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8d74d4:::benchmark_data_smithi159_102124_object29224:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8eee4a:::benchmark_data_smithi159_102124_object17452:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d8f83fd:::benchmark_data_smithi159_102124_object15623:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d90ebb8:::benchmark_data_smithi159_102124_object9663:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d920ea3:::benchmark_data_smithi159_102124_object32939:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d921af6:::benchmark_data_smithi159_102124_object36129:head# 2024-04-15T18:22:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d926f4f:::benchmark_data_smithi159_102124_object11470:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d93312b:::benchmark_data_smithi159_102124_object14551:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d93ff7c:::benchmark_data_smithi159_102124_object45354:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9467de:::benchmark_data_smithi159_102124_object23646:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d949e00:::benchmark_data_smithi159_102124_object11436:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d94d7ed:::benchmark_data_smithi159_102124_object37282:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d95058f:::benchmark_data_smithi159_102124_object7006:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d95479b:::benchmark_data_smithi159_102124_object55742:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d95915e:::benchmark_data_smithi159_102124_object28110:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d959b83:::benchmark_data_smithi159_102124_object32686:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d95c314:::benchmark_data_smithi159_102124_object21571:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9774d0:::benchmark_data_smithi159_102124_object2103:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d993acc:::benchmark_data_smithi159_102124_object9703:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9ad885:::benchmark_data_smithi159_102124_object7990:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9c56c2:::benchmark_data_smithi159_102124_object21717:head# 2024-04-15T18:22:17.844 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9d31c7:::benchmark_data_smithi159_102124_object28591:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9d9ac4:::benchmark_data_smithi159_102124_object9358:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9e16e8:::benchmark_data_smithi159_102124_object34901:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2d9f06c6:::benchmark_data_smithi159_102124_object55192:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da2d0c5:::benchmark_data_smithi159_102124_object4608:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da31785:::benchmark_data_smithi159_102124_object44805:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da3b979:::benchmark_data_smithi159_102124_object27684:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da485ad:::benchmark_data_smithi159_102124_object33935:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da56d74:::benchmark_data_smithi159_102124_object14570:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da6a83e:::benchmark_data_smithi159_102124_object31318:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2da7d452:::benchmark_data_smithi159_102124_object6988:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2daa37f9:::benchmark_data_smithi159_102124_object18740:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2daa83e2:::benchmark_data_smithi159_102124_object35022:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2daaf184:::benchmark_data_smithi159_102124_object55597:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dab32b8:::benchmark_data_smithi159_102124_object56203:head# 2024-04-15T18:22:17.845 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dacd597:::benchmark_data_smithi159_102124_object5381:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dae74bd:::benchmark_data_smithi159_102124_object38583:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db1d6a3:::benchmark_data_smithi159_102124_object3143:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db258fb:::benchmark_data_smithi159_102124_object3520:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db3453c:::benchmark_data_smithi159_102124_object7382:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db450dd:::benchmark_data_smithi159_102124_object10191:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db5b1db:::benchmark_data_smithi159_102124_object21668:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db8023c:::benchmark_data_smithi159_102124_object13258:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db87032:::benchmark_data_smithi159_102124_object28317:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db8721e:::benchmark_data_smithi159_102124_object46678:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2db8f90d:::benchmark_data_smithi159_102124_object4738:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbaf307:::benchmark_data_smithi159_102124_object41266:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbc3bba:::benchmark_data_smithi159_102124_object53211:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbd8023:::benchmark_data_smithi159_102124_object49510:head# 2024-04-15T18:22:17.846 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbdf2bb:::benchmark_data_smithi159_102124_object1230:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbe2d0b:::benchmark_data_smithi159_102124_object16134:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbee9f0:::benchmark_data_smithi159_102124_object32742:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dbfe2a4:::benchmark_data_smithi159_102124_object22470:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc0a81c:::benchmark_data_smithi159_102124_object14711:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc11cba:::benchmark_data_smithi159_102124_object29300:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc15f5e:::benchmark_data_smithi159_102124_object33690:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc17e07:::benchmark_data_smithi159_102124_object31671:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc1f5bd:::benchmark_data_smithi159_102124_object46635:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc2dc4e:::benchmark_data_smithi159_102124_object33250:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc4511e:::benchmark_data_smithi159_102124_object47470:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc4f8c3:::benchmark_data_smithi159_102124_object41865:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc51043:::benchmark_data_smithi159_102124_object29089:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc5f8e7:::benchmark_data_smithi159_102124_object35101:head# 2024-04-15T18:22:17.847 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc60e50:::benchmark_data_smithi159_102124_object1454:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc61017:::benchmark_data_smithi159_102124_object12021:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc67838:::benchmark_data_smithi159_102124_object13562:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dc959fa:::benchmark_data_smithi159_102124_object1282:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dca6dad:::benchmark_data_smithi159_102124_object15539:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dca8075:::benchmark_data_smithi159_102124_object6572:head# 2024-04-15T18:22:17.848 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dcaaf6a:::benchmark_data_smithi159_102124_object34712:head# 2024-04-15T18:22:17.849 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dcbcf58:::benchmark_data_smithi159_102124_object12247:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dcc0422:::benchmark_data_smithi159_102124_object42689:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dcd3556:::benchmark_data_smithi159_102124_object15135:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dcdcb87:::benchmark_data_smithi159_102124_object24799:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd29f04:::benchmark_data_smithi159_102124_object29829:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd3d463:::benchmark_data_smithi159_102124_object23022:head# 2024-04-15T18:22:17.850 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd4002f:::benchmark_data_smithi159_102124_object35457:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd418db:::benchmark_data_smithi159_102124_object9416:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd4e65e:::benchmark_data_smithi159_102124_object56451:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd73696:::benchmark_data_smithi159_102124_object42409:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd7ac97:::benchmark_data_smithi159_102124_object24329:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd7e1a2:::benchmark_data_smithi159_102124_object30724:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd8fad7:::benchmark_data_smithi159_102124_object28565:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd91df1:::benchmark_data_smithi159_102124_object37081:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dd99af3:::benchmark_data_smithi159_102124_object468:head# 2024-04-15T18:22:17.851 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ddc44d7:::benchmark_data_smithi159_102124_object26737:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ddd5f8d:::benchmark_data_smithi159_102124_object48349:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dde027a:::benchmark_data_smithi159_102124_object22616:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dde88c7:::benchmark_data_smithi159_102124_object21742:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ddfa2a6:::benchmark_data_smithi159_102124_object10711:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de06695:::benchmark_data_smithi159_102124_object32974:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de08087:::benchmark_data_smithi159_102124_object49013:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de0c1a9:::benchmark_data_smithi159_102124_object9109:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de269b6:::benchmark_data_smithi159_102124_object42369:head# 2024-04-15T18:22:17.852 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de2789a:::benchmark_data_smithi159_102124_object38169:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de3007b:::benchmark_data_smithi159_102124_object44113:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de496f6:::benchmark_data_smithi159_102124_object26037:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2de59470:::benchmark_data_smithi159_102124_object51222:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2deb92b5:::benchmark_data_smithi159_102124_object2420:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2debb448:::benchmark_data_smithi159_102124_object15206:head# 2024-04-15T18:22:17.855 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dee0d04:::benchmark_data_smithi159_102124_object19616:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dee8ed6:::benchmark_data_smithi159_102124_object31094:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df09334:::benchmark_data_smithi159_102124_object22017:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df0f591:::benchmark_data_smithi159_102124_object14157:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df1c795:::benchmark_data_smithi159_102124_object14316:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df4015c:::benchmark_data_smithi159_102124_object12322:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df42c19:::benchmark_data_smithi159_102124_object25607:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df4855d:::benchmark_data_smithi159_102124_object51988:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df5b85f:::benchmark_data_smithi159_102124_object43564:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df658d8:::benchmark_data_smithi159_102124_object35284:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df76caa:::benchmark_data_smithi159_102124_object4634:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df88a1f:::benchmark_data_smithi159_102124_object31847:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2df97c22:::benchmark_data_smithi159_102124_object57769:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dfa5589:::benchmark_data_smithi159_102124_object3606:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dfa7e50:::benchmark_data_smithi159_102124_object51111:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dfe4c12:::benchmark_data_smithi159_102124_object52519:head# 2024-04-15T18:22:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dff26e6:::benchmark_data_smithi159_102124_object6286:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2dffcff6:::benchmark_data_smithi159_102124_object49573:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e008afd:::benchmark_data_smithi159_102124_object32191:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e013e21:::benchmark_data_smithi159_102124_object16263:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e018f67:::benchmark_data_smithi159_102124_object28514:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e01c2d7:::benchmark_data_smithi159_102124_object19147:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e02b32c:::benchmark_data_smithi159_102124_object38434:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e044dda:::benchmark_data_smithi159_102124_object49661:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e04d077:::benchmark_data_smithi159_102124_object34482:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e06bf17:::benchmark_data_smithi159_102124_object35493:head# 2024-04-15T18:22:17.857 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e06f1c2:::benchmark_data_smithi159_102124_object21400:head# 2024-04-15T18:22:17.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e081d59:::benchmark_data_smithi159_102124_object25455:head# 2024-04-15T18:22:17.858 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e087b34:::benchmark_data_smithi159_102124_object24150:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e0890cf:::benchmark_data_smithi159_102124_object18194:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e0897ea:::benchmark_data_smithi159_102124_object41186:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e08a5a3:::benchmark_data_smithi159_102124_object17534:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e095326:::benchmark_data_smithi159_102124_object31183:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e0d131f:::benchmark_data_smithi159_102124_object34843:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e0ed41b:::benchmark_data_smithi159_102124_object18313:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e0ff9ce:::benchmark_data_smithi159_102124_object6136:head# 2024-04-15T18:22:17.866 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e104606:::benchmark_data_smithi159_102124_object27163:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1152d8:::benchmark_data_smithi159_102124_object28253:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1169a7:::benchmark_data_smithi159_102124_object5214:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1274f1:::benchmark_data_smithi159_102124_object9619:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e12a604:::benchmark_data_smithi159_102124_object40269:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e133747:::benchmark_data_smithi159_102124_object22615:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e13e599:::benchmark_data_smithi159_102124_object41904:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e14512f:::benchmark_data_smithi159_102124_object35223:head# 2024-04-15T18:22:17.867 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e15ad3a:::benchmark_data_smithi159_102124_object14191:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1661dd:::benchmark_data_smithi159_102124_object56384:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e174470:::benchmark_data_smithi159_102124_object32491:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1819d0:::benchmark_data_smithi159_102124_object38565:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e18ff05:::benchmark_data_smithi159_102124_object728:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e192ee7:::benchmark_data_smithi159_102124_object6243:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1987cf:::benchmark_data_smithi159_102124_object49021:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1b00a1:::benchmark_data_smithi159_102124_object14621:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1cd124:::benchmark_data_smithi159_102124_object21636:head# 2024-04-15T18:22:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e1f7664:::benchmark_data_smithi159_102124_object17773:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e200ddf:::benchmark_data_smithi159_102124_object3972:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e22b0e5:::benchmark_data_smithi159_102124_object40559:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e23ce6d:::benchmark_data_smithi159_102124_object14425:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e2546c5:::benchmark_data_smithi159_102124_object50116:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e2600fb:::benchmark_data_smithi159_102124_object58314:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e2780ab:::benchmark_data_smithi159_102124_object48610:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e27831e:::benchmark_data_smithi159_102124_object10953:head# 2024-04-15T18:22:17.869 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e27bf86:::benchmark_data_smithi159_102124_object15717:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e281ce3:::benchmark_data_smithi159_102124_object8798:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e287af2:::benchmark_data_smithi159_102124_object32784:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e28c535:::benchmark_data_smithi159_102124_object30358:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e28eac4:::benchmark_data_smithi159_102124_object10835:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e298070:::benchmark_data_smithi159_102124_object314:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e29e447:::benchmark_data_smithi159_102124_object56094:head# 2024-04-15T18:22:17.870 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e2b9a02:::benchmark_data_smithi159_102124_object57690:head# 2024-04-15T18:22:17.871 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e2ceb93:::benchmark_data_smithi159_102124_object30776:head# 2024-04-15T18:22:17.871 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e319bea:::benchmark_data_smithi159_102124_object49830:head# 2024-04-15T18:22:17.871 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e334bfe:::benchmark_data_smithi159_102124_object37125:head# 2024-04-15T18:22:17.871 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e33fd92:::benchmark_data_smithi159_102124_object12214:head# 2024-04-15T18:22:17.872 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e348006:::benchmark_data_smithi159_102124_object35119:head# 2024-04-15T18:22:17.872 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e34a938:::benchmark_data_smithi159_102124_object23273:head# 2024-04-15T18:22:17.872 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e35ffd7:::benchmark_data_smithi159_102124_object8359:head# 2024-04-15T18:22:17.872 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e362205:::benchmark_data_smithi159_102124_object47501:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e37ae56:::benchmark_data_smithi159_102124_object32246:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e3a635e:::benchmark_data_smithi159_102124_object48896:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e3c51d8:::benchmark_data_smithi159_102124_object13402:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e3d5e14:::benchmark_data_smithi159_102124_object10592:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4084bb:::benchmark_data_smithi159_102124_object40148:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e41c1d6:::benchmark_data_smithi159_102124_object4925:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4256a3:::benchmark_data_smithi159_102124_object12183:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4296ca:::benchmark_data_smithi159_102124_object2178:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e431365:::benchmark_data_smithi159_102124_object55395:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e43c00c:::benchmark_data_smithi159_102124_object29574:head# 2024-04-15T18:22:17.873 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e465ad4:::benchmark_data_smithi159_102124_object25216:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e468202:::benchmark_data_smithi159_102124_object26904:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e46afa5:::benchmark_data_smithi159_102124_object11695:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e48273c:::benchmark_data_smithi159_102124_object5940:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4928a7:::benchmark_data_smithi159_102124_object40845:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e49fb2c:::benchmark_data_smithi159_102124_object55380:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4ad857:::benchmark_data_smithi159_102124_object13141:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4bf266:::benchmark_data_smithi159_102124_object32295:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4c6487:::benchmark_data_smithi159_102124_object41567:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4ccbd2:::benchmark_data_smithi159_102124_object52000:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4d3028:::benchmark_data_smithi159_102124_object47024:head# 2024-04-15T18:22:17.874 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e4e60fb:::benchmark_data_smithi159_102124_object9693:head# 2024-04-15T18:22:17.881 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e5023e3:::benchmark_data_smithi159_102124_object4773:head# 2024-04-15T18:22:17.881 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e50349f:::benchmark_data_smithi159_102124_object18216:head# 2024-04-15T18:22:17.881 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e508e46:::benchmark_data_smithi159_102124_object978:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e512164:::benchmark_data_smithi159_102124_object45603:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e51666b:::benchmark_data_smithi159_102124_object53922:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e521643:::benchmark_data_smithi159_102124_object15436:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e52fdbc:::benchmark_data_smithi159_102124_object33101:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e536ed7:::benchmark_data_smithi159_102124_object56118:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e54f2d6:::benchmark_data_smithi159_102124_object28487:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e587619:::benchmark_data_smithi159_102124_object54902:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e5d585f:::benchmark_data_smithi159_102124_object38237:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e5d8636:::benchmark_data_smithi159_102124_object8010:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e5fa462:::benchmark_data_smithi159_102124_object26871:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e602a3a:::benchmark_data_smithi159_102124_object30853:head# 2024-04-15T18:22:17.882 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e60667e:::benchmark_data_smithi159_102124_object262:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e607395:::benchmark_data_smithi159_102124_object37528:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e62dfdb:::benchmark_data_smithi159_102124_object54904:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e631bce:::benchmark_data_smithi159_102124_object19195:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e63384b:::benchmark_data_smithi159_102124_object16689:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e65c385:::benchmark_data_smithi159_102124_object53692:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e667c95:::benchmark_data_smithi159_102124_object24497:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e67ce03:::benchmark_data_smithi159_102124_object54955:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e681e1d:::benchmark_data_smithi159_102124_object41149:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e69e522:::benchmark_data_smithi159_102124_object1502:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6a66ae:::benchmark_data_smithi159_102124_object39648:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6b3285:::benchmark_data_smithi159_102124_object6761:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6dd0d9:::benchmark_data_smithi159_102124_object42131:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6e4f93:::benchmark_data_smithi159_102124_object33641:head# 2024-04-15T18:22:17.883 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6ed91a:::benchmark_data_smithi159_102124_object1133:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6f8764:::benchmark_data_smithi159_102124_object30284:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e6f9cd5:::benchmark_data_smithi159_102124_object2175:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e701e6a:::benchmark_data_smithi159_102124_object30416:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e70584f:::benchmark_data_smithi159_102124_object4818:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e70bc1a:::benchmark_data_smithi159_102124_object47511:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7287e1:::benchmark_data_smithi159_102124_object11066:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e742313:::benchmark_data_smithi159_102124_object19884:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e74ad1d:::benchmark_data_smithi159_102124_object1080:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e74cd95:::benchmark_data_smithi159_102124_object7089:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7528f4:::benchmark_data_smithi159_102124_object27447:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e75ddd6:::benchmark_data_smithi159_102124_object16396:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7810d1:::benchmark_data_smithi159_102124_object23598:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e790579:::benchmark_data_smithi159_102124_object1640:head# 2024-04-15T18:22:17.884 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e791612:::benchmark_data_smithi159_102124_object40188:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7916e8:::benchmark_data_smithi159_102124_object22910:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7a345f:::benchmark_data_smithi159_102124_object57449:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7aa186:::benchmark_data_smithi159_102124_object25248:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7c6552:::benchmark_data_smithi159_102124_object12014:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7cc7b2:::benchmark_data_smithi159_102124_object21109:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7dc9f8:::benchmark_data_smithi159_102124_object4785:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7f0056:::benchmark_data_smithi159_102124_object22674:head# 2024-04-15T18:22:17.885 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e7f1d39:::benchmark_data_smithi159_102124_object44146:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e80d66f:::benchmark_data_smithi159_102124_object27150:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e811815:::benchmark_data_smithi159_102124_object34202:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e840ae7:::benchmark_data_smithi159_102124_object26958:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e845507:::benchmark_data_smithi159_102124_object37665:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e86d54b:::benchmark_data_smithi159_102124_object41512:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e86d9b8:::benchmark_data_smithi159_102124_object37872:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e87415a:::benchmark_data_smithi159_102124_object42164:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e88000c:::benchmark_data_smithi159_102124_object32310:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e880bd9:::benchmark_data_smithi159_102124_object14346:head# 2024-04-15T18:22:17.887 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e881561:::benchmark_data_smithi159_102124_object32644:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e8ad852:::benchmark_data_smithi159_102124_object46883:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e8eed32:::benchmark_data_smithi159_102124_object29628:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e90a558:::benchmark_data_smithi159_102124_object31049:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e925a98:::benchmark_data_smithi159_102124_object14338:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e95513a:::benchmark_data_smithi159_102124_object39518:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9604e3:::benchmark_data_smithi159_102124_object43321:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e971bf9:::benchmark_data_smithi159_102124_object37319:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e97aad2:::benchmark_data_smithi159_102124_object50537:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9a124c:::benchmark_data_smithi159_102124_object14692:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9a1dd4:::benchmark_data_smithi159_102124_object498:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9a2630:::benchmark_data_smithi159_102124_object7722:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9b9935:::benchmark_data_smithi159_102124_object37713:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9d4b44:::benchmark_data_smithi159_102124_object19449:head# 2024-04-15T18:22:17.888 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9d4b6c:::benchmark_data_smithi159_102124_object7694:head# 2024-04-15T18:22:17.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9db2e6:::benchmark_data_smithi159_102124_object47549:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2e9dc99a:::benchmark_data_smithi159_102124_object46132:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea1f955:::benchmark_data_smithi159_102124_object36437:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea23c83:::benchmark_data_smithi159_102124_object22732:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea45c2b:::benchmark_data_smithi159_102124_object43013:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea52ff9:::benchmark_data_smithi159_102124_object46079:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea6a90b:::benchmark_data_smithi159_102124_object54756:head# 2024-04-15T18:22:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea72237:::benchmark_data_smithi159_102124_object5674:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea842ed:::benchmark_data_smithi159_102124_object11607:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea850b8:::benchmark_data_smithi159_102124_object31434:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ea9d5dc:::benchmark_data_smithi159_102124_object25093:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eaaf3b5:::benchmark_data_smithi159_102124_object42778:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eadeb52:::benchmark_data_smithi159_102124_object46734:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb172c8:::benchmark_data_smithi159_102124_object53349:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb27101:::benchmark_data_smithi159_102124_object50006:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb2a2f3:::benchmark_data_smithi159_102124_object46255:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb4c34a:::benchmark_data_smithi159_102124_object23209:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb7820d:::benchmark_data_smithi159_102124_object26915:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb92089:::benchmark_data_smithi159_102124_object29830:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb9a71a:::benchmark_data_smithi159_102124_object9054:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eb9bfb7:::benchmark_data_smithi159_102124_object6846:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eba65bc:::benchmark_data_smithi159_102124_object42102:head# 2024-04-15T18:22:17.892 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ebc42df:::benchmark_data_smithi159_102124_object45811:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ebcafa4:::benchmark_data_smithi159_102124_object37247:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ebd6396:::benchmark_data_smithi159_102124_object42887:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ebeea2e:::benchmark_data_smithi159_102124_object44776:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec46b69:::benchmark_data_smithi159_102124_object34746:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec4d846:::benchmark_data_smithi159_102124_object50721:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec50562:::benchmark_data_smithi159_102124_object52719:head# 2024-04-15T18:22:17.893 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec5f8a4:::benchmark_data_smithi159_102124_object52538:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec68ab5:::benchmark_data_smithi159_102124_object40783:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec749fa:::benchmark_data_smithi159_102124_object8747:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ec9e0da:::benchmark_data_smithi159_102124_object28783:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eca7b01:::benchmark_data_smithi159_102124_object1209:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ecf4dd3:::benchmark_data_smithi159_102124_object22556:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ecf8329:::benchmark_data_smithi159_102124_object13095:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed02566:::benchmark_data_smithi159_102124_object28558:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed0724a:::benchmark_data_smithi159_102124_object42839:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed0cbe9:::benchmark_data_smithi159_102124_object38469:head# 2024-04-15T18:22:17.898 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed46e75:::benchmark_data_smithi159_102124_object14899:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed5681f:::benchmark_data_smithi159_102124_object35147:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed61ded:::benchmark_data_smithi159_102124_object11189:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ed652e5:::benchmark_data_smithi159_102124_object44426:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2edb2510:::benchmark_data_smithi159_102124_object16150:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2edb2704:::benchmark_data_smithi159_102124_object27505:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2edb589f:::benchmark_data_smithi159_102124_object26079:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee0bb3e:::benchmark_data_smithi159_102124_object21271:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee1366a:::benchmark_data_smithi159_102124_object12998:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee2b00d:::benchmark_data_smithi159_102124_object35940:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee3c7a0:::benchmark_data_smithi159_102124_object25368:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee51eeb:::benchmark_data_smithi159_102124_object6287:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee8ee11:::benchmark_data_smithi159_102124_object35080:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ee97dfe:::benchmark_data_smithi159_102124_object40484:head# 2024-04-15T18:22:17.899 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eea458b:::benchmark_data_smithi159_102124_object55629:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eea45e4:::benchmark_data_smithi159_102124_object38615:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eeab11d:::benchmark_data_smithi159_102124_object3748:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eeb23a9:::benchmark_data_smithi159_102124_object48815:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eeb8df5:::benchmark_data_smithi159_102124_object15009:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eec0a1a:::benchmark_data_smithi159_102124_object46988:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eec9297:::benchmark_data_smithi159_102124_object1482:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eecc21b:::benchmark_data_smithi159_102124_object35466:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eedc7c7:::benchmark_data_smithi159_102124_object11848:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eee1fa7:::benchmark_data_smithi159_102124_object21589:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eee3ac5:::benchmark_data_smithi159_102124_object58021:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eeeda35:::benchmark_data_smithi159_102124_object15133:head# 2024-04-15T18:22:17.900 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef11938:::benchmark_data_smithi159_102124_object11049:head# 2024-04-15T18:22:17.903 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef27316:::benchmark_data_smithi159_102124_object17377:head# 2024-04-15T18:22:17.903 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef37237:::benchmark_data_smithi159_102124_object34832:head# 2024-04-15T18:22:17.903 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef409f9:::benchmark_data_smithi159_102124_object5830:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef4308c:::benchmark_data_smithi159_102124_object7594:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef5b993:::benchmark_data_smithi159_102124_object54947:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef66a14:::benchmark_data_smithi159_102124_object20483:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef74f72:::benchmark_data_smithi159_102124_object1094:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef7b2e4:::benchmark_data_smithi159_102124_object36384:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ef7ebc6:::benchmark_data_smithi159_102124_object54429:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efa10a7:::benchmark_data_smithi159_102124_object29757:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efb2570:::benchmark_data_smithi159_102124_object54593:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efbd016:::benchmark_data_smithi159_102124_object55387:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efc108f:::benchmark_data_smithi159_102124_object45953:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efe7e30:::benchmark_data_smithi159_102124_object39537:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2efeaf58:::benchmark_data_smithi159_102124_object22426:head# 2024-04-15T18:22:17.904 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2eff3992:::benchmark_data_smithi159_102124_object57224:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2effd6c4:::benchmark_data_smithi159_102124_object20015:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0278f1:::benchmark_data_smithi159_102124_object5233:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f02efe7:::benchmark_data_smithi159_102124_object37541:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f033268:::benchmark_data_smithi159_102124_object32486:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f038c43:::benchmark_data_smithi159_102124_object43978:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0836b5:::benchmark_data_smithi159_102124_object38821:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f08c4b5:::benchmark_data_smithi159_102124_object22862:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0b2926:::benchmark_data_smithi159_102124_object9584:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0b8706:::benchmark_data_smithi159_102124_object13780:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0b9988:::benchmark_data_smithi159_102124_object20905:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0c2f44:::benchmark_data_smithi159_102124_object26192:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0f0c3c:::benchmark_data_smithi159_102124_object5351:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0f4a61:::benchmark_data_smithi159_102124_object30548:head# 2024-04-15T18:22:17.905 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f0faf79:::benchmark_data_smithi159_102124_object22078:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f11df07:::benchmark_data_smithi159_102124_object25156:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f129fd5:::benchmark_data_smithi159_102124_object11850:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f13f2ac:::benchmark_data_smithi159_102124_object22471:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f142a3c:::benchmark_data_smithi159_102124_object34591:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f14ece7:::benchmark_data_smithi159_102124_object39106:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f15bee1:::benchmark_data_smithi159_102124_object48030:head# 2024-04-15T18:22:17.909 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f160b0b:::benchmark_data_smithi159_102124_object52793:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f17d1da:::benchmark_data_smithi159_102124_object31197:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f19a9a9:::benchmark_data_smithi159_102124_object48498:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f1dbf82:::benchmark_data_smithi159_102124_object42582:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f1ddd90:::benchmark_data_smithi159_102124_object50501:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f1dfde1:::benchmark_data_smithi159_102124_object17768:head# 2024-04-15T18:22:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f1fa06b:::benchmark_data_smithi159_102124_object41817:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f20a258:::benchmark_data_smithi159_102124_object10041:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f218eed:::benchmark_data_smithi159_102124_object28824:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f23ff26:::benchmark_data_smithi159_102124_object18300:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f244efa:::benchmark_data_smithi159_102124_object57759:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f2660d7:::benchmark_data_smithi159_102124_object57695:head# 2024-04-15T18:22:17.911 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f270fae:::benchmark_data_smithi159_102124_object31296:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f293efc:::benchmark_data_smithi159_102124_object1403:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f299c39:::benchmark_data_smithi159_102124_object31868:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f2c3556:::benchmark_data_smithi159_102124_object15052:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f2cd0c4:::benchmark_data_smithi159_102124_object31547:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f2e1808:::benchmark_data_smithi159_102124_object17387:head# 2024-04-15T18:22:17.912 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f2f2526:::benchmark_data_smithi159_102124_object17201:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f30274b:::benchmark_data_smithi159_102124_object10275:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f31099f:::benchmark_data_smithi159_102124_object18041:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f312ba4:::benchmark_data_smithi159_102124_object34514:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f362b04:::benchmark_data_smithi159_102124_object2078:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f364bdb:::benchmark_data_smithi159_102124_object41717:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f36669c:::benchmark_data_smithi159_102124_object16878:head# 2024-04-15T18:22:17.913 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f36d0b0:::benchmark_data_smithi159_102124_object27941:head# 2024-04-15T18:22:17.923 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f372955:::benchmark_data_smithi159_102124_object25958:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f37418f:::benchmark_data_smithi159_102124_object8492:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3805ef:::benchmark_data_smithi159_102124_object24604:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f383704:::benchmark_data_smithi159_102124_object2703:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f39ce02:::benchmark_data_smithi159_102124_object8841:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3a8691:::benchmark_data_smithi159_102124_object39787:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3b73de:::benchmark_data_smithi159_102124_object33571:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3c46b4:::benchmark_data_smithi159_102124_object26174:head# 2024-04-15T18:22:17.939 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3de9c1:::benchmark_data_smithi159_102124_object12261:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3e2443:::benchmark_data_smithi159_102124_object42456:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f3e85f5:::benchmark_data_smithi159_102124_object33931:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f415768:::benchmark_data_smithi159_102124_object23837:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4360d9:::benchmark_data_smithi159_102124_object10705:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f43a0af:::benchmark_data_smithi159_102124_object49601:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f44da17:::benchmark_data_smithi159_102124_object12435:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4540c4:::benchmark_data_smithi159_102124_object16720:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4726b0:::benchmark_data_smithi159_102124_object22341:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f478d6b:::benchmark_data_smithi159_102124_object41076:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4a8fe9:::benchmark_data_smithi159_102124_object19897:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4ac296:::benchmark_data_smithi159_102124_object16764:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4bf954:::benchmark_data_smithi159_102124_object42210:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4d1eba:::benchmark_data_smithi159_102124_object49958:head# 2024-04-15T18:22:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f4e77e1:::benchmark_data_smithi159_102124_object5222:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f51ff1a:::benchmark_data_smithi159_102124_object8618:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f52160b:::benchmark_data_smithi159_102124_object10303:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f527dca:::benchmark_data_smithi159_102124_object879:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f52e432:::benchmark_data_smithi159_102124_object28798:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f5421c7:::benchmark_data_smithi159_102124_object48121:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f54d7fa:::benchmark_data_smithi159_102124_object37864:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f55addd:::benchmark_data_smithi159_102124_object29789:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f56277a:::benchmark_data_smithi159_102124_object14904:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f56f92e:::benchmark_data_smithi159_102124_object43340:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f5e9b72:::benchmark_data_smithi159_102124_object30929:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f60755a:::benchmark_data_smithi159_102124_object50557:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f611364:::benchmark_data_smithi159_102124_object10448:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f623434:::benchmark_data_smithi159_102124_object52150:head# 2024-04-15T18:22:17.941 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f62dc0a:::benchmark_data_smithi159_102124_object6636:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f63c681:::benchmark_data_smithi159_102124_object24810:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f649c73:::benchmark_data_smithi159_102124_object9843:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f68794b:::benchmark_data_smithi159_102124_object23139:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f69594d:::benchmark_data_smithi159_102124_object24203:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f6b46c5:::benchmark_data_smithi159_102124_object1490:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f6d2d46:::benchmark_data_smithi159_102124_object56829:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f6d81c4:::benchmark_data_smithi159_102124_object40755:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f6d9333:::benchmark_data_smithi159_102124_object7454:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f6fab1a:::benchmark_data_smithi159_102124_object44979:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f70124d:::benchmark_data_smithi159_102124_object43663:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f70dc19:::benchmark_data_smithi159_102124_object8646:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f720a96:::benchmark_data_smithi159_102124_object36684:head# 2024-04-15T18:22:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f724ce3:::benchmark_data_smithi159_102124_object23857:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f73bbb0:::benchmark_data_smithi159_102124_object22794:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f74509f:::benchmark_data_smithi159_102124_object11550:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f74e5ec:::benchmark_data_smithi159_102124_object37504:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f764916:::benchmark_data_smithi159_102124_object45374:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f78b306:::benchmark_data_smithi159_102124_object20839:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f7a5188:::benchmark_data_smithi159_102124_object41092:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f7b8b40:::benchmark_data_smithi159_102124_object57458:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f7dfb89:::benchmark_data_smithi159_102124_object10024:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f7e29d6:::benchmark_data_smithi159_102124_object35196:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f7f7c59:::benchmark_data_smithi159_102124_object14580:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f807b36:::benchmark_data_smithi159_102124_object56523:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f809533:::benchmark_data_smithi159_102124_object45967:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f837f50:::benchmark_data_smithi159_102124_object12564:head# 2024-04-15T18:22:17.943 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f83ee5b:::benchmark_data_smithi159_102124_object31665:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f859373:::benchmark_data_smithi159_102124_object26054:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f86fa33:::benchmark_data_smithi159_102124_object17942:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f874b7e:::benchmark_data_smithi159_102124_object38808:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8ae2e1:::benchmark_data_smithi159_102124_object22865:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8b3f66:::benchmark_data_smithi159_102124_object12784:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8bf5b7:::benchmark_data_smithi159_102124_object3045:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8f3aed:::benchmark_data_smithi159_102124_object56455:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8f953c:::benchmark_data_smithi159_102124_object47619:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f8fd23d:::benchmark_data_smithi159_102124_object35857:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f9344a8:::benchmark_data_smithi159_102124_object15496:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f970dd0:::benchmark_data_smithi159_102124_object28554:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f97c770:::benchmark_data_smithi159_102124_object45294:head# 2024-04-15T18:22:17.944 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f98fb3d:::benchmark_data_smithi159_102124_object52544:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f99e5f6:::benchmark_data_smithi159_102124_object28140:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f99e70c:::benchmark_data_smithi159_102124_object54793:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f9cda81:::benchmark_data_smithi159_102124_object35930:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f9dc702:::benchmark_data_smithi159_102124_object54967:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2f9ff58a:::benchmark_data_smithi159_102124_object36527:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa22310:::benchmark_data_smithi159_102124_object20936:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa39c67:::benchmark_data_smithi159_102124_object41040:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa3a4b5:::benchmark_data_smithi159_102124_object8624:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa3c01c:::benchmark_data_smithi159_102124_object17857:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa5312a:::benchmark_data_smithi159_102124_object26942:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa5b8a8:::benchmark_data_smithi159_102124_object48363:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa60005:::benchmark_data_smithi159_102124_object34726:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa63d37:::benchmark_data_smithi159_102124_object19265:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa7d4ba:::benchmark_data_smithi159_102124_object23527:head# 2024-04-15T18:22:17.945 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fa9318c:::benchmark_data_smithi159_102124_object56245:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2faae820:::benchmark_data_smithi159_102124_object18493:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fab6849:::benchmark_data_smithi159_102124_object7287:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fabc312:::benchmark_data_smithi159_102124_object14542:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fac3e90:::benchmark_data_smithi159_102124_object24084:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fac512b:::benchmark_data_smithi159_102124_object1772:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fad57cb:::benchmark_data_smithi159_102124_object5497:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fae636a:::benchmark_data_smithi159_102124_object11892:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2faebfce:::benchmark_data_smithi159_102124_object22228:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2faf8d42:::benchmark_data_smithi159_102124_object29064:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb12cab:::benchmark_data_smithi159_102124_object25044:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb184d8:::benchmark_data_smithi159_102124_object56917:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb1b0f0:::benchmark_data_smithi159_102124_object47187:head# 2024-04-15T18:22:17.946 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb2012d:::benchmark_data_smithi159_102124_object778:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb23f89:::benchmark_data_smithi159_102124_object53587:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb2733e:::benchmark_data_smithi159_102124_object22184:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb4f65b:::benchmark_data_smithi159_102124_object51429:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb69160:::benchmark_data_smithi159_102124_object41740:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb70ca1:::benchmark_data_smithi159_102124_object22903:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fb90a25:::benchmark_data_smithi159_102124_object13657:head# 2024-04-15T18:22:17.947 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fba481d:::benchmark_data_smithi159_102124_object37217:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fba876f:::benchmark_data_smithi159_102124_object57486:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fbdabf9:::benchmark_data_smithi159_102124_object32678:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fbe9342:::benchmark_data_smithi159_102124_object37441:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc22832:::benchmark_data_smithi159_102124_object30470:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc2596a:::benchmark_data_smithi159_102124_object2118:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc2c7d9:::benchmark_data_smithi159_102124_object35154:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc30b7a:::benchmark_data_smithi159_102124_object21811:head# 2024-04-15T18:22:17.948 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc33bca:::benchmark_data_smithi159_102124_object33372:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc35d99:::benchmark_data_smithi159_102124_object9738:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc5cb3c:::benchmark_data_smithi159_102124_object37443:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fc67def:::benchmark_data_smithi159_102124_object20300:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fcacdef:::benchmark_data_smithi159_102124_object54339:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fcb0120:::benchmark_data_smithi159_102124_object6578:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fcb14a9:::benchmark_data_smithi159_102124_object35603:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fcec6a3:::benchmark_data_smithi159_102124_object41257:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd18ab9:::benchmark_data_smithi159_102124_object1538:head# 2024-04-15T18:22:17.949 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd19f1b:::benchmark_data_smithi159_102124_object31109:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd23f6f:::benchmark_data_smithi159_102124_object1075:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd4ca13:::benchmark_data_smithi159_102124_object55577:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd84bdd:::benchmark_data_smithi159_102124_object12459:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fd9892f:::benchmark_data_smithi159_102124_object41984:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fda3ed7:::benchmark_data_smithi159_102124_object17255:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fdb1365:::benchmark_data_smithi159_102124_object5364:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fdb63bd:::benchmark_data_smithi159_102124_object7666:head# 2024-04-15T18:22:17.950 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fdfca84:::benchmark_data_smithi159_102124_object40776:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fe2955f:::benchmark_data_smithi159_102124_object32740:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fe61221:::benchmark_data_smithi159_102124_object16668:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fe701fc:::benchmark_data_smithi159_102124_object55678:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fe7486e:::benchmark_data_smithi159_102124_object40035:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fe77adc:::benchmark_data_smithi159_102124_object22746:head# 2024-04-15T18:22:17.951 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fec0fce:::benchmark_data_smithi159_102124_object32817:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fec747d:::benchmark_data_smithi159_102124_object56588:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fed04b0:::benchmark_data_smithi159_102124_object31775:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fedcd96:::benchmark_data_smithi159_102124_object37925:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fee11ce:::benchmark_data_smithi159_102124_object48463:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2feee8a2:::benchmark_data_smithi159_102124_object51159:head# 2024-04-15T18:22:17.952 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fef5fd0:::benchmark_data_smithi159_102124_object25057:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fef78ec:::benchmark_data_smithi159_102124_object54257:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fefec6c:::benchmark_data_smithi159_102124_object18559:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff07bb9:::benchmark_data_smithi159_102124_object46438:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff2605f:::benchmark_data_smithi159_102124_object55334:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff344c0:::benchmark_data_smithi159_102124_object47333:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff3b058:::benchmark_data_smithi159_102124_object47290:head# 2024-04-15T18:22:17.953 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff4110c:::benchmark_data_smithi159_102124_object48579:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff645a3:::benchmark_data_smithi159_102124_object7148:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff674b0:::benchmark_data_smithi159_102124_object8429:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff69e15:::benchmark_data_smithi159_102124_object43153:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff6bbf8:::benchmark_data_smithi159_102124_object50223:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff71c7e:::benchmark_data_smithi159_102124_object43717:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ff7bf3f:::benchmark_data_smithi159_102124_object246:head# 2024-04-15T18:22:17.954 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ffa18e6:::benchmark_data_smithi159_102124_object26787:head# 2024-04-15T18:22:17.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ffb6205:::benchmark_data_smithi159_102124_object32885:head# 2024-04-15T18:22:17.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ffc567c:::benchmark_data_smithi159_102124_object46423:head# 2024-04-15T18:22:17.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2ffd0a2d:::benchmark_data_smithi159_102124_object4809:head# 2024-04-15T18:22:17.955 INFO:teuthology.orchestra.run.smithi049.stderr:Read #4:2fff5f73:::benchmark_data_smithi159_102124_object14040:head# 2024-04-15T18:22:17.955 INFO:teuthology.orchestra.run.smithi049.stderr:Export successful 2024-04-15T18:22:17.956 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:17.957 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:17.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:18.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:18.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:22:18.094 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:18.206 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:18.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:18.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:18.367 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:18.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:18 smithi049 ceph-mon[40087]: pgmap v888: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 34 MiB/s wr, 737 op/s; 7735/263556 objects degraded (2.935%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:18.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:18 smithi049 ceph-mon[40087]: 4.1 scrub starts 2024-04-15T18:22:18.562 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:18 smithi082 ceph-mon[44206]: pgmap v888: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 34 MiB/s wr, 737 op/s; 7735/263556 objects degraded (2.935%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:18.563 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:18 smithi082 ceph-mon[44206]: 4.1 scrub starts 2024-04-15T18:22:18.607 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:18.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:18.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:18 smithi159 ceph-mon[45336]: pgmap v888: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 34 MiB/s wr, 737 op/s; 7735/263556 objects degraded (2.935%); 7.0 MiB/s, 111 objects/s recovering 2024-04-15T18:22:18.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:18 smithi159 ceph-mon[45336]: 4.1 scrub starts 2024-04-15T18:22:18.711 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:18.750 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:18.780 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:18.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:18.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:18.946 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:18.984 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:18.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:19.086 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:19.123 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:19.146 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:19.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:19.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:19.279 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:19.300 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:19.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3315478583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:19.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:19.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:22:19.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:22:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:22:19.444 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:19.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3315478583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3315478583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:20.194 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:20.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:20.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:20.334 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:20.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: 4.1 scrub ok 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: pgmap v889: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 717 op/s; 7418/259191 objects degraded (2.862%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:22:20.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: 4.1 scrub ok 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: pgmap v889: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 717 op/s; 7418/259191 objects degraded (2.862%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:22:20.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: 4.1 scrub ok 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: pgmap v889: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 22 MiB/s wr, 717 op/s; 7418/259191 objects degraded (2.862%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:22:20.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:22:20.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:22:21.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.2 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.4 2024-04-15T18:22:21.329 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:21.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:21.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:21.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:21.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:21.695 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:21.699 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.2/config 2024-04-15T18:22:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:21.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:21.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:21.825 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:22:21.825 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:22:21.854 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:22.062 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:22.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:22.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:22.219 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:22.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:22 smithi049 ceph-mon[40087]: pgmap v890: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 4.4 MiB/s wr, 642 op/s; 7418/254319 objects degraded (2.917%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:22:22.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:22 smithi049 ceph-mon[40087]: osdmap e325: 12 total, 11 up, 11 in 2024-04-15T18:22:22.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 7418/254319 objects degraded (2.917%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:22.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:22 smithi082 ceph-mon[44206]: pgmap v890: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 4.4 MiB/s wr, 642 op/s; 7418/254319 objects degraded (2.917%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:22:22.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:22 smithi082 ceph-mon[44206]: osdmap e325: 12 total, 11 up, 11 in 2024-04-15T18:22:22.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 7418/254319 objects degraded (2.917%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:22.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:22:22.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:22 smithi159 ceph-mon[45336]: pgmap v890: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 4.4 MiB/s wr, 642 op/s; 7418/254319 objects degraded (2.917%); 4.7 MiB/s, 75 objects/s recovering 2024-04-15T18:22:22.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:22 smithi159 ceph-mon[45336]: osdmap e325: 12 total, 11 up, 11 in 2024-04-15T18:22:22.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 7418/254319 objects degraded (2.917%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:22.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:22:23.010 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:23.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:23.115 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:23.161 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:23.197 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:23.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:23.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:23 smithi049 ceph-mon[40087]: 4.9 scrub starts 2024-04-15T18:22:23.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:23.338 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:23.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:23.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:23.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:23.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:23 smithi082 ceph-mon[44206]: 4.9 scrub starts 2024-04-15T18:22:23.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:23 smithi159 ceph-mon[45336]: 4.9 scrub starts 2024-04-15T18:22:24.341 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:24.361 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:24.367 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:24.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:24.469 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:24.505 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:24 smithi082 ceph-mon[44206]: pgmap v892: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 1.2 MiB/s wr, 881 op/s; 6859/246204 objects degraded (2.786%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:22:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:24 smithi082 ceph-mon[44206]: 4.9 scrub ok 2024-04-15T18:22:24.537 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:24.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:24.542 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:24 smithi159 ceph-mon[45336]: pgmap v892: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 1.2 MiB/s wr, 881 op/s; 6859/246204 objects degraded (2.786%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:22:24.542 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:24 smithi159 ceph-mon[45336]: 4.9 scrub ok 2024-04-15T18:22:24.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:24.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:24 smithi049 ceph-mon[40087]: pgmap v892: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 1.2 MiB/s wr, 881 op/s; 6859/246204 objects degraded (2.786%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:22:24.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:24 smithi049 ceph-mon[40087]: 4.9 scrub ok 2024-04-15T18:22:24.679 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:25.943 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:25.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:26.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:26.084 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:26.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:26 smithi082 ceph-mon[44206]: pgmap v893: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 102 B/s rd, 960 KiB/s wr, 821 op/s; 6859/242790 objects degraded (2.825%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:22:26.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:26 smithi159 ceph-mon[45336]: pgmap v893: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 102 B/s rd, 960 KiB/s wr, 821 op/s; 6859/242790 objects degraded (2.825%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:22:26.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:26 smithi049 ceph-mon[40087]: pgmap v893: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 102 B/s rd, 960 KiB/s wr, 821 op/s; 6859/242790 objects degraded (2.825%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:22:27.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:22:27.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 6859/242790 objects degraded (2.825%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:27.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:22:27.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 6859/242790 objects degraded (2.825%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:27.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:22:27.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 6859/242790 objects degraded (2.825%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:28.074 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:28.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:28.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:28.214 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:28.236 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:28.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:28.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:28 smithi049 ceph-mon[40087]: pgmap v894: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 102 B/s rd, 102 B/s wr, 866 op/s; 6643/237552 objects degraded (2.796%); 6.8 MiB/s, 109 objects/s recovering 2024-04-15T18:22:28.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:28 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:28.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:22:28.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:28 smithi049 ceph-mon[40087]: osdmap e326: 12 total, 11 up, 11 in 2024-04-15T18:22:28.341 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:28.378 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:28.406 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:28.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:28.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:28.533 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:22:28.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:28 smithi082 ceph-mon[44206]: pgmap v894: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 102 B/s rd, 102 B/s wr, 866 op/s; 6643/237552 objects degraded (2.796%); 6.8 MiB/s, 109 objects/s recovering 2024-04-15T18:22:28.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:28 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:28.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:22:28.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:28 smithi082 ceph-mon[44206]: osdmap e326: 12 total, 11 up, 11 in 2024-04-15T18:22:28.558 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:28.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:28 smithi159 ceph-mon[45336]: pgmap v894: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 102 B/s rd, 102 B/s wr, 866 op/s; 6643/237552 objects degraded (2.796%); 6.8 MiB/s, 109 objects/s recovering 2024-04-15T18:22:28.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:28 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:28.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/14479921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:22:28.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:28 smithi159 ceph-mon[45336]: osdmap e326: 12 total, 11 up, 11 in 2024-04-15T18:22:28.823 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:28.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:28.934 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:28.972 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:28.996 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:28.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:29.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:29.119 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:22:29.119 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:103840 4.4 4:20e07980:::benchmark_data_smithi159_102124_object14486:head [delete] snapc 0=[] ondisk+write+known_if_redirected e326)", 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:29.102889+0000", 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.019746146999999999, 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019775694, 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:22:29.120 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 103840 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102889+0000", 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102889+0000", 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102890+0000", 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7579999999999999e-06 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102898+0000", 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5229999999999998e-06 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102913+0000", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5088e-05 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.102923+0000", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0034000000000001e-05 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.103252+0000", 2024-04-15T18:22:29.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032910200000000001 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.103381+0000", 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129321 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.103466+0000", 2024-04-15T18:22:29.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4295000000000005e-05 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.112509+0000", 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0090436300000000004 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:22:29.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.121003+0000", 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084941429999999991 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:103841 4.4 4:22caf849:::benchmark_data_smithi159_102124_object14487:head [delete] snapc 0=[] ondisk+write+known_if_redirected e326)", 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:29.106670+0000", 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.015964496000000002, 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016090966000000002, 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:22:29.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 103841 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106670+0000", 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106670+0000", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106672+0000", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.979e-06 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106692+0000", 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0211e-05 2024-04-15T18:22:29.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106711+0000", 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593e-05 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.106723+0000", 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2295e-05 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:22:29.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.112548+0000", 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058244029999999997 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.112710+0000", 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016234800000000001 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.112852+0000", 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000141674 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.118743+0000", 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058914029999999999 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.121026+0000", 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002282582 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.131 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:103843 4.4 4:2ba5dcce:::benchmark_data_smithi159_102124_object14489:head [delete] snapc 0=[] ondisk+write+known_if_redirected e326)", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:29.110895+0000", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.011739827, 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011961382, 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 103843 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110895+0000", 2024-04-15T18:22:29.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110895+0000", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110897+0000", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.68e-06 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110903+0000", 2024-04-15T18:22:29.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1079999999999998e-06 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110916+0000", 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3281999999999999e-05 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.110926+0000", 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0069e-05 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:22:29.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.115974+0000", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050483780000000001 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.116124+0000", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014967299999999999 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.116260+0000", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013616400000000001 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:22:29.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.120979+0000", 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047190579999999999 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:103846 4.4 4:2b52212a:::benchmark_data_smithi159_102124_object14492:head [delete] snapc 0=[] ondisk+write+known_if_redirected e326)", 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:29.119587+0000", 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.003047728, 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033513589999999999, 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:22:29.136 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 103846 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119587+0000", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119587+0000", 2024-04-15T18:22:29.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119588+0000", 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.536e-06 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119594+0000", 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7450000000000001e-06 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:22:29.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119607+0000", 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.325e-05 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.119616+0000", 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.028e-06 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.121043+0000", 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001427909 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.121144+0000", 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010052699999999999 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:29.121228+0000", 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3634000000000006e-05 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 4 2024-04-15T18:22:29.140 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:22:29.152 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:29.289 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:29.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:29.299 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1024322437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:29.300 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:29.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:29.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1024322437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:29.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:29.449 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:29.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:29.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1024322437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:29.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:29.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:29.848 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:29.873 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:29.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:30.021 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:30.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: pgmap v896: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 904 op/s; 6284/232605 objects degraded (2.702%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.304 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: pgmap v896: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 904 op/s; 6284/232605 objects degraded (2.702%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.656 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: pgmap v896: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 904 op/s; 6284/232605 objects degraded (2.702%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:30.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:31.141 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:22:31.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:22:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:22:31.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:22:31.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:22:31.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:22:31.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:22:31.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:22:31.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:22:31.790 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:31.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:22:31.834 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:22:31.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:31.927 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:31.956 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:31.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:32.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:32.123 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:32.162 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:32.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:32.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:32.304 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:32.332 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:32.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:32.432 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:32.436 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:32.451 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:32.470 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:32 smithi049 ceph-mon[40087]: pgmap v897: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 6284/227175 objects degraded (2.766%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 6284/227175 objects degraded (2.766%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:32.499 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:32.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:32.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:32.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:32 smithi082 ceph-mon[44206]: pgmap v897: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 6284/227175 objects degraded (2.766%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:32.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 6284/227175 objects degraded (2.766%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:32.636 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:32 smithi159 ceph-mon[45336]: pgmap v897: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 6284/227175 objects degraded (2.766%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:32.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 6284/227175 objects degraded (2.766%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:32.660 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:32.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:32.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:32.825 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:33.402 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:33.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:33.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:33.572 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:33.705 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-04-15T18:22:33.705 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-04-15T18:22:33.705 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 4.4_head removing 4.4 2024-04-15T18:22:33.787 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-04-15T18:22:33.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:22:34.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:34 smithi049 ceph-mon[40087]: pgmap v898: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5687/218244 objects degraded (2.606%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:34.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:34 smithi159 ceph-mon[45336]: pgmap v898: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5687/218244 objects degraded (2.606%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:34.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:34 smithi082 ceph-mon[44206]: pgmap v898: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5687/218244 objects degraded (2.606%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:36.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:36 smithi049 ceph-mon[40087]: pgmap v899: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5687/214677 objects degraded (2.649%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:36.542 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:36.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:36.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.2 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.4.2 2024-04-15T18:22:36.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:36 smithi159 ceph-mon[45336]: pgmap v899: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5687/214677 objects degraded (2.649%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:36.657 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:36.699 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:36.727 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:36.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:36.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:36 smithi082 ceph-mon[44206]: pgmap v899: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5687/214677 objects degraded (2.649%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:22:36.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.051 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:37.159 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.2/config 2024-04-15T18:22:37.194 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.216 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:22:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:22:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:22:37.268 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:22:37.268 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:22:37.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:37.336 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5687/214677 objects degraded (2.649%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:37.353 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.378 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:37.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:37.514 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5687/214677 objects degraded (2.649%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:37.515 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.539 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:37.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:37.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5687/214677 objects degraded (2.649%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:37.679 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.704 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:37.810 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:37.839 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:37.866 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:37.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:37.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:38.010 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:38.035 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:38.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:38.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:22:38.138 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:38.173 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:38.197 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:38.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:38.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:38.363 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:22:38.364 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:38.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:38.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:38.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:38.557 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:38.586 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:38.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:38.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:38 smithi159 ceph-mon[45336]: pgmap v900: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5420/209595 objects degraded (2.586%); 7.6 MiB/s, 122 objects/s recovering 2024-04-15T18:22:38.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:38.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:38.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:38 smithi049 ceph-mon[40087]: pgmap v900: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5420/209595 objects degraded (2.586%); 7.6 MiB/s, 122 objects/s recovering 2024-04-15T18:22:38.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:38.734 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:38.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:38 smithi082 ceph-mon[44206]: pgmap v900: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 931 op/s; 5420/209595 objects degraded (2.586%); 7.6 MiB/s, 122 objects/s recovering 2024-04-15T18:22:38.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:39.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:39 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:39.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:39.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:39 smithi049 ceph-mon[40087]: osdmap e327: 12 total, 11 up, 11 in 2024-04-15T18:22:39.609 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:22:39.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:39 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:39.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:39 smithi159 ceph-mon[45336]: osdmap e327: 12 total, 11 up, 11 in 2024-04-15T18:22:39.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:39 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:39.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1378507989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:39.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:39 smithi082 ceph-mon[44206]: osdmap e327: 12 total, 11 up, 11 in 2024-04-15T18:22:40.018 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:40.304 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:40.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:40.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:40.461 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:40.536 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:40.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:40.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:40.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:40 smithi049 ceph-mon[40087]: pgmap v902: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5033/204498 objects degraded (2.461%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:22:40.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:40 smithi049 ceph-mon[40087]: osdmap e328: 12 total, 11 up, 11 in 2024-04-15T18:22:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:40.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:40.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:40.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:40.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:40.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:40 smithi159 ceph-mon[45336]: pgmap v902: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5033/204498 objects degraded (2.461%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:22:40.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:40 smithi159 ceph-mon[45336]: osdmap e328: 12 total, 11 up, 11 in 2024-04-15T18:22:40.675 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:40.696 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:40.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:40.775 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:40.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:40.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:40.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:40 smithi082 ceph-mon[44206]: pgmap v902: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 5033/204498 objects degraded (2.461%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:22:40.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:40 smithi082 ceph-mon[44206]: osdmap e328: 12 total, 11 up, 11 in 2024-04-15T18:22:40.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:40.841 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:40.866 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:40.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:40.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:41.025 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e307 pg_nums {1={19=1},3={199=16},4={284=16}} deleted_pools 280,3) 2024-04-15T18:22:41.025 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 4.4 2024-04-15T18:22:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:41.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2002b423:::benchmark_data_smithi159_102124_object14057:head# 2024-04-15T18:22:41.030 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.030 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20034ce2:::benchmark_data_smithi159_102124_object30971:head# 2024-04-15T18:22:41.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200485e2:::benchmark_data_smithi159_102124_object21128:head# 2024-04-15T18:22:41.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.031 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2005116f:::benchmark_data_smithi159_102124_object27139:head# 2024-04-15T18:22:41.031 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2005c650:::benchmark_data_smithi159_102124_object12512:head# 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20069ff3:::benchmark_data_smithi159_102124_object17449:head# 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2007ebb4:::benchmark_data_smithi159_102124_object35755:head# 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2008198e:::benchmark_data_smithi159_102124_object46986:head# 2024-04-15T18:22:41.032 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2008c0bd:::benchmark_data_smithi159_102124_object12874:head# 2024-04-15T18:22:41.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200c01d8:::benchmark_data_smithi159_102124_object50121:head# 2024-04-15T18:22:41.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200d2aa7:::benchmark_data_smithi159_102124_object39058:head# 2024-04-15T18:22:41.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200e2b33:::benchmark_data_smithi159_102124_object43455:head# 2024-04-15T18:22:41.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200e2b35:::benchmark_data_smithi159_102124_object32979:head# 2024-04-15T18:22:41.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200e92a8:::benchmark_data_smithi159_102124_object24538:head# 2024-04-15T18:22:41.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:200ead45:::benchmark_data_smithi159_102124_object15740:head# 2024-04-15T18:22:41.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20124c03:::benchmark_data_smithi159_102124_object27553:head# 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20128afa:::benchmark_data_smithi159_102124_object42690:head# 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2012c4cf:::benchmark_data_smithi159_102124_object15287:head# 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2012ca35:::benchmark_data_smithi159_102124_object36486:head# 2024-04-15T18:22:41.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2015a747:::benchmark_data_smithi159_102124_object51617:head# 2024-04-15T18:22:41.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.050 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2015c385:::benchmark_data_smithi159_102124_object27400:head# 2024-04-15T18:22:41.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201758c7:::benchmark_data_smithi159_102124_object21390:head# 2024-04-15T18:22:41.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.051 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201827db:::benchmark_data_smithi159_102124_object36460:head# 2024-04-15T18:22:41.051 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201945a3:::benchmark_data_smithi159_102124_object27824:head# 2024-04-15T18:22:41.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20198f2a:::benchmark_data_smithi159_102124_object25295:head# 2024-04-15T18:22:41.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201c5c6e:::benchmark_data_smithi159_102124_object5739:head# 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201eea46:::benchmark_data_smithi159_102124_object28513:head# 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:201ef650:::benchmark_data_smithi159_102124_object41882:head# 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2020e8ed:::benchmark_data_smithi159_102124_object25858:head# 2024-04-15T18:22:41.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2023c2e5:::benchmark_data_smithi159_102124_object57657:head# 2024-04-15T18:22:41.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20255032:::benchmark_data_smithi159_102124_object50145:head# 2024-04-15T18:22:41.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20258d40:::benchmark_data_smithi159_102124_object45326:head# 2024-04-15T18:22:41.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2025df64:::benchmark_data_smithi159_102124_object52586:head# 2024-04-15T18:22:41.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2026ea31:::benchmark_data_smithi159_102124_object5326:head# 2024-04-15T18:22:41.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202a2903:::benchmark_data_smithi159_102124_object30610:head# 2024-04-15T18:22:41.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202acfc7:::benchmark_data_smithi159_102124_object38797:head# 2024-04-15T18:22:41.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.057 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202c814a:::benchmark_data_smithi159_102124_object50573:head# 2024-04-15T18:22:41.057 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202d3099:::benchmark_data_smithi159_102124_object27081:head# 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202e5c09:::benchmark_data_smithi159_102124_object8480:head# 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202eb7d5:::benchmark_data_smithi159_102124_object17694:head# 2024-04-15T18:22:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:202f5b20:::benchmark_data_smithi159_102124_object23565:head# 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20300349:::benchmark_data_smithi159_102124_object26069:head# 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2030ab34:::benchmark_data_smithi159_102124_object4913:head# 2024-04-15T18:22:41.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2031a3a7:::benchmark_data_smithi159_102124_object8840:head# 2024-04-15T18:22:41.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.065 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2031bc86:::benchmark_data_smithi159_102124_object12033:head# 2024-04-15T18:22:41.065 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20336a88:::benchmark_data_smithi159_102124_object17633:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20344fee:::benchmark_data_smithi159_102124_object29997:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20345a40:::benchmark_data_smithi159_102124_object4838:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2034b69a:::benchmark_data_smithi159_102124_object14981:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2035824c:::benchmark_data_smithi159_102124_object53453:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203608eb:::benchmark_data_smithi159_102124_object2093:head# 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.066 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2036aa85:::benchmark_data_smithi159_102124_object25617:head# 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20390c6e:::benchmark_data_smithi159_102124_object53439:head# 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20398480:::benchmark_data_smithi159_102124_object239:head# 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.067 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203a6d3c:::benchmark_data_smithi159_102124_object51633:head# 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203ae260:::benchmark_data_smithi159_102124_object27762:head# 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203bfddb:::benchmark_data_smithi159_102124_object50243:head# 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203cc020:::benchmark_data_smithi159_102124_object19707:head# 2024-04-15T18:22:41.068 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203d3bea:::benchmark_data_smithi159_102124_object11342:head# 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203e6a4c:::benchmark_data_smithi159_102124_object24601:head# 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:203fa210:::benchmark_data_smithi159_102124_object24881:head# 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20417cf1:::benchmark_data_smithi159_102124_object13297:head# 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.069 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2041bd7c:::benchmark_data_smithi159_102124_object2557:head# 2024-04-15T18:22:41.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2042404f:::benchmark_data_smithi159_102124_object8955:head# 2024-04-15T18:22:41.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20425048:::benchmark_data_smithi159_102124_object15991:head# 2024-04-15T18:22:41.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2042589f:::benchmark_data_smithi159_102124_object4825:head# 2024-04-15T18:22:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20441dec:::benchmark_data_smithi159_102124_object27774:head# 2024-04-15T18:22:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:204455cd:::benchmark_data_smithi159_102124_object39666:head# 2024-04-15T18:22:41.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2047bd30:::benchmark_data_smithi159_102124_object16977:head# 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2048866c:::benchmark_data_smithi159_102124_object5078:head# 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2048c8d6:::benchmark_data_smithi159_102124_object18731:head# 2024-04-15T18:22:41.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20495530:::benchmark_data_smithi159_102124_object11873:head# 2024-04-15T18:22:41.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:204b7191:::benchmark_data_smithi159_102124_object43021:head# 2024-04-15T18:22:41.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:204bd1c8:::benchmark_data_smithi159_102124_object15372:head# 2024-04-15T18:22:41.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:204c059f:::benchmark_data_smithi159_102124_object35483:head# 2024-04-15T18:22:41.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:204f8154:::benchmark_data_smithi159_102124_object43462:head# 2024-04-15T18:22:41.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20523036:::benchmark_data_smithi159_102124_object5239:head# 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2052de54:::benchmark_data_smithi159_102124_object22054:head# 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20531996:::benchmark_data_smithi159_102124_object5834:head# 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205454da:::benchmark_data_smithi159_102124_object46974:head# 2024-04-15T18:22:41.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.084 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205565ff:::benchmark_data_smithi159_102124_object34903:head# 2024-04-15T18:22:41.084 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20556f64:::benchmark_data_smithi159_102124_object41820:head# 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2055b78f:::benchmark_data_smithi159_102124_object40211:head# 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2055ebf3:::benchmark_data_smithi159_102124_object36473:head# 2024-04-15T18:22:41.087 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2056d863:::benchmark_data_smithi159_102124_object5776:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2056dc94:::benchmark_data_smithi159_102124_object17662:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20577af0:::benchmark_data_smithi159_102124_object40027:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205889c1:::benchmark_data_smithi159_102124_object17858:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20590408:::benchmark_data_smithi159_102124_object1363:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205ab778:::benchmark_data_smithi159_102124_object17621:head# 2024-04-15T18:22:41.088 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205b40e6:::benchmark_data_smithi159_102124_object19579:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205baad0:::benchmark_data_smithi159_102124_object7601:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205cf416:::benchmark_data_smithi159_102124_object28128:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205df261:::benchmark_data_smithi159_102124_object19490:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205e6d3a:::benchmark_data_smithi159_102124_object55275:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205f0788:::benchmark_data_smithi159_102124_object11154:head# 2024-04-15T18:22:41.089 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205f2217:::benchmark_data_smithi159_102124_object17618:head# 2024-04-15T18:22:41.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.090 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:205f4f56:::benchmark_data_smithi159_102124_object3924:head# 2024-04-15T18:22:41.090 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20607971:::benchmark_data_smithi159_102124_object42872:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20610c77:::benchmark_data_smithi159_102124_object23614:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20615d1d:::benchmark_data_smithi159_102124_object17668:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206386ab:::benchmark_data_smithi159_102124_object49676:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206524a0:::benchmark_data_smithi159_102124_object42563:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206624f3:::benchmark_data_smithi159_102124_object17120:head# 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20677688:::benchmark_data_smithi159_102124_object57316:head# 2024-04-15T18:22:41.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206974cc:::benchmark_data_smithi159_102124_object53293:head# 2024-04-15T18:22:41.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2069fb79:::benchmark_data_smithi159_102124_object1096:head# 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206c12e4:::benchmark_data_smithi159_102124_object27614:head# 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:206eb8a6:::benchmark_data_smithi159_102124_object36408:head# 2024-04-15T18:22:41.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20703b1f:::benchmark_data_smithi159_102124_object34172:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20724302:::benchmark_data_smithi159_102124_object17797:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20739c55:::benchmark_data_smithi159_102124_object50592:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2073ea2d:::benchmark_data_smithi159_102124_object2133:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207757cd:::benchmark_data_smithi159_102124_object33714:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20779e60:::benchmark_data_smithi159_102124_object55158:head# 2024-04-15T18:22:41.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207a3a2c:::benchmark_data_smithi159_102124_object6387:head# 2024-04-15T18:22:41.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207b11a7:::benchmark_data_smithi159_102124_object18272:head# 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207d39f8:::benchmark_data_smithi159_102124_object11330:head# 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207d757e:::benchmark_data_smithi159_102124_object32346:head# 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207ea96d:::benchmark_data_smithi159_102124_object37092:head# 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207ed40c:::benchmark_data_smithi159_102124_object27670:head# 2024-04-15T18:22:41.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:207ffb29:::benchmark_data_smithi159_102124_object36426:head# 2024-04-15T18:22:41.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2080807d:::benchmark_data_smithi159_102124_object7828:head# 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2080d55c:::benchmark_data_smithi159_102124_object39213:head# 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2081c31a:::benchmark_data_smithi159_102124_object42050:head# 2024-04-15T18:22:41.101 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2081d3b6:::benchmark_data_smithi159_102124_object5144:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2082d402:::benchmark_data_smithi159_102124_object56537:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20830ab0:::benchmark_data_smithi159_102124_object480:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208318fd:::benchmark_data_smithi159_102124_object11011:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20857169:::benchmark_data_smithi159_102124_object56020:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208b2b49:::benchmark_data_smithi159_102124_object57852:head# 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208c097c:::benchmark_data_smithi159_102124_object2964:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208c1a44:::benchmark_data_smithi159_102124_object42579:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208c4dab:::benchmark_data_smithi159_102124_object6635:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208c81a4:::benchmark_data_smithi159_102124_object52432:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208caa9d:::benchmark_data_smithi159_102124_object37682:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208cbcfa:::benchmark_data_smithi159_102124_object54502:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208de8c8:::benchmark_data_smithi159_102124_object58424:head# 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:208def1b:::benchmark_data_smithi159_102124_object10018:head# 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20902f8d:::benchmark_data_smithi159_102124_object22983:head# 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2091044e:::benchmark_data_smithi159_102124_object44129:head# 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:209311ac:::benchmark_data_smithi159_102124_object28384:head# 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20982c70:::benchmark_data_smithi159_102124_object22063:head# 2024-04-15T18:22:41.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:209926f8:::benchmark_data_smithi159_102124_object34416:head# 2024-04-15T18:22:41.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:209cd857:::benchmark_data_smithi159_102124_object48211:head# 2024-04-15T18:22:41.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a2f36d:::benchmark_data_smithi159_102124_object48966:head# 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a2f572:::benchmark_data_smithi159_102124_object55257:head# 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a3ae84:::benchmark_data_smithi159_102124_object44235:head# 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a52623:::benchmark_data_smithi159_102124_object47631:head# 2024-04-15T18:22:41.108 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a538d8:::benchmark_data_smithi159_102124_object26502:head# 2024-04-15T18:22:41.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.109 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a6c7c3:::benchmark_data_smithi159_102124_object44614:head# 2024-04-15T18:22:41.109 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.111 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a729b7:::benchmark_data_smithi159_102124_object10166:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20a929d8:::benchmark_data_smithi159_102124_object30223:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20aa2640:::benchmark_data_smithi159_102124_object3212:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ae15d4:::benchmark_data_smithi159_102124_object47168:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20aff8f2:::benchmark_data_smithi159_102124_object40257:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b014f4:::benchmark_data_smithi159_102124_object17661:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b05ff5:::benchmark_data_smithi159_102124_object38310:head# 2024-04-15T18:22:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b06f52:::benchmark_data_smithi159_102124_object11630:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b2be78:::benchmark_data_smithi159_102124_object58053:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b35b4b:::benchmark_data_smithi159_102124_object25779:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b3769b:::benchmark_data_smithi159_102124_object49247:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b37caa:::benchmark_data_smithi159_102124_object41148:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b42c23:::benchmark_data_smithi159_102124_object44121:head# 2024-04-15T18:22:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b62b51:::benchmark_data_smithi159_102124_object19670:head# 2024-04-15T18:22:41.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b65f51:::benchmark_data_smithi159_102124_object4134:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b679ae:::benchmark_data_smithi159_102124_object35689:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b7d5be:::benchmark_data_smithi159_102124_object32836:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b854be:::benchmark_data_smithi159_102124_object20357:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b8d337:::benchmark_data_smithi159_102124_object8926:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b907e1:::benchmark_data_smithi159_102124_object9787:head# 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20b9fb2b:::benchmark_data_smithi159_102124_object864:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ba8b00:::benchmark_data_smithi159_102124_object55394:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20bc1027:::benchmark_data_smithi159_102124_object19448:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20bc5247:::benchmark_data_smithi159_102124_object28839:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20be3603:::benchmark_data_smithi159_102124_object13024:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20bf4b43:::benchmark_data_smithi159_102124_object2999:head# 2024-04-15T18:22:41.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c18f39:::benchmark_data_smithi159_102124_object19236:head# 2024-04-15T18:22:41.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c24a78:::benchmark_data_smithi159_102124_object26655:head# 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c271e4:::benchmark_data_smithi159_102124_object54991:head# 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c32e97:::benchmark_data_smithi159_102124_object757:head# 2024-04-15T18:22:41.121 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c46fe1:::benchmark_data_smithi159_102124_object20860:head# 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c66b8c:::benchmark_data_smithi159_102124_object12720:head# 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c6b1e8:::benchmark_data_smithi159_102124_object48768:head# 2024-04-15T18:22:41.122 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c7b055:::benchmark_data_smithi159_102124_object49920:head# 2024-04-15T18:22:41.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c81674:::benchmark_data_smithi159_102124_object9359:head# 2024-04-15T18:22:41.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c92a65:::benchmark_data_smithi159_102124_object40223:head# 2024-04-15T18:22:41.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20c96c31:::benchmark_data_smithi159_102124_object37575:head# 2024-04-15T18:22:41.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20cab4da:::benchmark_data_smithi159_102124_object40742:head# 2024-04-15T18:22:41.129 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20cb4dfd:::benchmark_data_smithi159_102124_object27477:head# 2024-04-15T18:22:41.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.134 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20cb7a8a:::benchmark_data_smithi159_102124_object7882:head# 2024-04-15T18:22:41.134 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.139 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ced747:::benchmark_data_smithi159_102124_object57165:head# 2024-04-15T18:22:41.139 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.140 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20cfa261:::benchmark_data_smithi159_102124_object52555:head# 2024-04-15T18:22:41.140 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.141 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d0930d:::benchmark_data_smithi159_102124_object11383:head# 2024-04-15T18:22:41.141 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d53418:::benchmark_data_smithi159_102124_object22105:head# 2024-04-15T18:22:41.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.145 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d5355c:::benchmark_data_smithi159_102124_object36607:head# 2024-04-15T18:22:41.145 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d552e9:::benchmark_data_smithi159_102124_object50398:head# 2024-04-15T18:22:41.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d71de0:::benchmark_data_smithi159_102124_object23175:head# 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d8068d:::benchmark_data_smithi159_102124_object54564:head# 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d80fc1:::benchmark_data_smithi159_102124_object46614:head# 2024-04-15T18:22:41.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20d8a32f:::benchmark_data_smithi159_102124_object37271:head# 2024-04-15T18:22:41.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20dbfa1c:::benchmark_data_smithi159_102124_object51889:head# 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e07980:::benchmark_data_smithi159_102124_object14486:head# 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e0e09a:::benchmark_data_smithi159_102124_object52301:head# 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e2eaec:::benchmark_data_smithi159_102124_object46289:head# 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e2f91b:::benchmark_data_smithi159_102124_object46513:head# 2024-04-15T18:22:41.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e4848e:::benchmark_data_smithi159_102124_object23974:head# 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e5587c:::benchmark_data_smithi159_102124_object40972:head# 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e6e9e7:::benchmark_data_smithi159_102124_object19619:head# 2024-04-15T18:22:41.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e7d20c:::benchmark_data_smithi159_102124_object42257:head# 2024-04-15T18:22:41.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e95155:::benchmark_data_smithi159_102124_object37982:head# 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e99436:::benchmark_data_smithi159_102124_object48198:head# 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20e9b9ca:::benchmark_data_smithi159_102124_object24211:head# 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ea7107:::benchmark_data_smithi159_102124_object38655:head# 2024-04-15T18:22:41.164 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ea7da1:::benchmark_data_smithi159_102124_object28980:head# 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ea97ed:::benchmark_data_smithi159_102124_object45060:head# 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20eac977:::benchmark_data_smithi159_102124_object22590:head# 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ec04b1:::benchmark_data_smithi159_102124_object13107:head# 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ecc601:::benchmark_data_smithi159_102124_object26453:head# 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ed23d2:::benchmark_data_smithi159_102124_object53641:head# 2024-04-15T18:22:41.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ed4268:::benchmark_data_smithi159_102124_object19261:head# 2024-04-15T18:22:41.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f0938a:::benchmark_data_smithi159_102124_object22787:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f19296:::benchmark_data_smithi159_102124_object7064:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f1eff3:::benchmark_data_smithi159_102124_object44072:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f32bbb:::benchmark_data_smithi159_102124_object32698:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f34009:::benchmark_data_smithi159_102124_object3985:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f41d99:::benchmark_data_smithi159_102124_object46269:head# 2024-04-15T18:22:41.169 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.170 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f657ea:::benchmark_data_smithi159_102124_object57053:head# 2024-04-15T18:22:41.170 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f7446e:::benchmark_data_smithi159_102124_object23190:head# 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f96faf:::benchmark_data_smithi159_102124_object36069:head# 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20f9d68a:::benchmark_data_smithi159_102124_object13576:head# 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20fcb9be:::benchmark_data_smithi159_102124_object25950:head# 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20fdf52a:::benchmark_data_smithi159_102124_object22930:head# 2024-04-15T18:22:41.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:20ff45a1:::benchmark_data_smithi159_102124_object9143:head# 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2100c3b8:::benchmark_data_smithi159_102124_object1001:head# 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21029175:::benchmark_data_smithi159_102124_object16010:head# 2024-04-15T18:22:41.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21032fcb:::benchmark_data_smithi159_102124_object1278:head# 2024-04-15T18:22:41.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21039aaa:::benchmark_data_smithi159_102124_object18478:head# 2024-04-15T18:22:41.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2103ba37:::benchmark_data_smithi159_102124_object29878:head# 2024-04-15T18:22:41.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.175 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210500bb:::benchmark_data_smithi159_102124_object21264:head# 2024-04-15T18:22:41.175 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21056823:::benchmark_data_smithi159_102124_object4290:head# 2024-04-15T18:22:41.179 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2105e3f0:::benchmark_data_smithi159_102124_object15791:head# 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21080ad4:::benchmark_data_smithi159_102124_object21338:head# 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2108d6ba:::benchmark_data_smithi159_102124_object27800:head# 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210955e4:::benchmark_data_smithi159_102124_object34321:head# 2024-04-15T18:22:41.180 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210abee9:::benchmark_data_smithi159_102124_object52623:head# 2024-04-15T18:22:41.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210cde2e:::benchmark_data_smithi159_102124_object40360:head# 2024-04-15T18:22:41.181 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.181 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210e0003:::benchmark_data_smithi159_102124_object44886:head# 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210e9b26:::benchmark_data_smithi159_102124_object42547:head# 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210ec994:::benchmark_data_smithi159_102124_object47362:head# 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210ed748:::benchmark_data_smithi159_102124_object16425:head# 2024-04-15T18:22:41.182 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.183 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:210f5a5e:::benchmark_data_smithi159_102124_object22173:head# 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21119a05:::benchmark_data_smithi159_102124_object23401:head# 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21158612:::benchmark_data_smithi159_102124_object13237:head# 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211727b8:::benchmark_data_smithi159_102124_object14592:head# 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2117f622:::benchmark_data_smithi159_102124_object11718:head# 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2117fd0b:::benchmark_data_smithi159_102124_object17956:head# 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211b544b:::benchmark_data_smithi159_102124_object19908:head# 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211c0d93:::benchmark_data_smithi159_102124_object29780:head# 2024-04-15T18:22:41.186 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211c5f23:::benchmark_data_smithi159_102124_object54321:head# 2024-04-15T18:22:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211d3a83:::benchmark_data_smithi159_102124_object961:head# 2024-04-15T18:22:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.187 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211db8e2:::benchmark_data_smithi159_102124_object47510:head# 2024-04-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:211e59ed:::benchmark_data_smithi159_102124_object387:head# 2024-04-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.195 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212063af:::benchmark_data_smithi159_102124_object8957:head# 2024-04-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212263d2:::benchmark_data_smithi159_102124_object29752:head# 2024-04-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212496c3:::benchmark_data_smithi159_102124_object26091:head# 2024-04-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21256713:::benchmark_data_smithi159_102124_object149:head# 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2127500f:::benchmark_data_smithi159_102124_object32025:head# 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2129a303:::benchmark_data_smithi159_102124_object30412:head# 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212a6839:::benchmark_data_smithi159_102124_object4120:head# 2024-04-15T18:22:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212a7969:::benchmark_data_smithi159_102124_object23776:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212b1d80:::benchmark_data_smithi159_102124_object14003:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212b2837:::benchmark_data_smithi159_102124_object58315:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212b2ac9:::benchmark_data_smithi159_102124_object15676:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212ba875:::benchmark_data_smithi159_102124_object31129:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212bb32d:::benchmark_data_smithi159_102124_object18027:head# 2024-04-15T18:22:41.198 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212cff9a:::benchmark_data_smithi159_102124_object23718:head# 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212d1f8d:::benchmark_data_smithi159_102124_object54215:head# 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:212d2466:::benchmark_data_smithi159_102124_object27866:head# 2024-04-15T18:22:41.199 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21307b55:::benchmark_data_smithi159_102124_object5383:head# 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2130a8dc:::benchmark_data_smithi159_102124_object22375:head# 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2130aa32:::benchmark_data_smithi159_102124_object30275:head# 2024-04-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21316319:::benchmark_data_smithi159_102124_object27267:head# 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2132d0da:::benchmark_data_smithi159_102124_object52001:head# 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21355151:::benchmark_data_smithi159_102124_object52577:head# 2024-04-15T18:22:41.201 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2136f866:::benchmark_data_smithi159_102124_object13139:head# 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2137a84a:::benchmark_data_smithi159_102124_object9776:head# 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213820c4:::benchmark_data_smithi159_102124_object18121:head# 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2139128c:::benchmark_data_smithi159_102124_object51409:head# 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213acd08:::benchmark_data_smithi159_102124_object51205:head# 2024-04-15T18:22:41.202 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213b7ad4:::benchmark_data_smithi159_102124_object58282:head# 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213b9087:::benchmark_data_smithi159_102124_object31877:head# 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213cf459:::benchmark_data_smithi159_102124_object10708:head# 2024-04-15T18:22:41.203 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213f46f6:::benchmark_data_smithi159_102124_object7836:head# 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:213fd50a:::benchmark_data_smithi159_102124_object12673:head# 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:214055c6:::benchmark_data_smithi159_102124_object45632:head# 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.204 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21409619:::benchmark_data_smithi159_102124_object25675:head# 2024-04-15T18:22:41.205 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2140f065:::benchmark_data_smithi159_102124_object726:head# 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2142327c:::benchmark_data_smithi159_102124_object22962:head# 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21430502:::benchmark_data_smithi159_102124_object8606:head# 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2143cf58:::benchmark_data_smithi159_102124_object42326:head# 2024-04-15T18:22:41.208 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2144c5e9:::benchmark_data_smithi159_102124_object46492:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21450511:::benchmark_data_smithi159_102124_object29553:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21472c3c:::benchmark_data_smithi159_102124_object690:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2149db68:::benchmark_data_smithi159_102124_object48695:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:214a93aa:::benchmark_data_smithi159_102124_object26912:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:214b1d5f:::benchmark_data_smithi159_102124_object35460:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:214c1641:::benchmark_data_smithi159_102124_object5649:head# 2024-04-15T18:22:41.209 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215011bd:::benchmark_data_smithi159_102124_object42192:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21503753:::benchmark_data_smithi159_102124_object31352:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2150924b:::benchmark_data_smithi159_102124_object4490:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21511b04:::benchmark_data_smithi159_102124_object55808:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21516d3e:::benchmark_data_smithi159_102124_object15856:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215449ad:::benchmark_data_smithi159_102124_object37484:head# 2024-04-15T18:22:41.210 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21550dcc:::benchmark_data_smithi159_102124_object42365:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21552d7b:::benchmark_data_smithi159_102124_object33204:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21589327:::benchmark_data_smithi159_102124_object43601:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215939a8:::benchmark_data_smithi159_102124_object15382:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215b27a1:::benchmark_data_smithi159_102124_object56026:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215d0e2a:::benchmark_data_smithi159_102124_object15131:head# 2024-04-15T18:22:41.211 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215d8615:::benchmark_data_smithi159_102124_object20445:head# 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215e974c:::benchmark_data_smithi159_102124_object45159:head# 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:215faa8d:::benchmark_data_smithi159_102124_object40904:head# 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2162a325:::benchmark_data_smithi159_102124_object15219:head# 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.212 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2162fc02:::benchmark_data_smithi159_102124_object5582:head# 2024-04-15T18:22:41.213 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.213 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2163882f:::benchmark_data_smithi159_102124_object57278:head# 2024-04-15T18:22:41.213 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.213 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21641e34:::benchmark_data_smithi159_102124_object49395:head# 2024-04-15T18:22:41.213 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.220 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216424e9:::benchmark_data_smithi159_102124_object49156:head# 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2164a658:::benchmark_data_smithi159_102124_object44761:head# 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21650214:::benchmark_data_smithi159_102124_object116:head# 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2167af4f:::benchmark_data_smithi159_102124_object58171:head# 2024-04-15T18:22:41.221 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2168b20e:::benchmark_data_smithi159_102124_object37719:head# 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2169320c:::benchmark_data_smithi159_102124_object44242:head# 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21694f72:::benchmark_data_smithi159_102124_object37819:head# 2024-04-15T18:22:41.222 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216a1960:::benchmark_data_smithi159_102124_object11013:head# 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216a523c:::benchmark_data_smithi159_102124_object11307:head# 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216ae693:::benchmark_data_smithi159_102124_object5251:head# 2024-04-15T18:22:41.223 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216c5ae0:::benchmark_data_smithi159_102124_object26065:head# 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216e1101:::benchmark_data_smithi159_102124_object26576:head# 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216e20f6:::benchmark_data_smithi159_102124_object3703:head# 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.224 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216e2512:::benchmark_data_smithi159_102124_object40780:head# 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:216fe97b:::benchmark_data_smithi159_102124_object14951:head# 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21713d86:::benchmark_data_smithi159_102124_object38842:head# 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.225 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2172cfd6:::benchmark_data_smithi159_102124_object46892:head# 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2174db49:::benchmark_data_smithi159_102124_object9415:head# 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2175c08a:::benchmark_data_smithi159_102124_object47639:head# 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2176dae6:::benchmark_data_smithi159_102124_object41772:head# 2024-04-15T18:22:41.226 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:217813af:::benchmark_data_smithi159_102124_object57448:head# 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21787f2d:::benchmark_data_smithi159_102124_object43890:head# 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21794478:::benchmark_data_smithi159_102124_object35801:head# 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.227 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:217a39d4:::benchmark_data_smithi159_102124_object20977:head# 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:217a75bf:::benchmark_data_smithi159_102124_object50927:head# 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2180526c:::benchmark_data_smithi159_102124_object21981:head# 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218148dc:::benchmark_data_smithi159_102124_object33967:head# 2024-04-15T18:22:41.228 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2181b82a:::benchmark_data_smithi159_102124_object52265:head# 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21820892:::benchmark_data_smithi159_102124_object33839:head# 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21827e4f:::benchmark_data_smithi159_102124_object41071:head# 2024-04-15T18:22:41.232 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2182af08:::benchmark_data_smithi159_102124_object16885:head# 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2182d27d:::benchmark_data_smithi159_102124_object22128:head# 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2184bfa7:::benchmark_data_smithi159_102124_object44792:head# 2024-04-15T18:22:41.233 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21858c83:::benchmark_data_smithi159_102124_object10588:head# 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2185b428:::benchmark_data_smithi159_102124_object53861:head# 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218792ed:::benchmark_data_smithi159_102124_object31699:head# 2024-04-15T18:22:41.234 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2187feff:::benchmark_data_smithi159_102124_object12684:head# 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2188e7d7:::benchmark_data_smithi159_102124_object21449:head# 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218c6f77:::benchmark_data_smithi159_102124_object9642:head# 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218e813a:::benchmark_data_smithi159_102124_object13699:head# 2024-04-15T18:22:41.235 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218e9416:::benchmark_data_smithi159_102124_object11683:head# 2024-04-15T18:22:41.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.236 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:218f0f5d:::benchmark_data_smithi159_102124_object1040:head# 2024-04-15T18:22:41.236 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.237 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21903bbf:::benchmark_data_smithi159_102124_object5439:head# 2024-04-15T18:22:41.237 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.237 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21909b02:::benchmark_data_smithi159_102124_object47834:head# 2024-04-15T18:22:41.237 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2190e58d:::benchmark_data_smithi159_102124_object39893:head# 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21917caa:::benchmark_data_smithi159_102124_object37805:head# 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219269cd:::benchmark_data_smithi159_102124_object26366:head# 2024-04-15T18:22:41.238 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.239 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2192ee1a:::benchmark_data_smithi159_102124_object50117:head# 2024-04-15T18:22:41.239 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2193ac2f:::benchmark_data_smithi159_102124_object14564:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2194fb9c:::benchmark_data_smithi159_102124_object47975:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219711c8:::benchmark_data_smithi159_102124_object30314:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21978e80:::benchmark_data_smithi159_102124_object37662:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2199906f:::benchmark_data_smithi159_102124_object20320:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2199e215:::benchmark_data_smithi159_102124_object44915:head# 2024-04-15T18:22:41.243 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219b1150:::benchmark_data_smithi159_102124_object32766:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219c6000:::benchmark_data_smithi159_102124_object39600:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219d3fac:::benchmark_data_smithi159_102124_object35418:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219def4c:::benchmark_data_smithi159_102124_object11194:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219e509a:::benchmark_data_smithi159_102124_object40654:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219e5594:::benchmark_data_smithi159_102124_object34058:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219e594d:::benchmark_data_smithi159_102124_object3269:head# 2024-04-15T18:22:41.244 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:219fcc22:::benchmark_data_smithi159_102124_object16745:head# 2024-04-15T18:22:41.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.245 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a03b7b:::benchmark_data_smithi159_102124_object8598:head# 2024-04-15T18:22:41.245 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a0c15a:::benchmark_data_smithi159_102124_object1865:head# 2024-04-15T18:22:41.247 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.247 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a28b3e:::benchmark_data_smithi159_102124_object1638:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a2fbee:::benchmark_data_smithi159_102124_object15896:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a3d4a3:::benchmark_data_smithi159_102124_object29788:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a4b1ad:::benchmark_data_smithi159_102124_object32033:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a714ab:::benchmark_data_smithi159_102124_object47735:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a740d5:::benchmark_data_smithi159_102124_object35604:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21a760b4:::benchmark_data_smithi159_102124_object16044:head# 2024-04-15T18:22:41.248 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21aa3209:::benchmark_data_smithi159_102124_object731:head# 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ab302f:::benchmark_data_smithi159_102124_object20259:head# 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21abd52d:::benchmark_data_smithi159_102124_object51752:head# 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21abda6d:::benchmark_data_smithi159_102124_object23476:head# 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.249 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ac223a:::benchmark_data_smithi159_102124_object51379:head# 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ac6da5:::benchmark_data_smithi159_102124_object6282:head# 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21adb0d3:::benchmark_data_smithi159_102124_object16890:head# 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ae6e86:::benchmark_data_smithi159_102124_object19289:head# 2024-04-15T18:22:41.250 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ae884c:::benchmark_data_smithi159_102124_object95:head# 2024-04-15T18:22:41.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b1673e:::benchmark_data_smithi159_102124_object33951:head# 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b18526:::benchmark_data_smithi159_102124_object23090:head# 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b1dc79:::benchmark_data_smithi159_102124_object20772:head# 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b1e576:::benchmark_data_smithi159_102124_object41472:head# 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b3c028:::benchmark_data_smithi159_102124_object18025:head# 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b4204d:::benchmark_data_smithi159_102124_object14953:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b517fe:::benchmark_data_smithi159_102124_object26620:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b5236e:::benchmark_data_smithi159_102124_object24012:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b56ed3:::benchmark_data_smithi159_102124_object29454:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b59813:::benchmark_data_smithi159_102124_object2734:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b75c47:::benchmark_data_smithi159_102124_object3102:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b77eac:::benchmark_data_smithi159_102124_object36311:head# 2024-04-15T18:22:41.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b86237:::benchmark_data_smithi159_102124_object3829:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b8ac2a:::benchmark_data_smithi159_102124_object11209:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b9012f:::benchmark_data_smithi159_102124_object17853:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21b9d344:::benchmark_data_smithi159_102124_object39400:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21bc061e:::benchmark_data_smithi159_102124_object28747:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21bcd092:::benchmark_data_smithi159_102124_object48820:head# 2024-04-15T18:22:41.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21bd168b:::benchmark_data_smithi159_102124_object37388:head# 2024-04-15T18:22:41.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21bd9eec:::benchmark_data_smithi159_102124_object41535:head# 2024-04-15T18:22:41.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21bde605:::benchmark_data_smithi159_102124_object37817:head# 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21c03b1b:::benchmark_data_smithi159_102124_object57621:head# 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21c3f8e9:::benchmark_data_smithi159_102124_object47447:head# 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21c6eb93:::benchmark_data_smithi159_102124_object33830:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21c8502b:::benchmark_data_smithi159_102124_object53875:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21cb7591:::benchmark_data_smithi159_102124_object31736:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21cb795b:::benchmark_data_smithi159_102124_object845:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21cbec35:::benchmark_data_smithi159_102124_object11942:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21cc583c:::benchmark_data_smithi159_102124_object14336:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21cd1bc5:::benchmark_data_smithi159_102124_object31171:head# 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d23982:::benchmark_data_smithi159_102124_object31612:head# 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d27383:::benchmark_data_smithi159_102124_object57459:head# 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d2f2d8:::benchmark_data_smithi159_102124_object40689:head# 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d5e170:::benchmark_data_smithi159_102124_object16854:head# 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d5ede0:::benchmark_data_smithi159_102124_object12764:head# 2024-04-15T18:22:41.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d69091:::benchmark_data_smithi159_102124_object45791:head# 2024-04-15T18:22:41.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d6a3cb:::benchmark_data_smithi159_102124_object52302:head# 2024-04-15T18:22:41.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d6abc7:::benchmark_data_smithi159_102124_object8756:head# 2024-04-15T18:22:41.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d7a63a:::benchmark_data_smithi159_102124_object53902:head# 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d83379:::benchmark_data_smithi159_102124_object7184:head# 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d927ca:::benchmark_data_smithi159_102124_object36040:head# 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21d977e5:::benchmark_data_smithi159_102124_object8447:head# 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21dac676:::benchmark_data_smithi159_102124_object4130:head# 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21dc984d:::benchmark_data_smithi159_102124_object4419:head# 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ddf846:::benchmark_data_smithi159_102124_object40998:head# 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21dec006:::benchmark_data_smithi159_102124_object16577:head# 2024-04-15T18:22:41.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21dfc55a:::benchmark_data_smithi159_102124_object14317:head# 2024-04-15T18:22:41.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e059ec:::benchmark_data_smithi159_102124_object41638:head# 2024-04-15T18:22:41.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e16c18:::benchmark_data_smithi159_102124_object38544:head# 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e16ee8:::benchmark_data_smithi159_102124_object28134:head# 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e300b2:::benchmark_data_smithi159_102124_object41275:head# 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e33427:::benchmark_data_smithi159_102124_object30011:head# 2024-04-15T18:22:41.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e4300e:::benchmark_data_smithi159_102124_object47164:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e4b672:::benchmark_data_smithi159_102124_object17803:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e4c11b:::benchmark_data_smithi159_102124_object32100:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e59bf4:::benchmark_data_smithi159_102124_object49949:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e6aaf1:::benchmark_data_smithi159_102124_object36193:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21e83f38:::benchmark_data_smithi159_102124_object35869:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ea96cb:::benchmark_data_smithi159_102124_object12523:head# 2024-04-15T18:22:41.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21eab6b2:::benchmark_data_smithi159_102124_object3231:head# 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ec200a:::benchmark_data_smithi159_102124_object17475:head# 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f08d2f:::benchmark_data_smithi159_102124_object25072:head# 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f0f705:::benchmark_data_smithi159_102124_object5917:head# 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f178c6:::benchmark_data_smithi159_102124_object55976:head# 2024-04-15T18:22:41.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f1e2de:::benchmark_data_smithi159_102124_object39771:head# 2024-04-15T18:22:41.272 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f3161d:::benchmark_data_smithi159_102124_object37151:head# 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f37e4c:::benchmark_data_smithi159_102124_object32738:head# 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f4826a:::benchmark_data_smithi159_102124_object16291:head# 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f6a652:::benchmark_data_smithi159_102124_object51759:head# 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21f75f14:::benchmark_data_smithi159_102124_object57687:head# 2024-04-15T18:22:41.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21fa5862:::benchmark_data_smithi159_102124_object7543:head# 2024-04-15T18:22:41.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21fa72bb:::benchmark_data_smithi159_102124_object19248:head# 2024-04-15T18:22:41.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21fd0ab5:::benchmark_data_smithi159_102124_object35175:head# 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21fdc14a:::benchmark_data_smithi159_102124_object39285:head# 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21fdcc52:::benchmark_data_smithi159_102124_object24558:head# 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ff0600:::benchmark_data_smithi159_102124_object33382:head# 2024-04-15T18:22:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:21ff2c75:::benchmark_data_smithi159_102124_object22997:head# 2024-04-15T18:22:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2200293e:::benchmark_data_smithi159_102124_object16495:head# 2024-04-15T18:22:41.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22049ef5:::benchmark_data_smithi159_102124_object14780:head# 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2205c8bb:::benchmark_data_smithi159_102124_object34963:head# 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22060836:::benchmark_data_smithi159_102124_object28094:head# 2024-04-15T18:22:41.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2206633f:::benchmark_data_smithi159_102124_object34622:head# 2024-04-15T18:22:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22078d7c:::benchmark_data_smithi159_102124_object16113:head# 2024-04-15T18:22:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2207de25:::benchmark_data_smithi159_102124_object24583:head# 2024-04-15T18:22:41.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22083332:::benchmark_data_smithi159_102124_object1621:head# 2024-04-15T18:22:41.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:220aa8dc:::benchmark_data_smithi159_102124_object36207:head# 2024-04-15T18:22:41.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:220ae28c:::benchmark_data_smithi159_102124_object26731:head# 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:220ceeee:::benchmark_data_smithi159_102124_object30446:head# 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:220de4b0:::benchmark_data_smithi159_102124_object41168:head# 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22101277:::benchmark_data_smithi159_102124_object1119:head# 2024-04-15T18:22:41.285 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.286 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22102823:::benchmark_data_smithi159_102124_object53455:head# 2024-04-15T18:22:41.286 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2210692f:::benchmark_data_smithi159_102124_object35330:head# 2024-04-15T18:22:41.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:221079dc:::benchmark_data_smithi159_102124_object49465:head# 2024-04-15T18:22:41.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22110bbf:::benchmark_data_smithi159_102124_object10761:head# 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2212a912:::benchmark_data_smithi159_102124_object29704:head# 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22134df1:::benchmark_data_smithi159_102124_object31014:head# 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22161c69:::benchmark_data_smithi159_102124_object44946:head# 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2216ca5e:::benchmark_data_smithi159_102124_object16671:head# 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22176d37:::benchmark_data_smithi159_102124_object14122:head# 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2218da03:::benchmark_data_smithi159_102124_object34467:head# 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:221caa05:::benchmark_data_smithi159_102124_object12013:head# 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:221e5e53:::benchmark_data_smithi159_102124_object5745:head# 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22205566:::benchmark_data_smithi159_102124_object41395:head# 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22215996:::benchmark_data_smithi159_102124_object45794:head# 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222263d6:::benchmark_data_smithi159_102124_object31897:head# 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2222a8fc:::benchmark_data_smithi159_102124_object24395:head# 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222341ba:::benchmark_data_smithi159_102124_object29244:head# 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22239e70:::benchmark_data_smithi159_102124_object7788:head# 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2223d4ae:::benchmark_data_smithi159_102124_object2483:head# 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2224796a:::benchmark_data_smithi159_102124_object15494:head# 2024-04-15T18:22:41.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2225014b:::benchmark_data_smithi159_102124_object11533:head# 2024-04-15T18:22:41.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22253667:::benchmark_data_smithi159_102124_object58410:head# 2024-04-15T18:22:41.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2225e192:::benchmark_data_smithi159_102124_object18122:head# 2024-04-15T18:22:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22277e59:::benchmark_data_smithi159_102124_object11534:head# 2024-04-15T18:22:41.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22281e59:::benchmark_data_smithi159_102124_object37233:head# 2024-04-15T18:22:41.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222a96e5:::benchmark_data_smithi159_102124_object9245:head# 2024-04-15T18:22:41.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222af27e:::benchmark_data_smithi159_102124_object34543:head# 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222c3121:::benchmark_data_smithi159_102124_object10292:head# 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222ce673:::benchmark_data_smithi159_102124_object51608:head# 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222f61ac:::benchmark_data_smithi159_102124_object46952:head# 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222fa31c:::benchmark_data_smithi159_102124_object24023:head# 2024-04-15T18:22:41.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:222ff400:::benchmark_data_smithi159_102124_object1416:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22304711:::benchmark_data_smithi159_102124_object43318:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22309825:::benchmark_data_smithi159_102124_object5600:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223173c0:::benchmark_data_smithi159_102124_object28069:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22324699:::benchmark_data_smithi159_102124_object24779:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2233ffd7:::benchmark_data_smithi159_102124_object7435:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22345c6b:::benchmark_data_smithi159_102124_object5218:head# 2024-04-15T18:22:41.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22350011:::benchmark_data_smithi159_102124_object31365:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2237cb5a:::benchmark_data_smithi159_102124_object55931:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2237dbe9:::benchmark_data_smithi159_102124_object32293:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22381958:::benchmark_data_smithi159_102124_object25391:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22397911:::benchmark_data_smithi159_102124_object12434:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223c04fa:::benchmark_data_smithi159_102124_object51667:head# 2024-04-15T18:22:41.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223c1fa5:::benchmark_data_smithi159_102124_object46479:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223cb702:::benchmark_data_smithi159_102124_object21465:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223cd645:::benchmark_data_smithi159_102124_object21331:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223d0831:::benchmark_data_smithi159_102124_object45341:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223dded3:::benchmark_data_smithi159_102124_object32526:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:223e728c:::benchmark_data_smithi159_102124_object2119:head# 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.309 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22417712:::benchmark_data_smithi159_102124_object45852:head# 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224269c5:::benchmark_data_smithi159_102124_object17177:head# 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2244ae10:::benchmark_data_smithi159_102124_object47993:head# 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2244be9d:::benchmark_data_smithi159_102124_object32350:head# 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.310 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2244c41c:::benchmark_data_smithi159_102124_object49383:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2245bebc:::benchmark_data_smithi159_102124_object57610:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2245c90d:::benchmark_data_smithi159_102124_object55053:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224b23e3:::benchmark_data_smithi159_102124_object25146:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224c239e:::benchmark_data_smithi159_102124_object24687:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224c3a96:::benchmark_data_smithi159_102124_object17874:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224c5665:::benchmark_data_smithi159_102124_object16993:head# 2024-04-15T18:22:41.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224cdf50:::benchmark_data_smithi159_102124_object23290:head# 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224def1c:::benchmark_data_smithi159_102124_object16652:head# 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:224fa1ac:::benchmark_data_smithi159_102124_object23715:head# 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2251c540:::benchmark_data_smithi159_102124_object32079:head# 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225323d2:::benchmark_data_smithi159_102124_object37471:head# 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.312 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22555ba2:::benchmark_data_smithi159_102124_object42757:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225571d5:::benchmark_data_smithi159_102124_object15507:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2255ea62:::benchmark_data_smithi159_102124_object30288:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2257027e:::benchmark_data_smithi159_102124_object2699:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22578455:::benchmark_data_smithi159_102124_object57231:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22579749:::benchmark_data_smithi159_102124_object3439:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22582b04:::benchmark_data_smithi159_102124_object27654:head# 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22596a03:::benchmark_data_smithi159_102124_object18544:head# 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225abe1a:::benchmark_data_smithi159_102124_object24630:head# 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225b5a95:::benchmark_data_smithi159_102124_object48412:head# 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225bb1f4:::benchmark_data_smithi159_102124_object26623:head# 2024-04-15T18:22:41.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:225ef7ca:::benchmark_data_smithi159_102124_object30296:head# 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2260379b:::benchmark_data_smithi159_102124_object4694:head# 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22610ba6:::benchmark_data_smithi159_102124_object45833:head# 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2263d5d6:::benchmark_data_smithi159_102124_object13478:head# 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2264d497:::benchmark_data_smithi159_102124_object52079:head# 2024-04-15T18:22:41.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22661d5b:::benchmark_data_smithi159_102124_object23097:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2266af19:::benchmark_data_smithi159_102124_object23630:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22676369:::benchmark_data_smithi159_102124_object55265:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2267b2db:::benchmark_data_smithi159_102124_object53107:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2269812d:::benchmark_data_smithi159_102124_object51291:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22699bfd:::benchmark_data_smithi159_102124_object9001:head# 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:226c35ea:::benchmark_data_smithi159_102124_object24473:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:226db46e:::benchmark_data_smithi159_102124_object32127:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:226e83fe:::benchmark_data_smithi159_102124_object34677:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:226e9e16:::benchmark_data_smithi159_102124_object27131:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227060cc:::benchmark_data_smithi159_102124_object49391:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22709d25:::benchmark_data_smithi159_102124_object39395:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22712f50:::benchmark_data_smithi159_102124_object32015:head# 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.324 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22728f92:::benchmark_data_smithi159_102124_object30954:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22729018:::benchmark_data_smithi159_102124_object2254:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2272cfb3:::benchmark_data_smithi159_102124_object46211:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227493b0:::benchmark_data_smithi159_102124_object45718:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22768453:::benchmark_data_smithi159_102124_object53928:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2279d5d2:::benchmark_data_smithi159_102124_object28366:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227a7588:::benchmark_data_smithi159_102124_object15252:head# 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.325 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227ad279:::benchmark_data_smithi159_102124_object19644:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227bc86d:::benchmark_data_smithi159_102124_object33043:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227c057f:::benchmark_data_smithi159_102124_object24036:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227cf5d2:::benchmark_data_smithi159_102124_object44790:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227d6e08:::benchmark_data_smithi159_102124_object9943:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227f5ddd:::benchmark_data_smithi159_102124_object30079:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:227f8354:::benchmark_data_smithi159_102124_object37056:head# 2024-04-15T18:22:41.326 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2280b99b:::benchmark_data_smithi159_102124_object41887:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228196d2:::benchmark_data_smithi159_102124_object5579:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22835a60:::benchmark_data_smithi159_102124_object16415:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2283c79a:::benchmark_data_smithi159_102124_object20155:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22850a58:::benchmark_data_smithi159_102124_object84:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2286b553:::benchmark_data_smithi159_102124_object22599:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2286ea79:::benchmark_data_smithi159_102124_object31375:head# 2024-04-15T18:22:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22887d2c:::benchmark_data_smithi159_102124_object57712:head# 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228a1764:::benchmark_data_smithi159_102124_object43627:head# 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228a1e55:::benchmark_data_smithi159_102124_object25789:head# 2024-04-15T18:22:41.328 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228bee8f:::benchmark_data_smithi159_102124_object12091:head# 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228c90fd:::benchmark_data_smithi159_102124_object1438:head# 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228d03dc:::benchmark_data_smithi159_102124_object31374:head# 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228d5fd2:::benchmark_data_smithi159_102124_object26574:head# 2024-04-15T18:22:41.329 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228d6e0e:::benchmark_data_smithi159_102124_object13176:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228da68a:::benchmark_data_smithi159_102124_object23451:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:228e20ed:::benchmark_data_smithi159_102124_object3537:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2290c412:::benchmark_data_smithi159_102124_object50697:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2290fc42:::benchmark_data_smithi159_102124_object53464:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2292b55d:::benchmark_data_smithi159_102124_object56731:head# 2024-04-15T18:22:41.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2293464f:::benchmark_data_smithi159_102124_object20003:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229545e1:::benchmark_data_smithi159_102124_object55293:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2295d0ba:::benchmark_data_smithi159_102124_object37218:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22965aad:::benchmark_data_smithi159_102124_object536:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2296982b:::benchmark_data_smithi159_102124_object41941:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229737ae:::benchmark_data_smithi159_102124_object34049:head# 2024-04-15T18:22:41.331 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22978fb3:::benchmark_data_smithi159_102124_object105:head# 2024-04-15T18:22:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2297ee9e:::benchmark_data_smithi159_102124_object16859:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229961c9:::benchmark_data_smithi159_102124_object29509:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229bf4fb:::benchmark_data_smithi159_102124_object24088:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229e0146:::benchmark_data_smithi159_102124_object33796:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229e8955:::benchmark_data_smithi159_102124_object54096:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:229fa161:::benchmark_data_smithi159_102124_object8364:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a09009:::benchmark_data_smithi159_102124_object19924:head# 2024-04-15T18:22:41.333 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a1c67c:::benchmark_data_smithi159_102124_object4714:head# 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a32f6c:::benchmark_data_smithi159_102124_object38840:head# 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a5f81c:::benchmark_data_smithi159_102124_object37490:head# 2024-04-15T18:22:41.334 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.338 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a68264:::benchmark_data_smithi159_102124_object45215:head# 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a682b2:::benchmark_data_smithi159_102124_object37227:head# 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a72d30:::benchmark_data_smithi159_102124_object4132:head# 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.339 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a8791e:::benchmark_data_smithi159_102124_object46039:head# 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22a8eab5:::benchmark_data_smithi159_102124_object32361:head# 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ab0d26:::benchmark_data_smithi159_102124_object53949:head# 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22acb146:::benchmark_data_smithi159_102124_object30467:head# 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ad6836:::benchmark_data_smithi159_102124_object47681:head# 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22adde11:::benchmark_data_smithi159_102124_object40574:head# 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22aee9e7:::benchmark_data_smithi159_102124_object26154:head# 2024-04-15T18:22:41.341 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b18725:::benchmark_data_smithi159_102124_object33521:head# 2024-04-15T18:22:41.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b1edb0:::benchmark_data_smithi159_102124_object32757:head# 2024-04-15T18:22:41.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b34d31:::benchmark_data_smithi159_102124_object24424:head# 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b356c9:::benchmark_data_smithi159_102124_object15843:head# 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b35f2b:::benchmark_data_smithi159_102124_object10687:head# 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b38a98:::benchmark_data_smithi159_102124_object35983:head# 2024-04-15T18:22:41.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b61b55:::benchmark_data_smithi159_102124_object41254:head# 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b66f43:::benchmark_data_smithi159_102124_object56475:head# 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b8948f:::benchmark_data_smithi159_102124_object29102:head# 2024-04-15T18:22:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22b8a796:::benchmark_data_smithi159_102124_object4224:head# 2024-04-15T18:22:41.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ba0568:::benchmark_data_smithi159_102124_object17719:head# 2024-04-15T18:22:41.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22bb31f7:::benchmark_data_smithi159_102124_object23362:head# 2024-04-15T18:22:41.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22bbf757:::benchmark_data_smithi159_102124_object32612:head# 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22bcbce2:::benchmark_data_smithi159_102124_object11722:head# 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22be6a1a:::benchmark_data_smithi159_102124_object55150:head# 2024-04-15T18:22:41.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22bfafef:::benchmark_data_smithi159_102124_object51643:head# 2024-04-15T18:22:41.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c06b61:::benchmark_data_smithi159_102124_object20579:head# 2024-04-15T18:22:41.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c0e96d:::benchmark_data_smithi159_102124_object22885:head# 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c53e85:::benchmark_data_smithi159_102124_object35854:head# 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c5a37d:::benchmark_data_smithi159_102124_object19972:head# 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c6315a:::benchmark_data_smithi159_102124_object1882:head# 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c6f04c:::benchmark_data_smithi159_102124_object12815:head# 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c89c4d:::benchmark_data_smithi159_102124_object33387:head# 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22c8e92f:::benchmark_data_smithi159_102124_object689:head# 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22caa839:::benchmark_data_smithi159_102124_object35052:head# 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22cac605:::benchmark_data_smithi159_102124_object720:head# 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22caf849:::benchmark_data_smithi159_102124_object14487:head# 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22cbb502:::benchmark_data_smithi159_102124_object22408:head# 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ccd54c:::benchmark_data_smithi159_102124_object1737:head# 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.355 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22cf6424:::benchmark_data_smithi159_102124_object50033:head# 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22cff58e:::benchmark_data_smithi159_102124_object30973:head# 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d3383f:::benchmark_data_smithi159_102124_object7136:head# 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.356 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d4d4c8:::benchmark_data_smithi159_102124_object20682:head# 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d5631c:::benchmark_data_smithi159_102124_object43005:head# 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d59193:::benchmark_data_smithi159_102124_object35013:head# 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d5964d:::benchmark_data_smithi159_102124_object8296:head# 2024-04-15T18:22:41.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d621fb:::benchmark_data_smithi159_102124_object53029:head# 2024-04-15T18:22:41.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d701b5:::benchmark_data_smithi159_102124_object51331:head# 2024-04-15T18:22:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d87210:::benchmark_data_smithi159_102124_object57875:head# 2024-04-15T18:22:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d8acd8:::benchmark_data_smithi159_102124_object17877:head# 2024-04-15T18:22:41.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22d9c6af:::benchmark_data_smithi159_102124_object8718:head# 2024-04-15T18:22:41.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22dabf3c:::benchmark_data_smithi159_102124_object20165:head# 2024-04-15T18:22:41.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22db7d19:::benchmark_data_smithi159_102124_object34782:head# 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22dc17f4:::benchmark_data_smithi159_102124_object16201:head# 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22dc6b51:::benchmark_data_smithi159_102124_object48701:head# 2024-04-15T18:22:41.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22dd38fb:::benchmark_data_smithi159_102124_object50620:head# 2024-04-15T18:22:41.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ddc4f6:::benchmark_data_smithi159_102124_object15170:head# 2024-04-15T18:22:41.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22decd46:::benchmark_data_smithi159_102124_object22691:head# 2024-04-15T18:22:41.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e0d06a:::benchmark_data_smithi159_102124_object19769:head# 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e0fd5e:::benchmark_data_smithi159_102124_object7021:head# 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e14720:::benchmark_data_smithi159_102124_object22170:head# 2024-04-15T18:22:41.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e3ba63:::benchmark_data_smithi159_102124_object24655:head# 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e5a84e:::benchmark_data_smithi159_102124_object55632:head# 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e5ae8b:::benchmark_data_smithi159_102124_object29471:head# 2024-04-15T18:22:41.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e7c0f4:::benchmark_data_smithi159_102124_object27052:head# 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22e7e2ff:::benchmark_data_smithi159_102124_object6707:head# 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ea418b:::benchmark_data_smithi159_102124_object17519:head# 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ea7b7e:::benchmark_data_smithi159_102124_object1891:head# 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22eae519:::benchmark_data_smithi159_102124_object17034:head# 2024-04-15T18:22:41.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22eb03e9:::benchmark_data_smithi159_102124_object53285:head# 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ed7a41:::benchmark_data_smithi159_102124_object37330:head# 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22edce7b:::benchmark_data_smithi159_102124_object35072:head# 2024-04-15T18:22:41.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.372 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f133de:::benchmark_data_smithi159_102124_object2980:head# 2024-04-15T18:22:41.372 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f468c0:::benchmark_data_smithi159_102124_object5691:head# 2024-04-15T18:22:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f478f6:::benchmark_data_smithi159_102124_object1205:head# 2024-04-15T18:22:41.373 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f47948:::benchmark_data_smithi159_102124_object43799:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f554b1:::benchmark_data_smithi159_102124_object34740:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f562e2:::benchmark_data_smithi159_102124_object2366:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f62dc8:::benchmark_data_smithi159_102124_object26813:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f71e04:::benchmark_data_smithi159_102124_object44435:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f7d010:::benchmark_data_smithi159_102124_object47321:head# 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f7de4e:::benchmark_data_smithi159_102124_object54952:head# 2024-04-15T18:22:41.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f80ec8:::benchmark_data_smithi159_102124_object15725:head# 2024-04-15T18:22:41.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22f96599:::benchmark_data_smithi159_102124_object15257:head# 2024-04-15T18:22:41.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22fc79d7:::benchmark_data_smithi159_102124_object16656:head# 2024-04-15T18:22:41.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22fc88f1:::benchmark_data_smithi159_102124_object34397:head# 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22fd9c47:::benchmark_data_smithi159_102124_object55313:head# 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22fe256f:::benchmark_data_smithi159_102124_object4640:head# 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:22ffe434:::benchmark_data_smithi159_102124_object26854:head# 2024-04-15T18:22:41.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23003721:::benchmark_data_smithi159_102124_object14831:head# 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23010443:::benchmark_data_smithi159_102124_object6267:head# 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2301280a:::benchmark_data_smithi159_102124_object18064:head# 2024-04-15T18:22:41.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230394dc:::benchmark_data_smithi159_102124_object42741:head# 2024-04-15T18:22:41.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23054ddb:::benchmark_data_smithi159_102124_object15493:head# 2024-04-15T18:22:41.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23068378:::benchmark_data_smithi159_102124_object44019:head# 2024-04-15T18:22:41.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2306ae60:::benchmark_data_smithi159_102124_object16251:head# 2024-04-15T18:22:41.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23081414:::benchmark_data_smithi159_102124_object31630:head# 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2308141d:::benchmark_data_smithi159_102124_object32924:head# 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23082383:::benchmark_data_smithi159_102124_object15783:head# 2024-04-15T18:22:41.392 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230afdfe:::benchmark_data_smithi159_102124_object10172:head# 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230b224b:::benchmark_data_smithi159_102124_object22474:head# 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230b88ab:::benchmark_data_smithi159_102124_object12682:head# 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230c9a4d:::benchmark_data_smithi159_102124_object20250:head# 2024-04-15T18:22:41.393 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230d48cb:::benchmark_data_smithi159_102124_object8022:head# 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230ef031:::benchmark_data_smithi159_102124_object8587:head# 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:230faa12:::benchmark_data_smithi159_102124_object3987:head# 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.395 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2310c803:::benchmark_data_smithi159_102124_object28759:head# 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:231210cc:::benchmark_data_smithi159_102124_object22107:head# 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23126c04:::benchmark_data_smithi159_102124_object10833:head# 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2312fc4c:::benchmark_data_smithi159_102124_object45588:head# 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23136eea:::benchmark_data_smithi159_102124_object8595:head# 2024-04-15T18:22:41.396 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2313a5b9:::benchmark_data_smithi159_102124_object55965:head# 2024-04-15T18:22:41.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.397 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2314b3eb:::benchmark_data_smithi159_102124_object4332:head# 2024-04-15T18:22:41.397 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2314ba57:::benchmark_data_smithi159_102124_object9247:head# 2024-04-15T18:22:41.398 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.398 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2314bb3c:::benchmark_data_smithi159_102124_object3553:head# 2024-04-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23155e38:::benchmark_data_smithi159_102124_object2497:head# 2024-04-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23171b7f:::benchmark_data_smithi159_102124_object17154:head# 2024-04-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:231982c1:::benchmark_data_smithi159_102124_object16924:head# 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:231a82b1:::benchmark_data_smithi159_102124_object58279:head# 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:231dac79:::benchmark_data_smithi159_102124_object8784:head# 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:231e4cff:::benchmark_data_smithi159_102124_object40116:head# 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:232035eb:::benchmark_data_smithi159_102124_object17607:head# 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2326691a:::benchmark_data_smithi159_102124_object45050:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23267662:::benchmark_data_smithi159_102124_object44420:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2326bd21:::benchmark_data_smithi159_102124_object24435:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:232d56dd:::benchmark_data_smithi159_102124_object52562:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:232eb208:::benchmark_data_smithi159_102124_object11597:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:232fc7da:::benchmark_data_smithi159_102124_object28090:head# 2024-04-15T18:22:41.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23300135:::benchmark_data_smithi159_102124_object25919:head# 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2331dd22:::benchmark_data_smithi159_102124_object24928:head# 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2332535a:::benchmark_data_smithi159_102124_object36185:head# 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23327e76:::benchmark_data_smithi159_102124_object3447:head# 2024-04-15T18:22:41.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:233291d9:::benchmark_data_smithi159_102124_object619:head# 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2333a353:::benchmark_data_smithi159_102124_object20814:head# 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2333bb5b:::benchmark_data_smithi159_102124_object53370:head# 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23354655:::benchmark_data_smithi159_102124_object48364:head# 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23359168:::benchmark_data_smithi159_102124_object43305:head# 2024-04-15T18:22:41.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2336dcc2:::benchmark_data_smithi159_102124_object26266:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23382c77:::benchmark_data_smithi159_102124_object44025:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2338837a:::benchmark_data_smithi159_102124_object30678:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23389e0e:::benchmark_data_smithi159_102124_object46802:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:233a07d1:::benchmark_data_smithi159_102124_object2799:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:233c2e6f:::benchmark_data_smithi159_102124_object53747:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:233c4a7b:::benchmark_data_smithi159_102124_object37478:head# 2024-04-15T18:22:41.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:233f8ffa:::benchmark_data_smithi159_102124_object36427:head# 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2340c90b:::benchmark_data_smithi159_102124_object20380:head# 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2341de5d:::benchmark_data_smithi159_102124_object38704:head# 2024-04-15T18:22:41.412 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23429c39:::benchmark_data_smithi159_102124_object51770:head# 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2344cca7:::benchmark_data_smithi159_102124_object39271:head# 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2346396d:::benchmark_data_smithi159_102124_object42849:head# 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23469344:::benchmark_data_smithi159_102124_object8508:head# 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23470439:::benchmark_data_smithi159_102124_object31167:head# 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2347af81:::benchmark_data_smithi159_102124_object21847:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2347cf5c:::benchmark_data_smithi159_102124_object24796:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23482e80:::benchmark_data_smithi159_102124_object2461:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2348794b:::benchmark_data_smithi159_102124_object32253:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23498a0e:::benchmark_data_smithi159_102124_object10956:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:234a3e59:::benchmark_data_smithi159_102124_object10659:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:234a4d7d:::benchmark_data_smithi159_102124_object38422:head# 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:234a8a35:::benchmark_data_smithi159_102124_object5184:head# 2024-04-15T18:22:41.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:234e9797:::benchmark_data_smithi159_102124_object44653:head# 2024-04-15T18:22:41.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:234f4a48:::benchmark_data_smithi159_102124_object45404:head# 2024-04-15T18:22:41.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23503925:::benchmark_data_smithi159_102124_object49967:head# 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2350d1c3:::benchmark_data_smithi159_102124_object12529:head# 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23522fe9:::benchmark_data_smithi159_102124_object32568:head# 2024-04-15T18:22:41.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2353d0fe:::benchmark_data_smithi159_102124_object6402:head# 2024-04-15T18:22:41.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.420 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2355f357:::benchmark_data_smithi159_102124_object6531:head# 2024-04-15T18:22:41.420 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:235604d2:::benchmark_data_smithi159_102124_object4526:head# 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:235760ba:::benchmark_data_smithi159_102124_object18440:head# 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23589158:::benchmark_data_smithi159_102124_object32779:head# 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:235957b8:::benchmark_data_smithi159_102124_object53351:head# 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:235eb46c:::benchmark_data_smithi159_102124_object28189:head# 2024-04-15T18:22:41.421 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:235ec080:::benchmark_data_smithi159_102124_object56016:head# 2024-04-15T18:22:41.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2361e1e4:::benchmark_data_smithi159_102124_object18984:head# 2024-04-15T18:22:41.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23635d00:::benchmark_data_smithi159_102124_object3433:head# 2024-04-15T18:22:41.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2363c626:::benchmark_data_smithi159_102124_object34329:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2366a09b:::benchmark_data_smithi159_102124_object6443:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236825b6:::benchmark_data_smithi159_102124_object56911:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23687d62:::benchmark_data_smithi159_102124_object21014:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236a28b3:::benchmark_data_smithi159_102124_object13443:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236a29c0:::benchmark_data_smithi159_102124_object46446:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236a380a:::benchmark_data_smithi159_102124_object55241:head# 2024-04-15T18:22:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236a6719:::benchmark_data_smithi159_102124_object22442:head# 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236b4bcb:::benchmark_data_smithi159_102124_object34932:head# 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236ee358:::benchmark_data_smithi159_102124_object51642:head# 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236f89cd:::benchmark_data_smithi159_102124_object4638:head# 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:236fbecd:::benchmark_data_smithi159_102124_object29193:head# 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23711a47:::benchmark_data_smithi159_102124_object34512:head# 2024-04-15T18:22:41.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237238a6:::benchmark_data_smithi159_102124_object56241:head# 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237506c3:::benchmark_data_smithi159_102124_object6643:head# 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23753ead:::benchmark_data_smithi159_102124_object41513:head# 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237545fa:::benchmark_data_smithi159_102124_object16545:head# 2024-04-15T18:22:41.429 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2375d4df:::benchmark_data_smithi159_102124_object25832:head# 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23763849:::benchmark_data_smithi159_102124_object42576:head# 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237ae13f:::benchmark_data_smithi159_102124_object55602:head# 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237ae55f:::benchmark_data_smithi159_102124_object44690:head# 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237b7e0b:::benchmark_data_smithi159_102124_object7272:head# 2024-04-15T18:22:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:237ca780:::benchmark_data_smithi159_102124_object7038:head# 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2380d63b:::benchmark_data_smithi159_102124_object50135:head# 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23814050:::benchmark_data_smithi159_102124_object36951:head# 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2381698b:::benchmark_data_smithi159_102124_object3149:head# 2024-04-15T18:22:41.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2382c2c6:::benchmark_data_smithi159_102124_object9040:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2382c63f:::benchmark_data_smithi159_102124_object10432:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23846830:::benchmark_data_smithi159_102124_object41028:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23876b92:::benchmark_data_smithi159_102124_object31595:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2387c2a4:::benchmark_data_smithi159_102124_object2932:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2388e6da:::benchmark_data_smithi159_102124_object45002:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238ab000:::benchmark_data_smithi159_102124_object5803:head# 2024-04-15T18:22:41.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238ab736:::benchmark_data_smithi159_102124_object20584:head# 2024-04-15T18:22:41.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238b68c5:::benchmark_data_smithi159_102124_object37333:head# 2024-04-15T18:22:41.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238df0da:::benchmark_data_smithi159_102124_object32901:head# 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238eb1ca:::benchmark_data_smithi159_102124_object19600:head# 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238ed87d:::benchmark_data_smithi159_102124_object16037:head# 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:238fca02:::benchmark_data_smithi159_102124_object33595:head# 2024-04-15T18:22:41.435 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23901480:::benchmark_data_smithi159_102124_object42892:head# 2024-04-15T18:22:41.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2390dd68:::benchmark_data_smithi159_102124_object55696:head# 2024-04-15T18:22:41.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2391ad2f:::benchmark_data_smithi159_102124_object614:head# 2024-04-15T18:22:41.444 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.447 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23926efc:::benchmark_data_smithi159_102124_object34691:head# 2024-04-15T18:22:41.447 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.447 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23936205:::benchmark_data_smithi159_102124_object54401:head# 2024-04-15T18:22:41.447 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.448 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23936c64:::benchmark_data_smithi159_102124_object34388:head# 2024-04-15T18:22:41.448 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239480f3:::benchmark_data_smithi159_102124_object13687:head# 2024-04-15T18:22:41.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23960e50:::benchmark_data_smithi159_102124_object35529:head# 2024-04-15T18:22:41.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2398ee2e:::benchmark_data_smithi159_102124_object16168:head# 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239960e1:::benchmark_data_smithi159_102124_object23339:head# 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239b606f:::benchmark_data_smithi159_102124_object48370:head# 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239b6cf8:::benchmark_data_smithi159_102124_object40519:head# 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239bbace:::benchmark_data_smithi159_102124_object40034:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239bd7a9:::benchmark_data_smithi159_102124_object53035:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239cabe1:::benchmark_data_smithi159_102124_object50634:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239e0285:::benchmark_data_smithi159_102124_object19122:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239e304c:::benchmark_data_smithi159_102124_object13683:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239e5bff:::benchmark_data_smithi159_102124_object34241:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239f8387:::benchmark_data_smithi159_102124_object55099:head# 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:239f99ff:::benchmark_data_smithi159_102124_object13268:head# 2024-04-15T18:22:41.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a018c7:::benchmark_data_smithi159_102124_object46536:head# 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a09b43:::benchmark_data_smithi159_102124_object54659:head# 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a19f46:::benchmark_data_smithi159_102124_object19152:head# 2024-04-15T18:22:41.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a237be:::benchmark_data_smithi159_102124_object45376:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a2e0c4:::benchmark_data_smithi159_102124_object34117:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a3c860:::benchmark_data_smithi159_102124_object20830:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a54080:::benchmark_data_smithi159_102124_object29698:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a5949c:::benchmark_data_smithi159_102124_object29521:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a5e1b6:::benchmark_data_smithi159_102124_object38302:head# 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a775c8:::benchmark_data_smithi159_102124_object34423:head# 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a781cc:::benchmark_data_smithi159_102124_object24534:head# 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23a92416:::benchmark_data_smithi159_102124_object58431:head# 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23aa19f8:::benchmark_data_smithi159_102124_object18372:head# 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23aceba5:::benchmark_data_smithi159_102124_object51418:head# 2024-04-15T18:22:41.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23adbbff:::benchmark_data_smithi159_102124_object14362:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ae33a8:::benchmark_data_smithi159_102124_object18607:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ae402d:::benchmark_data_smithi159_102124_object5136:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ae68ab:::benchmark_data_smithi159_102124_object48532:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23aefca2:::benchmark_data_smithi159_102124_object17453:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23af260a:::benchmark_data_smithi159_102124_object6304:head# 2024-04-15T18:22:41.466 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23afc2b7:::benchmark_data_smithi159_102124_object45359:head# 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23b2d684:::benchmark_data_smithi159_102124_object10063:head# 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23b30297:::benchmark_data_smithi159_102124_object37757:head# 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23b4741c:::benchmark_data_smithi159_102124_object8077:head# 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23b51bf6:::benchmark_data_smithi159_102124_object6889:head# 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.467 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23b6d454:::benchmark_data_smithi159_102124_object32761:head# 2024-04-15T18:22:41.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ba5816:::benchmark_data_smithi159_102124_object34164:head# 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23bd0b5b:::benchmark_data_smithi159_102124_object46810:head# 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23bd3806:::benchmark_data_smithi159_102124_object31682:head# 2024-04-15T18:22:41.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23bdc1ac:::benchmark_data_smithi159_102124_object57211:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23be5a0d:::benchmark_data_smithi159_102124_object43952:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23bee57b:::benchmark_data_smithi159_102124_object2295:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c04ecc:::benchmark_data_smithi159_102124_object40192:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c296c2:::benchmark_data_smithi159_102124_object37933:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c2d73d:::benchmark_data_smithi159_102124_object53323:head# 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.474 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c3bb58:::benchmark_data_smithi159_102124_object34658:head# 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c6e50d:::benchmark_data_smithi159_102124_object13348:head# 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c71243:::benchmark_data_smithi159_102124_object49265:head# 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c79350:::benchmark_data_smithi159_102124_object16640:head# 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.475 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23c99aaa:::benchmark_data_smithi159_102124_object23943:head# 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ca5823:::benchmark_data_smithi159_102124_object36186:head# 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d0c5fe:::benchmark_data_smithi159_102124_object4713:head# 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d19dca:::benchmark_data_smithi159_102124_object53975:head# 2024-04-15T18:22:41.476 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d2a51c:::benchmark_data_smithi159_102124_object19733:head# 2024-04-15T18:22:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d2a7ec:::benchmark_data_smithi159_102124_object44724:head# 2024-04-15T18:22:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d3515b:::benchmark_data_smithi159_102124_object41128:head# 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d39ab5:::benchmark_data_smithi159_102124_object30560:head# 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d3a905:::benchmark_data_smithi159_102124_object31617:head# 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d742c2:::benchmark_data_smithi159_102124_object27234:head# 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d83481:::benchmark_data_smithi159_102124_object17651:head# 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d83a3c:::benchmark_data_smithi159_102124_object1677:head# 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23d90a2b:::benchmark_data_smithi159_102124_object15986:head# 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23dae957:::benchmark_data_smithi159_102124_object45155:head# 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23db3857:::benchmark_data_smithi159_102124_object54076:head# 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23dde881:::benchmark_data_smithi159_102124_object23958:head# 2024-04-15T18:22:41.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23de9d04:::benchmark_data_smithi159_102124_object32972:head# 2024-04-15T18:22:41.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23e1f3c8:::benchmark_data_smithi159_102124_object14790:head# 2024-04-15T18:22:41.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23e373aa:::benchmark_data_smithi159_102124_object37168:head# 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23e73318:::benchmark_data_smithi159_102124_object55071:head# 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23e843d1:::benchmark_data_smithi159_102124_object33189:head# 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23eab097:::benchmark_data_smithi159_102124_object12903:head# 2024-04-15T18:22:41.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.483 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23eab6d0:::benchmark_data_smithi159_102124_object21705:head# 2024-04-15T18:22:41.483 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23eb5e2c:::benchmark_data_smithi159_102124_object52266:head# 2024-04-15T18:22:41.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ecde47:::benchmark_data_smithi159_102124_object14669:head# 2024-04-15T18:22:41.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ed6d4a:::benchmark_data_smithi159_102124_object29925:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23ee845b:::benchmark_data_smithi159_102124_object28959:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23eea0b8:::benchmark_data_smithi159_102124_object47377:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f0ea57:::benchmark_data_smithi159_102124_object38985:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f105b4:::benchmark_data_smithi159_102124_object23972:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f1dad2:::benchmark_data_smithi159_102124_object57848:head# 2024-04-15T18:22:41.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f78234:::benchmark_data_smithi159_102124_object11097:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f8160a:::benchmark_data_smithi159_102124_object1049:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f8e527:::benchmark_data_smithi159_102124_object5601:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f8fe43:::benchmark_data_smithi159_102124_object45824:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f9483d:::benchmark_data_smithi159_102124_object58088:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f97a1a:::benchmark_data_smithi159_102124_object22420:head# 2024-04-15T18:22:41.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23f9b414:::benchmark_data_smithi159_102124_object49905:head# 2024-04-15T18:22:41.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23fa3c49:::benchmark_data_smithi159_102124_object17710:head# 2024-04-15T18:22:41.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23fc3678:::benchmark_data_smithi159_102124_object20058:head# 2024-04-15T18:22:41.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23fd3371:::benchmark_data_smithi159_102124_object48104:head# 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:23feaaa3:::benchmark_data_smithi159_102124_object48526:head# 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2400aa4d:::benchmark_data_smithi159_102124_object32433:head# 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.535 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2400b57a:::benchmark_data_smithi159_102124_object18214:head# 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240111f9:::benchmark_data_smithi159_102124_object34233:head# 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2402bf12:::benchmark_data_smithi159_102124_object1563:head# 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.536 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24031a84:::benchmark_data_smithi159_102124_object14392:head# 2024-04-15T18:22:41.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2404112b:::benchmark_data_smithi159_102124_object30328:head# 2024-04-15T18:22:41.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2405231a:::benchmark_data_smithi159_102124_object45815:head# 2024-04-15T18:22:41.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24055895:::benchmark_data_smithi159_102124_object44368:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2405d718:::benchmark_data_smithi159_102124_object9067:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240826c8:::benchmark_data_smithi159_102124_object22618:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2409cf22:::benchmark_data_smithi159_102124_object14563:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240afd1e:::benchmark_data_smithi159_102124_object58142:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240b1522:::benchmark_data_smithi159_102124_object17577:head# 2024-04-15T18:22:41.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240bcdc4:::benchmark_data_smithi159_102124_object49436:head# 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240c40be:::benchmark_data_smithi159_102124_object36118:head# 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240c5125:::benchmark_data_smithi159_102124_object38079:head# 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240d3124:::benchmark_data_smithi159_102124_object17347:head# 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:240db244:::benchmark_data_smithi159_102124_object7755:head# 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241093a1:::benchmark_data_smithi159_102124_object14906:head# 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2410a43a:::benchmark_data_smithi159_102124_object14875:head# 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2410a986:::benchmark_data_smithi159_102124_object49402:head# 2024-04-15T18:22:41.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241499d1:::benchmark_data_smithi159_102124_object20774:head# 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2414dc36:::benchmark_data_smithi159_102124_object8119:head# 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2416f41c:::benchmark_data_smithi159_102124_object36602:head# 2024-04-15T18:22:41.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2417bd4d:::benchmark_data_smithi159_102124_object12034:head# 2024-04-15T18:22:41.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241a235b:::benchmark_data_smithi159_102124_object40108:head# 2024-04-15T18:22:41.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241c0979:::benchmark_data_smithi159_102124_object47821:head# 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241d6663:::benchmark_data_smithi159_102124_object1407:head# 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241e5d14:::benchmark_data_smithi159_102124_object50416:head# 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241ecf79:::benchmark_data_smithi159_102124_object11824:head# 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241f4ece:::benchmark_data_smithi159_102124_object1940:head# 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:241f7674:::benchmark_data_smithi159_102124_object48365:head# 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2420065f:::benchmark_data_smithi159_102124_object19968:head# 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24206164:::benchmark_data_smithi159_102124_object47546:head# 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2420c839:::benchmark_data_smithi159_102124_object55119:head# 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2420f283:::benchmark_data_smithi159_102124_object29962:head# 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24221503:::benchmark_data_smithi159_102124_object48784:head# 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2422343d:::benchmark_data_smithi159_102124_object24843:head# 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24247687:::benchmark_data_smithi159_102124_object33426:head# 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242af99a:::benchmark_data_smithi159_102124_object27425:head# 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242bc438:::benchmark_data_smithi159_102124_object25127:head# 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242cddcf:::benchmark_data_smithi159_102124_object43708:head# 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242d4f6e:::benchmark_data_smithi159_102124_object10400:head# 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242da722:::benchmark_data_smithi159_102124_object36621:head# 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.548 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242e4c1e:::benchmark_data_smithi159_102124_object46804:head# 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:242e6aa7:::benchmark_data_smithi159_102124_object55187:head# 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2433961e:::benchmark_data_smithi159_102124_object26350:head# 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2433babd:::benchmark_data_smithi159_102124_object44662:head# 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2433bbf7:::benchmark_data_smithi159_102124_object45535:head# 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243423ff:::benchmark_data_smithi159_102124_object38541:head# 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24385f18:::benchmark_data_smithi159_102124_object30872:head# 2024-04-15T18:22:41.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2438cc75:::benchmark_data_smithi159_102124_object19624:head# 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2438efe4:::benchmark_data_smithi159_102124_object41051:head# 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24391320:::benchmark_data_smithi159_102124_object45612:head# 2024-04-15T18:22:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243976ff:::benchmark_data_smithi159_102124_object10510:head# 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243b1916:::benchmark_data_smithi159_102124_object28945:head# 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243c6750:::benchmark_data_smithi159_102124_object15328:head# 2024-04-15T18:22:41.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243cd8bf:::benchmark_data_smithi159_102124_object33285:head# 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243d8876:::benchmark_data_smithi159_102124_object18596:head# 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:243f8448:::benchmark_data_smithi159_102124_object331:head# 2024-04-15T18:22:41.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24408322:::benchmark_data_smithi159_102124_object26964:head# 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2440e49a:::benchmark_data_smithi159_102124_object57400:head# 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2441ed50:::benchmark_data_smithi159_102124_object13030:head# 2024-04-15T18:22:41.554 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24452a42:::benchmark_data_smithi159_102124_object50330:head# 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2446bd36:::benchmark_data_smithi159_102124_object39876:head# 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244717e2:::benchmark_data_smithi159_102124_object46405:head# 2024-04-15T18:22:41.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244765d6:::benchmark_data_smithi159_102124_object22296:head# 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2447df84:::benchmark_data_smithi159_102124_object55391:head# 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2447edee:::benchmark_data_smithi159_102124_object24548:head# 2024-04-15T18:22:41.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2447f717:::benchmark_data_smithi159_102124_object35840:head# 2024-04-15T18:22:41.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2448a514:::benchmark_data_smithi159_102124_object13181:head# 2024-04-15T18:22:41.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2449596c:::benchmark_data_smithi159_102124_object51691:head# 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244a6176:::benchmark_data_smithi159_102124_object47789:head# 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244b14f3:::benchmark_data_smithi159_102124_object7598:head# 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244badd8:::benchmark_data_smithi159_102124_object17517:head# 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244c479e:::benchmark_data_smithi159_102124_object35705:head# 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:244d08ba:::benchmark_data_smithi159_102124_object13745:head# 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24503217:::benchmark_data_smithi159_102124_object208:head# 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2454a9c1:::benchmark_data_smithi159_102124_object11664:head# 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24560bab:::benchmark_data_smithi159_102124_object27519:head# 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24563ae7:::benchmark_data_smithi159_102124_object42630:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2456de53:::benchmark_data_smithi159_102124_object43042:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24573dfe:::benchmark_data_smithi159_102124_object9566:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245791fd:::benchmark_data_smithi159_102124_object55593:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24586a7a:::benchmark_data_smithi159_102124_object28895:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245881c7:::benchmark_data_smithi159_102124_object47716:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245a7fbf:::benchmark_data_smithi159_102124_object28168:head# 2024-04-15T18:22:41.561 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245b7e50:::benchmark_data_smithi159_102124_object20177:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245bda00:::benchmark_data_smithi159_102124_object44020:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:245f942e:::benchmark_data_smithi159_102124_object34127:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24610067:::benchmark_data_smithi159_102124_object30635:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24693dda:::benchmark_data_smithi159_102124_object3832:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:246a747e:::benchmark_data_smithi159_102124_object53495:head# 2024-04-15T18:22:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:246b992a:::benchmark_data_smithi159_102124_object43681:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:246c1309:::benchmark_data_smithi159_102124_object46698:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:246ea550:::benchmark_data_smithi159_102124_object23240:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:246f8254:::benchmark_data_smithi159_102124_object32338:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24706ae5:::benchmark_data_smithi159_102124_object31283:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24716843:::benchmark_data_smithi159_102124_object56938:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2471b541:::benchmark_data_smithi159_102124_object20203:head# 2024-04-15T18:22:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24724f08:::benchmark_data_smithi159_102124_object48542:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2472501b:::benchmark_data_smithi159_102124_object10591:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247271fe:::benchmark_data_smithi159_102124_object38612:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24750515:::benchmark_data_smithi159_102124_object9674:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247515b3:::benchmark_data_smithi159_102124_object16513:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24770182:::benchmark_data_smithi159_102124_object38634:head# 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24775c3c:::benchmark_data_smithi159_102124_object24983:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2477aef6:::benchmark_data_smithi159_102124_object47938:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24789045:::benchmark_data_smithi159_102124_object10564:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247910ee:::benchmark_data_smithi159_102124_object1665:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2479d3bd:::benchmark_data_smithi159_102124_object51552:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247c61a7:::benchmark_data_smithi159_102124_object1496:head# 2024-04-15T18:22:41.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247d351f:::benchmark_data_smithi159_102124_object20291:head# 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247db6c6:::benchmark_data_smithi159_102124_object1236:head# 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247db81d:::benchmark_data_smithi159_102124_object7871:head# 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247f02a8:::benchmark_data_smithi159_102124_object6339:head# 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:247f2a4c:::benchmark_data_smithi159_102124_object7787:head# 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.566 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248033c5:::benchmark_data_smithi159_102124_object51560:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2481d5ae:::benchmark_data_smithi159_102124_object676:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2481da43:::benchmark_data_smithi159_102124_object3010:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2483c0b7:::benchmark_data_smithi159_102124_object41111:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24842daf:::benchmark_data_smithi159_102124_object44454:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2484a84c:::benchmark_data_smithi159_102124_object16408:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2484e898:::benchmark_data_smithi159_102124_object14331:head# 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2486dc53:::benchmark_data_smithi159_102124_object29600:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24871165:::benchmark_data_smithi159_102124_object16676:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24874cd1:::benchmark_data_smithi159_102124_object24746:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2487aef5:::benchmark_data_smithi159_102124_object44909:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2488b633:::benchmark_data_smithi159_102124_object34267:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24898609:::benchmark_data_smithi159_102124_object43623:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24899708:::benchmark_data_smithi159_102124_object43996:head# 2024-04-15T18:22:41.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248c1613:::benchmark_data_smithi159_102124_object23096:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248c3a5f:::benchmark_data_smithi159_102124_object21460:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248cf66e:::benchmark_data_smithi159_102124_object12638:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248d732f:::benchmark_data_smithi159_102124_object49039:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248e3302:::benchmark_data_smithi159_102124_object13660:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248e8b89:::benchmark_data_smithi159_102124_object51324:head# 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:248faffa:::benchmark_data_smithi159_102124_object39474:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2490520b:::benchmark_data_smithi159_102124_object10703:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2490fc31:::benchmark_data_smithi159_102124_object5089:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2494322d:::benchmark_data_smithi159_102124_object52018:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2494a8d8:::benchmark_data_smithi159_102124_object58400:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24960890:::benchmark_data_smithi159_102124_object29794:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249664ac:::benchmark_data_smithi159_102124_object36089:head# 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.570 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2496b6a3:::benchmark_data_smithi159_102124_object18701:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2496b70d:::benchmark_data_smithi159_102124_object2892:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2496bb0f:::benchmark_data_smithi159_102124_object3617:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2496f6d2:::benchmark_data_smithi159_102124_object11386:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249b96c2:::benchmark_data_smithi159_102124_object42562:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249c31bf:::benchmark_data_smithi159_102124_object55164:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249cc05e:::benchmark_data_smithi159_102124_object52473:head# 2024-04-15T18:22:41.571 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249d0b82:::benchmark_data_smithi159_102124_object29588:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249e22e6:::benchmark_data_smithi159_102124_object43407:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249ef69b:::benchmark_data_smithi159_102124_object37696:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:249f75e1:::benchmark_data_smithi159_102124_object20162:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a05eef:::benchmark_data_smithi159_102124_object52872:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a0e50f:::benchmark_data_smithi159_102124_object4628:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a14f45:::benchmark_data_smithi159_102124_object13654:head# 2024-04-15T18:22:41.572 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a4d67a:::benchmark_data_smithi159_102124_object14097:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a58fe2:::benchmark_data_smithi159_102124_object42381:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a59b2d:::benchmark_data_smithi159_102124_object54684:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24a92c5e:::benchmark_data_smithi159_102124_object45405:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ab2312:::benchmark_data_smithi159_102124_object12343:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ac0de2:::benchmark_data_smithi159_102124_object49589:head# 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ae3fed:::benchmark_data_smithi159_102124_object24437:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24afe7b2:::benchmark_data_smithi159_102124_object50156:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b12eef:::benchmark_data_smithi159_102124_object17079:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b1729f:::benchmark_data_smithi159_102124_object42334:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b46801:::benchmark_data_smithi159_102124_object41348:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b6841f:::benchmark_data_smithi159_102124_object42220:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b875ef:::benchmark_data_smithi159_102124_object8394:head# 2024-04-15T18:22:41.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24b93f55:::benchmark_data_smithi159_102124_object38382:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24bcc68e:::benchmark_data_smithi159_102124_object46866:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24bdc1d0:::benchmark_data_smithi159_102124_object29983:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24bef0b7:::benchmark_data_smithi159_102124_object57040:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24bf0b82:::benchmark_data_smithi159_102124_object43530:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c04908:::benchmark_data_smithi159_102124_object33699:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c08788:::benchmark_data_smithi159_102124_object54961:head# 2024-04-15T18:22:41.575 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c264cf:::benchmark_data_smithi159_102124_object48912:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c3cb2b:::benchmark_data_smithi159_102124_object14026:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c4ae62:::benchmark_data_smithi159_102124_object22999:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c54f10:::benchmark_data_smithi159_102124_object47564:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c604ff:::benchmark_data_smithi159_102124_object6372:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c66413:::benchmark_data_smithi159_102124_object17497:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c71958:::benchmark_data_smithi159_102124_object39508:head# 2024-04-15T18:22:41.576 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c79eea:::benchmark_data_smithi159_102124_object8455:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c86666:::benchmark_data_smithi159_102124_object14718:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c89e1c:::benchmark_data_smithi159_102124_object41977:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c8adc3:::benchmark_data_smithi159_102124_object42925:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c91788:::benchmark_data_smithi159_102124_object20896:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24c9da37:::benchmark_data_smithi159_102124_object39183:head# 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24cbe103:::benchmark_data_smithi159_102124_object18517:head# 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24cce2e3:::benchmark_data_smithi159_102124_object52885:head# 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ce21a7:::benchmark_data_smithi159_102124_object46418:head# 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ceb84c:::benchmark_data_smithi159_102124_object24383:head# 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24cf8142:::benchmark_data_smithi159_102124_object23654:head# 2024-04-15T18:22:41.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24cff4dc:::benchmark_data_smithi159_102124_object26273:head# 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d2079c:::benchmark_data_smithi159_102124_object29572:head# 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d20aaa:::benchmark_data_smithi159_102124_object7979:head# 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d2b694:::benchmark_data_smithi159_102124_object35317:head# 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d46cec:::benchmark_data_smithi159_102124_object5704:head# 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d4b646:::benchmark_data_smithi159_102124_object35842:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d5d130:::benchmark_data_smithi159_102124_object12775:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d70cfc:::benchmark_data_smithi159_102124_object4467:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d818df:::benchmark_data_smithi159_102124_object14364:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24d90399:::benchmark_data_smithi159_102124_object15657:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24dc3fa2:::benchmark_data_smithi159_102124_object15077:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24dc41d9:::benchmark_data_smithi159_102124_object2174:head# 2024-04-15T18:22:41.597 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24df7421:::benchmark_data_smithi159_102124_object54355:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e15138:::benchmark_data_smithi159_102124_object10351:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e1cf99:::benchmark_data_smithi159_102124_object16367:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e2624e:::benchmark_data_smithi159_102124_object3048:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e4801f:::benchmark_data_smithi159_102124_object2561:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e50b43:::benchmark_data_smithi159_102124_object5491:head# 2024-04-15T18:22:41.598 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e5eb5f:::benchmark_data_smithi159_102124_object18038:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e6502f:::benchmark_data_smithi159_102124_object49175:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e66a90:::benchmark_data_smithi159_102124_object26163:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e680d0:::benchmark_data_smithi159_102124_object44595:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e69c1c:::benchmark_data_smithi159_102124_object22287:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e6a209:::benchmark_data_smithi159_102124_object20305:head# 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.599 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e77108:::benchmark_data_smithi159_102124_object46953:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e77e40:::benchmark_data_smithi159_102124_object40532:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e79ef5:::benchmark_data_smithi159_102124_object56292:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e7a718:::benchmark_data_smithi159_102124_object6086:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24e962fa:::benchmark_data_smithi159_102124_object26558:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ea3dd8:::benchmark_data_smithi159_102124_object34672:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24eb132d:::benchmark_data_smithi159_102124_object8867:head# 2024-04-15T18:22:41.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24eb37d5:::benchmark_data_smithi159_102124_object41402:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ec3702:::benchmark_data_smithi159_102124_object23081:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ed70b5:::benchmark_data_smithi159_102124_object18488:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24eda6b0:::benchmark_data_smithi159_102124_object42004:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24eefa2c:::benchmark_data_smithi159_102124_object35781:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ef7bec:::benchmark_data_smithi159_102124_object44922:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f14b7c:::benchmark_data_smithi159_102124_object28840:head# 2024-04-15T18:22:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f283c4:::benchmark_data_smithi159_102124_object31539:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f2c30d:::benchmark_data_smithi159_102124_object24745:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f34111:::benchmark_data_smithi159_102124_object41769:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f4b46a:::benchmark_data_smithi159_102124_object49314:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f64790:::benchmark_data_smithi159_102124_object50942:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24f811e4:::benchmark_data_smithi159_102124_object27898:head# 2024-04-15T18:22:41.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24fa4906:::benchmark_data_smithi159_102124_object52174:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24faaf60:::benchmark_data_smithi159_102124_object37464:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24fc1738:::benchmark_data_smithi159_102124_object21641:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24fea26e:::benchmark_data_smithi159_102124_object20374:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ff9076:::benchmark_data_smithi159_102124_object976:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:24ffd7d5:::benchmark_data_smithi159_102124_object44721:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25020080:::benchmark_data_smithi159_102124_object26290:head# 2024-04-15T18:22:41.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25023da0:::benchmark_data_smithi159_102124_object33120:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2502accf:::benchmark_data_smithi159_102124_object14950:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2503f0b2:::benchmark_data_smithi159_102124_object6838:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25041349:::benchmark_data_smithi159_102124_object39472:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25060b7b:::benchmark_data_smithi159_102124_object24762:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2509d9a9:::benchmark_data_smithi159_102124_object57436:head# 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250bfff2:::benchmark_data_smithi159_102124_object49263:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250c06ca:::benchmark_data_smithi159_102124_object56233:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250c65c6:::benchmark_data_smithi159_102124_object47588:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250d756e:::benchmark_data_smithi159_102124_object14401:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250f05c2:::benchmark_data_smithi159_102124_object56404:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:250fa7d5:::benchmark_data_smithi159_102124_object28499:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25134435:::benchmark_data_smithi159_102124_object18963:head# 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25152f87:::benchmark_data_smithi159_102124_object13636:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251a620d:::benchmark_data_smithi159_102124_object3939:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251a7d19:::benchmark_data_smithi159_102124_object32090:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251a8021:::benchmark_data_smithi159_102124_object28641:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251a8259:::benchmark_data_smithi159_102124_object43394:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251ad9b3:::benchmark_data_smithi159_102124_object46829:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251b8a5b:::benchmark_data_smithi159_102124_object10808:head# 2024-04-15T18:22:41.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251bd5af:::benchmark_data_smithi159_102124_object2326:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251d767d:::benchmark_data_smithi159_102124_object18519:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251e338d:::benchmark_data_smithi159_102124_object14558:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:251f10f6:::benchmark_data_smithi159_102124_object32767:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25238af3:::benchmark_data_smithi159_102124_object22681:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25243d1e:::benchmark_data_smithi159_102124_object12283:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25258dd4:::benchmark_data_smithi159_102124_object49581:head# 2024-04-15T18:22:41.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25259a75:::benchmark_data_smithi159_102124_object51719:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25276d6d:::benchmark_data_smithi159_102124_object23407:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25279f66:::benchmark_data_smithi159_102124_object53090:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2527d8e9:::benchmark_data_smithi159_102124_object24274:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:252a2be4:::benchmark_data_smithi159_102124_object39079:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:252a7a26:::benchmark_data_smithi159_102124_object9881:head# 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.608 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:252b179f:::benchmark_data_smithi159_102124_object17088:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:252ffc52:::benchmark_data_smithi159_102124_object50173:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25316e15:::benchmark_data_smithi159_102124_object40105:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2533ab46:::benchmark_data_smithi159_102124_object12488:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253404b8:::benchmark_data_smithi159_102124_object26516:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25342295:::benchmark_data_smithi159_102124_object10391:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2534bf5f:::benchmark_data_smithi159_102124_object1983:head# 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.609 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2538f5d2:::benchmark_data_smithi159_102124_object18073:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25397205:::benchmark_data_smithi159_102124_object44854:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253a08f0:::benchmark_data_smithi159_102124_object20089:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253aeba5:::benchmark_data_smithi159_102124_object16963:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253b77fc:::benchmark_data_smithi159_102124_object28410:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253c7def:::benchmark_data_smithi159_102124_object26358:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253c9202:::benchmark_data_smithi159_102124_object27038:head# 2024-04-15T18:22:41.610 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253dc802:::benchmark_data_smithi159_102124_object51975:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253e7572:::benchmark_data_smithi159_102124_object52573:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:253e8918:::benchmark_data_smithi159_102124_object51092:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2541ad49:::benchmark_data_smithi159_102124_object16572:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25425b56:::benchmark_data_smithi159_102124_object43265:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25449c6e:::benchmark_data_smithi159_102124_object32547:head# 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.611 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25456f01:::benchmark_data_smithi159_102124_object2931:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25457956:::benchmark_data_smithi159_102124_object15452:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2545b206:::benchmark_data_smithi159_102124_object27861:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2546556c:::benchmark_data_smithi159_102124_object33214:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2547340a:::benchmark_data_smithi159_102124_object2123:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25476a0d:::benchmark_data_smithi159_102124_object55959:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2547ae62:::benchmark_data_smithi159_102124_object54863:head# 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.612 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25489681:::benchmark_data_smithi159_102124_object13157:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25490b97:::benchmark_data_smithi159_102124_object34844:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25498638:::benchmark_data_smithi159_102124_object55836:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254b9de9:::benchmark_data_smithi159_102124_object25988:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254be9ea:::benchmark_data_smithi159_102124_object5055:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254ceb1f:::benchmark_data_smithi159_102124_object40923:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254dbfcc:::benchmark_data_smithi159_102124_object11526:head# 2024-04-15T18:22:41.613 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254e62aa:::benchmark_data_smithi159_102124_object25136:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254f4565:::benchmark_data_smithi159_102124_object37776:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:254f58a2:::benchmark_data_smithi159_102124_object5854:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25502405:::benchmark_data_smithi159_102124_object10008:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25518dfb:::benchmark_data_smithi159_102124_object1003:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2554c3db:::benchmark_data_smithi159_102124_object7723:head# 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.614 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25555ead:::benchmark_data_smithi159_102124_object38477:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25567a06:::benchmark_data_smithi159_102124_object16551:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2557d0ec:::benchmark_data_smithi159_102124_object8186:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2559fdfb:::benchmark_data_smithi159_102124_object41922:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:255cc214:::benchmark_data_smithi159_102124_object56355:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:255e3e37:::benchmark_data_smithi159_102124_object41511:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:255ffaa1:::benchmark_data_smithi159_102124_object30710:head# 2024-04-15T18:22:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2560aaca:::benchmark_data_smithi159_102124_object26738:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:256153dc:::benchmark_data_smithi159_102124_object22125:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25632061:::benchmark_data_smithi159_102124_object27950:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2563c8fe:::benchmark_data_smithi159_102124_object37642:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2563cc1c:::benchmark_data_smithi159_102124_object39987:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2567f34c:::benchmark_data_smithi159_102124_object32932:head# 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.616 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25694f4e:::benchmark_data_smithi159_102124_object36074:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:256b34ae:::benchmark_data_smithi159_102124_object2563:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:256c8f45:::benchmark_data_smithi159_102124_object34197:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:256ee4bc:::benchmark_data_smithi159_102124_object45501:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:256fad91:::benchmark_data_smithi159_102124_object16175:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2572200e:::benchmark_data_smithi159_102124_object10780:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25725786:::benchmark_data_smithi159_102124_object41686:head# 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.617 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2573cc46:::benchmark_data_smithi159_102124_object10989:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:257513a0:::benchmark_data_smithi159_102124_object32317:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2576929c:::benchmark_data_smithi159_102124_object52127:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25791d08:::benchmark_data_smithi159_102124_object5337:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2579d6ea:::benchmark_data_smithi159_102124_object289:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:257a3263:::benchmark_data_smithi159_102124_object6029:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25838eb6:::benchmark_data_smithi159_102124_object1843:head# 2024-04-15T18:22:41.618 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2583a2e5:::benchmark_data_smithi159_102124_object21785:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2587a1b4:::benchmark_data_smithi159_102124_object2065:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2587d1db:::benchmark_data_smithi159_102124_object12532:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25898c9b:::benchmark_data_smithi159_102124_object48858:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2589b953:::benchmark_data_smithi159_102124_object16229:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:258ab7c2:::benchmark_data_smithi159_102124_object56074:head# 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.619 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:258bebc8:::benchmark_data_smithi159_102124_object31886:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:258d09fd:::benchmark_data_smithi159_102124_object15173:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:258d5339:::benchmark_data_smithi159_102124_object892:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259049f5:::benchmark_data_smithi159_102124_object36324:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25914139:::benchmark_data_smithi159_102124_object15337:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259143bc:::benchmark_data_smithi159_102124_object8375:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259257c8:::benchmark_data_smithi159_102124_object45209:head# 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.620 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25934347:::benchmark_data_smithi159_102124_object35542:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259361bc:::benchmark_data_smithi159_102124_object56434:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259429fe:::benchmark_data_smithi159_102124_object28985:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25970dfc:::benchmark_data_smithi159_102124_object6778:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2597452a:::benchmark_data_smithi159_102124_object27342:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2597a43d:::benchmark_data_smithi159_102124_object31140:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2597b5ae:::benchmark_data_smithi159_102124_object42282:head# 2024-04-15T18:22:41.621 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259a0014:::benchmark_data_smithi159_102124_object47114:head# 2024-04-15T18:22:41.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.622 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259a7809:::benchmark_data_smithi159_102124_object10264:head# 2024-04-15T18:22:41.622 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259b131f:::benchmark_data_smithi159_102124_object57248:head# 2024-04-15T18:22:41.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259bcfe7:::benchmark_data_smithi159_102124_object25448:head# 2024-04-15T18:22:41.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259d253d:::benchmark_data_smithi159_102124_object43928:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259d8afc:::benchmark_data_smithi159_102124_object9262:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:259ddfa1:::benchmark_data_smithi159_102124_object616:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a0995d:::benchmark_data_smithi159_102124_object57945:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a0eaa7:::benchmark_data_smithi159_102124_object27203:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a113e1:::benchmark_data_smithi159_102124_object18986:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a1b827:::benchmark_data_smithi159_102124_object40348:head# 2024-04-15T18:22:41.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a471d3:::benchmark_data_smithi159_102124_object6552:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a4cb92:::benchmark_data_smithi159_102124_object56701:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a58c8f:::benchmark_data_smithi159_102124_object15366:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a7b8f9:::benchmark_data_smithi159_102124_object43464:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25a9876e:::benchmark_data_smithi159_102124_object12277:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ac892b:::benchmark_data_smithi159_102124_object38782:head# 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ae0d25:::benchmark_data_smithi159_102124_object20908:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25af0b04:::benchmark_data_smithi159_102124_object38621:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25af45fe:::benchmark_data_smithi159_102124_object23402:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25af8739:::benchmark_data_smithi159_102124_object9846:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b07421:::benchmark_data_smithi159_102124_object24703:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b24461:::benchmark_data_smithi159_102124_object6093:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b37472:::benchmark_data_smithi159_102124_object33212:head# 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b4469b:::benchmark_data_smithi159_102124_object7147:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b5c051:::benchmark_data_smithi159_102124_object57646:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b69594:::benchmark_data_smithi159_102124_object41802:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b6c422:::benchmark_data_smithi159_102124_object40000:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25b88842:::benchmark_data_smithi159_102124_object23442:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ba4665:::benchmark_data_smithi159_102124_object17094:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bb466f:::benchmark_data_smithi159_102124_object20102:head# 2024-04-15T18:22:41.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bb9470:::benchmark_data_smithi159_102124_object17775:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bc05f4:::benchmark_data_smithi159_102124_object6245:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bcb4d2:::benchmark_data_smithi159_102124_object45978:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25be5acf:::benchmark_data_smithi159_102124_object16177:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bf15dd:::benchmark_data_smithi159_102124_object23854:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bf2d36:::benchmark_data_smithi159_102124_object48075:head# 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25bf2ef4:::benchmark_data_smithi159_102124_object15532:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c07640:::benchmark_data_smithi159_102124_object25117:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c0d554:::benchmark_data_smithi159_102124_object10922:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c0d69a:::benchmark_data_smithi159_102124_object53001:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c166ac:::benchmark_data_smithi159_102124_object26320:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c1ccfc:::benchmark_data_smithi159_102124_object55110:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c34a01:::benchmark_data_smithi159_102124_object36804:head# 2024-04-15T18:22:41.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c40fc7:::benchmark_data_smithi159_102124_object16228:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c4691b:::benchmark_data_smithi159_102124_object29020:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c5f83e:::benchmark_data_smithi159_102124_object25507:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c731f0:::benchmark_data_smithi159_102124_object16876:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c84788:::benchmark_data_smithi159_102124_object22993:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c863fa:::benchmark_data_smithi159_102124_object48637:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25c8c514:::benchmark_data_smithi159_102124_object53577:head# 2024-04-15T18:22:41.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ca7d03:::benchmark_data_smithi159_102124_object50032:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cb5629:::benchmark_data_smithi159_102124_object8214:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cc3120:::benchmark_data_smithi159_102124_object9656:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cc5c98:::benchmark_data_smithi159_102124_object53811:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cc61ac:::benchmark_data_smithi159_102124_object50875:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cc91f7:::benchmark_data_smithi159_102124_object2864:head# 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cd4c5f:::benchmark_data_smithi159_102124_object24344:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cd602c:::benchmark_data_smithi159_102124_object8260:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cd6887:::benchmark_data_smithi159_102124_object26616:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ce5ece:::benchmark_data_smithi159_102124_object48213:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cedb88:::benchmark_data_smithi159_102124_object29328:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cf4945:::benchmark_data_smithi159_102124_object18437:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25cfe85d:::benchmark_data_smithi159_102124_object28378:head# 2024-04-15T18:22:41.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d0c053:::benchmark_data_smithi159_102124_object7622:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d0f225:::benchmark_data_smithi159_102124_object44872:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d335e4:::benchmark_data_smithi159_102124_object36143:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d353d3:::benchmark_data_smithi159_102124_object43037:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d523b6:::benchmark_data_smithi159_102124_object26022:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d5c3ba:::benchmark_data_smithi159_102124_object48366:head# 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d659d5:::benchmark_data_smithi159_102124_object18058:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d74737:::benchmark_data_smithi159_102124_object41983:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d76d73:::benchmark_data_smithi159_102124_object37482:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d7a933:::benchmark_data_smithi159_102124_object7530:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d9c4a8:::benchmark_data_smithi159_102124_object50646:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25d9c4cc:::benchmark_data_smithi159_102124_object2298:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25daf94e:::benchmark_data_smithi159_102124_object10230:head# 2024-04-15T18:22:41.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25dd133e:::benchmark_data_smithi159_102124_object4892:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ddf9b5:::benchmark_data_smithi159_102124_object35435:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25df1ac1:::benchmark_data_smithi159_102124_object29923:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e15e72:::benchmark_data_smithi159_102124_object41931:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e445df:::benchmark_data_smithi159_102124_object9110:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e55333:::benchmark_data_smithi159_102124_object41398:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e65800:::benchmark_data_smithi159_102124_object31420:head# 2024-04-15T18:22:41.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e70ffc:::benchmark_data_smithi159_102124_object54875:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25e8cd7d:::benchmark_data_smithi159_102124_object33185:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ea01ce:::benchmark_data_smithi159_102124_object10767:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25eb0d92:::benchmark_data_smithi159_102124_object55784:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25ed154a:::benchmark_data_smithi159_102124_object54644:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25eedc3d:::benchmark_data_smithi159_102124_object44313:head# 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.647 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f022b0:::benchmark_data_smithi159_102124_object9805:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f02e4c:::benchmark_data_smithi159_102124_object51340:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f0db1d:::benchmark_data_smithi159_102124_object22331:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f16f73:::benchmark_data_smithi159_102124_object49428:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f2d509:::benchmark_data_smithi159_102124_object24306:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f4ecc4:::benchmark_data_smithi159_102124_object26322:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f76982:::benchmark_data_smithi159_102124_object51594:head# 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.648 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f7d69b:::benchmark_data_smithi159_102124_object25914:head# 2024-04-15T18:22:41.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f8f7a4:::benchmark_data_smithi159_102124_object44599:head# 2024-04-15T18:22:41.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.649 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25f9d393:::benchmark_data_smithi159_102124_object29203:head# 2024-04-15T18:22:41.649 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fa4f03:::benchmark_data_smithi159_102124_object15709:head# 2024-04-15T18:22:41.652 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.652 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25faa1e8:::benchmark_data_smithi159_102124_object32987:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fb255b:::benchmark_data_smithi159_102124_object4005:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fbad0c:::benchmark_data_smithi159_102124_object44504:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fbced7:::benchmark_data_smithi159_102124_object43575:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fd3d22:::benchmark_data_smithi159_102124_object40363:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fd5b69:::benchmark_data_smithi159_102124_object42474:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fdd9ae:::benchmark_data_smithi159_102124_object11911:head# 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fe5e39:::benchmark_data_smithi159_102124_object342:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:25fe75a7:::benchmark_data_smithi159_102124_object9862:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2600e64a:::benchmark_data_smithi159_102124_object8502:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2601d2b1:::benchmark_data_smithi159_102124_object40103:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260353b2:::benchmark_data_smithi159_102124_object38610:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26037583:::benchmark_data_smithi159_102124_object42127:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2603c7c1:::benchmark_data_smithi159_102124_object47167:head# 2024-04-15T18:22:41.654 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2603eb2d:::benchmark_data_smithi159_102124_object44842:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2604bfe2:::benchmark_data_smithi159_102124_object52467:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26054a2a:::benchmark_data_smithi159_102124_object17812:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2605734b:::benchmark_data_smithi159_102124_object11133:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26057b53:::benchmark_data_smithi159_102124_object53384:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26057bcb:::benchmark_data_smithi159_102124_object42416:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2605c5a0:::benchmark_data_smithi159_102124_object22978:head# 2024-04-15T18:22:41.655 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26061adf:::benchmark_data_smithi159_102124_object40323:head# 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2606a2a8:::benchmark_data_smithi159_102124_object22979:head# 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26084dbc:::benchmark_data_smithi159_102124_object41796:head# 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26086127:::benchmark_data_smithi159_102124_object53069:head# 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2608da5e:::benchmark_data_smithi159_102124_object50668:head# 2024-04-15T18:22:41.656 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26090209:::benchmark_data_smithi159_102124_object46598:head# 2024-04-15T18:22:41.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.657 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26097ea4:::benchmark_data_smithi159_102124_object2402:head# 2024-04-15T18:22:41.657 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260980ce:::benchmark_data_smithi159_102124_object24293:head# 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260a32e8:::benchmark_data_smithi159_102124_object58189:head# 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260a7898:::benchmark_data_smithi159_102124_object36649:head# 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260c44d9:::benchmark_data_smithi159_102124_object35821:head# 2024-04-15T18:22:41.661 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260d7b49:::benchmark_data_smithi159_102124_object2836:head# 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260f6f96:::benchmark_data_smithi159_102124_object14419:head# 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260f8d86:::benchmark_data_smithi159_102124_object51523:head# 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:260fcc7e:::benchmark_data_smithi159_102124_object7617:head# 2024-04-15T18:22:41.664 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26109614:::benchmark_data_smithi159_102124_object52194:head# 2024-04-15T18:22:41.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26121853:::benchmark_data_smithi159_102124_object51933:head# 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2612fb78:::benchmark_data_smithi159_102124_object27880:head# 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2613b6d7:::benchmark_data_smithi159_102124_object46441:head# 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2613e361:::benchmark_data_smithi159_102124_object3782:head# 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2614d264:::benchmark_data_smithi159_102124_object18642:head# 2024-04-15T18:22:41.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2615e903:::benchmark_data_smithi159_102124_object29295:head# 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26182b62:::benchmark_data_smithi159_102124_object2968:head# 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261845ab:::benchmark_data_smithi159_102124_object57037:head# 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2618ae41:::benchmark_data_smithi159_102124_object49652:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2619b3c5:::benchmark_data_smithi159_102124_object4205:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261c1910:::benchmark_data_smithi159_102124_object17947:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261d0106:::benchmark_data_smithi159_102124_object40450:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261d340f:::benchmark_data_smithi159_102124_object44775:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261d8b33:::benchmark_data_smithi159_102124_object9171:head# 2024-04-15T18:22:41.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261da60a:::benchmark_data_smithi159_102124_object30360:head# 2024-04-15T18:22:41.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261dee27:::benchmark_data_smithi159_102124_object13265:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:261e4d8a:::benchmark_data_smithi159_102124_object22598:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2620190f:::benchmark_data_smithi159_102124_object36112:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262050af:::benchmark_data_smithi159_102124_object25751:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2620df56:::benchmark_data_smithi159_102124_object31734:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2620dfe6:::benchmark_data_smithi159_102124_object40168:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26216631:::benchmark_data_smithi159_102124_object20533:head# 2024-04-15T18:22:41.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26227863:::benchmark_data_smithi159_102124_object55723:head# 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26247b22:::benchmark_data_smithi159_102124_object54919:head# 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2624839e:::benchmark_data_smithi159_102124_object18163:head# 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26269f72:::benchmark_data_smithi159_102124_object5920:head# 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2626c1e7:::benchmark_data_smithi159_102124_object1974:head# 2024-04-15T18:22:41.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2626f15c:::benchmark_data_smithi159_102124_object38154:head# 2024-04-15T18:22:41.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2629291b:::benchmark_data_smithi159_102124_object19995:head# 2024-04-15T18:22:41.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262957bf:::benchmark_data_smithi159_102124_object36155:head# 2024-04-15T18:22:41.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2629bd5b:::benchmark_data_smithi159_102124_object15339:head# 2024-04-15T18:22:41.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262a2bde:::benchmark_data_smithi159_102124_object19684:head# 2024-04-15T18:22:41.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.687 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262d2b3b:::benchmark_data_smithi159_102124_object34312:head# 2024-04-15T18:22:41.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.688 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262d53a8:::benchmark_data_smithi159_102124_object26860:head# 2024-04-15T18:22:41.688 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262d6750:::benchmark_data_smithi159_102124_object16982:head# 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262de795:::benchmark_data_smithi159_102124_object30055:head# 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262e4a95:::benchmark_data_smithi159_102124_object56784:head# 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:262f27c2:::benchmark_data_smithi159_102124_object3593:head# 2024-04-15T18:22:41.689 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26319d53:::benchmark_data_smithi159_102124_object27638:head# 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26325c18:::benchmark_data_smithi159_102124_object284:head# 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26341814:::benchmark_data_smithi159_102124_object57024:head# 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26348456:::benchmark_data_smithi159_102124_object44576:head# 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:263770e0:::benchmark_data_smithi159_102124_object15956:head# 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2637b3dd:::benchmark_data_smithi159_102124_object18930:head# 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2638fa82:::benchmark_data_smithi159_102124_object42659:head# 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:263bd974:::benchmark_data_smithi159_102124_object24656:head# 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:263c0ba7:::benchmark_data_smithi159_102124_object23528:head# 2024-04-15T18:22:41.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:263e1376:::benchmark_data_smithi159_102124_object18396:head# 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:263f4853:::benchmark_data_smithi159_102124_object12224:head# 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26402021:::benchmark_data_smithi159_102124_object49671:head# 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:264298c4:::benchmark_data_smithi159_102124_object14309:head# 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26436d16:::benchmark_data_smithi159_102124_object27826:head# 2024-04-15T18:22:41.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26440dee:::benchmark_data_smithi159_102124_object8440:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26444ef7:::benchmark_data_smithi159_102124_object20873:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2644caa5:::benchmark_data_smithi159_102124_object41596:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26455909:::benchmark_data_smithi159_102124_object50873:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26465148:::benchmark_data_smithi159_102124_object20364:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:264779a6:::benchmark_data_smithi159_102124_object45447:head# 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2647e06a:::benchmark_data_smithi159_102124_object51028:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26498561:::benchmark_data_smithi159_102124_object58013:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:264a95f7:::benchmark_data_smithi159_102124_object24167:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:264e9b98:::benchmark_data_smithi159_102124_object54471:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2650064c:::benchmark_data_smithi159_102124_object58304:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26505a1f:::benchmark_data_smithi159_102124_object44375:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2652b99f:::benchmark_data_smithi159_102124_object15305:head# 2024-04-15T18:22:41.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2652f022:::benchmark_data_smithi159_102124_object46213:head# 2024-04-15T18:22:41.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2653d349:::benchmark_data_smithi159_102124_object56799:head# 2024-04-15T18:22:41.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26545915:::benchmark_data_smithi159_102124_object33611:head# 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26561c7e:::benchmark_data_smithi159_102124_object4960:head# 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26577c94:::benchmark_data_smithi159_102124_object19485:head# 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26595fbb:::benchmark_data_smithi159_102124_object38816:head# 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2659ac8e:::benchmark_data_smithi159_102124_object16925:head# 2024-04-15T18:22:41.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265a6680:::benchmark_data_smithi159_102124_object18206:head# 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265b5ca0:::benchmark_data_smithi159_102124_object32125:head# 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265c796b:::benchmark_data_smithi159_102124_object27619:head# 2024-04-15T18:22:41.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.701 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265e0307:::benchmark_data_smithi159_102124_object10101:head# 2024-04-15T18:22:41.701 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.705 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265e152f:::benchmark_data_smithi159_102124_object19409:head# 2024-04-15T18:22:41.705 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:265f683b:::benchmark_data_smithi159_102124_object33404:head# 2024-04-15T18:22:41.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.706 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2660795d:::benchmark_data_smithi159_102124_object4772:head# 2024-04-15T18:22:41.706 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2663037e:::benchmark_data_smithi159_102124_object15236:head# 2024-04-15T18:22:41.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2664f5c2:::benchmark_data_smithi159_102124_object20043:head# 2024-04-15T18:22:41.707 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.707 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26686cbb:::benchmark_data_smithi159_102124_object43165:head# 2024-04-15T18:22:41.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266921d3:::benchmark_data_smithi159_102124_object20667:head# 2024-04-15T18:22:41.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.708 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266a253c:::benchmark_data_smithi159_102124_object22241:head# 2024-04-15T18:22:41.708 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266a6785:::benchmark_data_smithi159_102124_object50359:head# 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266bfe7d:::benchmark_data_smithi159_102124_object22747:head# 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266c1ac8:::benchmark_data_smithi159_102124_object22944:head# 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.709 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266ce958:::benchmark_data_smithi159_102124_object14199:head# 2024-04-15T18:22:41.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266d240e:::benchmark_data_smithi159_102124_object42163:head# 2024-04-15T18:22:41.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.710 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266e2cc6:::benchmark_data_smithi159_102124_object51321:head# 2024-04-15T18:22:41.710 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.711 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266f2273:::benchmark_data_smithi159_102124_object37082:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:266f2708:::benchmark_data_smithi159_102124_object30085:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26710614:::benchmark_data_smithi159_102124_object18819:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2672a879:::benchmark_data_smithi159_102124_object854:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:267772b7:::benchmark_data_smithi159_102124_object26952:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26783ced:::benchmark_data_smithi159_102124_object47226:head# 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.712 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26789f8b:::benchmark_data_smithi159_102124_object43078:head# 2024-04-15T18:22:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:267a1467:::benchmark_data_smithi159_102124_object22032:head# 2024-04-15T18:22:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:267cd7b6:::benchmark_data_smithi159_102124_object349:head# 2024-04-15T18:22:41.713 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26818217:::benchmark_data_smithi159_102124_object48816:head# 2024-04-15T18:22:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26818584:::benchmark_data_smithi159_102124_object1819:head# 2024-04-15T18:22:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268368f7:::benchmark_data_smithi159_102124_object55866:head# 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26842183:::benchmark_data_smithi159_102124_object55507:head# 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268623f4:::benchmark_data_smithi159_102124_object49186:head# 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26863674:::benchmark_data_smithi159_102124_object30270:head# 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.720 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2686adf5:::benchmark_data_smithi159_102124_object10129:head# 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2687851b:::benchmark_data_smithi159_102124_object12657:head# 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26881390:::benchmark_data_smithi159_102124_object49902:head# 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.721 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26884aad:::benchmark_data_smithi159_102124_object53010:head# 2024-04-15T18:22:41.722 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2688ce40:::benchmark_data_smithi159_102124_object28886:head# 2024-04-15T18:22:41.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2688fd8c:::benchmark_data_smithi159_102124_object50907:head# 2024-04-15T18:22:41.732 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.732 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268bc4bd:::benchmark_data_smithi159_102124_object23952:head# 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268bd48b:::benchmark_data_smithi159_102124_object3200:head# 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268c0b61:::benchmark_data_smithi159_102124_object50057:head# 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.733 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268e5104:::benchmark_data_smithi159_102124_object8935:head# 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:268f6a26:::benchmark_data_smithi159_102124_object24008:head# 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26900f18:::benchmark_data_smithi159_102124_object43176:head# 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.734 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26910caa:::benchmark_data_smithi159_102124_object19533:head# 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269165d3:::benchmark_data_smithi159_102124_object7348:head# 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269460e4:::benchmark_data_smithi159_102124_object11972:head# 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.735 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2696131a:::benchmark_data_smithi159_102124_object53319:head# 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2696d3a3:::benchmark_data_smithi159_102124_object54229:head# 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269836bb:::benchmark_data_smithi159_102124_object33936:head# 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.736 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26988030:::benchmark_data_smithi159_102124_object33150:head# 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269aa7fa:::benchmark_data_smithi159_102124_object25811:head# 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269bfefd:::benchmark_data_smithi159_102124_object31844:head# 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.737 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269c1860:::benchmark_data_smithi159_102124_object45207:head# 2024-04-15T18:22:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269c52b1:::benchmark_data_smithi159_102124_object13019:head# 2024-04-15T18:22:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:269d6197:::benchmark_data_smithi159_102124_object11218:head# 2024-04-15T18:22:41.738 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a2129a:::benchmark_data_smithi159_102124_object12467:head# 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a23738:::benchmark_data_smithi159_102124_object32499:head# 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a2412f:::benchmark_data_smithi159_102124_object42732:head# 2024-04-15T18:22:41.739 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a2b190:::benchmark_data_smithi159_102124_object46415:head# 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a2b2cf:::benchmark_data_smithi159_102124_object25353:head# 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a39721:::benchmark_data_smithi159_102124_object35238:head# 2024-04-15T18:22:41.740 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a3e49f:::benchmark_data_smithi159_102124_object7134:head# 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a441be:::benchmark_data_smithi159_102124_object47766:head# 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26a4f6e1:::benchmark_data_smithi159_102124_object11985:head# 2024-04-15T18:22:41.741 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26aa3768:::benchmark_data_smithi159_102124_object5525:head# 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26aa6bda:::benchmark_data_smithi159_102124_object35906:head# 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26aaadf7:::benchmark_data_smithi159_102124_object42769:head# 2024-04-15T18:22:41.742 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26abee1b:::benchmark_data_smithi159_102124_object12257:head# 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ae4ed7:::benchmark_data_smithi159_102124_object33319:head# 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26aeef9a:::benchmark_data_smithi159_102124_object41571:head# 2024-04-15T18:22:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26af391c:::benchmark_data_smithi159_102124_object13798:head# 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b1667e:::benchmark_data_smithi159_102124_object28409:head# 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b3aeb1:::benchmark_data_smithi159_102124_object25257:head# 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b444a4:::benchmark_data_smithi159_102124_object7240:head# 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b51318:::benchmark_data_smithi159_102124_object30672:head# 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.754 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b51875:::benchmark_data_smithi159_102124_object57520:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b720c8:::benchmark_data_smithi159_102124_object45245:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b76b6d:::benchmark_data_smithi159_102124_object43449:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b88e07:::benchmark_data_smithi159_102124_object41224:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b8a6a4:::benchmark_data_smithi159_102124_object19171:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26b96c26:::benchmark_data_smithi159_102124_object9405:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26bbd3ca:::benchmark_data_smithi159_102124_object57333:head# 2024-04-15T18:22:41.755 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26bcca55:::benchmark_data_smithi159_102124_object27571:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26be1c16:::benchmark_data_smithi159_102124_object34806:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26be35ec:::benchmark_data_smithi159_102124_object6085:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26bf35f9:::benchmark_data_smithi159_102124_object37506:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26bf366c:::benchmark_data_smithi159_102124_object43336:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26bf5943:::benchmark_data_smithi159_102124_object3163:head# 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.756 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c06ed1:::benchmark_data_smithi159_102124_object56992:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c14039:::benchmark_data_smithi159_102124_object26849:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c18a13:::benchmark_data_smithi159_102124_object16995:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c21978:::benchmark_data_smithi159_102124_object41009:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c287cb:::benchmark_data_smithi159_102124_object31408:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c2a143:::benchmark_data_smithi159_102124_object34477:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c2d76c:::benchmark_data_smithi159_102124_object36966:head# 2024-04-15T18:22:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c35952:::benchmark_data_smithi159_102124_object12522:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c3664a:::benchmark_data_smithi159_102124_object52557:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c46bb3:::benchmark_data_smithi159_102124_object47513:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c4eabc:::benchmark_data_smithi159_102124_object16110:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c56358:::benchmark_data_smithi159_102124_object10500:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26c8c39f:::benchmark_data_smithi159_102124_object25487:head# 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.758 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ca8fdd:::benchmark_data_smithi159_102124_object9401:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d01817:::benchmark_data_smithi159_102124_object35576:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d0d614:::benchmark_data_smithi159_102124_object38215:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d181e8:::benchmark_data_smithi159_102124_object53753:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d2bc54:::benchmark_data_smithi159_102124_object28790:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d2d019:::benchmark_data_smithi159_102124_object20071:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d42480:::benchmark_data_smithi159_102124_object50133:head# 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d972bb:::benchmark_data_smithi159_102124_object20704:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d98fd6:::benchmark_data_smithi159_102124_object18828:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26d9edaf:::benchmark_data_smithi159_102124_object17722:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26da1c25:::benchmark_data_smithi159_102124_object1533:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26db00ea:::benchmark_data_smithi159_102124_object41035:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26db1a3e:::benchmark_data_smithi159_102124_object54365:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26db53ba:::benchmark_data_smithi159_102124_object50302:head# 2024-04-15T18:22:41.760 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26dc61d1:::benchmark_data_smithi159_102124_object52377:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26dc71a9:::benchmark_data_smithi159_102124_object23458:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26dc8c6d:::benchmark_data_smithi159_102124_object29214:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26de62f4:::benchmark_data_smithi159_102124_object42671:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e10318:::benchmark_data_smithi159_102124_object19946:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e18b44:::benchmark_data_smithi159_102124_object47379:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e33cce:::benchmark_data_smithi159_102124_object45879:head# 2024-04-15T18:22:41.761 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e3e463:::benchmark_data_smithi159_102124_object26577:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e51d20:::benchmark_data_smithi159_102124_object11369:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e5cab5:::benchmark_data_smithi159_102124_object56800:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e5da03:::benchmark_data_smithi159_102124_object52503:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e8c7a4:::benchmark_data_smithi159_102124_object53308:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26e94e39:::benchmark_data_smithi159_102124_object28482:head# 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ea07f3:::benchmark_data_smithi159_102124_object12734:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26eb8caa:::benchmark_data_smithi159_102124_object44896:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ecf77a:::benchmark_data_smithi159_102124_object20455:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26eecdca:::benchmark_data_smithi159_102124_object28357:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f04e06:::benchmark_data_smithi159_102124_object53748:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f1c47f:::benchmark_data_smithi159_102124_object5763:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f34483:::benchmark_data_smithi159_102124_object3308:head# 2024-04-15T18:22:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f3ad31:::benchmark_data_smithi159_102124_object3745:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f55995:::benchmark_data_smithi159_102124_object9218:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f5a001:::benchmark_data_smithi159_102124_object25501:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f5a3ae:::benchmark_data_smithi159_102124_object29427:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f5e0e5:::benchmark_data_smithi159_102124_object57240:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f73cf2:::benchmark_data_smithi159_102124_object7703:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f7432e:::benchmark_data_smithi159_102124_object15722:head# 2024-04-15T18:22:41.764 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f7c8c9:::benchmark_data_smithi159_102124_object11600:head# 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26f91708:::benchmark_data_smithi159_102124_object17115:head# 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26fca5dc:::benchmark_data_smithi159_102124_object45176:head# 2024-04-15T18:22:41.765 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.770 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26fdd575:::benchmark_data_smithi159_102124_object52751:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26fe3855:::benchmark_data_smithi159_102124_object56264:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26fed884:::benchmark_data_smithi159_102124_object56955:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ff1e36:::benchmark_data_smithi159_102124_object418:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ff2947:::benchmark_data_smithi159_102124_object17050:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:26ff5fc6:::benchmark_data_smithi159_102124_object7489:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27007734:::benchmark_data_smithi159_102124_object53690:head# 2024-04-15T18:22:41.771 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2700f793:::benchmark_data_smithi159_102124_object25154:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27015e85:::benchmark_data_smithi159_102124_object8398:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27036371:::benchmark_data_smithi159_102124_object7537:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2703b112:::benchmark_data_smithi159_102124_object9503:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2703bdd4:::benchmark_data_smithi159_102124_object39214:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27053816:::benchmark_data_smithi159_102124_object32585:head# 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27058b27:::benchmark_data_smithi159_102124_object52228:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:270601a2:::benchmark_data_smithi159_102124_object41416:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27067f3c:::benchmark_data_smithi159_102124_object47099:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2708e514:::benchmark_data_smithi159_102124_object36788:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27093952:::benchmark_data_smithi159_102124_object28233:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:270adfb8:::benchmark_data_smithi159_102124_object7646:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:270d78c0:::benchmark_data_smithi159_102124_object34865:head# 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:270f04de:::benchmark_data_smithi159_102124_object48350:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:270f58f6:::benchmark_data_smithi159_102124_object56329:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271010fb:::benchmark_data_smithi159_102124_object9297:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2711c30c:::benchmark_data_smithi159_102124_object19131:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2713d3a4:::benchmark_data_smithi159_102124_object37584:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27161626:::benchmark_data_smithi159_102124_object31915:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27179507:::benchmark_data_smithi159_102124_object14631:head# 2024-04-15T18:22:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2718f3b8:::benchmark_data_smithi159_102124_object3865:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2719b95f:::benchmark_data_smithi159_102124_object58252:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2719cd29:::benchmark_data_smithi159_102124_object28024:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271af96c:::benchmark_data_smithi159_102124_object52582:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271cba92:::benchmark_data_smithi159_102124_object6131:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271e3564:::benchmark_data_smithi159_102124_object17159:head# 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271e89a9:::benchmark_data_smithi159_102124_object9612:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271ec047:::benchmark_data_smithi159_102124_object1914:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271f3363:::benchmark_data_smithi159_102124_object22631:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271f86f7:::benchmark_data_smithi159_102124_object48004:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:271f9ef0:::benchmark_data_smithi159_102124_object40866:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272015ff:::benchmark_data_smithi159_102124_object26784:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27208ee1:::benchmark_data_smithi159_102124_object16102:head# 2024-04-15T18:22:41.776 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27212dbb:::benchmark_data_smithi159_102124_object36753:head# 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2722c367:::benchmark_data_smithi159_102124_object36703:head# 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272317ef:::benchmark_data_smithi159_102124_object4462:head# 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272321df:::benchmark_data_smithi159_102124_object56132:head# 2024-04-15T18:22:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27233a01:::benchmark_data_smithi159_102124_object54884:head# 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2724f9b2:::benchmark_data_smithi159_102124_object18131:head# 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27266f42:::benchmark_data_smithi159_102124_object8180:head# 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2729ee32:::benchmark_data_smithi159_102124_object57915:head# 2024-04-15T18:22:41.778 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272ab165:::benchmark_data_smithi159_102124_object3460:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272ad91e:::benchmark_data_smithi159_102124_object39345:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272b734c:::benchmark_data_smithi159_102124_object776:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272b8520:::benchmark_data_smithi159_102124_object43270:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272b8b83:::benchmark_data_smithi159_102124_object41985:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272c9755:::benchmark_data_smithi159_102124_object54986:head# 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272df14e:::benchmark_data_smithi159_102124_object21346:head# 2024-04-15T18:22:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272e8009:::benchmark_data_smithi159_102124_object45120:head# 2024-04-15T18:22:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272ec561:::benchmark_data_smithi159_102124_object25913:head# 2024-04-15T18:22:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272faba7:::benchmark_data_smithi159_102124_object50820:head# 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:272fbd31:::benchmark_data_smithi159_102124_object35807:head# 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2733d69d:::benchmark_data_smithi159_102124_object30651:head# 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27345b28:::benchmark_data_smithi159_102124_object9832:head# 2024-04-15T18:22:41.781 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27351528:::benchmark_data_smithi159_102124_object28650:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27362fff:::benchmark_data_smithi159_102124_object56681:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2736425b:::benchmark_data_smithi159_102124_object55962:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27373213:::benchmark_data_smithi159_102124_object21238:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:273a8947:::benchmark_data_smithi159_102124_object30131:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:273cf0ba:::benchmark_data_smithi159_102124_object43196:head# 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.782 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:273d2e11:::benchmark_data_smithi159_102124_object32614:head# 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:273ee8dc:::benchmark_data_smithi159_102124_object44597:head# 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:274290f9:::benchmark_data_smithi159_102124_object58316:head# 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27433633:::benchmark_data_smithi159_102124_object27030:head# 2024-04-15T18:22:41.783 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.785 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2744abec:::benchmark_data_smithi159_102124_object4987:head# 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27458a8a:::benchmark_data_smithi159_102124_object1248:head# 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27459e45:::benchmark_data_smithi159_102124_object49111:head# 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27464a7c:::benchmark_data_smithi159_102124_object40198:head# 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27467d1c:::benchmark_data_smithi159_102124_object33414:head# 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.786 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2746d823:::benchmark_data_smithi159_102124_object54485:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2748d586:::benchmark_data_smithi159_102124_object14347:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2748edda:::benchmark_data_smithi159_102124_object5999:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2749adae:::benchmark_data_smithi159_102124_object51919:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:274af554:::benchmark_data_smithi159_102124_object27652:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:274f3a6c:::benchmark_data_smithi159_102124_object46615:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2750bf2b:::benchmark_data_smithi159_102124_object53279:head# 2024-04-15T18:22:41.787 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27510151:::benchmark_data_smithi159_102124_object53359:head# 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275102d9:::benchmark_data_smithi159_102124_object50150:head# 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275142c6:::benchmark_data_smithi159_102124_object3148:head# 2024-04-15T18:22:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2751f4a6:::benchmark_data_smithi159_102124_object1768:head# 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2752f8cd:::benchmark_data_smithi159_102124_object1958:head# 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275477e1:::benchmark_data_smithi159_102124_object43258:head# 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2754b925:::benchmark_data_smithi159_102124_object30513:head# 2024-04-15T18:22:41.789 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.795 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27555853:::benchmark_data_smithi159_102124_object26603:head# 2024-04-15T18:22:41.796 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.798 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2755b4a5:::benchmark_data_smithi159_102124_object7621:head# 2024-04-15T18:22:41.798 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.799 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275642b2:::benchmark_data_smithi159_102124_object45261:head# 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2756b8bc:::benchmark_data_smithi159_102124_object15689:head# 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2756de7d:::benchmark_data_smithi159_102124_object20275:head# 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2756e006:::benchmark_data_smithi159_102124_object39665:head# 2024-04-15T18:22:41.801 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2757beb8:::benchmark_data_smithi159_102124_object16128:head# 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2758efc9:::benchmark_data_smithi159_102124_object13085:head# 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275b8499:::benchmark_data_smithi159_102124_object38090:head# 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275bdde7:::benchmark_data_smithi159_102124_object29251:head# 2024-04-15T18:22:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275c7885:::benchmark_data_smithi159_102124_object53602:head# 2024-04-15T18:22:41.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.803 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275cc951:::benchmark_data_smithi159_102124_object12649:head# 2024-04-15T18:22:41.803 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275d28a0:::benchmark_data_smithi159_102124_object53893:head# 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:275edf99:::benchmark_data_smithi159_102124_object9895:head# 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:276087ed:::benchmark_data_smithi159_102124_object32423:head# 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2762aebf:::benchmark_data_smithi159_102124_object11460:head# 2024-04-15T18:22:41.804 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2767c40a:::benchmark_data_smithi159_102124_object50433:head# 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2768f3ec:::benchmark_data_smithi159_102124_object25778:head# 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:276a7dc4:::benchmark_data_smithi159_102124_object53634:head# 2024-04-15T18:22:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:276bca36:::benchmark_data_smithi159_102124_object48036:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:276bda7d:::benchmark_data_smithi159_102124_object16378:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27701011:::benchmark_data_smithi159_102124_object25514:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277118dd:::benchmark_data_smithi159_102124_object10503:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2771f51d:::benchmark_data_smithi159_102124_object44713:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27722e89:::benchmark_data_smithi159_102124_object14966:head# 2024-04-15T18:22:41.807 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27723b6c:::benchmark_data_smithi159_102124_object44927:head# 2024-04-15T18:22:41.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.808 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277573a0:::benchmark_data_smithi159_102124_object23019:head# 2024-04-15T18:22:41.808 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27772f54:::benchmark_data_smithi159_102124_object7053:head# 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27788afd:::benchmark_data_smithi159_102124_object21103:head# 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277919dd:::benchmark_data_smithi159_102124_object41372:head# 2024-04-15T18:22:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27796d94:::benchmark_data_smithi159_102124_object56219:head# 2024-04-15T18:22:41.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.814 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2779dafc:::benchmark_data_smithi159_102124_object47236:head# 2024-04-15T18:22:41.814 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2779e1b7:::benchmark_data_smithi159_102124_object21595:head# 2024-04-15T18:22:41.821 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.821 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277bea72:::benchmark_data_smithi159_102124_object48920:head# 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277c92eb:::benchmark_data_smithi159_102124_object6770:head# 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:277de83b:::benchmark_data_smithi159_102124_object38041:head# 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.822 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27806079:::benchmark_data_smithi159_102124_object3814:head# 2024-04-15T18:22:41.823 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2781f504:::benchmark_data_smithi159_102124_object13365:head# 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:278262ec:::benchmark_data_smithi159_102124_object33182:head# 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2787268a:::benchmark_data_smithi159_102124_object41702:head# 2024-04-15T18:22:41.825 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2788caf6:::benchmark_data_smithi159_102124_object8030:head# 2024-04-15T18:22:41.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.826 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:278c2598:::benchmark_data_smithi159_102124_object1128:head# 2024-04-15T18:22:41.826 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.829 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:278d69f3:::benchmark_data_smithi159_102124_object19583:head# 2024-04-15T18:22:41.829 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:278d9467:::benchmark_data_smithi159_102124_object29986:head# 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:278f86fc:::benchmark_data_smithi159_102124_object42075:head# 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2790317a:::benchmark_data_smithi159_102124_object21130:head# 2024-04-15T18:22:41.830 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.831 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27916548:::benchmark_data_smithi159_102124_object30183:head# 2024-04-15T18:22:41.831 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.833 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27922a58:::benchmark_data_smithi159_102124_object13996:head# 2024-04-15T18:22:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27947335:::benchmark_data_smithi159_102124_object56360:head# 2024-04-15T18:22:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:279494b4:::benchmark_data_smithi159_102124_object30761:head# 2024-04-15T18:22:41.834 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2795a81a:::benchmark_data_smithi159_102124_object49768:head# 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27988e45:::benchmark_data_smithi159_102124_object41835:head# 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2798ef35:::benchmark_data_smithi159_102124_object52716:head# 2024-04-15T18:22:41.835 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27999a33:::benchmark_data_smithi159_102124_object3063:head# 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:279ca42d:::benchmark_data_smithi159_102124_object28058:head# 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:279e2430:::benchmark_data_smithi159_102124_object12157:head# 2024-04-15T18:22:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:279fbcc1:::benchmark_data_smithi159_102124_object3067:head# 2024-04-15T18:22:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a2324a:::benchmark_data_smithi159_102124_object3504:head# 2024-04-15T18:22:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.838 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a27ff3:::benchmark_data_smithi159_102124_object18539:head# 2024-04-15T18:22:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a2de60:::benchmark_data_smithi159_102124_object40470:head# 2024-04-15T18:22:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a30f81:::benchmark_data_smithi159_102124_object27630:head# 2024-04-15T18:22:41.839 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a55016:::benchmark_data_smithi159_102124_object58160:head# 2024-04-15T18:22:41.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.842 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a5aa42:::benchmark_data_smithi159_102124_object3191:head# 2024-04-15T18:22:41.842 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a67a42:::benchmark_data_smithi159_102124_object47233:head# 2024-04-15T18:22:41.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.847 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a7dbee:::benchmark_data_smithi159_102124_object6078:head# 2024-04-15T18:22:41.847 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a7fe02:::benchmark_data_smithi159_102124_object33481:head# 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27a958e2:::benchmark_data_smithi159_102124_object54687:head# 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27aa4c4c:::benchmark_data_smithi159_102124_object53153:head# 2024-04-15T18:22:41.848 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27acd251:::benchmark_data_smithi159_102124_object38132:head# 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27adc61a:::benchmark_data_smithi159_102124_object50303:head# 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27addbef:::benchmark_data_smithi159_102124_object26458:head# 2024-04-15T18:22:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27af5893:::benchmark_data_smithi159_102124_object16529:head# 2024-04-15T18:22:41.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27b58ad5:::benchmark_data_smithi159_102124_object43376:head# 2024-04-15T18:22:41.850 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.850 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27b6c306:::benchmark_data_smithi159_102124_object50353:head# 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27b8c4d2:::benchmark_data_smithi159_102124_object12440:head# 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27b8e2c8:::benchmark_data_smithi159_102124_object58454:head# 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27b9e94b:::benchmark_data_smithi159_102124_object25010:head# 2024-04-15T18:22:41.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.852 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27bad7a7:::benchmark_data_smithi159_102124_object28133:head# 2024-04-15T18:22:41.852 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27bbf8aa:::benchmark_data_smithi159_102124_object14911:head# 2024-04-15T18:22:41.853 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.853 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27bd67f8:::benchmark_data_smithi159_102124_object12707:head# 2024-04-15T18:22:41.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c22a26:::benchmark_data_smithi159_102124_object18903:head# 2024-04-15T18:22:41.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.854 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c2da12:::benchmark_data_smithi159_102124_object442:head# 2024-04-15T18:22:41.854 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c2f271:::benchmark_data_smithi159_102124_object8978:head# 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c409dc:::benchmark_data_smithi159_102124_object53278:head# 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c49d23:::benchmark_data_smithi159_102124_object33255:head# 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.858 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c520c7:::benchmark_data_smithi159_102124_object31738:head# 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c54dd1:::benchmark_data_smithi159_102124_object57129:head# 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c6b935:::benchmark_data_smithi159_102124_object148:head# 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27c9d0d5:::benchmark_data_smithi159_102124_object10626:head# 2024-04-15T18:22:41.859 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ca3d31:::benchmark_data_smithi159_102124_object19413:head# 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ca83ee:::benchmark_data_smithi159_102124_object29832:head# 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27cad303:::benchmark_data_smithi159_102124_object45563:head# 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27cb45d0:::benchmark_data_smithi159_102124_object53208:head# 2024-04-15T18:22:41.860 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27cc8d7f:::benchmark_data_smithi159_102124_object2149:head# 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27cda5f3:::benchmark_data_smithi159_102124_object22207:head# 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27cff870:::benchmark_data_smithi159_102124_object57876:head# 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27d02aed:::benchmark_data_smithi159_102124_object32098:head# 2024-04-15T18:22:41.861 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27d305a7:::benchmark_data_smithi159_102124_object47616:head# 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27d60059:::benchmark_data_smithi159_102124_object7286:head# 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27d68337:::benchmark_data_smithi159_102124_object49415:head# 2024-04-15T18:22:41.862 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.863 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27d9e44f:::benchmark_data_smithi159_102124_object52599:head# 2024-04-15T18:22:41.863 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dae632:::benchmark_data_smithi159_102124_object8411:head# 2024-04-15T18:22:41.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.864 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dbfc63:::benchmark_data_smithi159_102124_object11010:head# 2024-04-15T18:22:41.864 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dc2ba1:::benchmark_data_smithi159_102124_object19664:head# 2024-04-15T18:22:41.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dc76d3:::benchmark_data_smithi159_102124_object1190:head# 2024-04-15T18:22:41.869 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.869 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dc9dc5:::benchmark_data_smithi159_102124_object2678:head# 2024-04-15T18:22:41.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27de7c14:::benchmark_data_smithi159_102124_object8789:head# 2024-04-15T18:22:41.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.870 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27de9289:::benchmark_data_smithi159_102124_object42090:head# 2024-04-15T18:22:41.870 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.872 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27dec9aa:::benchmark_data_smithi159_102124_object54030:head# 2024-04-15T18:22:41.872 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27df83f8:::benchmark_data_smithi159_102124_object43603:head# 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27df9c4e:::benchmark_data_smithi159_102124_object23581:head# 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e44fb3:::benchmark_data_smithi159_102124_object1270:head# 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e4accb:::benchmark_data_smithi159_102124_object35295:head# 2024-04-15T18:22:41.873 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e54f60:::benchmark_data_smithi159_102124_object5139:head# 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e59645:::benchmark_data_smithi159_102124_object13247:head# 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e5e168:::benchmark_data_smithi159_102124_object39316:head# 2024-04-15T18:22:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e5e7b5:::benchmark_data_smithi159_102124_object41851:head# 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e770a6:::benchmark_data_smithi159_102124_object34378:head# 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27e81695:::benchmark_data_smithi159_102124_object33423:head# 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ebbbad:::benchmark_data_smithi159_102124_object25307:head# 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ebebc0:::benchmark_data_smithi159_102124_object5069:head# 2024-04-15T18:22:41.875 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ec6c0e:::benchmark_data_smithi159_102124_object44369:head# 2024-04-15T18:22:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ece549:::benchmark_data_smithi159_102124_object15348:head# 2024-04-15T18:22:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27eee58a:::benchmark_data_smithi159_102124_object57007:head# 2024-04-15T18:22:41.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.878 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27f005ce:::benchmark_data_smithi159_102124_object40117:head# 2024-04-15T18:22:41.878 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27f2db8c:::benchmark_data_smithi159_102124_object11804:head# 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27f31783:::benchmark_data_smithi159_102124_object50827:head# 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27f915fd:::benchmark_data_smithi159_102124_object49046:head# 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27f958a5:::benchmark_data_smithi159_102124_object58085:head# 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fa7ab6:::benchmark_data_smithi159_102124_object14075:head# 2024-04-15T18:22:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fa9cee:::benchmark_data_smithi159_102124_object19821:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fabbee:::benchmark_data_smithi159_102124_object18367:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fcda11:::benchmark_data_smithi159_102124_object57642:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fd6e24:::benchmark_data_smithi159_102124_object17619:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27fe7963:::benchmark_data_smithi159_102124_object36863:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27febdca:::benchmark_data_smithi159_102124_object9998:head# 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.884 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:27ffd4a0:::benchmark_data_smithi159_102124_object46852:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2804f579:::benchmark_data_smithi159_102124_object44808:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28054bba:::benchmark_data_smithi159_102124_object48599:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2808219a:::benchmark_data_smithi159_102124_object47873:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:280c791b:::benchmark_data_smithi159_102124_object44461:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:280cdb01:::benchmark_data_smithi159_102124_object55640:head# 2024-04-15T18:22:41.885 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:280d6fac:::benchmark_data_smithi159_102124_object2765:head# 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:280ecbb6:::benchmark_data_smithi159_102124_object10265:head# 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2811b92f:::benchmark_data_smithi159_102124_object15032:head# 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2811fc02:::benchmark_data_smithi159_102124_object50024:head# 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2813b346:::benchmark_data_smithi159_102124_object39416:head# 2024-04-15T18:22:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28141a0b:::benchmark_data_smithi159_102124_object24083:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281443d3:::benchmark_data_smithi159_102124_object11363:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28146020:::benchmark_data_smithi159_102124_object33183:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281558e4:::benchmark_data_smithi159_102124_object2506:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28162f77:::benchmark_data_smithi159_102124_object29638:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2818ebb6:::benchmark_data_smithi159_102124_object38819:head# 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2819053d:::benchmark_data_smithi159_102124_object45845:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2819f403:::benchmark_data_smithi159_102124_object50094:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281a1221:::benchmark_data_smithi159_102124_object44716:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281a17f7:::benchmark_data_smithi159_102124_object47058:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281c7d37:::benchmark_data_smithi159_102124_object9761:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:281e511a:::benchmark_data_smithi159_102124_object39314:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28211b83:::benchmark_data_smithi159_102124_object10288:head# 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.891 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:282348d8:::benchmark_data_smithi159_102124_object33506:head# 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2827a7ef:::benchmark_data_smithi159_102124_object17573:head# 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:282a1525:::benchmark_data_smithi159_102124_object48883:head# 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:282be60b:::benchmark_data_smithi159_102124_object32186:head# 2024-04-15T18:22:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:282c23de:::benchmark_data_smithi159_102124_object57932:head# 2024-04-15T18:22:41.894 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.894 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:282fa01a:::benchmark_data_smithi159_102124_object15084:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28301f47:::benchmark_data_smithi159_102124_object30257:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2831bbc8:::benchmark_data_smithi159_102124_object13396:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28321232:::benchmark_data_smithi159_102124_object42585:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283342fe:::benchmark_data_smithi159_102124_object58083:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2834eaf2:::benchmark_data_smithi159_102124_object25213:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283623d4:::benchmark_data_smithi159_102124_object24430:head# 2024-04-15T18:22:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283694c4:::benchmark_data_smithi159_102124_object47396:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2837153f:::benchmark_data_smithi159_102124_object10310:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2838479c:::benchmark_data_smithi159_102124_object52738:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283a03e4:::benchmark_data_smithi159_102124_object17270:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283b891b:::benchmark_data_smithi159_102124_object36385:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283c596a:::benchmark_data_smithi159_102124_object57036:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283c9458:::benchmark_data_smithi159_102124_object49266:head# 2024-04-15T18:22:41.896 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:283de3e8:::benchmark_data_smithi159_102124_object55100:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2840a396:::benchmark_data_smithi159_102124_object3861:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2840d5b6:::benchmark_data_smithi159_102124_object13701:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2841859a:::benchmark_data_smithi159_102124_object37335:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2841dbeb:::benchmark_data_smithi159_102124_object46721:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284376d9:::benchmark_data_smithi159_102124_object3409:head# 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.897 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2843c2b0:::benchmark_data_smithi159_102124_object57542:head# 2024-04-15T18:22:41.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.898 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28461405:::benchmark_data_smithi159_102124_object33957:head# 2024-04-15T18:22:41.898 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.900 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284890db:::benchmark_data_smithi159_102124_object32296:head# 2024-04-15T18:22:41.900 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28489b12:::benchmark_data_smithi159_102124_object5647:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2849f21f:::benchmark_data_smithi159_102124_object33919:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284b7166:::benchmark_data_smithi159_102124_object33363:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284bf5e0:::benchmark_data_smithi159_102124_object47106:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284c3c7e:::benchmark_data_smithi159_102124_object46442:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284c62c3:::benchmark_data_smithi159_102124_object54528:head# 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:284cb472:::benchmark_data_smithi159_102124_object21509:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2853c927:::benchmark_data_smithi159_102124_object7657:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285624dd:::benchmark_data_smithi159_102124_object8342:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28564de4:::benchmark_data_smithi159_102124_object27661:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2856e5ae:::benchmark_data_smithi159_102124_object49856:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28588fb5:::benchmark_data_smithi159_102124_object6690:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2858cb7c:::benchmark_data_smithi159_102124_object43939:head# 2024-04-15T18:22:41.902 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2858d87e:::benchmark_data_smithi159_102124_object31587:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28597123:::benchmark_data_smithi159_102124_object20169:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285c33ab:::benchmark_data_smithi159_102124_object4658:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285c3c65:::benchmark_data_smithi159_102124_object1993:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285e3e08:::benchmark_data_smithi159_102124_object55258:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285e4c9c:::benchmark_data_smithi159_102124_object44885:head# 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.903 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:285fe546:::benchmark_data_smithi159_102124_object13138:head# 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28600a7f:::benchmark_data_smithi159_102124_object8749:head# 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2860a90c:::benchmark_data_smithi159_102124_object25410:head# 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2861114a:::benchmark_data_smithi159_102124_object58344:head# 2024-04-15T18:22:41.904 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.907 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2861634c:::benchmark_data_smithi159_102124_object5752:head# 2024-04-15T18:22:41.907 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2863786e:::benchmark_data_smithi159_102124_object12069:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2863a9f3:::benchmark_data_smithi159_102124_object4766:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2864985a:::benchmark_data_smithi159_102124_object11963:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28649c19:::benchmark_data_smithi159_102124_object40810:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2865217d:::benchmark_data_smithi159_102124_object57866:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28656e3b:::benchmark_data_smithi159_102124_object684:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2865fa83:::benchmark_data_smithi159_102124_object34653:head# 2024-04-15T18:22:41.908 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2866d997:::benchmark_data_smithi159_102124_object36295:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28698946:::benchmark_data_smithi159_102124_object15471:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2869e844:::benchmark_data_smithi159_102124_object6415:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:286a2b14:::benchmark_data_smithi159_102124_object20101:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:286da10c:::benchmark_data_smithi159_102124_object52229:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:286fa456:::benchmark_data_smithi159_102124_object53094:head# 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.909 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28706204:::benchmark_data_smithi159_102124_object42003:head# 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28716e86:::benchmark_data_smithi159_102124_object49296:head# 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28723a13:::benchmark_data_smithi159_102124_object36471:head# 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28732440:::benchmark_data_smithi159_102124_object36096:head# 2024-04-15T18:22:41.910 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28740818:::benchmark_data_smithi159_102124_object35817:head# 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2874ab16:::benchmark_data_smithi159_102124_object53409:head# 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2874d045:::benchmark_data_smithi159_102124_object15664:head# 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2876710d:::benchmark_data_smithi159_102124_object41480:head# 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.912 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2876d62a:::benchmark_data_smithi159_102124_object44137:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28774680:::benchmark_data_smithi159_102124_object11148:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2877720a:::benchmark_data_smithi159_102124_object33949:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28799aa4:::benchmark_data_smithi159_102124_object3543:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2879c63a:::benchmark_data_smithi159_102124_object33448:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:287b4309:::benchmark_data_smithi159_102124_object17929:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:287d0e01:::benchmark_data_smithi159_102124_object281:head# 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.913 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:287d8e9d:::benchmark_data_smithi159_102124_object19430:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:287e0dee:::benchmark_data_smithi159_102124_object13833:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28805fc6:::benchmark_data_smithi159_102124_object46041:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28811d21:::benchmark_data_smithi159_102124_object12651:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28817faf:::benchmark_data_smithi159_102124_object27422:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2881e2a4:::benchmark_data_smithi159_102124_object9793:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28835db1:::benchmark_data_smithi159_102124_object31379:head# 2024-04-15T18:22:41.914 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:288406e2:::benchmark_data_smithi159_102124_object30029:head# 2024-04-15T18:22:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2884bd70:::benchmark_data_smithi159_102124_object39559:head# 2024-04-15T18:22:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28856995:::benchmark_data_smithi159_102124_object22453:head# 2024-04-15T18:22:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28869b4f:::benchmark_data_smithi159_102124_object27909:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2886c5c4:::benchmark_data_smithi159_102124_object44640:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28872f8d:::benchmark_data_smithi159_102124_object41862:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28888d65:::benchmark_data_smithi159_102124_object51578:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2888a057:::benchmark_data_smithi159_102124_object40255:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:288a28d2:::benchmark_data_smithi159_102124_object56593:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:288a9797:::benchmark_data_smithi159_102124_object23845:head# 2024-04-15T18:22:41.917 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2892342e:::benchmark_data_smithi159_102124_object47551:head# 2024-04-15T18:22:41.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.918 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28923b6b:::benchmark_data_smithi159_102124_object3746:head# 2024-04-15T18:22:41.918 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28930c3f:::benchmark_data_smithi159_102124_object12678:head# 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28939a20:::benchmark_data_smithi159_102124_object27179:head# 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2893d59a:::benchmark_data_smithi159_102124_object22023:head# 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28940a69:::benchmark_data_smithi159_102124_object46357:head# 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289544c1:::benchmark_data_smithi159_102124_object13148:head# 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2896725c:::benchmark_data_smithi159_102124_object4088:head# 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2896a64a:::benchmark_data_smithi159_102124_object41270:head# 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2897722e:::benchmark_data_smithi159_102124_object27906:head# 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.925 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28982beb:::benchmark_data_smithi159_102124_object39089:head# 2024-04-15T18:22:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2898807d:::benchmark_data_smithi159_102124_object54860:head# 2024-04-15T18:22:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2898a71e:::benchmark_data_smithi159_102124_object8024:head# 2024-04-15T18:22:41.926 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2899f4ec:::benchmark_data_smithi159_102124_object7273:head# 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289b9891:::benchmark_data_smithi159_102124_object28776:head# 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289be15b:::benchmark_data_smithi159_102124_object40062:head# 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289ce0f7:::benchmark_data_smithi159_102124_object37139:head# 2024-04-15T18:22:41.927 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289e6ec6:::benchmark_data_smithi159_102124_object37321:head# 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289ec5b5:::benchmark_data_smithi159_102124_object3509:head# 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289f45c9:::benchmark_data_smithi159_102124_object54609:head# 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:289f54c2:::benchmark_data_smithi159_102124_object27484:head# 2024-04-15T18:22:41.928 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a1ef2b:::benchmark_data_smithi159_102124_object11412:head# 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a2dffc:::benchmark_data_smithi159_102124_object11954:head# 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a4108d:::benchmark_data_smithi159_102124_object54602:head# 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a4332a:::benchmark_data_smithi159_102124_object40476:head# 2024-04-15T18:22:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a43830:::benchmark_data_smithi159_102124_object4235:head# 2024-04-15T18:22:41.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.930 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a452a4:::benchmark_data_smithi159_102124_object46244:head# 2024-04-15T18:22:41.930 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a4aa21:::benchmark_data_smithi159_102124_object42209:head# 2024-04-15T18:22:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a732cc:::benchmark_data_smithi159_102124_object38670:head# 2024-04-15T18:22:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a7e558:::benchmark_data_smithi159_102124_object38002:head# 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a84679:::benchmark_data_smithi159_102124_object41175:head# 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a85bca:::benchmark_data_smithi159_102124_object21244:head# 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a8d544:::benchmark_data_smithi159_102124_object58247:head# 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28a9a587:::benchmark_data_smithi159_102124_object4684:head# 2024-04-15T18:22:41.932 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ad0a95:::benchmark_data_smithi159_102124_object3917:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28aef23c:::benchmark_data_smithi159_102124_object53994:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28af2e5a:::benchmark_data_smithi159_102124_object29012:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28af6264:::benchmark_data_smithi159_102124_object48524:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28af9b7f:::benchmark_data_smithi159_102124_object15115:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28afac13:::benchmark_data_smithi159_102124_object7746:head# 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.934 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28aff461:::benchmark_data_smithi159_102124_object1440:head# 2024-04-15T18:22:41.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.935 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b2338e:::benchmark_data_smithi159_102124_object17416:head# 2024-04-15T18:22:41.935 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b37fea:::benchmark_data_smithi159_102124_object52222:head# 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b51a43:::benchmark_data_smithi159_102124_object8425:head# 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b5a1a5:::benchmark_data_smithi159_102124_object11254:head# 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b6cb6c:::benchmark_data_smithi159_102124_object19830:head# 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28b964e6:::benchmark_data_smithi159_102124_object34757:head# 2024-04-15T18:22:41.937 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28bb8e07:::benchmark_data_smithi159_102124_object55162:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28bc0d76:::benchmark_data_smithi159_102124_object19803:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28bc80f6:::benchmark_data_smithi159_102124_object5441:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28bcf45a:::benchmark_data_smithi159_102124_object19552:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28bd6daa:::benchmark_data_smithi159_102124_object40475:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28be9b66:::benchmark_data_smithi159_102124_object9713:head# 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.938 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c1cbce:::benchmark_data_smithi159_102124_object53904:head# 2024-04-15T18:22:41.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.939 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c35a2d:::benchmark_data_smithi159_102124_object18299:head# 2024-04-15T18:22:41.939 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c44d4d:::benchmark_data_smithi159_102124_object1850:head# 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c60d89:::benchmark_data_smithi159_102124_object46911:head# 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c65422:::benchmark_data_smithi159_102124_object42219:head# 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c7a6ab:::benchmark_data_smithi159_102124_object6495:head# 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.941 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c7b479:::benchmark_data_smithi159_102124_object47927:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28c7b984:::benchmark_data_smithi159_102124_object26978:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28cb40fb:::benchmark_data_smithi159_102124_object27245:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28cc73a2:::benchmark_data_smithi159_102124_object53195:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28cde145:::benchmark_data_smithi159_102124_object24466:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28cf86d6:::benchmark_data_smithi159_102124_object32424:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d1957d:::benchmark_data_smithi159_102124_object10466:head# 2024-04-15T18:22:41.942 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d234af:::benchmark_data_smithi159_102124_object29367:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d3c058:::benchmark_data_smithi159_102124_object40039:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d40bbc:::benchmark_data_smithi159_102124_object18698:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d42c21:::benchmark_data_smithi159_102124_object771:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d42f98:::benchmark_data_smithi159_102124_object48948:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d51385:::benchmark_data_smithi159_102124_object12356:head# 2024-04-15T18:22:41.943 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.944 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d540a0:::benchmark_data_smithi159_102124_object38053:head# 2024-04-15T18:22:41.944 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d5a277:::benchmark_data_smithi159_102124_object7256:head# 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d6e71c:::benchmark_data_smithi159_102124_object48107:head# 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d7b436:::benchmark_data_smithi159_102124_object35532:head# 2024-04-15T18:22:41.946 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d819d1:::benchmark_data_smithi159_102124_object37666:head# 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28d8e5b5:::benchmark_data_smithi159_102124_object23712:head# 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28dab7d5:::benchmark_data_smithi159_102124_object16188:head# 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.947 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28dd2d60:::benchmark_data_smithi159_102124_object51623:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28dd4ff8:::benchmark_data_smithi159_102124_object8189:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28de1165:::benchmark_data_smithi159_102124_object10493:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28de6012:::benchmark_data_smithi159_102124_object41299:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28df9699:::benchmark_data_smithi159_102124_object52457:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e2210a:::benchmark_data_smithi159_102124_object9424:head# 2024-04-15T18:22:41.948 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e3428d:::benchmark_data_smithi159_102124_object12969:head# 2024-04-15T18:22:41.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.949 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e43dbf:::benchmark_data_smithi159_102124_object46156:head# 2024-04-15T18:22:41.949 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e4b3c3:::benchmark_data_smithi159_102124_object55137:head# 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e56421:::benchmark_data_smithi159_102124_object24303:head# 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e57d53:::benchmark_data_smithi159_102124_object23923:head# 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e5a754:::benchmark_data_smithi159_102124_object5246:head# 2024-04-15T18:22:41.950 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e5bfe7:::benchmark_data_smithi159_102124_object7702:head# 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e61b7f:::benchmark_data_smithi159_102124_object4610:head# 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e854b2:::benchmark_data_smithi159_102124_object27664:head# 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e86cf7:::benchmark_data_smithi159_102124_object37397:head# 2024-04-15T18:22:41.951 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e91d70:::benchmark_data_smithi159_102124_object34638:head# 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28e9e5fa:::benchmark_data_smithi159_102124_object43777:head# 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ea5d9d:::benchmark_data_smithi159_102124_object10734:head# 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ea7afc:::benchmark_data_smithi159_102124_object20797:head# 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ee287e:::benchmark_data_smithi159_102124_object49959:head# 2024-04-15T18:22:41.952 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28eedd4b:::benchmark_data_smithi159_102124_object42774:head# 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ef4b80:::benchmark_data_smithi159_102124_object48023:head# 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f157a6:::benchmark_data_smithi159_102124_object23436:head# 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f17920:::benchmark_data_smithi159_102124_object15487:head# 2024-04-15T18:22:41.959 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.960 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f2d23a:::benchmark_data_smithi159_102124_object55594:head# 2024-04-15T18:22:41.960 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.963 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f38320:::benchmark_data_smithi159_102124_object32217:head# 2024-04-15T18:22:41.963 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.967 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f3ee94:::benchmark_data_smithi159_102124_object5972:head# 2024-04-15T18:22:41.967 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.969 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f51469:::benchmark_data_smithi159_102124_object46095:head# 2024-04-15T18:22:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f577d5:::benchmark_data_smithi159_102124_object18161:head# 2024-04-15T18:22:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f5faa7:::benchmark_data_smithi159_102124_object5050:head# 2024-04-15T18:22:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28f646c0:::benchmark_data_smithi159_102124_object12426:head# 2024-04-15T18:22:41.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.971 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28fab7cd:::benchmark_data_smithi159_102124_object42024:head# 2024-04-15T18:22:41.971 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28fc9bcc:::benchmark_data_smithi159_102124_object48990:head# 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:28ffc6f1:::benchmark_data_smithi159_102124_object14130:head# 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2900d976:::benchmark_data_smithi159_102124_object7791:head# 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2901e907:::benchmark_data_smithi159_102124_object49874:head# 2024-04-15T18:22:41.976 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2903f2e0:::benchmark_data_smithi159_102124_object29085:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290608dd:::benchmark_data_smithi159_102124_object13693:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2906d5b1:::benchmark_data_smithi159_102124_object31244:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29088614:::benchmark_data_smithi159_102124_object45254:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290a50ee:::benchmark_data_smithi159_102124_object39465:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290a817d:::benchmark_data_smithi159_102124_object46108:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290e1965:::benchmark_data_smithi159_102124_object27600:head# 2024-04-15T18:22:41.977 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290e81ee:::benchmark_data_smithi159_102124_object27365:head# 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:290f624c:::benchmark_data_smithi159_102124_object38067:head# 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29112543:::benchmark_data_smithi159_102124_object39951:head# 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2911ebeb:::benchmark_data_smithi159_102124_object22291:head# 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291269d8:::benchmark_data_smithi159_102124_object39219:head# 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.978 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291473a5:::benchmark_data_smithi159_102124_object12042:head# 2024-04-15T18:22:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29150851:::benchmark_data_smithi159_102124_object26898:head# 2024-04-15T18:22:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291509ea:::benchmark_data_smithi159_102124_object6807:head# 2024-04-15T18:22:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.982 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291545cb:::benchmark_data_smithi159_102124_object41129:head# 2024-04-15T18:22:41.982 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29170503:::benchmark_data_smithi159_102124_object12971:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29176131:::benchmark_data_smithi159_102124_object31751:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2917ecc9:::benchmark_data_smithi159_102124_object3572:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2918aa87:::benchmark_data_smithi159_102124_object18211:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29192076:::benchmark_data_smithi159_102124_object16834:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291950f9:::benchmark_data_smithi159_102124_object51476:head# 2024-04-15T18:22:41.983 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291a2f72:::benchmark_data_smithi159_102124_object3564:head# 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:291da2af:::benchmark_data_smithi159_102124_object25492:head# 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29213861:::benchmark_data_smithi159_102124_object11681:head# 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.984 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2921dd55:::benchmark_data_smithi159_102124_object43103:head# 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2921ee4f:::benchmark_data_smithi159_102124_object36163:head# 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2922921a:::benchmark_data_smithi159_102124_object2339:head# 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2922b9d8:::benchmark_data_smithi159_102124_object45307:head# 2024-04-15T18:22:41.985 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29242d6e:::benchmark_data_smithi159_102124_object8582:head# 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2924841b:::benchmark_data_smithi159_102124_object3273:head# 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2924d3fb:::benchmark_data_smithi159_102124_object57924:head# 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29258667:::benchmark_data_smithi159_102124_object52502:head# 2024-04-15T18:22:41.986 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292773b4:::benchmark_data_smithi159_102124_object58148:head# 2024-04-15T18:22:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29292d23:::benchmark_data_smithi159_102124_object40482:head# 2024-04-15T18:22:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.987 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292b2645:::benchmark_data_smithi159_102124_object54240:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292b480b:::benchmark_data_smithi159_102124_object15149:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292c7ae1:::benchmark_data_smithi159_102124_object51927:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292e0da3:::benchmark_data_smithi159_102124_object25231:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:292f1c8c:::benchmark_data_smithi159_102124_object47143:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29301c2b:::benchmark_data_smithi159_102124_object714:head# 2024-04-15T18:22:41.988 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:293092f3:::benchmark_data_smithi159_102124_object2961:head# 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29324b67:::benchmark_data_smithi159_102124_object53019:head# 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2934af95:::benchmark_data_smithi159_102124_object6833:head# 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2934e750:::benchmark_data_smithi159_102124_object58362:head# 2024-04-15T18:22:41.989 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.990 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29350b09:::benchmark_data_smithi159_102124_object6502:head# 2024-04-15T18:22:41.990 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29378f71:::benchmark_data_smithi159_102124_object10329:head# 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2937c043:::benchmark_data_smithi159_102124_object37101:head# 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29389a63:::benchmark_data_smithi159_102124_object8479:head# 2024-04-15T18:22:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:293c476c:::benchmark_data_smithi159_102124_object3764:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:293d3bcf:::benchmark_data_smithi159_102124_object44706:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:293e9fdb:::benchmark_data_smithi159_102124_object33512:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:293eea1d:::benchmark_data_smithi159_102124_object8660:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29410e8b:::benchmark_data_smithi159_102124_object26788:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29414ceb:::benchmark_data_smithi159_102124_object38003:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29429887:::benchmark_data_smithi159_102124_object48905:head# 2024-04-15T18:22:41.992 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2943d102:::benchmark_data_smithi159_102124_object49489:head# 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29450678:::benchmark_data_smithi159_102124_object24217:head# 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294716e6:::benchmark_data_smithi159_102124_object3442:head# 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29485bcc:::benchmark_data_smithi159_102124_object25253:head# 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2948baad:::benchmark_data_smithi159_102124_object54517:head# 2024-04-15T18:22:41.993 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294978c8:::benchmark_data_smithi159_102124_object32588:head# 2024-04-15T18:22:41.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294b3ba4:::benchmark_data_smithi159_102124_object24097:head# 2024-04-15T18:22:41.995 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.995 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294b734c:::benchmark_data_smithi159_102124_object7995:head# 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294d4844:::benchmark_data_smithi159_102124_object24961:head# 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294de263:::benchmark_data_smithi159_102124_object54918:head# 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294fa931:::benchmark_data_smithi159_102124_object13548:head# 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:294fff30:::benchmark_data_smithi159_102124_object19594:head# 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.996 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29511bae:::benchmark_data_smithi159_102124_object41215:head# 2024-04-15T18:22:41.997 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2953c256:::benchmark_data_smithi159_102124_object31508:head# 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2953ce5b:::benchmark_data_smithi159_102124_object52726:head# 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29550f78:::benchmark_data_smithi159_102124_object1314:head# 2024-04-15T18:22:41.998 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2956edd6:::benchmark_data_smithi159_102124_object30474:head# 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:295adb9b:::benchmark_data_smithi159_102124_object19768:head# 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:295ba06d:::benchmark_data_smithi159_102124_object50320:head# 2024-04-15T18:22:41.999 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:295d087e:::benchmark_data_smithi159_102124_object54737:head# 2024-04-15T18:22:42.000 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.000 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:295fdb5b:::benchmark_data_smithi159_102124_object55998:head# 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29609070:::benchmark_data_smithi159_102124_object5248:head# 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29618a54:::benchmark_data_smithi159_102124_object10866:head# 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2962b9fa:::benchmark_data_smithi159_102124_object56998:head# 2024-04-15T18:22:42.001 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296645c8:::benchmark_data_smithi159_102124_object48865:head# 2024-04-15T18:22:42.002 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.002 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2967d8fc:::benchmark_data_smithi159_102124_object13445:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29694337:::benchmark_data_smithi159_102124_object21746:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296ad758:::benchmark_data_smithi159_102124_object18999:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296b6fc4:::benchmark_data_smithi159_102124_object5603:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296baf39:::benchmark_data_smithi159_102124_object4871:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296c24fc:::benchmark_data_smithi159_102124_object32598:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296ccd25:::benchmark_data_smithi159_102124_object48437:head# 2024-04-15T18:22:42.003 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296e5abc:::benchmark_data_smithi159_102124_object22992:head# 2024-04-15T18:22:42.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.004 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:296e83e5:::benchmark_data_smithi159_102124_object18157:head# 2024-04-15T18:22:42.004 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.006 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29713a28:::benchmark_data_smithi159_102124_object9831:head# 2024-04-15T18:22:42.007 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29717281:::benchmark_data_smithi159_102124_object51628:head# 2024-04-15T18:22:42.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.008 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2971be78:::benchmark_data_smithi159_102124_object42850:head# 2024-04-15T18:22:42.008 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.011 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2972d60e:::benchmark_data_smithi159_102124_object36403:head# 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29730e7d:::benchmark_data_smithi159_102124_object19434:head# 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29748bdd:::benchmark_data_smithi159_102124_object12774:head# 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29757ece:::benchmark_data_smithi159_102124_object19469:head# 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2975ba14:::benchmark_data_smithi159_102124_object46061:head# 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.012 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2976add4:::benchmark_data_smithi159_102124_object12100:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297884b9:::benchmark_data_smithi159_102124_object41339:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2979a442:::benchmark_data_smithi159_102124_object13174:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297a6227:::benchmark_data_smithi159_102124_object30411:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297b99dd:::benchmark_data_smithi159_102124_object7425:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297bfa37:::benchmark_data_smithi159_102124_object1101:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297c04cd:::benchmark_data_smithi159_102124_object3727:head# 2024-04-15T18:22:42.013 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297c4a1e:::benchmark_data_smithi159_102124_object55373:head# 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:297d7374:::benchmark_data_smithi159_102124_object4463:head# 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2980289e:::benchmark_data_smithi159_102124_object56797:head# 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2981c10b:::benchmark_data_smithi159_102124_object219:head# 2024-04-15T18:22:42.015 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298290db:::benchmark_data_smithi159_102124_object35079:head# 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2982cbee:::benchmark_data_smithi159_102124_object26913:head# 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298556bf:::benchmark_data_smithi159_102124_object28563:head# 2024-04-15T18:22:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2986c0c4:::benchmark_data_smithi159_102124_object13747:head# 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298731c1:::benchmark_data_smithi159_102124_object9940:head# 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298824e8:::benchmark_data_smithi159_102124_object8050:head# 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.017 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29882f80:::benchmark_data_smithi159_102124_object49454:head# 2024-04-15T18:22:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2988321a:::benchmark_data_smithi159_102124_object2507:head# 2024-04-15T18:22:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298b7cb9:::benchmark_data_smithi159_102124_object34196:head# 2024-04-15T18:22:42.018 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298c57cf:::benchmark_data_smithi159_102124_object15368:head# 2024-04-15T18:22:42.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.020 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298cb614:::benchmark_data_smithi159_102124_object52951:head# 2024-04-15T18:22:42.020 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298e2f3c:::benchmark_data_smithi159_102124_object40961:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:298f66be:::benchmark_data_smithi159_102124_object40407:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2990e56c:::benchmark_data_smithi159_102124_object23195:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2992abb4:::benchmark_data_smithi159_102124_object41146:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2992f8f9:::benchmark_data_smithi159_102124_object26301:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:299734aa:::benchmark_data_smithi159_102124_object11130:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29973a9a:::benchmark_data_smithi159_102124_object8619:head# 2024-04-15T18:22:42.021 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29994a6c:::benchmark_data_smithi159_102124_object6288:head# 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2999bfdd:::benchmark_data_smithi159_102124_object28644:head# 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2999d446:::benchmark_data_smithi159_102124_object37324:head# 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:299aaa4d:::benchmark_data_smithi159_102124_object11646:head# 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:299bbe24:::benchmark_data_smithi159_102124_object6381:head# 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.022 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:299d5207:::benchmark_data_smithi159_102124_object56192:head# 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:299eae23:::benchmark_data_smithi159_102124_object26891:head# 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a004ed:::benchmark_data_smithi159_102124_object21659:head# 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a1030b:::benchmark_data_smithi159_102124_object17562:head# 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a1f4a2:::benchmark_data_smithi159_102124_object47487:head# 2024-04-15T18:22:42.023 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a27731:::benchmark_data_smithi159_102124_object14793:head# 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a321f4:::benchmark_data_smithi159_102124_object36448:head# 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a3f001:::benchmark_data_smithi159_102124_object39042:head# 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a4cf58:::benchmark_data_smithi159_102124_object35661:head# 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.026 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a571fb:::benchmark_data_smithi159_102124_object52446:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a58c91:::benchmark_data_smithi159_102124_object33034:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a85d06:::benchmark_data_smithi159_102124_object26202:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a8b9e6:::benchmark_data_smithi159_102124_object19793:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29a99f58:::benchmark_data_smithi159_102124_object26268:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ae4fb1:::benchmark_data_smithi159_102124_object52711:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b35ae7:::benchmark_data_smithi159_102124_object57837:head# 2024-04-15T18:22:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b377f7:::benchmark_data_smithi159_102124_object55315:head# 2024-04-15T18:22:42.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b3f47d:::benchmark_data_smithi159_102124_object48943:head# 2024-04-15T18:22:42.028 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.028 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b45e6f:::benchmark_data_smithi159_102124_object47481:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b47a4b:::benchmark_data_smithi159_102124_object53346:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b49bd2:::benchmark_data_smithi159_102124_object36278:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b4c73d:::benchmark_data_smithi159_102124_object3423:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b52756:::benchmark_data_smithi159_102124_object55535:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b588cd:::benchmark_data_smithi159_102124_object10420:head# 2024-04-15T18:22:42.029 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b5e172:::benchmark_data_smithi159_102124_object27040:head# 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b627bd:::benchmark_data_smithi159_102124_object25273:head# 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29b90d16:::benchmark_data_smithi159_102124_object15156:head# 2024-04-15T18:22:42.035 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.036 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ba7521:::benchmark_data_smithi159_102124_object21201:head# 2024-04-15T18:22:42.036 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.038 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bc11ed:::benchmark_data_smithi159_102124_object52874:head# 2024-04-15T18:22:42.038 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bd401c:::benchmark_data_smithi159_102124_object27398:head# 2024-04-15T18:22:42.040 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.040 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bd40f8:::benchmark_data_smithi159_102124_object52114:head# 2024-04-15T18:22:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bd58a3:::benchmark_data_smithi159_102124_object123:head# 2024-04-15T18:22:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bd72ab:::benchmark_data_smithi159_102124_object45320:head# 2024-04-15T18:22:42.041 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bdadf6:::benchmark_data_smithi159_102124_object17725:head# 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29bef1d3:::benchmark_data_smithi159_102124_object45343:head# 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c04b7f:::benchmark_data_smithi159_102124_object21842:head# 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.042 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c0aece:::benchmark_data_smithi159_102124_object35262:head# 2024-04-15T18:22:42.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c13e24:::benchmark_data_smithi159_102124_object25689:head# 2024-04-15T18:22:42.043 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.043 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c31b4f:::benchmark_data_smithi159_102124_object40435:head# 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c3d5b5:::benchmark_data_smithi159_102124_object10126:head# 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c4f012:::benchmark_data_smithi159_102124_object12172:head# 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c5ab5c:::benchmark_data_smithi159_102124_object47640:head# 2024-04-15T18:22:42.044 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c6e229:::benchmark_data_smithi159_102124_object57812:head# 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29c9243a:::benchmark_data_smithi159_102124_object11666:head# 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ca76a5:::benchmark_data_smithi159_102124_object16489:head# 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29cdc663:::benchmark_data_smithi159_102124_object35861:head# 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29cdece9:::benchmark_data_smithi159_102124_object24993:head# 2024-04-15T18:22:42.045 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ce0471:::benchmark_data_smithi159_102124_object50255:head# 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ce441b:::benchmark_data_smithi159_102124_object50675:head# 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29cefe4c:::benchmark_data_smithi159_102124_object27806:head# 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.047 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29cf2ff8:::benchmark_data_smithi159_102124_object4727:head# 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29cfbbb6:::benchmark_data_smithi159_102124_object20960:head# 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29d00218:::benchmark_data_smithi159_102124_object9190:head# 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29d318c9:::benchmark_data_smithi159_102124_object30498:head# 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29d4919a:::benchmark_data_smithi159_102124_object732:head# 2024-04-15T18:22:42.048 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29d70c1e:::benchmark_data_smithi159_102124_object25155:head# 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29d7ee73:::benchmark_data_smithi159_102124_object37315:head# 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29da06f4:::benchmark_data_smithi159_102124_object34651:head# 2024-04-15T18:22:42.049 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29dda44c:::benchmark_data_smithi159_102124_object14879:head# 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ddc04f:::benchmark_data_smithi159_102124_object6950:head# 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29de9d12:::benchmark_data_smithi159_102124_object20038:head# 2024-04-15T18:22:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29deb49d:::benchmark_data_smithi159_102124_object14017:head# 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29defe21:::benchmark_data_smithi159_102124_object35641:head# 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29df806a:::benchmark_data_smithi159_102124_object54120:head# 2024-04-15T18:22:42.053 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e1ccc1:::benchmark_data_smithi159_102124_object745:head# 2024-04-15T18:22:42.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e1eb78:::benchmark_data_smithi159_102124_object53642:head# 2024-04-15T18:22:42.054 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.054 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e25d2a:::benchmark_data_smithi159_102124_object14584:head# 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e55a45:::benchmark_data_smithi159_102124_object20218:head# 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e684d4:::benchmark_data_smithi159_102124_object7210:head# 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.055 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29e9f9e4:::benchmark_data_smithi159_102124_object29230:head# 2024-04-15T18:22:42.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.056 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ea7f39:::benchmark_data_smithi159_102124_object47736:head# 2024-04-15T18:22:42.056 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ecfb77:::benchmark_data_smithi159_102124_object38629:head# 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ed4db8:::benchmark_data_smithi159_102124_object50103:head# 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29eed757:::benchmark_data_smithi159_102124_object57541:head# 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f221cf:::benchmark_data_smithi159_102124_object19640:head# 2024-04-15T18:22:42.058 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f52161:::benchmark_data_smithi159_102124_object25746:head# 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f5c898:::benchmark_data_smithi159_102124_object52412:head# 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f5d320:::benchmark_data_smithi159_102124_object240:head# 2024-04-15T18:22:42.059 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f88fc3:::benchmark_data_smithi159_102124_object51391:head# 2024-04-15T18:22:42.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.060 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f8ad9b:::benchmark_data_smithi159_102124_object55803:head# 2024-04-15T18:22:42.060 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29f93092:::benchmark_data_smithi159_102124_object38957:head# 2024-04-15T18:22:42.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.070 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29fa58fd:::benchmark_data_smithi159_102124_object31053:head# 2024-04-15T18:22:42.070 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29fa763b:::benchmark_data_smithi159_102124_object16853:head# 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29fab3e6:::benchmark_data_smithi159_102124_object22584:head# 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29fd2404:::benchmark_data_smithi159_102124_object37674:head# 2024-04-15T18:22:42.071 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:29ff967f:::benchmark_data_smithi159_102124_object36541:head# 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a011332:::benchmark_data_smithi159_102124_object39341:head# 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a02ac1f:::benchmark_data_smithi159_102124_object52772:head# 2024-04-15T18:22:42.072 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0349d5:::benchmark_data_smithi159_102124_object36896:head# 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0383bf:::benchmark_data_smithi159_102124_object7879:head# 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0386b8:::benchmark_data_smithi159_102124_object7852:head# 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.073 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a041957:::benchmark_data_smithi159_102124_object5597:head# 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a04bdce:::benchmark_data_smithi159_102124_object36387:head# 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a055f4b:::benchmark_data_smithi159_102124_object8530:head# 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.074 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0cbbce:::benchmark_data_smithi159_102124_object50901:head# 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0cfd83:::benchmark_data_smithi159_102124_object29091:head# 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a0fc756:::benchmark_data_smithi159_102124_object40490:head# 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.075 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1040d5:::benchmark_data_smithi159_102124_object48208:head# 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a114e3f:::benchmark_data_smithi159_102124_object14104:head# 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a12fcea:::benchmark_data_smithi159_102124_object13061:head# 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1302a1:::benchmark_data_smithi159_102124_object11479:head# 2024-04-15T18:22:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1431ee:::benchmark_data_smithi159_102124_object26772:head# 2024-04-15T18:22:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a149915:::benchmark_data_smithi159_102124_object48652:head# 2024-04-15T18:22:42.077 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a14e08f:::benchmark_data_smithi159_102124_object19119:head# 2024-04-15T18:22:42.078 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.078 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a15a3c9:::benchmark_data_smithi159_102124_object15788:head# 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a163f70:::benchmark_data_smithi159_102124_object39637:head# 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a16d09b:::benchmark_data_smithi159_102124_object1903:head# 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.079 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1711f9:::benchmark_data_smithi159_102124_object28307:head# 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a17acd4:::benchmark_data_smithi159_102124_object36116:head# 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a19abbf:::benchmark_data_smithi159_102124_object25073:head# 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.080 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1a67c5:::benchmark_data_smithi159_102124_object13257:head# 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1ac8b0:::benchmark_data_smithi159_102124_object29675:head# 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1ae91a:::benchmark_data_smithi159_102124_object43587:head# 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.081 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1b3614:::benchmark_data_smithi159_102124_object18406:head# 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1c355d:::benchmark_data_smithi159_102124_object49598:head# 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1c6595:::benchmark_data_smithi159_102124_object57110:head# 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.082 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1ddaf3:::benchmark_data_smithi159_102124_object37839:head# 2024-04-15T18:22:42.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.083 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1e32ec:::benchmark_data_smithi159_102124_object39277:head# 2024-04-15T18:22:42.083 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:22:42.091 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a1f4124:::benchmark_data_smithi159_102124_object17758:head# 2024-04-15T18:22:42.091 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a2142c3:::benchmark_data_smithi159_102124_object49631:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a223801:::benchmark_data_smithi159_102124_object51599:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a243356:::benchmark_data_smithi159_102124_object13346:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a24dde6:::benchmark_data_smithi159_102124_object52671:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a258f49:::benchmark_data_smithi159_102124_object14085:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a259f24:::benchmark_data_smithi159_102124_object15972:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a26e0a1:::benchmark_data_smithi159_102124_object13429:head# 2024-04-15T18:22:42.092 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a27bc86:::benchmark_data_smithi159_102124_object5644:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a2c47b0:::benchmark_data_smithi159_102124_object34046:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a2cedf6:::benchmark_data_smithi159_102124_object382:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a2dbf57:::benchmark_data_smithi159_102124_object39904:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a318388:::benchmark_data_smithi159_102124_object32214:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a32d443:::benchmark_data_smithi159_102124_object14738:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a34a73b:::benchmark_data_smithi159_102124_object25896:head# 2024-04-15T18:22:42.093 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a35bcff:::benchmark_data_smithi159_102124_object7315:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a35e879:::benchmark_data_smithi159_102124_object46150:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a369a55:::benchmark_data_smithi159_102124_object32461:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a39acb9:::benchmark_data_smithi159_102124_object14295:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a3a0818:::benchmark_data_smithi159_102124_object15294:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a3ad63a:::benchmark_data_smithi159_102124_object28582:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a3afaae:::benchmark_data_smithi159_102124_object29964:head# 2024-04-15T18:22:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a3e0ea6:::benchmark_data_smithi159_102124_object48224:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a3f61e2:::benchmark_data_smithi159_102124_object6676:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a40c377:::benchmark_data_smithi159_102124_object55885:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a42c9b6:::benchmark_data_smithi159_102124_object42883:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a45d502:::benchmark_data_smithi159_102124_object33421:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a461f0d:::benchmark_data_smithi159_102124_object32944:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a462c58:::benchmark_data_smithi159_102124_object27269:head# 2024-04-15T18:22:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a46cda8:::benchmark_data_smithi159_102124_object218:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4708ba:::benchmark_data_smithi159_102124_object15247:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4a6f6e:::benchmark_data_smithi159_102124_object1334:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4b9937:::benchmark_data_smithi159_102124_object32737:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4ba8c9:::benchmark_data_smithi159_102124_object15758:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4be079:::benchmark_data_smithi159_102124_object40309:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4c058d:::benchmark_data_smithi159_102124_object17367:head# 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.096 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4d1aa6:::benchmark_data_smithi159_102124_object41423:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a4fba35:::benchmark_data_smithi159_102124_object1753:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5042ae:::benchmark_data_smithi159_102124_object36940:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5057b0:::benchmark_data_smithi159_102124_object11830:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5058bd:::benchmark_data_smithi159_102124_object25032:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a520868:::benchmark_data_smithi159_102124_object56253:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a53b99e:::benchmark_data_smithi159_102124_object40582:head# 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.097 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a54bef3:::benchmark_data_smithi159_102124_object878:head# 2024-04-15T18:22:42.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a564b36:::benchmark_data_smithi159_102124_object35311:head# 2024-04-15T18:22:42.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.098 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5673a4:::benchmark_data_smithi159_102124_object36056:head# 2024-04-15T18:22:42.098 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.102 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a578652:::benchmark_data_smithi159_102124_object54423:head# 2024-04-15T18:22:42.102 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a58d642:::benchmark_data_smithi159_102124_object25190:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5940aa:::benchmark_data_smithi159_102124_object36809:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a595d85:::benchmark_data_smithi159_102124_object52729:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a596a29:::benchmark_data_smithi159_102124_object2049:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5a0740:::benchmark_data_smithi159_102124_object39388:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5a1891:::benchmark_data_smithi159_102124_object55992:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5ae8dc:::benchmark_data_smithi159_102124_object48691:head# 2024-04-15T18:22:42.103 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5c32db:::benchmark_data_smithi159_102124_object17589:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5f2472:::benchmark_data_smithi159_102124_object29523:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5f9606:::benchmark_data_smithi159_102124_object54314:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a5fd56a:::benchmark_data_smithi159_102124_object11311:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a614e7e:::benchmark_data_smithi159_102124_object50548:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a615edf:::benchmark_data_smithi159_102124_object19555:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a618ee2:::benchmark_data_smithi159_102124_object42685:head# 2024-04-15T18:22:42.104 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a61bc56:::benchmark_data_smithi159_102124_object24990:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6426db:::benchmark_data_smithi159_102124_object55773:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a670722:::benchmark_data_smithi159_102124_object33457:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6efd1d:::benchmark_data_smithi159_102124_object9259:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6f4906:::benchmark_data_smithi159_102124_object2891:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6f9ee1:::benchmark_data_smithi159_102124_object31901:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6fa41d:::benchmark_data_smithi159_102124_object47298:head# 2024-04-15T18:22:42.105 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a6fdeab:::benchmark_data_smithi159_102124_object49499:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a70b0a4:::benchmark_data_smithi159_102124_object52908:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a714cdc:::benchmark_data_smithi159_102124_object57443:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a71f6a5:::benchmark_data_smithi159_102124_object30313:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a75b475:::benchmark_data_smithi159_102124_object5223:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a764c83:::benchmark_data_smithi159_102124_object32552:head# 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.106 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a76cb9b:::benchmark_data_smithi159_102124_object12557:head# 2024-04-15T18:22:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a780740:::benchmark_data_smithi159_102124_object37738:head# 2024-04-15T18:22:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a780d3e:::benchmark_data_smithi159_102124_object51664:head# 2024-04-15T18:22:42.107 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a79cfa2:::benchmark_data_smithi159_102124_object23276:head# 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a7cadc4:::benchmark_data_smithi159_102124_object20931:head# 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a7d4f5d:::benchmark_data_smithi159_102124_object41079:head# 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a7dd419:::benchmark_data_smithi159_102124_object26549:head# 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a7eaa90:::benchmark_data_smithi159_102124_object11350:head# 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.114 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8050e6:::benchmark_data_smithi159_102124_object47565:head# 2024-04-15T18:22:42.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a82e7ce:::benchmark_data_smithi159_102124_object5412:head# 2024-04-15T18:22:42.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.115 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8582cb:::benchmark_data_smithi159_102124_object12078:head# 2024-04-15T18:22:42.115 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a874abb:::benchmark_data_smithi159_102124_object9340:head# 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8a9352:::benchmark_data_smithi159_102124_object20914:head# 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8b857d:::benchmark_data_smithi159_102124_object36671:head# 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8d2347:::benchmark_data_smithi159_102124_object18444:head# 2024-04-15T18:22:42.116 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.117 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8f27a8:::benchmark_data_smithi159_102124_object13513:head# 2024-04-15T18:22:42.117 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a8f7476:::benchmark_data_smithi159_102124_object32224:head# 2024-04-15T18:22:42.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.118 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a91ce0a:::benchmark_data_smithi159_102124_object9859:head# 2024-04-15T18:22:42.118 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a91e010:::benchmark_data_smithi159_102124_object33704:head# 2024-04-15T18:22:42.119 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.119 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a94ee45:::benchmark_data_smithi159_102124_object37061:head# 2024-04-15T18:22:42.120 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.122 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a971cac:::benchmark_data_smithi159_102124_object49470:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2a9b0702:::benchmark_data_smithi159_102124_object53541:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa2d9b5:::benchmark_data_smithi159_102124_object40233:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa3a278:::benchmark_data_smithi159_102124_object6941:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa40d40:::benchmark_data_smithi159_102124_object43369:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa4dcd2:::benchmark_data_smithi159_102124_object4995:head# 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.123 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa5af50:::benchmark_data_smithi159_102124_object14140:head# 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa61948:::benchmark_data_smithi159_102124_object55555:head# 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa7c1bb:::benchmark_data_smithi159_102124_object9216:head# 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aa98635:::benchmark_data_smithi159_102124_object32558:head# 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aacdbeb:::benchmark_data_smithi159_102124_object34502:head# 2024-04-15T18:22:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aad8df4:::benchmark_data_smithi159_102124_object49301:head# 2024-04-15T18:22:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aae93de:::benchmark_data_smithi159_102124_object50869:head# 2024-04-15T18:22:42.125 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aaf1faa:::benchmark_data_smithi159_102124_object36345:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab192db:::benchmark_data_smithi159_102124_object24781:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab31a93:::benchmark_data_smithi159_102124_object2098:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab394ef:::benchmark_data_smithi159_102124_object34831:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab41b1b:::benchmark_data_smithi159_102124_object34407:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab4dcee:::benchmark_data_smithi159_102124_object34627:head# 2024-04-15T18:22:42.126 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab5a92f:::benchmark_data_smithi159_102124_object55457:head# 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab80dab:::benchmark_data_smithi159_102124_object53140:head# 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ab85767:::benchmark_data_smithi159_102124_object10900:head# 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2abaed09:::benchmark_data_smithi159_102124_object1261:head# 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2abd88eb:::benchmark_data_smithi159_102124_object13968:head# 2024-04-15T18:22:42.128 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2abf4e7f:::benchmark_data_smithi159_102124_object34887:head# 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ac1901a:::benchmark_data_smithi159_102124_object22766:head# 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ac1a7ab:::benchmark_data_smithi159_102124_object21959:head# 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ac22792:::benchmark_data_smithi159_102124_object34460:head# 2024-04-15T18:22:42.129 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.130 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ac4d593:::benchmark_data_smithi159_102124_object45136:head# 2024-04-15T18:22:42.130 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.130 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aca46ec:::benchmark_data_smithi159_102124_object46495:head# 2024-04-15T18:22:42.131 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.131 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2acdbaea:::benchmark_data_smithi159_102124_object49725:head# 2024-04-15T18:22:42.131 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2acf4aca:::benchmark_data_smithi159_102124_object27054:head# 2024-04-15T18:22:42.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.132 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad00546:::benchmark_data_smithi159_102124_object24783:head# 2024-04-15T18:22:42.132 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad0b16c:::benchmark_data_smithi159_102124_object26928:head# 2024-04-15T18:22:42.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.133 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad34f04:::benchmark_data_smithi159_102124_object18276:head# 2024-04-15T18:22:42.133 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad54a20:::benchmark_data_smithi159_102124_object13756:head# 2024-04-15T18:22:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad6d616:::benchmark_data_smithi159_102124_object17863:head# 2024-04-15T18:22:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad8d686:::benchmark_data_smithi159_102124_object15885:head# 2024-04-15T18:22:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad8d830:::benchmark_data_smithi159_102124_object5499:head# 2024-04-15T18:22:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ad921ec:::benchmark_data_smithi159_102124_object45444:head# 2024-04-15T18:22:42.150 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2adade22:::benchmark_data_smithi159_102124_object11842:head# 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2adb41f5:::benchmark_data_smithi159_102124_object43263:head# 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2adb6e61:::benchmark_data_smithi159_102124_object54029:head# 2024-04-15T18:22:42.151 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2adc43de:::benchmark_data_smithi159_102124_object40069:head# 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2add7d37:::benchmark_data_smithi159_102124_object53371:head# 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ade7616:::benchmark_data_smithi159_102124_object47242:head# 2024-04-15T18:22:42.152 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2adfc4ca:::benchmark_data_smithi159_102124_object50700:head# 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae1f355:::benchmark_data_smithi159_102124_object41094:head# 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae23e6a:::benchmark_data_smithi159_102124_object18261:head# 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae30fc3:::benchmark_data_smithi159_102124_object52050:head# 2024-04-15T18:22:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae5c39d:::benchmark_data_smithi159_102124_object51634:head# 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae741be:::benchmark_data_smithi159_102124_object35551:head# 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ae947a5:::benchmark_data_smithi159_102124_object8377:head# 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aea876c:::benchmark_data_smithi159_102124_object40816:head# 2024-04-15T18:22:42.154 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aed2c26:::benchmark_data_smithi159_102124_object4778:head# 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aeef69a:::benchmark_data_smithi159_102124_object42288:head# 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2aef1513:::benchmark_data_smithi159_102124_object15220:head# 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af03667:::benchmark_data_smithi159_102124_object13571:head# 2024-04-15T18:22:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af26222:::benchmark_data_smithi159_102124_object26324:head# 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af2fda8:::benchmark_data_smithi159_102124_object47910:head# 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af39c44:::benchmark_data_smithi159_102124_object44571:head# 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.156 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af40b00:::benchmark_data_smithi159_102124_object25917:head# 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af41ddf:::benchmark_data_smithi159_102124_object57327:head# 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af461d8:::benchmark_data_smithi159_102124_object30623:head# 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.157 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af85a53:::benchmark_data_smithi159_102124_object3157:head# 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af96443:::benchmark_data_smithi159_102124_object23505:head# 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af9774e:::benchmark_data_smithi159_102124_object9022:head# 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.158 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af9ba50:::benchmark_data_smithi159_102124_object11730:head# 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2af9d04e:::benchmark_data_smithi159_102124_object24133:head# 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2afa2104:::benchmark_data_smithi159_102124_object17782:head# 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.159 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2afbbda8:::benchmark_data_smithi159_102124_object9575:head# 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2afc78d8:::benchmark_data_smithi159_102124_object47856:head# 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2afec6d0:::benchmark_data_smithi159_102124_object32415:head# 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b01d215:::benchmark_data_smithi159_102124_object44318:head# 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0293b8:::benchmark_data_smithi159_102124_object30693:head# 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.160 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b077384:::benchmark_data_smithi159_102124_object17972:head# 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b07c639:::benchmark_data_smithi159_102124_object18043:head# 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b08bcb6:::benchmark_data_smithi159_102124_object42988:head# 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b09c394:::benchmark_data_smithi159_102124_object52410:head# 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0b3b4f:::benchmark_data_smithi159_102124_object18735:head# 2024-04-15T18:22:42.161 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0cd36d:::benchmark_data_smithi159_102124_object38517:head# 2024-04-15T18:22:42.162 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.162 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0e9f55:::benchmark_data_smithi159_102124_object45976:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0ecd12:::benchmark_data_smithi159_102124_object19896:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b0fa1b3:::benchmark_data_smithi159_102124_object855:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b104eb7:::benchmark_data_smithi159_102124_object6327:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b107d97:::benchmark_data_smithi159_102124_object28252:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b140fc6:::benchmark_data_smithi159_102124_object57772:head# 2024-04-15T18:22:42.163 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.167 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b14ff30:::benchmark_data_smithi159_102124_object44717:head# 2024-04-15T18:22:42.167 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b16c5a1:::benchmark_data_smithi159_102124_object38999:head# 2024-04-15T18:22:42.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.168 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b18c370:::benchmark_data_smithi159_102124_object40631:head# 2024-04-15T18:22:42.168 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.171 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b18f6f4:::benchmark_data_smithi159_102124_object15661:head# 2024-04-15T18:22:42.171 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.172 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1a751c:::benchmark_data_smithi159_102124_object55101:head# 2024-04-15T18:22:42.172 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1a89e5:::benchmark_data_smithi159_102124_object16452:head# 2024-04-15T18:22:42.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.173 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1aba86:::benchmark_data_smithi159_102124_object841:head# 2024-04-15T18:22:42.173 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.174 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1b0911:::benchmark_data_smithi159_102124_object55920:head# 2024-04-15T18:22:42.174 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1c4aa5:::benchmark_data_smithi159_102124_object35902:head# 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1c7b9d:::benchmark_data_smithi159_102124_object10:head# 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1cee15:::benchmark_data_smithi159_102124_object39414:head# 2024-04-15T18:22:42.176 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b1e05d1:::benchmark_data_smithi159_102124_object55677:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b20542c:::benchmark_data_smithi159_102124_object6938:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b20b40f:::benchmark_data_smithi159_102124_object49789:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b21bb80:::benchmark_data_smithi159_102124_object6431:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b239e0f:::benchmark_data_smithi159_102124_object16390:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b260fde:::benchmark_data_smithi159_102124_object35116:head# 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.177 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b266861:::benchmark_data_smithi159_102124_object5323:head# 2024-04-15T18:22:42.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.178 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b27a878:::benchmark_data_smithi159_102124_object15074:head# 2024-04-15T18:22:42.178 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b283ab6:::benchmark_data_smithi159_102124_object47251:head# 2024-04-15T18:22:42.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b292160:::benchmark_data_smithi159_102124_object25014:head# 2024-04-15T18:22:42.251 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.251 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b29dc3d:::benchmark_data_smithi159_102124_object11776:head# 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b2c6c07:::benchmark_data_smithi159_102124_object9845:head# 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b2e90eb:::benchmark_data_smithi159_102124_object25996:head# 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b311f7b:::benchmark_data_smithi159_102124_object11336:head# 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3176c6:::benchmark_data_smithi159_102124_object6174:head# 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b32b234:::benchmark_data_smithi159_102124_object23809:head# 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b32cd58:::benchmark_data_smithi159_102124_object44936:head# 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b340ae4:::benchmark_data_smithi159_102124_object47663:head# 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.253 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b34fd5e:::benchmark_data_smithi159_102124_object26383:head# 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b352c07:::benchmark_data_smithi159_102124_object32437:head# 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b36b00c:::benchmark_data_smithi159_102124_object52399:head# 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b397bdd:::benchmark_data_smithi159_102124_object57823:head# 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.254 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3aab73:::benchmark_data_smithi159_102124_object7889:head# 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3acb80:::benchmark_data_smithi159_102124_object6868:head# 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3b4d68:::benchmark_data_smithi159_102124_object56098:head# 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3b9351:::benchmark_data_smithi159_102124_object18448:head# 2024-04-15T18:22:42.255 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3d3572:::benchmark_data_smithi159_102124_object2328:head# 2024-04-15T18:22:42.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3d49d0:::benchmark_data_smithi159_102124_object13896:head# 2024-04-15T18:22:42.256 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.256 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b3f6d7e:::benchmark_data_smithi159_102124_object39629:head# 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b40e0e1:::benchmark_data_smithi159_102124_object49764:head# 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b413e4c:::benchmark_data_smithi159_102124_object2913:head# 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b46ddd6:::benchmark_data_smithi159_102124_object41050:head# 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b46f26d:::benchmark_data_smithi159_102124_object48697:head# 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.257 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b47b4e2:::benchmark_data_smithi159_102124_object3549:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4ba63e:::benchmark_data_smithi159_102124_object52871:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4c0bba:::benchmark_data_smithi159_102124_object47424:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4c50e2:::benchmark_data_smithi159_102124_object36098:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4c74d2:::benchmark_data_smithi159_102124_object47753:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4ecc93:::benchmark_data_smithi159_102124_object2723:head# 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.258 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b4f7e10:::benchmark_data_smithi159_102124_object43222:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b504d1c:::benchmark_data_smithi159_102124_object3188:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b506f2f:::benchmark_data_smithi159_102124_object36469:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b51cc6a:::benchmark_data_smithi159_102124_object32244:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b52212a:::benchmark_data_smithi159_102124_object14492:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b54e88a:::benchmark_data_smithi159_102124_object32900:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b55d42f:::benchmark_data_smithi159_102124_object133:head# 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.259 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b57ead3:::benchmark_data_smithi159_102124_object20648:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b584a56:::benchmark_data_smithi159_102124_object35211:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b58fa1d:::benchmark_data_smithi159_102124_object4711:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b59b37d:::benchmark_data_smithi159_102124_object9078:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5a2c15:::benchmark_data_smithi159_102124_object34387:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5ae70d:::benchmark_data_smithi159_102124_object49042:head# 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.260 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5b6bcd:::benchmark_data_smithi159_102124_object49722:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5e7564:::benchmark_data_smithi159_102124_object17268:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5e8168:::benchmark_data_smithi159_102124_object32145:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b5f1c2f:::benchmark_data_smithi159_102124_object42387:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b602335:::benchmark_data_smithi159_102124_object55089:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b60ad51:::benchmark_data_smithi159_102124_object6769:head# 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.261 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b617b83:::benchmark_data_smithi159_102124_object28443:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b62855c:::benchmark_data_smithi159_102124_object41691:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b62ef58:::benchmark_data_smithi159_102124_object16556:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b640085:::benchmark_data_smithi159_102124_object22804:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b650441:::benchmark_data_smithi159_102124_object19864:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b66eb65:::benchmark_data_smithi159_102124_object13436:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b67ae66:::benchmark_data_smithi159_102124_object13844:head# 2024-04-15T18:22:42.262 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6917da:::benchmark_data_smithi159_102124_object36038:head# 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b69c64b:::benchmark_data_smithi159_102124_object8892:head# 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b69cfe4:::benchmark_data_smithi159_102124_object5697:head# 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b69e141:::benchmark_data_smithi159_102124_object46276:head# 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.263 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6a8bce:::benchmark_data_smithi159_102124_object11563:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6b5d84:::benchmark_data_smithi159_102124_object50112:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6d8b31:::benchmark_data_smithi159_102124_object35772:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6e95ea:::benchmark_data_smithi159_102124_object10843:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b6ef8fb:::benchmark_data_smithi159_102124_object45260:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b717f53:::benchmark_data_smithi159_102124_object17236:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b71b694:::benchmark_data_smithi159_102124_object3671:head# 2024-04-15T18:22:42.264 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b71d8bb:::benchmark_data_smithi159_102124_object14751:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b71ea9c:::benchmark_data_smithi159_102124_object40607:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b737566:::benchmark_data_smithi159_102124_object6210:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b73e814:::benchmark_data_smithi159_102124_object35735:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b73ebfe:::benchmark_data_smithi159_102124_object48634:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b76225c:::benchmark_data_smithi159_102124_object42047:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b763015:::benchmark_data_smithi159_102124_object38822:head# 2024-04-15T18:22:42.265 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b78dff3:::benchmark_data_smithi159_102124_object21207:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b7a5347:::benchmark_data_smithi159_102124_object9111:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b7a986b:::benchmark_data_smithi159_102124_object6448:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b7d15f2:::benchmark_data_smithi159_102124_object17503:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b7e5c36:::benchmark_data_smithi159_102124_object14868:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b7f1ba9:::benchmark_data_smithi159_102124_object48635:head# 2024-04-15T18:22:42.266 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b801e91:::benchmark_data_smithi159_102124_object35664:head# 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b80dd77:::benchmark_data_smithi159_102124_object47334:head# 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b81110e:::benchmark_data_smithi159_102124_object14786:head# 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b8459d5:::benchmark_data_smithi159_102124_object19524:head# 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.267 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b880430:::benchmark_data_smithi159_102124_object45800:head# 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b88b966:::benchmark_data_smithi159_102124_object29031:head# 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b897b1b:::benchmark_data_smithi159_102124_object22275:head# 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b89d914:::benchmark_data_smithi159_102124_object53234:head# 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b8b68f4:::benchmark_data_smithi159_102124_object6064:head# 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.268 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b8b75f3:::benchmark_data_smithi159_102124_object45780:head# 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b8dc708:::benchmark_data_smithi159_102124_object23741:head# 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b8fe340:::benchmark_data_smithi159_102124_object14281:head# 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b913374:::benchmark_data_smithi159_102124_object43768:head# 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b918a44:::benchmark_data_smithi159_102124_object10781:head# 2024-04-15T18:22:42.269 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b926f28:::benchmark_data_smithi159_102124_object55697:head# 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b949390:::benchmark_data_smithi159_102124_object51068:head# 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b969718:::benchmark_data_smithi159_102124_object55393:head# 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b98527d:::benchmark_data_smithi159_102124_object30484:head# 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.270 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9aef4c:::benchmark_data_smithi159_102124_object8857:head# 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9bdb19:::benchmark_data_smithi159_102124_object56057:head# 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9cb646:::benchmark_data_smithi159_102124_object10853:head# 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9cee68:::benchmark_data_smithi159_102124_object37309:head# 2024-04-15T18:22:42.271 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.272 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9cfba6:::benchmark_data_smithi159_102124_object34074:head# 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9d7770:::benchmark_data_smithi159_102124_object21138:head# 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9e58f6:::benchmark_data_smithi159_102124_object43807:head# 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2b9f0cbe:::benchmark_data_smithi159_102124_object11962:head# 2024-04-15T18:22:42.273 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba0785b:::benchmark_data_smithi159_102124_object37468:head# 2024-04-15T18:22:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba07996:::benchmark_data_smithi159_102124_object37042:head# 2024-04-15T18:22:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.274 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba07b7b:::benchmark_data_smithi159_102124_object44841:head# 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba1d493:::benchmark_data_smithi159_102124_object27877:head# 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba1ecbe:::benchmark_data_smithi159_102124_object54012:head# 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba2a8be:::benchmark_data_smithi159_102124_object29184:head# 2024-04-15T18:22:42.275 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba3955c:::benchmark_data_smithi159_102124_object3037:head# 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba3f0ad:::benchmark_data_smithi159_102124_object52500:head# 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba48627:::benchmark_data_smithi159_102124_object11531:head# 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba5dcce:::benchmark_data_smithi159_102124_object14489:head# 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.276 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba65bc6:::benchmark_data_smithi159_102124_object45766:head# 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba6f703:::benchmark_data_smithi159_102124_object30410:head# 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba7034d:::benchmark_data_smithi159_102124_object49850:head# 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ba883dc:::benchmark_data_smithi159_102124_object37098:head# 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.277 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2baaef77:::benchmark_data_smithi159_102124_object39762:head# 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2babb627:::benchmark_data_smithi159_102124_object38433:head# 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bac54fb:::benchmark_data_smithi159_102124_object28994:head# 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2badf088:::benchmark_data_smithi159_102124_object57625:head# 2024-04-15T18:22:42.278 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bae46a6:::benchmark_data_smithi159_102124_object33794:head# 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb1130b:::benchmark_data_smithi159_102124_object40335:head# 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb229e0:::benchmark_data_smithi159_102124_object41899:head# 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.279 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb2557f:::benchmark_data_smithi159_102124_object6215:head# 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb2edce:::benchmark_data_smithi159_102124_object18797:head# 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb3343f:::benchmark_data_smithi159_102124_object10911:head# 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.280 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb3ec4d:::benchmark_data_smithi159_102124_object1089:head# 2024-04-15T18:22:42.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb55c5c:::benchmark_data_smithi159_102124_object15092:head# 2024-04-15T18:22:42.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.281 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb56f47:::benchmark_data_smithi159_102124_object17650:head# 2024-04-15T18:22:42.281 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb7acaa:::benchmark_data_smithi159_102124_object48105:head# 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb81552:::benchmark_data_smithi159_102124_object25576:head# 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb87f7f:::benchmark_data_smithi159_102124_object17327:head# 2024-04-15T18:22:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb8fdfa:::benchmark_data_smithi159_102124_object21618:head# 2024-04-15T18:22:42.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.283 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb9478f:::benchmark_data_smithi159_102124_object18889:head# 2024-04-15T18:22:42.283 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.284 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:42.284 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bb9c265:::benchmark_data_smithi159_102124_object47841:head# 2024-04-15T18:22:42.284 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:42.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbb40b1:::benchmark_data_smithi159_102124_object582:head# 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbb7a15:::benchmark_data_smithi159_102124_object58188:head# 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbc0ff6:::benchmark_data_smithi159_102124_object13338:head# 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbe09a7:::benchmark_data_smithi159_102124_object48009:head# 2024-04-15T18:22:42.287 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbe1149:::benchmark_data_smithi159_102124_object5085:head# 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbe3a82:::benchmark_data_smithi159_102124_object10471:head# 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbe7e17:::benchmark_data_smithi159_102124_object50054:head# 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbe7ed1:::benchmark_data_smithi159_102124_object24924:head# 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbed100:::benchmark_data_smithi159_102124_object13421:head# 2024-04-15T18:22:42.288 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bbfcad8:::benchmark_data_smithi159_102124_object49409:head# 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc16117:::benchmark_data_smithi159_102124_object1004:head# 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc165f3:::benchmark_data_smithi159_102124_object28933:head# 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc3f45c:::benchmark_data_smithi159_102124_object44286:head# 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc52b99:::benchmark_data_smithi159_102124_object50074:head# 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.289 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc54062:::benchmark_data_smithi159_102124_object26850:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bc7a4fd:::benchmark_data_smithi159_102124_object40399:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bcab63a:::benchmark_data_smithi159_102124_object42045:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bcaf92b:::benchmark_data_smithi159_102124_object46609:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bcc556d:::benchmark_data_smithi159_102124_object1820:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bce104b:::benchmark_data_smithi159_102124_object46744:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bcee70b:::benchmark_data_smithi159_102124_object16740:head# 2024-04-15T18:22:42.290 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd2c80f:::benchmark_data_smithi159_102124_object53873:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd3c296:::benchmark_data_smithi159_102124_object31334:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd62d64:::benchmark_data_smithi159_102124_object19196:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd666d8:::benchmark_data_smithi159_102124_object23729:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd6c4bf:::benchmark_data_smithi159_102124_object28072:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd8091e:::benchmark_data_smithi159_102124_object37574:head# 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.291 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd99b6e:::benchmark_data_smithi159_102124_object4114:head# 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd9a016:::benchmark_data_smithi159_102124_object54007:head# 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bd9ecb2:::benchmark_data_smithi159_102124_object57401:head# 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bdb1722:::benchmark_data_smithi159_102124_object57386:head# 2024-04-15T18:22:42.292 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.293 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bdd27aa:::benchmark_data_smithi159_102124_object44637:head# 2024-04-15T18:22:42.293 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bddf287:::benchmark_data_smithi159_102124_object54934:head# 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bde0188:::benchmark_data_smithi159_102124_object36433:head# 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bdf5c41:::benchmark_data_smithi159_102124_object22392:head# 2024-04-15T18:22:42.294 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.295 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be07e57:::benchmark_data_smithi159_102124_object24733:head# 2024-04-15T18:22:42.295 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be0d078:::benchmark_data_smithi159_102124_object26270:head# 2024-04-15T18:22:42.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be1c426:::benchmark_data_smithi159_102124_object23260:head# 2024-04-15T18:22:42.296 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.296 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be54003:::benchmark_data_smithi159_102124_object11624:head# 2024-04-15T18:22:42.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be62614:::benchmark_data_smithi159_102124_object51422:head# 2024-04-15T18:22:42.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.297 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be66ae4:::benchmark_data_smithi159_102124_object49701:head# 2024-04-15T18:22:42.297 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be6cc22:::benchmark_data_smithi159_102124_object3552:head# 2024-04-15T18:22:42.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be73411:::benchmark_data_smithi159_102124_object335:head# 2024-04-15T18:22:42.298 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.298 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be8d40d:::benchmark_data_smithi159_102124_object54489:head# 2024-04-15T18:22:42.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be8e310:::benchmark_data_smithi159_102124_object1838:head# 2024-04-15T18:22:42.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.299 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be92522:::benchmark_data_smithi159_102124_object45186:head# 2024-04-15T18:22:42.299 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2be946a7:::benchmark_data_smithi159_102124_object26691:head# 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2beb5cf1:::benchmark_data_smithi159_102124_object58006:head# 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bebe79b:::benchmark_data_smithi159_102124_object23404:head# 2024-04-15T18:22:42.300 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.301 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bed016d:::benchmark_data_smithi159_102124_object21729:head# 2024-04-15T18:22:42.301 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bed54d7:::benchmark_data_smithi159_102124_object9900:head# 2024-04-15T18:22:42.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.303 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf0c5a5:::benchmark_data_smithi159_102124_object41543:head# 2024-04-15T18:22:42.303 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf0dffb:::benchmark_data_smithi159_102124_object12836:head# 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf108ef:::benchmark_data_smithi159_102124_object14153:head# 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf15f2f:::benchmark_data_smithi159_102124_object13879:head# 2024-04-15T18:22:42.304 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf1f891:::benchmark_data_smithi159_102124_object46667:head# 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf267aa:::benchmark_data_smithi159_102124_object52652:head# 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf4181b:::benchmark_data_smithi159_102124_object26291:head# 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.305 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf73d6a:::benchmark_data_smithi159_102124_object20766:head# 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf75a66:::benchmark_data_smithi159_102124_object7498:head# 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bf9dd2b:::benchmark_data_smithi159_102124_object22835:head# 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bfa7e4d:::benchmark_data_smithi159_102124_object18031:head# 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bfb5f2f:::benchmark_data_smithi159_102124_object8007:head# 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bfdb7ed:::benchmark_data_smithi159_102124_object3954:head# 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bfdf58a:::benchmark_data_smithi159_102124_object5653:head# 2024-04-15T18:22:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.308 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bff1e86:::benchmark_data_smithi159_102124_object50153:head# 2024-04-15T18:22:42.308 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.311 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bff7033:::benchmark_data_smithi159_102124_object35732:head# 2024-04-15T18:22:42.311 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.313 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2bff9158:::benchmark_data_smithi159_102124_object41713:head# 2024-04-15T18:22:42.314 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.314 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0377fc:::benchmark_data_smithi159_102124_object7945:head# 2024-04-15T18:22:42.315 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c03dc24:::benchmark_data_smithi159_102124_object19575:head# 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c05de86:::benchmark_data_smithi159_102124_object38661:head# 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c05fa76:::benchmark_data_smithi159_102124_object7775:head# 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c08a09b:::benchmark_data_smithi159_102124_object604:head# 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0939ab:::benchmark_data_smithi159_102124_object11378:head# 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.320 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c09797a:::benchmark_data_smithi159_102124_object42277:head# 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0c123f:::benchmark_data_smithi159_102124_object5618:head# 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0c5df5:::benchmark_data_smithi159_102124_object26077:head# 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0ceb6a:::benchmark_data_smithi159_102124_object10321:head# 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0d9d22:::benchmark_data_smithi159_102124_object6861:head# 2024-04-15T18:22:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0e3428:::benchmark_data_smithi159_102124_object28096:head# 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0e5fce:::benchmark_data_smithi159_102124_object57268:head# 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0e7496:::benchmark_data_smithi159_102124_object37854:head# 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.322 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c0f97ab:::benchmark_data_smithi159_102124_object13067:head# 2024-04-15T18:22:42.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.323 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1021a2:::benchmark_data_smithi159_102124_object49193:head# 2024-04-15T18:22:42.323 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c10fc59:::benchmark_data_smithi159_102124_object25274:head# 2024-04-15T18:22:42.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.330 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1214c3:::benchmark_data_smithi159_102124_object14471:head# 2024-04-15T18:22:42.330 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.332 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c131807:::benchmark_data_smithi159_102124_object1031:head# 2024-04-15T18:22:42.332 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1346ba:::benchmark_data_smithi159_102124_object45021:head# 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c136f96:::benchmark_data_smithi159_102124_object20911:head# 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c15f559:::benchmark_data_smithi159_102124_object9600:head# 2024-04-15T18:22:42.340 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.342 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1838b3:::benchmark_data_smithi159_102124_object50235:head# 2024-04-15T18:22:42.342 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1a3c26:::benchmark_data_smithi159_102124_object17058:head# 2024-04-15T18:22:42.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1c980a:::benchmark_data_smithi159_102124_object26758:head# 2024-04-15T18:22:42.343 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.343 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1d27fc:::benchmark_data_smithi159_102124_object10323:head# 2024-04-15T18:22:42.344 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1e1333:::benchmark_data_smithi159_102124_object16969:head# 2024-04-15T18:22:42.345 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.345 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c1fa461:::benchmark_data_smithi159_102124_object16742:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c22675b:::benchmark_data_smithi159_102124_object3263:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2320e9:::benchmark_data_smithi159_102124_object39813:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c23be16:::benchmark_data_smithi159_102124_object34795:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c243c17:::benchmark_data_smithi159_102124_object54476:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c263f53:::benchmark_data_smithi159_102124_object46386:head# 2024-04-15T18:22:42.346 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c26fe39:::benchmark_data_smithi159_102124_object5708:head# 2024-04-15T18:22:42.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.347 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c277297:::benchmark_data_smithi159_102124_object5707:head# 2024-04-15T18:22:42.347 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c279fe9:::benchmark_data_smithi159_102124_object58260:head# 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c281e5c:::benchmark_data_smithi159_102124_object46288:head# 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c293f29:::benchmark_data_smithi159_102124_object44932:head# 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c29921a:::benchmark_data_smithi159_102124_object44895:head# 2024-04-15T18:22:42.348 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2a7a12:::benchmark_data_smithi159_102124_object41630:head# 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2a82d2:::benchmark_data_smithi159_102124_object30675:head# 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2bbe8b:::benchmark_data_smithi159_102124_object37326:head# 2024-04-15T18:22:42.349 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2c1683:::benchmark_data_smithi159_102124_object7634:head# 2024-04-15T18:22:42.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.350 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2c86b7:::benchmark_data_smithi159_102124_object970:head# 2024-04-15T18:22:42.350 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.351 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2df90d:::benchmark_data_smithi159_102124_object55269:head# 2024-04-15T18:22:42.351 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2f198a:::benchmark_data_smithi159_102124_object11779:head# 2024-04-15T18:22:42.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c2ff0f7:::benchmark_data_smithi159_102124_object805:head# 2024-04-15T18:22:42.352 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.352 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c301e5c:::benchmark_data_smithi159_102124_object42315:head# 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c316bbc:::benchmark_data_smithi159_102124_object29402:head# 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c31c45f:::benchmark_data_smithi159_102124_object24004:head# 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c355253:::benchmark_data_smithi159_102124_object51325:head# 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.353 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c35dc7b:::benchmark_data_smithi159_102124_object53934:head# 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c362912:::benchmark_data_smithi159_102124_object53933:head# 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c36a667:::benchmark_data_smithi159_102124_object7589:head# 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c36b3d2:::benchmark_data_smithi159_102124_object36413:head# 2024-04-15T18:22:42.354 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c382025:::benchmark_data_smithi159_102124_object26123:head# 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c387770:::benchmark_data_smithi159_102124_object1869:head# 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c3b32f2:::benchmark_data_smithi159_102124_object36942:head# 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c3d2301:::benchmark_data_smithi159_102124_object8182:head# 2024-04-15T18:22:42.357 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c3e160b:::benchmark_data_smithi159_102124_object21850:head# 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c3e53f0:::benchmark_data_smithi159_102124_object1577:head# 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c3f4336:::benchmark_data_smithi159_102124_object50792:head# 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4174b0:::benchmark_data_smithi159_102124_object55626:head# 2024-04-15T18:22:42.358 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4240b8:::benchmark_data_smithi159_102124_object31397:head# 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c441656:::benchmark_data_smithi159_102124_object24804:head# 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c463b4c:::benchmark_data_smithi159_102124_object40778:head# 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4b3e95:::benchmark_data_smithi159_102124_object8040:head# 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.359 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4c15be:::benchmark_data_smithi159_102124_object33523:head# 2024-04-15T18:22:42.360 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4d17fd:::benchmark_data_smithi159_102124_object8344:head# 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4d3edb:::benchmark_data_smithi159_102124_object28017:head# 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4dc74d:::benchmark_data_smithi159_102124_object49971:head# 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4e1964:::benchmark_data_smithi159_102124_object57033:head# 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.361 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c4fdec0:::benchmark_data_smithi159_102124_object173:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c518f05:::benchmark_data_smithi159_102124_object6340:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c519f62:::benchmark_data_smithi159_102124_object26436:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c544c0b:::benchmark_data_smithi159_102124_object41025:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c54d11f:::benchmark_data_smithi159_102124_object3195:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c56045b:::benchmark_data_smithi159_102124_object52944:head# 2024-04-15T18:22:42.362 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c585119:::benchmark_data_smithi159_102124_object1673:head# 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c599eb4:::benchmark_data_smithi159_102124_object6047:head# 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c5a9cfc:::benchmark_data_smithi159_102124_object25610:head# 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c5ae149:::benchmark_data_smithi159_102124_object21010:head# 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c5b1b3c:::benchmark_data_smithi159_102124_object27303:head# 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.363 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c5eb338:::benchmark_data_smithi159_102124_object36681:head# 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c603e82:::benchmark_data_smithi159_102124_object32220:head# 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c61fe08:::benchmark_data_smithi159_102124_object31020:head# 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c62171e:::benchmark_data_smithi159_102124_object36430:head# 2024-04-15T18:22:42.364 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c62d3e7:::benchmark_data_smithi159_102124_object26070:head# 2024-04-15T18:22:42.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.365 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c637672:::benchmark_data_smithi159_102124_object1455:head# 2024-04-15T18:22:42.365 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c64bf05:::benchmark_data_smithi159_102124_object22826:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6540ba:::benchmark_data_smithi159_102124_object4675:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c65f2a2:::benchmark_data_smithi159_102124_object33982:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c669c34:::benchmark_data_smithi159_102124_object49526:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c693623:::benchmark_data_smithi159_102124_object6885:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6a4e81:::benchmark_data_smithi159_102124_object17847:head# 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.366 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6d3ae5:::benchmark_data_smithi159_102124_object22980:head# 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6d7fc3:::benchmark_data_smithi159_102124_object3768:head# 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6e7e03:::benchmark_data_smithi159_102124_object1492:head# 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6ebd94:::benchmark_data_smithi159_102124_object48574:head# 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c6f7e5c:::benchmark_data_smithi159_102124_object29001:head# 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.367 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c700560:::benchmark_data_smithi159_102124_object21614:head# 2024-04-15T18:22:42.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c704f1c:::benchmark_data_smithi159_102124_object15000:head# 2024-04-15T18:22:42.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.368 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c70e0f7:::benchmark_data_smithi159_102124_object58115:head# 2024-04-15T18:22:42.368 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c729cf8:::benchmark_data_smithi159_102124_object21816:head# 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c741096:::benchmark_data_smithi159_102124_object4555:head# 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c745390:::benchmark_data_smithi159_102124_object45437:head# 2024-04-15T18:22:42.369 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c74e43b:::benchmark_data_smithi159_102124_object13655:head# 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c793b30:::benchmark_data_smithi159_102124_object18762:head# 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c79ed40:::benchmark_data_smithi159_102124_object13730:head# 2024-04-15T18:22:42.370 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7a2d0d:::benchmark_data_smithi159_102124_object21410:head# 2024-04-15T18:22:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7a9313:::benchmark_data_smithi159_102124_object26764:head# 2024-04-15T18:22:42.371 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7b393f:::benchmark_data_smithi159_102124_object29919:head# 2024-04-15T18:22:42.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7bca1a:::benchmark_data_smithi159_102124_object45300:head# 2024-04-15T18:22:42.374 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.374 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7c49ce:::benchmark_data_smithi159_102124_object12671:head# 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7de5bc:::benchmark_data_smithi159_102124_object18215:head# 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c7f3aba:::benchmark_data_smithi159_102124_object35096:head# 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c800f01:::benchmark_data_smithi159_102124_object44920:head# 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.375 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c806133:::benchmark_data_smithi159_102124_object2009:head# 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8084bb:::benchmark_data_smithi159_102124_object47503:head# 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8096b2:::benchmark_data_smithi159_102124_object17401:head# 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c80c134:::benchmark_data_smithi159_102124_object33564:head# 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c80f7e9:::benchmark_data_smithi159_102124_object31226:head# 2024-04-15T18:22:42.376 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c811f2c:::benchmark_data_smithi159_102124_object36489:head# 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c815361:::benchmark_data_smithi159_102124_object3994:head# 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c822898:::benchmark_data_smithi159_102124_object24400:head# 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c826d49:::benchmark_data_smithi159_102124_object36415:head# 2024-04-15T18:22:42.377 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c83fcb2:::benchmark_data_smithi159_102124_object25441:head# 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c854215:::benchmark_data_smithi159_102124_object53758:head# 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c85b109:::benchmark_data_smithi159_102124_object23643:head# 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c876316:::benchmark_data_smithi159_102124_object46169:head# 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.378 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8773cc:::benchmark_data_smithi159_102124_object20164:head# 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8862d4:::benchmark_data_smithi159_102124_object8854:head# 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8892cd:::benchmark_data_smithi159_102124_object2297:head# 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c88e3ad:::benchmark_data_smithi159_102124_object7699:head# 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.379 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c898cd5:::benchmark_data_smithi159_102124_object19867:head# 2024-04-15T18:22:42.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.380 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c89d72a:::benchmark_data_smithi159_102124_object50049:head# 2024-04-15T18:22:42.380 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8c5513:::benchmark_data_smithi159_102124_object14998:head# 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8c754b:::benchmark_data_smithi159_102124_object49182:head# 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8d3787:::benchmark_data_smithi159_102124_object42648:head# 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8fc6a1:::benchmark_data_smithi159_102124_object6881:head# 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.382 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c8fdbdc:::benchmark_data_smithi159_102124_object40859:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c94168a:::benchmark_data_smithi159_102124_object23381:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c949bfc:::benchmark_data_smithi159_102124_object6050:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c951df6:::benchmark_data_smithi159_102124_object30425:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c96936e:::benchmark_data_smithi159_102124_object27696:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c96a291:::benchmark_data_smithi159_102124_object46359:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c9721e0:::benchmark_data_smithi159_102124_object10222:head# 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.383 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c97c88b:::benchmark_data_smithi159_102124_object26160:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c983428:::benchmark_data_smithi159_102124_object43909:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c98bf45:::benchmark_data_smithi159_102124_object24657:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c99c138:::benchmark_data_smithi159_102124_object46824:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c9ae883:::benchmark_data_smithi159_102124_object17359:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c9c6b1e:::benchmark_data_smithi159_102124_object43855:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2c9dd683:::benchmark_data_smithi159_102124_object18833:head# 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.384 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca12cf9:::benchmark_data_smithi159_102124_object24854:head# 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca30a50:::benchmark_data_smithi159_102124_object49588:head# 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca3897f:::benchmark_data_smithi159_102124_object36515:head# 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca4bfac:::benchmark_data_smithi159_102124_object12954:head# 2024-04-15T18:22:42.385 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca75f27:::benchmark_data_smithi159_102124_object21147:head# 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca8e869:::benchmark_data_smithi159_102124_object44415:head# 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ca94455:::benchmark_data_smithi159_102124_object18351:head# 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2caa387d:::benchmark_data_smithi159_102124_object44579:head# 2024-04-15T18:22:42.386 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cab598f:::benchmark_data_smithi159_102124_object40625:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cab9597:::benchmark_data_smithi159_102124_object12290:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cace85b:::benchmark_data_smithi159_102124_object41572:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb311a7:::benchmark_data_smithi159_102124_object48811:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb4b905:::benchmark_data_smithi159_102124_object22574:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb4c584:::benchmark_data_smithi159_102124_object40180:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb5248a:::benchmark_data_smithi159_102124_object46741:head# 2024-04-15T18:22:42.387 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb54347:::benchmark_data_smithi159_102124_object28743:head# 2024-04-15T18:22:42.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.388 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb5ded8:::benchmark_data_smithi159_102124_object37356:head# 2024-04-15T18:22:42.388 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:42.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb5e216:::benchmark_data_smithi159_102124_object27183:head# 2024-04-15T18:22:42.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.389 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb6bec9:::benchmark_data_smithi159_102124_object41487:head# 2024-04-15T18:22:42.389 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb77e11:::benchmark_data_smithi159_102124_object8021:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb89287:::benchmark_data_smithi159_102124_object17542:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb9982b:::benchmark_data_smithi159_102124_object32265:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cb9f8a2:::benchmark_data_smithi159_102124_object50588:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cba64b8:::benchmark_data_smithi159_102124_object56349:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cbabc1e:::benchmark_data_smithi159_102124_object45987:head# 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.390 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cbb162e:::benchmark_data_smithi159_102124_object51774:head# 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cbb21a5:::benchmark_data_smithi159_102124_object7258:head# 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cbdc4c9:::benchmark_data_smithi159_102124_object18033:head# 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cbea0b8:::benchmark_data_smithi159_102124_object17754:head# 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc0456a:::benchmark_data_smithi159_102124_object17448:head# 2024-04-15T18:22:42.391 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.400 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc073a7:::benchmark_data_smithi159_102124_object50920:head# 2024-04-15T18:22:42.400 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc12789:::benchmark_data_smithi159_102124_object749:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc26535:::benchmark_data_smithi159_102124_object57484:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc28418:::benchmark_data_smithi159_102124_object56498:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc4236c:::benchmark_data_smithi159_102124_object13686:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc4d8d2:::benchmark_data_smithi159_102124_object1319:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc5b2a2:::benchmark_data_smithi159_102124_object13008:head# 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.401 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc6d418:::benchmark_data_smithi159_102124_object329:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc70281:::benchmark_data_smithi159_102124_object50758:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc7453f:::benchmark_data_smithi159_102124_object5123:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc8d54e:::benchmark_data_smithi159_102124_object5013:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cc94911:::benchmark_data_smithi159_102124_object12364:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ccc0aa1:::benchmark_data_smithi159_102124_object36081:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ccd660d:::benchmark_data_smithi159_102124_object58318:head# 2024-04-15T18:22:42.402 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cce19a6:::benchmark_data_smithi159_102124_object11894:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ccece3a:::benchmark_data_smithi159_102124_object57707:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ccf157b:::benchmark_data_smithi159_102124_object2046:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ccf5d29:::benchmark_data_smithi159_102124_object57019:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd3a492:::benchmark_data_smithi159_102124_object41325:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd3d47e:::benchmark_data_smithi159_102124_object11670:head# 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.403 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd43f6d:::benchmark_data_smithi159_102124_object1965:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd4f9a1:::benchmark_data_smithi159_102124_object633:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd6ee99:::benchmark_data_smithi159_102124_object8326:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd6eeb8:::benchmark_data_smithi159_102124_object48201:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd6f2b7:::benchmark_data_smithi159_102124_object44608:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd8151d:::benchmark_data_smithi159_102124_object46290:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd88e4c:::benchmark_data_smithi159_102124_object20801:head# 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.404 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd8a235:::benchmark_data_smithi159_102124_object51261:head# 2024-04-15T18:22:42.405 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cd8a5e9:::benchmark_data_smithi159_102124_object18659:head# 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cdc7f98:::benchmark_data_smithi159_102124_object1025:head# 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cdd22e6:::benchmark_data_smithi159_102124_object52487:head# 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cdd59e2:::benchmark_data_smithi159_102124_object40752:head# 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.407 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cdd6def:::benchmark_data_smithi159_102124_object38232:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce20d9e:::benchmark_data_smithi159_102124_object36794:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce27d7b:::benchmark_data_smithi159_102124_object17903:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce3c41c:::benchmark_data_smithi159_102124_object31406:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce68792:::benchmark_data_smithi159_102124_object23346:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce6f03f:::benchmark_data_smithi159_102124_object35984:head# 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce85d69:::benchmark_data_smithi159_102124_object9945:head# 2024-04-15T18:22:42.409 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ce9b720:::benchmark_data_smithi159_102124_object21245:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cea65f9:::benchmark_data_smithi159_102124_object53264:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ceaea08:::benchmark_data_smithi159_102124_object27578:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ceb716c:::benchmark_data_smithi159_102124_object52524:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ced5823:::benchmark_data_smithi159_102124_object7821:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cef3150:::benchmark_data_smithi159_102124_object57942:head# 2024-04-15T18:22:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cef54ae:::benchmark_data_smithi159_102124_object19179:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cefc9fc:::benchmark_data_smithi159_102124_object47504:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf29023:::benchmark_data_smithi159_102124_object27156:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf2d720:::benchmark_data_smithi159_102124_object43743:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf416fa:::benchmark_data_smithi159_102124_object26688:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf4fa27:::benchmark_data_smithi159_102124_object1362:head# 2024-04-15T18:22:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf699a7:::benchmark_data_smithi159_102124_object11485:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf6dd4c:::benchmark_data_smithi159_102124_object17063:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf7b73c:::benchmark_data_smithi159_102124_object943:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf7e4eb:::benchmark_data_smithi159_102124_object27488:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf82a07:::benchmark_data_smithi159_102124_object33187:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cf96bdc:::benchmark_data_smithi159_102124_object15338:head# 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.413 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cfc267b:::benchmark_data_smithi159_102124_object53295:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cfd1eee:::benchmark_data_smithi159_102124_object49555:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cfd8cb7:::benchmark_data_smithi159_102124_object49218:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2cfea787:::benchmark_data_smithi159_102124_object33877:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d019969:::benchmark_data_smithi159_102124_object3329:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d01de47:::benchmark_data_smithi159_102124_object15157:head# 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.414 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d02d9ad:::benchmark_data_smithi159_102124_object51064:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d02f887:::benchmark_data_smithi159_102124_object23169:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d0336e8:::benchmark_data_smithi159_102124_object7913:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d03ad5a:::benchmark_data_smithi159_102124_object31418:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d06b7c0:::benchmark_data_smithi159_102124_object1708:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d08323e:::benchmark_data_smithi159_102124_object35619:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d0900c1:::benchmark_data_smithi159_102124_object53306:head# 2024-04-15T18:22:42.415 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.416 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d097500:::benchmark_data_smithi159_102124_object8382:head# 2024-04-15T18:22:42.416 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d0ad771:::benchmark_data_smithi159_102124_object6171:head# 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d10f8d0:::benchmark_data_smithi159_102124_object21805:head# 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d11414a:::benchmark_data_smithi159_102124_object21900:head# 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d11bcad:::benchmark_data_smithi159_102124_object38770:head# 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d126ae8:::benchmark_data_smithi159_102124_object37769:head# 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.417 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d12e4e3:::benchmark_data_smithi159_102124_object3030:head# 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d137f6f:::benchmark_data_smithi159_102124_object54328:head# 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d143580:::benchmark_data_smithi159_102124_object25329:head# 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.418 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d147355:::benchmark_data_smithi159_102124_object12480:head# 2024-04-15T18:22:42.419 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d165f70:::benchmark_data_smithi159_102124_object20489:head# 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d17afb5:::benchmark_data_smithi159_102124_object7313:head# 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d19089b:::benchmark_data_smithi159_102124_object40411:head# 2024-04-15T18:22:42.422 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d1a2af4:::benchmark_data_smithi159_102124_object7762:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d1f2f0e:::benchmark_data_smithi159_102124_object31172:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2303c5:::benchmark_data_smithi159_102124_object32673:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d23463d:::benchmark_data_smithi159_102124_object18990:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2388dc:::benchmark_data_smithi159_102124_object16662:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d244655:::benchmark_data_smithi159_102124_object56965:head# 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.423 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d287c52:::benchmark_data_smithi159_102124_object36412:head# 2024-04-15T18:22:42.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d298e83:::benchmark_data_smithi159_102124_object9540:head# 2024-04-15T18:22:42.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.424 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2b5615:::benchmark_data_smithi159_102124_object26636:head# 2024-04-15T18:22:42.424 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.426 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2caf6e:::benchmark_data_smithi159_102124_object29322:head# 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2e7544:::benchmark_data_smithi159_102124_object9307:head# 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d2edc92:::benchmark_data_smithi159_102124_object12245:head# 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d304e3d:::benchmark_data_smithi159_102124_object58439:head# 2024-04-15T18:22:42.427 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.431 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d323713:::benchmark_data_smithi159_102124_object36938:head# 2024-04-15T18:22:42.432 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.433 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d33a16d:::benchmark_data_smithi159_102124_object39549:head# 2024-04-15T18:22:42.433 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3478ee:::benchmark_data_smithi159_102124_object45642:head# 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3657be:::benchmark_data_smithi159_102124_object14957:head# 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3766dc:::benchmark_data_smithi159_102124_object2271:head# 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3b4abb:::benchmark_data_smithi159_102124_object37899:head# 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3c909b:::benchmark_data_smithi159_102124_object24785:head# 2024-04-15T18:22:42.434 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3d0e74:::benchmark_data_smithi159_102124_object25510:head# 2024-04-15T18:22:42.436 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.436 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d3e999d:::benchmark_data_smithi159_102124_object44943:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d424dcb:::benchmark_data_smithi159_102124_object4167:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4262a5:::benchmark_data_smithi159_102124_object50616:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d427b73:::benchmark_data_smithi159_102124_object14528:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d439abb:::benchmark_data_smithi159_102124_object6684:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d458361:::benchmark_data_smithi159_102124_object43437:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d46c5b7:::benchmark_data_smithi159_102124_object15224:head# 2024-04-15T18:22:42.437 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d471174:::benchmark_data_smithi159_102124_object26240:head# 2024-04-15T18:22:42.438 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.438 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d498b4e:::benchmark_data_smithi159_102124_object44097:head# 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d49f804:::benchmark_data_smithi159_102124_object4226:head# 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4b78e2:::benchmark_data_smithi159_102124_object39464:head# 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4c9304:::benchmark_data_smithi159_102124_object53666:head# 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4c986d:::benchmark_data_smithi159_102124_object17738:head# 2024-04-15T18:22:42.439 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: pgmap v904: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 5033/199545 objects degraded (2.522%); 5.1 MiB/s, 81 objects/s recovering 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5033/199545 objects degraded (2.522%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:22:42.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:22:42.441 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4da26a:::benchmark_data_smithi159_102124_object16247:head# 2024-04-15T18:22:42.441 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.441 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4e0139:::benchmark_data_smithi159_102124_object101:head# 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4e4973:::benchmark_data_smithi159_102124_object30395:head# 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4f9b21:::benchmark_data_smithi159_102124_object15936:head# 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d4fda6c:::benchmark_data_smithi159_102124_object12519:head# 2024-04-15T18:22:42.442 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d502388:::benchmark_data_smithi159_102124_object3846:head# 2024-04-15T18:22:42.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.443 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d505b44:::benchmark_data_smithi159_102124_object4851:head# 2024-04-15T18:22:42.443 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.444 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:42.448 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d512ddb:::benchmark_data_smithi159_102124_object12820:head# 2024-04-15T18:22:42.448 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.448 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d52db52:::benchmark_data_smithi159_102124_object33562:head# 2024-04-15T18:22:42.448 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.448 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d543dc7:::benchmark_data_smithi159_102124_object6109:head# 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d544294:::benchmark_data_smithi159_102124_object19741:head# 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d54e06a:::benchmark_data_smithi159_102124_object22074:head# 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.449 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d558533:::benchmark_data_smithi159_102124_object11395:head# 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d56dfed:::benchmark_data_smithi159_102124_object33470:head# 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d57b6d7:::benchmark_data_smithi159_102124_object10056:head# 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d57dc81:::benchmark_data_smithi159_102124_object47538:head# 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d57f4e5:::benchmark_data_smithi159_102124_object25260:head# 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d58d7b1:::benchmark_data_smithi159_102124_object1729:head# 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d590181:::benchmark_data_smithi159_102124_object34289:head# 2024-04-15T18:22:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5b9a0a:::benchmark_data_smithi159_102124_object14582:head# 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5c1fea:::benchmark_data_smithi159_102124_object11663:head# 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5d05fc:::benchmark_data_smithi159_102124_object15458:head# 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5d377c:::benchmark_data_smithi159_102124_object33545:head# 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5d7c79:::benchmark_data_smithi159_102124_object2779:head# 2024-04-15T18:22:42.452 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.453 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d5f1c00:::benchmark_data_smithi159_102124_object4446:head# 2024-04-15T18:22:42.453 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d60392e:::benchmark_data_smithi159_102124_object25725:head# 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d63ef0d:::benchmark_data_smithi159_102124_object43245:head# 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6653ce:::benchmark_data_smithi159_102124_object48242:head# 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d671192:::benchmark_data_smithi159_102124_object23966:head# 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.454 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d687289:::benchmark_data_smithi159_102124_object57361:head# 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d68c404:::benchmark_data_smithi159_102124_object57357:head# 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d69e0b6:::benchmark_data_smithi159_102124_object49645:head# 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6a28ca:::benchmark_data_smithi159_102124_object22211:head# 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.455 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6b603d:::benchmark_data_smithi159_102124_object5533:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6b9b7c:::benchmark_data_smithi159_102124_object35098:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6eb96c:::benchmark_data_smithi159_102124_object32768:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6f3b5b:::benchmark_data_smithi159_102124_object18242:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6fa2e2:::benchmark_data_smithi159_102124_object52325:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d6fce81:::benchmark_data_smithi159_102124_object28515:head# 2024-04-15T18:22:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7208a2:::benchmark_data_smithi159_102124_object43644:head# 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d72841a:::benchmark_data_smithi159_102124_object25531:head# 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d735676:::benchmark_data_smithi159_102124_object40791:head# 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.457 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d73cc15:::benchmark_data_smithi159_102124_object48772:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d75d240:::benchmark_data_smithi159_102124_object8409:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d76b441:::benchmark_data_smithi159_102124_object51445:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d77697f:::benchmark_data_smithi159_102124_object1977:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7976d4:::benchmark_data_smithi159_102124_object41576:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7b8064:::benchmark_data_smithi159_102124_object23311:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7c0c6e:::benchmark_data_smithi159_102124_object12469:head# 2024-04-15T18:22:42.458 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7ee665:::benchmark_data_smithi159_102124_object39071:head# 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d7f1939:::benchmark_data_smithi159_102124_object7967:head# 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d802165:::benchmark_data_smithi159_102124_object6193:head# 2024-04-15T18:22:42.459 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.461 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d81c176:::benchmark_data_smithi159_102124_object47539:head# 2024-04-15T18:22:42.462 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d81c23a:::benchmark_data_smithi159_102124_object54186:head# 2024-04-15T18:22:42.463 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.463 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d825975:::benchmark_data_smithi159_102124_object51107:head# 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d836bc0:::benchmark_data_smithi159_102124_object32903:head# 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d859ca6:::benchmark_data_smithi159_102124_object15154:head# 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.464 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d860ffe:::benchmark_data_smithi159_102124_object3087:head# 2024-04-15T18:22:42.465 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.468 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d872812:::benchmark_data_smithi159_102124_object26985:head# 2024-04-15T18:22:42.468 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8762f2:::benchmark_data_smithi159_102124_object21609:head# 2024-04-15T18:22:42.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d878dac:::benchmark_data_smithi159_102124_object50887:head# 2024-04-15T18:22:42.469 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.469 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d88da51:::benchmark_data_smithi159_102124_object36307:head# 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8ab5b5:::benchmark_data_smithi159_102124_object47885:head# 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8b56f8:::benchmark_data_smithi159_102124_object20726:head# 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8d51d8:::benchmark_data_smithi159_102124_object12560:head# 2024-04-15T18:22:42.470 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8d74d4:::benchmark_data_smithi159_102124_object29224:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8eee4a:::benchmark_data_smithi159_102124_object17452:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d8f83fd:::benchmark_data_smithi159_102124_object15623:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d90ebb8:::benchmark_data_smithi159_102124_object9663:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d920ea3:::benchmark_data_smithi159_102124_object32939:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d921af6:::benchmark_data_smithi159_102124_object36129:head# 2024-04-15T18:22:42.471 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d926f4f:::benchmark_data_smithi159_102124_object11470:head# 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d93312b:::benchmark_data_smithi159_102124_object14551:head# 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d93ff7c:::benchmark_data_smithi159_102124_object45354:head# 2024-04-15T18:22:42.472 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9467de:::benchmark_data_smithi159_102124_object23646:head# 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d949e00:::benchmark_data_smithi159_102124_object11436:head# 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d94d7ed:::benchmark_data_smithi159_102124_object37282:head# 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d95058f:::benchmark_data_smithi159_102124_object7006:head# 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.473 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d95479b:::benchmark_data_smithi159_102124_object55742:head# 2024-04-15T18:22:42.474 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.476 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d95915e:::benchmark_data_smithi159_102124_object28110:head# 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d959b83:::benchmark_data_smithi159_102124_object32686:head# 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d95c314:::benchmark_data_smithi159_102124_object21571:head# 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9774d0:::benchmark_data_smithi159_102124_object2103:head# 2024-04-15T18:22:42.477 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d993acc:::benchmark_data_smithi159_102124_object9703:head# 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9ad885:::benchmark_data_smithi159_102124_object7990:head# 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9c56c2:::benchmark_data_smithi159_102124_object21717:head# 2024-04-15T18:22:42.478 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9d31c7:::benchmark_data_smithi159_102124_object28591:head# 2024-04-15T18:22:42.479 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9d9ac4:::benchmark_data_smithi159_102124_object9358:head# 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9e16e8:::benchmark_data_smithi159_102124_object34901:head# 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2d9f06c6:::benchmark_data_smithi159_102124_object55192:head# 2024-04-15T18:22:42.480 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da2d0c5:::benchmark_data_smithi159_102124_object4608:head# 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da31785:::benchmark_data_smithi159_102124_object44805:head# 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da3b979:::benchmark_data_smithi159_102124_object27684:head# 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.481 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da485ad:::benchmark_data_smithi159_102124_object33935:head# 2024-04-15T18:22:42.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da56d74:::benchmark_data_smithi159_102124_object14570:head# 2024-04-15T18:22:42.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.482 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da6a83e:::benchmark_data_smithi159_102124_object31318:head# 2024-04-15T18:22:42.482 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.485 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2da7d452:::benchmark_data_smithi159_102124_object6988:head# 2024-04-15T18:22:42.485 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2daa37f9:::benchmark_data_smithi159_102124_object18740:head# 2024-04-15T18:22:42.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.486 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2daa83e2:::benchmark_data_smithi159_102124_object35022:head# 2024-04-15T18:22:42.486 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2daaf184:::benchmark_data_smithi159_102124_object55597:head# 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dab32b8:::benchmark_data_smithi159_102124_object56203:head# 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dacd597:::benchmark_data_smithi159_102124_object5381:head# 2024-04-15T18:22:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dae74bd:::benchmark_data_smithi159_102124_object38583:head# 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db1d6a3:::benchmark_data_smithi159_102124_object3143:head# 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db258fb:::benchmark_data_smithi159_102124_object3520:head# 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db3453c:::benchmark_data_smithi159_102124_object7382:head# 2024-04-15T18:22:42.488 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.489 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db450dd:::benchmark_data_smithi159_102124_object10191:head# 2024-04-15T18:22:42.489 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db5b1db:::benchmark_data_smithi159_102124_object21668:head# 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db8023c:::benchmark_data_smithi159_102124_object13258:head# 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db87032:::benchmark_data_smithi159_102124_object28317:head# 2024-04-15T18:22:42.490 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db8721e:::benchmark_data_smithi159_102124_object46678:head# 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2db8f90d:::benchmark_data_smithi159_102124_object4738:head# 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbaf307:::benchmark_data_smithi159_102124_object41266:head# 2024-04-15T18:22:42.491 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.492 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbc3bba:::benchmark_data_smithi159_102124_object53211:head# 2024-04-15T18:22:42.492 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.492 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbd8023:::benchmark_data_smithi159_102124_object49510:head# 2024-04-15T18:22:42.492 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbdf2bb:::benchmark_data_smithi159_102124_object1230:head# 2024-04-15T18:22:42.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.493 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbe2d0b:::benchmark_data_smithi159_102124_object16134:head# 2024-04-15T18:22:42.493 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.494 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbee9f0:::benchmark_data_smithi159_102124_object32742:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dbfe2a4:::benchmark_data_smithi159_102124_object22470:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc0a81c:::benchmark_data_smithi159_102124_object14711:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc11cba:::benchmark_data_smithi159_102124_object29300:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc15f5e:::benchmark_data_smithi159_102124_object33690:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc17e07:::benchmark_data_smithi159_102124_object31671:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc1f5bd:::benchmark_data_smithi159_102124_object46635:head# 2024-04-15T18:22:42.495 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc2dc4e:::benchmark_data_smithi159_102124_object33250:head# 2024-04-15T18:22:42.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc4511e:::benchmark_data_smithi159_102124_object47470:head# 2024-04-15T18:22:42.496 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.496 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc4f8c3:::benchmark_data_smithi159_102124_object41865:head# 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc51043:::benchmark_data_smithi159_102124_object29089:head# 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc5f8e7:::benchmark_data_smithi159_102124_object35101:head# 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc60e50:::benchmark_data_smithi159_102124_object1454:head# 2024-04-15T18:22:42.498 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.500 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:42.500 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc61017:::benchmark_data_smithi159_102124_object12021:head# 2024-04-15T18:22:42.500 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.501 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc67838:::benchmark_data_smithi159_102124_object13562:head# 2024-04-15T18:22:42.501 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dc959fa:::benchmark_data_smithi159_102124_object1282:head# 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dca6dad:::benchmark_data_smithi159_102124_object15539:head# 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dca8075:::benchmark_data_smithi159_102124_object6572:head# 2024-04-15T18:22:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dcaaf6a:::benchmark_data_smithi159_102124_object34712:head# 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dcbcf58:::benchmark_data_smithi159_102124_object12247:head# 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dcc0422:::benchmark_data_smithi159_102124_object42689:head# 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.503 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dcd3556:::benchmark_data_smithi159_102124_object15135:head# 2024-04-15T18:22:42.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.504 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dcdcb87:::benchmark_data_smithi159_102124_object24799:head# 2024-04-15T18:22:42.504 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd29f04:::benchmark_data_smithi159_102124_object29829:head# 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd3d463:::benchmark_data_smithi159_102124_object23022:head# 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd4002f:::benchmark_data_smithi159_102124_object35457:head# 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd418db:::benchmark_data_smithi159_102124_object9416:head# 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd4e65e:::benchmark_data_smithi159_102124_object56451:head# 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.512 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd73696:::benchmark_data_smithi159_102124_object42409:head# 2024-04-15T18:22:42.513 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:42.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd7ac97:::benchmark_data_smithi159_102124_object24329:head# 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd7e1a2:::benchmark_data_smithi159_102124_object30724:head# 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd8fad7:::benchmark_data_smithi159_102124_object28565:head# 2024-04-15T18:22:42.516 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd91df1:::benchmark_data_smithi159_102124_object37081:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dd99af3:::benchmark_data_smithi159_102124_object468:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ddc44d7:::benchmark_data_smithi159_102124_object26737:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ddd5f8d:::benchmark_data_smithi159_102124_object48349:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dde027a:::benchmark_data_smithi159_102124_object22616:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dde88c7:::benchmark_data_smithi159_102124_object21742:head# 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.517 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ddfa2a6:::benchmark_data_smithi159_102124_object10711:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de06695:::benchmark_data_smithi159_102124_object32974:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de08087:::benchmark_data_smithi159_102124_object49013:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de0c1a9:::benchmark_data_smithi159_102124_object9109:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de269b6:::benchmark_data_smithi159_102124_object42369:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de2789a:::benchmark_data_smithi159_102124_object38169:head# 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.518 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de3007b:::benchmark_data_smithi159_102124_object44113:head# 2024-04-15T18:22:42.519 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.519 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de496f6:::benchmark_data_smithi159_102124_object26037:head# 2024-04-15T18:22:42.519 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2de59470:::benchmark_data_smithi159_102124_object51222:head# 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2deb92b5:::benchmark_data_smithi159_102124_object2420:head# 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2debb448:::benchmark_data_smithi159_102124_object15206:head# 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dee0d04:::benchmark_data_smithi159_102124_object19616:head# 2024-04-15T18:22:42.522 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dee8ed6:::benchmark_data_smithi159_102124_object31094:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df09334:::benchmark_data_smithi159_102124_object22017:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df0f591:::benchmark_data_smithi159_102124_object14157:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df1c795:::benchmark_data_smithi159_102124_object14316:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df4015c:::benchmark_data_smithi159_102124_object12322:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df42c19:::benchmark_data_smithi159_102124_object25607:head# 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.523 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df4855d:::benchmark_data_smithi159_102124_object51988:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df5b85f:::benchmark_data_smithi159_102124_object43564:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df658d8:::benchmark_data_smithi159_102124_object35284:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df76caa:::benchmark_data_smithi159_102124_object4634:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df88a1f:::benchmark_data_smithi159_102124_object31847:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2df97c22:::benchmark_data_smithi159_102124_object57769:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dfa5589:::benchmark_data_smithi159_102124_object3606:head# 2024-04-15T18:22:42.524 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dfa7e50:::benchmark_data_smithi159_102124_object51111:head# 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dfe4c12:::benchmark_data_smithi159_102124_object52519:head# 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dff26e6:::benchmark_data_smithi159_102124_object6286:head# 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2dffcff6:::benchmark_data_smithi159_102124_object49573:head# 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e008afd:::benchmark_data_smithi159_102124_object32191:head# 2024-04-15T18:22:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.527 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e013e21:::benchmark_data_smithi159_102124_object16263:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e018f67:::benchmark_data_smithi159_102124_object28514:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e01c2d7:::benchmark_data_smithi159_102124_object19147:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e02b32c:::benchmark_data_smithi159_102124_object38434:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e044dda:::benchmark_data_smithi159_102124_object49661:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e04d077:::benchmark_data_smithi159_102124_object34482:head# 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e06bf17:::benchmark_data_smithi159_102124_object35493:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e06f1c2:::benchmark_data_smithi159_102124_object21400:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e081d59:::benchmark_data_smithi159_102124_object25455:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e087b34:::benchmark_data_smithi159_102124_object24150:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e0890cf:::benchmark_data_smithi159_102124_object18194:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e0897ea:::benchmark_data_smithi159_102124_object41186:head# 2024-04-15T18:22:42.529 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e08a5a3:::benchmark_data_smithi159_102124_object17534:head# 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e095326:::benchmark_data_smithi159_102124_object31183:head# 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e0d131f:::benchmark_data_smithi159_102124_object34843:head# 2024-04-15T18:22:42.530 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.531 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e0ed41b:::benchmark_data_smithi159_102124_object18313:head# 2024-04-15T18:22:42.531 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e0ff9ce:::benchmark_data_smithi159_102124_object6136:head# 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e104606:::benchmark_data_smithi159_102124_object27163:head# 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1152d8:::benchmark_data_smithi159_102124_object28253:head# 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1169a7:::benchmark_data_smithi159_102124_object5214:head# 2024-04-15T18:22:42.532 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1274f1:::benchmark_data_smithi159_102124_object9619:head# 2024-04-15T18:22:42.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e12a604:::benchmark_data_smithi159_102124_object40269:head# 2024-04-15T18:22:42.533 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.533 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e133747:::benchmark_data_smithi159_102124_object22615:head# 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e13e599:::benchmark_data_smithi159_102124_object41904:head# 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e14512f:::benchmark_data_smithi159_102124_object35223:head# 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e15ad3a:::benchmark_data_smithi159_102124_object14191:head# 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1661dd:::benchmark_data_smithi159_102124_object56384:head# 2024-04-15T18:22:42.534 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e174470:::benchmark_data_smithi159_102124_object32491:head# 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1819d0:::benchmark_data_smithi159_102124_object38565:head# 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e18ff05:::benchmark_data_smithi159_102124_object728:head# 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e192ee7:::benchmark_data_smithi159_102124_object6243:head# 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.537 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1987cf:::benchmark_data_smithi159_102124_object49021:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1b00a1:::benchmark_data_smithi159_102124_object14621:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1cd124:::benchmark_data_smithi159_102124_object21636:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e1f7664:::benchmark_data_smithi159_102124_object17773:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e200ddf:::benchmark_data_smithi159_102124_object3972:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e22b0e5:::benchmark_data_smithi159_102124_object40559:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e23ce6d:::benchmark_data_smithi159_102124_object14425:head# 2024-04-15T18:22:42.538 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e2546c5:::benchmark_data_smithi159_102124_object50116:head# 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e2600fb:::benchmark_data_smithi159_102124_object58314:head# 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e2780ab:::benchmark_data_smithi159_102124_object48610:head# 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e27831e:::benchmark_data_smithi159_102124_object10953:head# 2024-04-15T18:22:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e27bf86:::benchmark_data_smithi159_102124_object15717:head# 2024-04-15T18:22:42.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.540 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e281ce3:::benchmark_data_smithi159_102124_object8798:head# 2024-04-15T18:22:42.540 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.541 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e287af2:::benchmark_data_smithi159_102124_object32784:head# 2024-04-15T18:22:42.541 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e28c535:::benchmark_data_smithi159_102124_object30358:head# 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e28eac4:::benchmark_data_smithi159_102124_object10835:head# 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e298070:::benchmark_data_smithi159_102124_object314:head# 2024-04-15T18:22:42.542 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e29e447:::benchmark_data_smithi159_102124_object56094:head# 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e2b9a02:::benchmark_data_smithi159_102124_object57690:head# 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e2ceb93:::benchmark_data_smithi159_102124_object30776:head# 2024-04-15T18:22:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e319bea:::benchmark_data_smithi159_102124_object49830:head# 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e334bfe:::benchmark_data_smithi159_102124_object37125:head# 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e33fd92:::benchmark_data_smithi159_102124_object12214:head# 2024-04-15T18:22:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e348006:::benchmark_data_smithi159_102124_object35119:head# 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e34a938:::benchmark_data_smithi159_102124_object23273:head# 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e35ffd7:::benchmark_data_smithi159_102124_object8359:head# 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.545 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e362205:::benchmark_data_smithi159_102124_object47501:head# 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e37ae56:::benchmark_data_smithi159_102124_object32246:head# 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e3a635e:::benchmark_data_smithi159_102124_object48896:head# 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e3c51d8:::benchmark_data_smithi159_102124_object13402:head# 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e3d5e14:::benchmark_data_smithi159_102124_object10592:head# 2024-04-15T18:22:42.546 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.547 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4084bb:::benchmark_data_smithi159_102124_object40148:head# 2024-04-15T18:22:42.547 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.549 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e41c1d6:::benchmark_data_smithi159_102124_object4925:head# 2024-04-15T18:22:42.549 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4256a3:::benchmark_data_smithi159_102124_object12183:head# 2024-04-15T18:22:42.550 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.550 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4296ca:::benchmark_data_smithi159_102124_object2178:head# 2024-04-15T18:22:42.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e431365:::benchmark_data_smithi159_102124_object55395:head# 2024-04-15T18:22:42.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.551 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e43c00c:::benchmark_data_smithi159_102124_object29574:head# 2024-04-15T18:22:42.551 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e465ad4:::benchmark_data_smithi159_102124_object25216:head# 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e468202:::benchmark_data_smithi159_102124_object26904:head# 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e46afa5:::benchmark_data_smithi159_102124_object11695:head# 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e48273c:::benchmark_data_smithi159_102124_object5940:head# 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4928a7:::benchmark_data_smithi159_102124_object40845:head# 2024-04-15T18:22:42.552 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.553 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e49fb2c:::benchmark_data_smithi159_102124_object55380:head# 2024-04-15T18:22:42.553 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4ad857:::benchmark_data_smithi159_102124_object13141:head# 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4bf266:::benchmark_data_smithi159_102124_object32295:head# 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4c6487:::benchmark_data_smithi159_102124_object41567:head# 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4ccbd2:::benchmark_data_smithi159_102124_object52000:head# 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4d3028:::benchmark_data_smithi159_102124_object47024:head# 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.555 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e4e60fb:::benchmark_data_smithi159_102124_object9693:head# 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e5023e3:::benchmark_data_smithi159_102124_object4773:head# 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e50349f:::benchmark_data_smithi159_102124_object18216:head# 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e508e46:::benchmark_data_smithi159_102124_object978:head# 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e512164:::benchmark_data_smithi159_102124_object45603:head# 2024-04-15T18:22:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.557 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e51666b:::benchmark_data_smithi159_102124_object53922:head# 2024-04-15T18:22:42.557 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.558 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e521643:::benchmark_data_smithi159_102124_object15436:head# 2024-04-15T18:22:42.558 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e52fdbc:::benchmark_data_smithi159_102124_object33101:head# 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e536ed7:::benchmark_data_smithi159_102124_object56118:head# 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e54f2d6:::benchmark_data_smithi159_102124_object28487:head# 2024-04-15T18:22:42.559 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e587619:::benchmark_data_smithi159_102124_object54902:head# 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e5d585f:::benchmark_data_smithi159_102124_object38237:head# 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e5d8636:::benchmark_data_smithi159_102124_object8010:head# 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e5fa462:::benchmark_data_smithi159_102124_object26871:head# 2024-04-15T18:22:42.560 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.563 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e602a3a:::benchmark_data_smithi159_102124_object30853:head# 2024-04-15T18:22:42.563 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e60667e:::benchmark_data_smithi159_102124_object262:head# 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e607395:::benchmark_data_smithi159_102124_object37528:head# 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e62dfdb:::benchmark_data_smithi159_102124_object54904:head# 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e631bce:::benchmark_data_smithi159_102124_object19195:head# 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.564 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e63384b:::benchmark_data_smithi159_102124_object16689:head# 2024-04-15T18:22:42.565 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.567 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e65c385:::benchmark_data_smithi159_102124_object53692:head# 2024-04-15T18:22:42.567 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e667c95:::benchmark_data_smithi159_102124_object24497:head# 2024-04-15T18:22:42.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.568 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e67ce03:::benchmark_data_smithi159_102124_object54955:head# 2024-04-15T18:22:42.568 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e681e1d:::benchmark_data_smithi159_102124_object41149:head# 2024-04-15T18:22:42.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.569 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e69e522:::benchmark_data_smithi159_102124_object1502:head# 2024-04-15T18:22:42.569 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.573 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6a66ae:::benchmark_data_smithi159_102124_object39648:head# 2024-04-15T18:22:42.573 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6b3285:::benchmark_data_smithi159_102124_object6761:head# 2024-04-15T18:22:42.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.574 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6dd0d9:::benchmark_data_smithi159_102124_object42131:head# 2024-04-15T18:22:42.574 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.576 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6e4f93:::benchmark_data_smithi159_102124_object33641:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6ed91a:::benchmark_data_smithi159_102124_object1133:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6f8764:::benchmark_data_smithi159_102124_object30284:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e6f9cd5:::benchmark_data_smithi159_102124_object2175:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e701e6a:::benchmark_data_smithi159_102124_object30416:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e70584f:::benchmark_data_smithi159_102124_object4818:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e70bc1a:::benchmark_data_smithi159_102124_object47511:head# 2024-04-15T18:22:42.577 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7287e1:::benchmark_data_smithi159_102124_object11066:head# 2024-04-15T18:22:42.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.578 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e742313:::benchmark_data_smithi159_102124_object19884:head# 2024-04-15T18:22:42.578 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e74ad1d:::benchmark_data_smithi159_102124_object1080:head# 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e74cd95:::benchmark_data_smithi159_102124_object7089:head# 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7528f4:::benchmark_data_smithi159_102124_object27447:head# 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e75ddd6:::benchmark_data_smithi159_102124_object16396:head# 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7810d1:::benchmark_data_smithi159_102124_object23598:head# 2024-04-15T18:22:42.579 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e790579:::benchmark_data_smithi159_102124_object1640:head# 2024-04-15T18:22:42.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.580 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e791612:::benchmark_data_smithi159_102124_object40188:head# 2024-04-15T18:22:42.580 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.589 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7916e8:::benchmark_data_smithi159_102124_object22910:head# 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7a345f:::benchmark_data_smithi159_102124_object57449:head# 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7aa186:::benchmark_data_smithi159_102124_object25248:head# 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7c6552:::benchmark_data_smithi159_102124_object12014:head# 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7cc7b2:::benchmark_data_smithi159_102124_object21109:head# 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7dc9f8:::benchmark_data_smithi159_102124_object4785:head# 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7f0056:::benchmark_data_smithi159_102124_object22674:head# 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e7f1d39:::benchmark_data_smithi159_102124_object44146:head# 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e80d66f:::benchmark_data_smithi159_102124_object27150:head# 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e811815:::benchmark_data_smithi159_102124_object34202:head# 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e840ae7:::benchmark_data_smithi159_102124_object26958:head# 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e845507:::benchmark_data_smithi159_102124_object37665:head# 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e86d54b:::benchmark_data_smithi159_102124_object41512:head# 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e86d9b8:::benchmark_data_smithi159_102124_object37872:head# 2024-04-15T18:22:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e87415a:::benchmark_data_smithi159_102124_object42164:head# 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e88000c:::benchmark_data_smithi159_102124_object32310:head# 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e880bd9:::benchmark_data_smithi159_102124_object14346:head# 2024-04-15T18:22:42.594 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e881561:::benchmark_data_smithi159_102124_object32644:head# 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e8ad852:::benchmark_data_smithi159_102124_object46883:head# 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e8eed32:::benchmark_data_smithi159_102124_object29628:head# 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.595 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e90a558:::benchmark_data_smithi159_102124_object31049:head# 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e925a98:::benchmark_data_smithi159_102124_object14338:head# 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e95513a:::benchmark_data_smithi159_102124_object39518:head# 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9604e3:::benchmark_data_smithi159_102124_object43321:head# 2024-04-15T18:22:42.596 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e971bf9:::benchmark_data_smithi159_102124_object37319:head# 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e97aad2:::benchmark_data_smithi159_102124_object50537:head# 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9a124c:::benchmark_data_smithi159_102124_object14692:head# 2024-04-15T18:22:42.600 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9a1dd4:::benchmark_data_smithi159_102124_object498:head# 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9a2630:::benchmark_data_smithi159_102124_object7722:head# 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9b9935:::benchmark_data_smithi159_102124_object37713:head# 2024-04-15T18:22:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9d4b44:::benchmark_data_smithi159_102124_object19449:head# 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9d4b6c:::benchmark_data_smithi159_102124_object7694:head# 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9db2e6:::benchmark_data_smithi159_102124_object47549:head# 2024-04-15T18:22:42.602 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2e9dc99a:::benchmark_data_smithi159_102124_object46132:head# 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea1f955:::benchmark_data_smithi159_102124_object36437:head# 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea23c83:::benchmark_data_smithi159_102124_object22732:head# 2024-04-15T18:22:42.603 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea45c2b:::benchmark_data_smithi159_102124_object43013:head# 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea52ff9:::benchmark_data_smithi159_102124_object46079:head# 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea6a90b:::benchmark_data_smithi159_102124_object54756:head# 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.604 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea72237:::benchmark_data_smithi159_102124_object5674:head# 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea842ed:::benchmark_data_smithi159_102124_object11607:head# 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea850b8:::benchmark_data_smithi159_102124_object31434:head# 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ea9d5dc:::benchmark_data_smithi159_102124_object25093:head# 2024-04-15T18:22:42.605 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eaaf3b5:::benchmark_data_smithi159_102124_object42778:head# 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eadeb52:::benchmark_data_smithi159_102124_object46734:head# 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb172c8:::benchmark_data_smithi159_102124_object53349:head# 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.606 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb27101:::benchmark_data_smithi159_102124_object50006:head# 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb2a2f3:::benchmark_data_smithi159_102124_object46255:head# 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb4c34a:::benchmark_data_smithi159_102124_object23209:head# 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb7820d:::benchmark_data_smithi159_102124_object26915:head# 2024-04-15T18:22:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:42.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb92089:::benchmark_data_smithi159_102124_object29830:head# 2024-04-15T18:22:42.625 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.625 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb9a71a:::benchmark_data_smithi159_102124_object9054:head# 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eb9bfb7:::benchmark_data_smithi159_102124_object6846:head# 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eba65bc:::benchmark_data_smithi159_102124_object42102:head# 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ebc42df:::benchmark_data_smithi159_102124_object45811:head# 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.626 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ebcafa4:::benchmark_data_smithi159_102124_object37247:head# 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ebd6396:::benchmark_data_smithi159_102124_object42887:head# 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ebeea2e:::benchmark_data_smithi159_102124_object44776:head# 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec46b69:::benchmark_data_smithi159_102124_object34746:head# 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.627 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec4d846:::benchmark_data_smithi159_102124_object50721:head# 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec50562:::benchmark_data_smithi159_102124_object52719:head# 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec5f8a4:::benchmark_data_smithi159_102124_object52538:head# 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec68ab5:::benchmark_data_smithi159_102124_object40783:head# 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.628 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec749fa:::benchmark_data_smithi159_102124_object8747:head# 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ec9e0da:::benchmark_data_smithi159_102124_object28783:head# 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eca7b01:::benchmark_data_smithi159_102124_object1209:head# 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ecf4dd3:::benchmark_data_smithi159_102124_object22556:head# 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.629 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ecf8329:::benchmark_data_smithi159_102124_object13095:head# 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed02566:::benchmark_data_smithi159_102124_object28558:head# 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed0724a:::benchmark_data_smithi159_102124_object42839:head# 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed0cbe9:::benchmark_data_smithi159_102124_object38469:head# 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.630 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed46e75:::benchmark_data_smithi159_102124_object14899:head# 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed5681f:::benchmark_data_smithi159_102124_object35147:head# 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed61ded:::benchmark_data_smithi159_102124_object11189:head# 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ed652e5:::benchmark_data_smithi159_102124_object44426:head# 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.631 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2edb2510:::benchmark_data_smithi159_102124_object16150:head# 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2edb2704:::benchmark_data_smithi159_102124_object27505:head# 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2edb589f:::benchmark_data_smithi159_102124_object26079:head# 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee0bb3e:::benchmark_data_smithi159_102124_object21271:head# 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee1366a:::benchmark_data_smithi159_102124_object12998:head# 2024-04-15T18:22:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee2b00d:::benchmark_data_smithi159_102124_object35940:head# 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee3c7a0:::benchmark_data_smithi159_102124_object25368:head# 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee51eeb:::benchmark_data_smithi159_102124_object6287:head# 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee8ee11:::benchmark_data_smithi159_102124_object35080:head# 2024-04-15T18:22:42.633 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ee97dfe:::benchmark_data_smithi159_102124_object40484:head# 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eea458b:::benchmark_data_smithi159_102124_object55629:head# 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eea45e4:::benchmark_data_smithi159_102124_object38615:head# 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eeab11d:::benchmark_data_smithi159_102124_object3748:head# 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eeb23a9:::benchmark_data_smithi159_102124_object48815:head# 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eeb8df5:::benchmark_data_smithi159_102124_object15009:head# 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eec0a1a:::benchmark_data_smithi159_102124_object46988:head# 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.635 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eec9297:::benchmark_data_smithi159_102124_object1482:head# 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eecc21b:::benchmark_data_smithi159_102124_object35466:head# 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eedc7c7:::benchmark_data_smithi159_102124_object11848:head# 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eee1fa7:::benchmark_data_smithi159_102124_object21589:head# 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.636 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eee3ac5:::benchmark_data_smithi159_102124_object58021:head# 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eeeda35:::benchmark_data_smithi159_102124_object15133:head# 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef11938:::benchmark_data_smithi159_102124_object11049:head# 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef27316:::benchmark_data_smithi159_102124_object17377:head# 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef37237:::benchmark_data_smithi159_102124_object34832:head# 2024-04-15T18:22:42.637 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef409f9:::benchmark_data_smithi159_102124_object5830:head# 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef4308c:::benchmark_data_smithi159_102124_object7594:head# 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef5b993:::benchmark_data_smithi159_102124_object54947:head# 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef66a14:::benchmark_data_smithi159_102124_object20483:head# 2024-04-15T18:22:42.638 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef74f72:::benchmark_data_smithi159_102124_object1094:head# 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef7b2e4:::benchmark_data_smithi159_102124_object36384:head# 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ef7ebc6:::benchmark_data_smithi159_102124_object54429:head# 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efa10a7:::benchmark_data_smithi159_102124_object29757:head# 2024-04-15T18:22:42.639 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efb2570:::benchmark_data_smithi159_102124_object54593:head# 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efbd016:::benchmark_data_smithi159_102124_object55387:head# 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efc108f:::benchmark_data_smithi159_102124_object45953:head# 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efe7e30:::benchmark_data_smithi159_102124_object39537:head# 2024-04-15T18:22:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2efeaf58:::benchmark_data_smithi159_102124_object22426:head# 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2eff3992:::benchmark_data_smithi159_102124_object57224:head# 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2effd6c4:::benchmark_data_smithi159_102124_object20015:head# 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0278f1:::benchmark_data_smithi159_102124_object5233:head# 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f02efe7:::benchmark_data_smithi159_102124_object37541:head# 2024-04-15T18:22:42.641 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f033268:::benchmark_data_smithi159_102124_object32486:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f038c43:::benchmark_data_smithi159_102124_object43978:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0836b5:::benchmark_data_smithi159_102124_object38821:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f08c4b5:::benchmark_data_smithi159_102124_object22862:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0b2926:::benchmark_data_smithi159_102124_object9584:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0b8706:::benchmark_data_smithi159_102124_object13780:head# 2024-04-15T18:22:42.642 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0b9988:::benchmark_data_smithi159_102124_object20905:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0c2f44:::benchmark_data_smithi159_102124_object26192:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0f0c3c:::benchmark_data_smithi159_102124_object5351:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0f4a61:::benchmark_data_smithi159_102124_object30548:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f0faf79:::benchmark_data_smithi159_102124_object22078:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f11df07:::benchmark_data_smithi159_102124_object25156:head# 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.643 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f129fd5:::benchmark_data_smithi159_102124_object11850:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f13f2ac:::benchmark_data_smithi159_102124_object22471:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f142a3c:::benchmark_data_smithi159_102124_object34591:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f14ece7:::benchmark_data_smithi159_102124_object39106:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f15bee1:::benchmark_data_smithi159_102124_object48030:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f160b0b:::benchmark_data_smithi159_102124_object52793:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f17d1da:::benchmark_data_smithi159_102124_object31197:head# 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.644 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f19a9a9:::benchmark_data_smithi159_102124_object48498:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f1dbf82:::benchmark_data_smithi159_102124_object42582:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f1ddd90:::benchmark_data_smithi159_102124_object50501:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f1dfde1:::benchmark_data_smithi159_102124_object17768:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f1fa06b:::benchmark_data_smithi159_102124_object41817:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f20a258:::benchmark_data_smithi159_102124_object10041:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f218eed:::benchmark_data_smithi159_102124_object28824:head# 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f23ff26:::benchmark_data_smithi159_102124_object18300:head# 2024-04-15T18:22:42.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.646 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f244efa:::benchmark_data_smithi159_102124_object57759:head# 2024-04-15T18:22:42.646 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: pgmap v904: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 5033/199545 objects degraded (2.522%); 5.1 MiB/s, 81 objects/s recovering 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5033/199545 objects degraded (2.522%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:22:42.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:22:42.653 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:22:42.653 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:22:42.653 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:42.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: pgmap v904: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 5033/199545 objects degraded (2.522%); 5.1 MiB/s, 81 objects/s recovering 2024-04-15T18:22:42.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5033/199545 objects degraded (2.522%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:42.658 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:22:42.659 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f2660d7:::benchmark_data_smithi159_102124_object57695:head# 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f270fae:::benchmark_data_smithi159_102124_object31296:head# 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f293efc:::benchmark_data_smithi159_102124_object1403:head# 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.665 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f299c39:::benchmark_data_smithi159_102124_object31868:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f2c3556:::benchmark_data_smithi159_102124_object15052:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f2cd0c4:::benchmark_data_smithi159_102124_object31547:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f2e1808:::benchmark_data_smithi159_102124_object17387:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f2f2526:::benchmark_data_smithi159_102124_object17201:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f30274b:::benchmark_data_smithi159_102124_object10275:head# 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.666 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f31099f:::benchmark_data_smithi159_102124_object18041:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f312ba4:::benchmark_data_smithi159_102124_object34514:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f362b04:::benchmark_data_smithi159_102124_object2078:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f364bdb:::benchmark_data_smithi159_102124_object41717:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f36669c:::benchmark_data_smithi159_102124_object16878:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f36d0b0:::benchmark_data_smithi159_102124_object27941:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f372955:::benchmark_data_smithi159_102124_object25958:head# 2024-04-15T18:22:42.667 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f37418f:::benchmark_data_smithi159_102124_object8492:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3805ef:::benchmark_data_smithi159_102124_object24604:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f383704:::benchmark_data_smithi159_102124_object2703:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f39ce02:::benchmark_data_smithi159_102124_object8841:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3a8691:::benchmark_data_smithi159_102124_object39787:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3b73de:::benchmark_data_smithi159_102124_object33571:head# 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3c46b4:::benchmark_data_smithi159_102124_object26174:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3de9c1:::benchmark_data_smithi159_102124_object12261:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3e2443:::benchmark_data_smithi159_102124_object42456:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f3e85f5:::benchmark_data_smithi159_102124_object33931:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f415768:::benchmark_data_smithi159_102124_object23837:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4360d9:::benchmark_data_smithi159_102124_object10705:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f43a0af:::benchmark_data_smithi159_102124_object49601:head# 2024-04-15T18:22:42.669 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f44da17:::benchmark_data_smithi159_102124_object12435:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4540c4:::benchmark_data_smithi159_102124_object16720:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4726b0:::benchmark_data_smithi159_102124_object22341:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f478d6b:::benchmark_data_smithi159_102124_object41076:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4a8fe9:::benchmark_data_smithi159_102124_object19897:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4ac296:::benchmark_data_smithi159_102124_object16764:head# 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.670 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4bf954:::benchmark_data_smithi159_102124_object42210:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4d1eba:::benchmark_data_smithi159_102124_object49958:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f4e77e1:::benchmark_data_smithi159_102124_object5222:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f51ff1a:::benchmark_data_smithi159_102124_object8618:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f52160b:::benchmark_data_smithi159_102124_object10303:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f527dca:::benchmark_data_smithi159_102124_object879:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f52e432:::benchmark_data_smithi159_102124_object28798:head# 2024-04-15T18:22:42.671 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f5421c7:::benchmark_data_smithi159_102124_object48121:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f54d7fa:::benchmark_data_smithi159_102124_object37864:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f55addd:::benchmark_data_smithi159_102124_object29789:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f56277a:::benchmark_data_smithi159_102124_object14904:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f56f92e:::benchmark_data_smithi159_102124_object43340:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f5e9b72:::benchmark_data_smithi159_102124_object30929:head# 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.672 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f60755a:::benchmark_data_smithi159_102124_object50557:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f611364:::benchmark_data_smithi159_102124_object10448:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f623434:::benchmark_data_smithi159_102124_object52150:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f62dc0a:::benchmark_data_smithi159_102124_object6636:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f63c681:::benchmark_data_smithi159_102124_object24810:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f649c73:::benchmark_data_smithi159_102124_object9843:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f68794b:::benchmark_data_smithi159_102124_object23139:head# 2024-04-15T18:22:42.673 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f69594d:::benchmark_data_smithi159_102124_object24203:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f6b46c5:::benchmark_data_smithi159_102124_object1490:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f6d2d46:::benchmark_data_smithi159_102124_object56829:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f6d81c4:::benchmark_data_smithi159_102124_object40755:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f6d9333:::benchmark_data_smithi159_102124_object7454:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f6fab1a:::benchmark_data_smithi159_102124_object44979:head# 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.674 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f70124d:::benchmark_data_smithi159_102124_object43663:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f70dc19:::benchmark_data_smithi159_102124_object8646:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f720a96:::benchmark_data_smithi159_102124_object36684:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f724ce3:::benchmark_data_smithi159_102124_object23857:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f73bbb0:::benchmark_data_smithi159_102124_object22794:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f74509f:::benchmark_data_smithi159_102124_object11550:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f74e5ec:::benchmark_data_smithi159_102124_object37504:head# 2024-04-15T18:22:42.675 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f764916:::benchmark_data_smithi159_102124_object45374:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f78b306:::benchmark_data_smithi159_102124_object20839:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f7a5188:::benchmark_data_smithi159_102124_object41092:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f7b8b40:::benchmark_data_smithi159_102124_object57458:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f7dfb89:::benchmark_data_smithi159_102124_object10024:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f7e29d6:::benchmark_data_smithi159_102124_object35196:head# 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.676 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f7f7c59:::benchmark_data_smithi159_102124_object14580:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f807b36:::benchmark_data_smithi159_102124_object56523:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f809533:::benchmark_data_smithi159_102124_object45967:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f837f50:::benchmark_data_smithi159_102124_object12564:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f83ee5b:::benchmark_data_smithi159_102124_object31665:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f859373:::benchmark_data_smithi159_102124_object26054:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f86fa33:::benchmark_data_smithi159_102124_object17942:head# 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.677 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f874b7e:::benchmark_data_smithi159_102124_object38808:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8ae2e1:::benchmark_data_smithi159_102124_object22865:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8b3f66:::benchmark_data_smithi159_102124_object12784:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8bf5b7:::benchmark_data_smithi159_102124_object3045:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8f3aed:::benchmark_data_smithi159_102124_object56455:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8f953c:::benchmark_data_smithi159_102124_object47619:head# 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.678 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f8fd23d:::benchmark_data_smithi159_102124_object35857:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f9344a8:::benchmark_data_smithi159_102124_object15496:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f970dd0:::benchmark_data_smithi159_102124_object28554:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f97c770:::benchmark_data_smithi159_102124_object45294:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f98fb3d:::benchmark_data_smithi159_102124_object52544:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f99e5f6:::benchmark_data_smithi159_102124_object28140:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f99e70c:::benchmark_data_smithi159_102124_object54793:head# 2024-04-15T18:22:42.679 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f9cda81:::benchmark_data_smithi159_102124_object35930:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f9dc702:::benchmark_data_smithi159_102124_object54967:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2f9ff58a:::benchmark_data_smithi159_102124_object36527:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa22310:::benchmark_data_smithi159_102124_object20936:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa39c67:::benchmark_data_smithi159_102124_object41040:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa3a4b5:::benchmark_data_smithi159_102124_object8624:head# 2024-04-15T18:22:42.680 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa3c01c:::benchmark_data_smithi159_102124_object17857:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa5312a:::benchmark_data_smithi159_102124_object26942:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa5b8a8:::benchmark_data_smithi159_102124_object48363:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa60005:::benchmark_data_smithi159_102124_object34726:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa63d37:::benchmark_data_smithi159_102124_object19265:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa7d4ba:::benchmark_data_smithi159_102124_object23527:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fa9318c:::benchmark_data_smithi159_102124_object56245:head# 2024-04-15T18:22:42.681 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2faae820:::benchmark_data_smithi159_102124_object18493:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fab6849:::benchmark_data_smithi159_102124_object7287:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fabc312:::benchmark_data_smithi159_102124_object14542:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fac3e90:::benchmark_data_smithi159_102124_object24084:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fac512b:::benchmark_data_smithi159_102124_object1772:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fad57cb:::benchmark_data_smithi159_102124_object5497:head# 2024-04-15T18:22:42.682 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fae636a:::benchmark_data_smithi159_102124_object11892:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2faebfce:::benchmark_data_smithi159_102124_object22228:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2faf8d42:::benchmark_data_smithi159_102124_object29064:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb12cab:::benchmark_data_smithi159_102124_object25044:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb184d8:::benchmark_data_smithi159_102124_object56917:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb1b0f0:::benchmark_data_smithi159_102124_object47187:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb2012d:::benchmark_data_smithi159_102124_object778:head# 2024-04-15T18:22:42.683 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb23f89:::benchmark_data_smithi159_102124_object53587:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb2733e:::benchmark_data_smithi159_102124_object22184:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb4f65b:::benchmark_data_smithi159_102124_object51429:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb69160:::benchmark_data_smithi159_102124_object41740:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb70ca1:::benchmark_data_smithi159_102124_object22903:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fb90a25:::benchmark_data_smithi159_102124_object13657:head# 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.684 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fba481d:::benchmark_data_smithi159_102124_object37217:head# 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fba876f:::benchmark_data_smithi159_102124_object57486:head# 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fbdabf9:::benchmark_data_smithi159_102124_object32678:head# 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fbe9342:::benchmark_data_smithi159_102124_object37441:head# 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc22832:::benchmark_data_smithi159_102124_object30470:head# 2024-04-15T18:22:42.685 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:42.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:42.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc2596a:::benchmark_data_smithi159_102124_object2118:head# 2024-04-15T18:22:42.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.691 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc2c7d9:::benchmark_data_smithi159_102124_object35154:head# 2024-04-15T18:22:42.691 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc30b7a:::benchmark_data_smithi159_102124_object21811:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc33bca:::benchmark_data_smithi159_102124_object33372:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc35d99:::benchmark_data_smithi159_102124_object9738:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc5cb3c:::benchmark_data_smithi159_102124_object37443:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fc67def:::benchmark_data_smithi159_102124_object20300:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fcacdef:::benchmark_data_smithi159_102124_object54339:head# 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.692 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fcb0120:::benchmark_data_smithi159_102124_object6578:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fcb14a9:::benchmark_data_smithi159_102124_object35603:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fcec6a3:::benchmark_data_smithi159_102124_object41257:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd18ab9:::benchmark_data_smithi159_102124_object1538:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd19f1b:::benchmark_data_smithi159_102124_object31109:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd23f6f:::benchmark_data_smithi159_102124_object1075:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd4ca13:::benchmark_data_smithi159_102124_object55577:head# 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.693 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd84bdd:::benchmark_data_smithi159_102124_object12459:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fd9892f:::benchmark_data_smithi159_102124_object41984:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fda3ed7:::benchmark_data_smithi159_102124_object17255:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fdb1365:::benchmark_data_smithi159_102124_object5364:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fdb63bd:::benchmark_data_smithi159_102124_object7666:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fdfca84:::benchmark_data_smithi159_102124_object40776:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fe2955f:::benchmark_data_smithi159_102124_object32740:head# 2024-04-15T18:22:42.694 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fe61221:::benchmark_data_smithi159_102124_object16668:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fe701fc:::benchmark_data_smithi159_102124_object55678:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fe7486e:::benchmark_data_smithi159_102124_object40035:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fe77adc:::benchmark_data_smithi159_102124_object22746:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fec0fce:::benchmark_data_smithi159_102124_object32817:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fec747d:::benchmark_data_smithi159_102124_object56588:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fed04b0:::benchmark_data_smithi159_102124_object31775:head# 2024-04-15T18:22:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fedcd96:::benchmark_data_smithi159_102124_object37925:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fee11ce:::benchmark_data_smithi159_102124_object48463:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2feee8a2:::benchmark_data_smithi159_102124_object51159:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fef5fd0:::benchmark_data_smithi159_102124_object25057:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fef78ec:::benchmark_data_smithi159_102124_object54257:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fefec6c:::benchmark_data_smithi159_102124_object18559:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff07bb9:::benchmark_data_smithi159_102124_object46438:head# 2024-04-15T18:22:42.696 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff2605f:::benchmark_data_smithi159_102124_object55334:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff344c0:::benchmark_data_smithi159_102124_object47333:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff3b058:::benchmark_data_smithi159_102124_object47290:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff4110c:::benchmark_data_smithi159_102124_object48579:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff645a3:::benchmark_data_smithi159_102124_object7148:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff674b0:::benchmark_data_smithi159_102124_object8429:head# 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.697 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff69e15:::benchmark_data_smithi159_102124_object43153:head# 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff6bbf8:::benchmark_data_smithi159_102124_object50223:head# 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff71c7e:::benchmark_data_smithi159_102124_object43717:head# 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ff7bf3f:::benchmark_data_smithi159_102124_object246:head# 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ffa18e6:::benchmark_data_smithi159_102124_object26787:head# 2024-04-15T18:22:42.698 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ffb6205:::benchmark_data_smithi159_102124_object32885:head# 2024-04-15T18:22:42.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ffc567c:::benchmark_data_smithi159_102124_object46423:head# 2024-04-15T18:22:42.699 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.699 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2ffd0a2d:::benchmark_data_smithi159_102124_object4809:head# 2024-04-15T18:22:42.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.700 INFO:teuthology.orchestra.run.smithi049.stdout:Write #4:2fff5f73:::benchmark_data_smithi159_102124_object14040:head# 2024-04-15T18:22:42.700 INFO:teuthology.orchestra.run.smithi049.stdout:snapset 0=[]:{} 2024-04-15T18:22:42.700 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 307 info 4.4( v 307'3739 (0'0,307'3739] local-lis/les=284/285 n=3739 ec=284/284 lis/c=284/284 les/c/f=285/285/0 sis=284) 2024-04-15T18:22:42.726 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-04-15T18:22:42.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:42.838 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:42.928 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:42.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:43.032 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:43.081 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:43.108 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:43.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:43.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:43.247 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:43.273 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:43.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:43.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:43.415 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:43.443 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:43.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:43.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:43.579 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:43.601 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:43.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:43.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:43.758 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.146 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:44.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:44.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/exp.4.4.2 2024-04-15T18:22:44.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:44.290 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.303 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-15T18:22:44.303 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:22:44.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:22:44.312 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:44.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:44.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:44.455 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:44 smithi049 ceph-mon[40087]: pgmap v905: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 4424/191673 objects degraded (2.308%); 9.9 MiB/s, 157 objects/s recovering 2024-04-15T18:22:44.477 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:44.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:44.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:44.610 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:44 smithi082 ceph-mon[44206]: pgmap v905: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 4424/191673 objects degraded (2.308%); 9.9 MiB/s, 157 objects/s recovering 2024-04-15T18:22:44.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.635 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:44.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:44 smithi159 ceph-mon[45336]: pgmap v905: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 958 op/s; 4424/191673 objects degraded (2.308%); 9.9 MiB/s, 157 objects/s recovering 2024-04-15T18:22:44.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:44.779 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.809 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:44.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:44.921 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:44.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:22:44.970 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:44.999 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:45.003 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:22:45.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:22:45.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:22:45.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:45.121 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:22:45.141 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:45.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:22:45.165 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:45.186 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:22:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:45.223 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:22:45.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:22:45.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:45.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:22:45.305 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:22:45.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:22:45.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:22:45.338 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:45.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:45.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:45.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3159230901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:45.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.444 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:45.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:45.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:45.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3159230901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:45.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.480 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:45.513 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:45.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:45.659 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:45.684 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:45.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:45.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:45.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3159230901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:45.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:45.790 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:45.826 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:45.852 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:45.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:45.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:45.997 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: pgmap v906: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 4424/188454 objects degraded (2.348%); 7.8 MiB/s, 124 objects/s recovering 2024-04-15T18:22:46.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:46 smithi049 ceph-mon[40087]: osdmap e329: 12 total, 11 up, 11 in 2024-04-15T18:22:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:22:46.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:22:46.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: pgmap v906: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 4424/188454 objects degraded (2.348%); 7.8 MiB/s, 124 objects/s recovering 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:46 smithi159 ceph-mon[45336]: osdmap e329: 12 total, 11 up, 11 in 2024-04-15T18:22:46.653 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:46.655 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:46.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:46.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:46.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: pgmap v906: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 4424/188454 objects degraded (2.348%); 7.8 MiB/s, 124 objects/s recovering 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:46.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:46 smithi082 ceph-mon[44206]: osdmap e329: 12 total, 11 up, 11 in 2024-04-15T18:22:46.789 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:46.813 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:46.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:46.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:46.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:46.984 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:46.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:47.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:47.127 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:47.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 4424/188454 objects degraded (2.348%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:47.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 4424/188454 objects degraded (2.348%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:47.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 4424/188454 objects degraded (2.348%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:47.887 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:47.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:47.994 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:48.023 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:48.063 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:22:48.063 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:22:48.163 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:48.192 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.223 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:48.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:48.367 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.397 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:48.503 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:22:48.532 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.563 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:22:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:22:48.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:22:48.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:48 smithi159 ceph-mon[45336]: pgmap v908: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 855 op/s; 4122/183966 objects degraded (2.241%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:48.668 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:48.703 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:48 smithi049 ceph-mon[40087]: pgmap v908: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 855 op/s; 4122/183966 objects degraded (2.241%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:48.731 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:48 smithi082 ceph-mon[44206]: pgmap v908: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 855 op/s; 4122/183966 objects degraded (2.241%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:22:48.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:48.873 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:48.901 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:48.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:49.008 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:49.050 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:49.090 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:49.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:49.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:49.229 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:49.266 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:49.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:49.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:49.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:22:49.439 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:49.441 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:22:49.716 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:49.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:22:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:49.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:49.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:22:49.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:49.856 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:50.066 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:50.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:50.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:22:50.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:50.216 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:50.354 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:50.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:50.403 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:50 smithi082 ceph-mon[44206]: pgmap v909: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 3707/179349 objects degraded (2.067%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:22:50.404 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:50 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:50.404 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:50.404 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:50 smithi082 ceph-mon[44206]: osdmap e330: 12 total, 11 up, 11 in 2024-04-15T18:22:50.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:22:50.502 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:50.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:50 smithi049 ceph-mon[40087]: pgmap v909: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 3707/179349 objects degraded (2.067%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:22:50.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:50 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:50.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:50.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:50 smithi049 ceph-mon[40087]: osdmap e330: 12 total, 11 up, 11 in 2024-04-15T18:22:50.758 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:50.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:50.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:50.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:50 smithi159 ceph-mon[45336]: pgmap v909: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 3707/179349 objects degraded (2.067%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:22:50.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:50 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:22:50.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3809093841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:22:50.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:50 smithi159 ceph-mon[45336]: osdmap e330: 12 total, 11 up, 11 in 2024-04-15T18:22:50.893 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:50.914 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:50.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:51.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:51.043 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:22:51.047 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:51.051 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:22:51.061 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:51.067 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:51.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:51.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:51.208 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:51.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:51 smithi049 ceph-mon[40087]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:51.563 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:51.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:51.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:22:51.701 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:51 smithi159 ceph-mon[45336]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:51.704 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:51.735 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:22:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:51.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:22:51.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:51 smithi082 ceph-mon[44206]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:51.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:51.876 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:52.083 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:52.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:52.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:52.213 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":910,"stamp":"2024-04-15T18:22:49.412429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3918332472,"num_objects":59783,"num_object_clones":0,"num_object_copies":179349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59783,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":119104,"num_write_kb":5721403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5970,"num_bytes_recovered":393239200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121235,"ondisk_log_size":121235,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":14108156,"kb_used_data":11635752,"kb_used_omap":60,"kb_used_meta":2471619,"kb_avail":1017088516,"statfs":{"total":1055945392128,"available":1041498640384,"internally_reserved":0,"allocated":11915010048,"data_stored":11912096827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61481,"internal_metadata":2530938839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,5,1,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-441188352,"num_objects":-6732,"num_object_clones":0,"num_object_copies":-20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6732,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":86900736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.172448"},"pg_stats":[{"pgid":"4.f","version":"329'7512","reported_seq":7653,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.101138+0000","last_change":"2024-04-15T18:21:32.333996+0000","last_active":"2024-04-15T18:22:47.101138+0000","last_peered":"2024-04-15T18:22:47.101138+0000","last_clean":"2024-04-15T18:22:47.101138+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:22:47.101138+0000","last_undegraded":"2024-04-15T18:22:47.101138+0000","last_fullsized":"2024-04-15T18:22:47.101138+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'3320","last_scrub_stamp":"2024-04-15T18:21:32.333909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:21:32.333909+0000","objects_scrubbed":3301,"log_size":7512,"log_dups_size":0,"ondisk_log_size":7512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247332864,"num_objects":3774,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7512,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"329'7265","reported_seq":7422,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.533625+0000","last_change":"2024-04-15T18:22:03.765164+0000","last_active":"2024-04-15T18:22:47.533625+0000","last_peered":"2024-04-15T18:22:47.533625+0000","last_clean":"2024-04-15T18:22:47.533625+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:22:47.533625+0000","last_undegraded":"2024-04-15T18:22:47.533625+0000","last_fullsized":"2024-04-15T18:22:47.533625+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5045","last_scrub_stamp":"2024-04-15T18:22:03.764952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.764952+0000","objects_scrubbed":5010,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:14.312914+0000","scrub_duration":1265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":349440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"329'7367","reported_seq":7531,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.487839+0000","last_change":"2024-04-15T18:22:11.794263+0000","last_active":"2024-04-15T18:22:47.487839+0000","last_peered":"2024-04-15T18:22:47.487839+0000","last_clean":"2024-04-15T18:22:47.487839+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:22:47.487839+0000","last_undegraded":"2024-04-15T18:22:47.487839+0000","last_fullsized":"2024-04-15T18:22:47.487839+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5531","last_scrub_stamp":"2024-04-15T18:22:11.794127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.794127+0000","objects_scrubbed":5516,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:35.007436+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242221080,"num_objects":3697,"num_object_clones":0,"num_object_copies":11091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7367,"num_write_kb":353985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"329'7541","reported_seq":7699,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.574498+0000","last_change":"2024-04-15T18:22:00.355031+0000","last_active":"2024-04-15T18:22:47.574498+0000","last_peered":"2024-04-15T18:22:47.574498+0000","last_clean":"2024-04-15T18:22:47.574498+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:22:47.574498+0000","last_undegraded":"2024-04-15T18:22:47.574498+0000","last_fullsized":"2024-04-15T18:22:47.574498+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'5063","last_scrub_stamp":"2024-04-15T18:22:00.354917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:00.354917+0000","objects_scrubbed":5039,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:07.368746+0000","scrub_duration":791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":11319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7541,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"329'7424","reported_seq":7585,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:46.279075+0000","last_change":"2024-04-15T18:22:05.829296+0000","last_active":"2024-04-15T18:22:46.279075+0000","last_peered":"2024-04-15T18:22:46.279075+0000","last_clean":"2024-04-15T18:22:46.279075+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:22:46.279075+0000","last_undegraded":"2024-04-15T18:22:46.279075+0000","last_fullsized":"2024-04-15T18:22:46.279075+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5237","last_scrub_stamp":"2024-04-15T18:22:05.829122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.829122+0000","objects_scrubbed":5187,"log_size":7424,"log_dups_size":0,"ondisk_log_size":7424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.283179+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240123904,"num_objects":3664,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"328'7427","reported_seq":7587,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:45.210981+0000","last_change":"2024-04-15T18:22:05.753884+0000","last_active":"2024-04-15T18:22:45.210981+0000","last_peered":"2024-04-15T18:22:45.210981+0000","last_clean":"2024-04-15T18:22:45.210981+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:22:45.210981+0000","last_undegraded":"2024-04-15T18:22:45.210981+0000","last_fullsized":"2024-04-15T18:22:45.210981+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5298","last_scrub_stamp":"2024-04-15T18:22:05.753792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.753792+0000","objects_scrubbed":5262,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.075803+0000","scrub_duration":1367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253165568,"num_objects":3863,"num_object_clones":0,"num_object_copies":11589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"328'7221","reported_seq":7378,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:44.838162+0000","last_change":"2024-04-15T18:22:23.627521+0000","last_active":"2024-04-15T18:22:44.838162+0000","last_peered":"2024-04-15T18:22:44.838162+0000","last_clean":"2024-04-15T18:22:44.838162+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:22:44.838162+0000","last_undegraded":"2024-04-15T18:22:44.838162+0000","last_fullsized":"2024-04-15T18:22:44.838162+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'6038","last_scrub_stamp":"2024-04-15T18:22:23.627256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:23.627256+0000","objects_scrubbed":4987,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:31.345919+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"328'7302","reported_seq":7462,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:44.845227+0000","last_change":"2024-04-15T18:22:11.496215+0000","last_active":"2024-04-15T18:22:44.845227+0000","last_peered":"2024-04-15T18:22:44.845227+0000","last_clean":"2024-04-15T18:22:44.845227+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:22:44.845227+0000","last_undegraded":"2024-04-15T18:22:44.845227+0000","last_fullsized":"2024-04-15T18:22:44.845227+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5535","last_scrub_stamp":"2024-04-15T18:22:11.496110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.496110+0000","objects_scrubbed":5505,"log_size":7302,"log_dups_size":0,"ondisk_log_size":7302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.883032+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246939648,"num_objects":3768,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7302,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"329'7327","reported_seq":7425,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.136199+0000","last_change":"2024-04-15T18:20:31.649944+0000","last_active":"2024-04-15T18:22:47.136199+0000","last_peered":"2024-04-15T18:22:47.136199+0000","last_clean":"2024-04-15T18:22:47.136199+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:22:47.136199+0000","last_undegraded":"2024-04-15T18:22:47.136199+0000","last_fullsized":"2024-04-15T18:22:47.136199+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7327,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"329'7412","reported_seq":14128,"reported_epoch":329,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:22:47.412723+0000","last_change":"2024-04-15T18:21:57.693302+0000","last_active":"2024-04-15T18:22:47.412723+0000","last_peered":"2024-04-15T18:22:47.412723+0000","last_clean":"2024-04-15T18:21:37.132207+0000","last_became_active":"2024-04-15T18:21:57.661742+0000","last_became_peered":"2024-04-15T18:21:57.661742+0000","last_unstale":"2024-04-15T18:22:47.412723+0000","last_undegraded":"2024-04-15T18:21:57.238321+0000","last_fullsized":"2024-04-15T18:21:57.218020+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3306,"num_bytes_recovered":216662016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1],"avail_no_missing":["9","1"],"object_location_counts":[{"shards":"1,9","objects":2198}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"329'7605","reported_seq":13021,"reported_epoch":329,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:22:47.117373+0000","last_change":"2024-04-15T18:21:57.762236+0000","last_active":"2024-04-15T18:22:47.117373+0000","last_peered":"2024-04-15T18:22:47.117373+0000","last_clean":"2024-04-15T18:21:37.153258+0000","last_became_active":"2024-04-15T18:21:57.737029+0000","last_became_peered":"2024-04-15T18:21:57.737029+0000","last_unstale":"2024-04-15T18:22:47.117373+0000","last_undegraded":"2024-04-15T18:21:57.302505+0000","last_fullsized":"2024-04-15T18:21:57.216461+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7605,"log_dups_size":0,"ondisk_log_size":7605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7605,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2654,"num_bytes_recovered":173932544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,10],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1420}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"328'7452","reported_seq":7611,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:44.798746+0000","last_change":"2024-04-15T18:22:03.640303+0000","last_active":"2024-04-15T18:22:44.798746+0000","last_peered":"2024-04-15T18:22:44.798746+0000","last_clean":"2024-04-15T18:22:44.798746+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:22:44.798746+0000","last_undegraded":"2024-04-15T18:22:44.798746+0000","last_fullsized":"2024-04-15T18:22:44.798746+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5259","last_scrub_stamp":"2024-04-15T18:22:03.640098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.640098+0000","objects_scrubbed":5226,"log_size":7452,"log_dups_size":0,"ondisk_log_size":7452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:19.907355+0000","scrub_duration":1294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257163264,"num_objects":3924,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7452,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"329'7600","reported_seq":7700,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.370483+0000","last_change":"2024-04-15T18:20:31.187024+0000","last_active":"2024-04-15T18:22:47.370483+0000","last_peered":"2024-04-15T18:22:47.370483+0000","last_clean":"2024-04-15T18:22:47.370483+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:22:47.370483+0000","last_undegraded":"2024-04-15T18:22:47.370483+0000","last_fullsized":"2024-04-15T18:22:47.370483+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239206400,"num_objects":3650,"num_object_clones":0,"num_object_copies":10950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7600,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"328'7636","reported_seq":7797,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:44.823325+0000","last_change":"2024-04-15T18:22:19.093291+0000","last_active":"2024-04-15T18:22:44.823325+0000","last_peered":"2024-04-15T18:22:44.823325+0000","last_clean":"2024-04-15T18:22:44.823325+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:22:44.823325+0000","last_undegraded":"2024-04-15T18:22:44.823325+0000","last_fullsized":"2024-04-15T18:22:44.823325+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6103","last_scrub_stamp":"2024-04-15T18:22:19.093022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:19.093022+0000","objects_scrubbed":5424,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:27.551120+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7636,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"329'7414","reported_seq":7513,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:47.511906+0000","last_change":"2024-04-15T18:20:31.186380+0000","last_active":"2024-04-15T18:22:47.511906+0000","last_peered":"2024-04-15T18:22:47.511906+0000","last_clean":"2024-04-15T18:22:47.511906+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:22:47.511906+0000","last_undegraded":"2024-04-15T18:22:47.511906+0000","last_fullsized":"2024-04-15T18:22:47.511906+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239337472,"num_objects":3652,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7414,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"328'9593","reported_seq":9850,"reported_epoch":328,"state":"active+clean","last_fresh":"2024-04-15T18:22:45.248730+0000","last_change":"2024-04-15T18:21:44.278068+0000","last_active":"2024-04-15T18:22:45.248730+0000","last_peered":"2024-04-15T18:22:45.248730+0000","last_clean":"2024-04-15T18:22:45.248730+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:22:45.248730+0000","last_undegraded":"2024-04-15T18:22:45.248730+0000","last_fullsized":"2024-04-15T18:22:45.248730+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'6222","last_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_deep_scrub":"311'6222","last_deep_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_clean_scrub_stamp":"2024-04-15T18:21:44.277930+0000","objects_scrubbed":3695,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:13.074359+0000","scrub_duration":10061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":870,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-15T18:22:45.404707+0000","last_change":"2024-04-15T18:21:31.648147+0000","last_active":"2024-04-15T18:22:45.404707+0000","last_peered":"2024-04-15T18:22:45.404707+0000","last_clean":"2024-04-15T18:22:45.404707+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:22:45.404707+0000","last_undegraded":"2024-04-15T18:22:45.404707+0000","last_fullsized":"2024-04-15T18:22:45.404707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:21:31.648016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:21:31.648016+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3917742104,"num_objects":59781,"num_object_clones":0,"num_object_copies":179343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118923,"num_write_kb":5718465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5960,"num_bytes_recovered":390594560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11898466304,"data_stored":11898454088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121098,"ondisk_log_size":121098,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925988,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2127356,"kb_used_data":1839348,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91617796,"statfs":{"total":95995035648,"available":93816623104,"internally_reserved":0,"allocated":1883492352,"data_stored":1883197129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":294910404},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187624,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165000,"kb_used_data":1025536,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92580152,"statfs":{"total":95995035648,"available":94802075648,"internally_reserved":0,"allocated":1050148864,"data_stored":1049896942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927628,"kb_used_data":715972,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92817524,"statfs":{"total":95995035648,"available":95045144576,"internally_reserved":0,"allocated":733155328,"data_stored":732899334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":216725952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874417,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122172,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622980,"statfs":{"total":95995035648,"available":95869931520,"internally_reserved":0,"allocated":765952,"data_stored":474301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168758,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073420,"kb_used_data":867208,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":92671732,"statfs":{"total":95995035648,"available":94895853568,"internally_reserved":0,"allocated":888020992,"data_stored":887765518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":211024330},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463099,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1843132,"kb_used_data":1587060,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91902020,"statfs":{"total":95995035648,"available":94107668480,"internally_reserved":0,"allocated":1625149440,"data_stored":1624854237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2470088,"kb_used_data":2159748,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91275064,"statfs":{"total":95995035648,"available":93465665536,"internally_reserved":0,"allocated":2211581952,"data_stored":2211325958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969348,"kb_used_data":713088,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92775804,"statfs":{"total":95995035648,"available":95002423296,"internally_reserved":0,"allocated":730202112,"data_stored":729950190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608967,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475976,"kb_used_data":1188736,"kb_used_omap":44,"kb_used_meta":287187,"kb_avail":92269176,"statfs":{"total":95995035648,"available":94483636224,"internally_reserved":0,"allocated":1217265664,"data_stored":1217013742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":294080042},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986373,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1341132,"kb_used_data":1075140,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92404020,"statfs":{"total":95995035648,"available":94621716480,"internally_reserved":0,"allocated":1100943360,"data_stored":1100687366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724736,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592904,"kb_used_data":463168,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93152248,"statfs":{"total":95995035648,"available":95387901952,"internally_reserved":0,"allocated":474284032,"data_stored":474032110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":726728704,"data_stored":726728704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2183991296,"data_stored":2183987224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442302464,"data_stored":442302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1207173120,"data_stored":1207173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":964694016,"data_stored":964689944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":468647936,"data_stored":468647936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1498284032,"data_stored":1498284032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":882180096,"data_stored":882180096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":722866176,"data_stored":722862104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1047920640,"data_stored":1047920640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1753677824,"data_stored":1753677824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:22:52.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:22:52.220 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:52.244 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:52.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:52.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:22:52.392 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:52.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:52 smithi049 ceph-mon[40087]: pgmap v911: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3707/174525 objects degraded (2.124%); 5.6 MiB/s, 89 objects/s recovering 2024-04-15T18:22:52.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3707/174525 objects degraded (2.124%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:52.508 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:22:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:52.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:22:52.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:52.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:52 smithi159 ceph-mon[45336]: pgmap v911: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3707/174525 objects degraded (2.124%); 5.6 MiB/s, 89 objects/s recovering 2024-04-15T18:22:52.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3707/174525 objects degraded (2.124%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:52.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:52 smithi082 ceph-mon[44206]: pgmap v911: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3707/174525 objects degraded (2.124%); 5.6 MiB/s, 89 objects/s recovering 2024-04-15T18:22:52.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3707/174525 objects degraded (2.124%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:52.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:52.709 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:52.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:52.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:52.723 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:22:52.724 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:22:52.815 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:52.850 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:52.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:22:53.100 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:53.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:53.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:53.243 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:53.276 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:53.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:53.382 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:53.422 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:53.449 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:53.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:22:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:53.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:53.555 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:53.565 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:53.594 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:53.594 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:53.596 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:53.629 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:53.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:53.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:53.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:53.735 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:53.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:53.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:53.786 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:53.815 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:53.817 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:22:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:53.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:53.920 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:53.964 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:54.001 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:54.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:54.039 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:22:54.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:54.147 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:54.268 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:54.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:54.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:54.412 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:54.680 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:54.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:54 smithi082 ceph-mon[44206]: pgmap v912: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3003/166518 objects degraded (1.803%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:22:54.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:54 smithi082 ceph-mon[44206]: osdmap e331: 12 total, 11 up, 11 in 2024-04-15T18:22:54.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/323416205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:54.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:22:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:22:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:22:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:22:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:22:54.716 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:22:54.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:54 smithi049 ceph-mon[40087]: pgmap v912: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3003/166518 objects degraded (1.803%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:22:54.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:54 smithi049 ceph-mon[40087]: osdmap e331: 12 total, 11 up, 11 in 2024-04-15T18:22:54.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/323416205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:54.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:22:54.776 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:22:54.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:54.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:22:54.814 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:54.833 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:22:54.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:22:54.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:22:54.867 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:54.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:54.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:54 smithi159 ceph-mon[45336]: pgmap v912: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3003/166518 objects degraded (1.803%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:22:54.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:54 smithi159 ceph-mon[45336]: osdmap e331: 12 total, 11 up, 11 in 2024-04-15T18:22:54.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/323416205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:22:54.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:55.006 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:55.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:55.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:55.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:55.166 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:55.190 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:55.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:55.297 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:22:55.334 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:55.359 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:22:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:55.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:22:55.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:22:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:55.557 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:22:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:55.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:55.696 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:55.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:55.805 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:22:55.805 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:22:55.805 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:22:55.805 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:22:55.805 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:22:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:22:56.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:22:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: pgmap v914: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 846 op/s; 3003/163635 objects degraded (1.835%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:22:56.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: osdmap e332: 12 total, 11 up, 11 in 2024-04-15T18:22:56.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:56.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:56.460 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:56.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:56.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:22:56.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:56.620 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:56.652 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:56.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:56.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:56.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:56.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: pgmap v914: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 846 op/s; 3003/163635 objects degraded (1.835%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: osdmap e332: 12 total, 11 up, 11 in 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:56.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:56.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:22:56.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: pgmap v914: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 846 op/s; 3003/163635 objects degraded (1.835%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:22:56.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: osdmap e332: 12 total, 11 up, 11 in 2024-04-15T18:22:56.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:56.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:56.793 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:56.823 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:56.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:56.919 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:22:56.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:22:56.968 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:57.062 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:22:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:57.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:22:57.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:22:57.213 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:57.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:57 smithi049 ceph-mon[40087]: osdmap e333: 12 total, 11 up, 11 in 2024-04-15T18:22:57.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3003/163635 objects degraded (1.835%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:57.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:57 smithi082 ceph-mon[44206]: osdmap e333: 12 total, 11 up, 11 in 2024-04-15T18:22:57.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3003/163635 objects degraded (1.835%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:22:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:57.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:22:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:57 smithi159 ceph-mon[45336]: osdmap e333: 12 total, 11 up, 11 in 2024-04-15T18:22:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3003/163635 objects degraded (1.835%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:22:57.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:22:58.011 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:58.037 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:22:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:58.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:22:58.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:22:58.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:22:58.062 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:22:58.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:58.182 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:58.472 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: pgmap v917: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 856 op/s; 2659/159111 objects degraded (1.671%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:58.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:58.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: pgmap v917: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 856 op/s; 2659/159111 objects degraded (1.671%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:22:58.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:58.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:58.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:22:58.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:22:58.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:58.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:58.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: pgmap v917: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 856 op/s; 2659/159111 objects degraded (1.671%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:22:58.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:22:59.136 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:22:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:59.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:22:59.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:22:59.280 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:59.305 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:22:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:59.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:22:59.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:22:59.456 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:22:59.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:59 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:59.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:59.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:22:59 smithi159 ceph-mon[45336]: osdmap e334: 12 total, 11 up, 11 in 2024-04-15T18:22:59.666 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:22:59.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:59 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:59.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:59.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:22:59 smithi082 ceph-mon[44206]: osdmap e334: 12 total, 11 up, 11 in 2024-04-15T18:22:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:22:59.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:22:59.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:59 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:22:59.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2817148842' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:22:59.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:22:59 smithi049 ceph-mon[40087]: osdmap e334: 12 total, 11 up, 11 in 2024-04-15T18:22:59.772 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:22:59.814 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:00.172 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:00.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:00.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:00.313 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:00.447 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:00.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:00.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:153095 2.d e176/114 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head v 176'9796, mlcod=176'9793)", 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 509.60416247000001, 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25507247500000002, 2024-04-15T18:23:00.754 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150260+0000", 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150262+0000", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6479999999999999e-06 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.150272+0000", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6250000000000002e-06 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400544+0000", 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25027195200000002 2024-04-15T18:23:00.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400549+0000", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2229999999999998e-06 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400606+0000", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7339000000000001e-05 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.400629+0000", 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3059000000000001e-05 2024-04-15T18:23:00.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405295+0000", 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046661419999999999 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:31.405333+0000", 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7487000000000002e-05 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.758 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:82192 3.3 3:cf0d204e:::benchmark_data_smithi159_91621_object82191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 369.80741798899999, 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26314599599999999, 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.759 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82192 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947005+0000", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947006+0000", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4890000000000001e-06 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947096+0000", 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9630999999999996e-05 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947112+0000", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5571000000000001e-05 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.947119+0000", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6040000000000004e-06 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986537+0000", 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.039417671000000001 2024-04-15T18:23:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986667+0000", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013041100000000001 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.986813+0000", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014619599999999999 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991475+0000", 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046618220000000004 2024-04-15T18:23:00.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991584+0000", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010877799999999999 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210072+0000", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21848774500000001 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210107+0000", 2024-04-15T18:23:00.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5611999999999997e-05 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210151+0000", 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3466e-05 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:82205 3.3 3:cead4c8e:::benchmark_data_smithi159_91621_object82204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e222)", 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 369.79123876900002, 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24726364100000001, 2024-04-15T18:23:00.765 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82205 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963184+0000", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963186+0000", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7859999999999999e-06 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963315+0000", 2024-04-15T18:23:00.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012882900000000001 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963331+0000", 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6065e-05 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.963339+0000", 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5350000000000004e-06 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989027+0000", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.025687326 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989114+0000", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7418999999999998e-05 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.989211+0000", 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6518e-05 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991499+0000", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022884369999999999 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:50.991589+0000", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9748999999999997e-05 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210351+0000", 2024-04-15T18:23:00.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21876195400000001 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210406+0000", 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5059999999999998e-05 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:16:51.210448+0000", 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1962999999999998e-05 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:113573 3.a e234/199 3:55511c97:::benchmark_data_smithi159_91621_object7097:head v 234'7161, mlcod=234'7158)", 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 334.09070365899998, 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41958572, 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663719+0000", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663720+0000", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8800000000000002e-07 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:26.663727+0000", 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6490000000000002e-06 2024-04-15T18:23:00.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079390+0000", 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41566319699999998 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079401+0000", 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1106e-05 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079486+0000", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5329999999999998e-05 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.079506+0000", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9528e-05 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083269+0000", 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037634790000000001 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083305+0000", 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5642999999999997e-05 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113580 3.3 3:c4e755da:::benchmark_data_smithi159_91621_object7104:head [delete] snapc 0=[] ondisk+write+known_if_redirected e235)", 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 333.67063384699998, 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27359631099999998, 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113580 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083789+0000", 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083791+0000", 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6479999999999999e-06 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083796+0000", 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.485e-06 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083810+0000", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3864e-05 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083818+0000", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5460000000000003e-06 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083866+0000", 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8829999999999998e-05 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.083982+0000", 2024-04-15T18:23:00.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0001153 2024-04-15T18:23:00.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.084071+0000", 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9450000000000006e-05 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087618+0000", 2024-04-15T18:23:00.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035471399999999998 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087772+0000", 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015348900000000001 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357160+0000", 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26938846300000002 2024-04-15T18:23:00.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357259+0000", 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8851000000000006e-05 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357385+0000", 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012624500000000001 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24793.0:113580 3.3 e235/199 ondisk, result = 0)", 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 333.666663153, 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26963310400000001, 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087760+0000", 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087761+0000", 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.122e-06 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.087768+0000", 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.356968+0000", 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26919998699999997 2024-04-15T18:23:00.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.356985+0000", 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6609999999999999e-05 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357132+0000", 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147217 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357149+0000", 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7127000000000001e-05 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:27.357393+0000", 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000243553 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:15499 4.c 4:33492281:::benchmark_data_smithi159_102124_object15498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:50.394471+0000", 2024-04-15T18:23:00.789 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.35995233599999, 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22943941800000001, 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 15499 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394471+0000", 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394471+0000", 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394471+0000", 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0100000000000005e-07 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394518+0000", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6539000000000003e-05 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394526+0000", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.602e-06 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394531+0000", 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4819999999999996e-06 2024-04-15T18:23:00.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394564+0000", 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3025000000000003e-05 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394637+0000", 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3118e-05 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.394748+0000", 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110874 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395248+0000", 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00050015699999999999 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395758+0000", 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00051011999999999997 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623847+0000", 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22808932300000001 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623882+0000", 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4819e-05 2024-04-15T18:23:00.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623910+0000", 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7758000000000001e-05 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25126.0:15499 4.c e292/284 ondisk, result = 0)", 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:50.395553+0000", 2024-04-15T18:23:00.797 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.35886993099999, 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22836225199999999, 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395553+0000", 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395553+0000", 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395554+0000", 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8699999999999995e-07 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.395557+0000", 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9530000000000001e-06 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623793+0000", 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22823541 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623797+0000", 2024-04-15T18:23:00.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4710000000000001e-06 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623839+0000", 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1353999999999997e-05 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623844+0000", 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0980000000000001e-06 2024-04-15T18:23:00.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.623915+0000", 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1378999999999995e-05 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:15503 4.c 4:3c9fe988:::benchmark_data_smithi159_102124_object15502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:50.397837+0000", 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 130.35658606999999, 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23480457800000001, 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 15503 2024-04-15T18:23:00.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397837+0000", 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397837+0000", 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397838+0000", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2600000000000001e-07 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397864+0000", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6622e-05 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397875+0000", 2024-04-15T18:23:00.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1165000000000001e-05 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397881+0000", 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.604e-06 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.397922+0000", 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0518000000000001e-05 2024-04-15T18:23:00.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.398006+0000", 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4875000000000003e-05 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.398118+0000", 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111774 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.398645+0000", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00052673900000000003 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.399113+0000", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00046778999999999999 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.632578+0000", 2024-04-15T18:23:00.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23346509300000001 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.632612+0000", 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4137999999999997e-05 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.632641+0000", 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9434e-05 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:24793 4.e 4:744641f7:::benchmark_data_smithi159_102124_object24792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:56.541690+0000", 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 124.212733375, 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24914171299999999, 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.810 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 24793 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541690+0000", 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541690+0000", 2024-04-15T18:23:00.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541690+0000", 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7699999999999992e-07 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541755+0000", 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4508e-05 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.812 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541765+0000", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0271999999999999e-05 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541771+0000", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3279999999999996e-06 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541835+0000", 2024-04-15T18:23:00.813 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4837000000000003e-05 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.541973+0000", 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000137126 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,10", 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.542137+0000", 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016433600000000001 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.814 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543039+0000", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090209299999999997 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543204+0000", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016477899999999999 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790768+0000", 2024-04-15T18:23:00.815 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24756435299999999 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790804+0000", 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5531999999999998e-05 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790831+0000", 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7572e-05 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.816 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25126.0:24793 4.e e293/284 ondisk, result = 0)", 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:56.543279+0000", 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 124.21114436000001, 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.247557207, 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:00.817 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543279+0000", 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543279+0000", 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.818 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543279+0000", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.68e-07 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.543282+0000", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.242e-06 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790735+0000", 2024-04-15T18:23:00.819 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24745220400000001 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790738+0000", 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6619999999999998e-06 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790761+0000", 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3122999999999999e-05 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.820 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790765+0000", 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6679999999999999e-06 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:56.790836+0000", 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0740000000000004e-05 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.821 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:39789 4.c 4:381e7168:::benchmark_data_smithi159_102124_object39788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:21:22.087309+0000", 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 98.667113572999995, 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22685892799999999, 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39789 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.822 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087309+0000", 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087309+0000", 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.823 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087310+0000", 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0999999999999997e-07 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087331+0000", 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1231999999999999e-05 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087342+0000", 2024-04-15T18:23:00.824 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0491e-05 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087350+0000", 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6020000000000002e-06 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087390+0000", 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9966000000000003e-05 2024-04-15T18:23:00.825 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087481+0000", 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1841000000000004e-05 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.087566+0000", 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4835999999999999e-05 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.826 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.088771+0000", 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001204812 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.098416+0000", 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0096448599999999999 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.827 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.314081+0000", 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21566480699999999 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.314121+0000", 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0221e-05 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.314168+0000", 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7349999999999999e-05 2024-04-15T18:23:00.828 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:39801 4.c 4:3ea5c1e4:::benchmark_data_smithi159_102124_object39800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:21:22.096360+0000", 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 98.658062572000006, 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22681623000000001, 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.829 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 39801 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096360+0000", 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.830 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096360+0000", 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096361+0000", 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0809999999999999e-06 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.831 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096412+0000", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0182999999999997e-05 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096423+0000", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1055e-05 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096430+0000", 2024-04-15T18:23:00.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5449999999999998e-06 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096478+0000", 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8269000000000002e-05 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096584+0000", 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105333 2024-04-15T18:23:00.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.096686+0000", 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102494 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.097866+0000", 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001180054 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.098578+0000", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071120299999999999 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.323106+0000", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.224528903 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.323140+0000", 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3422e-05 2024-04-15T18:23:00.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.323177+0000", 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6687999999999998e-05 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:71501 4.e 4:774826b2:::benchmark_data_smithi159_102124_object71500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:21:49.921086+0000", 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 70.833336731000003, 2024-04-15T18:23:00.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.342742716, 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 71501 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921086+0000", 2024-04-15T18:23:00.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921086+0000", 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921087+0000", 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5300000000000003e-07 2024-04-15T18:23:00.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921107+0000", 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0202000000000001e-05 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921117+0000", 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8539999999999992e-06 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921123+0000", 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3249999999999996e-06 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921192+0000", 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8283999999999999e-05 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921281+0000", 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9370000000000007e-05 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,10", 2024-04-15T18:23:00.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.921369+0000", 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7837000000000006e-05 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.923149+0000", 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017797430000000001 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.923511+0000", 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000361978 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.263746+0000", 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34023537999999998 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.263788+0000", 2024-04-15T18:23:00.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2048000000000001e-05 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.263829+0000", 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0942000000000002e-05 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:89317 4.c 4:3d350ac9:::benchmark_data_smithi159_102124_object89316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.386142+0000", 2024-04-15T18:23:00.844 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.368281162000002, 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30004350899999999, 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 89317 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386142+0000", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386142+0000", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386143+0000", 2024-04-15T18:23:00.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.598e-06 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386214+0000", 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0505000000000003e-05 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386225+0000", 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0681e-05 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386230+0000", 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.784e-06 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386273+0000", 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2913000000000001e-05 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386379+0000", 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105466 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.386509+0000", 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012984800000000001 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.389096+0000", 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025870770000000001 2024-04-15T18:23:00.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.389792+0000", 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069596299999999996 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686096+0000", 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29630427500000001 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686153+0000", 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.681e-05 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686185+0000", 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2589000000000003e-05 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:89341 4.c 4:33e79400:::benchmark_data_smithi159_102124_object89340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.420669+0000", 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.333753987000001, 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.65495880299999998, 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.852 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 89341 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420669+0000", 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420669+0000", 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420670+0000", 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0920000000000001e-06 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420731+0000", 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1486000000000001e-05 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420743+0000", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1892999999999999e-05 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420749+0000", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7350000000000001e-06 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420793+0000", 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4116000000000002e-05 2024-04-15T18:23:00.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420893+0000", 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100272 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.420993+0000", 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9308999999999994e-05 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.423268+0000", 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022748209999999998 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.423559+0000", 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029142600000000001 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075548+0000", 2024-04-15T18:23:00.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.65198875599999995 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075591+0000", 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3659000000000003e-05 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075628+0000", 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6238000000000001e-05 2024-04-15T18:23:00.858 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25126.0:89317 4.c e322/284 ondisk, result = 0)", 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.424001+0000", 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.330422357000003, 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26219130299999999, 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:00.859 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.424001+0000", 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.424001+0000", 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.424001+0000", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6700000000000003e-07 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.424005+0000", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2529999999999998e-06 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686009+0000", 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26200378200000002 2024-04-15T18:23:00.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686014+0000", 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0939999999999998e-06 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686078+0000", 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3927e-05 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686091+0000", 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2921999999999999e-05 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.686192+0000", 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100658 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25126.0:89341 4.c e322/284 ondisk, result = 0)", 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.685986+0000", 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 49.068436843000001, 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.389647574, 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.685986+0000", 2024-04-15T18:23:00.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.685986+0000", 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.685987+0000", 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2500000000000004e-07 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.685991+0000", 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3619999999999999e-06 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075477+0000", 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.389485841 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075484+0000", 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8669999999999996e-06 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075536+0000", 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1666999999999998e-05 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075543+0000", 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7980000000000006e-06 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.075634+0000", 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0214000000000001e-05 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:89352 4.d 4:b8615e71:::benchmark_last_metadata:head [writefull 0~24] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:12.449831+0000", 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 48.304592307999997, 2024-04-15T18:23:00.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.846638102, 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 89352 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449831+0000", 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449831+0000", 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449833+0000", 2024-04-15T18:23:00.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5500000000000001e-06 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449848+0000", 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4686e-05 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449879+0000", 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1653999999999999e-05 2024-04-15T18:23:00.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.449896+0000", 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6869000000000001e-05 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.450066+0000", 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000169844 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.450243+0000", 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017714200000000001 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.450346+0000", 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102217 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.452776+0000", 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024309079999999999 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.453259+0000", 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00048222800000000002 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.296376+0000", 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.843117695 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.296422+0000", 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5172000000000001e-05 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.296469+0000", 2024-04-15T18:23:00.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7136999999999999e-05 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:129174 4.c 4:3ab79f80:::benchmark_data_smithi159_102124_object39820:head [delete] snapc 0=[] ondisk+write+known_if_redirected e334)", 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:58.734483+0000", 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 2.0199394609999999, 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216297978, 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:00.876 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 129174 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734483+0000", 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734483+0000", 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734485+0000", 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.22e-06 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.878 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734490+0000", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8949999999999997e-06 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734499+0000", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8160000000000005e-06 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734506+0000", 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8589999999999999e-06 2024-04-15T18:23:00.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734559+0000", 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2423000000000003e-05 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734682+0000", 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012374900000000001 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.734806+0000", 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000123874 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.738461+0000", 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036543959999999999 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.740447+0000", 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019862080000000002 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.950675+0000", 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.210228055 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:00.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.950722+0000", 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6907000000000003e-05 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:58.950781+0000", 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9576e-05 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:00.883 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:00.884 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:00.884 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:00 smithi082 ceph-mon[44206]: pgmap v919: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 2222/154602 objects degraded (1.437%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:23:00.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:00 smithi159 ceph-mon[45336]: pgmap v919: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 2222/154602 objects degraded (1.437%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:23:00.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:00 smithi049 ceph-mon[40087]: pgmap v919: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 2222/154602 objects degraded (1.437%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:23:01.473 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:01.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:01.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:23:01.582 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:01.615 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:01.639 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:01.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:01.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:01.786 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:01.819 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:01.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:01.885 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:01.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:01.963 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:02.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:02.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:02.324 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:02.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:02.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:02.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:02 smithi049 ceph-mon[40087]: pgmap v920: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 2222/149769 objects degraded (1.484%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:23:02.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2222/149769 objects degraded (1.484%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:02.646 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:02.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:02 smithi082 ceph-mon[44206]: pgmap v920: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 2222/149769 objects degraded (1.484%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:23:02.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2222/149769 objects degraded (1.484%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:02.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:02 smithi159 ceph-mon[45336]: pgmap v920: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 2222/149769 objects degraded (1.484%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:23:02.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2222/149769 objects degraded (1.484%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:03.851 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:03.851 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:23:03.852 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:23:03.852 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:04 smithi082 ceph-mon[44206]: pgmap v921: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 876 op/s; 1489/141651 objects degraded (1.051%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:23:04.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:23:04.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:04 smithi159 ceph-mon[45336]: pgmap v921: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 876 op/s; 1489/141651 objects degraded (1.051%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:23:04.952 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:04.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:04 smithi049 ceph-mon[40087]: pgmap v921: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 876 op/s; 1489/141651 objects degraded (1.051%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: pgmap v922: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 1489/138573 objects degraded (1.075%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: osdmap e335: 12 total, 11 up, 11 in 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: 4.2 scrub starts 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:23:06.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:06 smithi082 ceph-mon[44206]: 4.2 scrub ok 2024-04-15T18:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:23:06.847 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: pgmap v922: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 1489/138573 objects degraded (1.075%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: osdmap e335: 12 total, 11 up, 11 in 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: 4.2 scrub starts 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:23:06.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:06 smithi159 ceph-mon[45336]: 4.2 scrub ok 2024-04-15T18:23:06.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: pgmap v922: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 865 op/s; 1489/138573 objects degraded (1.075%); 11 MiB/s, 172 objects/s recovering 2024-04-15T18:23:06.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: osdmap e335: 12 total, 11 up, 11 in 2024-04-15T18:23:06.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:23:06.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: 4.2 scrub starts 2024-04-15T18:23:06.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:23:06.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:06 smithi049 ceph-mon[40087]: 4.2 scrub ok 2024-04-15T18:23:07.513 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:07 smithi082 ceph-mon[44206]: osdmap e336: 12 total, 11 up, 11 in 2024-04-15T18:23:07.513 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1489/138573 objects degraded (1.075%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:07.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:23:07.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:07 smithi159 ceph-mon[45336]: osdmap e336: 12 total, 11 up, 11 in 2024-04-15T18:23:07.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1489/138573 objects degraded (1.075%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:07.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:07 smithi049 ceph-mon[40087]: osdmap e336: 12 total, 11 up, 11 in 2024-04-15T18:23:07.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1489/138573 objects degraded (1.075%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:23:07.994 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:09 smithi082 ceph-mon[44206]: pgmap v925: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 1004/134466 objects degraded (0.747%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:23:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:09 smithi082 ceph-mon[44206]: 4.c scrub starts 2024-04-15T18:23:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:09 smithi082 ceph-mon[44206]: 4.c scrub ok 2024-04-15T18:23:09.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:09 smithi049 ceph-mon[40087]: pgmap v925: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 1004/134466 objects degraded (0.747%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:23:09.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:09 smithi049 ceph-mon[40087]: 4.c scrub starts 2024-04-15T18:23:09.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:09 smithi049 ceph-mon[40087]: 4.c scrub ok 2024-04-15T18:23:09.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:09 smithi159 ceph-mon[45336]: pgmap v925: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 1004/134466 objects degraded (0.747%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:23:09.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:09 smithi159 ceph-mon[45336]: 4.c scrub starts 2024-04-15T18:23:09.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:09 smithi159 ceph-mon[45336]: 4.c scrub ok 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:89108 2.3 e156/114 ondisk, result = 0)", 2024-04-15T18:23:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 592.98135307099994, 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40252157700000002, 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138632+0000", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6599999999999996e-07 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.138636+0000", 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.348e-06 2024-04-15T18:23:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540769+0000", 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40213289499999999 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540782+0000", 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3692999999999999e-05 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540945+0000", 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000162227 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.540962+0000", 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7819999999999999e-05 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:13:17.541153+0000", 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019102799999999999 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157649 2.3 2:c89d4810:::benchmark_data_smithi159_80260_object68536:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 511.08037762399999, 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26908227299999998, 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157649 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039607+0000", 2024-04-15T18:23:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039609+0000", 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6050000000000001e-06 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039617+0000", 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4190000000000002e-06 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039636+0000", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8454e-05 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039645+0000", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2110000000000004e-06 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039704+0000", 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8634999999999997e-05 2024-04-15T18:23:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.039708+0000", 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5199999999999999e-06 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043065+0000", 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033569559999999999 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043169+0000", 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103738 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.043236+0000", 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6959999999999996e-05 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.047179+0000", 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039432829999999997 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.048394+0000", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00121502 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308578+0000", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26018443400000002 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308639+0000", 2024-04-15T18:23:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0284000000000001e-05 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.308690+0000", 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0754000000000002e-05 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157667 2.3 2:ce409a58:::benchmark_data_smithi159_80260_object68554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:23:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 511.037286435, 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25352618900000001, 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157667 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082698+0000", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082699+0000", 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.64e-07 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082704+0000", 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9840000000000001e-06 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082715+0000", 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0516999999999999e-05 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082721+0000", 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6300000000000003e-06 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082759+0000", 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8463000000000001e-05 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082852+0000", 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3418999999999994e-05 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.082928+0000", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5815999999999994e-05 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086226+0000", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032979839999999999 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.086901+0000", 2024-04-15T18:23:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067478099999999995 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336152+0000", 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.249250834 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336190+0000", 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7737999999999997e-05 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.336225+0000", 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5058999999999998e-05 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:157688 2.3 2:c5c2b454:::benchmark_data_smithi159_80260_object68575:head [delete] snapc 0=[] ondisk+write+known_if_redirected e178)", 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 510.773501936, 2024-04-15T18:23:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.316335582, 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157688 2024-04-15T18:23:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:23:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346483+0000", 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346485+0000", 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.097e-06 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346496+0000", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1211e-05 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346520+0000", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4124000000000001e-05 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346531+0000", 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0691e-05 2024-04-15T18:23:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346590+0000", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9417999999999999e-05 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346693+0000", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010207700000000001 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.346776+0000", 2024-04-15T18:23:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3354000000000002e-05 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.350571+0000", 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037947829999999999 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.351115+0000", 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054438000000000002 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662733+0000", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.311618171 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662772+0000", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9242000000000001e-05 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:39.662819+0000", 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6034e-05 2024-04-15T18:23:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:158069 2.2 2:4af888cc:::benchmark_data_smithi159_80260_object68956:head [delete] snapc 0=[] ondisk+write+known_if_redirected e179)", 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 509.54966941399999, 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27351397799999999, 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 158069 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570315+0000", 2024-04-15T18:23:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570316+0000", 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7300000000000004e-07 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570321+0000", 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5569999999999997e-06 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570330+0000", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3389999999999999e-06 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570337+0000", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5250000000000002e-06 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570370+0000", 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2932999999999998e-05 2024-04-15T18:23:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570459+0000", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8788000000000004e-05 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.570531+0000", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2027999999999995e-05 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573921+0000", 2024-04-15T18:23:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033902889999999999 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.601514+0000", 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027593414 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843733+0000", 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24221832400000001 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843785+0000", 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2438999999999997e-05 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843829+0000", 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4369000000000002e-05 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:158069 2.2 e179/114 ondisk, result = 0)", 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 509.54609644599998, 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26994574500000001, 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573888+0000", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573889+0000", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4499999999999996e-07 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.573895+0000", 2024-04-15T18:23:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6389999999999997e-06 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843667+0000", 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26977213900000002 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843673+0000", 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5960000000000003e-06 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843719+0000", 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6405e-05 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843726+0000", 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.323e-06 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:40.843834+0000", 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000107898 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:17054 3.5 3:ab47d494:::benchmark_data_smithi159_91621_object17053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e206)", 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 431.51619209199998, 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26525695500000002, 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 17054 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603793+0000", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1200000000000003e-07 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603877+0000", 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3918999999999994e-05 2024-04-15T18:23:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603888+0000", 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1080999999999999e-05 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603894+0000", 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5860000000000004e-06 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.603940+0000", 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5918000000000003e-05 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604029+0000", 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9003000000000001e-05 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,6", 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604111+0000", 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2146999999999998e-05 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.604861+0000", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075021899999999999 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.606482+0000", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001620687 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.868983+0000", 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.262501182 2024-04-15T18:23:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.869017+0000", 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4103999999999997e-05 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:58.869050+0000", 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2496999999999997e-05 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113524 3.c 3:30a9f32f:::benchmark_data_smithi159_91621_object7048:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.115667+0000", 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.004317684, 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34058279400000002, 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113524 2024-04-15T18:23:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115667+0000", 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115667+0000", 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115669+0000", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.917e-06 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115678+0000", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2089999999999994e-06 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115695+0000", 2024-04-15T18:23:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6294e-05 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115706+0000", 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1396999999999999e-05 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118767+0000", 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030612130000000001 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118778+0000", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0720000000000001e-05 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.456090+0000", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.33731155899999998 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.456250+0000", 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016048500000000001 2024-04-15T18:23:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113571 3.c 3:3f83a0a6:::benchmark_data_smithi159_91621_object7095:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.115730+0000", 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00425483399999, 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36950692899999998, 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113571 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115730+0000", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115730+0000", 2024-04-15T18:23:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115730+0000", 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0899999999999997e-07 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115732+0000", 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1390000000000002e-06 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115737+0000", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5229999999999997e-06 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115740+0000", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3670000000000001e-06 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118804+0000", 2024-04-15T18:23:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030640559999999999 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118812+0000", 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2189999999999998e-06 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.458835+0000", 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34002348399999999 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.459002+0000", 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016645700000000001 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.459169+0000", 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000167436 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.465148+0000", 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0059791100000000002 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.465186+0000", 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8129e-05 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481658+0000", 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016472067 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481700+0000", 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1309e-05 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485237+0000", 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035373240000000001 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113584 3.c 3:3816a81a:::benchmark_data_smithi159_91621_object7108:head [delete] snapc 0=[] ondisk+write+known_if_redirected e236)", 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.115744+0000", 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00424041299999, 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37350634900000002, 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113584 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115744+0000", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115744+0000", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115745+0000", 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.29e-07 2024-04-15T18:23:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115746+0000", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.64e-06 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115750+0000", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7160000000000001e-06 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.115753+0000", 2024-04-15T18:23:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.881e-06 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.119090+0000", 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033365920000000002 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.119100+0000", 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0733999999999999e-05 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.462086+0000", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.342985394 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.462251+0000", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165494 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.462355+0000", 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103748 2024-04-15T18:23:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.465218+0000", 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028626989999999998 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.465330+0000", 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112847 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485285+0000", 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.019954897999999999 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485320+0000", 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4177999999999997e-05 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.489251+0000", 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039312990000000001 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113532 3.5 3:a3af6274:::benchmark_data_smithi159_91621_object7056:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.118474+0000", 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00151085700003, 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32809179900000002, 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113532 2024-04-15T18:23:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118474+0000", 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118474+0000", 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118476+0000", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.103e-06 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118486+0000", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7170000000000003e-06 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118500+0000", 2024-04-15T18:23:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3927999999999999e-05 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118516+0000", 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6002e-05 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.120898+0000", 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023822549999999998 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.120910+0000", 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1933000000000001e-05 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.446381+0000", 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32547073300000001 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.446566+0000", 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018512800000000001 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113541 3.5 3:a553a5ef:::benchmark_data_smithi159_91621_object7065:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.118542+0000", 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00144328800002, 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.33058079299999998, 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.194 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113541 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118542+0000", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118542+0000", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118542+0000", 2024-04-15T18:23:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6199999999999998e-07 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118545+0000", 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2260000000000001e-06 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118552+0000", 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5089999999999999e-06 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118559+0000", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4240000000000004e-06 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121486+0000", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029263179999999998 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121491+0000", 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1000000000000003e-06 2024-04-15T18:23:10.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.449000+0000", 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32750889999999999 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.449122+0000", 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012275400000000001 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113558 3.5 3:ae8e5dfd:::benchmark_data_smithi159_91621_object7082:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.118566+0000", 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00141862200002, 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36643874900000001, 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.200 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113558 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118566+0000", 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118566+0000", 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118567+0000", 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9499999999999998e-07 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118569+0000", 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5500000000000001e-06 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118577+0000", 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0700000000000007e-06 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118583+0000", 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6250000000000004e-06 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121535+0000", 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029519500000000001 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121547+0000", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2625e-05 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.449158+0000", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32761032600000001 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.449289+0000", 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013148599999999999 2024-04-15T18:23:10.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.449398+0000", 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010843 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.460843+0000", 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.011445498 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.460858+0000", 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.52e-05 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481573+0000", 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.020714264 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.481655+0000", 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2013000000000002e-05 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485005+0000", 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033503169999999998 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:113574 3.5 3:a379b0de:::benchmark_data_smithi159_91621_object7098:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e236)", 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:28.118590+0000", 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.00139444500002, 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36969342199999999, 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:10.208 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113574 2024-04-15T18:23:10.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.209 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118590+0000", 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118590+0000", 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118591+0000", 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2599999999999998e-07 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118593+0000", 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3300000000000001e-06 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118599+0000", 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1929999999999998e-06 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.118604+0000", 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9570000000000001e-06 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121564+0000", 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029595749999999999 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.121568+0000", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.279e-06 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.452210+0000", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.33064227200000001 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.452377+0000", 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000166906 2024-04-15T18:23:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.452560+0000", 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000183055 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.460863+0000", 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0083028970000000001 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.460956+0000", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3105999999999999e-05 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485081+0000", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.024124491000000001 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.485146+0000", 2024-04-15T18:23:10.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4585000000000006e-05 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:28.488284+0000", 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031383499999999998 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:39780 4.1 e301/284 4:86de28a1:::benchmark_data_smithi159_102124_object39779:head v 301'2631, mlcod=301'2628)", 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:21:22.078579+0000", 2024-04-15T18:23:10.216 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 108.041405634, 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23869781000000001, 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.078579+0000", 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.078579+0000", 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.078580+0000", 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4600000000000004e-07 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.078756+0000", 2024-04-15T18:23:10.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017590100000000001 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.315826+0000", 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23707020500000001 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.315832+0000", 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.959e-06 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.315910+0000", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8442999999999994e-05 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.315938+0000", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7639000000000001e-05 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.317242+0000", 2024-04-15T18:23:10.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013043040000000001 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:22.317277+0000", 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4613000000000002e-05 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:71486 4.2 e315/284 4:46c4a592:::benchmark_data_smithi159_102124_object71485:head v 315'4483, mlcod=315'4480)", 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:21:49.876606+0000", 2024-04-15T18:23:10.221 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 80.243378784000001, 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.308087628, 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.876606+0000", 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.876606+0000", 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.876607+0000", 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7499999999999998e-07 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:49.876669+0000", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2288999999999999e-05 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.182702+0000", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30603279700000002 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.182712+0000", 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6830000000000003e-06 2024-04-15T18:23:10.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.182806+0000", 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3837000000000003e-05 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.182828+0000", 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1916000000000002e-05 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.184666+0000", 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001838059 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:21:50.184694+0000", 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8072000000000002e-05 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:133412 4.2 e334/284 4:4cc08a99:::benchmark_data_smithi159_102124_object44058:head v 334'8417, mlcod=334'8413)", 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:03.655641+0000", 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 6.464343779, 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.096228293000000006, 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.655641+0000", 2024-04-15T18:23:10.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.655641+0000", 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.655642+0000", 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5499999999999998e-07 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.655646+0000", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5220000000000001e-06 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.748213+0000", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.092566457000000005 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.748223+0000", 2024-04-15T18:23:10.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0572e-05 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.748287+0000", 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3342000000000004e-05 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.748313+0000", 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6506999999999999e-05 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.751817+0000", 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035040409999999998 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.751869+0000", 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2296999999999998e-05 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.231 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:133641 4.2 e334/284 4:413cd078:::benchmark_data_smithi159_102124_object44287:head v 334'8427, mlcod=334'8424)", 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:03.955584+0000", 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 6.1644011990000003, 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.078010460000000004, 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.955584+0000", 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.955584+0000", 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.955585+0000", 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3960000000000001e-06 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:03.955592+0000", 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3250000000000002e-06 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.026948+0000", 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.071355162999999999 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.026963+0000", 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5495999999999999e-05 2024-04-15T18:23:10.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.027085+0000", 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122371 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.027134+0000", 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8242999999999999e-05 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.033562+0000", 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0064281809999999998 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.033594+0000", 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2285e-05 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:133666 4.2 e334/284 4:46282705:::benchmark_data_smithi159_102124_object44312:head v 334'8430, mlcod=334'8427)", 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:04.036907+0000", 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 6.0830780200000003, 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.094635828000000005, 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.036907+0000", 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.036907+0000", 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.036908+0000", 2024-04-15T18:23:10.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1399999999999996e-07 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.036915+0000", 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4089999999999997e-06 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.128483+0000", 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.091567554999999995 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.128489+0000", 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6810000000000001e-06 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.128536+0000", 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6888999999999998e-05 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.128552+0000", 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5739999999999998e-05 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.131506+0000", 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002953776 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:04.131543+0000", 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6964e-05 2024-04-15T18:23:10.241 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:136606 4.2 e336/284 4:4e582662:::benchmark_data_smithi159_102124_object47252:head v 336'8606, mlcod=336'8603)", 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:08.319486+0000", 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 1.8004984369999999, 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18953832400000001, 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:10.242 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.319486+0000", 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.319486+0000", 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.319487+0000", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4499999999999996e-07 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.319493+0000", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7080000000000002e-06 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.505604+0000", 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18611064999999999 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.505610+0000", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3810000000000004e-06 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.505666+0000", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5519000000000002e-05 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.505691+0000", 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5442000000000001e-05 2024-04-15T18:23:10.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.508976+0000", 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032853610000000001 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.509025+0000", 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8418000000000003e-05 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.246 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.247 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:10.247 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:10.247 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:10.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:10 smithi049 ceph-mon[40087]: 4.6 scrub starts 2024-04-15T18:23:10.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:23:10.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:23:10.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:10 smithi082 ceph-mon[44206]: 4.6 scrub starts 2024-04-15T18:23:10.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:23:10.489 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:23:10.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:10 smithi159 ceph-mon[45336]: 4.6 scrub starts 2024-04-15T18:23:10.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:23:10.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:23:10.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:23:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:11 smithi082 ceph-mon[44206]: pgmap v926: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 720/130530 objects degraded (0.552%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:23:11.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:11 smithi082 ceph-mon[44206]: 4.6 scrub ok 2024-04-15T18:23:11.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:11 smithi082 ceph-mon[44206]: osdmap e337: 12 total, 11 up, 11 in 2024-04-15T18:23:11.212 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:11.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:11 smithi049 ceph-mon[40087]: pgmap v926: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 720/130530 objects degraded (0.552%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:23:11.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:11 smithi049 ceph-mon[40087]: 4.6 scrub ok 2024-04-15T18:23:11.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:11 smithi049 ceph-mon[40087]: osdmap e337: 12 total, 11 up, 11 in 2024-04-15T18:23:11.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:11.634 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:11.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:11.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:11 smithi159 ceph-mon[45336]: pgmap v926: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 720/130530 objects degraded (0.552%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:23:11.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:11 smithi159 ceph-mon[45336]: 4.6 scrub ok 2024-04-15T18:23:11.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:11 smithi159 ceph-mon[45336]: osdmap e337: 12 total, 11 up, 11 in 2024-04-15T18:23:11.674 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":917,"stamp":"2024-04-15T18:22:57.243366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3476226616,"num_objects":53037,"num_object_clones":0,"num_object_copies":159111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53037,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":125850,"num_write_kb":5721403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7018,"num_bytes_recovered":461920928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127881,"ondisk_log_size":127881,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":12868180,"kb_used_data":10382432,"kb_used_omap":60,"kb_used_meta":2485507,"kb_avail":1018328492,"statfs":{"total":1055945392128,"available":1042768375808,"internally_reserved":0,"allocated":10631610368,"data_stored":10628721536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61481,"internal_metadata":2545160151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-336723968,"num_objects":-5138,"num_object_clones":0,"num_object_copies":-15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":68681728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001354"},"pg_stats":[{"pgid":"4.f","version":"333'8025","reported_seq":8174,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-04-15T18:22:57.109084+0000","last_change":"2024-04-15T18:21:32.333996+0000","last_active":"2024-04-15T18:22:57.109084+0000","last_peered":"2024-04-15T18:22:57.109084+0000","last_clean":"2024-04-15T18:22:57.109084+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:22:57.109084+0000","last_undegraded":"2024-04-15T18:22:57.109084+0000","last_fullsized":"2024-04-15T18:22:57.109084+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'3320","last_scrub_stamp":"2024-04-15T18:21:32.333909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:21:32.333909+0000","objects_scrubbed":3301,"log_size":8025,"log_dups_size":0,"ondisk_log_size":8025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8025,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"330'7540","reported_seq":7700,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-04-15T18:22:52.532741+0000","last_change":"2024-04-15T18:22:03.765164+0000","last_active":"2024-04-15T18:22:52.532741+0000","last_peered":"2024-04-15T18:22:52.532741+0000","last_clean":"2024-04-15T18:22:52.532741+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:22:52.532741+0000","last_undegraded":"2024-04-15T18:22:52.532741+0000","last_fullsized":"2024-04-15T18:22:52.532741+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5045","last_scrub_stamp":"2024-04-15T18:22:03.764952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.764952+0000","objects_scrubbed":5010,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:14.312914+0000","scrub_duration":1265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221511680,"num_objects":3380,"num_object_clones":0,"num_object_copies":10140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":349440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"330'7624","reported_seq":7790,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-04-15T18:22:52.536712+0000","last_change":"2024-04-15T18:22:11.794263+0000","last_active":"2024-04-15T18:22:52.536712+0000","last_peered":"2024-04-15T18:22:52.536712+0000","last_clean":"2024-04-15T18:22:52.536712+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:22:52.536712+0000","last_undegraded":"2024-04-15T18:22:52.536712+0000","last_fullsized":"2024-04-15T18:22:52.536712+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5531","last_scrub_stamp":"2024-04-15T18:22:11.794127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.794127+0000","objects_scrubbed":5516,"log_size":7624,"log_dups_size":0,"ondisk_log_size":7624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:35.007436+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225378328,"num_objects":3440,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7624,"num_write_kb":353985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"330'7818","reported_seq":7979,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-04-15T18:22:52.565924+0000","last_change":"2024-04-15T18:22:00.355031+0000","last_active":"2024-04-15T18:22:52.565924+0000","last_peered":"2024-04-15T18:22:52.565924+0000","last_clean":"2024-04-15T18:22:52.565924+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:22:52.565924+0000","last_undegraded":"2024-04-15T18:22:52.565924+0000","last_fullsized":"2024-04-15T18:22:52.565924+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'5063","last_scrub_stamp":"2024-04-15T18:22:00.354917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:00.354917+0000","objects_scrubbed":5039,"log_size":7818,"log_dups_size":0,"ondisk_log_size":7818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:07.368746+0000","scrub_duration":791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229113856,"num_objects":3496,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"332'7944","reported_seq":8110,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-04-15T18:22:56.296816+0000","last_change":"2024-04-15T18:22:05.829296+0000","last_active":"2024-04-15T18:22:56.296816+0000","last_peered":"2024-04-15T18:22:56.296816+0000","last_clean":"2024-04-15T18:22:56.296816+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:22:56.296816+0000","last_undegraded":"2024-04-15T18:22:56.296816+0000","last_fullsized":"2024-04-15T18:22:56.296816+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5237","last_scrub_stamp":"2024-04-15T18:22:05.829122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.829122+0000","objects_scrubbed":5187,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.283179+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045184,"num_objects":3144,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7944,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"331'7934","reported_seq":8100,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:55.286563+0000","last_change":"2024-04-15T18:22:05.753884+0000","last_active":"2024-04-15T18:22:55.286563+0000","last_peered":"2024-04-15T18:22:55.286563+0000","last_clean":"2024-04-15T18:22:55.286563+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:22:55.286563+0000","last_undegraded":"2024-04-15T18:22:55.286563+0000","last_fullsized":"2024-04-15T18:22:55.286563+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5298","last_scrub_stamp":"2024-04-15T18:22:05.753792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.753792+0000","objects_scrubbed":5262,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.075803+0000","scrub_duration":1367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219938816,"num_objects":3356,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7934,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"331'7766","reported_seq":7928,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:54.761239+0000","last_change":"2024-04-15T18:22:23.627521+0000","last_active":"2024-04-15T18:22:54.761239+0000","last_peered":"2024-04-15T18:22:54.761239+0000","last_clean":"2024-04-15T18:22:54.761239+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:22:54.761239+0000","last_undegraded":"2024-04-15T18:22:54.761239+0000","last_fullsized":"2024-04-15T18:22:54.761239+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'6038","last_scrub_stamp":"2024-04-15T18:22:23.627256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:23.627256+0000","objects_scrubbed":4987,"log_size":7766,"log_dups_size":0,"ondisk_log_size":7766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:31.345919+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211812352,"num_objects":3232,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"331'7779","reported_seq":7947,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:54.747537+0000","last_change":"2024-04-15T18:22:11.496215+0000","last_active":"2024-04-15T18:22:54.747537+0000","last_peered":"2024-04-15T18:22:54.747537+0000","last_clean":"2024-04-15T18:22:54.747537+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:22:54.747537+0000","last_undegraded":"2024-04-15T18:22:54.747537+0000","last_fullsized":"2024-04-15T18:22:54.747537+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5535","last_scrub_stamp":"2024-04-15T18:22:11.496110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.496110+0000","objects_scrubbed":5505,"log_size":7779,"log_dups_size":0,"ondisk_log_size":7779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.883032+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215678976,"num_objects":3291,"num_object_clones":0,"num_object_copies":9873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"333'7869","reported_seq":7977,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-04-15T18:22:57.097396+0000","last_change":"2024-04-15T18:20:31.649944+0000","last_active":"2024-04-15T18:22:57.097396+0000","last_peered":"2024-04-15T18:22:57.097396+0000","last_clean":"2024-04-15T18:22:57.097396+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:22:57.097396+0000","last_undegraded":"2024-04-15T18:22:57.097396+0000","last_fullsized":"2024-04-15T18:22:57.097396+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204013568,"num_objects":3113,"num_object_clones":0,"num_object_copies":9339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"330'7682","reported_seq":15173,"reported_epoch":330,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:22:52.416742+0000","last_change":"2024-04-15T18:21:57.693302+0000","last_active":"2024-04-15T18:22:52.416742+0000","last_peered":"2024-04-15T18:22:52.416742+0000","last_clean":"2024-04-15T18:21:37.132207+0000","last_became_active":"2024-04-15T18:21:57.661742+0000","last_became_peered":"2024-04-15T18:21:57.661742+0000","last_unstale":"2024-04-15T18:22:52.416742+0000","last_undegraded":"2024-04-15T18:21:57.238321+0000","last_fullsized":"2024-04-15T18:21:57.218020+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223477760,"num_objects":3410,"num_object_clones":0,"num_object_copies":10230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3693,"num_bytes_recovered":242024448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1],"avail_no_missing":["9","1"],"object_location_counts":[{"shards":"1,9","objects":2315}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"333'8130","reported_seq":14876,"reported_epoch":333,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:22:57.145358+0000","last_change":"2024-04-15T18:21:57.762236+0000","last_active":"2024-04-15T18:22:57.145358+0000","last_peered":"2024-04-15T18:22:57.145358+0000","last_clean":"2024-04-15T18:21:37.153258+0000","last_became_active":"2024-04-15T18:21:57.737029+0000","last_became_peered":"2024-04-15T18:21:57.737029+0000","last_unstale":"2024-04-15T18:22:57.145358+0000","last_undegraded":"2024-04-15T18:21:57.302505+0000","last_fullsized":"2024-04-15T18:21:57.216461+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204472320,"num_objects":3120,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8130,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3315,"num_bytes_recovered":217251840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,10],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"331'7958","reported_seq":8121,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:54.780074+0000","last_change":"2024-04-15T18:22:03.640303+0000","last_active":"2024-04-15T18:22:54.780074+0000","last_peered":"2024-04-15T18:22:54.780074+0000","last_clean":"2024-04-15T18:22:54.780074+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:22:54.780074+0000","last_undegraded":"2024-04-15T18:22:54.780074+0000","last_fullsized":"2024-04-15T18:22:54.780074+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5259","last_scrub_stamp":"2024-04-15T18:22:03.640098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.640098+0000","objects_scrubbed":5226,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:19.907355+0000","scrub_duration":1294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224002048,"num_objects":3418,"num_object_clones":0,"num_object_copies":10254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"330'7851","reported_seq":7953,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-04-15T18:22:52.394252+0000","last_change":"2024-04-15T18:20:31.187024+0000","last_active":"2024-04-15T18:22:52.394252+0000","last_peered":"2024-04-15T18:22:52.394252+0000","last_clean":"2024-04-15T18:22:52.394252+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:22:52.394252+0000","last_undegraded":"2024-04-15T18:22:52.394252+0000","last_fullsized":"2024-04-15T18:22:52.394252+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222756864,"num_objects":3399,"num_object_clones":0,"num_object_copies":10197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"331'8144","reported_seq":8310,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:54.774048+0000","last_change":"2024-04-15T18:22:19.093291+0000","last_active":"2024-04-15T18:22:54.774048+0000","last_peered":"2024-04-15T18:22:54.774048+0000","last_clean":"2024-04-15T18:22:54.774048+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:22:54.774048+0000","last_undegraded":"2024-04-15T18:22:54.774048+0000","last_fullsized":"2024-04-15T18:22:54.774048+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6103","last_scrub_stamp":"2024-04-15T18:22:19.093022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:19.093022+0000","objects_scrubbed":5424,"log_size":8144,"log_dups_size":0,"ondisk_log_size":8144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:27.551120+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219152384,"num_objects":3344,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8144,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"330'7668","reported_seq":7769,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-04-15T18:22:52.552266+0000","last_change":"2024-04-15T18:20:31.186380+0000","last_active":"2024-04-15T18:22:52.552266+0000","last_peered":"2024-04-15T18:22:52.552266+0000","last_clean":"2024-04-15T18:22:52.552266+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:22:52.552266+0000","last_undegraded":"2024-04-15T18:22:52.552266+0000","last_fullsized":"2024-04-15T18:22:52.552266+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"331'10112","reported_seq":10375,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-04-15T18:22:55.255276+0000","last_change":"2024-04-15T18:21:44.278068+0000","last_active":"2024-04-15T18:22:55.255276+0000","last_peered":"2024-04-15T18:22:55.255276+0000","last_clean":"2024-04-15T18:22:55.255276+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:22:55.255276+0000","last_undegraded":"2024-04-15T18:22:55.255276+0000","last_fullsized":"2024-04-15T18:22:55.255276+0000","mapping_epoch":284,"log_start":"290'100","ondisk_log_start":"290'100","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"311'6222","last_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_deep_scrub":"311'6222","last_deep_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_clean_scrub_stamp":"2024-04-15T18:21:44.277930+0000","objects_scrubbed":3695,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:13.074359+0000","scrub_duration":10061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7937,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":876,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-04-15T18:22:55.449782+0000","last_change":"2024-04-15T18:21:31.648147+0000","last_active":"2024-04-15T18:22:55.449782+0000","last_peered":"2024-04-15T18:22:55.449782+0000","last_clean":"2024-04-15T18:22:55.449782+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:22:55.449782+0000","last_undegraded":"2024-04-15T18:22:55.449782+0000","last_fullsized":"2024-04-15T18:22:55.449782+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:21:31.648016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:21:31.648016+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3475636248,"num_objects":53035,"num_object_clones":0,"num_object_copies":159105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125669,"num_write_kb":5718465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7008,"num_bytes_recovered":459276288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10668158976,"data_stored":10668146760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127744,"ondisk_log_size":127744,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925990,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891240,"kb_used_data":1589344,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91853912,"statfs":{"total":95995035648,"available":94058405888,"internally_reserved":0,"allocated":1627488256,"data_stored":1627223736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":309131716},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187625,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110740,"kb_used_data":962956,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92634412,"statfs":{"total":95995035648,"available":94857637888,"internally_reserved":0,"allocated":986066944,"data_stored":985810484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882396,"kb_used_data":666576,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92862756,"statfs":{"total":95995035648,"available":95091462144,"internally_reserved":0,"allocated":682573824,"data_stored":682313292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":220985792},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874419,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122084,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623068,"statfs":{"total":95995035648,"available":95870021632,"internally_reserved":0,"allocated":679936,"data_stored":418988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168760,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":985720,"kb_used_data":787948,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92759432,"statfs":{"total":95995035648,"available":94985658368,"internally_reserved":0,"allocated":806858752,"data_stored":806589764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463101,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621740,"kb_used_data":1373920,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92123412,"statfs":{"total":95995035648,"available":94334373888,"internally_reserved":0,"allocated":1406894080,"data_stored":1406629580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":253688262},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2173552,"kb_used_data":1859176,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":91571600,"statfs":{"total":95995035648,"available":93769318400,"internally_reserved":0,"allocated":1903796224,"data_stored":1903526716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860976,"kb_used_data":613220,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92884176,"statfs":{"total":95995035648,"available":95113396224,"internally_reserved":0,"allocated":627937280,"data_stored":627671844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608968,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1393748,"kb_used_data":1102348,"kb_used_omap":44,"kb_used_meta":291347,"kb_avail":92351404,"statfs":{"total":95995035648,"available":94567837696,"internally_reserved":0,"allocated":1128804352,"data_stored":1128547892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":298339882},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986374,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265244,"kb_used_data":995280,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92479908,"statfs":{"total":95995035648,"available":94699425792,"internally_reserved":0,"allocated":1019166720,"data_stored":1018906188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724737,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560740,"kb_used_data":431000,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93184412,"statfs":{"total":95995035648,"available":95420837888,"internally_reserved":0,"allocated":441344000,"data_stored":441083052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":623640576,"data_stored":623640576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1874202624,"data_stored":1874198552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442302464,"data_stored":442302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1120534528,"data_stored":1120534528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":893194240,"data_stored":893190168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":436600832,"data_stored":436600832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1297481728,"data_stored":1297481728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":802750464,"data_stored":802750464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":672337920,"data_stored":672333848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":984612864,"data_stored":984612864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1520500736,"data_stored":1520500736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:23:11.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:23:11.739 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:11.773 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:12.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 720/130530 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:12.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 720/130530 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:12.574 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:12.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:12.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:23:12.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 720/130530 objects degraded (0.552%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:12.692 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:12.756 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:12.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:12.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:12.861 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:12.892 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:12.948 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:13.053 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:13.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:13.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:13.209 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:13.212 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:13 smithi082 ceph-mon[44206]: pgmap v928: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 720/126636 objects degraded (0.569%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:140523 4.a 4:5e7437b5:::benchmark_data_smithi159_102124_object51169:head [delete] snapc 0=[] ondisk+write+known_if_redirected e337)", 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:13.202160+0000", 2024-04-15T18:23:13.226 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.027363930000000002, 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027395579, 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140523 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202160+0000", 2024-04-15T18:23:13.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202160+0000", 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202161+0000", 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.879999999999999e-07 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:13.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202166+0000", 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9400000000000001e-06 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202175+0000", 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1609999999999992e-06 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202181+0000", 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9170000000000003e-06 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202359+0000", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017792099999999999 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202463+0000", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103959 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.202561+0000", 2024-04-15T18:23:13.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8422000000000001e-05 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.205956+0000", 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033951440000000001 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.210055+0000", 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040988149999999996 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:13.231 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:140543 4.5 4:ae503f9d:::benchmark_data_smithi159_102124_object51189:head [delete] snapc 0=[] ondisk+write+known_if_redirected e337)", 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:13.225160+0000", 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.004363647, 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044980000000000003, 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140543 2024-04-15T18:23:13.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225160+0000", 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225160+0000", 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225161+0000", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.406e-06 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225168+0000", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2589999999999997e-06 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225181+0000", 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2958e-05 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225189+0000", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0790000000000001e-06 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225235+0000", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5927000000000003e-05 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225348+0000", 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011352 2024-04-15T18:23:13.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 10,11", 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225434+0000", 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6068000000000003e-05 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:140544 4.5 4:a42965fa:::benchmark_data_smithi159_102124_object51190:head [delete] snapc 0=[] ondisk+write+known_if_redirected e337)", 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:13.225365+0000", 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0041587079999999997, 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043761520000000003, 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:13.236 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 140544 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225365+0000", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:13.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225365+0000", 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225366+0000", 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0699999999999996e-07 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225369+0000", 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8299999999999998e-06 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225378+0000", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4670000000000004e-06 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225384+0000", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1689999999999997e-06 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.229561+0000", 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041768530000000003 2024-04-15T18:23:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.229695+0000", 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000134411 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:140546 4.8 e337/284 4:129af29b:::benchmark_data_smithi159_102124_object51192:head v 337'8701, mlcod=337'8698)", 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:13.225802+0000", 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0037215740000000001, 2024-04-15T18:23:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040134230000000003, 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225802+0000", 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225802+0000", 2024-04-15T18:23:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225803+0000", 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4099999999999998e-07 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225808+0000", 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9150000000000004e-06 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225823+0000", 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5129e-05 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225830+0000", 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7449999999999998e-06 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225877+0000", 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7781000000000002e-05 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.225897+0000", 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9494999999999999e-05 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:13.229795+0000", 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003898501 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 4 2024-04-15T18:23:13.244 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:13.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:13 smithi049 ceph-mon[40087]: pgmap v928: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 720/126636 objects degraded (0.569%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:23:13.417 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:13.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:13.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:13.561 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:13.602 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:13.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:13 smithi159 ceph-mon[45336]: pgmap v928: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 720/126636 objects degraded (0.569%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:23:13.713 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:13.745 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:13.899 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:13.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:13.922 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:23:14.004 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:14.044 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:14.074 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:14.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:14.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:14.223 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:14.312 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:14.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:14.534 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:14.560 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:14.578 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:14.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:14.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:14.711 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:14.963 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:14.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:15.071 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:15.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:15.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:15.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:15.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:15.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:15.079 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:15.107 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:15.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: pgmap v929: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 255/119148 objects degraded (0.214%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:23:15.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: 4.5 scrub starts 2024-04-15T18:23:15.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2515301050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:15.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:15 smithi082 ceph-mon[44206]: 4.5 scrub ok 2024-04-15T18:23:15.221 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: pgmap v929: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 255/119148 objects degraded (0.214%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: 4.5 scrub starts 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2515301050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:15 smithi049 ceph-mon[40087]: 4.5 scrub ok 2024-04-15T18:23:15.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:15.270 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.293 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:15.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:15.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: pgmap v929: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 255/119148 objects degraded (0.214%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: 4.5 scrub starts 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2515301050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:15.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:15 smithi159 ceph-mon[45336]: 4.5 scrub ok 2024-04-15T18:23:15.426 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.444 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:15.548 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:15.577 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.595 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:15.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:15.728 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.788 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:15.801 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:15.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:15.906 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:15.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:15.953 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:15.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:16.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:16.088 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:16.092 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:16.093 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:16.109 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:16.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:16.213 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:16.240 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:16.260 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:16.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:16.292 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: osdmap e338: 12 total, 11 up, 11 in 2024-04-15T18:23:16.293 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:16 smithi049 ceph-mon[40087]: pgmap v931: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 255/116283 objects degraded (0.219%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:23:16.364 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:16.406 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:16.435 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:16.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:16.441 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.441 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.442 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.442 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.442 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.442 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: osdmap e338: 12 total, 11 up, 11 in 2024-04-15T18:23:16.442 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:16 smithi082 ceph-mon[44206]: pgmap v931: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 255/116283 objects degraded (0.219%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:23:16.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:16.591 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: osdmap e338: 12 total, 11 up, 11 in 2024-04-15T18:23:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:16 smithi159 ceph-mon[45336]: pgmap v931: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 2.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 255/116283 objects degraded (0.219%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:23:16.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:23:16.954 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:16.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:17.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:17.129 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:17.149 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:17.169 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:17.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:17.273 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:17.309 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:17.334 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:17.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:17.399 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:17 smithi082 ceph-mon[44206]: osdmap e339: 12 total, 11 up, 11 in 2024-04-15T18:23:17.399 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:17 smithi082 ceph-mon[44206]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:17.399 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:17.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:17.484 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:23:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:23:17.511 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:17.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:17.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:17.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:17 smithi159 ceph-mon[45336]: osdmap e339: 12 total, 11 up, 11 in 2024-04-15T18:23:17.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:17 smithi159 ceph-mon[45336]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:17.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:17.653 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:17.681 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:17.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:17 smithi049 ceph-mon[40087]: osdmap e339: 12 total, 11 up, 11 in 2024-04-15T18:23:17.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:17 smithi049 ceph-mon[40087]: from='client.25370 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:17.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:23:17.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:17.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:23:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:18.177 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":932,"stamp":"2024-04-15T18:23:16.236241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2508522040,"num_objects":38271,"num_object_clones":0,"num_object_copies":114813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38271,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":140616,"num_write_kb":5721403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9422,"num_bytes_recovered":619469472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141747,"ondisk_log_size":141747,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":10163524,"kb_used_data":7583312,"kb_used_omap":60,"kb_used_meta":2579907,"kb_avail":1021033148,"statfs":{"total":1055945392128,"available":1045537943552,"internally_reserved":0,"allocated":7765311488,"data_stored":7762116293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61481,"internal_metadata":2641825751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,7,7,7,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-343343104,"num_objects":-5239,"num_object_clones":0,"num_object_copies":-15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-465,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":30474240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.990411"},"pg_stats":[{"pgid":"4.f","version":"337'8785","reported_seq":8942,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.122412+0000","last_change":"2024-04-15T18:21:32.333996+0000","last_active":"2024-04-15T18:23:12.122412+0000","last_peered":"2024-04-15T18:23:12.122412+0000","last_clean":"2024-04-15T18:23:12.122412+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:23:12.122412+0000","last_undegraded":"2024-04-15T18:23:12.122412+0000","last_fullsized":"2024-04-15T18:23:12.122412+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'3320","last_scrub_stamp":"2024-04-15T18:21:32.333909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:21:32.333909+0000","objects_scrubbed":3301,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"337'8477","reported_seq":8651,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.588988+0000","last_change":"2024-04-15T18:22:03.765164+0000","last_active":"2024-04-15T18:23:12.588988+0000","last_peered":"2024-04-15T18:23:12.588988+0000","last_clean":"2024-04-15T18:23:12.588988+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:23:12.588988+0000","last_undegraded":"2024-04-15T18:23:12.588988+0000","last_fullsized":"2024-04-15T18:23:12.588988+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5045","last_scrub_stamp":"2024-04-15T18:22:03.764952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.764952+0000","objects_scrubbed":5010,"log_size":8477,"log_dups_size":0,"ondisk_log_size":8477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:14.312914+0000","scrub_duration":1265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160104448,"num_objects":2443,"num_object_clones":0,"num_object_copies":7329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8477,"num_write_kb":349440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"337'8642","reported_seq":8822,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.547197+0000","last_change":"2024-04-15T18:22:11.794263+0000","last_active":"2024-04-15T18:23:12.547197+0000","last_peered":"2024-04-15T18:23:12.547197+0000","last_clean":"2024-04-15T18:23:12.547197+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:23:12.547197+0000","last_undegraded":"2024-04-15T18:23:12.547197+0000","last_fullsized":"2024-04-15T18:23:12.547197+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5531","last_scrub_stamp":"2024-04-15T18:22:11.794127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.794127+0000","objects_scrubbed":5516,"log_size":8642,"log_dups_size":0,"ondisk_log_size":8642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:35.007436+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158662680,"num_objects":2422,"num_object_clones":0,"num_object_copies":7266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8642,"num_write_kb":353985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"337'8854","reported_seq":9064,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.579695+0000","last_change":"2024-04-15T18:23:08.359021+0000","last_active":"2024-04-15T18:23:12.579695+0000","last_peered":"2024-04-15T18:23:12.579695+0000","last_clean":"2024-04-15T18:23:12.579695+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:23:12.579695+0000","last_undegraded":"2024-04-15T18:23:12.579695+0000","last_fullsized":"2024-04-15T18:23:12.579695+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8637","last_scrub_stamp":"2024-04-15T18:23:08.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:08.358887+0000","objects_scrubbed":2688,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:18.649813+0000","scrub_duration":570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161218560,"num_objects":2460,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"337'8716","reported_seq":8893,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:11.285679+0000","last_change":"2024-04-15T18:22:05.829296+0000","last_active":"2024-04-15T18:23:11.285679+0000","last_peered":"2024-04-15T18:23:11.285679+0000","last_clean":"2024-04-15T18:23:11.285679+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:23:11.285679+0000","last_undegraded":"2024-04-15T18:23:11.285679+0000","last_fullsized":"2024-04-15T18:23:11.285679+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5237","last_scrub_stamp":"2024-04-15T18:22:05.829122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.829122+0000","objects_scrubbed":5187,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.283179+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155451392,"num_objects":2372,"num_object_clones":0,"num_object_copies":7116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8716,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"337'8946","reported_seq":9126,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-15T18:23:15.240628+0000","last_change":"2024-04-15T18:22:05.753884+0000","last_active":"2024-04-15T18:23:15.240628+0000","last_peered":"2024-04-15T18:23:15.240628+0000","last_clean":"2024-04-15T18:23:15.240628+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:23:15.240628+0000","last_undegraded":"2024-04-15T18:23:15.240628+0000","last_fullsized":"2024-04-15T18:23:15.240628+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5298","last_scrub_stamp":"2024-04-15T18:22:05.753792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.753792+0000","objects_scrubbed":5262,"log_size":8946,"log_dups_size":0,"ondisk_log_size":8946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.075803+0000","scrub_duration":1367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153616384,"num_objects":2344,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8946,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"337'8686","reported_seq":8861,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:14.830868+0000","last_change":"2024-04-15T18:22:23.627521+0000","last_active":"2024-04-15T18:23:14.830868+0000","last_peered":"2024-04-15T18:23:14.830868+0000","last_clean":"2024-04-15T18:23:14.830868+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:23:14.830868+0000","last_undegraded":"2024-04-15T18:23:14.830868+0000","last_fullsized":"2024-04-15T18:23:14.830868+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'6038","last_scrub_stamp":"2024-04-15T18:22:23.627256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:23.627256+0000","objects_scrubbed":4987,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:31.345919+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"337'8754","reported_seq":8934,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:14.524158+0000","last_change":"2024-04-15T18:22:11.496215+0000","last_active":"2024-04-15T18:23:14.524158+0000","last_peered":"2024-04-15T18:23:14.524158+0000","last_clean":"2024-04-15T18:23:14.524158+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:23:14.524158+0000","last_undegraded":"2024-04-15T18:23:14.524158+0000","last_fullsized":"2024-04-15T18:23:14.524158+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5535","last_scrub_stamp":"2024-04-15T18:22:11.496110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.496110+0000","objects_scrubbed":5505,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.883032+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"337'8607","reported_seq":8723,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.164376+0000","last_change":"2024-04-15T18:20:31.649944+0000","last_active":"2024-04-15T18:23:12.164376+0000","last_peered":"2024-04-15T18:23:12.164376+0000","last_clean":"2024-04-15T18:23:12.164376+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:23:12.164376+0000","last_undegraded":"2024-04-15T18:23:12.164376+0000","last_fullsized":"2024-04-15T18:23:12.164376+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155648000,"num_objects":2375,"num_object_clones":0,"num_object_copies":7125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8607,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"337'8650","reported_seq":18386,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.322182+0000","last_change":"2024-04-15T18:23:09.475875+0000","last_active":"2024-04-15T18:23:12.322182+0000","last_peered":"2024-04-15T18:23:12.322182+0000","last_clean":"2024-04-15T18:23:12.322182+0000","last_became_active":"2024-04-15T18:23:06.512970+0000","last_became_peered":"2024-04-15T18:23:06.512970+0000","last_unstale":"2024-04-15T18:23:12.322182+0000","last_undegraded":"2024-04-15T18:23:12.322182+0000","last_fullsized":"2024-04-15T18:23:12.322182+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8513","last_scrub_stamp":"2024-04-15T18:23:09.475588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:09.475588+0000","objects_scrubbed":2599,"log_size":8650,"log_dups_size":0,"ondisk_log_size":8650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:22.506266+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160038912,"num_objects":2442,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4788,"num_bytes_recovered":313786368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"337'8879","reported_seq":18251,"reported_epoch":337,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:23:12.168657+0000","last_change":"2024-04-15T18:21:57.762236+0000","last_active":"2024-04-15T18:23:12.168657+0000","last_peered":"2024-04-15T18:23:12.168657+0000","last_clean":"2024-04-15T18:21:37.153258+0000","last_became_active":"2024-04-15T18:21:57.737029+0000","last_became_peered":"2024-04-15T18:21:57.737029+0000","last_unstale":"2024-04-15T18:23:12.168657+0000","last_undegraded":"2024-04-15T18:21:57.302505+0000","last_fullsized":"2024-04-15T18:21:57.216461+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155385856,"num_objects":2371,"num_object_clones":0,"num_object_copies":7113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8879,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4624,"num_bytes_recovered":303038464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,10],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":2116}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"337'9013","reported_seq":9190,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:14.835391+0000","last_change":"2024-04-15T18:22:03.640303+0000","last_active":"2024-04-15T18:23:14.835391+0000","last_peered":"2024-04-15T18:23:14.835391+0000","last_clean":"2024-04-15T18:23:14.835391+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:23:14.835391+0000","last_undegraded":"2024-04-15T18:23:14.835391+0000","last_fullsized":"2024-04-15T18:23:14.835391+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5259","last_scrub_stamp":"2024-04-15T18:22:03.640098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.640098+0000","objects_scrubbed":5226,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:19.907355+0000","scrub_duration":1294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154861568,"num_objects":2363,"num_object_clones":0,"num_object_copies":7089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"337'8806","reported_seq":8956,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.400988+0000","last_change":"2024-04-15T18:23:05.903965+0000","last_active":"2024-04-15T18:23:12.400988+0000","last_peered":"2024-04-15T18:23:12.400988+0000","last_clean":"2024-04-15T18:23:12.400988+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:23:12.400988+0000","last_undegraded":"2024-04-15T18:23:12.400988+0000","last_fullsized":"2024-04-15T18:23:12.400988+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'8504","last_scrub_stamp":"2024-04-15T18:23:05.903829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.903829+0000","objects_scrubbed":2750,"log_size":8806,"log_dups_size":0,"ondisk_log_size":8806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:17.161701+0000","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160169984,"num_objects":2444,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8806,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"337'9144","reported_seq":9323,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:14.788267+0000","last_change":"2024-04-15T18:22:19.093291+0000","last_active":"2024-04-15T18:23:14.788267+0000","last_peered":"2024-04-15T18:23:14.788267+0000","last_clean":"2024-04-15T18:23:14.788267+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:23:14.788267+0000","last_undegraded":"2024-04-15T18:23:14.788267+0000","last_fullsized":"2024-04-15T18:23:14.788267+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6103","last_scrub_stamp":"2024-04-15T18:22:19.093022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:19.093022+0000","objects_scrubbed":5424,"log_size":9144,"log_dups_size":0,"ondisk_log_size":9144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:27.551120+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153616384,"num_objects":2344,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9144,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"337'8632","reported_seq":8747,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-04-15T18:23:12.525088+0000","last_change":"2024-04-15T18:20:31.186380+0000","last_active":"2024-04-15T18:23:12.525088+0000","last_peered":"2024-04-15T18:23:12.525088+0000","last_clean":"2024-04-15T18:23:12.525088+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:23:12.525088+0000","last_undegraded":"2024-04-15T18:23:12.525088+0000","last_fullsized":"2024-04-15T18:23:12.525088+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":8632,"log_dups_size":0,"ondisk_log_size":8632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159514624,"num_objects":2434,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8632,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"338'11019","reported_seq":11327,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-15T18:23:15.271709+0000","last_change":"2024-04-15T18:23:14.957703+0000","last_active":"2024-04-15T18:23:15.271709+0000","last_peered":"2024-04-15T18:23:15.271709+0000","last_clean":"2024-04-15T18:23:15.271709+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:23:15.271709+0000","last_undegraded":"2024-04-15T18:23:15.271709+0000","last_fullsized":"2024-04-15T18:23:15.271709+0000","mapping_epoch":284,"log_start":"292'1000","ondisk_log_start":"292'1000","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'10999","last_scrub_stamp":"2024-04-15T18:23:14.957443+0000","last_deep_scrub":"311'6222","last_deep_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_clean_scrub_stamp":"2024-04-15T18:23:14.957443+0000","objects_scrubbed":2362,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:31.738507+0000","scrub_duration":842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":6978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":896,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-15T18:23:15.234097+0000","last_change":"2024-04-15T18:23:05.604093+0000","last_active":"2024-04-15T18:23:15.234097+0000","last_peered":"2024-04-15T18:23:15.234097+0000","last_clean":"2024-04-15T18:23:15.234097+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:23:15.234097+0000","last_undegraded":"2024-04-15T18:23:15.234097+0000","last_fullsized":"2024-04-15T18:23:15.234097+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:23:05.603936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.603936+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:30.294200+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2507931672,"num_objects":38269,"num_object_clones":0,"num_object_copies":114807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140435,"num_write_kb":5718465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9412,"num_bytes_recovered":616824832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7929147392,"data_stored":7929135176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141610,"ondisk_log_size":141610,"up":48,"acting":47,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925994,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1421680,"kb_used_data":1111464,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92323472,"statfs":{"total":95995035648,"available":94539235328,"internally_reserved":0,"allocated":1138139136,"data_stored":1137847588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":317651396},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187629,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917096,"kb_used_data":759456,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92828056,"statfs":{"total":95995035648,"available":95055929344,"internally_reserved":0,"allocated":777682944,"data_stored":777394968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":161282496},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514674,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687472,"kb_used_data":475812,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93057680,"statfs":{"total":95995035648,"available":95291064320,"internally_reserved":0,"allocated":487231488,"data_stored":486939440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":216725952},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874423,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122156,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622996,"statfs":{"total":95995035648,"available":95869947904,"internally_reserved":0,"allocated":753664,"data_stored":465688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168764,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829888,"kb_used_data":591860,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92915264,"statfs":{"total":95995035648,"available":95145230336,"internally_reserved":0,"allocated":606064640,"data_stored":605768545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":243726794},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463105,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216624,"kb_used_data":960552,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92528528,"statfs":{"total":95995035648,"available":94749212672,"internally_reserved":0,"allocated":983605248,"data_stored":983313720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1732080,"kb_used_data":1426020,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92013072,"statfs":{"total":95995035648,"available":94221385728,"internally_reserved":0,"allocated":1460244480,"data_stored":1459952432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738591,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":725932,"kb_used_data":469856,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93019220,"statfs":{"total":95995035648,"available":95251681280,"internally_reserved":0,"allocated":481132544,"data_stored":480844568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608972,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099884,"kb_used_data":786272,"kb_used_omap":44,"kb_used_meta":313555,"kb_avail":92645268,"statfs":{"total":95995035648,"available":94868754432,"internally_reserved":0,"allocated":805142528,"data_stored":804854552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":321080874},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986378,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966896,"kb_used_data":696932,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92778256,"statfs":{"total":95995035648,"available":95004934144,"internally_reserved":0,"allocated":713658368,"data_stored":713366320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724741,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":443816,"kb_used_data":304352,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93301336,"statfs":{"total":95995035648,"available":95540568064,"internally_reserved":0,"allocated":311656448,"data_stored":311368472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476708864,"data_stored":476708864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1435308032,"data_stored":1435303960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442302464,"data_stored":442302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":795475968,"data_stored":795475968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":631443456,"data_stored":631439384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":307625984,"data_stored":307625984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":913047552,"data_stored":913047552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":604372992,"data_stored":604372992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":479203328,"data_stored":479199256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":770572288,"data_stored":770572288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1073086464,"data_stored":1073086464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:23:18.178 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.4 is not active+clean 2024-04-15T18:23:18.178 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.4', 'version': "337'8879", 'reported_seq': 18251, 'reported_epoch': 337, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:23:12.168657+0000', 'last_change': '2024-04-15T18:21:57.762236+0000', 'last_active': '2024-04-15T18:23:12.168657+0000', 'last_peered': '2024-04-15T18:23:12.168657+0000', 'last_clean': '2024-04-15T18:21:37.153258+0000', 'last_became_active': '2024-04-15T18:21:57.737029+0000', 'last_became_peered': '2024-04-15T18:21:57.737029+0000', 'last_unstale': '2024-04-15T18:23:12.168657+0000', 'last_undegraded': '2024-04-15T18:21:57.302505+0000', 'last_fullsized': '2024-04-15T18:21:57.216461+0000', 'mapping_epoch': 318, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 284, 'last_epoch_clean': 285, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:20:30.170849+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:20:30.170849+0000', 'last_clean_scrub_stamp': '2024-04-15T18:20:30.170849+0000', 'objects_scrubbed': 0, 'log_size': 8879, 'log_dups_size': 0, 'ondisk_log_size': 8879, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 155385856, 'num_objects': 2371, 'num_object_clones': 0, 'num_object_copies': 7113, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8879, 'num_write_kb': 360000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4624, 'num_bytes_recovered': 303038464, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 10], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [{'shards': '1,3', 'objects': 2116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T18:23:18.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:23:18.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:18 smithi082 ceph-mon[44206]: 4.e scrub starts 2024-04-15T18:23:18.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:18 smithi082 ceph-mon[44206]: pgmap v933: 17 pgs: 17 active+clean; 2.3 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:23:18.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:18 smithi082 ceph-mon[44206]: 4.e scrub ok 2024-04-15T18:23:18.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:18 smithi049 ceph-mon[40087]: 4.e scrub starts 2024-04-15T18:23:18.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:18 smithi049 ceph-mon[40087]: pgmap v933: 17 pgs: 17 active+clean; 2.3 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:23:18.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:18 smithi049 ceph-mon[40087]: 4.e scrub ok 2024-04-15T18:23:18.405 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:18.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:18.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:18.547 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:18.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:18 smithi159 ceph-mon[45336]: 4.e scrub starts 2024-04-15T18:23:18.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:18 smithi159 ceph-mon[45336]: pgmap v933: 17 pgs: 17 active+clean; 2.3 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:23:18.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:18 smithi159 ceph-mon[45336]: 4.e scrub ok 2024-04-15T18:23:18.697 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:18.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:18.733 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:18.749 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:18.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:18.846 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:18.901 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:18.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:19.004 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:19.004 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:19.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:19.034 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:19.059 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:19.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:19.164 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:19.210 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:19.242 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:19.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:19.274 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:23:19.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: 4.0 scrub starts 2024-04-15T18:23:19.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:23:19.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:23:19.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: 4.4 scrub starts 2024-04-15T18:23:19.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/305524736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:19.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:19 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:19.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:19.398 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: 4.0 scrub starts 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: 4.4 scrub starts 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/305524736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:19.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:19 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:19.706 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: 4.0 scrub starts 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 255/116283 objects degraded (0.219%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: 4.4 scrub starts 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/305524736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:19 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:20.102 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: 4.0 scrub ok 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: 4.4 scrub ok 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: pgmap v934: 17 pgs: 17 active+clean; 2.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: 4.7 scrub starts 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:20.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:20 smithi082 ceph-mon[44206]: osdmap e340: 12 total, 11 up, 11 in 2024-04-15T18:23:20.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: 4.0 scrub ok 2024-04-15T18:23:20.536 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: 4.4 scrub ok 2024-04-15T18:23:20.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: pgmap v934: 17 pgs: 17 active+clean; 2.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-15T18:23:20.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:20.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: 4.7 scrub starts 2024-04-15T18:23:20.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:20.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:20 smithi049 ceph-mon[40087]: osdmap e340: 12 total, 11 up, 11 in 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: 4.0 scrub ok 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: 4.4 scrub ok 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: pgmap v934: 17 pgs: 17 active+clean; 2.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: 4.7 scrub starts 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:20.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:20 smithi159 ceph-mon[45336]: osdmap e340: 12 total, 11 up, 11 in 2024-04-15T18:23:20.973 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:20.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:21.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:21.122 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:21.287 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:21.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:21.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:21.434 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:21.513 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:21.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:21.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:21.648 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:21.862 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:21.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:21.970 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:22.010 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:22.034 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:22.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:22.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:22.175 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:22.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:22.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:22.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:22.443 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:22.527 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:22.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:22.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:22.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:22.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:22.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:22.634 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:22.636 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:22.677 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:22.679 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:22.706 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:22.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:22.724 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:22.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:22.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:22.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:22.841 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:22.859 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:22.859 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:22.859 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:22.861 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:22.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:22.948 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:23:22.948 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:23:22.948 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:22.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:23.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:23.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:23.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:23.130 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:23.172 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:23.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: pgmap v936: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 2.6 MiB/s, 42 objects/s recovering 2024-04-15T18:23:23.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2910710490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:23.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:23:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.196 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:23.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:23.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:23:23.268 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1150,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":341,"num_osds":12,"num_up_osds":11,"osd_up_since":1713205297,"num_in_osds":11,"osd_in_since":1713205315,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":34986,"data_bytes":2293236280,"bytes_used":9779191808,"bytes_avail":1046166200320,"bytes_total":1055945392128,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":2777743,"recovering_keys_per_sec":0,"num_objects_recovered":256,"num_bytes_recovered":16711680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":627},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":28,"modified":"2024-04-15T18:23:01.244316+0000","services":{"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","7":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: pgmap v936: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 2.6 MiB/s, 42 objects/s recovering 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2910710490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:23:23.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:23:23.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:23.337 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:23.367 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:23.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:23.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:23.512 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:23.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:23.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:23.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:23.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: pgmap v936: 17 pgs: 17 active+clean; 2.1 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 2.6 MiB/s, 42 objects/s recovering 2024-04-15T18:23:23.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2910710490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:23:23.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:23.656 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:23:23.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:23.755 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:23.784 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:23.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:23.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:23:23.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:23.934 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:24.001 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:24.285 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:24.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:24.297 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:23:24.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:23:24.392 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:24.397 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:24.425 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: osdmap e341: 12 total, 11 up, 11 in 2024-04-15T18:23:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2713361283' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:23:24.486 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: osdmap e341: 12 total, 11 up, 11 in 2024-04-15T18:23:24.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2713361283' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:23:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:24.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:24.590 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:24.635 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: osdmap e341: 12 total, 11 up, 11 in 2024-04-15T18:23:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2713361283' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:23:25.290 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:25.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:25.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:25.447 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:25 smithi049 ceph-mon[40087]: pgmap v938: 17 pgs: 17 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:23:25.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:25 smithi082 ceph-mon[44206]: pgmap v938: 17 pgs: 17 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:23:25.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:23:25.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:25 smithi159 ceph-mon[45336]: pgmap v938: 17 pgs: 17 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 833 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:23:25.746 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:25.808 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:25.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:25.837 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:25.838 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:23:25.838 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:23:25.838 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:23:25.838 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:25.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:25.964 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:26.120 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:26.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:26.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:26.254 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:26.277 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:26.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:26.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:26.440 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:26.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:23:27.085 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:23:27.262 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:27 smithi049 ceph-mon[40087]: pgmap v939: 17 pgs: 17 active+clean; 1.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:23:27.262 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:27.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:27 smithi082 ceph-mon[44206]: pgmap v939: 17 pgs: 17 active+clean; 1.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:23:27.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:27.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:27 smithi159 ceph-mon[45336]: pgmap v939: 17 pgs: 17 active+clean; 1.9 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:23:27.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:27.707 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:23:28.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:28 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:28.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:28.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:28 smithi082 ceph-mon[44206]: osdmap e342: 12 total, 11 up, 11 in 2024-04-15T18:23:28.499 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:28 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:28.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:28.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:28 smithi049 ceph-mon[40087]: osdmap e342: 12 total, 11 up, 11 in 2024-04-15T18:23:28.501 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:28.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:28.513 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:28.610 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:28.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:28 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:28.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/732718031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:28.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:28 smithi159 ceph-mon[45336]: osdmap e342: 12 total, 11 up, 11 in 2024-04-15T18:23:28.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:28.666 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:28.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:28.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:28.807 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:28.833 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:28.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:28.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:29.004 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24511.0:135361 2.0 2:08a010d6:::benchmark_data_smithi159_80260_object46248:head [delete] snapc 0=[] ondisk+write+known_if_redirected e172)", 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:07.533916+0000", 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 561.47300899300001, 2024-04-15T18:23:29.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22979191400000001, 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24511", 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 135361 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533916+0000", 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533916+0000", 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533917+0000", 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.71e-07 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533922+0000", 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2899999999999996e-06 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533932+0000", 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0764000000000001e-05 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533938+0000", 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8839999999999999e-06 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.533980+0000", 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2074999999999999e-05 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.534063+0000", 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2169000000000006e-05 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.534130+0000", 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7459000000000006e-05 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537290+0000", 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003160174 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537297+0000", 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5130000000000002e-06 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763644+0000", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22634754400000001 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763677+0000", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2784000000000003e-05 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763708+0000", 2024-04-15T18:23:29.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1387000000000003e-05 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.018 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.24511.0:135361 2.0 e172/114 ondisk, result = 0)", 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:07.537556+0000", 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 561.46936955199999, 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22615627399999999, 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537556+0000", 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537556+0000", 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537556+0000", 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2e-07 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.537560+0000", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.315e-06 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763602+0000", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.226042459 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763607+0000", 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3549999999999998e-06 2024-04-15T18:23:29.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763636+0000", 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9804000000000001e-05 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763641+0000", 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4610000000000001e-06 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:07.763712+0000", 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1359999999999994e-05 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:156377 2.3 e177/114 2:c8c3c9ce:::benchmark_data_smithi159_80260_object67264:head v 177'9962, mlcod=177'9958)", 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:36.482512+0000", 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 532.52441388299997, 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19978027100000001, 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.023 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.482512+0000", 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.482512+0000", 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.482512+0000", 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0900000000000001e-07 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.482517+0000", 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1510000000000003e-06 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.677801+0000", 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19528404599999999 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.677808+0000", 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.99e-06 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.677862+0000", 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4088000000000001e-05 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.677886+0000", 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3357999999999999e-05 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.682247+0000", 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043608199999999996 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.682292+0000", 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5209e-05 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.028 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24511.0:156378 2.3 e177/114 2:c6cc3ec9:::benchmark_data_smithi159_80260_object67265:head v 177'9963, mlcod=177'9958)", 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:14:36.486152+0000", 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 532.52077340999995, 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.205533048, 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.486152+0000", 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.486152+0000", 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.486153+0000", 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1069999999999999e-06 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.486160+0000", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7900000000000002e-06 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.688040+0000", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20187983900000001 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.688046+0000", 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7150000000000003e-06 2024-04-15T18:23:29.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.688089+0000", 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3881000000000002e-05 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.688106+0000", 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6786000000000002e-05 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.691650+0000", 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003544307 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:14:36.691685+0000", 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4622999999999997e-05 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.24793.0:15742 3.9 e205/199 3:98ecea81:::benchmark_data_smithi159_91621_object15741:head v 205'936, mlcod=205'934)", 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:15:57.612484+0000", 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 451.394441798, 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21010641899999999, 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.612484+0000", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.612484+0000", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.612484+0000", 2024-04-15T18:23:29.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1999999999999999e-07 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.612539+0000", 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4574999999999998e-05 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.821694+0000", 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20915557700000001 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.821702+0000", 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.464e-06 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.821771+0000", 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9474000000000006e-05 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.821794+0000", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2612999999999998e-05 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.822553+0000", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075864400000000005 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:15:57.822590+0000", 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7452e-05 2024-04-15T18:23:29.038 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121149 3.d 3:bf94904e:::benchmark_data_smithi159_91621_object14673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 346.88838960599998, 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25102950499999999, 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.039 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121149 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118537+0000", 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2309999999999999e-06 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118542+0000", 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9400000000000001e-06 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.041 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118552+0000", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.044e-05 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118561+0000", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7979999999999995e-06 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118604+0000", 2024-04-15T18:23:29.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3111000000000001e-05 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118765+0000", 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016077999999999999 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118879+0000", 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011379600000000001 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.122472+0000", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035933610000000002 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.131902+0000", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0094298760000000002 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369472+0000", 2024-04-15T18:23:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23756957000000001 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369526+0000", 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3842999999999999e-05 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369565+0000", 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759000000000003e-05 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.045 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121171 3.d 3:b5145494:::benchmark_data_smithi159_91621_object14695:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 346.83844973100003, 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.201212849, 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.046 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121171 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168477+0000", 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6840000000000001e-06 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168484+0000", 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6630000000000004e-06 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168501+0000", 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6441000000000001e-05 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168509+0000", 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6619999999999994e-06 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168562+0000", 2024-04-15T18:23:29.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3244e-05 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168700+0000", 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000137231 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168833+0000", 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013358200000000001 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.171844+0000", 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030106540000000002 2024-04-15T18:23:29.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.188649+0000", 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016804927000000001 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369603+0000", 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18095407099999999 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369650+0000", 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7243000000000002e-05 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369689+0000", 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8447e-05 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.056 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121461 3.f 3:f41f1665:::benchmark_data_smithi159_91621_object14985:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 346.12007603299998, 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19883951899999999, 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.057 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121461 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.058 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886851+0000", 2024-04-15T18:23:29.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2580000000000001e-06 2024-04-15T18:23:29.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886857+0000", 2024-04-15T18:23:29.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9100000000000002e-06 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886870+0000", 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2928999999999999e-05 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886878+0000", 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9629999999999998e-06 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886929+0000", 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.126e-05 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.887062+0000", 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013309800000000001 2024-04-15T18:23:29.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 8,11", 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.887193+0000", 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000131502 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.890852+0000", 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036585580000000001 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.890892+0000", 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9793999999999999e-05 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085577+0000", 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19468490299999999 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085637+0000", 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0449999999999999e-05 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085689+0000", 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1894000000000002e-05 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.066 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:144275 3.d 3:b0290c5b:::benchmark_data_smithi159_91621_object37799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 306.84176042500002, 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203547589, 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.067 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 144275 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165166+0000", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2e-07 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165169+0000", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.27e-06 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165176+0000", 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:23:29.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165182+0000", 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165235+0000", 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2642000000000002e-05 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165361+0000", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012657799999999999 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165472+0000", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111254 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.169817+0000", 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043446539999999999 2024-04-15T18:23:29.071 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.170617+0000", 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00079937499999999996 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368612+0000", 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.197995223 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.072 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368666+0000", 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4616e-05 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368713+0000", 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6276999999999998e-05 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.073 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:144542 3.d 3:ba20a02c:::benchmark_data_smithi159_91621_object38066:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 306.38373534599998, 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.205631183, 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 144542 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.074 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623191+0000", 2024-04-15T18:23:29.075 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4399999999999999e-07 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623195+0000", 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1659999999999996e-06 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623206+0000", 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1415e-05 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.076 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623213+0000", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8349999999999998e-06 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626803+0000", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035900490000000001 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626920+0000", 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116906 2024-04-15T18:23:29.077 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626997+0000", 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7090999999999998e-05 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.078 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.630647+0000", 2024-04-15T18:23:29.079 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036496509999999999 2024-04-15T18:23:29.079 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.079 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.079 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.079 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.630721+0000", 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4012999999999998e-05 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828707+0000", 2024-04-15T18:23:29.080 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19798600699999999 2024-04-15T18:23:29.081 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.081 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.081 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.081 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828776+0000", 2024-04-15T18:23:29.081 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8727999999999997e-05 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828821+0000", 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5578000000000003e-05 2024-04-15T18:23:29.082 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.083 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.083 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.083 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.083 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.083 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:185262 3.d 3:b4a17326:::benchmark_data_smithi159_91621_object78786:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 236.70904511000001, 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23775624300000001, 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.084 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185262 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.085 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297882+0000", 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.733e-06 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297889+0000", 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3819999999999998e-06 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297906+0000", 2024-04-15T18:23:29.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6565000000000001e-05 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297915+0000", 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1519999999999998e-06 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297975+0000", 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0296e-05 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.298116+0000", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.298242+0000", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000126458 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.303601+0000", 2024-04-15T18:23:29.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0053590319999999997 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.310158+0000", 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006557149 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535525+0000", 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22536619899999999 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535586+0000", 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1457999999999993e-05 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535637+0000", 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0547999999999998e-05 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:212946 3.d 3:b1715a3b:::benchmark_data_smithi159_91621_object106470:head [delete] snapc 0=[] ondisk+write+known_if_redirected e275)", 2024-04-15T18:23:29.092 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.00388230799999, 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29758224900000002, 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 212946 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003044+0000", 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.19e-06 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003050+0000", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7379999999999999e-06 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003063+0000", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2772000000000001e-05 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003072+0000", 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6980000000000004e-06 2024-04-15T18:23:29.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003128+0000", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6187000000000003e-05 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003250+0000", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122142 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003346+0000", 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5867999999999998e-05 2024-04-15T18:23:29.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008011+0000", 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046649550000000001 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008401+0000", 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039006300000000001 2024-04-15T18:23:29.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300548+0000", 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29214726699999999 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300587+0000", 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9243999999999999e-05 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.098 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300625+0000", 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8124999999999998e-05 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:15519 4.3 e292/284 4:cd27a75b:::benchmark_data_smithi159_102124_object15518:head v 292'960, mlcod=292'958)", 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 158.58290130500001, 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.212814323, 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.099 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424025+0000", 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6899999999999997e-07 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424047+0000", 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2731e-05 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636060+0000", 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21201231600000001 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636066+0000", 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8560000000000003e-06 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636119+0000", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3171999999999998e-05 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636141+0000", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1848000000000002e-05 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636805+0000", 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066410000000000004 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636838+0000", 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3730999999999999e-05 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:89299 4.0 e322/284 4:078bede3:::benchmark_data_smithi159_102124_object89298:head v 322'5533, mlcod=322'5530)", 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 77.642579413999997, 2024-04-15T18:23:29.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53392096499999997, 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364347+0000", 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.004e-06 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364416+0000", 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8975999999999998e-05 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895471+0000", 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53105475999999996 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895477+0000", 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1619999999999996e-06 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895526+0000", 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9008999999999998e-05 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895549+0000", 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3104000000000001e-05 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898235+0000", 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026861400000000001 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898267+0000", 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1810000000000002e-05 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(4.6 322/318 [PushOp(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head, version: 292'910, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head@292'910, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 76.293746467999995, 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47774007400000001, 2024-04-15T18:23:29.109 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:23:29.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713180+0000", 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713250+0000", 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0133000000000001e-05 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189486+0000", 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47623641300000003 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189505+0000", 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8240000000000002e-05 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189590+0000", 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5097000000000002e-05 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189600+0000", 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0101e-05 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.190919+0000", 2024-04-15T18:23:29.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001319204 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(4.6 328/318 [PushOp(4:695a50f3:::benchmark_data_smithi159_102124_object46694:head, version: 304'2886, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:695a50f3:::benchmark_data_smithi159_102124_object46694:head@304'2886, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:41.984364+0000", 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 47.022561201999999, 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.098843840000000002, 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.114 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:41.984364+0000", 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:41.984364+0000", 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:41.984365+0000", 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9400000000000004e-07 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:41.984428+0000", 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3034999999999995e-05 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:42.081002+0000", 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.096574187000000006 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:42.081020+0000", 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7839999999999999e-05 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:42.081112+0000", 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2033000000000005e-05 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:42.081126+0000", 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3526000000000001e-05 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:42.083208+0000", 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020823249999999999 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:134866 4.8 e335/284 4:1fd83f8c:::benchmark_data_smithi159_102124_object45512:head v 335'8348, mlcod=335'8346)", 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 23.128050938000001, 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21941387600000001, 2024-04-15T18:23:29.119 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7000000000000004e-07 2024-04-15T18:23:29.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878880+0000", 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0379999999999999e-06 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.094952+0000", 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216071508 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.094958+0000", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7760000000000003e-06 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.095008+0000", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0250000000000002e-05 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.095035+0000", 2024-04-15T18:23:29.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7151000000000001e-05 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.098260+0000", 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032249570000000001 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.098288+0000", 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8526000000000001e-05 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:136570 4.0 e336/284 4:0b6648ef:::benchmark_data_smithi159_102124_object47216:head v 336'8436, mlcod=336'8433)", 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:08.269990+0000", 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 20.736935811999999, 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.154858949, 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.269990+0000", 2024-04-15T18:23:29.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.269990+0000", 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.269990+0000", 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4000000000000001e-07 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.269995+0000", 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1900000000000003e-06 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.421530+0000", 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15153409700000001 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.421554+0000", 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4593e-05 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.421604+0000", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9867999999999998e-05 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.421624+0000", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9465000000000001e-05 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.424808+0000", 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031842519999999998 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:08.424849+0000", 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0843999999999997e-05 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:138890 4.0 e337/284 4:06b62592:::benchmark_data_smithi159_102124_object49536:head v 337'8576, mlcod=337'8572)", 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:11.247022+0000", 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.759903529999999, 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.09216684, 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.247022+0000", 2024-04-15T18:23:29.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.247022+0000", 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.247023+0000", 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5099999999999998e-07 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.131 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.247030+0000", 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0389999999999999e-06 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.335059+0000", 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.088029069000000001 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.335067+0000", 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6120000000000001e-06 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.335119+0000", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2865999999999998e-05 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.335141+0000", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1041999999999999e-05 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.339140+0000", 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039991699999999998 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.339189+0000", 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9091000000000001e-05 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:139383 4.6 e337/335 4:69e988dc:::benchmark_data_smithi159_102124_object50029:head v 337'8630, mlcod=337'8627)", 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:11.811341+0000", 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 17.195584706999998, 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.091065454000000004, 2024-04-15T18:23:29.134 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.811341+0000", 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.811341+0000", 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.811342+0000", 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0699999999999996e-07 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.811348+0000", 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6089999999999997e-06 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:23:29.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.899835+0000", 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.088486475999999994 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.899842+0000", 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7100000000000007e-06 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.899890+0000", 2024-04-15T18:23:29.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7763999999999999e-05 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.899907+0000", 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6614000000000001e-05 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:23:29.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.902375+0000", 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024680230000000002 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:11.902406+0000", 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1350999999999998e-05 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:23:29.139 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:23:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:29.141 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:29.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:29.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:29.277 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:29.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:29 smithi082 ceph-mon[44206]: pgmap v941: 17 pgs: 17 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-04-15T18:23:29.303 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:29.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:29.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:29.440 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:29.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:29 smithi049 ceph-mon[40087]: pgmap v941: 17 pgs: 17 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-04-15T18:23:29.613 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:23:29.622 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:23:29.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:29 smithi159 ceph-mon[45336]: pgmap v941: 17 pgs: 17 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 793 op/s 2024-04-15T18:23:29.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:23:29.798 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:29.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:29.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:29.938 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:29.967 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:29.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:30.071 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:30.108 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:30.133 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:30.164 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:30.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:30.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:30.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:30 smithi049 ceph-mon[40087]: pgmap v942: 17 pgs: 17 active+clean; 1.7 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-04-15T18:23:30.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:30 smithi049 ceph-mon[40087]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:30.328 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:30.363 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:30.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:30.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:30 smithi159 ceph-mon[45336]: pgmap v942: 17 pgs: 17 active+clean; 1.7 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-04-15T18:23:30.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:30 smithi159 ceph-mon[45336]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:30.411 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":942,"stamp":"2024-04-15T18:23:29.250154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1862926904,"num_objects":28420,"num_object_clones":0,"num_object_copies":85260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28420,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":150467,"num_write_kb":5721403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9678,"num_bytes_recovered":636181152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151198,"ondisk_log_size":151198,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":8265948,"kb_used_data":5666292,"kb_used_omap":60,"kb_used_meta":2599363,"kb_avail":1022930724,"statfs":{"total":1055945392128,"available":1047481061376,"internally_reserved":0,"allocated":5802283008,"data_stored":5799444866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61481,"internal_metadata":2661748695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,4,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-430309376,"num_objects":-6566,"num_object_clones":0,"num_object_copies":-19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6566,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001799"},"pg_stats":[{"pgid":"4.f","version":"341'9526","reported_seq":9691,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.180617+0000","last_change":"2024-04-15T18:21:32.333996+0000","last_active":"2024-04-15T18:23:27.180617+0000","last_peered":"2024-04-15T18:23:27.180617+0000","last_clean":"2024-04-15T18:23:27.180617+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:23:27.180617+0000","last_undegraded":"2024-04-15T18:23:27.180617+0000","last_fullsized":"2024-04-15T18:23:27.180617+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'3320","last_scrub_stamp":"2024-04-15T18:21:32.333909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:21:32.333909+0000","objects_scrubbed":3301,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.278232+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115343360,"num_objects":1760,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"342'9175","reported_seq":9388,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.581607+0000","last_change":"2024-04-15T18:23:17.585526+0000","last_active":"2024-04-15T18:23:27.581607+0000","last_peered":"2024-04-15T18:23:27.581607+0000","last_clean":"2024-04-15T18:23:27.581607+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:23:27.581607+0000","last_undegraded":"2024-04-15T18:23:27.581607+0000","last_fullsized":"2024-04-15T18:23:27.581607+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8692","last_scrub_stamp":"2024-04-15T18:23:17.585204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:17.585204+0000","objects_scrubbed":2251,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:39.572162+0000","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114360320,"num_objects":1745,"num_object_clones":0,"num_object_copies":5235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9175,"num_write_kb":349440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"342'9366","reported_seq":9555,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.577118+0000","last_change":"2024-04-15T18:22:11.794263+0000","last_active":"2024-04-15T18:23:27.577118+0000","last_peered":"2024-04-15T18:23:27.577118+0000","last_clean":"2024-04-15T18:23:27.577118+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:23:27.577118+0000","last_undegraded":"2024-04-15T18:23:27.577118+0000","last_fullsized":"2024-04-15T18:23:27.577118+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5531","last_scrub_stamp":"2024-04-15T18:22:11.794127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.794127+0000","objects_scrubbed":5516,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:35.007436+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214616,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9366,"num_write_kb":353985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"342'9548","reported_seq":9768,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.498606+0000","last_change":"2024-04-15T18:23:08.359021+0000","last_active":"2024-04-15T18:23:27.498606+0000","last_peered":"2024-04-15T18:23:27.498606+0000","last_clean":"2024-04-15T18:23:27.498606+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:23:27.498606+0000","last_undegraded":"2024-04-15T18:23:27.498606+0000","last_fullsized":"2024-04-15T18:23:27.498606+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8637","last_scrub_stamp":"2024-04-15T18:23:08.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:08.358887+0000","objects_scrubbed":2688,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:18.649813+0000","scrub_duration":570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115736576,"num_objects":1766,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9548,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"341'9367","reported_seq":9551,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:26.289591+0000","last_change":"2024-04-15T18:22:05.829296+0000","last_active":"2024-04-15T18:23:26.289591+0000","last_peered":"2024-04-15T18:23:26.289591+0000","last_clean":"2024-04-15T18:23:26.289591+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:23:26.289591+0000","last_undegraded":"2024-04-15T18:23:26.289591+0000","last_fullsized":"2024-04-15T18:23:26.289591+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5237","last_scrub_stamp":"2024-04-15T18:22:05.829122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.829122+0000","objects_scrubbed":5187,"log_size":9367,"log_dups_size":0,"ondisk_log_size":9367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.283179+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"341'9429","reported_seq":9615,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:25.213362+0000","last_change":"2024-04-15T18:22:05.753884+0000","last_active":"2024-04-15T18:23:25.213362+0000","last_peered":"2024-04-15T18:23:25.213362+0000","last_clean":"2024-04-15T18:23:25.213362+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:23:25.213362+0000","last_undegraded":"2024-04-15T18:23:25.213362+0000","last_fullsized":"2024-04-15T18:23:25.213362+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5298","last_scrub_stamp":"2024-04-15T18:22:05.753792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.753792+0000","objects_scrubbed":5262,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.075803+0000","scrub_duration":1367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121962496,"num_objects":1861,"num_object_clones":0,"num_object_copies":5583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"341'9129","reported_seq":9312,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:24.859913+0000","last_change":"2024-04-15T18:22:23.627521+0000","last_active":"2024-04-15T18:23:24.859913+0000","last_peered":"2024-04-15T18:23:24.859913+0000","last_clean":"2024-04-15T18:23:24.859913+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:23:24.859913+0000","last_undegraded":"2024-04-15T18:23:24.859913+0000","last_fullsized":"2024-04-15T18:23:24.859913+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'6038","last_scrub_stamp":"2024-04-15T18:22:23.627256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:23.627256+0000","objects_scrubbed":4987,"log_size":9129,"log_dups_size":0,"ondisk_log_size":9129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:31.345919+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122486784,"num_objects":1869,"num_object_clones":0,"num_object_copies":5607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9129,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"341'9222","reported_seq":9409,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:24.859491+0000","last_change":"2024-04-15T18:22:11.496215+0000","last_active":"2024-04-15T18:23:24.859491+0000","last_peered":"2024-04-15T18:23:24.859491+0000","last_clean":"2024-04-15T18:23:24.859491+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:23:24.859491+0000","last_undegraded":"2024-04-15T18:23:24.859491+0000","last_fullsized":"2024-04-15T18:23:24.859491+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5535","last_scrub_stamp":"2024-04-15T18:22:11.496110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.496110+0000","objects_scrubbed":5505,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:30.883032+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121110528,"num_objects":1848,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"341'9298","reported_seq":9425,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.144486+0000","last_change":"2024-04-15T18:23:19.280474+0000","last_active":"2024-04-15T18:23:27.144486+0000","last_peered":"2024-04-15T18:23:27.144486+0000","last_clean":"2024-04-15T18:23:27.144486+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:23:27.144486+0000","last_undegraded":"2024-04-15T18:23:27.144486+0000","last_fullsized":"2024-04-15T18:23:27.144486+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.275206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"342'9380","reported_seq":19124,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.435771+0000","last_change":"2024-04-15T18:23:09.475875+0000","last_active":"2024-04-15T18:23:27.435771+0000","last_peered":"2024-04-15T18:23:27.435771+0000","last_clean":"2024-04-15T18:23:27.435771+0000","last_became_active":"2024-04-15T18:23:06.512970+0000","last_became_peered":"2024-04-15T18:23:06.512970+0000","last_unstale":"2024-04-15T18:23:27.435771+0000","last_undegraded":"2024-04-15T18:23:27.435771+0000","last_fullsized":"2024-04-15T18:23:27.435771+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8513","last_scrub_stamp":"2024-04-15T18:23:09.475588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:09.475588+0000","objects_scrubbed":2599,"log_size":9380,"log_dups_size":0,"ondisk_log_size":9380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:22.506266+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4788,"num_bytes_recovered":313786368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"341'9552","reported_seq":19482,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.180885+0000","last_change":"2024-04-15T18:23:18.942151+0000","last_active":"2024-04-15T18:23:27.180885+0000","last_peered":"2024-04-15T18:23:27.180885+0000","last_clean":"2024-04-15T18:23:27.180885+0000","last_became_active":"2024-04-15T18:23:16.252888+0000","last_became_peered":"2024-04-15T18:23:16.252888+0000","last_unstale":"2024-04-15T18:23:27.180885+0000","last_undegraded":"2024-04-15T18:23:27.180885+0000","last_fullsized":"2024-04-15T18:23:27.180885+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'9138","last_scrub_stamp":"2024-04-15T18:23:18.941879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:18.941879+0000","objects_scrubbed":2119,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:36.894557+0000","scrub_duration":664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4880,"num_bytes_recovered":319750144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,10],"acting":[1,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"341'9516","reported_seq":9701,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:24.871149+0000","last_change":"2024-04-15T18:22:03.640303+0000","last_active":"2024-04-15T18:23:24.871149+0000","last_peered":"2024-04-15T18:23:24.871149+0000","last_clean":"2024-04-15T18:23:24.871149+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:23:24.871149+0000","last_undegraded":"2024-04-15T18:23:24.871149+0000","last_fullsized":"2024-04-15T18:23:24.871149+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5259","last_scrub_stamp":"2024-04-15T18:22:03.640098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.640098+0000","objects_scrubbed":5226,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:29.609289+0000","scrub_duration":1294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":1860,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9516,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"342'9530","reported_seq":9690,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.418060+0000","last_change":"2024-04-15T18:23:05.903965+0000","last_active":"2024-04-15T18:23:27.418060+0000","last_peered":"2024-04-15T18:23:27.418060+0000","last_clean":"2024-04-15T18:23:27.418060+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:23:27.418060+0000","last_undegraded":"2024-04-15T18:23:27.418060+0000","last_fullsized":"2024-04-15T18:23:27.418060+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'8504","last_scrub_stamp":"2024-04-15T18:23:05.903829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.903829+0000","objects_scrubbed":2750,"log_size":9530,"log_dups_size":0,"ondisk_log_size":9530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:17.161701+0000","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112721920,"num_objects":1720,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9530,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"341'9609","reported_seq":9796,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:24.842012+0000","last_change":"2024-04-15T18:22:19.093291+0000","last_active":"2024-04-15T18:23:24.842012+0000","last_peered":"2024-04-15T18:23:24.842012+0000","last_clean":"2024-04-15T18:23:24.842012+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:23:24.842012+0000","last_undegraded":"2024-04-15T18:23:24.842012+0000","last_fullsized":"2024-04-15T18:23:24.842012+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6103","last_scrub_stamp":"2024-04-15T18:22:19.093022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:19.093022+0000","objects_scrubbed":5424,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:27.551120+0000","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123142144,"num_objects":1879,"num_object_clones":0,"num_object_copies":5637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9609,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"342'9337","reported_seq":9491,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.571017+0000","last_change":"2024-04-15T18:23:18.792596+0000","last_active":"2024-04-15T18:23:27.571017+0000","last_peered":"2024-04-15T18:23:27.571017+0000","last_clean":"2024-04-15T18:23:27.571017+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:23:27.571017+0000","last_undegraded":"2024-04-15T18:23:27.571017+0000","last_fullsized":"2024-04-15T18:23:27.571017+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8868","last_scrub_stamp":"2024-04-15T18:23:18.792430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:18.792430+0000","objects_scrubbed":2216,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:32.076768+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"341'11477","reported_seq":11791,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:25.240633+0000","last_change":"2024-04-15T18:23:14.957703+0000","last_active":"2024-04-15T18:23:25.240633+0000","last_peered":"2024-04-15T18:23:25.240633+0000","last_clean":"2024-04-15T18:23:25.240633+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:23:25.240633+0000","last_undegraded":"2024-04-15T18:23:25.240633+0000","last_fullsized":"2024-04-15T18:23:25.240633+0000","mapping_epoch":284,"log_start":"293'1400","ondisk_log_start":"293'1400","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'10999","last_scrub_stamp":"2024-04-15T18:23:14.957443+0000","last_deep_scrub":"311'6222","last_deep_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_clean_scrub_stamp":"2024-04-15T18:23:14.957443+0000","objects_scrubbed":2362,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:31.738507+0000","scrub_duration":842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122421248,"num_objects":1868,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9302,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":902,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-15T18:23:23.209852+0000","last_change":"2024-04-15T18:23:05.604093+0000","last_active":"2024-04-15T18:23:23.209852+0000","last_peered":"2024-04-15T18:23:23.209852+0000","last_clean":"2024-04-15T18:23:23.209852+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:23:23.209852+0000","last_undegraded":"2024-04-15T18:23:23.209852+0000","last_fullsized":"2024-04-15T18:23:23.209852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:23:05.603936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.603936+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:30.294200+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1862336536,"num_objects":28418,"num_object_clones":0,"num_object_copies":85254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150286,"num_write_kb":5718465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9668,"num_bytes_recovered":633536512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6021656576,"data_stored":6021644360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151061,"ondisk_log_size":151061,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925996,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197212,"kb_used_data":886996,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92547940,"statfs":{"total":95995035648,"available":94769090560,"internally_reserved":0,"allocated":908283904,"data_stored":908024818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":317651396},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187632,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":724388,"kb_used_data":548824,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93020764,"statfs":{"total":95995035648,"available":95253262336,"internally_reserved":0,"allocated":561995776,"data_stored":561735666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514677,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":558488,"kb_used_data":338512,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93186664,"statfs":{"total":95995035648,"available":95423143936,"internally_reserved":0,"allocated":346636288,"data_stored":346376702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":225245632},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874425,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122072,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623080,"statfs":{"total":95995035648,"available":95870033920,"internally_reserved":0,"allocated":667648,"data_stored":412134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168766,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695260,"kb_used_data":465556,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93049892,"statfs":{"total":95995035648,"available":95283089408,"internally_reserved":0,"allocated":476729344,"data_stored":476470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463107,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1014624,"kb_used_data":762708,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92730528,"statfs":{"total":95995035648,"available":94956060672,"internally_reserved":0,"allocated":781012992,"data_stored":780753926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575073,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345944,"kb_used_data":1017552,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92399208,"statfs":{"total":95995035648,"available":94616788992,"internally_reserved":0,"allocated":1041973248,"data_stored":1041713662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587540,"kb_used_data":339788,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93157612,"statfs":{"total":95995035648,"available":95393394688,"internally_reserved":0,"allocated":347942912,"data_stored":347687398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608975,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870932,"kb_used_data":561484,"kb_used_omap":44,"kb_used_meta":309395,"kb_avail":92874220,"statfs":{"total":95995035648,"available":95103201280,"internally_reserved":0,"allocated":574959616,"data_stored":574704102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":316821034},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986381,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758872,"kb_used_data":497232,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92986280,"statfs":{"total":95995035648,"available":95217950720,"internally_reserved":0,"allocated":509165568,"data_stored":508905982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724743,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390616,"kb_used_data":246988,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93354536,"statfs":{"total":95995035648,"available":95595044864,"internally_reserved":0,"allocated":252915712,"data_stored":252660198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":343932928,"data_stored":343932928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1015681024,"data_stored":1015676952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442302464,"data_stored":442302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":566820864,"data_stored":566820864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":449908736,"data_stored":449904664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":248840192,"data_stored":248840192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":729808896,"data_stored":729808896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":473890816,"data_stored":473890816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":336662528,"data_stored":336658456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":555417600,"data_stored":555417600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":858390528,"data_stored":858390528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:23:30.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:30.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:23:30.505 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:30.507 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:30 smithi082 ceph-mon[44206]: pgmap v942: 17 pgs: 17 active+clean; 1.7 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-04-15T18:23:30.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:30 smithi082 ceph-mon[44206]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:30.533 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:30.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:30.638 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:30.687 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:30.710 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:30.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:30.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:30.864 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:30.914 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:30.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:30.968 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:31.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:31.065 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:31.097 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:31.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:31.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:31.244 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:31.597 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:31.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:31.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:31.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:31.708 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:31.732 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:31.742 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:31.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:31.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:31.801 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:31.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:31.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:31.869 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:31.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:31.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:31.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:31.959 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:31.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:32.029 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:32.222 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:32.222 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:32.324 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:32.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:32.430 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:32.466 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:32.490 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:32.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:32.597 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:32.629 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: pgmap v943: 17 pgs: 17 active+clean; 1.6 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:23:32.629 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/232672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:32.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.633 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:32.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: pgmap v943: 17 pgs: 17 active+clean; 1.6 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:23:32.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/232672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:32.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: pgmap v943: 17 pgs: 17 active+clean; 1.6 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/232672614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:32.998 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:33.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:33.002 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:23:33.102 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:33.151 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:33.180 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:33.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:33.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:33.317 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.318 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.330 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:33.351 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:33.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:33.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:23:33.431 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:33.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:33.492 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:33.518 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:33.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:33.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.653 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:33.674 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:33.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:33.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:33.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:34.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:34.476 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:34.572 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:34.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:34 smithi082 ceph-mon[44206]: pgmap v944: 17 pgs: 17 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-04-15T18:23:34.613 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:34.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:34 smithi049 ceph-mon[40087]: pgmap v944: 17 pgs: 17 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-04-15T18:23:34.645 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:34.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:34.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:34 smithi159 ceph-mon[45336]: pgmap v944: 17 pgs: 17 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 816 op/s 2024-04-15T18:23:34.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:34.789 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:35.044 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:35.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:35.113 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:35.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:35.180 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:35.206 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:35.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:35.311 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:35.352 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:35.375 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:35.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:35.396 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:35.397 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:35.483 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:35.515 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:35.725 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:35.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:35.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:35.896 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:35.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:36.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:36.133 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:36.171 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:23:36.289 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:36.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:36.347 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:23:36.397 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:36.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:36 smithi159 ceph-mon[45336]: pgmap v945: 17 pgs: 17 active+clean; 1.4 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-04-15T18:23:36.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:23:36.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:23:36.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/940234127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:36 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.429 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:36.452 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:36.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:36.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:23:36.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:36.560 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:36.589 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:36 smithi082 ceph-mon[44206]: pgmap v945: 17 pgs: 17 active+clean; 1.4 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-04-15T18:23:36.589 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:23:36.589 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:23:36.590 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/940234127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.590 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:36 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.593 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:36.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:36.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:36 smithi049 ceph-mon[40087]: pgmap v945: 17 pgs: 17 active+clean; 1.4 GiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s 2024-04-15T18:23:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:23:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:23:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/940234127' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:36 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:23:36.728 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:36.760 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:36.823 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:36.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:36.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:36.979 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:37.136 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:37.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:37.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:37.275 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:37.298 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:37.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:37.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:37 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:37.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:37 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:23:37.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:37 smithi159 ceph-mon[45336]: osdmap e343: 12 total, 11 up, 11 in 2024-04-15T18:23:37.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:37.439 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:37.474 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:37.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:37.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:37.580 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:37.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:37 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:37.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:37 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:23:37.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:37 smithi082 ceph-mon[44206]: osdmap e343: 12 total, 11 up, 11 in 2024-04-15T18:23:37.609 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:37.635 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:37.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:37.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:37 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:37.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:37 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:23:37.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:37 smithi049 ceph-mon[40087]: osdmap e343: 12 total, 11 up, 11 in 2024-04-15T18:23:37.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:37.774 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:37.908 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:23:37.921 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:23:38.039 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:38.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:38.143 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:38.178 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:38.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:23:38.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:23:38.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:38 smithi159 ceph-mon[45336]: pgmap v947: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s 2024-04-15T18:23:38.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:38 smithi159 ceph-mon[45336]: osdmap e344: 12 total, 11 up, 11 in 2024-04-15T18:23:38.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:38 smithi159 ceph-mon[45336]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:38.390 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:38.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:38.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:38.530 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:38.563 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:38.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:38.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:23:38.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:23:38.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:38 smithi049 ceph-mon[40087]: pgmap v947: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s 2024-04-15T18:23:38.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:38 smithi049 ceph-mon[40087]: osdmap e344: 12 total, 11 up, 11 in 2024-04-15T18:23:38.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:38 smithi049 ceph-mon[40087]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:38.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:38.711 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:38.713 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:38.713 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:38.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:23:38.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:23:38.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:38 smithi082 ceph-mon[44206]: pgmap v947: 17 pgs: 17 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s 2024-04-15T18:23:38.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:38 smithi082 ceph-mon[44206]: osdmap e344: 12 total, 11 up, 11 in 2024-04-15T18:23:38.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:38 smithi082 ceph-mon[44206]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:23:39.077 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":948,"stamp":"2024-04-15T18:23:37.344611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1426195000,"num_objects":21756,"num_object_clones":0,"num_object_copies":65268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21756,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":157131,"num_write_kb":5721403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9678,"num_bytes_recovered":636181152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157162,"ondisk_log_size":157162,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":6930696,"kb_used_data":4318700,"kb_used_omap":60,"kb_used_meta":2611843,"kb_avail":1024265976,"statfs":{"total":1055945392128,"available":1048848359424,"internally_reserved":0,"allocated":4422348800,"data_stored":4419464698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61481,"internal_metadata":2674528215},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-436731904,"num_objects":-6664,"num_object_clones":0,"num_object_copies":-19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6664,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.094457"},"pg_stats":[{"pgid":"4.f","version":"343'10047","reported_seq":10216,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-15T18:23:37.159225+0000","last_change":"2024-04-15T18:21:32.333996+0000","last_active":"2024-04-15T18:23:37.159225+0000","last_peered":"2024-04-15T18:23:37.159225+0000","last_clean":"2024-04-15T18:23:37.159225+0000","last_became_active":"2024-04-15T18:20:31.187051+0000","last_became_peered":"2024-04-15T18:20:31.187051+0000","last_unstale":"2024-04-15T18:23:37.159225+0000","last_undegraded":"2024-04-15T18:23:37.159225+0000","last_fullsized":"2024-04-15T18:23:37.159225+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'3320","last_scrub_stamp":"2024-04-15T18:21:32.333909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:21:32.333909+0000","objects_scrubbed":3301,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:37.309276+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"342'9431","reported_seq":9646,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.563179+0000","last_change":"2024-04-15T18:23:17.585526+0000","last_active":"2024-04-15T18:23:32.563179+0000","last_peered":"2024-04-15T18:23:32.563179+0000","last_clean":"2024-04-15T18:23:32.563179+0000","last_became_active":"2024-04-15T18:20:31.649475+0000","last_became_peered":"2024-04-15T18:20:31.649475+0000","last_unstale":"2024-04-15T18:23:32.563179+0000","last_undegraded":"2024-04-15T18:23:32.563179+0000","last_fullsized":"2024-04-15T18:23:32.563179+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8692","last_scrub_stamp":"2024-04-15T18:23:17.585204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:17.585204+0000","objects_scrubbed":2251,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:39.572162+0000","scrub_duration":690,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97583104,"num_objects":1489,"num_object_clones":0,"num_object_copies":4467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":349440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"342'9615","reported_seq":9805,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.544540+0000","last_change":"2024-04-15T18:22:11.794263+0000","last_active":"2024-04-15T18:23:32.544540+0000","last_peered":"2024-04-15T18:23:32.544540+0000","last_clean":"2024-04-15T18:23:32.544540+0000","last_became_active":"2024-04-15T18:20:31.190547+0000","last_became_peered":"2024-04-15T18:20:31.190547+0000","last_unstale":"2024-04-15T18:23:32.544540+0000","last_undegraded":"2024-04-15T18:23:32.544540+0000","last_fullsized":"2024-04-15T18:23:32.544540+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5531","last_scrub_stamp":"2024-04-15T18:22:11.794127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.794127+0000","objects_scrubbed":5516,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:35.007436+0000","scrub_duration":1253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896152,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9615,"num_write_kb":353985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"342'9829","reported_seq":10049,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.568159+0000","last_change":"2024-04-15T18:23:08.359021+0000","last_active":"2024-04-15T18:23:32.568159+0000","last_peered":"2024-04-15T18:23:32.568159+0000","last_clean":"2024-04-15T18:23:32.568159+0000","last_became_active":"2024-04-15T18:20:31.193821+0000","last_became_peered":"2024-04-15T18:20:31.193821+0000","last_unstale":"2024-04-15T18:23:32.568159+0000","last_undegraded":"2024-04-15T18:23:32.568159+0000","last_fullsized":"2024-04-15T18:23:32.568159+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8637","last_scrub_stamp":"2024-04-15T18:23:08.358887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:08.358887+0000","objects_scrubbed":2688,"log_size":9829,"log_dups_size":0,"ondisk_log_size":9829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:18.649813+0000","scrub_duration":570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97320960,"num_objects":1485,"num_object_clones":0,"num_object_copies":4455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9829,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"342'9838","reported_seq":10025,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:36.276753+0000","last_change":"2024-04-15T18:22:05.829296+0000","last_active":"2024-04-15T18:23:36.276753+0000","last_peered":"2024-04-15T18:23:36.276753+0000","last_clean":"2024-04-15T18:23:36.276753+0000","last_became_active":"2024-04-15T18:20:31.187972+0000","last_became_peered":"2024-04-15T18:20:31.187972+0000","last_unstale":"2024-04-15T18:23:36.276753+0000","last_undegraded":"2024-04-15T18:23:36.276753+0000","last_fullsized":"2024-04-15T18:23:36.276753+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5237","last_scrub_stamp":"2024-04-15T18:22:05.829122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.829122+0000","objects_scrubbed":5187,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"342'9956","reported_seq":10144,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:35.185471+0000","last_change":"2024-04-15T18:22:05.753884+0000","last_active":"2024-04-15T18:23:35.185471+0000","last_peered":"2024-04-15T18:23:35.185471+0000","last_clean":"2024-04-15T18:23:35.185471+0000","last_became_active":"2024-04-15T18:20:31.649533+0000","last_became_peered":"2024-04-15T18:20:31.649533+0000","last_unstale":"2024-04-15T18:23:35.185471+0000","last_undegraded":"2024-04-15T18:23:35.185471+0000","last_fullsized":"2024-04-15T18:23:35.185471+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5298","last_scrub_stamp":"2024-04-15T18:22:05.753792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:05.753792+0000","objects_scrubbed":5262,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:37.159574+0000","scrub_duration":1367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"342'9645","reported_seq":9829,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:34.863855+0000","last_change":"2024-04-15T18:22:23.627521+0000","last_active":"2024-04-15T18:23:34.863855+0000","last_peered":"2024-04-15T18:23:34.863855+0000","last_clean":"2024-04-15T18:23:34.863855+0000","last_became_active":"2024-04-15T18:20:31.187160+0000","last_became_peered":"2024-04-15T18:20:31.187160+0000","last_unstale":"2024-04-15T18:23:34.863855+0000","last_undegraded":"2024-04-15T18:23:34.863855+0000","last_fullsized":"2024-04-15T18:23:34.863855+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'6038","last_scrub_stamp":"2024-04-15T18:22:23.627256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:23.627256+0000","objects_scrubbed":4987,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:36.778056+0000","scrub_duration":1161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670208,"num_objects":1353,"num_object_clones":0,"num_object_copies":4059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":351936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"342'9728","reported_seq":9918,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:34.828533+0000","last_change":"2024-04-15T18:22:11.496215+0000","last_active":"2024-04-15T18:23:34.828533+0000","last_peered":"2024-04-15T18:23:34.828533+0000","last_clean":"2024-04-15T18:23:34.828533+0000","last_became_active":"2024-04-15T18:20:31.188231+0000","last_became_peered":"2024-04-15T18:20:31.188231+0000","last_unstale":"2024-04-15T18:23:34.828533+0000","last_undegraded":"2024-04-15T18:23:34.828533+0000","last_fullsized":"2024-04-15T18:23:34.828533+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'5535","last_scrub_stamp":"2024-04-15T18:22:11.496110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:11.496110+0000","objects_scrubbed":5505,"log_size":9728,"log_dups_size":0,"ondisk_log_size":9728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:36.778020+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9728,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"343'9816","reported_seq":9947,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-15T18:23:37.138956+0000","last_change":"2024-04-15T18:23:19.280474+0000","last_active":"2024-04-15T18:23:37.138956+0000","last_peered":"2024-04-15T18:23:37.138956+0000","last_clean":"2024-04-15T18:23:37.138956+0000","last_became_active":"2024-04-15T18:20:31.649598+0000","last_became_peered":"2024-04-15T18:20:31.649598+0000","last_unstale":"2024-04-15T18:23:37.138956+0000","last_undegraded":"2024-04-15T18:23:37.138956+0000","last_fullsized":"2024-04-15T18:23:37.138956+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:20:30.170849+0000","objects_scrubbed":0,"log_size":9816,"log_dups_size":0,"ondisk_log_size":9816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:23:37.309255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76414976,"num_objects":1166,"num_object_clones":0,"num_object_copies":3498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"342'9641","reported_seq":19387,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.373022+0000","last_change":"2024-04-15T18:23:09.475875+0000","last_active":"2024-04-15T18:23:32.373022+0000","last_peered":"2024-04-15T18:23:32.373022+0000","last_clean":"2024-04-15T18:23:32.373022+0000","last_became_active":"2024-04-15T18:23:06.512970+0000","last_became_peered":"2024-04-15T18:23:06.512970+0000","last_unstale":"2024-04-15T18:23:32.373022+0000","last_undegraded":"2024-04-15T18:23:32.373022+0000","last_fullsized":"2024-04-15T18:23:32.373022+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'8513","last_scrub_stamp":"2024-04-15T18:23:09.475588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:09.475588+0000","objects_scrubbed":2599,"log_size":9641,"log_dups_size":0,"ondisk_log_size":9641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:22.506266+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":4353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9641,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4788,"num_bytes_recovered":313786368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"343'10046","reported_seq":19980,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-04-15T18:23:37.134498+0000","last_change":"2024-04-15T18:23:18.942151+0000","last_active":"2024-04-15T18:23:37.134498+0000","last_peered":"2024-04-15T18:23:37.134498+0000","last_clean":"2024-04-15T18:23:37.134498+0000","last_became_active":"2024-04-15T18:23:16.252888+0000","last_became_peered":"2024-04-15T18:23:16.252888+0000","last_unstale":"2024-04-15T18:23:37.134498+0000","last_undegraded":"2024-04-15T18:23:37.134498+0000","last_fullsized":"2024-04-15T18:23:37.134498+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'9138","last_scrub_stamp":"2024-04-15T18:23:18.941879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:18.941879+0000","objects_scrubbed":2119,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:36.894557+0000","scrub_duration":664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78905344,"num_objects":1204,"num_object_clones":0,"num_object_copies":3612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4880,"num_bytes_recovered":319750144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,10],"acting":[1,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"342'10035","reported_seq":10222,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:34.830276+0000","last_change":"2024-04-15T18:22:03.640303+0000","last_active":"2024-04-15T18:23:34.830276+0000","last_peered":"2024-04-15T18:23:34.830276+0000","last_clean":"2024-04-15T18:23:34.830276+0000","last_became_active":"2024-04-15T18:20:31.188923+0000","last_became_peered":"2024-04-15T18:20:31.188923+0000","last_unstale":"2024-04-15T18:23:34.830276+0000","last_undegraded":"2024-04-15T18:23:34.830276+0000","last_fullsized":"2024-04-15T18:23:34.830276+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'5259","last_scrub_stamp":"2024-04-15T18:22:03.640098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:03.640098+0000","objects_scrubbed":5226,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"342'9759","reported_seq":9920,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.383042+0000","last_change":"2024-04-15T18:23:05.903965+0000","last_active":"2024-04-15T18:23:32.383042+0000","last_peered":"2024-04-15T18:23:32.383042+0000","last_clean":"2024-04-15T18:23:32.383042+0000","last_became_active":"2024-04-15T18:20:31.186645+0000","last_became_peered":"2024-04-15T18:20:31.186645+0000","last_unstale":"2024-04-15T18:23:32.383042+0000","last_undegraded":"2024-04-15T18:23:32.383042+0000","last_fullsized":"2024-04-15T18:23:32.383042+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'8504","last_scrub_stamp":"2024-04-15T18:23:05.903829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.903829+0000","objects_scrubbed":2750,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:17.161701+0000","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97714176,"num_objects":1491,"num_object_clones":0,"num_object_copies":4473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"342'10119","reported_seq":10308,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:34.816712+0000","last_change":"2024-04-15T18:22:19.093291+0000","last_active":"2024-04-15T18:23:34.816712+0000","last_peered":"2024-04-15T18:23:34.816712+0000","last_clean":"2024-04-15T18:23:34.816712+0000","last_became_active":"2024-04-15T18:20:31.188210+0000","last_became_peered":"2024-04-15T18:20:31.188210+0000","last_unstale":"2024-04-15T18:23:34.816712+0000","last_undegraded":"2024-04-15T18:23:34.816712+0000","last_fullsized":"2024-04-15T18:23:34.816712+0000","mapping_epoch":284,"log_start":"290'100","ondisk_log_start":"290'100","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'6103","last_scrub_stamp":"2024-04-15T18:22:19.093022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:22:19.093022+0000","objects_scrubbed":5424,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"342'9588","reported_seq":9743,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:32.569230+0000","last_change":"2024-04-15T18:23:18.792596+0000","last_active":"2024-04-15T18:23:32.569230+0000","last_peered":"2024-04-15T18:23:32.569230+0000","last_clean":"2024-04-15T18:23:32.569230+0000","last_became_active":"2024-04-15T18:20:31.186004+0000","last_became_peered":"2024-04-15T18:20:31.186004+0000","last_unstale":"2024-04-15T18:23:32.569230+0000","last_undegraded":"2024-04-15T18:23:32.569230+0000","last_fullsized":"2024-04-15T18:23:32.569230+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"339'8868","last_scrub_stamp":"2024-04-15T18:23:18.792430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:20:30.170849+0000","last_clean_scrub_stamp":"2024-04-15T18:23:18.792430+0000","objects_scrubbed":2216,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:32.076768+0000","scrub_duration":794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96862208,"num_objects":1478,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9588,"num_write_kb":354112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"342'12032","reported_seq":12348,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:35.243823+0000","last_change":"2024-04-15T18:23:14.957703+0000","last_active":"2024-04-15T18:23:35.243823+0000","last_peered":"2024-04-15T18:23:35.243823+0000","last_clean":"2024-04-15T18:23:35.243823+0000","last_became_active":"2024-04-15T18:20:31.649500+0000","last_became_peered":"2024-04-15T18:20:31.649500+0000","last_unstale":"2024-04-15T18:23:35.243823+0000","last_undegraded":"2024-04-15T18:23:35.243823+0000","last_fullsized":"2024-04-15T18:23:35.243823+0000","mapping_epoch":284,"log_start":"294'2000","ondisk_log_start":"294'2000","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'10999","last_scrub_stamp":"2024-04-15T18:23:14.957443+0000","last_deep_scrub":"311'6222","last_deep_scrub_stamp":"2024-04-15T18:21:44.277930+0000","last_clean_scrub_stamp":"2024-04-15T18:23:14.957443+0000","objects_scrubbed":2362,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:31.738507+0000","scrub_duration":842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9857,"num_write_kb":357440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":904,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-04-15T18:23:27.244014+0000","last_change":"2024-04-15T18:23:05.604093+0000","last_active":"2024-04-15T18:23:27.244014+0000","last_peered":"2024-04-15T18:23:27.244014+0000","last_clean":"2024-04-15T18:23:27.244014+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:23:27.244014+0000","last_undegraded":"2024-04-15T18:23:27.244014+0000","last_fullsized":"2024-04-15T18:23:27.244014+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:23:05.603936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:23:05.603936+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:24:30.294200+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1425604632,"num_objects":21754,"num_object_clones":0,"num_object_copies":65262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21754,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156950,"num_write_kb":5718465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9668,"num_bytes_recovered":633536512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4692520960,"data_stored":4692508744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157025,"ondisk_log_size":157025,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726925998,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938920,"kb_used_data":632864,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92806232,"statfs":{"total":95995035648,"available":95033581568,"internally_reserved":0,"allocated":648052736,"data_stored":647789054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":313391556},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641504,"kb_used_data":461784,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93103648,"statfs":{"total":95995035648,"available":95338135552,"internally_reserved":0,"allocated":472866816,"data_stored":472606706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514678,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496548,"kb_used_data":289052,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93248604,"statfs":{"total":95995035648,"available":95486570496,"internally_reserved":0,"allocated":295989248,"data_stored":295725066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":212466112},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874427,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122084,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623068,"statfs":{"total":95995035648,"available":95870021632,"internally_reserved":0,"allocated":679936,"data_stored":419826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168768,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573480,"kb_used_data":335456,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93171672,"statfs":{"total":95995035648,"available":95407792128,"internally_reserved":0,"allocated":343506944,"data_stored":343243282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":243726794},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463109,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791144,"kb_used_data":543392,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92954008,"statfs":{"total":95995035648,"available":95184904192,"internally_reserved":0,"allocated":556433408,"data_stored":556169746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":253688262},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058980,"kb_used_data":730716,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92686172,"statfs":{"total":95995035648,"available":94910640128,"internally_reserved":0,"allocated":748253184,"data_stored":747989002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495520,"kb_used_data":243608,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93249632,"statfs":{"total":95995035648,"available":95487623168,"internally_reserved":0,"allocated":249454592,"data_stored":249194482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662608976,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794464,"kb_used_data":476696,"kb_used_omap":44,"kb_used_meta":317715,"kb_avail":92950688,"statfs":{"total":95995035648,"available":95181504512,"internally_reserved":0,"allocated":488136704,"data_stored":487876594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":325340714},"hb_peers":[0,1,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986382,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687268,"kb_used_data":421468,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93057884,"statfs":{"total":95995035648,"available":95291273216,"internally_reserved":0,"allocated":431583232,"data_stored":431319050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724745,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330784,"kb_used_data":183000,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93414368,"statfs":{"total":95995035648,"available":95656312832,"internally_reserved":0,"allocated":187392000,"data_stored":187131890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245039104,"data_stored":245039104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":720441344,"data_stored":720437272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442302464,"data_stored":442302464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":479592448,"data_stored":479592448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":383520768,"data_stored":383516696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":182845440,"data_stored":182845440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":522977280,"data_stored":522977280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":340852736,"data_stored":340852736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":287969280,"data_stored":287965208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":466223104,"data_stored":466223104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":620756992,"data_stored":620756992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:23:39.079 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:23:39.079 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-15T18:23:39.080 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:39.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:39.187 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:39.244 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:39.407 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:39.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:39.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:23:39.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:39.588 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:39.617 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:39.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:39.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:39.757 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:39.859 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:39.921 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:39.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:40.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:40.061 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:40.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:23:40.097 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:40.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:40.204 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:40.238 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:40.264 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:40.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:40.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:40.400 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:40.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:40 smithi159 ceph-mon[45336]: pgmap v949: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:23:40.429 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:40.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:40.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:40.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:40 smithi049 ceph-mon[40087]: pgmap v949: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:23:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:40.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:40.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:40.695 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:40.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:40.772 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:40.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:40.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:40 smithi082 ceph-mon[44206]: pgmap v949: 17 pgs: 17 active+clean; 1.2 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-04-15T18:23:40.803 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:40.811 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:40.831 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:40.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:40.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:40.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:40.930 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:40.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:40.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:41.012 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:41.025 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:41.056 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:41.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:41.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:41.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:41.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:41.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:41.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:41.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:41.198 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:41.224 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:41.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:41.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:41.397 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:41.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:41.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: 4.a scrub starts 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: 4.7 scrub starts 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: 4.a scrub ok 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: 4.7 scrub ok 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/668040706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: 4.3 scrub starts 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: 4.a scrub starts 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: 4.7 scrub starts 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: 4.a scrub ok 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: 4.7 scrub ok 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/668040706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: 4.3 scrub starts 2024-04-15T18:23:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.535 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:41.560 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:41.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:41.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:41.708 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: 4.a scrub starts 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: 4.7 scrub starts 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: 4.a scrub ok 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: 4.7 scrub ok 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/668040706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:41.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: 4.3 scrub starts 2024-04-15T18:23:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:41.812 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:41.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:162564 4.6 4:65320e7d:::benchmark_data_smithi159_102124_object73210:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:41.812533+0000", 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0080429750000000008, 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080678280000000008, 2024-04-15T18:23:41.816 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 162564 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812533+0000", 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:41.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812533+0000", 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812534+0000", 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5799999999999999e-06 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812539+0000", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0889999999999999e-06 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812552+0000", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2754e-05 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.812558+0000", 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1399999999999997e-06 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.813101+0000", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00054303299999999995 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.813229+0000", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012797899999999999 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.813377+0000", 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014784299999999999 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818591+0000", 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052140729999999996 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:162570 4.2 4:40a4fa2d:::benchmark_data_smithi159_102124_object73216:head [delete] snapc 0=[] ondisk+write+known_if_redirected e344)", 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:41.818742+0000", 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.001833477, 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019122799999999999, 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:41.821 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 162570 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818742+0000", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818742+0000", 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:41.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818743+0000", 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.062e-06 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818748+0000", 2024-04-15T18:23:41.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5279999999999997e-06 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818758+0000", 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0209e-05 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818764+0000", 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.604e-06 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:41.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818827+0000", 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2889e-05 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.818962+0000", 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013548299999999999 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:41.819086+0000", 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000123666 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:41.825 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:23:41.826 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 2 2024-04-15T18:23:41.826 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:23:41.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:41.949 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:42.049 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:42.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:42.155 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:42.187 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: 4.b scrub starts 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.568 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: pgmap v950: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:23:42.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: 4.b scrub ok 2024-04-15T18:23:42.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:42 smithi159 ceph-mon[45336]: 4.3 scrub ok 2024-04-15T18:23:42.584 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:42.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:42.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:23:42.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: 4.b scrub starts 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: pgmap v950: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:23:42.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: 4.b scrub ok 2024-04-15T18:23:42.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:42 smithi049 ceph-mon[40087]: 4.3 scrub ok 2024-04-15T18:23:42.774 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:42.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: 4.b scrub starts 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: pgmap v950: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: 4.b scrub ok 2024-04-15T18:23:42.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:42 smithi082 ceph-mon[44206]: 4.3 scrub ok 2024-04-15T18:23:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:42.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:42.820 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:42.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:42.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:42.967 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.006 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:43.056 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:43.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:43.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.168 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:43.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:43.304 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.335 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:43.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:23:43.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:43 smithi049 ceph-mon[40087]: 4.1 scrub starts 2024-04-15T18:23:43.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.442 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:43.478 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:23:43.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:43 smithi082 ceph-mon[44206]: 4.1 scrub starts 2024-04-15T18:23:43.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.557 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:43.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:23:43.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:43 smithi159 ceph-mon[45336]: 4.1 scrub starts 2024-04-15T18:23:43.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:43.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:43.740 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.774 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:43.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:43.908 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:43.935 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:43.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:44.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:44.083 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:44.113 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:44.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:44.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:44.258 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: 4.1 scrub ok 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: pgmap v951: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.0 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: 4.8 scrub starts 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: 4.8 scrub ok 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:23:44.365 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:23:44.366 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:44.389 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:23:44.415 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:44.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:44.520 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:44.551 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:44.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: 4.1 scrub ok 2024-04-15T18:23:44.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: pgmap v951: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.0 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-04-15T18:23:44.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: 4.8 scrub starts 2024-04-15T18:23:44.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:44.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: 4.8 scrub ok 2024-04-15T18:23:44.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:23:44.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:23:44.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:44.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: 4.1 scrub ok 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: pgmap v951: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 1.0 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: 4.8 scrub starts 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: 4.8 scrub ok 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:23:44.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:23:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:44.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:44.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:23:44.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:44.929 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:44.957 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:44.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:45.062 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:45.092 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:45.115 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:45.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:45.120 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:23:45.120 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:23:45.120 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:23:45.120 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:23:45.120 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:23:45.218 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:45.250 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:45.380 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:45.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:45.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:45 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:23:45.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:45 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:23:45.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:45.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:45 smithi159 ceph-mon[45336]: osdmap e345: 12 total, 11 up, 11 in 2024-04-15T18:23:45.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:45.567 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:45.585 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:45.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:45.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:45.716 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:45 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:23:45.716 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:45 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:23:45.716 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:45.716 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:45 smithi082 ceph-mon[44206]: osdmap e345: 12 total, 11 up, 11 in 2024-04-15T18:23:45.718 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:45.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:45 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:23:45.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:45 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:23:45.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2039533978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:23:45.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:45 smithi049 ceph-mon[40087]: osdmap e345: 12 total, 11 up, 11 in 2024-04-15T18:23:45.736 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:45.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:45.842 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:45.887 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:45.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:23:45.914 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:45.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:46.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:46.049 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:46.068 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:46.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:46.172 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:46.209 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:46.235 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:46.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:46.259 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:46.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:46.370 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:46.393 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:46.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:46.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:46.523 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:46.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:46 smithi159 ceph-mon[45336]: 4.f deep-scrub starts 2024-04-15T18:23:46.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:46 smithi159 ceph-mon[45336]: pgmap v953: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 970 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-04-15T18:23:46.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:46 smithi159 ceph-mon[45336]: 4.9 deep-scrub starts 2024-04-15T18:23:46.543 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:46.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:46.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:46 smithi082 ceph-mon[44206]: 4.f deep-scrub starts 2024-04-15T18:23:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:46 smithi082 ceph-mon[44206]: pgmap v953: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 970 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-04-15T18:23:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:46 smithi082 ceph-mon[44206]: 4.9 deep-scrub starts 2024-04-15T18:23:46.675 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:46.695 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:46.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:46.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:46 smithi049 ceph-mon[40087]: 4.f deep-scrub starts 2024-04-15T18:23:46.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:46 smithi049 ceph-mon[40087]: pgmap v953: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 970 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 780 op/s 2024-04-15T18:23:46.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:46 smithi049 ceph-mon[40087]: 4.9 deep-scrub starts 2024-04-15T18:23:46.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:46.829 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:47.041 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:47.285 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:47.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:47.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:47.427 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:47.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:47.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:47.887 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:47.913 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:47.932 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:47.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:48.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:48.065 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:48.106 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:48.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:48.211 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:48.247 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:48.274 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:48.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:48.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:48.412 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:153095 2.d 2:b145a6a1:::benchmark_data_smithi159_80260_object63982:head [delete] snapc 0=[] ondisk+write+known_if_redirected e176)", 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 557.293005268, 2024-04-15T18:23:48.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25579912100000002, 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 153095 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149690+0000", 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149692+0000", 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3999999999999999e-06 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149703+0000", 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0703000000000001e-05 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149731+0000", 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8107e-05 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149747+0000", 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6135999999999999e-05 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149885+0000", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013756900000000001 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.149982+0000", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6799000000000006e-05 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.150060+0000", 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7757999999999997e-05 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.153583+0000", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035231580000000002 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.154591+0000", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010079780000000001 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405390+0000", 2024-04-15T18:23:48.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25079957800000002 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405442+0000", 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2023e-05 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:31.405489+0000", 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6912000000000001e-05 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24511.0:156890 2.d 2:b8159b5c:::benchmark_data_smithi159_80260_object67777:head [delete] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 550.88726615600001, 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23222245999999999, 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.451 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24511", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3200587502", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 156890 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555429+0000", 2024-04-15T18:23:48.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555431+0000", 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.838e-06 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555437+0000", 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555453+0000", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6079e-05 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555463+0000", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7210000000000006e-06 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555538+0000", 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4938999999999995e-05 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555631+0000", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2980999999999996e-05 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.555714+0000", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2655000000000001e-05 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.559420+0000", 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037066360000000001 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.583952+0000", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024531976 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787555+0000", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.203603273 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787605+0000", 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9555000000000002e-05 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:37.787652+0000", 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6606000000000001e-05 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95270 3.2 3:489b241f:::benchmark_data_smithi159_91621_object95269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 403.268222874, 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.224337434, 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.457 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95270 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174472+0000", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174475+0000", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1100000000000001e-06 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174500+0000", 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5009999999999999e-05 2024-04-15T18:23:48.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174518+0000", 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8921000000000001e-05 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174529+0000", 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0518e-05 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174601+0000", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2020000000000005e-05 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174741+0000", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014028700000000001 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.174933+0000", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019125199999999999 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.177734+0000", 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028019450000000001 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.193915+0000", 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016180439000000001 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398737+0000", 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20482228699999999 2024-04-15T18:23:48.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398777+0000", 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9320999999999998e-05 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.398810+0000", 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3324000000000001e-05 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.24793.0:95280 3.2 3:4dc79632:::benchmark_data_smithi159_91621_object95279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e228)", 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 403.25695591700003, 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22356709, 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.464 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.24793", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 95280 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185739+0000", 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185740+0000", 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5499999999999998e-07 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185758+0000", 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7938e-05 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185767+0000", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0690000000000008e-06 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.185772+0000", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3059999999999997e-06 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188088+0000", 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00231525 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188222+0000", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013395799999999999 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,6", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.188380+0000", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158768 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.190273+0000", 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018926030000000001 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.214298+0000", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024025517 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409241+0000", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19494304000000001 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409277+0000", 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5970000000000003e-05 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.409306+0000", 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9016000000000001e-05 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:158593 3.0 e255/199 3:066955e5:::benchmark_data_smithi159_91621_object52117:head v 255'10025, mlcod=255'10022)", 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 301.14613205400002, 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.239038166, 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.470 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296564+0000", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2600000000000001e-07 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296571+0000", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7870000000000004e-06 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530850+0000", 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.234278768 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530857+0000", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530914+0000", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7123000000000002e-05 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530935+0000", 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1597e-05 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535555+0000", 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046197399999999998 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535601+0000", 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6391999999999999e-05 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:185262 3.d e267/199 3:b4a17326:::benchmark_data_smithi159_91621_object78786:head v 267'11416, mlcod=267'11412)", 2024-04-15T18:23:48.474 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 256.14435559899999, 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23697396100000001, 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:23:48.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298340+0000", 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.06e-07 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.298345+0000", 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1680000000000003e-06 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531088+0000", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.232742949 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531098+0000", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5899999999999997e-06 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531159+0000", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0865000000000001e-05 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.531176+0000", 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7589e-05 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.535265+0000", 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040884110000000001 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:32.535314+0000", 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8782999999999997e-05 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.478 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:212948 3.2 e275/235 3:409c861b:::benchmark_data_smithi159_91621_object106472:head v 275'13490, mlcod=242'8024)", 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 213.43669722000001, 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56367495999999995, 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:23:48.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005999+0000", 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3680000000000001e-06 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.006010+0000", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0786e-05 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565792+0000", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55978132000000003 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565799+0000", 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.255e-06 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565863+0000", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4552000000000001e-05 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565893+0000", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9329999999999999e-05 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569633+0000", 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037404080000000002 2024-04-15T18:23:48.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569673+0000", 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9941000000000003e-05 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15515 4.2 4:4aec9481:::benchmark_data_smithi159_102124_object15514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 178.02536999599999, 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.233061504, 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.483 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15515 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417325+0000", 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417327+0000", 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3090000000000001e-06 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417347+0000", 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0038999999999999e-05 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417367+0000", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0679e-05 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417375+0000", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4080000000000001e-06 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417423+0000", 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8251000000000003e-05 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417512+0000", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8677999999999994e-05 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.417602+0000", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0295000000000002e-05 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.418326+0000", 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00072434099999999998 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.418645+0000", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000318762 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650261+0000", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.231616023 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650324+0000", 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2487e-05 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650387+0000", 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3231999999999994e-05 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15517 4.2 4:4c161e0b:::benchmark_data_smithi159_102124_object15516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 178.022288976, 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26252755999999999, 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.489 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15517 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420407+0000", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3099999999999996e-07 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420426+0000", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8371000000000001e-05 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420435+0000", 2024-04-15T18:23:48.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.448e-06 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420440+0000", 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5149999999999997e-06 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420481+0000", 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0311000000000001e-05 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420569+0000", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8140999999999995e-05 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420654+0000", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5535e-05 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421174+0000", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00051933100000000005 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421526+0000", 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035268700000000002 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682869+0000", 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26134240600000003 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682904+0000", 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682934+0000", 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9674000000000001e-05 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15520 4.2 4:4779e579:::benchmark_data_smithi159_102124_object15519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 178.017634832, 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27309090499999999, 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.495 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15520 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425061+0000", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8800000000000001e-07 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425080+0000", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8600000000000001e-05 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425090+0000", 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9219999999999992e-06 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425096+0000", 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.84e-06 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425134+0000", 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8637000000000003e-05 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425212+0000", 2024-04-15T18:23:48.499 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:48.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7958000000000001e-05 2024-04-15T18:23:48.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425297+0000", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4568999999999996e-05 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425802+0000", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050520399999999998 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426334+0000", 2024-04-15T18:23:48.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00053202999999999996 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698085+0000", 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27175072900000002 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698115+0000", 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0343000000000001e-05 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698151+0000", 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6185e-05 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25126.0:15515 4.2 e292/284 ondisk, result = 0)", 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 178.016689383, 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22438891599999999, 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.502 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426006+0000", 2024-04-15T18:23:48.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8299999999999997e-07 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426009+0000", 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.543e-06 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650191+0000", 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22418169800000001 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650198+0000", 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.046e-06 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650248+0000", 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9849000000000003e-05 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650255+0000", 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4850000000000003e-06 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.650395+0000", 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013981200000000001 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:39779 4.2 4:4b8eaf80:::benchmark_data_smithi159_102124_object39778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:22.078163+0000", 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.36453196400001, 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.234329543, 2024-04-15T18:23:48.506 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 39779 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078163+0000", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078163+0000", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078165+0000", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3289999999999999e-06 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078189+0000", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3998e-05 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078200+0000", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1022000000000001e-05 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078208+0000", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8180000000000004e-06 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078263+0000", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5071000000000002e-05 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078407+0000", 2024-04-15T18:23:48.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000144893 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.078559+0000", 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015144999999999999 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080468+0000", 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019085949999999999 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080764+0000", 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029694600000000003 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312374+0000", 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23160956799999999 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312439+0000", 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5052e-05 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312493+0000", 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3801000000000002e-05 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:39783 4.2 4:408c9a27:::benchmark_data_smithi159_102124_object39782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e301)", 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:22.080849+0000", 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.36184648299999, 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23180234899999999, 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.512 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 39783 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080849+0000", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080849+0000", 2024-04-15T18:23:48.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080850+0000", 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.418e-06 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080884+0000", 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3893999999999998e-05 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080896+0000", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2245e-05 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080902+0000", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0730000000000001e-06 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.080954+0000", 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1150000000000003e-05 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.081060+0000", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106112 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.081155+0000", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5384999999999996e-05 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082688+0000", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001532872 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082821+0000", 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013262199999999999 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312544+0000", 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22972298499999999 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312605+0000", 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1705000000000006e-05 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312651+0000", 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5887999999999998e-05 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25126.0:39779 4.2 e301/284 ondisk, result = 0)", 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:22.082375+0000", 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.36032033699999, 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23012791199999999, 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.518 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082375+0000", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082375+0000", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082376+0000", 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0399999999999995e-07 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.082382+0000", 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1800000000000001e-06 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312304+0000", 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22992172499999999 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312309+0000", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0869999999999997e-06 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312355+0000", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6502999999999998e-05 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312367+0000", 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1837e-05 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.312503+0000", 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000135876 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:39871 4.2 4:4b8084d4:::benchmark_data_smithi159_102124_object39870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:22.353222+0000", 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.089472965, 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22576115799999999, 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 39871 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353222+0000", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353222+0000", 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353224+0000", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.626e-06 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353276+0000", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2096999999999999e-05 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353291+0000", 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5133999999999999e-05 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353300+0000", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5290000000000008e-06 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353365+0000", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5605999999999995e-05 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353521+0000", 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015587899999999999 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.353653+0000", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013134799999999999 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.355140+0000", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001487216 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.356697+0000", 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015572579999999999 2024-04-15T18:23:48.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.578911+0000", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.222214035 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.578945+0000", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4208999999999997e-05 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:22.578983+0000", 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8220999999999999e-05 2024-04-15T18:23:48.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:71486 4.2 4:46c4a592:::benchmark_data_smithi159_102124_object71485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.566479469, 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.308684452, 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 71486 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876217+0000", 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3880000000000001e-06 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876262+0000", 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4391000000000002e-05 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876277+0000", 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5481e-05 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876284+0000", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.374e-06 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876338+0000", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3653000000000003e-05 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876457+0000", 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118548 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876557+0000", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010054 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878524+0000", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019665110000000002 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878860+0000", 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00033609900000000002 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184840+0000", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30597998300000001 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184876+0000", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5997000000000001e-05 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184900+0000", 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4487000000000001e-05 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:89342 4.2 4:4b07b843:::benchmark_data_smithi159_102124_object89341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 97.021264252999998, 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36570210600000003, 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 89342 2024-04-15T18:23:48.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421433+0000", 2024-04-15T18:23:48.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.609e-06 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421458+0000", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5007999999999998e-05 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421472+0000", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4518999999999999e-05 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421481+0000", 2024-04-15T18:23:48.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.507e-06 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421547+0000", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5950000000000004e-05 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421686+0000", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013925 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421818+0000", 2024-04-15T18:23:48.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013244400000000001 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.428345+0000", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065266689999999997 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.435961+0000", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076156009999999996 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787063+0000", 2024-04-15T18:23:48.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.351102101 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787102+0000", 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9351000000000003e-05 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787133+0000", 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1096999999999997e-05 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.539 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:133645 4.6 4:60972f28:::benchmark_data_smithi159_102124_object44291:head [delete] snapc 0=[] ondisk+write+known_if_redirected e334)", 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:03.994369+0000", 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.448326715, 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18467803099999999, 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.540 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 133645 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994369+0000", 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994369+0000", 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994370+0000", 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.544e-06 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994380+0000", 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0356999999999999e-05 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994396+0000", 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5474000000000001e-05 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994405+0000", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8480000000000007e-06 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994472+0000", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7068000000000002e-05 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994617+0000", 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014471300000000001 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.994753+0000", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001362 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.998855+0000", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041023099999999996 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:03.999150+0000", 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029536299999999998 2024-04-15T18:23:48.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:04.178921+0000", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17977026099999999 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:04.178965+0000", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4601999999999997e-05 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:04.179047+0000", 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1291000000000005e-05 2024-04-15T18:23:48.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:136606 4.2 4:4e582662:::benchmark_data_smithi159_102124_object47252:head [delete] snapc 0=[] ondisk+write+known_if_redirected e336)", 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:08.319082+0000", 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.123613552000002, 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.194850097, 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:23:48.546 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 136606 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319082+0000", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319082+0000", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319084+0000", 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.759e-06 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319092+0000", 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4759999999999998e-06 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319110+0000", 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7839e-05 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319120+0000", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.007e-05 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319186+0000", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6130000000000006e-05 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319335+0000", 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014857600000000001 2024-04-15T18:23:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.319442+0000", 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010757299999999999 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:23:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.322710+0000", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032680029999999998 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513786+0000", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.191075623 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513833+0000", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7512000000000002e-05 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:23:48.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513889+0000", 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5674000000000003e-05 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513932+0000", 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2861999999999997e-05 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25126.0:136606 4.2 e336/284 ondisk, result = 0)", 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:08.323690+0000", 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.119004967000002, 2024-04-15T18:23:48.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19024616899999999, 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.323690+0000", 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.323690+0000", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.323691+0000", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4600000000000004e-07 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.323696+0000", 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:23:48.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.509953+0000", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18625665299999999 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.509960+0000", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5250000000000002e-06 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513823+0000", 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038629129999999999 2024-04-15T18:23:48.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513828+0000", 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9460000000000003e-06 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:08.513937+0000", 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010801799999999999 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:23:48.556 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:23:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:48.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:48.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:48.717 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:48.786 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:48.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:48.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:48.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:23:48.944 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.000 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:49.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:49.132 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.154 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:49.209 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:23:49.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:49.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:49.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:49.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:49 smithi049 ceph-mon[40087]: pgmap v954: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 868 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-04-15T18:23:49.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:23:49.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:23:49.329 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.355 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:49.386 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:49.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:49.402 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:49.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:49.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:49.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:49.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:49.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:49 smithi082 ceph-mon[44206]: pgmap v954: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 868 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-04-15T18:23:49.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:23:49.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:23:49.521 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.542 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:49.554 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:49.559 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:49.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:49.630 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:49.648 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:49 smithi159 ceph-mon[45336]: pgmap v954: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 868 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 794 op/s 2024-04-15T18:23:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:23:49.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:23:49.668 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:49.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:49.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:49.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:49.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:49.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:49.693 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.823 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:49.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:49.955 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:49.977 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:49.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:50.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:50.146 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1151681272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osdmap e346: 12 total, 11 up, 11 in 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.239 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.361 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:50.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:50.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1151681272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:50.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osdmap e346: 12 total, 11 up, 11 in 2024-04-15T18:23:50.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1151681272' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:50.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osdmap e346: 12 total, 11 up, 11 in 2024-04-15T18:23:50.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.403 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:50.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:50.504 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:50.965 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:50.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:51.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:51.120 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:51.179 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:51.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:51.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:51.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:51.343 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:51.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:51.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: pgmap v956: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 821 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 708 op/s 2024-04-15T18:23:51.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: 4.d scrub starts 2024-04-15T18:23:51.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:51 smithi159 ceph-mon[45336]: 4.d scrub ok 2024-04-15T18:23:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:51.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:51.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:51.500 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:51.501 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:23:51.501 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:23:51.530 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:51.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: pgmap v956: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 821 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 708 op/s 2024-04-15T18:23:51.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: 4.d scrub starts 2024-04-15T18:23:51.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:51 smithi082 ceph-mon[44206]: 4.d scrub ok 2024-04-15T18:23:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:51.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:51.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:51.671 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:51.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: pgmap v956: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 821 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 708 op/s 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: 4.d scrub starts 2024-04-15T18:23:51.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:51 smithi049 ceph-mon[40087]: 4.d scrub ok 2024-04-15T18:23:51.699 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:51.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:51.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:51.831 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:51.850 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:51.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:51.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:51.991 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:52.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:52 smithi082 ceph-mon[44206]: 4.f deep-scrub ok 2024-04-15T18:23:52.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:52 smithi082 ceph-mon[44206]: 4.9 deep-scrub ok 2024-04-15T18:23:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:52 smithi159 ceph-mon[45336]: 4.f deep-scrub ok 2024-04-15T18:23:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:52 smithi159 ceph-mon[45336]: 4.9 deep-scrub ok 2024-04-15T18:23:52.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:52 smithi049 ceph-mon[40087]: 4.f deep-scrub ok 2024-04-15T18:23:52.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:52 smithi049 ceph-mon[40087]: 4.9 deep-scrub ok 2024-04-15T18:23:53.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:23:53.253 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:23:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:53.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:53.358 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:53.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:53.418 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:53.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:53.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:23:53.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:53.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:53 smithi082 ceph-mon[44206]: pgmap v957: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 767 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:23:53.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1677138722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:53 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.553 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:53.583 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:53.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:53.604 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:53.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:53 smithi049 ceph-mon[40087]: pgmap v957: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 767 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:23:53.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1677138722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:53 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:53 smithi159 ceph-mon[45336]: pgmap v957: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 767 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:23:53.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1677138722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:53 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:23:53.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:23:53.752 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:54.110 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:23:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:54.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:23:54.181 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:54.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:54.253 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:54.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:54 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:54.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:54 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:54.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:54 smithi049 ceph-mon[40087]: osdmap e347: 12 total, 11 up, 11 in 2024-04-15T18:23:54.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:54 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:54.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:54 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:54.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:54 smithi082 ceph-mon[44206]: osdmap e347: 12 total, 11 up, 11 in 2024-04-15T18:23:54.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:54 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:23:54.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:54 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:23:54.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:54 smithi159 ceph-mon[45336]: osdmap e347: 12 total, 11 up, 11 in 2024-04-15T18:23:55.312 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:55.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:55.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:55.457 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:55.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:55 smithi049 ceph-mon[40087]: pgmap v959: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 669 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 601 op/s 2024-04-15T18:23:55.479 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:55.484 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:55 smithi159 ceph-mon[45336]: pgmap v959: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 669 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 601 op/s 2024-04-15T18:23:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:55.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:55.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:55 smithi082 ceph-mon[44206]: pgmap v959: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 669 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 601 op/s 2024-04-15T18:23:55.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:55.629 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:55.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:55.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:55.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:55.926 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:56.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:23:56.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:23:56.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:23:56.377 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:23:56.381 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:56.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:56.413 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:23:56.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:23:56.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:23:56.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:23:56.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:23:56.516 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:56.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:56 smithi082 ceph-mon[44206]: pgmap v960: 17 pgs: 17 active+clean; 617 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 502 op/s 2024-04-15T18:23:56.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1396993043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:56.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:56.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:56.627 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:56 smithi159 ceph-mon[45336]: pgmap v960: 17 pgs: 17 active+clean; 617 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 502 op/s 2024-04-15T18:23:56.628 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1396993043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:56.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:23:56.682 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:56.710 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:23:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:56.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:23:56.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:56 smithi049 ceph-mon[40087]: pgmap v960: 17 pgs: 17 active+clean; 617 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 502 op/s 2024-04-15T18:23:56.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1396993043' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:23:56.752 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:23:56.809 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:23:56.818 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:56.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:23:56.858 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:56.887 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:56.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:23:56.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:23:56.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:23:57.024 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:23:57.024 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:23:57.024 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:23:57.024 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:23:57.024 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:23:57.041 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:57.071 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:23:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:57.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:23:57.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:23:57.221 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:57.246 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:23:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:57.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.353 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.391 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:57.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:57.806 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:23:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:23:57.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:23:57.857 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:57.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:57.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:23:57.993 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:58.015 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:23:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:58.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:23:58.120 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:58.159 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:58.188 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:58.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:58.216 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:23:58.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:23:58.325 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:58.362 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:23:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:58.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:23:58.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:23:58.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:23:58 smithi049 ceph-mon[40087]: pgmap v961: 17 pgs: 17 active+clean; 501 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 636 op/s 2024-04-15T18:23:58.509 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:58.520 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.520 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.520 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.520 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.520 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:23:58 smithi159 ceph-mon[45336]: pgmap v961: 17 pgs: 17 active+clean; 501 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 636 op/s 2024-04-15T18:23:58.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:23:58.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:23:58 smithi082 ceph-mon[44206]: pgmap v961: 17 pgs: 17 active+clean; 501 MiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 636 op/s 2024-04-15T18:23:58.568 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:23:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:58.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:23:58.674 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:58.713 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:58.756 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:58.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:58.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:23:58.910 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:59.164 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:23:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:59.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:23:59.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:23:59.328 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:59.355 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:23:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:59.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:23:59.463 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:23:59.495 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:59.606 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:23:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:59.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:23:59.711 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:23:59.745 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:23:59.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:23:59.852 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:23:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:23:59.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:23:59.959 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:00.011 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:00.121 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:00.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:00.194 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:00.232 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:00.284 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:00.310 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:24:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:24:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:24:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:24:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:133614 2.7 e172/114 2:ef127ea4:::benchmark_data_smithi159_80260_object44501:head v 172'8316, mlcod=172'8313)", 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 595.04760151100004, 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19239903699999999, 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263899+0000", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263900+0000", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1039999999999999e-06 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.263909+0000", 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1810000000000007e-06 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453017+0000", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18910781300000001 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453026+0000", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3239999999999992e-06 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453087+0000", 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0713e-05 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.453111+0000", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4711000000000001e-05 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.456254+0000", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031426890000000002 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:05.456298+0000", 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3501999999999998e-05 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.314 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24511.0:157249 2.4 e177/114 2:266d3c4f:::benchmark_data_smithi159_80260_object68136:head v 177'9918, mlcod=177'9915)", 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:14:38.242957+0000", 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 562.06854291900004, 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.159099354, 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.242957+0000", 2024-04-15T18:24:00.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.242957+0000", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.242958+0000", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.85e-07 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.242967+0000", 2024-04-15T18:24:00.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9150000000000002e-06 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.398586+0000", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.15561931500000001 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.398596+0000", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4099999999999997e-06 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.398663+0000", 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6956000000000001e-05 2024-04-15T18:24:00.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.398679+0000", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6480000000000001e-05 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.402014+0000", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033344120000000001 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:14:38.402057+0000", 2024-04-15T18:24:00.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2981000000000001e-05 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:17328 3.c e206/199 3:35e737d2:::benchmark_data_smithi159_91621_object17327:head v 206'1098, mlcod=206'1095)", 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:59.094941+0000", 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 481.21655879999997, 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13605590300000001, 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.094941+0000", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.094941+0000", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.094942+0000", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.139e-06 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.094987+0000", 2024-04-15T18:24:00.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4929999999999998e-05 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230077+0000", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.135089929 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230086+0000", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3489999999999992e-06 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230140+0000", 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4744000000000003e-05 2024-04-15T18:24:00.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230163+0000", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2663999999999998e-05 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230955+0000", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00079190499999999997 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.230997+0000", 2024-04-15T18:24:00.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2243000000000002e-05 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:17331 3.c e206/199 3:3f6b6e0d:::benchmark_data_smithi159_91621_object17330:head v 206'1099, mlcod=206'1095)", 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:15:59.099398+0000", 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 481.21210166600002, 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13778400699999999, 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.099398+0000", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.099398+0000", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.099399+0000", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8700000000000005e-07 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.099429+0000", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0264999999999999e-05 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.236377+0000", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13694799399999999 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.236384+0000", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5950000000000004e-06 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.236435+0000", 2024-04-15T18:24:00.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0693999999999999e-05 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.236452+0000", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7609999999999999e-05 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.237126+0000", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00067331800000000001 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:15:59.237182+0000", 2024-04-15T18:24:00.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6844e-05 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:19463 3.1 e206/199 3:8ba0b04d:::benchmark_data_smithi159_91621_object19462:head v 206'1183, mlcod=206'1180)", 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:16:00.749932+0000", 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 479.56156800799999, 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.131892132, 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.749932+0000", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.749932+0000", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.749933+0000", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.142e-06 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.750010+0000", 2024-04-15T18:24:00.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6947999999999996e-05 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881036+0000", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13102562400000001 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881044+0000", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3999999999999992e-06 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881094+0000", 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0009999999999997e-05 2024-04-15T18:24:00.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881111+0000", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7110999999999999e-05 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881796+0000", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068439200000000003 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:16:00.881824+0000", 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8504999999999999e-05 2024-04-15T18:24:00.330 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:95294 3.c e228/199 3:30e75263:::benchmark_data_smithi159_91621_object95293:head v 228'5859, mlcod=228'5856)", 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:17:05.205981+0000", 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 415.10551863500001, 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13562581400000001, 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.205981+0000", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.205981+0000", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.205982+0000", 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3699999999999999e-07 2024-04-15T18:24:00.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.206007+0000", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4477999999999998e-05 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.339338+0000", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13333096999999999 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.339347+0000", 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0890000000000006e-06 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.339414+0000", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7151999999999997e-05 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.339433+0000", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9151e-05 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.341568+0000", 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002134702 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:17:05.341607+0000", 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9335000000000002e-05 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:157078 3.1 e255/199 3:893625ff:::benchmark_data_smithi159_91621_object50602:head v 255'9896, mlcod=255'9893)", 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:44.517300+0000", 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 315.79419991100002, 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14355564900000001, 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.335 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.517300+0000", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.517300+0000", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.517302+0000", 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.302e-06 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.517312+0000", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0166999999999999e-05 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.653739+0000", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.136427203 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.653745+0000", 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2179999999999996e-06 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.653801+0000", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5821e-05 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.653826+0000", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4766999999999999e-05 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.660817+0000", 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069917929999999996 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:44.660856+0000", 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8377999999999997e-05 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:167874 3.1 e259/199 3:806e9cd7:::benchmark_data_smithi159_91621_object61398:head v 259'10553, mlcod=258'10550)", 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:19:03.485073+0000", 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 296.82642716499998, 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14009526, 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.339 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.485073+0000", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.485073+0000", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.485074+0000", 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3480000000000001e-06 2024-04-15T18:24:00.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.485086+0000", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.167e-05 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.620779+0000", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.135693434 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.620788+0000", 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.371e-06 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.620847+0000", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8903000000000002e-05 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.620868+0000", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0877e-05 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.625123+0000", 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004255633 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:03.625168+0000", 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5024000000000001e-05 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:169925 3.1 e260/199 3:8862914a:::benchmark_data_smithi159_91621_object63449:head v 260'10690, mlcod=260'10686)", 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:19:06.924511+0000", 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 293.38698880499999, 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14659908599999999, 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.343 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:06.924511+0000", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:06.924511+0000", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:06.924512+0000", 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0380000000000001e-06 2024-04-15T18:24:00.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:06.924519+0000", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1119999999999998e-06 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.066510+0000", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.141990222 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.066517+0000", 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9840000000000004e-06 2024-04-15T18:24:00.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.066574+0000", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7428e-05 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.066601+0000", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6879000000000001e-05 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.071052+0000", 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044508710000000003 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:07.071110+0000", 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8551999999999998e-05 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:192127 3.1 e269/199 3:85270e68:::benchmark_data_smithi159_91621_object85651:head v 269'12132, mlcod=269'12129)", 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:19:43.228685+0000", 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 257.08281463600002, 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13541866399999999, 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.228685+0000", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.228685+0000", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.228687+0000", 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.127e-06 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.228695+0000", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0320000000000003e-06 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.360431+0000", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13173652799999999 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.360446+0000", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.523e-05 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.360577+0000", 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013056799999999999 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.360640+0000", 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3251999999999997e-05 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.364051+0000", 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034107809999999999 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:19:43.364104+0000", 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3146000000000002e-05 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:24833 4.7 e293/284 4:e3858863:::benchmark_data_smithi159_102124_object24832:head v 293'1494, mlcod=293'1492)", 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:56.604879+0000", 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 183.706621205, 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19330673000000001, 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.604879+0000", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.604879+0000", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.604880+0000", 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.17e-06 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.604930+0000", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0185999999999997e-05 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.797207+0000", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.192276326 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.797213+0000", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0859999999999998e-06 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.797303+0000", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0272999999999994e-05 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.797327+0000", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4202999999999998e-05 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.798137+0000", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00080944600000000002 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:56.798186+0000", 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9039999999999998e-05 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:26048 4.e e294/284 4:7d493ce7:::benchmark_data_smithi159_102124_object26047:head v 294'1546, mlcod=294'1543)", 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 182.640225507, 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20374900400000001, 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671341+0000", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5304999999999999e-05 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874017+0000", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.202676935 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874025+0000", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3660000000000003e-06 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874080+0000", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5652999999999997e-05 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874108+0000", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7107999999999999e-05 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874990+0000", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000882594 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.875024+0000", 2024-04-15T18:24:00.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.341e-05 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:70756 4.e e314/284 4:7abf6187:::benchmark_data_smithi159_102124_object70755:head v 314'4281, mlcod=314'4277)", 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.114705+0000", 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 131.196795533, 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13969288999999999, 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.114705+0000", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.114705+0000", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.114705+0000", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.114764+0000", 2024-04-15T18:24:00.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9237999999999998e-05 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.251181+0000", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13641679400000001 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.251190+0000", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7190000000000007e-06 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.251272+0000", 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2201999999999997e-05 2024-04-15T18:24:00.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.251288+0000", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5909999999999998e-05 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.254347+0000", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030584890000000002 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.254397+0000", 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0905000000000001e-05 2024-04-15T18:24:00.362 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:71501 4.e e315/284 4:774826b2:::benchmark_data_smithi159_102124_object71500:head v 315'4323, mlcod=315'4320)", 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.389943136, 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.341239132, 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:24:00.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921558+0000", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3e-07 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921622+0000", 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4045000000000001e-05 2024-04-15T18:24:00.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260680+0000", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.339058835 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260689+0000", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3729999999999993e-06 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260771+0000", 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2348999999999994e-05 2024-04-15T18:24:00.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260792+0000", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1185000000000001e-05 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.262759+0000", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019667769999999998 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.262796+0000", 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6937999999999998e-05 2024-04-15T18:24:00.366 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:72425 4.7 e316/284 4:e39e774d:::benchmark_data_smithi159_102124_object72424:head v 316'4498, mlcod=316'4494)", 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:50.957739+0000", 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.35376100400001, 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13622215100000001, 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.957739+0000", 2024-04-15T18:24:00.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.957739+0000", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.957740+0000", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1370000000000001e-06 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.957796+0000", 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5327000000000001e-05 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.092090+0000", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13429459599999999 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.092097+0000", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.652e-06 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.092151+0000", 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4540000000000003e-05 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.092168+0000", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6951000000000001e-05 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.093929+0000", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001760445 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.093961+0000", 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2502999999999997e-05 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.370 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:72548 4.7 e316/284 4:e397c969:::benchmark_data_smithi159_102124_object72547:head v 316'4508, mlcod=316'4504)", 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:51.262167+0000", 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.04933276400001, 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13570058099999999, 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.262167+0000", 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.262167+0000", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.262168+0000", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8100000000000001e-07 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.262232+0000", 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3697999999999999e-05 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.395881+0000", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13364847599999999 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.395887+0000", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1480000000000003e-06 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.395932+0000", 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5633000000000002e-05 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.395948+0000", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6066000000000002e-05 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.397835+0000", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001887031 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:51.397868+0000", 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2548000000000001e-05 2024-04-15T18:24:00.374 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:141420 4.a e337/284 4:5258642a:::benchmark_data_smithi159_102124_object52066:head v 337'8914, mlcod=337'8911)", 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:14.530479+0000", 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.781021484999997, 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.129125024, 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.530479+0000", 2024-04-15T18:24:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.530479+0000", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.530480+0000", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0839999999999999e-06 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.530489+0000", 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5680000000000005e-06 2024-04-15T18:24:00.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.656380+0000", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.12589044499999999 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.656403+0000", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3581999999999999e-05 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.656452+0000", 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8767000000000003e-05 2024-04-15T18:24:00.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.656472+0000", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9585e-05 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.659560+0000", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003088286 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:14.659604+0000", 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3707e-05 2024-04-15T18:24:00.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:148267 4.a e341/284 4:5a3846de:::benchmark_data_smithi159_102124_object58913:head v 341'9371, mlcod=341'9369)", 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:23.934844+0000", 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.376655872999997, 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.12204506, 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:23.934844+0000", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:23.934844+0000", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:23.934845+0000", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7000000000000003e-07 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:23.934854+0000", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9190000000000005e-06 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.052734+0000", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.117880454 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.052741+0000", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6529999999999997e-06 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.052793+0000", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2219000000000003e-05 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.052815+0000", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1840000000000001e-05 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.056845+0000", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040299710000000002 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:24.056889+0000", 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4134000000000001e-05 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:150462 4.e e341/284 4:78722d31:::benchmark_data_smithi159_102124_object61108:head v 341'9133, mlcod=341'9130)", 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:26.616411+0000", 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.695088738999999, 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.13002181400000001, 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.616411+0000", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.616411+0000", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.616412+0000", 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6300000000000004e-07 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.616420+0000", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4680000000000003e-06 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.743018+0000", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.126598192 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.743024+0000", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6760000000000004e-06 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.743069+0000", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5581000000000003e-05 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.743086+0000", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.66e-05 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.746395+0000", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033089600000000001 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:26.746433+0000", 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8473999999999998e-05 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:174517 4.e e347/284 4:71ecda1a:::benchmark_data_smithi159_102124_object85163:head v 347'10646, mlcod=347'10643)", 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:23:59.931676+0000", 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.37982435199999998, 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.055343476000000003, 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.931676+0000", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.931676+0000", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.931677+0000", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.221e-06 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.931685+0000", 2024-04-15T18:24:00.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2029999999999992e-06 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.978986+0000", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.047301041000000002 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.978995+0000", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.568e-06 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.979053+0000", 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8412000000000002e-05 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.979075+0000", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1554000000000001e-05 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.986972+0000", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078973670000000006 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:23:59.987019+0000", 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7110000000000001e-05 2024-04-15T18:24:00.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.391 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:24:00.391 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:24:00.391 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:24:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:00.392 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:00.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:00.493 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:00.522 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:00.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:00 smithi049 ceph-mon[40087]: pgmap v962: 17 pgs: 17 active+clean; 437 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 613 op/s 2024-04-15T18:24:00.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2841966534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.560 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:00.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:00 smithi159 ceph-mon[45336]: pgmap v962: 17 pgs: 17 active+clean; 437 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 613 op/s 2024-04-15T18:24:00.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2841966534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:00.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:00.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:00.704 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:00 smithi082 ceph-mon[44206]: pgmap v962: 17 pgs: 17 active+clean; 437 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 613 op/s 2024-04-15T18:24:00.704 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2841966534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.704 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:00.707 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:00.732 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:00.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:00.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:00.870 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:00.892 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:00.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:00.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:01.024 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:01.045 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:01.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:01.149 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:01.181 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:01.206 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:01.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:01.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:01.341 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:01.369 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:01.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:01.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:01 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:01.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:01 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:01.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:01 smithi049 ceph-mon[40087]: osdmap e348: 12 total, 11 up, 11 in 2024-04-15T18:24:01.473 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:01.500 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:01.529 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:01.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:01.637 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:01.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:01 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:01.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:01 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:01.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:01 smithi159 ceph-mon[45336]: osdmap e348: 12 total, 11 up, 11 in 2024-04-15T18:24:01.672 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:01.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:01 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:01.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:01 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:01.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:01 smithi082 ceph-mon[44206]: osdmap e348: 12 total, 11 up, 11 in 2024-04-15T18:24:01.847 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:01.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:01.874 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:24:01.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:02.011 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:02.040 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:02.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:02.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:02.205 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:02.288 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:24:02.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:02 smithi159 ceph-mon[45336]: pgmap v964: 17 pgs: 17 active+clean; 338 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 658 op/s 2024-04-15T18:24:02.337 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:02.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:02.442 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:02.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:02 smithi049 ceph-mon[40087]: pgmap v964: 17 pgs: 17 active+clean; 338 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 658 op/s 2024-04-15T18:24:02.475 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:02.499 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:02.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:02.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:02.633 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:02.662 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:02.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:02.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:02.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:02 smithi082 ceph-mon[44206]: pgmap v964: 17 pgs: 17 active+clean; 338 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 658 op/s 2024-04-15T18:24:02.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:02.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:02.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:03.029 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:03.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:03.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:24:03.129 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:03.152 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:03.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:03.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:03.213 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:03.241 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:03.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:03.307 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:03.358 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:03.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:03.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:03.489 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:03.492 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:03.520 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:03.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:03.529 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:03.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:03.585 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2584812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:03.585 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.585 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:03.657 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2584812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:03.658 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.658 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.660 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:03.685 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:03.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:03.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2584812606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:03.790 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:03.825 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:03.853 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:03.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:03.957 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:03.987 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:04.012 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:04.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:04.120 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:04.157 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:04.242 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:04.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:04.299 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:24:04.299 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:24:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:24:04.328 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:24:04.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:04.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:24:04.380 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:04.415 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:04.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:04.524 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: pgmap v965: 17 pgs: 17 active+clean; 194 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:24:04.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.570 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: pgmap v965: 17 pgs: 17 active+clean; 194 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:24:04.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: pgmap v965: 17 pgs: 17 active+clean; 194 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-04-15T18:24:04.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:04.784 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:05.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:05.028 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:24:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:05.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:05.132 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:05.180 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:05.213 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:05.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:05.318 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:05.363 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:05.391 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:05.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:05.460 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:24:05.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:05.523 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:05.542 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:05.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:05.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:05.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.228 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:06.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:06.274 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 89351 objects 2024-04-15T18:24:06.274 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :112.976 2024-04-15T18:24:06.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:24:06.333 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:06.363 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:06.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:06.489 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:06.518 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.538 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:06.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:06.641 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:06.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:06 smithi159 ceph-mon[45336]: pgmap v966: 17 pgs: 17 active+clean; 134 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:24:06.670 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.690 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:06.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:06.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:06 smithi049 ceph-mon[40087]: pgmap v966: 17 pgs: 17 active+clean; 134 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:24:06.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:06 smithi082 ceph-mon[44206]: pgmap v966: 17 pgs: 17 active+clean; 134 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:24:06.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:06.821 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.841 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:06.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:06.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:06.977 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:06.997 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:07.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:07.101 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:07.130 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:07.149 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:07.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:07.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:07.280 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:24:07.281 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:24:07.283 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:07.337 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:24:07.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:07.570 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:07.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:07.943 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:24:08.203 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:24:08.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:08 smithi159 ceph-mon[45336]: pgmap v967: 17 pgs: 17 active+clean; 43 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:24:08.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:08 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:08.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:08.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:08 smithi159 ceph-mon[45336]: osdmap e349: 12 total, 11 up, 11 in 2024-04-15T18:24:08.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:08 smithi049 ceph-mon[40087]: pgmap v967: 17 pgs: 17 active+clean; 43 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:24:08.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:08 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:08.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:08.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:08 smithi049 ceph-mon[40087]: osdmap e349: 12 total, 11 up, 11 in 2024-04-15T18:24:08.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:08 smithi082 ceph-mon[44206]: pgmap v967: 17 pgs: 17 active+clean; 43 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-04-15T18:24:08.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:08 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:08.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3936199335' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:08.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:08 smithi082 ceph-mon[44206]: osdmap e349: 12 total, 11 up, 11 in 2024-04-15T18:24:08.830 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-15T18:24:08.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-15T18:24:09.133 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:09.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:09.238 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:09.269 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:09.291 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:09.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:09.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:09.425 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:09.445 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:09.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:09.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:09.577 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:09.700 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:09.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:09.803 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:09.835 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:09.855 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:09.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:09.957 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:24:09.958 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:24:09.959 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:09.987 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:10.006 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:10.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:10.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:10.137 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:10.189 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:10.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:10.293 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:10.320 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:10.340 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:10.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:10.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:10.496 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:10.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:10.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:10.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:10 smithi049 ceph-mon[40087]: pgmap v969: 17 pgs: 17 active+clean; 8.9 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 646 op/s 2024-04-15T18:24:10.631 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:10.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:10 smithi159 ceph-mon[45336]: pgmap v969: 17 pgs: 17 active+clean; 8.9 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 646 op/s 2024-04-15T18:24:10.650 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:10.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:10.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:24:10.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:10.755 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:10.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:10 smithi082 ceph-mon[44206]: pgmap v969: 17 pgs: 17 active+clean; 8.9 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 646 op/s 2024-04-15T18:24:10.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:11.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:11.205 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.225 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:11.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:11.363 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.383 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:11.488 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:11.518 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:11.518 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:11.519 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:11 smithi159 ceph-mon[45336]: osdmap e350: 12 total, 11 up, 11 in 2024-04-15T18:24:11.519 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2457913315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:11.521 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.542 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:11.547 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:11.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:11.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:11.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:11.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:11.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:11 smithi049 ceph-mon[40087]: osdmap e350: 12 total, 11 up, 11 in 2024-04-15T18:24:11.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2457913315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:11.672 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:11.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:11.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:11.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:11 smithi082 ceph-mon[44206]: osdmap e350: 12 total, 11 up, 11 in 2024-04-15T18:24:11.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2457913315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:11.678 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:11.699 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:11.722 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:11.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:24:11.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:11.797 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:11.803 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:11.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:11.831 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:11.847 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:11.850 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:11.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:11.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:11.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:11.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:11.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:11.897 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:24:11.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:11.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:11.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:11.985 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:12.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: pgmap v971: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 387 op/s 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: pgmap v971: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 387 op/s 2024-04-15T18:24:12.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: pgmap v971: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 387 op/s 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:12.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.434 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=2303642217,config=1981513354,health=1997216375,kv=1580638300,logm=378681375}) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661267647}) 2024-04-15T18:24:13.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2445997866}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128376766}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517627527}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665468165}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876968645}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:24:13.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790500624}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018635117}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689843949}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359700823}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294486141}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062161047}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307891516}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145139276}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227492076}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056145104}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339764093}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277008716}) 2024-04-15T18:24:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942470351}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3682624949}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=621367901}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3697692228}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938510445}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476979255}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883648820}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800339504}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526086032}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620597859}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912368067}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4214998585}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310990745}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111277302}) 2024-04-15T18:24:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452527995}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605656306}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=64,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12} crc {logm=2553923131,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=372857809}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=4,mon_config_key=96} crc {mgrstat=1521512103,mon_config_key=388300720}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3371526220}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=1041185889,monmap=1029477071,osd_metadata=2999419095}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=3604443825,osd_pg_creating=230151435,osdmap=2004052147}) 2024-04-15T18:24:13.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3724040541}) 2024-04-15T18:24:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:13.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:13.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:24:13.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:24:13.576 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:13.650 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=2303642217,config=1981513354,health=1997216375,kv=1580638300,logm=378681375}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661267647}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2445997866}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128376766}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517627527}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665468165}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876968645}) 2024-04-15T18:24:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790500624}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018635117}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689843949}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359700823}) 2024-04-15T18:24:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294486141}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062161047}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307891516}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145139276}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227492076}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056145104}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339764093}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277008716}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942470351}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3682624949}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=621367901}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3697692228}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938510445}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476979255}) 2024-04-15T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883648820}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800339504}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526086032}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620597859}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912368067}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4214998585}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310990745}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111277302}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452527995}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605656306}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=64,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12} crc {logm=2553923131,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=372857809}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=4,mon_config_key=96} crc {mgrstat=1521512103,mon_config_key=388300720}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3371526220}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=1041185889,monmap=1029477071,osd_metadata=2999419095}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=3604443825,osd_pg_creating=230151435,osdmap=2004052147}) 2024-04-15T18:24:13.655 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3724040541}) 2024-04-15T18:24:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:13.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:13.757 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=53,logm=2} crc {auth=2303642217,config=1981513354,health=1997216375,kv=1580638300,logm=378681375}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661267647}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2445997866}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128376766}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1517627527}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665468165}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3876968645}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:24:13.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1790500624}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018635117}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3689843949}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359700823}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294486141}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1062161047}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307891516}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145139276}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227492076}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4056145104}) 2024-04-15T18:24:13.788 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339764093}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4277008716}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942470351}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3682624949}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=621367901}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3697692228}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938510445}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476979255}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883648820}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800339504}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2526086032}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=620597859}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912368067}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4214998585}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310990745}) 2024-04-15T18:24:13.789 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4111277302}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452527995}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3605656306}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=64,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12} crc {logm=2553923131,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=372857809}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=4,mon_config_key=96} crc {mgrstat=1521512103,mon_config_key=388300720}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3371526220}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=1041185889,monmap=1029477071,osd_metadata=2999419095}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=3604443825,osd_pg_creating=230151435,osdmap=2004052147}) 2024-04-15T18:24:13.790 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=33} crc {osdmap=3724040541}) 2024-04-15T18:24:13.791 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:13.810 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:13.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:13.914 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:13.945 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:13.965 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:13.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:14.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:14.099 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.152 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:14.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:14.286 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.307 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:14.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:14.439 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:14.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:14.595 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.617 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:14.619 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:14 smithi159 ceph-mon[45336]: pgmap v972: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 267 op/s 2024-04-15T18:24:14.619 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:14.619 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:14.662 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:14.703 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:14.703 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:24:14.704 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:24:14.704 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:14.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:14 smithi049 ceph-mon[40087]: pgmap v972: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 267 op/s 2024-04-15T18:24:14.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:14.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:14.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:14.747 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:14 smithi082 ceph-mon[44206]: pgmap v972: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 267 op/s 2024-04-15T18:24:14.748 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:14.748 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:14.749 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.769 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:14.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:14.906 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:14.925 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:14.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:15.029 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:15.059 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:15.079 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:15.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:15.182 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:15.211 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:15.231 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:15.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:15.334 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:15.368 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' removed 2024-04-15T18:24:15.370 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:15.403 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:15.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:15.512 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:15.539 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:15.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:24:15.558 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:15.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:15.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:24:15.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:15 smithi049 ceph-mon[40087]: osdmap e351: 12 total, 11 up, 11 in 2024-04-15T18:24:15.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:15.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:15.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:24:15.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:15 smithi159 ceph-mon[45336]: osdmap e351: 12 total, 11 up, 11 in 2024-04-15T18:24:15.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:15.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:15.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:24:15.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:15.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:15 smithi082 ceph-mon[44206]: osdmap e351: 12 total, 11 up, 11 in 2024-04-15T18:24:15.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:15.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:15.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:24:15.898 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:15.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:16.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:16.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.055 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:16.160 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:16.190 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:16.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:16.346 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.366 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:16.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:16.497 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.516 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:16.621 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:16 smithi159 ceph-mon[45336]: pgmap v974: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 85 op/s 2024-04-15T18:24:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:24:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:16 smithi159 ceph-mon[45336]: osdmap e352: 12 total, 11 up, 11 in 2024-04-15T18:24:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:16.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:16.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:16 smithi049 ceph-mon[40087]: pgmap v974: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 85 op/s 2024-04-15T18:24:16.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:24:16.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:16 smithi049 ceph-mon[40087]: osdmap e352: 12 total, 11 up, 11 in 2024-04-15T18:24:16.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:16.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:16.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:16 smithi082 ceph-mon[44206]: pgmap v974: 17 pgs: 17 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 85 op/s 2024-04-15T18:24:16.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1622947676' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:24:16.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:16 smithi082 ceph-mon[44206]: osdmap e352: 12 total, 11 up, 11 in 2024-04-15T18:24:16.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:16.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:16.672 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.692 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:16.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:16.824 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:16.843 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:16.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:16.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:16.977 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:17.014 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:17.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:17.039 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:24:17.040 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:24:17.040 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-15T18:24:17.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-04-15T18:24:17.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:17.150 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:17.276 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:17.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:17.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:17.431 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:17.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:17.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:17.564 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:17.584 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:17.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:17.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:17 smithi159 ceph-mon[45336]: osdmap e353: 12 total, 11 up, 11 in 2024-04-15T18:24:17.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:17.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:17.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:17.713 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:17 smithi082 ceph-mon[44206]: osdmap e353: 12 total, 11 up, 11 in 2024-04-15T18:24:17.714 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:17.714 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:17.715 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:17.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:17 smithi049 ceph-mon[40087]: osdmap e353: 12 total, 11 up, 11 in 2024-04-15T18:24:17.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:17.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:17.734 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:17.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:17.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:17.868 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:18.382 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:24:18.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:18 smithi159 ceph-mon[45336]: pgmap v977: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:18.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:18 smithi159 ceph-mon[45336]: osdmap e354: 12 total, 11 up, 11 in 2024-04-15T18:24:18.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1811432370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:18 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:18 smithi049 ceph-mon[40087]: pgmap v977: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:18.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:18 smithi049 ceph-mon[40087]: osdmap e354: 12 total, 11 up, 11 in 2024-04-15T18:24:18.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1811432370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:18 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:18 smithi082 ceph-mon[44206]: pgmap v977: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:18.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:18 smithi082 ceph-mon[44206]: osdmap e354: 12 total, 11 up, 11 in 2024-04-15T18:24:18.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1811432370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:18 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:18.828 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:18.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:18.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:18.963 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:19.080 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-15T18:24:19.080 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-04-15T18:24:19.080 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T18:24:19.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2 2024-04-15T18:24:19.112 INFO:journalctl@ceph.osd.2.smithi049.stdout:-- Logs begin at Mon 2024-04-15 17:01:55 UTC. -- 2024-04-15T18:24:19.220 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:19 smithi049 systemd[1]: Starting Ceph osd.2 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:24:19.241 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:19.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:19.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:19.378 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:19.400 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:19.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:19.504 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:19.533 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:19.553 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:19.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:19.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:24:19.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:24:19.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:19.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: osdmap e355: 12 total, 11 up, 11 in 2024-04-15T18:24:19.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:19.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:19 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:19.656 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: osdmap e355: 12 total, 11 up, 11 in 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:19.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:19 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:19.684 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:19.703 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:19.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: osdmap e355: 12 total, 11 up, 11 in 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:19.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:19 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:19.807 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:19.833 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:19.853 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:19.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:19.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:19.982 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:20.001 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:20.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:20.140 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:20.409 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:20.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:20 smithi159 ceph-mon[45336]: pgmap v980: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:20.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:20 smithi159 ceph-mon[45336]: osdmap e356: 12 total, 11 up, 11 in 2024-04-15T18:24:20.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:20 smithi049 ceph-mon[40087]: pgmap v980: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:20.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:20 smithi049 ceph-mon[40087]: osdmap e356: 12 total, 11 up, 11 in 2024-04-15T18:24:20.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:20 smithi082 ceph-mon[44206]: pgmap v980: 1 pgs: 1 active+clean; 577 KiB data, 2.5 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:20.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:20 smithi082 ceph-mon[44206]: osdmap e356: 12 total, 11 up, 11 in 2024-04-15T18:24:21.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:21.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:21.598 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:21.630 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:21.651 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:21.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:21.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:21.786 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:21.807 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:21.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:21.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:21.944 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:22.143 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:22.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:22.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:22.279 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:22.301 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:22.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:22.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:22.437 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:22.459 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:22.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:22.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:22.595 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:22.722 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:22.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:22.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:22.857 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:22.982 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:22.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:23.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:23.115 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T18:24:23.115 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-15T18:24:23.115 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-15T18:24:23.116 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-15T18:24:23.116 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-15T18:24:23.116 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T18:24:23.116 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:22 smithi049 bash[175349]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-15T18:24:23.118 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:23.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:24:23.317 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:23.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:23.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:23.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:23 smithi049 ceph-mon[40087]: pgmap v982: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:23.453 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:23.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:23.489 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:23.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:23.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:23 smithi082 ceph-mon[44206]: pgmap v982: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:23.540 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:23.558 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:23.558 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:24:23.559 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:24:23.559 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:24:23.559 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:23.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:24:23.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:23.616 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:23.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:23 smithi159 ceph-mon[45336]: pgmap v982: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:23.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:23.869 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:23.894 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:23.919 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osdmap e357: 12 total, 11 up, 11 in 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1758111800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:24.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:24.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osdmap e357: 12 total, 11 up, 11 in 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1758111800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:24:24.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:24.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:24.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osdmap e357: 12 total, 11 up, 11 in 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1758111800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:24.907 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:24.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:25.011 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:25.046 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.064 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:25.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:25.199 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.219 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:25.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: pgmap v984: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.352 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.371 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:25.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: pgmap v984: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:25.502 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.521 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:25.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: pgmap v984: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:25.652 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.671 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:25.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:25.779 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:24:25.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:24:25.802 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:25.821 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:25.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:25.825 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:25.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:25.957 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.030 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:26.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:26.164 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.183 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:26.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:26.319 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.341 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:26.445 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:26.473 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.492 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:26.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:26.629 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.650 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:26.754 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:26.785 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.807 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:26.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:26.940 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:26.959 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:26.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:27.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:27.095 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.116 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:27.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:27.248 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.267 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:27.371 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:27.400 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:27 smithi082 ceph-mon[44206]: pgmap v985: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:27.403 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.424 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:27.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:27 smithi049 ceph-mon[40087]: pgmap v985: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:27.527 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:27.559 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.581 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:27.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:27 smithi159 ceph-mon[45336]: pgmap v985: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:27.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:24:27.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:27.713 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.733 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:27.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:24:27.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:27.869 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:27.891 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:27.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:27.970 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:27 smithi049 ceph-osd[176347]: -- 172.21.15.49:0/3902486409 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5600a22c1680 con 0x5600a228b400 2024-04-15T18:24:27.970 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:27 smithi049 bash[176137]: 9debc039fca3f87a41d359e7bb8b9d30d4be4d69a05510c6065bc4ba06880532 2024-04-15T18:24:27.970 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:27 smithi049 systemd[1]: Started Ceph osd.2 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:24:27.994 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:28.027 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.048 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:28.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:28.181 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.201 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:28.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:28.332 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.352 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:28.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:28.469 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:28 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:24:28.114+0000 7fc125a6a7c0 -1 Falling back to public interface 2024-04-15T18:24:28.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:28.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:28.642 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.668 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:28.773 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:28.805 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.827 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:28.932 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:28.961 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:28.980 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:28.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:29.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:29.133 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:29.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' created 2024-04-15T18:24:29.237 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:29.248 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:24:29.248 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:24:29.265 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:29.284 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:29.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:29.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:29.420 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:29.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:29 smithi082 ceph-mon[44206]: pgmap v986: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:29.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-15T18:24:29.545 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:29.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:29 smithi049 ceph-mon[40087]: pgmap v986: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:29.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-15T18:24:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:29.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:29.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:29.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:29 smithi159 ceph-mon[45336]: pgmap v986: 1 pgs: 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:29.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-15T18:24:29.678 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:30.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:24:30.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-15T18:24:30.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:30 smithi049 ceph-mon[40087]: osdmap e358: 12 total, 11 up, 11 in 2024-04-15T18:24:30.267 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:24:30.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:24:29.701+0000 7fbcf68c1700 0 --2- 172.21.15.49:0/169100980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcf01034c0 0x7fbcf010f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:24:30.422 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:24:30.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-15T18:24:30.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:30 smithi082 ceph-mon[44206]: osdmap e358: 12 total, 11 up, 11 in 2024-04-15T18:24:30.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:24:30.627 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:30.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:30.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/229318382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-15T18:24:30.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:30 smithi159 ceph-mon[45336]: osdmap e358: 12 total, 11 up, 11 in 2024-04-15T18:24:30.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:24:30.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:30.763 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:30.785 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:30.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:30.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:30.920 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:30.942 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:30.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:30.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-15T18:24:31.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:31.073 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:31.094 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:31.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:31.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:31.220 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:24:31.014+0000 7fc125a6a7c0 -1 osd.2 307 log_to_monitors true 2024-04-15T18:24:31.230 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:31.250 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:31.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:31.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:31.385 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:31.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: pgmap v988: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:31.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:24:31.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:31.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:24:31.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: osdmap e359: 12 total, 11 up, 11 in 2024-04-15T18:24:31.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:24:31.510 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:31.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: pgmap v988: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: osdmap e359: 12 total, 11 up, 11 in 2024-04-15T18:24:31.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:31 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:24:31.539 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:31.547 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:31.616 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:31.648 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: pgmap v988: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4064662582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: osdmap e359: 12 total, 11 up, 11 in 2024-04-15T18:24:31.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:31 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T18:24:31.669 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:31.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:31.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:31.798 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:24:31 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:24:31.548+0000 7fc112eba700 -1 osd.2 360 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:24:31.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:31.820 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:31.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:31.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:31.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:31.954 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:32.027 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:32.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:32.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:32.161 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:32.424 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:32 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:24:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:32 smithi049 ceph-mon[40087]: osdmap e360: 12 total, 11 up, 11 in 2024-04-15T18:24:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:32 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:24:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:32 smithi049 ceph-mon[40087]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' 2024-04-15T18:24:32.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:32 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:24:32.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:32 smithi082 ceph-mon[44206]: osdmap e360: 12 total, 11 up, 11 in 2024-04-15T18:24:32.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:32 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:24:32.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:32.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:32 smithi082 ceph-mon[44206]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' 2024-04-15T18:24:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:32 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T18:24:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:32 smithi159 ceph-mon[45336]: osdmap e360: 12 total, 11 up, 11 in 2024-04-15T18:24:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:32 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-04-15T18:24:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:32 smithi159 ceph-mon[45336]: from='osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388]' entity='osd.2' 2024-04-15T18:24:33.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:33.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:33.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:33 smithi049 ceph-mon[40087]: pgmap v991: 17 pgs: 4 active+clean, 5 creating+peering, 8 unknown; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:33.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:33 smithi049 ceph-mon[40087]: osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388] boot 2024-04-15T18:24:33.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:33 smithi049 ceph-mon[40087]: osdmap e361: 12 total, 12 up, 11 in 2024-04-15T18:24:33.330 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:33.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:33.449 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:33.470 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:33.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:33.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:33 smithi082 ceph-mon[44206]: pgmap v991: 17 pgs: 4 active+clean, 5 creating+peering, 8 unknown; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:33.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:33 smithi082 ceph-mon[44206]: osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388] boot 2024-04-15T18:24:33.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:33 smithi082 ceph-mon[44206]: osdmap e361: 12 total, 12 up, 11 in 2024-04-15T18:24:33.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:33.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:24:33.573 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:33.605 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:33.608 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:33.627 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:33.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:33 smithi159 ceph-mon[45336]: pgmap v991: 17 pgs: 4 active+clean, 5 creating+peering, 8 unknown; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:33 smithi159 ceph-mon[45336]: osd.2 [v2:172.21.15.49:6818/2410603388,v1:172.21.15.49:6819/2410603388] boot 2024-04-15T18:24:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:33 smithi159 ceph-mon[45336]: osdmap e361: 12 total, 12 up, 11 in 2024-04-15T18:24:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T18:24:33.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:33.761 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:33.781 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:33.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:33.885 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:33.914 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:33.934 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:33.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:34.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:34.065 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:34.084 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:34.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:34.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:34.222 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:34.242 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:34.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:34.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:34.376 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:34.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:34.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:34.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: pgmap v993: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:34.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:34 smithi049 ceph-mon[40087]: osdmap e362: 12 total, 12 up, 11 in 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: pgmap v993: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:34 smithi082 ceph-mon[44206]: osdmap e362: 12 total, 12 up, 11 in 2024-04-15T18:24:34.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:34.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:34.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:34.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: pgmap v993: 17 pgs: 12 active+clean, 5 creating+peering; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:34.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:34 smithi159 ceph-mon[45336]: osdmap e362: 12 total, 12 up, 11 in 2024-04-15T18:24:34.929 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:34.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:35.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:35.166 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:35.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:35.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:35.299 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:35.425 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:35.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:35.530 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:35.561 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:35.583 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:35.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:35.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:24:35.611 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:35.636 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:35.661 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:35.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:35.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:35.715 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:35.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2210204188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:35.735 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:35.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:35.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:35.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:35.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2210204188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:35.839 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:35.869 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2210204188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:35.871 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:35.892 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:35.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:35.996 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:36.026 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:36.046 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:36.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:24:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:24:36.141 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:36.150 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:36.179 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:36.200 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:36.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:36.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:36.334 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:36.354 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:36.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:36.400 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:36.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:36.486 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:36.506 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:36.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:36.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:36.641 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:36.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: pgmap v995: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:36.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:36.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:36.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: pgmap v995: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:36.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:36.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:36.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: pgmap v995: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:36.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:36.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:37.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:24:37.390 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:37.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:37.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:37.523 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:37.722 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:37.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:37.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:37.864 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:37.865 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:37 smithi159 ceph-mon[45336]: osdmap e363: 12 total, 12 up, 11 in 2024-04-15T18:24:37.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:37.888 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:37.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:37 smithi049 ceph-mon[40087]: osdmap e363: 12 total, 12 up, 11 in 2024-04-15T18:24:37.992 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:38.021 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:38.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:37 smithi082 ceph-mon[44206]: osdmap e363: 12 total, 12 up, 11 in 2024-04-15T18:24:38.041 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:38.130 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:38.130 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:24:38.130 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:24:38.130 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:38.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:38.173 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.322 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:38.426 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:38.456 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.477 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:38.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:38.609 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.629 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:38.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.785 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:38.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:38.918 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:38.936 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:38.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:39.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:24:39.072 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:39.094 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:24:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:39.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:24:39.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:39.227 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:39.246 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:39.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:39.350 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:39.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:39 smithi082 ceph-mon[44206]: pgmap v997: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:39.382 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:39.404 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:39.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:39 smithi049 ceph-mon[40087]: pgmap v997: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:39.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:39.540 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:39 smithi159 ceph-mon[45336]: pgmap v997: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:39.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:24:39.786 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3, 2] 2024-04-15T18:24:39.786 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-15T18:24:39.787 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5 2024-04-15T18:24:40.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:24:40.069 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:39 smithi082 systemd[1]: Stopping Ceph osd.5 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:24:40.191 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:40.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:40.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:40.323 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:40 smithi082 ceph-mon[44206]: osd.5 marked itself down and dead 2024-04-15T18:24:40.323 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[57379]: 2024-04-15T18:24:40.073+0000 7fc24ffb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:24:40.323 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[57379]: 2024-04-15T18:24:40.073+0000 7fc24ffb2700 -1 osd.5 363 *** Got signal Terminated *** 2024-04-15T18:24:40.323 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[57379]: 2024-04-15T18:24:40.073+0000 7fc24ffb2700 -1 osd.5 363 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:24:40.325 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:40.344 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:40.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:40.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:40.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:40 smithi049 ceph-mon[40087]: osd.5 marked itself down and dead 2024-04-15T18:24:40.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:40 smithi159 ceph-mon[45336]: osd.5 marked itself down and dead 2024-04-15T18:24:40.481 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:40.503 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:40.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:40.607 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:40.637 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:40.657 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:40.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:40.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:40.791 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:40.990 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:40.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:41.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:41.124 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:41.217 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:41 smithi082 bash[116897]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5 2024-04-15T18:24:41.235 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:41 smithi049 ceph-mon[40087]: pgmap v998: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:41.235 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:41 smithi049 ceph-mon[40087]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:24:41.235 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:41 smithi049 ceph-mon[40087]: osdmap e364: 12 total, 11 up, 11 in 2024-04-15T18:24:41.261 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:41.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:41.365 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:41.394 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:41.413 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:41.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:41.466 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:41.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:41 smithi082 ceph-mon[44206]: pgmap v998: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:41 smithi082 ceph-mon[44206]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:24:41.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:41 smithi082 ceph-mon[44206]: osdmap e364: 12 total, 11 up, 11 in 2024-04-15T18:24:41.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:41 smithi159 ceph-mon[45336]: pgmap v998: 17 pgs: 17 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:41.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:41 smithi159 ceph-mon[45336]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:24:41.547 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:41 smithi159 ceph-mon[45336]: osdmap e364: 12 total, 11 up, 11 in 2024-04-15T18:24:41.549 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:41.571 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:41.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:41.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:41.707 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:42.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:42.061 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:42.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:42.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:42.196 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:42.231 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:24:42.231 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-15T18:24:42.395 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:42.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:42.403 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:42 smithi082 ceph-mon[44206]: osdmap e365: 12 total, 11 up, 11 in 2024-04-15T18:24:42.403 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:42.403 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.403 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:42 smithi082 ceph-mon[44206]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:42 smithi049 ceph-mon[40087]: osdmap e365: 12 total, 11 up, 11 in 2024-04-15T18:24:42.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:42.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:42 smithi049 ceph-mon[40087]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:42.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:42 smithi159 ceph-mon[45336]: osdmap e365: 12 total, 11 up, 11 in 2024-04-15T18:24:42.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:24:42.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.530 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:42 smithi159 ceph-mon[45336]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:24:42.532 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:42.554 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:42.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:42.629 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-15T18:24:42.655 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:42 smithi082 systemd[1]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5.service: Succeeded. 2024-04-15T18:24:42.656 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:24:42 smithi082 systemd[1]: Stopped Ceph osd.5 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:24:42.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:42.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:24:42.684 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-15T18:24:42.684 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-04-15T18:24:42.684 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi082.front.sepia.ceph.com 2024-04-15T18:24:42.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.5 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-04-15T18:24:42.686 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:42.708 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:42.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:42.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:42.813 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:42.845 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:42.867 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:42.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:42.889 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.5/config 2024-04-15T18:24:42.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:43.000 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:43.021 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:43.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:43.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:43.156 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:43.175 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:43.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:43.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:43.306 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:43.378 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:43.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:43.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:43 smithi082 ceph-mon[44206]: pgmap v1001: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:43.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:43 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:43.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:43.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:43 smithi082 ceph-mon[44206]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:24:43.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:43 smithi082 ceph-mon[44206]: osdmap e366: 12 total, 11 up, 11 in 2024-04-15T18:24:43.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:43.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:43 smithi049 ceph-mon[40087]: pgmap v1001: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:43.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:43 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:43.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:43.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:43 smithi049 ceph-mon[40087]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:24:43.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:43 smithi049 ceph-mon[40087]: osdmap e366: 12 total, 11 up, 11 in 2024-04-15T18:24:43.512 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:43.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:43 smithi159 ceph-mon[45336]: pgmap v1001: 17 pgs: 1 peering, 2 stale+active+clean, 14 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:43.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:43 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:43.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1584470319' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:24:43.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:43 smithi159 ceph-mon[45336]: from='client.15696 172.21.15.49:0/223762360' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:24:43.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:43 smithi159 ceph-mon[45336]: osdmap e366: 12 total, 11 up, 11 in 2024-04-15T18:24:43.762 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:43.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:43.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:43.894 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:44.092 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:44.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:44.126 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:24:44.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:44.224 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:44.344 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:44 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:24:44.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:44 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:24:44.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:44 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:24:44.688 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:24:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:24:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:24:44.689 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:45.087 INFO:teuthology.orchestra.run.smithi082.stdout:5.f 2024-04-15T18:24:45.087 INFO:teuthology.orchestra.run.smithi082.stdout:5.4 2024-04-15T18:24:45.088 INFO:teuthology.orchestra.run.smithi082.stdout:5.6 2024-04-15T18:24:45.088 INFO:teuthology.orchestra.run.smithi082.stdout:5.b 2024-04-15T18:24:45.088 INFO:teuthology.orchestra.run.smithi082.stdout:5.7 2024-04-15T18:24:45.088 INFO:teuthology.orchestra.run.smithi082.stdout:5.3 2024-04-15T18:24:45.088 INFO:teuthology.orchestra.run.smithi082.stdout:5.d 2024-04-15T18:24:45.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:24:45.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: pgmap v1003: 17 pgs: 1 peering, 5 active+undersized, 11 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:45.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:24:45.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: pgmap v1003: 17 pgs: 1 peering, 5 active+undersized, 11 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: pgmap v1003: 17 pgs: 1 peering, 5 active+undersized, 11 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:45.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:24:45.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:45.703 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:45.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:45.807 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:45.836 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:45.959 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:45.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:45.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.5 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 5.3 --file /var/log/ceph/exp.5.3.5 2024-04-15T18:24:46.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:46.093 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:46.112 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:46.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:46.217 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:46.225 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.5/config 2024-04-15T18:24:46.251 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:46.273 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:46.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:46.378 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:46.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:24:46.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:24:46.386 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:46.408 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:46.428 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:46.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:46.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:24:46.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:24:46.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:46.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:24:46.561 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:46.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:24:46.641 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:24:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:24:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:24:46.659 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:24:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:24:46.697 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-15T18:24:46.699 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:24:46.700 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:24:46.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:24:46.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:24:46.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_116570 2024-04-15T18:24:46.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:24:46.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:24:46.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:46.801 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:47.006 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:47.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:47.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:47.148 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:47.175 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:47.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:47.280 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:47.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: pgmap v1004: 17 pgs: 1 peering, 6 active+undersized, 10 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: osdmap e367: 12 total, 11 up, 11 in 2024-04-15T18:24:47.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3582592730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:47.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:47.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: pgmap v1004: 17 pgs: 1 peering, 6 active+undersized, 10 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:47.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:47.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:47.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:24:47.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:24:47.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: osdmap e367: 12 total, 11 up, 11 in 2024-04-15T18:24:47.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3582592730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:47.495 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:47.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:47.522 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:47.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:47.539 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:47.568 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:24:47.570 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:24:47.603 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:47.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: pgmap v1004: 17 pgs: 1 peering, 6 active+undersized, 10 active+clean; 577 KiB data, 2.6 GiB used, 981 GiB / 983 GiB avail 2024-04-15T18:24:47.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:47.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:47.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:24:47.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:24:47.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: osdmap e367: 12 total, 11 up, 11 in 2024-04-15T18:24:47.608 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3582592730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:47.629 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:47.639 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:47.666 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:47.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:47.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:47.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:47.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 15 3405 3390 211.869 211.875 0.00157221 0.00470189 2024-04-15T18:24:47.774 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:47.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:47.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:47.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:47.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:47.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:24:47.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:47.871 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:47.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:47.975 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:48.010 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:48.138 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:48.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:48.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:48.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.289 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:48.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:48.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:48.599 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:48.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.631 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:48.633 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:48.658 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:48.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:48.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 5978 5962 186.302 160.75 0.0031311 0.00534594 2024-04-15T18:24:48.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:48.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:48.821 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:48.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:48.828 INFO:teuthology.orchestra.run.smithi082.stderr:Exporting 5.3 info 5.3( empty local-lis/les=358/359 n=0 ec=358/358 lis/c=358/358 les/c/f=359/359/0 sis=358) 2024-04-15T18:24:48.828 INFO:teuthology.orchestra.run.smithi082.stderr:Export successful 2024-04-15T18:24:48.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:48.957 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:48.981 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:48.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:49.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:49.118 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:49.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: pgmap v1006: 17 pgs: 2 active+undersized+degraded, 5 active+undersized, 10 active+clean; 27 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 4.4 MiB/s wr, 70 op/s; 214/1290 objects degraded (16.589%) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:49 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 214/1290 objects degraded (16.589%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:49.460 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: pgmap v1006: 17 pgs: 2 active+undersized+degraded, 5 active+undersized, 10 active+clean; 27 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 4.4 MiB/s wr, 70 op/s; 214/1290 objects degraded (16.589%) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.461 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:49 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 214/1290 objects degraded (16.589%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:49.469 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:49.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:24:49.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:24:49.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:49.609 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: pgmap v1006: 17 pgs: 2 active+undersized+degraded, 5 active+undersized, 10 active+clean; 27 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 4.4 MiB/s wr, 70 op/s; 214/1290 objects degraded (16.589%) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:49 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 214/1290 objects degraded (16.589%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:49.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 8544 8528 177.656 160.375 0.00403273 0.0056047 2024-04-15T18:24:49.808 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:49.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:49.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.5 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 5.3 2024-04-15T18:24:49.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:49.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:49.961 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:49.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:50.065 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:50.066 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.5/config 2024-04-15T18:24:50.096 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:50.119 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:50.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:50.223 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:50.258 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:50.364 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:50.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:50.468 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:50.505 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:50.531 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:50.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:50.594 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:24:50.594 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:24:50.635 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:50.667 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:50.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 10536 10520 164.365 124.5 0.0175821 0.00607229 2024-04-15T18:24:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:50.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:51.023 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:51.049 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:51.069 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:51.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:51.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:51.205 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:51.227 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:51.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:51.332 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:51.363 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:51 smithi159 ceph-mon[45336]: pgmap v1007: 17 pgs: 4 active+undersized+degraded, 3 active+undersized, 10 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 10 MiB/s wr, 166 op/s; 667/4014 objects degraded (16.617%) 2024-04-15T18:24:51.363 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:51.366 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:51.391 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:51.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:51.413 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:24:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:24:51.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:24:51.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:51.521 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:51 smithi049 ceph-mon[40087]: pgmap v1007: 17 pgs: 4 active+undersized+degraded, 3 active+undersized, 10 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 10 MiB/s wr, 166 op/s; 667/4014 objects degraded (16.617%) 2024-04-15T18:24:51.521 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:51.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:51 smithi082 ceph-mon[44206]: pgmap v1007: 17 pgs: 4 active+undersized+degraded, 3 active+undersized, 10 active+clean; 84 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 10 MiB/s wr, 166 op/s; 667/4014 objects degraded (16.617%) 2024-04-15T18:24:51.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:24:51.555 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:51.638 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:51.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:51.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:51.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 12274 12258 153.215 108.625 0.00494601 0.00651437 2024-04-15T18:24:51.779 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:52.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:52 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:52.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:52.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:52 smithi049 ceph-mon[40087]: osdmap e368: 12 total, 11 up, 11 in 2024-04-15T18:24:52.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:52 smithi049 ceph-mon[40087]: osdmap e369: 12 total, 11 up, 11 in 2024-04-15T18:24:52.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:24:52.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:52 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:52.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:52.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:52 smithi082 ceph-mon[44206]: osdmap e368: 12 total, 11 up, 11 in 2024-04-15T18:24:52.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:52 smithi082 ceph-mon[44206]: osdmap e369: 12 total, 11 up, 11 in 2024-04-15T18:24:52.534 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:52.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:52.579 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:52.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:52 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:24:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2207506950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:24:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:52 smithi159 ceph-mon[45336]: osdmap e368: 12 total, 11 up, 11 in 2024-04-15T18:24:52.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:52 smithi159 ceph-mon[45336]: osdmap e369: 12 total, 11 up, 11 in 2024-04-15T18:24:52.679 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:52.702 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:52.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:52.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 14019 14003 145.854 109.062 0.00295225 0.00683794 2024-04-15T18:24:52.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:24:52.834 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:52.854 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:24:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:52.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:24:52.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:52.992 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.020 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:53.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:53.161 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.185 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:53.290 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:53.319 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.342 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:53.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:53.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:53 smithi049 ceph-mon[40087]: pgmap v1009: 17 pgs: 6 active+undersized+degraded, 1 active+undersized, 10 active+clean; 310 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 39 MiB/s wr, 618 op/s; 1948/14850 objects degraded (13.118%) 2024-04-15T18:24:53.478 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.498 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:53.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:53 smithi082 ceph-mon[44206]: pgmap v1009: 17 pgs: 6 active+undersized+degraded, 1 active+undersized, 10 active+clean; 310 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 39 MiB/s wr, 618 op/s; 1948/14850 objects degraded (13.118%) 2024-04-15T18:24:53.585 INFO:teuthology.orchestra.run.smithi082.stdout: marking collection for removal 2024-04-15T18:24:53.585 INFO:teuthology.orchestra.run.smithi082.stdout:setting '_remove' omap key 2024-04-15T18:24:53.590 INFO:teuthology.orchestra.run.smithi082.stdout:finish_remove_pgs 5.3_head removing 5.3 2024-04-15T18:24:53.590 INFO:teuthology.orchestra.run.smithi082.stdout:Remove successful 2024-04-15T18:24:53.602 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:24:53.628 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.649 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:24:53.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:53 smithi159 ceph-mon[45336]: pgmap v1009: 17 pgs: 6 active+undersized+degraded, 1 active+undersized, 10 active+clean; 310 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 39 MiB/s wr, 618 op/s; 1948/14850 objects degraded (13.118%) 2024-04-15T18:24:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:24:53.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:53.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 15 15508 15493 138.321 93.125 0.0114632 0.00721821 2024-04-15T18:24:53.783 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.803 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:53.819 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:53.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:53.937 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:53.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:53.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:54.060 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:24:54.096 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:54.121 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:24:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:54.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:24:54.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:24:54.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:54.292 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:54.349 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:54.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:54.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:54.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:54 smithi082 ceph-mon[44206]: pgmap v1011: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 744 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s; 5450/35685 objects degraded (15.273%) 2024-04-15T18:24:54.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:54 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5450/35685 objects degraded (15.273%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:54.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:54 smithi049 ceph-mon[40087]: pgmap v1011: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 744 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s; 5450/35685 objects degraded (15.273%) 2024-04-15T18:24:54.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:54 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5450/35685 objects degraded (15.273%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:54.498 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:54.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:54 smithi159 ceph-mon[45336]: pgmap v1011: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 744 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 110 MiB/s wr, 1.76k op/s; 5450/35685 objects degraded (15.273%) 2024-04-15T18:24:54.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:54 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5450/35685 objects degraded (15.273%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:24:54.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 17266 17250 134.756 109.812 0.00319437 0.0074097 2024-04-15T18:24:54.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:24:54.798 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:24:54.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.5 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.3.5 2024-04-15T18:24:54.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:24:54.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:24:54.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:24:54.899 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:24:54.925 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:24:55.027 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.5/config 2024-04-15T18:24:55.047 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:24:55.053 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:55.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:55.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:55.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:24:55.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:24:55.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:24:55.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:24:55.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:55.191 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:55.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3681855574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:55.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3681855574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:55.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3681855574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:24:55.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:55.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 18340 18324 127.241 67.125 0.0260366 0.00784251 2024-04-15T18:24:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:24:55.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:24:56.141 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:56.148 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:56.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:56.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:24:56.286 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:56.390 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:24:56.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:56 smithi049 ceph-mon[40087]: pgmap v1012: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 844 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s; 5948/40497 objects degraded (14.688%) 2024-04-15T18:24:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:56.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:24:56.429 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:24:56.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:56.529 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:56.552 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:56.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:56 smithi159 ceph-mon[45336]: pgmap v1012: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 844 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s; 5948/40497 objects degraded (14.688%) 2024-04-15T18:24:56.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:24:56.692 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.710 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:24:56.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:56 smithi082 ceph-mon[44206]: pgmap v1012: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 844 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s; 5948/40497 objects degraded (14.688%) 2024-04-15T18:24:56.717 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:24:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:56.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:24:56.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 20160 20144 125.891 113.75 0.009387 0.00793735 2024-04-15T18:24:56.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:56.855 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:56.879 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:56.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:56.982 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:57.012 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:57.033 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:57.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:57.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:57.196 INFO:teuthology.orchestra.run.smithi082.stdout:get_pg_num_history pg_num_history pg_num_history(e363 pg_nums {1={19=1},4={284=16},5={358=16}} deleted_pools 352,4) 2024-04-15T18:24:57.196 INFO:teuthology.orchestra.run.smithi082.stdout:Importing pgid 5.3 2024-04-15T18:24:57.196 INFO:teuthology.orchestra.run.smithi082.stdout:write_pg epoch 359 info 5.3( empty local-lis/les=358/359 n=0 ec=358/358 lis/c=358/358 les/c/f=359/359/0 sis=358) 2024-04-15T18:24:57.196 INFO:teuthology.orchestra.run.smithi082.stdout:Import successful 2024-04-15T18:24:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:57.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:57.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:24:57.415 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:21135 5.c 5:39233d89:::benchmark_data_smithi159_116570_object21134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e370)", 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:24:57.434512+0000", 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0026623520000000002, 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026961419999999999, 2024-04-15T18:24:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 21135 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434512+0000", 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434512+0000", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434512+0000", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8199999999999999e-07 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434579+0000", 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6925e-05 2024-04-15T18:24:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434588+0000", 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8009999999999993e-06 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434593+0000", 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.242e-06 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:24:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434641+0000", 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7388000000000001e-05 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434733+0000", 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2245999999999997e-05 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.434835+0000", 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101739 2024-04-15T18:24:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:24:57.435674+0000", 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00083894500000000001 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-04-15T18:24:57.437 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:24:57.438 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:24:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:57.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:24:57.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:57.573 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:57.597 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:24:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:57.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:24:57.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:24:57.743 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:57.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 21816 21800 123.852 103.5 0.00703995 0.00806201 2024-04-15T18:24:57.769 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:24:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:57.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:24:57.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:24:57.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:57.913 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:58.086 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:58.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:58.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:24:58.229 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:58.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:58 smithi049 ceph-mon[40087]: osdmap e370: 12 total, 11 up, 11 in 2024-04-15T18:24:58.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:58 smithi082 ceph-mon[44206]: osdmap e370: 12 total, 11 up, 11 in 2024-04-15T18:24:58.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:24:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:58 smithi159 ceph-mon[45336]: osdmap e370: 12 total, 11 up, 11 in 2024-04-15T18:24:58.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 23597 23581 122.806 111.312 0.0154463 0.00813242 2024-04-15T18:24:58.781 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:24:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:58.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:24:58.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:24:58.923 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:59.028 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:24:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:24:59.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:24:59.059 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:59.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:24:59.163 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:24:59.343 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:24:59.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:24:59 smithi082 ceph-mon[44206]: pgmap v1014: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.1 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s; 7897/53547 objects degraded (14.748%) 2024-04-15T18:24:59.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:24:59 smithi049 ceph-mon[40087]: pgmap v1014: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.1 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s; 7897/53547 objects degraded (14.748%) 2024-04-15T18:24:59.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:24:59 smithi159 ceph-mon[45336]: pgmap v1014: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.1 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 134 MiB/s wr, 2.14k op/s; 7897/53547 objects degraded (14.748%) 2024-04-15T18:24:59.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 24921 24905 119.725 82.75 0.00747424 0.00834125 2024-04-15T18:25:00.237 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:25:00.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/934573109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:00.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:00.572 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/exp.5.3.5 2024-04-15T18:25:00.617 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:00.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:00.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/934573109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:00.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/934573109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:00.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:00.749 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:00.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 26794 26778 119.534 117.062 0.00175524 0.00835628 2024-04-15T18:25:00.769 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:00.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:00.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:00.900 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:00.919 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:00.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:01.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:01.059 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:01.082 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:01.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:01.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:01.218 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:01.242 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:01.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:01.346 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:01.374 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:01 smithi159 ceph-mon[45336]: pgmap v1015: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.2 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 8900/59484 objects degraded (14.962%) 2024-04-15T18:25:01.374 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:01 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8900/59484 objects degraded (14.962%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:01.374 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:01 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:01.374 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:01 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:01.374 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:01 smithi159 ceph-mon[45336]: osdmap e371: 12 total, 11 up, 11 in 2024-04-15T18:25:01.376 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:01.399 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:01.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:01.504 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:01.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:01 smithi082 ceph-mon[44206]: pgmap v1015: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.2 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 8900/59484 objects degraded (14.962%) 2024-04-15T18:25:01.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:01 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8900/59484 objects degraded (14.962%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:01.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:01 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:01.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:01 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:01.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:01 smithi082 ceph-mon[44206]: osdmap e371: 12 total, 11 up, 11 in 2024-04-15T18:25:01.535 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:01.558 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:01.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:01.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:01.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:01 smithi049 ceph-mon[40087]: pgmap v1015: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.2 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 8900/59484 objects degraded (14.962%) 2024-04-15T18:25:01.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:01 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8900/59484 objects degraded (14.962%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:01.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:01 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:01.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:01 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:01.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:01 smithi049 ceph-mon[40087]: osdmap e371: 12 total, 11 up, 11 in 2024-04-15T18:25:01.696 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:01.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 28201 28185 117.427 87.9375 0.00373019 0.00850396 2024-04-15T18:25:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:01.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:01.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:01.982 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:02.004 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:02.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:02.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:02.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:02.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:02.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:02.529 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:02.555 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:02.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:02.611 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:02.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:02.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:02.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:02.714 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:02.738 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:02.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:02.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 29371 29355 114.658 73.125 0.152643 0.00866374 2024-04-15T18:25:03.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: pgmap v1017: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 9931/68406 objects degraded (14.518%) 2024-04-15T18:25:03.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osdmap e372: 12 total, 11 up, 11 in 2024-04-15T18:25:03.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/7962137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:03.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:25:03.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: pgmap v1017: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 9931/68406 objects degraded (14.518%) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osdmap e372: 12 total, 11 up, 11 in 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/7962137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: pgmap v1017: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 9931/68406 objects degraded (14.518%) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osdmap e372: 12 total, 11 up, 11 in 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/7962137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:03.702 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:03.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:03.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 30583 30567 112.368 75.75 0.00167209 0.008855 2024-04-15T18:25:03.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:03.837 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:03.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:25:03.940 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:03.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:04.001 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:04.001 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:04.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:04.077 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:04.094 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:04.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:04.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:04.230 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:04.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:04.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:04.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:04.394 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:04.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:25:04.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:04 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:25:04.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:04 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:25:04.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:04 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:25:04.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:25:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:04.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:04.758 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:04.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 32111 32095 111.431 95.5 0.0181406 0.0089668 2024-04-15T18:25:04.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:04.882 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:05.030 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:05.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:05.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:05.165 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:05.187 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:05.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:05.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:05.299 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:05 smithi049 ceph-mon[40087]: pgmap v1019: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s; 12437/83409 objects degraded (14.911%) 2024-04-15T18:25:05.324 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:05.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:25:05.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:05 smithi082 ceph-mon[44206]: pgmap v1019: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s; 12437/83409 objects degraded (14.911%) 2024-04-15T18:25:05.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [2] dead_osds: [5] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2] 2024-04-15T18:25:05.650 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:25:05.651 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-04-15T18:25:05.651 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-04-15T18:25:05.651 DEBUG:teuthology.orchestra.run.smithi082:osd.5> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T18:25:05.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:05 smithi159 ceph-mon[45336]: pgmap v1019: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 122 MiB/s wr, 1.95k op/s; 12437/83409 objects degraded (14.911%) 2024-04-15T18:25:05.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl start ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5 2024-04-15T18:25:05.688 INFO:journalctl@ceph.osd.5.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:25:05.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 33549 33533 110.296 89.875 0.00539388 0.00906054 2024-04-15T18:25:05.978 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:05.978 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:05.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:06.032 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:05 smithi082 systemd[1]: Starting Ceph osd.5 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:25:06.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:06.121 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:06.147 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:06.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:06.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:06.293 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:06.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:06 smithi159 ceph-mon[45336]: pgmap v1020: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.8 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 12884/87351 objects degraded (14.750%) 2024-04-15T18:25:06.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:06 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12884/87351 objects degraded (14.750%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:06.421 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:06.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:06 smithi082 ceph-mon[44206]: pgmap v1020: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.8 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 12884/87351 objects degraded (14.750%) 2024-04-15T18:25:06.425 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:06 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12884/87351 objects degraded (14.750%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:06.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:06.519 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:06 smithi049 ceph-mon[40087]: pgmap v1020: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 1.8 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 12884/87351 objects degraded (14.750%) 2024-04-15T18:25:06.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:06 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12884/87351 objects degraded (14.750%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:06.526 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:06.562 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:06.587 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:06.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:06.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:06.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:06.755 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:06.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:06.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:25:06.767127+0000 min lat: 0.00106705 max lat: 0.24196 avg lat: 0.0091101 2024-04-15T18:25:06.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:25:06.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 35122 35106 109.696 98.3125 0.0148738 0.0091101 2024-04-15T18:25:06.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:06.897 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:07.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:07.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:07.204 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:07.253 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:07.270 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T18:25:07.270 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-15T18:25:07.270 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-15T18:25:07.270 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-15T18:25:07.270 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-15T18:25:07.271 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T18:25:07.271 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:06 smithi082 bash[120205]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-15T18:25:07.279 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:07.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:07.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:07.423 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:07.628 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:07.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:07.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:07.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 36851 36835 109.618 108.062 0.0273849 0.0091174 2024-04-15T18:25:07.772 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:07.798 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:07.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:07.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:07.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:07.966 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:07.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:08.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:08.107 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:08.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:08.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:08.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:25:08.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:08 smithi049 ceph-mon[40087]: osdmap e373: 12 total, 11 up, 11 in 2024-04-15T18:25:08.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:08.235 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:08.240 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:25:08.265 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:08.286 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:08.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:08.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:08.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:08 smithi159 ceph-mon[45336]: osdmap e373: 12 total, 11 up, 11 in 2024-04-15T18:25:08.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:08.418 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:08.436 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:08.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:08.502 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:08.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:08 smithi082 ceph-mon[44206]: osdmap e373: 12 total, 11 up, 11 in 2024-04-15T18:25:08.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:08.532 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:08 smithi082 ceph-osd[120552]: -- 172.21.15.82:0/3726958806 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560fe4f3680 con 0x5560fe4bd000 2024-04-15T18:25:08.533 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:08 smithi082 bash[120515]: e6da92ef3304db4ebe784b18794dcf4d0060bae31ad614ffa9a1a39d04b8ed18 2024-04-15T18:25:08.533 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:08 smithi082 systemd[1]: Started Ceph osd.5 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:25:08.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:08.570 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:08.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 38173 38157 108.391 82.625 0.00253944 0.0092189 2024-04-15T18:25:08.989 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:08 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:25:08.655+0000 7f6a8f7847c0 -1 Falling back to public interface 2024-04-15T18:25:09.021 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:09.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:09.124 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:09.150 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:09.169 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:09.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:09.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:09.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:09 smithi082 ceph-mon[44206]: pgmap v1022: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 14647/99255 objects degraded (14.757%) 2024-04-15T18:25:09.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:09 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:09.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:09.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:09 smithi082 ceph-mon[44206]: osdmap e374: 12 total, 11 up, 11 in 2024-04-15T18:25:09.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:09 smithi049 ceph-mon[40087]: pgmap v1022: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 14647/99255 objects degraded (14.757%) 2024-04-15T18:25:09.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:09 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:09.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:09.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:09 smithi049 ceph-mon[40087]: osdmap e374: 12 total, 11 up, 11 in 2024-04-15T18:25:09.305 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:09.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:09 smithi159 ceph-mon[45336]: pgmap v1022: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 14647/99255 objects degraded (14.757%) 2024-04-15T18:25:09.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:09 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:09.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1974314324' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:09.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:09 smithi159 ceph-mon[45336]: osdmap e374: 12 total, 11 up, 11 in 2024-04-15T18:25:09.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 39549 39533 107.417 86 0.00238988 0.00930319 2024-04-15T18:25:09.857 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:09.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:09.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:09.988 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:09.988 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:09.994 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:10.246 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:10.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:10.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:10.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-04-15T18:25:10.413 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-04-15T18:25:10.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:25:10.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 40934 40918 106.548 86.5625 0.0474424 0.00936913 2024-04-15T18:25:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-15T18:25:11.128 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-04-15T18:25:11.156 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:11.360 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:11 smithi082 ceph-mon[44206]: pgmap v1024: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 15629/104907 objects degraded (14.898%) 2024-04-15T18:25:11.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:11.445 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:11 smithi049 ceph-mon[40087]: pgmap v1024: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 15629/104907 objects degraded (14.898%) 2024-04-15T18:25:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:11.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:11.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:11.577 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:11.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:11.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:11 smithi159 ceph-mon[45336]: pgmap v1024: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 15629/104907 objects degraded (14.898%) 2024-04-15T18:25:11.658 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:11 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:25:11.363+0000 7f6a8f7847c0 -1 osd.5 363 log_to_monitors true 2024-04-15T18:25:11.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:11.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:11.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 42031 42015 105.028 68.5625 0.00517781 0.00951349 2024-04-15T18:25:11.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:11.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:11.828 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:11.860 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:11.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:25:11.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:11.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:11.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:11.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:12.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:12.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:12.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:12.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:12.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:12.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:12.086 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:12.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:12.139 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:12.163 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:12.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:12.218 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:12.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:12.282 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:25:12.246+0000 7f6a80bdc700 -1 osd.5 363 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:25:12.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1927274359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:12.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 15629/104907 objects degraded (14.898%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:12.298 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:12.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1927274359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:12.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 15629/104907 objects degraded (14.898%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:12.460 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:12.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T18:25:12.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1927274359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:12.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:12.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 15629/104907 objects degraded (14.898%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:12.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:12.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:12.637 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:12.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 43222 43206 103.852 74.4375 0.00464196 0.00962221 2024-04-15T18:25:12.891 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:12.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:12.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:13.025 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:13.149 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:13.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43587 5.1 5:863143b5:::benchmark_data_smithi159_116570_object43586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:13.163441+0000", 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.010645706, 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01067739, 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:25:13.174 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43587 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163441+0000", 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163441+0000", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163442+0000", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8899999999999998e-07 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163493+0000", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0667999999999997e-05 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163504+0000", 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0934999999999999e-05 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163511+0000", 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2660000000000004e-06 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163563+0000", 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1709999999999998e-05 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163651+0000", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8178999999999997e-05 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.163737+0000", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6210000000000003e-05 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.164734+0000", 2024-04-15T18:25:13.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00099682299999999998 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.171092+0000", 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0063584590000000003 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:43604 5.a e375/358 5:5d92a293:::benchmark_data_smithi159_116570_object43603:head v 375'2752, mlcod=375'2749)", 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:13.173123+0000", 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.00096361500000000002, 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010881350000000001, 2024-04-15T18:25:13.179 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173123+0000", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173123+0000", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:13.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173124+0000", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8000000000000005e-07 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173385+0000", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000261141 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173403+0000", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8185e-05 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173411+0000", 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3889999999999999e-06 2024-04-15T18:25:13.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173463+0000", 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2040999999999999e-05 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.173478+0000", 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5444999999999999e-05 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43606 5.1 5.b7dcfc01 (undecoded) ondisk+write+known_if_redirected e375)", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:13.174145+0000", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4294967295.9999413, 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000128909, 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43606 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174145+0000", 2024-04-15T18:25:13.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174145+0000", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174147+0000", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3480000000000001e-06 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174210+0000", 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3075000000000002e-05 2024-04-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174224+0000", 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3886000000000001e-05 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:13.174232+0000", 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1589999999999993e-06 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 3 2024-04-15T18:25:13.185 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:13.253 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:13.296 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:13.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:13.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:13.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:13.447 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:13.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: pgmap v1025: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 16510/113163 objects degraded (14.590%) 2024-04-15T18:25:13.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: osdmap e375: 12 total, 11 up, 11 in 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:13 smithi082 ceph-mon[44206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: pgmap v1025: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 16510/113163 objects degraded (14.590%) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: osdmap e375: 12 total, 11 up, 11 in 2024-04-15T18:25:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:13 smithi159 ceph-mon[45336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: pgmap v1025: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 16510/113163 objects degraded (14.590%) 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: osdmap e375: 12 total, 11 up, 11 in 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: from='osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:13 smithi049 ceph-mon[40087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:25:13.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 43734 43718 101.19 32 0.00259861 0.00972765 2024-04-15T18:25:13.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:25:14.168 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:14.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:14.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:14.301 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:14.328 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:14.427 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:14.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:25:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481] boot 2024-04-15T18:25:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: osdmap e376: 12 total, 12 up, 11 in 2024-04-15T18:25:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:25:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:14 smithi049 ceph-mon[40087]: osdmap e377: 12 total, 12 up, 11 in 2024-04-15T18:25:14.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:25:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481] boot 2024-04-15T18:25:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: osdmap e376: 12 total, 12 up, 11 in 2024-04-15T18:25:14.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:25:14.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:14.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:14.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:14 smithi082 ceph-mon[44206]: osdmap e377: 12 total, 12 up, 11 in 2024-04-15T18:25:14.562 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:14.584 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:14.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: osd.5 [v2:172.21.15.82:6808/1630882481,v1:172.21.15.82:6809/1630882481] boot 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: osdmap e376: 12 total, 12 up, 11 in 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:14.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:14 smithi159 ceph-mon[45336]: osdmap e377: 12 total, 12 up, 11 in 2024-04-15T18:25:14.688 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:14.716 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:14.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 43734 43718 97.576 0 - 0.00972765 2024-04-15T18:25:14.913 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:14.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:15.017 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:25:15.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:25:15.049 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:15.070 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:15.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:15.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:15.204 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:15.251 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:15.252 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:15 smithi082 ceph-mon[44206]: pgmap v1028: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 18603/125421 objects degraded (14.832%) 2024-04-15T18:25:15.252 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:15.252 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:15.252 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:15 smithi082 ceph-mon[44206]: osdmap e378: 12 total, 12 up, 11 in 2024-04-15T18:25:15.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:15.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:15 smithi159 ceph-mon[45336]: pgmap v1028: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 18603/125421 objects degraded (14.832%) 2024-04-15T18:25:15.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:15.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:15.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:15 smithi159 ceph-mon[45336]: osdmap e378: 12 total, 12 up, 11 in 2024-04-15T18:25:15.456 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:15 smithi049 ceph-mon[40087]: pgmap v1028: 17 pgs: 7 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 18603/125421 objects degraded (14.832%) 2024-04-15T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:15.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:15 smithi049 ceph-mon[40087]: osdmap e378: 12 total, 12 up, 11 in 2024-04-15T18:25:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:15.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:15.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:15.601 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:15.629 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:15.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:15.735 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:15.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 43988 43972 94.7586 7.9375 0.00364997 0.010546 2024-04-15T18:25:15.794 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:15.837 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:15.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:15.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:15.984 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.010 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:16.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:16.120 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:16.120 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:25:16.121 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:25:16.121 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:25:16.121 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:16.129 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:25:16.153 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:16.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:16.396 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:16.548 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:16.583 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.608 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:16.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:16.750 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 45018 45002 93.7457 64.375 0.0264387 0.0106588 2024-04-15T18:25:16.776 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:16.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:16.909 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:16.930 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:16.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:16.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:25:17.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:17.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:25:17.080 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:17.106 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:17.115 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:17.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:17.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:17.229 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:25:17.278 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: pgmap v1031: 17 pgs: 1 remapped+peering, 6 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 16129/127626 objects degraded (12.638%) 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: osdmap e379: 12 total, 12 up, 11 in 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 16129/127626 objects degraded (12.638%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:17.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:17 smithi049 ceph-mon[40087]: osdmap e380: 12 total, 12 up, 11 in 2024-04-15T18:25:17.486 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:17.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: pgmap v1031: 17 pgs: 1 remapped+peering, 6 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 16129/127626 objects degraded (12.638%) 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: osdmap e379: 12 total, 12 up, 11 in 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 16129/127626 objects degraded (12.638%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:17.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:17 smithi082 ceph-mon[44206]: osdmap e380: 12 total, 12 up, 11 in 2024-04-15T18:25:17.590 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:17.615 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: pgmap v1031: 17 pgs: 1 remapped+peering, 6 active+undersized+degraded, 10 active+clean; 2.6 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 16129/127626 objects degraded (12.638%) 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: osdmap e379: 12 total, 12 up, 11 in 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 16129/127626 objects degraded (12.638%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2097659391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:17.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:17 smithi159 ceph-mon[45336]: osdmap e380: 12 total, 12 up, 11 in 2024-04-15T18:25:17.618 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:17.639 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:17.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:17.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:17.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 45591 45575 91.8768 35.8125 0.00971729 0.0108755 2024-04-15T18:25:17.781 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:18.113 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:18.138 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:18.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:18.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:18 smithi082 ceph-mon[44206]: pgmap v1034: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 584 op/s; 16377/132441 objects degraded (12.366%); 2.8 MiB/s, 44 objects/s recovering 2024-04-15T18:25:18.296 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:18.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:18 smithi049 ceph-mon[40087]: pgmap v1034: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 584 op/s; 16377/132441 objects degraded (12.366%); 2.8 MiB/s, 44 objects/s recovering 2024-04-15T18:25:18.438 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:18.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:18.488 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:18.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:18.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:18 smithi159 ceph-mon[45336]: pgmap v1034: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 584 op/s; 16377/132441 objects degraded (12.366%); 2.8 MiB/s, 44 objects/s recovering 2024-04-15T18:25:18.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 46503 46487 90.7868 57 0.145158 0.0109851 2024-04-15T18:25:18.880 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:18.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:18.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:19.014 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:19.033 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:19.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:19.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:19.169 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:19.190 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:19.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:19.295 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:19.338 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:19.448 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:19.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:19.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:19.596 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:19.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 47336 47320 89.6132 52.0625 0.0484719 0.0111511 2024-04-15T18:25:19.771 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:19.771 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:25:19.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.772 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:47332 5.f e380/377 5:fa13ed7c:::benchmark_data_smithi159_116570_object47331:head v 380'3053, mlcod=367'219)", 2024-04-15T18:25:19.772 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:19.768241+0000", 2024-04-15T18:25:19.772 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0043333759999999999, 2024-04-15T18:25:19.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043582250000000003, 2024-04-15T18:25:19.772 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:19.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768241+0000", 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768241+0000", 2024-04-15T18:25:19.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768242+0000", 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.175e-06 2024-04-15T18:25:19.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768268+0000", 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5434000000000001e-05 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768287+0000", 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9593999999999999e-05 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:19.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768297+0000", 2024-04-15T18:25:19.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2969999999999992e-06 2024-04-15T18:25:19.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:19.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.769972+0000", 2024-04-15T18:25:19.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001675258 2024-04-15T18:25:19.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.769997+0000", 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4879e-05 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(5.3 380/377 [PushOp(5:c9196089:::benchmark_data_smithi159_116570_object4399:head, version: 367'279, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c9196089:::benchmark_data_smithi159_116570_object4399:head@367'279, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:19.768457+0000", 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0041177109999999996, 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042009940000000004, 2024-04-15T18:25:19.781 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:19.782 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:19.782 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:25:19.782 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:19.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768457+0000", 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:19.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768457+0000", 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768458+0000", 2024-04-15T18:25:19.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3099999999999996e-07 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768541+0000", 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2837999999999995e-05 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:19.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768576+0000", 2024-04-15T18:25:19.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5033999999999997e-05 2024-04-15T18:25:19.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.768592+0000", 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6481999999999999e-05 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:19.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.788 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(5.d 380/377 [PushOp(5:b08037b4:::benchmark_data_smithi159_116570_object4127:head, version: 367'278, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b08037b4:::benchmark_data_smithi159_116570_object4127:head@367'278, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:25:19.788 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:19.771512+0000", 2024-04-15T18:25:19.789 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0010628160000000001, 2024-04-15T18:25:19.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011919140000000001, 2024-04-15T18:25:19.789 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:19.789 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:19.789 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771512+0000", 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771512+0000", 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:19.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771512+0000", 2024-04-15T18:25:19.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3e-07 2024-04-15T18:25:19.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771560+0000", 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7543999999999997e-05 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:19.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771701+0000", 2024-04-15T18:25:19.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014147699999999999 2024-04-15T18:25:19.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771719+0000", 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7226000000000001e-05 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:19.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771874+0000", 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015493500000000001 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771887+0000", 2024-04-15T18:25:19.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3661e-05 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.797 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:47337 5.3 e380/377)", 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:19.771572+0000", 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.001002887, 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011643510000000001, 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:19.798 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771572+0000", 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771572+0000", 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:19.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771572+0000", 2024-04-15T18:25:19.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2199999999999999e-07 2024-04-15T18:25:19.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771608+0000", 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5580999999999997e-05 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771752+0000", 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144325 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:19.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771757+0000", 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3279999999999996e-06 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:47339 5.3 e380/377)", 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:19.771621+0000", 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.00095336599999999998, 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001144878, 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:19.805 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:19.806 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:25:19.806 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:19.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:19.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771621+0000", 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771621+0000", 2024-04-15T18:25:19.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:19.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:19.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771622+0000", 2024-04-15T18:25:19.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1300000000000001e-07 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771780+0000", 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015883 2024-04-15T18:25:19.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.810 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.810 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:19.810 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771799+0000", 2024-04-15T18:25:19.810 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8159999999999999e-05 2024-04-15T18:25:19.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:19.771803+0000", 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7310000000000001e-06 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.811 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:19.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:19.812 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:25:19.812 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 5 2024-04-15T18:25:19.812 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:19.854 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:19.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:19.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:19.999 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:20.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:25:20.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:20.234 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:20.264 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:20.297 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:20.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:20.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:25:20.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:20.439 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:20.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:20 smithi049 ceph-mon[40087]: pgmap v1035: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 589 op/s; 16170/134340 objects degraded (12.037%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:25:20.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/173267126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:20.471 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:20.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:20.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:20.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:20.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:20.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:20.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:20.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:20.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:20.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:20 smithi159 ceph-mon[45336]: pgmap v1035: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 589 op/s; 16170/134340 objects degraded (12.037%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:25:20.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/173267126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:20.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:20 smithi082 ceph-mon[44206]: pgmap v1035: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 37 MiB/s wr, 589 op/s; 16170/134340 objects degraded (12.037%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:25:20.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/173267126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:20.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 48226 48210 88.6134 55.625 0.00385645 0.0112395 2024-04-15T18:25:20.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:21.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:21.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:21.151 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:21.509 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:21.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:21.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osdmap e381: 12 total, 12 up, 11 in 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.595 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:21.595 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:21.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:21.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osdmap e381: 12 total, 12 up, 11 in 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.679 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:21.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osdmap e381: 12 total, 12 up, 11 in 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:21.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 49219 49203 87.8546 62.0625 0.0177764 0.0113767 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: pgmap v1037: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 18230/138852 objects degraded (13.129%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 18230/138852 objects degraded (13.129%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:22.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:22 smithi159 ceph-mon[45336]: osdmap e382: 12 total, 12 up, 11 in 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: pgmap v1037: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 18230/138852 objects degraded (13.129%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 18230/138852 objects degraded (13.129%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:22.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:22 smithi082 ceph-mon[44206]: osdmap e382: 12 total, 12 up, 11 in 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: pgmap v1037: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 18230/138852 objects degraded (13.129%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 18230/138852 objects degraded (13.129%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:22 smithi049 ceph-mon[40087]: osdmap e382: 12 total, 12 up, 11 in 2024-04-15T18:25:22.740 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:22.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:22.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 50315 50299 87.3169 68.5 0.00785517 0.011446 2024-04-15T18:25:22.769 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:25:22.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:22.909 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:22.936 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:22.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:23.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:23.107 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:23.138 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:23.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:23.239 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:23.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:23.293 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:23.318 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:23.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:23.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:23.465 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:23.492 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:23.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:23.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:23.645 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:23.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 51246 51230 86.5295 58.1875 0.0157971 0.01155 2024-04-15T18:25:23.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:25:24.307 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:24.364 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:25:24.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:24 smithi082 ceph-mon[44206]: pgmap v1039: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s; 16840/147327 objects degraded (11.430%); 26 MiB/s, 408 objects/s recovering 2024-04-15T18:25:24.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:25:24.608 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:24.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:24.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:24 smithi049 ceph-mon[40087]: pgmap v1039: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s; 16840/147327 objects degraded (11.430%); 26 MiB/s, 408 objects/s recovering 2024-04-15T18:25:24.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:25:24.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:24 smithi159 ceph-mon[45336]: pgmap v1039: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 57 MiB/s wr, 916 op/s; 16840/147327 objects degraded (11.430%); 26 MiB/s, 408 objects/s recovering 2024-04-15T18:25:24.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:25:24.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:24.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 52123 52107 85.6947 54.8125 0.0381408 0.0116644 2024-04-15T18:25:25.239 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:25.239 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:25.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:25 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:25.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:25:25.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:25 smithi082 ceph-mon[44206]: osdmap e383: 12 total, 12 up, 11 in 2024-04-15T18:25:25.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:25 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:25:25.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:25 smithi159 ceph-mon[45336]: osdmap e383: 12 total, 12 up, 11 in 2024-04-15T18:25:25.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:25 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:25.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3500276562' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:25:25.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:25 smithi049 ceph-mon[40087]: osdmap e383: 12 total, 12 up, 11 in 2024-04-15T18:25:25.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 15 52902 52887 84.7473 48.75 0.00589929 0.0117928 2024-04-15T18:25:26.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:25:26.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:25:26.028 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-15T18:25:26.028 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-04-15T18:25:26.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 3 0 2024-04-15T18:25:26.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:25:26.491 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:26 smithi159 ceph-mon[45336]: pgmap v1041: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 858 op/s; 16441/149802 objects degraded (10.975%); 26 MiB/s, 412 objects/s recovering 2024-04-15T18:25:26.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:26 smithi049 ceph-mon[40087]: pgmap v1041: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 858 op/s; 16441/149802 objects degraded (10.975%); 26 MiB/s, 412 objects/s recovering 2024-04-15T18:25:26.759 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:26 smithi082 ceph-mon[44206]: pgmap v1041: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 858 op/s; 16441/149802 objects degraded (10.975%); 26 MiB/s, 412 objects/s recovering 2024-04-15T18:25:26.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:25:26.768867+0000 min lat: 0.00106705 max lat: 2.32562 avg lat: 0.0118732 2024-04-15T18:25:26.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:25:26.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 53891 53875 84.1723 61.75 0.0106709 0.0118732 2024-04-15T18:25:27.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 16441/149802 objects degraded (10.975%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:27.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:27 smithi159 ceph-mon[45336]: osdmap e384: 12 total, 12 up, 11 in 2024-04-15T18:25:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 16441/149802 objects degraded (10.975%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:27 smithi049 ceph-mon[40087]: osdmap e384: 12 total, 12 up, 11 in 2024-04-15T18:25:27.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 54819 54803 83.5338 58 0.00470806 0.0119624 2024-04-15T18:25:27.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 16441/149802 objects degraded (10.975%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:27.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:27 smithi082 ceph-mon[44206]: osdmap e384: 12 total, 12 up, 11 in 2024-04-15T18:25:28.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 55519 55503 82.5865 43.75 0.0109491 0.0120992 2024-04-15T18:25:29.119 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:29.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:29.143 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:29.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:29.258 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:29.290 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:29.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:29.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:29.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:29 smithi049 ceph-mon[40087]: pgmap v1043: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 15533/157863 objects degraded (9.840%); 28 MiB/s, 449 objects/s recovering 2024-04-15T18:25:29.436 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:29.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:29 smithi082 ceph-mon[44206]: pgmap v1043: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 15533/157863 objects degraded (9.840%); 28 MiB/s, 449 objects/s recovering 2024-04-15T18:25:29.636 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:29.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:29.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:29 smithi159 ceph-mon[45336]: pgmap v1043: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 15533/157863 objects degraded (9.840%); 28 MiB/s, 449 objects/s recovering 2024-04-15T18:25:29.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:29.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 56487 56471 82.0703 60.5 0.0309765 0.0121779 2024-04-15T18:25:29.777 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:29.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:29.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:29.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:29.939 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:30.043 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:30.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:30.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:30.175 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:30.201 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:30.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:30.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:30.359 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:30.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:25:30.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 57466 57450 81.5956 61.1875 0.00701163 0.0122495 2024-04-15T18:25:31.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:31 smithi082 ceph-mon[44206]: pgmap v1044: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s; 14924/161169 objects degraded (9.260%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:31.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:31 smithi159 ceph-mon[45336]: pgmap v1044: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s; 14924/161169 objects degraded (9.260%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:31.656 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:31.680 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:31.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:31.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:31 smithi049 ceph-mon[40087]: pgmap v1044: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 980 op/s; 14924/161169 objects degraded (9.260%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:31.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:31.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 58400 58384 81.0795 58.375 0.0280639 0.0123263 2024-04-15T18:25:31.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:31.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:25:31.859 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:31.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:31.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:31.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:31.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:31.993 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:32.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:32.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:32.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:32.253 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.3 primary-affinity to 0 (802) 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/348458124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1416283607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 14924/161169 objects degraded (9.260%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/348458124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1416283607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 14924/161169 objects degraded (9.260%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:32.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/348458124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1416283607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]: dispatch 2024-04-15T18:25:32.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 14924/161169 objects degraded (9.260%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:32.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 59224 59208 80.4364 51.5 0.0407883 0.0124243 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: pgmap v1045: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 14158/166065 objects degraded (8.526%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-04-15T18:25:33.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:33 smithi049 ceph-mon[40087]: osdmap e385: 12 total, 12 up, 11 in 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: pgmap v1045: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 14158/166065 objects degraded (8.526%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-04-15T18:25:33.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:33 smithi082 ceph-mon[44206]: osdmap e385: 12 total, 12 up, 11 in 2024-04-15T18:25:33.574 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:33.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: pgmap v1045: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 14158/166065 objects degraded (8.526%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.0}]': finished 2024-04-15T18:25:33.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:33 smithi159 ceph-mon[45336]: osdmap e385: 12 total, 12 up, 11 in 2024-04-15T18:25:33.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:33.715 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:33.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 59964 59948 79.709 46.25 0.0256699 0.0125334 2024-04-15T18:25:33.770 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:33.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:33.917 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:34.073 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:34.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:34.183 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:34.219 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:34.244 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:34.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:34.349 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:34.387 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:34.412 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:34.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:34.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:34.548 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:34.650 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:34.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:34.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:34.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 60822 60806 79.1655 53.625 0.0440235 0.0126249 2024-04-15T18:25:34.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:34.845 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:34.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:34.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:34.988 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:35.016 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:35.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:35.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:25:35.120 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:35.149 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:35.172 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:35.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:35.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:35.313 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:35.345 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:35.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:35.424 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:35.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:35.481 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:35.505 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:35.506 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:35 smithi159 ceph-mon[45336]: pgmap v1047: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 12775/174594 objects degraded (7.317%); 29 MiB/s, 458 objects/s recovering 2024-04-15T18:25:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:35.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:35.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:35 smithi082 ceph-mon[44206]: pgmap v1047: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 12775/174594 objects degraded (7.317%); 29 MiB/s, 458 objects/s recovering 2024-04-15T18:25:35.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:35.663 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:35.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:35 smithi049 ceph-mon[40087]: pgmap v1047: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 12775/174594 objects degraded (7.317%); 29 MiB/s, 458 objects/s recovering 2024-04-15T18:25:35.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 61639 61623 78.5919 51.0625 0.00302712 0.012718 2024-04-15T18:25:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:25:36.325 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:36.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:36.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:36.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:36.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3435413933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:36 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3435413933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:36 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.705 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:36.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:36.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3435413933' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:36 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:36.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 62670 62654 78.3087 64.4375 0.00557183 0.0127624 2024-04-15T18:25:36.770 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:36.770 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:25:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:25:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:25:36.771 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:36.811 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:36.855 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:37.011 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:37.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:37.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:37.153 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:37.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:37 smithi082 ceph-mon[44206]: pgmap v1048: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 12387/176973 objects degraded (6.999%); 27 MiB/s, 424 objects/s recovering 2024-04-15T18:25:37.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:37 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:37.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:37 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:37.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:37 smithi082 ceph-mon[44206]: osdmap e386: 12 total, 12 up, 11 in 2024-04-15T18:25:37.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12387/176973 objects degraded (6.999%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:37.306 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:37.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:37.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:37.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:37 smithi049 ceph-mon[40087]: pgmap v1048: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 12387/176973 objects degraded (6.999%); 27 MiB/s, 424 objects/s recovering 2024-04-15T18:25:37.433 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:37 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:37.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:37 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:37.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:37 smithi049 ceph-mon[40087]: osdmap e386: 12 total, 12 up, 11 in 2024-04-15T18:25:37.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12387/176973 objects degraded (6.999%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:37.448 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:37.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:25:37.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:37 smithi159 ceph-mon[45336]: pgmap v1048: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 59 MiB/s wr, 940 op/s; 12387/176973 objects degraded (6.999%); 27 MiB/s, 424 objects/s recovering 2024-04-15T18:25:37.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:37 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:25:37.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:37 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:37.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:37 smithi159 ceph-mon[45336]: osdmap e386: 12 total, 12 up, 11 in 2024-04-15T18:25:37.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12387/176973 objects degraded (6.999%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:37.700 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:37.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:37.752 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:37.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 63557 63541 77.8602 55.4375 0.0283101 0.012836 2024-04-15T18:25:37.781 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:37.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:37.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:37.838 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:37.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:37.863 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:37.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:37.875 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:37.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:25:37.931 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:37.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:37.967 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:37.983 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:37.996 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:38.011 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:38.012 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:38.037 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:38.041 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:38.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:38.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:38.068 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:38.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:38.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:38.177 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:38.202 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:38.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:38.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:38.419 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1966584305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:38.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:38.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:38.535 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1966584305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:38.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:38.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:38.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.576 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1966584305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:38.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 64465 64449 77.4542 56.75 0.0132654 0.0129055 2024-04-15T18:25:38.835 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:38.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:38.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:38.983 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:39.104 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:25:39.104 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:25:39.104 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-15T18:25:39.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:25:39.269 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: pgmap v1050: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 11368/184218 objects degraded (6.171%); 28 MiB/s, 444 objects/s recovering 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.270 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:39 smithi082 ceph-mon[44206]: osdmap e387: 12 total, 12 up, 11 in 2024-04-15T18:25:39.405 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:39.444 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:39.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: pgmap v1050: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 11368/184218 objects degraded (6.171%); 28 MiB/s, 444 objects/s recovering 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:39 smithi049 ceph-mon[40087]: osdmap e387: 12 total, 12 up, 11 in 2024-04-15T18:25:39.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:39.590 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: pgmap v1050: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 11368/184218 objects degraded (6.171%); 28 MiB/s, 444 objects/s recovering 2024-04-15T18:25:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:39.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:39 smithi159 ceph-mon[45336]: osdmap e387: 12 total, 12 up, 11 in 2024-04-15T18:25:39.727 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:39.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:39.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 65185 65169 76.8418 45 0.0219519 0.0130077 2024-04-15T18:25:39.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:39.887 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:39.922 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:39.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:40.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:40.073 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:40.099 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:40.099 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:40.099 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:40.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:40.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:40.236 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:40.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:25:40.272 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:40.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:40.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:40 smithi082 ceph-mon[44206]: pgmap v1052: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 10538/187494 objects degraded (5.620%); 29 MiB/s, 469 objects/s recovering 2024-04-15T18:25:40.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:40.425 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:40.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:40 smithi159 ceph-mon[45336]: pgmap v1052: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 10538/187494 objects degraded (5.620%); 29 MiB/s, 469 objects/s recovering 2024-04-15T18:25:40.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:40 smithi049 ceph-mon[40087]: pgmap v1052: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 10538/187494 objects degraded (5.620%); 29 MiB/s, 469 objects/s recovering 2024-04-15T18:25:40.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 66138 66122 76.5217 59.5625 0.0119729 0.0130614 2024-04-15T18:25:40.775 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:40.901 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:40.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:40.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:25:41.004 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:41.067 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:41.096 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:41.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:41.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:25:41.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:41.228 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:41.247 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:41.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:41.320 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:41.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:41.381 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:41.402 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:41.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:41.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:41.534 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:41.557 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:41.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:41.662 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:41.706 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:41.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 66877 66861 75.9701 46.1875 0.0206719 0.013156 2024-04-15T18:25:41.852 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:41.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:41.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:41.997 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:42.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:42.250 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:42.294 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 388, 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:25:42.232395+0000", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:25:13.233296+0000", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:21:55.200437+0000", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-04-15T18:25:42.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 29, 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 5, 2024-04-15T18:25:42.322 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "jewel", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:25:42.323 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:25:42.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:25:42.325 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:25:42.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:25:42.327 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.328 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 11.109999656677246, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 11.109999656677246, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.27000001072883606, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-04-15T18:25:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_3", 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:24:28.921306+0000", 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:25:42.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:25:42.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:25:42.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:25:42.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:25:42.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:25:42.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:25:42.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:25:42.333 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "366", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:25:42.334 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:25:42.335 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:25:42.336 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.3799999952316284, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.059999942779541, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.3799999952316284, 2024-04-15T18:25:42.337 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.059999942779541, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.338 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.343 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.344 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.345 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 192, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:25:42.350 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.352 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.355 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 284, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:25:42.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:25:42.360 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 376, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:25:42.367 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.370 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:25:42.372 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 358, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:25:42.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.375 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1896143039 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/1896143039", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/1896143039", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/1896143039", 2024-04-15T18:25:42.377 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/1896143039", 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.378 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 358, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:25:42.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.389 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:25:42.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:25:42.399 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:25:42.400 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 94, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 377, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.401 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.402 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:25:42.404 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/4245248647", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/4245248647", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/4245248647", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/4245248647", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:25:42.405 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:24:46.351924+0000", 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:25:42.406 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:24:59.569400+0000", 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.407 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:24:32.025606+0000", 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:13.390994+0000", 2024-04-15T18:25:42.408 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:24:45.944993+0000", 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:19.472129+0000", 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:21.956981+0000", 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:20.255130+0000", 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.411 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:37.808926+0000", 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.412 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:15.619415+0000", 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:14.444254+0000", 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.413 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:25:37.752663+0000", 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:25:42.414 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.3", 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.4", 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: 8 2024-04-15T18:25:42.415 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.6", 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: 10 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.7", 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-04-15T18:25:42.416 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.b", 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.d", 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: 11, 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-04-15T18:25:42.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.f", 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: 11 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:25:42.418 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:25:42.419 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:25:42.420 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:25:42.421 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:25:42.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:25:42.421 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:25:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:42.422 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:42.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:42.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:42.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:42 smithi082 ceph-mon[44206]: pgmap v1053: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 9868/192471 objects degraded (5.127%); 23 MiB/s, 363 objects/s recovering 2024-04-15T18:25:42.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 9868/192471 objects degraded (5.127%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:42.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:42 smithi082 ceph-mon[44206]: osdmap e388: 12 total, 12 up, 11 in 2024-04-15T18:25:42.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/580095840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:25:42.579 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:42.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:42 smithi159 ceph-mon[45336]: pgmap v1053: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 9868/192471 objects degraded (5.127%); 23 MiB/s, 363 objects/s recovering 2024-04-15T18:25:42.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 9868/192471 objects degraded (5.127%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:42.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:42 smithi159 ceph-mon[45336]: osdmap e388: 12 total, 12 up, 11 in 2024-04-15T18:25:42.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/580095840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:25:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:42 smithi049 ceph-mon[40087]: pgmap v1053: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 9868/192471 objects degraded (5.127%); 23 MiB/s, 363 objects/s recovering 2024-04-15T18:25:42.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 9868/192471 objects degraded (5.127%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:42.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:42 smithi049 ceph-mon[40087]: osdmap e388: 12 total, 12 up, 11 in 2024-04-15T18:25:42.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/580095840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:25:42.721 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:42.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:42.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 67863 67847 75.714 61.625 0.00295106 0.0132006 2024-04-15T18:25:42.827 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:42.860 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:42.884 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:42.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:42.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:43.020 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:43.223 INFO:tasks.thrashosds.thrasher:j is {'epoch': 388, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:25:42.232395+0000', 'last_up_change': '2024-04-15T18:25:13.233296+0000', 'last_in_change': '2024-04-15T18:21:55.200437+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-04-15T18:24:28.921306+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '366', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 192, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 284, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 376, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 358, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 1896143039}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 1896143039}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 1896143039}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 1896143039}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 1896143039}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 1896143039}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 1896143039}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 1896143039}]}, 'public_addr': '172.21.15.82:6817/1896143039', 'cluster_addr': '172.21.15.82:6819/1896143039', 'heartbeat_back_addr': '172.21.15.82:6823/1896143039', 'heartbeat_front_addr': '172.21.15.82:6821/1896143039', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 358, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 94, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 4245248647}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 4245248647}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 4245248647}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 4245248647}]}, 'public_addr': '172.21.15.159:6825/4245248647', 'cluster_addr': '172.21.15.159:6827/4245248647', 'heartbeat_back_addr': '172.21.15.159:6831/4245248647', 'heartbeat_front_addr': '172.21.15.159:6829/4245248647', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:24:46.351924+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:24:59.569400+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-15T18:24:32.025606+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:13.390994+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:24:45.944993+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:19.472129+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:21.956981+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:20.255130+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:37.808926+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:15.619415+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:14.444254+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:25:37.752663+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.3', 'osds': [0, 6]}, {'pgid': '5.4', 'osds': [10, 8]}, {'pgid': '5.6', 'osds': [7, 10]}, {'pgid': '5.7', 'osds': [1, 9]}, {'pgid': '5.b', 'osds': [9, 0]}, {'pgid': '5.d', 'osds': [11, 7]}, {'pgid': '5.f', 'osds': [4, 11]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:25:43.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:25:43.380 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:25:43.381 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:25:43.472 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:43.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:43.578 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:25:43.686 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:43.727 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:25:43.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:43.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:25:43.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 68646 68630 75.2441 48.9375 0.0214138 0.0132832 2024-04-15T18:25:43.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:43.899 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:43.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:43.931 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:43.951 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:44.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:44.041 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:44.172 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:44.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:44.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:44.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:25:44.358 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:25:44.396 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:44.430 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:44.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:44 smithi049 ceph-mon[40087]: pgmap v1055: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 62 MiB/s wr, 997 op/s; 8292/200397 objects degraded (4.138%); 33 MiB/s, 533 objects/s recovering 2024-04-15T18:25:44.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:44.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:44.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:44 smithi082 ceph-mon[44206]: pgmap v1055: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 62 MiB/s wr, 997 op/s; 8292/200397 objects degraded (4.138%); 33 MiB/s, 533 objects/s recovering 2024-04-15T18:25:44.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:44.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:44.581 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:44.613 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:44.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:44 smithi159 ceph-mon[45336]: pgmap v1055: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 62 MiB/s wr, 997 op/s; 8292/200397 objects degraded (4.138%); 33 MiB/s, 533 objects/s recovering 2024-04-15T18:25:44.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:25:44.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:44.753 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:44.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 15 69604 69589 74.9802 59.9375 0.0176209 0.0133311 2024-04-15T18:25:44.777 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:44.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:44.864 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:44.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:44.914 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:45.168 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:45.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:45.275 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:45.313 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:45.343 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:45.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:45 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:45.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:45.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:45 smithi049 ceph-mon[40087]: osdmap e389: 12 total, 12 up, 11 in 2024-04-15T18:25:45.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:45.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:45.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:45.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:45.483 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:45 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:45 smithi159 ceph-mon[45336]: osdmap e389: 12 total, 12 up, 11 in 2024-04-15T18:25:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:45.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:45.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 70597 70581 74.7601 62 0.00608994 0.0133673 2024-04-15T18:25:45.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:45 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:45.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/782117478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:25:45.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:45 smithi082 ceph-mon[44206]: osdmap e389: 12 total, 12 up, 11 in 2024-04-15T18:25:45.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:45.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:46.042 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:46.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:46.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:46.186 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:46 smithi159 ceph-mon[45336]: pgmap v1057: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 7819/202974 objects degraded (3.852%); 24 MiB/s, 387 objects/s recovering 2024-04-15T18:25:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:46 smithi159 ceph-mon[45336]: osdmap e390: 12 total, 12 up, 11 in 2024-04-15T18:25:46.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:25:46.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:46 smithi049 ceph-mon[40087]: pgmap v1057: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 7819/202974 objects degraded (3.852%); 24 MiB/s, 387 objects/s recovering 2024-04-15T18:25:46.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:46 smithi049 ceph-mon[40087]: osdmap e390: 12 total, 12 up, 11 in 2024-04-15T18:25:46.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:25:46.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:25:46.771670+0000 min lat: 0.00106705 max lat: 2.32562 avg lat: 0.0134341 2024-04-15T18:25:46.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:25:46.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 71446 71430 74.3984 53.0625 0.0176117 0.0134341 2024-04-15T18:25:46.903 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:46 smithi082 ceph-mon[44206]: pgmap v1057: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 7819/202974 objects degraded (3.852%); 24 MiB/s, 387 objects/s recovering 2024-04-15T18:25:46.903 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:46 smithi082 ceph-mon[44206]: osdmap e390: 12 total, 12 up, 11 in 2024-04-15T18:25:46.903 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:25:47.147 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:47.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:47.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:25:47.289 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:47.316 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:25:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:47.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:25:47.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:25:47.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 7819/202974 objects degraded (3.852%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:47.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:25:47.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:25:47.444 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:25:47.462 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:47.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 7819/202974 objects degraded (3.852%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:47.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:25:47.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:25:47.519 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:25:47.592 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:25:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:47.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:25:47.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 7819/202974 objects degraded (3.852%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:47.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:25:47.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:25:47.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:25:47.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:47.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 72506 72490 74.2647 66.25 0.0152355 0.0134592 2024-04-15T18:25:47.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:25:47.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:25:47.832 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:47.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:25:47.891 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:47.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:47.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:25:47.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:25:48.001 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:25:48.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:25:48.004 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:25:48.006 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:25:48.040 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:25:48.053 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:48.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:25:48.109 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:25:48.137 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:25:48.171 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:25:48.201 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:25:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:25:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:25:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:25:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:25:48.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:25:48.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:25:48.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: pgmap v1059: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 6755/210624 objects degraded (3.207%); 32 MiB/s, 518 objects/s recovering 2024-04-15T18:25:48.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:25:48.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:25:48.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2806954980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:48.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:25:48.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:48 smithi049 ceph-mon[40087]: osdmap e391: 12 total, 12 up, 11 in 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: pgmap v1059: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 6755/210624 objects degraded (3.207%); 32 MiB/s, 518 objects/s recovering 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2806954980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:48 smithi159 ceph-mon[45336]: osdmap e391: 12 total, 12 up, 11 in 2024-04-15T18:25:48.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 73560 73544 74.1293 65.875 0.0137364 0.0134841 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: pgmap v1059: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 6755/210624 objects degraded (3.207%); 32 MiB/s, 518 objects/s recovering 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2806954980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:48.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:48 smithi082 ceph-mon[44206]: osdmap e391: 12 total, 12 up, 11 in 2024-04-15T18:25:48.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:25:49.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 74569 74553 73.9536 63.0625 0.0181481 0.0135146 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:49.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:25:50.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:50 smithi159 ceph-mon[45336]: pgmap v1061: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 5895/214164 objects degraded (2.753%); 25 MiB/s, 399 objects/s recovering 2024-04-15T18:25:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:50 smithi049 ceph-mon[40087]: pgmap v1061: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 5895/214164 objects degraded (2.753%); 25 MiB/s, 399 objects/s recovering 2024-04-15T18:25:50.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 75382 75366 73.5919 50.8125 0.0559323 0.0135552 2024-04-15T18:25:50.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:50 smithi082 ceph-mon[44206]: pgmap v1061: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 764 op/s; 5895/214164 objects degraded (2.753%); 25 MiB/s, 399 objects/s recovering 2024-04-15T18:25:51.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:51 smithi159 ceph-mon[45336]: osdmap e392: 12 total, 12 up, 11 in 2024-04-15T18:25:51.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:51 smithi049 ceph-mon[40087]: osdmap e392: 12 total, 12 up, 11 in 2024-04-15T18:25:51.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 75547 75531 72.6184 10.3125 0.00551727 0.0137295 2024-04-15T18:25:51.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:51 smithi082 ceph-mon[44206]: osdmap e392: 12 total, 12 up, 11 in 2024-04-15T18:25:52.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:52 smithi049 ceph-mon[40087]: pgmap v1063: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 5050/219651 objects degraded (2.299%); 29 MiB/s, 461 objects/s recovering 2024-04-15T18:25:52.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:52 smithi049 ceph-mon[40087]: osdmap e393: 12 total, 12 up, 11 in 2024-04-15T18:25:52.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5050/219651 objects degraded (2.299%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:52.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:52 smithi049 ceph-mon[40087]: osdmap e394: 12 total, 12 up, 11 in 2024-04-15T18:25:52.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 75711 75695 71.6733 10.25 0.00288892 0.0138596 2024-04-15T18:25:52.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:52 smithi082 ceph-mon[44206]: pgmap v1063: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 5050/219651 objects degraded (2.299%); 29 MiB/s, 461 objects/s recovering 2024-04-15T18:25:52.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:52 smithi082 ceph-mon[44206]: osdmap e393: 12 total, 12 up, 11 in 2024-04-15T18:25:52.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5050/219651 objects degraded (2.299%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:52.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:52 smithi082 ceph-mon[44206]: osdmap e394: 12 total, 12 up, 11 in 2024-04-15T18:25:52.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:52 smithi159 ceph-mon[45336]: pgmap v1063: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 10 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 5050/219651 objects degraded (2.299%); 29 MiB/s, 461 objects/s recovering 2024-04-15T18:25:52.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:52 smithi159 ceph-mon[45336]: osdmap e393: 12 total, 12 up, 11 in 2024-04-15T18:25:52.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5050/219651 objects degraded (2.299%), 7 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:52.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:52 smithi159 ceph-mon[45336]: osdmap e394: 12 total, 12 up, 11 in 2024-04-15T18:25:53.443 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:25:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:53.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:25:53.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:53.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 76367 76351 71.2154 41 0.0731347 0.0140362 2024-04-15T18:25:53.849 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:54.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:25:54.202 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.224 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:25:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:25:54.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:54.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.377 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:54.382 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:25:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:25:54.382 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:25:54.382 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:54.482 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:54.506 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:54 smithi159 ceph-mon[45336]: osdmap e395: 12 total, 12 up, 11 in 2024-04-15T18:25:54.508 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.526 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:54.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:54.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:54 smithi082 ceph-mon[44206]: osdmap e395: 12 total, 12 up, 11 in 2024-04-15T18:25:54.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:54.657 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.676 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:54.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:54 smithi049 ceph-mon[40087]: osdmap e395: 12 total, 12 up, 11 in 2024-04-15T18:25:54.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 76911 76895 70.668 34 0.0106883 0.0140857 2024-04-15T18:25:54.773 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1059,"stamp":"2024-04-15T18:25:47.290805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4601610784,"num_objects":70208,"num_object_clones":0,"num_object_copies":210624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70208,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":70387,"num_write_kb":4496122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12617,"num_bytes_recovered":828856992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70343,"ondisk_log_size":70343,"up":51,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16160132,"kb_used_data":13024400,"kb_used_omap":62,"kb_used_meta":3135553,"kb_avail":1015036540,"statfs":{"total":1055945392128,"available":1039397416960,"internally_reserved":0,"allocated":13336985600,"data_stored":13332471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64301,"internal_metadata":3210806483},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":18153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6051,"num_write_kb":387264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3113,"num_bytes_recovered":204013568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001528"},"pg_stats":[{"pgid":"5.f","version":"388'4315","reported_seq":8208,"reported_epoch":388,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:42.638105+0000","last_change":"2024-04-15T18:25:15.550575+0000","last_active":"2024-04-15T18:25:42.638105+0000","last_peered":"2024-04-15T18:25:42.638105+0000","last_clean":"2024-04-15T18:24:29.224778+0000","last_became_active":"2024-04-15T18:25:15.532307+0000","last_became_peered":"2024-04-15T18:25:15.532307+0000","last_unstale":"2024-04-15T18:25:42.638105+0000","last_undegraded":"2024-04-15T18:25:15.298206+0000","last_fullsized":"2024-04-15T18:25:15.244252+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":12945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":276160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1916,"num_bytes_recovered":125566976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[4,11],"avail_no_missing":["4","11"],"object_location_counts":[{"shards":"4,11","objects":3420}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"390'4524","reported_seq":4596,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-04-15T18:25:47.229134+0000","last_change":"2024-04-15T18:24:30.244877+0000","last_active":"2024-04-15T18:25:47.229134+0000","last_peered":"2024-04-15T18:25:47.229134+0000","last_clean":"2024-04-15T18:25:47.229134+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:25:47.229134+0000","last_undegraded":"2024-04-15T18:25:47.229134+0000","last_fullsized":"2024-04-15T18:25:47.229134+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:25:57.173190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296484864,"num_objects":4524,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4524,"num_write_kb":289536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"389'4314","reported_seq":9020,"reported_epoch":389,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:44.998565+0000","last_change":"2024-04-15T18:25:15.547428+0000","last_active":"2024-04-15T18:25:44.998565+0000","last_peered":"2024-04-15T18:25:44.998565+0000","last_clean":"2024-04-15T18:24:36.709314+0000","last_became_active":"2024-04-15T18:25:15.531256+0000","last_became_peered":"2024-04-15T18:25:15.531256+0000","last_unstale":"2024-04-15T18:25:44.998565+0000","last_undegraded":"2024-04-15T18:25:15.259774+0000","last_fullsized":"2024-04-15T18:25:15.245918+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282722304,"num_objects":4314,"num_object_clones":0,"num_object_copies":12942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":276096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2314,"num_bytes_recovered":151650304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[{"shards":"7,11","objects":3882}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"390'4531","reported_seq":4605,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-04-15T18:25:47.262721+0000","last_change":"2024-04-15T18:24:30.244856+0000","last_active":"2024-04-15T18:25:47.262721+0000","last_peered":"2024-04-15T18:25:47.262721+0000","last_clean":"2024-04-15T18:25:47.262721+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:25:47.262721+0000","last_undegraded":"2024-04-15T18:25:47.262721+0000","last_fullsized":"2024-04-15T18:25:47.262721+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:25:54.622360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296943616,"num_objects":4531,"num_object_clones":0,"num_object_copies":13593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":289984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"390'4548","reported_seq":4627,"reported_epoch":390,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:47.273241+0000","last_change":"2024-04-15T18:25:15.544901+0000","last_active":"2024-04-15T18:25:47.273241+0000","last_peered":"2024-04-15T18:25:47.273241+0000","last_clean":"2024-04-15T18:24:36.709012+0000","last_became_active":"2024-04-15T18:25:15.538224+0000","last_became_peered":"2024-04-15T18:25:15.538224+0000","last_unstale":"2024-04-15T18:25:47.273241+0000","last_undegraded":"2024-04-15T18:25:15.258532+0000","last_fullsized":"2024-04-15T18:25:15.243707+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[{"shards":"0,9","objects":1799}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"389'4328","reported_seq":4400,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-04-15T18:25:45.266966+0000","last_change":"2024-04-15T18:24:30.247430+0000","last_active":"2024-04-15T18:25:45.266966+0000","last_peered":"2024-04-15T18:25:45.266966+0000","last_clean":"2024-04-15T18:25:45.266966+0000","last_became_active":"2024-04-15T18:24:30.247051+0000","last_became_peered":"2024-04-15T18:24:30.247051+0000","last_unstale":"2024-04-15T18:25:45.266966+0000","last_undegraded":"2024-04-15T18:25:45.266966+0000","last_fullsized":"2024-04-15T18:25:45.266966+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:25:49.060129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":276992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,4],"acting":[8,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.9","version":"390'4454","reported_seq":4528,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-04-15T18:25:45.932202+0000","last_change":"2024-04-15T18:24:30.248024+0000","last_active":"2024-04-15T18:25:45.932202+0000","last_peered":"2024-04-15T18:25:45.932202+0000","last_clean":"2024-04-15T18:25:45.932202+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:25:45.932202+0000","last_undegraded":"2024-04-15T18:25:45.932202+0000","last_fullsized":"2024-04-15T18:25:45.932202+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291897344,"num_objects":4454,"num_object_clones":0,"num_object_copies":13362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4454,"num_write_kb":285056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"389'4378","reported_seq":4448,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-04-15T18:25:44.969317+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:25:44.969317+0000","last_peered":"2024-04-15T18:25:44.969317+0000","last_clean":"2024-04-15T18:25:44.969317+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:25:44.969317+0000","last_undegraded":"2024-04-15T18:25:44.969317+0000","last_fullsized":"2024-04-15T18:25:44.969317+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286916608,"num_objects":4378,"num_object_clones":0,"num_object_copies":13134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4378,"num_write_kb":280192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"390'4553","reported_seq":8788,"reported_epoch":390,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:47.264081+0000","last_change":"2024-04-15T18:25:15.548974+0000","last_active":"2024-04-15T18:25:47.264081+0000","last_peered":"2024-04-15T18:25:47.264081+0000","last_clean":"2024-04-15T18:24:29.224778+0000","last_became_active":"2024-04-15T18:25:15.537411+0000","last_became_peered":"2024-04-15T18:25:15.537411+0000","last_unstale":"2024-04-15T18:25:47.264081+0000","last_undegraded":"2024-04-15T18:25:15.256403+0000","last_fullsized":"2024-04-15T18:25:15.244223+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298385408,"num_objects":4553,"num_object_clones":0,"num_object_copies":13659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4553,"num_write_kb":291392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":136708096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[1,9],"avail_no_missing":["1","9"],"object_location_counts":[{"shards":"1,9","objects":3832}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"390'4330","reported_seq":8500,"reported_epoch":390,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:45.949097+0000","last_change":"2024-04-15T18:25:15.580738+0000","last_active":"2024-04-15T18:25:45.949097+0000","last_peered":"2024-04-15T18:25:45.949097+0000","last_clean":"2024-04-15T18:24:29.224778+0000","last_became_active":"2024-04-15T18:25:15.532723+0000","last_became_peered":"2024-04-15T18:25:15.532723+0000","last_unstale":"2024-04-15T18:25:45.949097+0000","last_undegraded":"2024-04-15T18:25:15.257816+0000","last_fullsized":"2024-04-15T18:25:15.244186+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4330,"log_dups_size":0,"ondisk_log_size":4330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283770880,"num_objects":4330,"num_object_clones":0,"num_object_copies":12990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":277120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2054,"num_bytes_recovered":134610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[7,10],"avail_no_missing":["7","10"],"object_location_counts":[{"shards":"7,10","objects":3673}],"blocked_by":[],"up_primary":5,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"390'4327","reported_seq":4401,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-04-15T18:25:46.346870+0000","last_change":"2024-04-15T18:24:30.247507+0000","last_active":"2024-04-15T18:25:46.346870+0000","last_peered":"2024-04-15T18:25:46.346870+0000","last_clean":"2024-04-15T18:25:46.346870+0000","last_became_active":"2024-04-15T18:24:30.247171+0000","last_became_peered":"2024-04-15T18:24:30.247171+0000","last_unstale":"2024-04-15T18:25:46.346870+0000","last_undegraded":"2024-04-15T18:25:46.346870+0000","last_fullsized":"2024-04-15T18:25:46.346870+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:25:49.162952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283574272,"num_objects":4327,"num_object_clones":0,"num_object_copies":12981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":276928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"390'4563","reported_seq":9006,"reported_epoch":390,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:46.339311+0000","last_change":"2024-04-15T18:25:15.550537+0000","last_active":"2024-04-15T18:25:46.339311+0000","last_peered":"2024-04-15T18:25:46.339311+0000","last_clean":"2024-04-15T18:24:36.536415+0000","last_became_active":"2024-04-15T18:25:15.531391+0000","last_became_peered":"2024-04-15T18:25:15.531391+0000","last_unstale":"2024-04-15T18:25:46.339311+0000","last_undegraded":"2024-04-15T18:25:15.253142+0000","last_fullsized":"2024-04-15T18:25:15.241587+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299040768,"num_objects":4563,"num_object_clones":0,"num_object_copies":13689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":619,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":292032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2182,"num_bytes_recovered":142999552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":3944}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"389'4354","reported_seq":4426,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-04-15T18:25:45.339946+0000","last_change":"2024-04-15T18:24:30.245935+0000","last_active":"2024-04-15T18:25:45.339946+0000","last_peered":"2024-04-15T18:25:45.339946+0000","last_clean":"2024-04-15T18:25:45.339946+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:25:45.339946+0000","last_undegraded":"2024-04-15T18:25:45.339946+0000","last_fullsized":"2024-04-15T18:25:45.339946+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:25:55.825018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285343744,"num_objects":4354,"num_object_clones":0,"num_object_copies":13062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4354,"num_write_kb":278656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"388'4131","reported_seq":4200,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-15T18:25:42.564867+0000","last_change":"2024-04-15T18:24:30.314295+0000","last_active":"2024-04-15T18:25:42.564867+0000","last_peered":"2024-04-15T18:25:42.564867+0000","last_clean":"2024-04-15T18:25:42.564867+0000","last_became_active":"2024-04-15T18:24:30.313912+0000","last_became_peered":"2024-04-15T18:24:30.313912+0000","last_unstale":"2024-04-15T18:25:42.564867+0000","last_undegraded":"2024-04-15T18:25:42.564867+0000","last_fullsized":"2024-04-15T18:25:42.564867+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270729216,"num_objects":4131,"num_object_clones":0,"num_object_copies":12393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":264384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"388'4321","reported_seq":4389,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-15T18:25:42.978001+0000","last_change":"2024-04-15T18:24:30.247246+0000","last_active":"2024-04-15T18:25:42.978001+0000","last_peered":"2024-04-15T18:25:42.978001+0000","last_clean":"2024-04-15T18:25:42.978001+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:25:42.978001+0000","last_undegraded":"2024-04-15T18:25:42.978001+0000","last_fullsized":"2024-04-15T18:25:42.978001+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283181056,"num_objects":4321,"num_object_clones":0,"num_object_copies":12963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":276544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"388'4235","reported_seq":8420,"reported_epoch":388,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:25:42.986380+0000","last_change":"2024-04-15T18:25:15.530188+0000","last_active":"2024-04-15T18:25:42.986380+0000","last_peered":"2024-04-15T18:25:42.986380+0000","last_clean":"2024-04-15T18:24:36.536929+0000","last_became_active":"2024-04-15T18:25:15.522658+0000","last_became_peered":"2024-04-15T18:25:15.522658+0000","last_unstale":"2024-04-15T18:25:42.986380+0000","last_undegraded":"2024-04-15T18:25:15.259694+0000","last_fullsized":"2024-04-15T18:25:15.244751+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277544960,"num_objects":4235,"num_object_clones":0,"num_object_copies":12705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4235,"num_write_kb":271040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2055,"num_bytes_recovered":134676480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,8],"avail_no_missing":["10","8"],"object_location_counts":[{"shards":"8,10","objects":3553}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1009,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-04-15T18:25:45.375794+0000","last_change":"2024-04-15T18:25:03.390730+0000","last_active":"2024-04-15T18:25:45.375794+0000","last_peered":"2024-04-15T18:25:45.375794+0000","last_clean":"2024-04-15T18:25:45.375794+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:25:45.375794+0000","last_undegraded":"2024-04-15T18:25:45.375794+0000","last_fullsized":"2024-04-15T18:25:45.375794+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:25:03.390582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:25:03.390582+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:26:26.290417+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4601020416,"num_objects":70206,"num_object_clones":0,"num_object_copies":210618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70206,"num_write_kb":4493184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12607,"num_bytes_recovered":826212352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13447135232,"data_stored":13447135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70206,"ondisk_log_size":70206,"up":48,"acting":41,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926024,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1481772,"kb_used_data":1107684,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92263380,"statfs":{"total":95995035648,"available":94477701120,"internally_reserved":0,"allocated":1134268416,"data_stored":1133859748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":383056324},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187659,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1886308,"kb_used_data":1620508,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91858844,"statfs":{"total":95995035648,"available":94063456256,"internally_reserved":0,"allocated":1659400192,"data_stored":1658995096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165884,"kb_used_data":863988,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92579268,"statfs":{"total":95995035648,"available":94801170432,"internally_reserved":0,"allocated":884723712,"data_stored":884309976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":309131712},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874453,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978868,"kb_used_data":835240,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92766284,"statfs":{"total":95995035648,"available":94992674816,"internally_reserved":0,"allocated":855285760,"data_stored":854876319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168794,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1408716,"kb_used_data":1110972,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92336436,"statfs":{"total":95995035648,"available":94552510464,"internally_reserved":0,"allocated":1137635328,"data_stored":1137218040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":304871882},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463135,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1147068,"kb_used_data":845168,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92598084,"statfs":{"total":95995035648,"available":94820438016,"internally_reserved":0,"allocated":865452032,"data_stored":865039039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309131718},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575101,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1500608,"kb_used_data":1148724,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92244544,"statfs":{"total":95995035648,"available":94458413056,"internally_reserved":0,"allocated":1176293376,"data_stored":1175879640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738622,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2307836,"kb_used_data":1992052,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91437316,"statfs":{"total":95995035648,"available":93631811584,"internally_reserved":0,"allocated":2039861248,"data_stored":2039447512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":1550483193873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598552,"kb_used_data":262736,"kb_used_omap":44,"kb_used_meta":335763,"kb_avail":93146600,"statfs":{"total":95995035648,"available":95382118400,"internally_reserved":0,"allocated":269041664,"data_stored":268640913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":343821866},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986408,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967464,"kb_used_data":1621148,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91777688,"statfs":{"total":95995035648,"available":93980352512,"internally_reserved":0,"allocated":1660055552,"data_stored":1659650456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717056,"kb_used_data":1616180,"kb_used_omap":4,"kb_used_meta":100859,"kb_avail":92028096,"statfs":{"total":95995035648,"available":94236770304,"internally_reserved":0,"allocated":1654968320,"data_stored":1654554584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":103280321},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2046427136,"data_stored":2046427136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1182334976,"data_stored":1182334976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270729216,"data_stored":270729216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1659830272,"data_stored":1659830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1706229760,"data_stored":1706229760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":869269504,"data_stored":869269504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1145044992,"data_stored":1145044992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":859439104,"data_stored":859439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":893321216,"data_stored":893321216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1666056192,"data_stored":1666056192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1148452864,"data_stored":1148452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:25:54.774 INFO:tasks.thrashosds.thrasher:Setting 5.1 to [0, 11, 6] 2024-04-15T18:25:54.774 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1', '0', '11', '6'] 2024-04-15T18:25:54.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pg-upmap 5.1 0 11 6 2024-04-15T18:25:54.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:54.844 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:54.924 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:54.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:54.959 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:55.029 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:25:55.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:55.059 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:55.202 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:55.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:55.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:55 smithi082 ceph-mon[44206]: pgmap v1067: 17 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 3007/226293 objects degraded (1.329%); 45 MiB/s, 721 objects/s recovering 2024-04-15T18:25:55.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:55 smithi082 ceph-mon[44206]: osdmap e396: 12 total, 12 up, 11 in 2024-04-15T18:25:55.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:55.339 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:55.367 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:55.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:55 smithi049 ceph-mon[40087]: pgmap v1067: 17 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 3007/226293 objects degraded (1.329%); 45 MiB/s, 721 objects/s recovering 2024-04-15T18:25:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:55 smithi049 ceph-mon[40087]: osdmap e396: 12 total, 12 up, 11 in 2024-04-15T18:25:55.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:25:55.497 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:25:55.504 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:55 smithi159 ceph-mon[45336]: pgmap v1067: 17 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 3007/226293 objects degraded (1.329%); 45 MiB/s, 721 objects/s recovering 2024-04-15T18:25:55.504 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:55 smithi159 ceph-mon[45336]: osdmap e396: 12 total, 12 up, 11 in 2024-04-15T18:25:55.506 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:55.532 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:25:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:55.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:25:55.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:55.676 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:55.754 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:55.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:55.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 77489 77473 70.1673 36.125 0.0142128 0.0142459 2024-04-15T18:25:55.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:55.893 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:56.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:56 smithi082 ceph-mon[44206]: osdmap e397: 12 total, 12 up, 11 in 2024-04-15T18:25:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:56.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:56 smithi049 ceph-mon[40087]: osdmap e397: 12 total, 12 up, 11 in 2024-04-15T18:25:56.547 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:56.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:56.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:25:56.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:56 smithi159 ceph-mon[45336]: osdmap e397: 12 total, 12 up, 11 in 2024-04-15T18:25:56.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 78492 78476 70.0604 62.6875 0.00978381 0.0142678 2024-04-15T18:25:56.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:25:57.108 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:57.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:57 smithi082 ceph-mon[44206]: pgmap v1070: 17 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 33 MiB/s wr, 531 op/s; 2998/227415 objects degraded (1.318%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:57.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2998/227415 objects degraded (1.318%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:57.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:57 smithi159 ceph-mon[45336]: pgmap v1070: 17 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 33 MiB/s wr, 531 op/s; 2998/227415 objects degraded (1.318%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:57.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2998/227415 objects degraded (1.318%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:57.656 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:25:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:57.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:25:57.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:57 smithi049 ceph-mon[40087]: pgmap v1070: 17 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 33 MiB/s wr, 531 op/s; 2998/227415 objects degraded (1.318%); 26 MiB/s, 414 objects/s recovering 2024-04-15T18:25:57.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2998/227415 objects degraded (1.318%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:25:57.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:25:57.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 79018 79002 69.5366 32.875 0.0174574 0.0143745 2024-04-15T18:25:57.805 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:57.841 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:25:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:57.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:25:57.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:25:57.965 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:25:57.987 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:58.244 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:25:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:58.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:25:58.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:58.270 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:25:58.309 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:25:58.309 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:25:58.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:25:58.309 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121149 3.d 3:bf94904e:::benchmark_data_smithi159_91621_object14673:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 496.19052767300002, 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25102950499999999, 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121149 2024-04-15T18:25:58.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118536+0000", 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118537+0000", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2309999999999999e-06 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118542+0000", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9400000000000001e-06 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118552+0000", 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.044e-05 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118561+0000", 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7979999999999995e-06 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118604+0000", 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3111000000000001e-05 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118765+0000", 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016077999999999999 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.118879+0000", 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011379600000000001 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.122472+0000", 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035933610000000002 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.131902+0000", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0094298760000000002 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369472+0000", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23756957000000001 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369526+0000", 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3842999999999999e-05 2024-04-15T18:25:58.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369565+0000", 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9759000000000003e-05 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121171 3.d 3:b5145494:::benchmark_data_smithi159_91621_object14695:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 496.14058779800001, 2024-04-15T18:25:58.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.201212849, 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121171 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168476+0000", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168477+0000", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6840000000000001e-06 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168484+0000", 2024-04-15T18:25:58.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6630000000000004e-06 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168501+0000", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6441000000000001e-05 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168509+0000", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6619999999999994e-06 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168562+0000", 2024-04-15T18:25:58.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3244e-05 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168700+0000", 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000137231 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.168833+0000", 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013358200000000001 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.171844+0000", 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030106540000000002 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.188649+0000", 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.016804927000000001 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369603+0000", 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18095407099999999 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369650+0000", 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7243000000000002e-05 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.369689+0000", 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8447e-05 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:121461 3.f 3:f41f1665:::benchmark_data_smithi159_91621_object14985:head [delete] snapc 0=[] ondisk+write+known_if_redirected e239)", 2024-04-15T18:25:58.322 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 495.42221410000002, 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19883951899999999, 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121461 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886849+0000", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886851+0000", 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2580000000000001e-06 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886857+0000", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9100000000000002e-06 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886870+0000", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2928999999999999e-05 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886878+0000", 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9629999999999998e-06 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.886929+0000", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.126e-05 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.887062+0000", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013309800000000001 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 8,11", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.887193+0000", 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000131502 2024-04-15T18:25:58.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.890852+0000", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036585580000000001 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:42.890892+0000", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9793999999999999e-05 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085577+0000", 2024-04-15T18:25:58.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19468490299999999 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085637+0000", 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0449999999999999e-05 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:17:43.085689+0000", 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1894000000000002e-05 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:144275 3.d 3:b0290c5b:::benchmark_data_smithi159_91621_object37799:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 456.14389849200001, 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203547589, 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 144275 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.329 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165165+0000", 2024-04-15T18:25:58.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165166+0000", 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2e-07 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165169+0000", 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.27e-06 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165176+0000", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165182+0000", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6919999999999999e-06 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165235+0000", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2642000000000002e-05 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165361+0000", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012657799999999999 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.165472+0000", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111254 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.169817+0000", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043446539999999999 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.170617+0000", 2024-04-15T18:25:58.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00079937499999999996 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368612+0000", 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.197995223 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368666+0000", 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4616e-05 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.368713+0000", 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6276999999999998e-05 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:144542 3.d 3:ba20a02c:::benchmark_data_smithi159_91621_object38066:head [delete] snapc 0=[] ondisk+write+known_if_redirected e250)", 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 455.68587341300002, 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.205631183, 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.335 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 144542 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623190+0000", 2024-04-15T18:25:58.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623191+0000", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4399999999999999e-07 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623195+0000", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1659999999999996e-06 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623206+0000", 2024-04-15T18:25:58.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1415e-05 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.623213+0000", 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8349999999999998e-06 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626803+0000", 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035900490000000001 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626920+0000", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116906 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.626997+0000", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7090999999999998e-05 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.630647+0000", 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036496509999999999 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.630721+0000", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4012999999999998e-05 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828707+0000", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19798600699999999 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828776+0000", 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8727999999999997e-05 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:18:22.828821+0000", 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5578000000000003e-05 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:185262 3.d 3:b4a17326:::benchmark_data_smithi159_91621_object78786:head [delete] snapc 0=[] ondisk+write+known_if_redirected e267)", 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 386.01118317700002, 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23775624300000001, 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.341 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185262 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297880+0000", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297882+0000", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.733e-06 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297889+0000", 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3819999999999998e-06 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297906+0000", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6565000000000001e-05 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297915+0000", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1519999999999998e-06 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.297975+0000", 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0296e-05 2024-04-15T18:25:58.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.298116+0000", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140271 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.298242+0000", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000126458 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.303601+0000", 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0053590319999999997 2024-04-15T18:25:58.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.310158+0000", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006557149 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535525+0000", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22536619899999999 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535586+0000", 2024-04-15T18:25:58.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1457999999999993e-05 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:19:32.535637+0000", 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0547999999999998e-05 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:212946 3.d 3:b1715a3b:::benchmark_data_smithi159_91621_object106470:head [delete] snapc 0=[] ondisk+write+known_if_redirected e275)", 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 343.306020375, 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29758224900000002, 2024-04-15T18:25:58.347 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 212946 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003044+0000", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.19e-06 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003050+0000", 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7379999999999999e-06 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003063+0000", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2772000000000001e-05 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003072+0000", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6980000000000004e-06 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003128+0000", 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6187000000000003e-05 2024-04-15T18:25:58.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003250+0000", 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122142 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003346+0000", 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5867999999999998e-05 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008011+0000", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046649550000000001 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008401+0000", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039006300000000001 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300548+0000", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29214726699999999 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300587+0000", 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9243999999999999e-05 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300625+0000", 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8124999999999998e-05 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:15519 4.3 e292/284 4:cd27a75b:::benchmark_data_smithi159_102124_object15518:head v 292'960, mlcod=292'958)", 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:25:58.353 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 307.88503937199999, 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.212814323, 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424024+0000", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424025+0000", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6899999999999997e-07 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.424047+0000", 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2731e-05 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636060+0000", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21201231600000001 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636066+0000", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8560000000000003e-06 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636119+0000", 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3171999999999998e-05 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636141+0000", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1848000000000002e-05 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636805+0000", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066410000000000004 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:50.636838+0000", 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3730999999999999e-05 2024-04-15T18:25:58.357 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:89299 4.0 e322/284 4:078bede3:::benchmark_data_smithi159_102124_object89298:head v 322'5533, mlcod=322'5530)", 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.944717481, 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53392096499999997, 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364347+0000", 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.004e-06 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364416+0000", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8975999999999998e-05 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895471+0000", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53105475999999996 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895477+0000", 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1619999999999996e-06 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895526+0000", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9008999999999998e-05 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895549+0000", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3104000000000001e-05 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898235+0000", 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026861400000000001 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898267+0000", 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1810000000000002e-05 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(4.6 322/318 [PushOp(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head, version: 292'910, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head@292'910, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.59588453500001, 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47774007400000001, 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.362 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713180+0000", 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713250+0000", 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0133000000000001e-05 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189486+0000", 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47623641300000003 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189505+0000", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8240000000000002e-05 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189590+0000", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5097000000000002e-05 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189600+0000", 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0101e-05 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.190919+0000", 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001319204 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:134866 4.8 e335/284 4:1fd83f8c:::benchmark_data_smithi159_102124_object45512:head v 335'8348, mlcod=335'8346)", 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 172.43018900499999, 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21941387600000001, 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.366 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878875+0000", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7000000000000004e-07 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.878880+0000", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0379999999999999e-06 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.094952+0000", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216071508 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.094958+0000", 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7760000000000003e-06 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.095008+0000", 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0250000000000002e-05 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.095035+0000", 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7151000000000001e-05 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.098260+0000", 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032249570000000001 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.098288+0000", 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8526000000000001e-05 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.370 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43697 5.6 5:62c05016:::benchmark_data_smithi159_116570_object43696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.050488647999998, 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27956367700000001, 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43697 2024-04-15T18:25:58.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258577+0000", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.866e-06 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258720+0000", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014362700000000001 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258737+0000", 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6290000000000002e-05 2024-04-15T18:25:58.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258749+0000", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2758999999999999e-05 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259906+0000", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011563929999999999 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259918+0000", 2024-04-15T18:25:58.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2048000000000001e-05 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.538024+0000", 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27810570899999998 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.538139+0000", 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000114985 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43706 5.6 5:6ce21a97:::benchmark_data_smithi159_116570_object43705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.050295726999998, 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.281325461, 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43706 2024-04-15T18:25:58.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.377 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.13e-07 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258809+0000", 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0833e-05 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258818+0000", 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7460000000000006e-06 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.378 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258825+0000", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0489999999999998e-06 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259943+0000", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011180140000000001 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259950+0000", 2024-04-15T18:25:58.379 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6649999999999997e-06 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.539990+0000", 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28003989200000001 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540093+0000", 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103749 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.380 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43717 5.6 5:6ddc9ba4:::benchmark_data_smithi159_116570_object43716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.050228986999997, 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29288951800000002, 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43717 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.381 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:25:58.382 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6899999999999998e-07 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258875+0000", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9944000000000002e-05 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258887+0000", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1799e-05 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258894+0000", 2024-04-15T18:25:58.383 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8159999999999996e-06 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259972+0000", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001078213 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259978+0000", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3330000000000002e-06 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540122+0000", 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28014370599999999 2024-04-15T18:25:58.384 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540281+0000", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000158703 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540483+0000", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020204700000000001 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545583+0000", 2024-04-15T18:25:58.385 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051007320000000002 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545631+0000", 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7271000000000002e-05 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549706+0000", 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040756949999999998 2024-04-15T18:25:58.386 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549764+0000", 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8183999999999997e-05 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551724+0000", 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001959606 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.387 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43723 5.6 5:6cb1f38e:::benchmark_data_smithi159_116570_object43722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.050142594, 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29697446900000002, 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43723 2024-04-15T18:25:58.388 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2600000000000002e-07 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258993+0000", 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1836999999999997e-05 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259010+0000", 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6433e-05 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.390 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259017+0000", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.238e-06 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259999+0000", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098238199999999996 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260007+0000", 2024-04-15T18:25:58.391 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1509999999999998e-06 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542068+0000", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28206113700000002 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542207+0000", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000139204 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542355+0000", 2024-04-15T18:25:58.392 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014846099999999999 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549548+0000", 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071922150000000001 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551744+0000", 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021962489999999999 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.393 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.553782+0000", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002037905 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.553869+0000", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7721999999999998e-05 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.555895+0000", 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020260090000000001 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.394 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43730 5.6 5:6b90e375:::benchmark_data_smithi159_116570_object43729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.050026287000001, 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29838256400000002, 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.395 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43730 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.396 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259038+0000", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259073+0000", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4836999999999999e-05 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259081+0000", 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5979999999999997e-06 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.397 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259088+0000", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6329999999999999e-06 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260032+0000", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00094387699999999998 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260039+0000", 2024-04-15T18:25:58.398 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4540000000000001e-06 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.543916+0000", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28387716299999999 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.544033+0000", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116709 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.544155+0000", 2024-04-15T18:25:58.399 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000121719 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549618+0000", 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0054628669999999997 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551754+0000", 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021362159999999998 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.400 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.555955+0000", 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004200713 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.556023+0000", 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8528000000000005e-05 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.401 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.557420+0000", 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001396771 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43734 5.6 5:69603c05:::benchmark_data_smithi159_116570_object43733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 43.049967138, 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30329634900000002, 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.402 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43734 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.403 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259097+0000", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2099999999999997e-07 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259124+0000", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7418000000000001e-05 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259132+0000", 2024-04-15T18:25:58.404 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.182e-06 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259138+0000", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2979999999999997e-06 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260130+0000", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0009926379999999999 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260141+0000", 2024-04-15T18:25:58.405 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0523000000000001e-05 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545663+0000", 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28552172300000001 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545815+0000", 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000152566 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:25:58.406 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545995+0000", 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000179326 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549688+0000", 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036929290000000002 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551762+0000", 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002074762 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.407 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.560176+0000", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084137169999999994 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.560236+0000", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9589000000000002e-05 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.562393+0000", 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021571569999999998 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.408 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:78847 5.9 5:97fe1fc4:::benchmark_data_smithi159_116570_object78846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e397)", 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:57.312358+0000", 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.99670603800000002, 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.179930434, 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.409 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 78847 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312358+0000", 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312358+0000", 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.410 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312359+0000", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.094e-06 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312416+0000", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7358000000000002e-05 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312426+0000", 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7310000000000005e-06 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.411 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312431+0000", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8679999999999998e-06 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312479+0000", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8168999999999999e-05 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312581+0000", 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102649 2024-04-15T18:25:58.412 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.312682+0000", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010050799999999999 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.315427+0000", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027447420000000001 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.315604+0000", 2024-04-15T18:25:58.413 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017771700000000001 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.492185+0000", 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17658079500000001 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.492237+0000", 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2352999999999998e-05 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.414 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.492288+0000", 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.045e-05 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:78863 5.9 5:9da4e27e:::benchmark_data_smithi159_116570_object78862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e397)", 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:57.333770+0000", 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.97529360399999998, 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.198934153, 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.415 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 78863 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333770+0000", 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.416 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333770+0000", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333771+0000", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5209999999999999e-06 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333808+0000", 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6850000000000001e-05 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.417 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333822+0000", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3551e-05 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333829+0000", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9349999999999997e-06 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333872+0000", 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3390999999999998e-05 2024-04-15T18:25:58.418 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.333997+0000", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012439100000000001 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.334091+0000", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4248999999999998e-05 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.336317+0000", 2024-04-15T18:25:58.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022262179999999999 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.336459+0000", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000142189 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.532596+0000", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19613715300000001 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.532659+0000", 2024-04-15T18:25:58.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2197000000000001e-05 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.532704+0000", 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5507999999999999e-05 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:78917 5.d e397/392 5:b42f7814:::benchmark_data_smithi159_116570_object78916:head v 397'4881, mlcod=397'4877)", 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:57.573576+0000", 2024-04-15T18:25:58.421 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.73548760499999999, 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.10971829, 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.573576+0000", 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.573576+0000", 2024-04-15T18:25:58.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.573577+0000", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3600000000000003e-07 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.573639+0000", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1964000000000005e-05 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.681057+0000", 2024-04-15T18:25:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.107418166 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.681062+0000", 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6740000000000002e-06 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.681124+0000", 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1513000000000005e-05 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:25:58.424 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.681138+0000", 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4535999999999999e-05 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.683252+0000", 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021136729999999999 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:57.683294+0000", 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2128000000000001e-05 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.425 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:25:58.426 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:25:58.426 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:25:58.426 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:25:58.461 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:58.492 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:25:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:58.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:25:58.499 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:58.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:58.637 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:25:58.715 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:58.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:58.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 80083 80067 69.4953 66.5625 0.0167132 0.0143824 2024-04-15T18:25:58.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:58.851 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:58.958 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:25:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:58.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:25:59.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:25:59.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:25:59.093 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:59.264 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:59 smithi082 ceph-mon[44206]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 2294/232794 objects degraded (0.985%); 29 MiB/s, 457 objects/s recovering 2024-04-15T18:25:59.264 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:59 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:59.264 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:59.264 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:25:59 smithi082 ceph-mon[44206]: osdmap e398: 12 total, 12 up, 11 in 2024-04-15T18:25:59.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:59 smithi159 ceph-mon[45336]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 2294/232794 objects degraded (0.985%); 29 MiB/s, 457 objects/s recovering 2024-04-15T18:25:59.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:59 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:59.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:59.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:25:59 smithi159 ceph-mon[45336]: osdmap e398: 12 total, 12 up, 11 in 2024-04-15T18:25:59.407 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:25:59.449 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:25:59.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:59 smithi049 ceph-mon[40087]: pgmap v1071: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 2294/232794 objects degraded (0.985%); 29 MiB/s, 457 objects/s recovering 2024-04-15T18:25:59.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:59 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:25:59.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1802318890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:25:59.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:25:59 smithi049 ceph-mon[40087]: osdmap e398: 12 total, 12 up, 11 in 2024-04-15T18:25:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:25:59.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:25:59.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:25:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:25:59.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 81105 81089 69.4182 63.875 0.0207737 0.0143985 2024-04-15T18:26:00.334 INFO:teuthology.orchestra.run.smithi049.stderr:set 5.1 pg_upmap mapping to [0,11,6] 2024-04-15T18:26:00.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:00.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:00.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]: dispatch 2024-04-15T18:26:00.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:00 smithi082 ceph-mon[44206]: osdmap e399: 12 total, 12 up, 11 in 2024-04-15T18:26:00.557 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:00.557 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:00.557 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]: dispatch 2024-04-15T18:26:00.557 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:00 smithi159 ceph-mon[45336]: osdmap e399: 12 total, 12 up, 11 in 2024-04-15T18:26:00.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]: dispatch 2024-04-15T18:26:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:00 smithi049 ceph-mon[40087]: osdmap e399: 12 total, 12 up, 11 in 2024-04-15T18:26:00.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 81973 81957 69.2132 54.25 0.00287121 0.0144184 2024-04-15T18:26:01.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:01 smithi082 ceph-mon[44206]: pgmap v1074: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 35 MiB/s wr, 557 op/s; 2294/234741 objects degraded (0.977%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:26:01.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]': finished 2024-04-15T18:26:01.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:01 smithi082 ceph-mon[44206]: osdmap e400: 12 total, 12 up, 11 in 2024-04-15T18:26:01.553 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:01.553 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:26:01.553 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:26:01.553 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:01.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:01 smithi159 ceph-mon[45336]: pgmap v1074: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 35 MiB/s wr, 557 op/s; 2294/234741 objects degraded (0.977%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:26:01.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]': finished 2024-04-15T18:26:01.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:01 smithi159 ceph-mon[45336]: osdmap e400: 12 total, 12 up, 11 in 2024-04-15T18:26:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:01 smithi049 ceph-mon[40087]: pgmap v1074: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 35 MiB/s wr, 557 op/s; 2294/234741 objects degraded (0.977%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:26:01.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3304537456' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1", "id": [0, 11, 6]}]': finished 2024-04-15T18:26:01.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:01 smithi049 ceph-mon[40087]: osdmap e400: 12 total, 12 up, 11 in 2024-04-15T18:26:01.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 82071 82055 68.372 6.125 0.33617 0.0144207 2024-04-15T18:26:02.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:26:02.244 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:26:02.541 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:02.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:02 smithi159 ceph-mon[45336]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 649 op/s; 2294/239106 objects degraded (0.959%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:26:02.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:02 smithi159 ceph-mon[45336]: osdmap e401: 12 total, 12 up, 11 in 2024-04-15T18:26:02.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2294/234741 objects degraded (0.977%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:02 smithi049 ceph-mon[40087]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 649 op/s; 2294/239106 objects degraded (0.959%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:26:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:02 smithi049 ceph-mon[40087]: osdmap e401: 12 total, 12 up, 11 in 2024-04-15T18:26:02.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2294/234741 objects degraded (0.977%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:02.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 82263 82247 67.6303 12 0.00526817 0.0147814 2024-04-15T18:26:02.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:02 smithi082 ceph-mon[44206]: pgmap v1076: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 41 MiB/s wr, 649 op/s; 2294/239106 objects degraded (0.959%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:26:02.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:02 smithi082 ceph-mon[44206]: osdmap e401: 12 total, 12 up, 11 in 2024-04-15T18:26:02.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2294/234741 objects degraded (0.977%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:03.344 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:03 smithi159 ceph-mon[45336]: osdmap e402: 12 total, 12 up, 11 in 2024-04-15T18:26:03.646 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:03.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:03 smithi049 ceph-mon[40087]: osdmap e402: 12 total, 12 up, 11 in 2024-04-15T18:26:03.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:03.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:03.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 83145 83129 67.4678 55.125 0.0166732 0.014816 2024-04-15T18:26:03.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:03 smithi082 ceph-mon[44206]: osdmap e402: 12 total, 12 up, 11 in 2024-04-15T18:26:03.789 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:03.918 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:03.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:04.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:26:04.055 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.082 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:26:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:04.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:26:04.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:04.243 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.277 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:04.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:04.382 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:04.418 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.439 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:04.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:04.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:26:04.577 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.588 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:26:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:26:04.588 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:26:04.588 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:04.604 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:26:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:04.608 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:04 smithi082 ceph-mon[44206]: pgmap v1079: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1965/245922 objects degraded (0.799%); 14901/245922 objects misplaced (6.059%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:26:04.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:26:04.616 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:04 smithi159 ceph-mon[45336]: pgmap v1079: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1965/245922 objects degraded (0.799%); 14901/245922 objects misplaced (6.059%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:26:04.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:04.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:04 smithi049 ceph-mon[40087]: pgmap v1079: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1965/245922 objects degraded (0.799%); 14901/245922 objects misplaced (6.059%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:26:04.743 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.765 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:04.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 84142 84126 67.4017 62.3125 0.0111822 0.0148286 2024-04-15T18:26:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:04.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:04.779 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:04.810 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:26:04.839 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:26:04.869 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:04.871 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:26:04.902 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:04.903 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:26:04.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:26:04.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:26:05.000 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:26:05.030 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:26:05.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:26:05.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:26:05.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:26:05.172 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:26:05.173 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:26:05.173 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:26:05.173 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:26:05.174 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:26:05.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3769726139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:05.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3769726139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:05.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 85042 85026 67.2604 56.25 0.00416891 0.0148622 2024-04-15T18:26:05.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3769726139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:05.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:05.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:06 smithi049 ceph-mon[40087]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 1965/247635 objects degraded (0.794%); 14901/247635 objects misplaced (6.017%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:06 smithi159 ceph-mon[45336]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 1965/247635 objects degraded (0.794%); 14901/247635 objects misplaced (6.017%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:06.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:26:06.773603+0000 min lat: 0.00106705 max lat: 2.3859 avg lat: 0.0148764 2024-04-15T18:26:06.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:26:06.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 86043 86027 67.2017 62.5625 0.0153115 0.0148764 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:06.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:06 smithi082 ceph-mon[44206]: pgmap v1080: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s; 1965/247635 objects degraded (0.794%); 14901/247635 objects misplaced (6.017%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:07.040 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:07.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:07.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:26:07.144 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:07.234 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:07.267 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:07.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:07.354 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1965/247635 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:07.374 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:26:07.409 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:07.434 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:26:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:07.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:26:07.493 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:07.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:07.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1965/247635 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:07.580 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:07.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:07.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:26:07.745 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1965/247635 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:07.747 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:07.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 86915 86899 67.0448 54.5 0.0123052 0.0149088 2024-04-15T18:26:07.774 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:26:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:07.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:26:07.879 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:07.922 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:08.355 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:08 smithi159 ceph-mon[45336]: pgmap v1081: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s; 1587/254961 objects degraded (0.622%); 14901/254961 objects misplaced (5.844%); 6.4 MiB/s, 101 objects/s recovering 2024-04-15T18:26:08.356 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:26:08.362 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:26:08.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:26:08.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:08 smithi049 ceph-mon[40087]: pgmap v1081: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s; 1587/254961 objects degraded (0.622%); 14901/254961 objects misplaced (5.844%); 6.4 MiB/s, 101 objects/s recovering 2024-04-15T18:26:08.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:26:08.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 87799 87783 66.9009 55.25 0.0065761 0.0149419 2024-04-15T18:26:08.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:08 smithi082 ceph-mon[44206]: pgmap v1081: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s; 1587/254961 objects degraded (0.622%); 14901/254961 objects misplaced (5.844%); 6.4 MiB/s, 101 objects/s recovering 2024-04-15T18:26:08.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:26:09.431 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:09.524 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3] out_osds: [2] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:26:09.524 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:26:09.524 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-15T18:26:09.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd in 2 2024-04-15T18:26:09.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:09 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:09.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:26:09.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:09 smithi049 ceph-mon[40087]: osdmap e403: 12 total, 12 up, 11 in 2024-04-15T18:26:09.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:09 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:26:09.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:09 smithi049 ceph-mon[40087]: 5.a scrub starts 2024-04-15T18:26:09.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:09 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:09.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:26:09.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:09 smithi159 ceph-mon[45336]: osdmap e403: 12 total, 12 up, 11 in 2024-04-15T18:26:09.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:09 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:26:09.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:09 smithi159 ceph-mon[45336]: 5.a scrub starts 2024-04-15T18:26:09.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 88545 88529 66.6565 46.625 0.0549272 0.0149959 2024-04-15T18:26:09.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:09 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:09.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1672429653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:26:09.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:09 smithi082 ceph-mon[44206]: osdmap e403: 12 total, 12 up, 11 in 2024-04-15T18:26:09.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:09 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:26:09.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:09 smithi082 ceph-mon[44206]: 5.a scrub starts 2024-04-15T18:26:09.985 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:09.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:10.089 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:26:10.126 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:10.155 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:26:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:10.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:26:10.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:10.307 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:10.567 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:10.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:10 smithi159 ceph-mon[45336]: pgmap v1083: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 1587/257571 objects degraded (0.616%); 14574/257571 objects misplaced (5.658%); 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:26:10.569 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:10 smithi159 ceph-mon[45336]: 5.a scrub ok 2024-04-15T18:26:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:10.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:10.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:10.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:10 smithi049 ceph-mon[40087]: pgmap v1083: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 1587/257571 objects degraded (0.616%); 14574/257571 objects misplaced (5.658%); 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:26:10.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:10 smithi049 ceph-mon[40087]: 5.a scrub ok 2024-04-15T18:26:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:10.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 89511 89495 66.5817 60.375 0.00536383 0.0150137 2024-04-15T18:26:10.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:10 smithi082 ceph-mon[44206]: pgmap v1083: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 5.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 1587/257571 objects degraded (0.616%); 14574/257571 objects misplaced (5.658%); 6.4 MiB/s, 102 objects/s recovering 2024-04-15T18:26:10.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:10 smithi082 ceph-mon[44206]: 5.a scrub ok 2024-04-15T18:26:10.898 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:10.919 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:10.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:11.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:11.057 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:11.190 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:11.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:11.298 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:11.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:26:11.333 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:11.382 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:11 smithi159 ceph-mon[45336]: 5.9 scrub ok 2024-04-15T18:26:11.441 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:11.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:11.546 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:11.589 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:11.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:11 smithi049 ceph-mon[40087]: 5.9 scrub ok 2024-04-15T18:26:11.701 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:11.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:11.763 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:26:11.764 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:26:11.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 90281 90265 66.3645 48.125 0.00512332 0.015061 2024-04-15T18:26:11.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:11 smithi082 ceph-mon[44206]: 5.9 scrub ok 2024-04-15T18:26:11.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:26:11.863 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:12.023 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:26:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:12.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:26:12.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:26:12.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:12.163 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:26:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:12.201 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:26:12.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:26:12.269 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:26:12.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:26:12.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:26:12.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:26:12.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:26:12.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:12 smithi049 ceph-mon[40087]: pgmap v1084: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 41 MiB/s wr, 648 op/s; 1587/261696 objects degraded (0.606%); 14574/261696 objects misplaced (5.569%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:12.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:12 smithi049 ceph-mon[40087]: 5.0 scrub starts 2024-04-15T18:26:12.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1827490199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:12.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1587/261696 objects degraded (0.606%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:12.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:26:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:12.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:12.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:26:12.639 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:12 smithi159 ceph-mon[45336]: pgmap v1084: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 41 MiB/s wr, 648 op/s; 1587/261696 objects degraded (0.606%); 14574/261696 objects misplaced (5.569%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:12.639 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:12 smithi159 ceph-mon[45336]: 5.0 scrub starts 2024-04-15T18:26:12.639 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1827490199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:12.640 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1587/261696 objects degraded (0.606%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:12.677 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:26:12.721 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:26:12.723 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:26:12.724 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:26:12.736 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:12.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 91075 91059 66.1681 49.625 0.0212578 0.015107 2024-04-15T18:26:12.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:12 smithi082 ceph-mon[44206]: pgmap v1084: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 41 MiB/s wr, 648 op/s; 1587/261696 objects degraded (0.606%); 14574/261696 objects misplaced (5.569%); 3.4 MiB/s, 54 objects/s recovering 2024-04-15T18:26:12.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:12 smithi082 ceph-mon[44206]: 5.0 scrub starts 2024-04-15T18:26:12.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1827490199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:12.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1587/261696 objects degraded (0.606%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:12.786 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:12.811 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:12.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:12.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:26:12.952 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:12.973 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:26:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:12.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:26:12.979 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:13.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:13.109 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:13.132 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:13.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:13.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:13.270 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.723 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:13.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:13.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 91688 91672 65.8479 38.3125 0.00975047 0.0151801 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:13.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:13.866 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:14 smithi159 ceph-mon[45336]: pgmap v1085: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 816 op/s; 1248/270444 objects degraded (0.461%); 14250/270444 objects misplaced (5.269%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:14.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:14.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:14 smithi049 ceph-mon[40087]: pgmap v1085: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 816 op/s; 1248/270444 objects degraded (0.461%); 14250/270444 objects misplaced (5.269%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:14.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 92545 92529 65.7082 53.5625 0.052689 0.0152126 2024-04-15T18:26:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:14 smithi082 ceph-mon[44206]: pgmap v1085: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 816 op/s; 1248/270444 objects degraded (0.461%); 14250/270444 objects misplaced (5.269%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:92720 5.b 5:d786755e:::benchmark_data_smithi159_116570_object92719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.006569+0000", 2024-04-15T18:26:15.018 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.016571227000000001, 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016604250000000001, 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 92720 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006569+0000", 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006569+0000", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006571+0000", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.77e-06 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006596+0000", 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4541000000000001e-05 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006608+0000", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2118e-05 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006617+0000", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2979999999999997e-06 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006693+0000", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6038999999999995e-05 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:15.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.006883+0000", 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000190316 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.007083+0000", 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019988100000000001 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.011399+0000", 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00431621 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.016489+0000", 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050898840000000002 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:92731 5.b 5:db391951:::benchmark_data_smithi159_116570_object92730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.015839+0000", 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0073009889999999999, 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074049290000000002, 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:26:15.023 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 92731 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015839+0000", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015839+0000", 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015841+0000", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5910000000000001e-06 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015867+0000", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5565e-05 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015876+0000", 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.781e-06 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015882+0000", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2739999999999998e-06 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.016533+0000", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00065103899999999996 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.016680+0000", 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014699400000000001 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.016846+0000", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016656900000000001 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.022713+0000", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058663949999999999 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:92732 5.b 5:d39adfd4:::benchmark_data_smithi159_116570_object92731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.015893+0000", 2024-04-15T18:26:15.027 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0072473640000000001, 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074013029999999997, 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 92732 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015893+0000", 2024-04-15T18:26:15.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015893+0000", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015893+0000", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6600000000000002e-07 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015911+0000", 2024-04-15T18:26:15.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7547e-05 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015916+0000", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5090000000000004e-06 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.015919+0000", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.264e-06 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.019308+0000", 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003389268 2024-04-15T18:26:15.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.019481+0000", 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000173141 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.019683+0000", 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00020186100000000001 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 3 2024-04-15T18:26:15.031 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:15.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:14 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:26:14.907+0000 7f8015e7e700 -1 mon.a@0(leader).osd e403 definitely_dead 0 2024-04-15T18:26:15.421 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.2. 2024-04-15T18:26:15.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:15.561 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:15.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:15.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:15 smithi159 ceph-mon[45336]: 5.c scrub starts 2024-04-15T18:26:15.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:15 smithi159 ceph-mon[45336]: 5.0 scrub ok 2024-04-15T18:26:15.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/967159314' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:15.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:15.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:15 smithi049 ceph-mon[40087]: 5.c scrub starts 2024-04-15T18:26:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:15 smithi049 ceph-mon[40087]: 5.0 scrub ok 2024-04-15T18:26:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/967159314' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:15.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:26:15.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 93146 93130 65.392 37.5625 0.00365374 0.0152319 2024-04-15T18:26:15.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:15 smithi082 ceph-mon[44206]: 5.c scrub starts 2024-04-15T18:26:15.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:15 smithi082 ceph-mon[44206]: 5.0 scrub ok 2024-04-15T18:26:15.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/967159314' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:15.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-15T18:26:16.013 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:16 smithi049 ceph-mon[40087]: 5.c scrub ok 2024-04-15T18:26:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:16 smithi049 ceph-mon[40087]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 1248/272805 objects degraded (0.457%); 14250/272805 objects misplaced (5.224%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-15T18:26:16.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:16 smithi049 ceph-mon[40087]: osdmap e404: 12 total, 12 up, 12 in 2024-04-15T18:26:16.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:16 smithi159 ceph-mon[45336]: 5.c scrub ok 2024-04-15T18:26:16.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:16 smithi159 ceph-mon[45336]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 1248/272805 objects degraded (0.457%); 14250/272805 objects misplaced (5.224%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:16.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-15T18:26:16.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:16 smithi159 ceph-mon[45336]: osdmap e404: 12 total, 12 up, 12 in 2024-04-15T18:26:16.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 16 93157 93141 64.6731 0.6875 0.00393665 0.015268 2024-04-15T18:26:16.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:16 smithi082 ceph-mon[44206]: 5.c scrub ok 2024-04-15T18:26:16.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:16 smithi082 ceph-mon[44206]: pgmap v1086: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 838 op/s; 1248/272805 objects degraded (0.457%); 14250/272805 objects misplaced (5.224%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:26:16.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-15T18:26:16.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:16 smithi082 ceph-mon[44206]: osdmap e404: 12 total, 12 up, 12 in 2024-04-15T18:26:17.227 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:26:17.524 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:17.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:17 smithi159 ceph-mon[45336]: osdmap e405: 12 total, 12 up, 12 in 2024-04-15T18:26:17.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1248/272805 objects degraded (0.457%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:17.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 91 16 93157 93141 63.9624 0 - 0.015268 2024-04-15T18:26:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:17 smithi082 ceph-mon[44206]: osdmap e405: 12 total, 12 up, 12 in 2024-04-15T18:26:17.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1248/272805 objects degraded (0.457%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:17.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:17 smithi049 ceph-mon[40087]: osdmap e405: 12 total, 12 up, 12 in 2024-04-15T18:26:17.814 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1248/272805 objects degraded (0.457%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:18.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:18 smithi049 ceph-mon[40087]: pgmap v1089: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 797 op/s; 1248/276705 objects degraded (0.451%); 14250/276705 objects misplaced (5.150%); 3.5 MiB/s, 55 objects/s recovering 2024-04-15T18:26:18.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:18 smithi049 ceph-mon[40087]: osdmap e406: 12 total, 12 up, 12 in 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 91.1393 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 93157 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 63.8836 2024-04-15T18:26:18.770 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 34.2738 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 211.875 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 1022 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 548.38 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 3390 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 0 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0156477 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.0556244 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 2.48637 2024-04-15T18:26:18.771 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00106705 2024-04-15T18:26:18.776 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:26:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:18 smithi082 ceph-mon[44206]: pgmap v1089: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 797 op/s; 1248/276705 objects degraded (0.451%); 14250/276705 objects misplaced (5.150%); 3.5 MiB/s, 55 objects/s recovering 2024-04-15T18:26:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:18 smithi082 ceph-mon[44206]: osdmap e406: 12 total, 12 up, 12 in 2024-04-15T18:26:18.839 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:18 smithi159 ceph-mon[45336]: pgmap v1089: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 797 op/s; 1248/276705 objects degraded (0.451%); 14250/276705 objects misplaced (5.150%); 3.5 MiB/s, 55 objects/s recovering 2024-04-15T18:26:18.839 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:18 smithi159 ceph-mon[45336]: osdmap e406: 12 total, 12 up, 12 in 2024-04-15T18:26:19.048 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:19.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:19.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:26:19.525 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:26:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:26:19.525 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:26:19.525 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:20.011 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:20.167 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:26:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:20.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:26:20.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:20.332 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:20.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:26:20.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:20 smithi159 ceph-mon[45336]: pgmap v1091: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 489 op/s; 818/279249 objects degraded (0.293%); 25400/279249 objects misplaced (9.096%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:20.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:20 smithi049 ceph-mon[40087]: pgmap v1091: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 489 op/s; 818/279249 objects degraded (0.293%); 25400/279249 objects misplaced (9.096%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:20.736 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:20.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:20 smithi082 ceph-mon[44206]: pgmap v1091: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 489 op/s; 818/279249 objects degraded (0.293%); 25400/279249 objects misplaced (9.096%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:22 smithi049 ceph-mon[40087]: pgmap v1092: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 23 MiB/s wr, 422 op/s; 818/278484 objects degraded (0.294%); 25400/278484 objects misplaced (9.121%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:22 smithi049 ceph-mon[40087]: osdmap e407: 12 total, 12 up, 12 in 2024-04-15T18:26:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 818/278484 objects degraded (0.294%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:22.725 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:158593 3.0 e255/199 3:066955e5:::benchmark_data_smithi159_91621_object52117:head v 255'10025, mlcod=255'10022)", 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 455.43237649299999, 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.239038166, 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296563+0000", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296564+0000", 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2600000000000001e-07 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.296571+0000", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7870000000000004e-06 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530850+0000", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.234278768 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530857+0000", 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530914+0000", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7123000000000002e-05 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.530935+0000", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1597e-05 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535555+0000", 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046197399999999998 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:18:47.535601+0000", 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6391999999999999e-05 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:212948 3.2 e275/235 3:409c861b:::benchmark_data_smithi159_91621_object106472:head v 275'13490, mlcod=242'8024)", 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 367.72294165900001, 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56367495999999995, 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.730 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005999+0000", 2024-04-15T18:26:22.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3680000000000001e-06 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.006010+0000", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0786e-05 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565792+0000", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55978132000000003 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565799+0000", 2024-04-15T18:26:22.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.255e-06 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565863+0000", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4552000000000001e-05 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565893+0000", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9329999999999999e-05 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569633+0000", 2024-04-15T18:26:22.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037404080000000002 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569673+0000", 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9941000000000003e-05 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15517 4.2 4:4c161e0b:::benchmark_data_smithi159_102124_object15516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:26:22.734 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 332.308533415, 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26252755999999999, 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15517 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420406+0000", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420407+0000", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3099999999999996e-07 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420426+0000", 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8371000000000001e-05 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420435+0000", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.448e-06 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420440+0000", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5149999999999997e-06 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420481+0000", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0311000000000001e-05 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420569+0000", 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8140999999999995e-05 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.420654+0000", 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5535e-05 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421174+0000", 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00051933100000000005 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.421526+0000", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035268700000000002 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682869+0000", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26134240600000003 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682904+0000", 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.682934+0000", 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9674000000000001e-05 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:15520 4.2 4:4779e579:::benchmark_data_smithi159_102124_object15519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e292)", 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 332.30387927100003, 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27309090499999999, 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.740 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 15520 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425060+0000", 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425061+0000", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8800000000000001e-07 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425080+0000", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8600000000000001e-05 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425090+0000", 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9219999999999992e-06 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425096+0000", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.84e-06 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425134+0000", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8637000000000003e-05 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425212+0000", 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7958000000000001e-05 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425297+0000", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4568999999999996e-05 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.425802+0000", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050520399999999998 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.426334+0000", 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00053202999999999996 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698085+0000", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27175072900000002 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698115+0000", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0343000000000001e-05 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:50.698151+0000", 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6185e-05 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.745 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:71486 4.2 4:46c4a592:::benchmark_data_smithi159_102124_object71485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 272.85272390799997, 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.308684452, 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 71486 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876217+0000", 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3880000000000001e-06 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876262+0000", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4391000000000002e-05 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876277+0000", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5481e-05 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876284+0000", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.374e-06 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876338+0000", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3653000000000003e-05 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876457+0000", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118548 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876557+0000", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010054 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878524+0000", 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019665110000000002 2024-04-15T18:26:22.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878860+0000", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00033609900000000002 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184840+0000", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30597998300000001 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184876+0000", 2024-04-15T18:26:22.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5997000000000001e-05 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184900+0000", 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4487000000000001e-05 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:89342 4.2 4:4b07b843:::benchmark_data_smithi159_102124_object89341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 251.307508692, 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36570210600000003, 2024-04-15T18:26:22.751 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 89342 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421433+0000", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.609e-06 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421458+0000", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5007999999999998e-05 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421472+0000", 2024-04-15T18:26:22.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4518999999999999e-05 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421481+0000", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.507e-06 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421547+0000", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5950000000000004e-05 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421686+0000", 2024-04-15T18:26:22.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013925 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421818+0000", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013244400000000001 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.428345+0000", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065266689999999997 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.435961+0000", 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076156009999999996 2024-04-15T18:26:22.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787063+0000", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.351102101 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787102+0000", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9351000000000003e-05 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787133+0000", 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1096999999999997e-05 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.756 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43719 5.b 5.2edd20bb (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.448004905999994, 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96247389699999997, 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43719 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.757 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280937+0000", 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7749999999999999e-06 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280959+0000", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2347000000000001e-05 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280972+0000", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3123e-05 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280981+0000", 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1179999999999998e-06 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281041+0000", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9335e-05 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281049+0000", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7730000000000005e-06 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243354+0000", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96230429100000003 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243409+0000", 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5135e-05 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43724 5.b 5.c9c3aa8b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.423816677000005, 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93864302600000005, 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.761 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43724 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305125+0000", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9e-06 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305150+0000", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4640000000000001e-05 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305167+0000", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.694e-05 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305176+0000", 2024-04-15T18:26:22.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6220000000000004e-06 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305242+0000", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5919000000000004e-05 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305251+0000", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9779999999999994e-06 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243757+0000", 2024-04-15T18:26:22.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93850576200000002 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243766+0000", 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2650000000000002e-06 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43728 5.b 5.b2d2527b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.240882791000004, 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75574586300000002, 2024-04-15T18:26:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43728 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488059+0000", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8610000000000001e-06 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488103+0000", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4598000000000002e-05 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488118+0000", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4708e-05 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488125+0000", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7630000000000003e-06 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488178+0000", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3477999999999999e-05 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488188+0000", 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2890000000000003e-06 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243796+0000", 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75560810199999995 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243803+0000", 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0639999999999996e-06 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43732 5.b 5.9c14c54b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.047775572000006, 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56280661600000004, 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43732 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681165+0000", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.243e-06 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681206+0000", 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0428e-05 2024-04-15T18:26:22.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681218+0000", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2269999999999999e-05 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681225+0000", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4529999999999999e-06 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681272+0000", 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7775999999999998e-05 2024-04-15T18:26:22.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681280+0000", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.892e-06 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243939+0000", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56265829300000003 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243971+0000", 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2261000000000002e-05 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43719 5.b 5:dd04bb74:::benchmark_data_smithi159_116570_object43718:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:15.258862+0000", 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.470077662999998, 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29592895000000002, 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43719 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.774 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258862+0000", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258862+0000", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258864+0000", 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.782e-06 2024-04-15T18:26:22.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258910+0000", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6553e-05 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258927+0000", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6504e-05 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258939+0000", 2024-04-15T18:26:22.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2374999999999999e-05 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260040+0000", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001100845 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260052+0000", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1418e-05 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.542877+0000", 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28282515899999999 2024-04-15T18:26:22.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.543011+0000", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013451199999999999 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.543140+0000", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000128682 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.548775+0000", 2024-04-15T18:26:22.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056347439999999997 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.548794+0000", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9205e-05 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.552800+0000", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040064610000000002 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.552861+0000", 2024-04-15T18:26:22.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0401e-05 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.554791+0000", 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019303090000000001 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43724 5.b 5:d155c393:::benchmark_data_smithi159_116570_object43723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:15.258961+0000", 2024-04-15T18:26:22.780 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.469978396000002, 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30137880500000003, 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43724 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258961+0000", 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258961+0000", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258962+0000", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6199999999999998e-07 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258982+0000", 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0344999999999999e-05 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258991+0000", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000003e-06 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.258997+0000", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2049999999999999e-06 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260075+0000", 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00107839 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260082+0000", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6549999999999998e-06 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.546506+0000", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28642444900000003 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.546678+0000", 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017167699999999999 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.546811+0000", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013315899999999999 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.548823+0000", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002011648 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.548831+0000", 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7810000000000005e-06 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.558254+0000", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094230719999999994 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.558318+0000", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4357999999999996e-05 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.560340+0000", 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020219639999999998 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43728 5.b 5:de4a4b4d:::benchmark_data_smithi159_116570_object43727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:15.259006+0000", 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.469933836999999, 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30277701600000001, 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43728 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259006+0000", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259006+0000", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259006+0000", 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:26:22.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259027+0000", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0823999999999999e-05 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259035+0000", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3379999999999999e-06 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259040+0000", 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4659999999999998e-06 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260104+0000", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010637929999999999 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260110+0000", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1480000000000003e-06 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.548927+0000", 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.288817041 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.549057+0000", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013042800000000001 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.549185+0000", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000127599 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.556401+0000", 2024-04-15T18:26:22.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072158650000000001 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.556444+0000", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2873000000000001e-05 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.560407+0000", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039630580000000002 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.560466+0000", 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9040999999999999e-05 2024-04-15T18:26:22.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.561783+0000", 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001317063 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43732 5.b 5:d2a32839:::benchmark_data_smithi159_116570_object43731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:15.259048+0000", 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.469891720000007, 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30380968600000002, 2024-04-15T18:26:22.793 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43732 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259048+0000", 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259048+0000", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259048+0000", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1899999999999998e-07 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259071+0000", 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2634999999999999e-05 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259079+0000", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1880000000000002e-06 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259084+0000", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0579999999999997e-06 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260130+0000", 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010458100000000001 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.260137+0000", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5629999999999997e-06 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.550709+0000", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29057187000000001 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.550881+0000", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017232900000000001 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:26:22.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.551085+0000", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000203673 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.556479+0000", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053946269999999999 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.556487+0000", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2619999999999998e-06 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.561854+0000", 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053667610000000003 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.561896+0000", 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2052999999999999e-05 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.562858+0000", 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096143800000000001 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.799 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 400/377 [PushReplyOp(5:d68de874:::benchmark_data_smithi159_116570_object10690:head)])", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.529236366999999, 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37573558899999998, 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:26:22.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199704+0000", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4300000000000002e-07 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199711+0000", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2540000000000003e-06 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574171+0000", 2024-04-15T18:26:22.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.374460026 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574179+0000", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1080000000000004e-06 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574267+0000", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8676000000000003e-05 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.575439+0000", 2024-04-15T18:26:22.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011717819999999999 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:82071 5.e 5:768b069e:::benchmark_data_smithi159_116570_object82070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e401)", 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.165286434999999, 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84584237600000001, 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 82071 2024-04-15T18:26:22.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563655+0000", 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0899999999999999e-06 2024-04-15T18:26:22.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563684+0000", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8977999999999999e-05 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563704+0000", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9874999999999999e-05 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563715+0000", 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0266e-05 2024-04-15T18:26:22.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563777+0000", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2502999999999994e-05 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563782+0000", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6979999999999997e-06 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565036+0000", 2024-04-15T18:26:22.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012537850000000001 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565179+0000", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014387199999999999 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565329+0000", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000150004 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568089+0000", 2024-04-15T18:26:22.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027595150000000001 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.569096+0000", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010070610000000001 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409426+0000", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84032972100000003 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409466+0000", 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0374999999999999e-05 2024-04-15T18:26:22.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409496+0000", 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9632999999999999e-05 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:82071 5.e e401/358 ondisk, result = 0)", 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.160909105999998, 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84147125499999997, 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568032+0000", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3790000000000001e-06 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568044+0000", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1654999999999999e-05 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409368+0000", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.841324552 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409372+0000", 2024-04-15T18:26:22.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.048e-06 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409416+0000", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3933000000000001e-05 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409422+0000", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6860000000000003e-06 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409502+0000", 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0001999999999997e-05 2024-04-15T18:26:22.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 404/377 [PushReplyOp(5:dd0ee7d7:::benchmark_data_smithi159_116570_object27502:head)])", 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.2596500299999995, 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39222512599999998, 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:26:22.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469295+0000", 2024-04-15T18:26:22.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6469999999999997e-06 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860402+0000", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39110686099999997 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860410+0000", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860475+0000", 2024-04-15T18:26:22.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5375000000000003e-05 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.861515+0000", 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010396470000000001 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 405/377 [PushReplyOp(5:d997bfb1:::benchmark_data_smithi159_116570_object30349:head)])", 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.9718527689999998, 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43079569699999998, 2024-04-15T18:26:22.816 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757088+0000", 2024-04-15T18:26:22.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7000000000000005e-07 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757092+0000", 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6079999999999998e-06 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186476+0000", 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42938425299999999 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186493+0000", 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6646e-05 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186666+0000", 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017301299999999999 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.187883+0000", 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001216607 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 406/377 [PushReplyOp(5:da19a5f9:::benchmark_data_smithi159_116570_object32780:head)])", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.8441711600000001, 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.322450403, 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:26:22.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884770+0000", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884778+0000", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9130000000000003e-06 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206398+0000", 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32162084699999999 2024-04-15T18:26:22.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206407+0000", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1780000000000003e-06 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206476+0000", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8959000000000002e-05 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.207219+0000", 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074334900000000003 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.822 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:22.823 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:22.823 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:22.823 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:22.823 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:22 smithi082 ceph-mon[44206]: pgmap v1092: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 23 MiB/s wr, 422 op/s; 818/278484 objects degraded (0.294%); 25400/278484 objects misplaced (9.121%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:22 smithi082 ceph-mon[44206]: osdmap e407: 12 total, 12 up, 12 in 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 818/278484 objects degraded (0.294%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:22 smithi159 ceph-mon[45336]: pgmap v1092: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 23 MiB/s wr, 422 op/s; 818/278484 objects degraded (0.294%); 25400/278484 objects misplaced (9.121%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:22 smithi159 ceph-mon[45336]: osdmap e407: 12 total, 12 up, 12 in 2024-04-15T18:26:22.824 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 818/278484 objects degraded (0.294%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:23.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:26:23.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:23.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:23.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:23 smithi049 ceph-mon[40087]: osdmap e408: 12 total, 12 up, 12 in 2024-04-15T18:26:23.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:23.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:23.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:23 smithi082 ceph-mon[44206]: osdmap e408: 12 total, 12 up, 12 in 2024-04-15T18:26:23.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:23.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:23.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:23 smithi159 ceph-mon[45336]: osdmap e408: 12 total, 12 up, 12 in 2024-04-15T18:26:23.901 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:23.980 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:23.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:24.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:24.141 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:24.165 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:24.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:24.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:26:24.373 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:24.452 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:24 smithi082 ceph-mon[44206]: pgmap v1095: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 9.6 MiB/s wr, 562 op/s; 241/272136 objects degraded (0.089%); 30173/272136 objects misplaced (11.087%); 20 MiB/s, 316 objects/s recovering 2024-04-15T18:26:24.452 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:24 smithi159 ceph-mon[45336]: pgmap v1095: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 9.6 MiB/s wr, 562 op/s; 241/272136 objects degraded (0.089%); 30173/272136 objects misplaced (11.087%); 20 MiB/s, 316 objects/s recovering 2024-04-15T18:26:24.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:24 smithi049 ceph-mon[40087]: pgmap v1095: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 9.6 MiB/s wr, 562 op/s; 241/272136 objects degraded (0.089%); 30173/272136 objects misplaced (11.087%); 20 MiB/s, 316 objects/s recovering 2024-04-15T18:26:24.966 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-15T18:26:25.821 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:25.821 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:26:25.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:98591 5.4 5:2f515ffe:::benchmark_data_smithi159_116570_object5431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-04-15T18:26:25.821 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:25.790604+0000", 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.034221438, 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.034246014999999998, 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 98591 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:25.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790604+0000", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790604+0000", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790606+0000", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.26e-06 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:25.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790615+0000", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8310000000000007e-06 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790628+0000", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3602e-05 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.790639+0000", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0318e-05 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:25.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.799641+0000", 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0090024730000000004 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.799770+0000", 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012896699999999999 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.799924+0000", 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015357000000000001 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.807303+0000", 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.007379661 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.811284+0000", 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003980674 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.826 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:98609 5.0 5:04a5eecd:::benchmark_data_smithi159_116570_object5449:head [delete] snapc 0=[] ondisk+write+known_if_redirected e408)", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:25.809653+0000", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.015172593, 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015300545, 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 98609 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:25.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809653+0000", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809653+0000", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809654+0000", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.319e-06 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:25.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809660+0000", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1290000000000001e-06 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809670+0000", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0311e-05 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809678+0000", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.588e-06 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809733+0000", 2024-04-15T18:26:25.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5090000000000003e-05 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809857+0000", 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012437700000000001 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.809976+0000", 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118621 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:25.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.813074+0000", 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030980809999999999 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.4 408/405 [PushReplyOp(5:2f41a22c:::benchmark_data_smithi159_116570_object5787:head)])", 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:25.822835+0000", 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0019896760000000001, 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022007509999999999, 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:25.831 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822835+0000", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822835+0000", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822837+0000", 2024-04-15T18:26:25.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.265e-06 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822852+0000", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5463000000000001e-05 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822882+0000", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9356000000000002e-05 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.822892+0000", 2024-04-15T18:26:25.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0723000000000001e-05 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.823080+0000", 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000187937 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:98591 5.4 e408/405)", 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:25.824291+0000", 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.00053441800000000005, 2024-04-15T18:26:25.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00078368800000000005, 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "queued for pg", 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824291+0000", 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824291+0000", 2024-04-15T18:26:25.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824292+0000", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4099999999999997e-07 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824298+0000", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3099999999999997e-06 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824340+0000", 2024-04-15T18:26:25.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2024000000000003e-05 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:25.824348+0000", 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0160000000000008e-06 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 4 2024-04-15T18:26:25.837 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:26.237 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:26:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:26.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:26:26.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:26.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:26 smithi049 ceph-mon[40087]: pgmap v1096: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 4.9 MiB/s wr, 449 op/s; 241/270687 objects degraded (0.089%); 30173/270687 objects misplaced (11.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:26:26.578 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:26.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:26 smithi159 ceph-mon[45336]: pgmap v1096: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 4.9 MiB/s wr, 449 op/s; 241/270687 objects degraded (0.089%); 30173/270687 objects misplaced (11.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:26:26.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:26:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:26 smithi082 ceph-mon[44206]: pgmap v1096: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 129 B/s rd, 4.9 MiB/s wr, 449 op/s; 241/270687 objects degraded (0.089%); 30173/270687 objects misplaced (11.147%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:26:26.980 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:27.497 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:27 smithi159 ceph-mon[45336]: osdmap e409: 12 total, 12 up, 12 in 2024-04-15T18:26:27.497 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 241/270687 objects degraded (0.089%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:27 smithi049 ceph-mon[40087]: osdmap e409: 12 total, 12 up, 12 in 2024-04-15T18:26:27.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 241/270687 objects degraded (0.089%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:27.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:27 smithi082 ceph-mon[44206]: osdmap e409: 12 total, 12 up, 12 in 2024-04-15T18:26:27.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 241/270687 objects degraded (0.089%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:26:27.926 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:27.933 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:27.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:28.038 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:28.073 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:28.430 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:28.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:28.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:28 smithi049 ceph-mon[40087]: pgmap v1098: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 738 op/s; 255/265251 objects degraded (0.096%); 29933/265251 objects misplaced (11.285%); 16 MiB/s, 254 objects/s recovering 2024-04-15T18:26:28.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:28 smithi049 ceph-mon[40087]: osdmap e410: 12 total, 12 up, 12 in 2024-04-15T18:26:28.538 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:28.574 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:28.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:28 smithi159 ceph-mon[45336]: pgmap v1098: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 738 op/s; 255/265251 objects degraded (0.096%); 29933/265251 objects misplaced (11.285%); 16 MiB/s, 254 objects/s recovering 2024-04-15T18:26:28.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:28 smithi159 ceph-mon[45336]: osdmap e410: 12 total, 12 up, 12 in 2024-04-15T18:26:28.709 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:28.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:28.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:28 smithi082 ceph-mon[44206]: pgmap v1098: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 738 op/s; 255/265251 objects degraded (0.096%); 29933/265251 objects misplaced (11.285%); 16 MiB/s, 254 objects/s recovering 2024-04-15T18:26:28.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:28 smithi082 ceph-mon[44206]: osdmap e410: 12 total, 12 up, 12 in 2024-04-15T18:26:28.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:28.856 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:28.898 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:28.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:29.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:29.053 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:29.077 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:29.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:29.086 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:26:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:26:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:26:29.087 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:29.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:26:29.227 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:29.336 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:26:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:29.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:26:29.444 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:29.484 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:29.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:29 smithi159 ceph-mon[45336]: 5.b scrub starts 2024-04-15T18:26:29.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:29 smithi049 ceph-mon[40087]: 5.b scrub starts 2024-04-15T18:26:29.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:29 smithi082 ceph-mon[44206]: 5.b scrub starts 2024-04-15T18:26:29.841 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:29.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:29.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:26:29.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:29.967 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:26:29.967 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:26:29.968 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-15T18:26:29.968 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-04-15T18:26:29.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 0 0 2024-04-15T18:26:30.004 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:30.037 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:30.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:30.146 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:30.183 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:30.213 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:30.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:30.287 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:30.320 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:30.356 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:30.561 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:30.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:30.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:30.709 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:30 smithi082 ceph-mon[44206]: pgmap v1100: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s wr, 627 op/s; 14/261627 objects degraded (0.005%); 29125/261627 objects misplaced (11.132%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:26:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:30 smithi082 ceph-mon[44206]: 5.e scrub starts 2024-04-15T18:26:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:30 smithi082 ceph-mon[44206]: 5.b scrub ok 2024-04-15T18:26:30.793 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:30 smithi159 ceph-mon[45336]: pgmap v1100: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s wr, 627 op/s; 14/261627 objects degraded (0.005%); 29125/261627 objects misplaced (11.132%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:26:30.793 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:30 smithi159 ceph-mon[45336]: 5.e scrub starts 2024-04-15T18:26:30.793 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:30 smithi159 ceph-mon[45336]: 5.b scrub ok 2024-04-15T18:26:30.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:30 smithi049 ceph-mon[40087]: pgmap v1100: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s wr, 627 op/s; 14/261627 objects degraded (0.005%); 29125/261627 objects misplaced (11.132%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:26:30.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:30 smithi049 ceph-mon[40087]: 5.e scrub starts 2024-04-15T18:26:30.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:30 smithi049 ceph-mon[40087]: 5.b scrub ok 2024-04-15T18:26:31.518 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:31 smithi159 ceph-mon[45336]: 5.e scrub ok 2024-04-15T18:26:31.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:31 smithi082 ceph-mon[44206]: 5.e scrub ok 2024-04-15T18:26:31.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:31 smithi049 ceph-mon[40087]: 5.e scrub ok 2024-04-15T18:26:32.133 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:26:32.133 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:26048 4.e e294/284 4:7d493ce7:::benchmark_data_smithi159_102124_object26047:head v 294'1546, mlcod=294'1543)", 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 334.46544009600001, 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20374900400000001, 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671275+0000", 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.671341+0000", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5304999999999999e-05 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874017+0000", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.202676935 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874025+0000", 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3660000000000003e-06 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874080+0000", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5652999999999997e-05 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874108+0000", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7107999999999999e-05 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.874990+0000", 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000882594 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:57.875024+0000", 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.341e-05 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25126.0:71501 4.e e315/284 4:774826b2:::benchmark_data_smithi159_102124_object71500:head v 315'4323, mlcod=315'4320)", 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 282.21515772499998, 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.341239132, 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.138 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921557+0000", 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921558+0000", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3e-07 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.921622+0000", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4045000000000001e-05 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260680+0000", 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.339058835 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260689+0000", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3729999999999993e-06 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260771+0000", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2348999999999994e-05 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.260792+0000", 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1185000000000001e-05 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.262759+0000", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019667769999999998 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.262796+0000", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6937999999999998e-05 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43727 5.4 5.e3032fd4 (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.327172+0000", 2024-04-15T18:26:32.142 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.809542355999994, 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.91690400000000005, 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43727 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327172+0000", 2024-04-15T18:26:32.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327172+0000", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327174+0000", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8989999999999999e-06 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327203+0000", 2024-04-15T18:26:32.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9091e-05 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327226+0000", 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2558e-05 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327237+0000", 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0672999999999999e-05 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327308+0000", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1363000000000001e-05 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.327318+0000", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5840000000000001e-06 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.244042+0000", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.91672452500000001 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.244076+0000", 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4307000000000002e-05 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43727 5.4 5:2bf4c0c7:::benchmark_data_smithi159_116570_object43726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:15.259875+0000", 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 76.876839723000003, 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27799535800000003, 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.147 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43727 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259875+0000", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259875+0000", 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259877+0000", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6780000000000001e-06 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259905+0000", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8002000000000001e-05 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259919+0000", 2024-04-15T18:26:32.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4721e-05 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.259930+0000", 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0609e-05 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.261274+0000", 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001344484 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.261285+0000", 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.043e-05 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.525829+0000", 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.264544 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.525955+0000", 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000126215 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,8", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.526082+0000", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000126604 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.528590+0000", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025078990000000001 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.528632+0000", 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2409e-05 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.535827+0000", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.007195402 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.535896+0000", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8623999999999999e-05 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:15.537870+0000", 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001974281 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.153 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:59922 5.0 5:046b50b5:::benchmark_data_smithi159_116570_object59921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e385)", 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:33.581086+0000", 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 58.555629015000001, 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20187013400000001, 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.154 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 59922 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581086+0000", 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581086+0000", 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581087+0000", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.46e-06 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581128+0000", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0643999999999999e-05 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581140+0000", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1824000000000001e-05 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581148+0000", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2460000000000003e-06 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581203+0000", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5041999999999999e-05 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581325+0000", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000121791 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.581467+0000", 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014201900000000001 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.583231+0000", 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017638179999999999 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.583643+0000", 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00041264200000000001 2024-04-15T18:26:32.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.782883+0000", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19924013700000001 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.782916+0000", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2530000000000002e-05 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:33.782956+0000", 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9981000000000002e-05 2024-04-15T18:26:32.159 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:75705 5.a e394/358 5:5a5ff8f8:::benchmark_data_smithi159_116570_object75704:head v 394'4723, mlcod=394'4721)", 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:52.419821+0000", 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.716893796000001, 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41672565499999997, 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419821+0000", 2024-04-15T18:26:32.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419821+0000", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419822+0000", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2580000000000001e-06 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419843+0000", 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1245000000000001e-05 2024-04-15T18:26:32.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.834375+0000", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.414532071 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.834381+0000", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.02e-06 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.834453+0000", 2024-04-15T18:26:32.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1575999999999994e-05 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.834472+0000", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.912e-05 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.836512+0000", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020395270000000002 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.836547+0000", 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4838000000000001e-05 2024-04-15T18:26:32.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:76844 5.0 5:0cf59c53:::benchmark_data_smithi159_116570_object76843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:54.310014+0000", 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.826700586000001, 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33517269500000002, 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 76844 2024-04-15T18:26:32.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310014+0000", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310014+0000", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310015+0000", 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7600000000000006e-07 2024-04-15T18:26:32.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310037+0000", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2359999999999999e-05 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310049+0000", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1715999999999999e-05 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.310057+0000", 2024-04-15T18:26:32.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3969999999999997e-06 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.311616+0000", 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015597930000000001 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.311706+0000", 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0025000000000006e-05 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.311797+0000", 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0816999999999995e-05 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.313522+0000", 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0017250449999999999 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.314644+0000", 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011220749999999999 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.645073+0000", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33042840299999998 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.645138+0000", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4844999999999993e-05 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.645187+0000", 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9243000000000003e-05 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:76885 5.0 5:05ac0d0b:::benchmark_data_smithi159_116570_object76884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:54.642051+0000", 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.494663864000003, 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.38782778000000001, 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 76885 2024-04-15T18:26:32.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642051+0000", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642051+0000", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642052+0000", 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-04-15T18:26:32.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642073+0000", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0818999999999998e-05 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642086+0000", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2846e-05 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642092+0000", 2024-04-15T18:26:32.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4230000000000002e-06 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642151+0000", 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8279999999999998e-05 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642304+0000", 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000153344 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.642472+0000", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016794800000000001 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.645279+0000", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028076680000000001 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.645299+0000", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9366000000000001e-05 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:55.029810+0000", 2024-04-15T18:26:32.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.384511565 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:55.029846+0000", 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5543000000000002e-05 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:55.029879+0000", 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2654000000000003e-05 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:91237 5.0 5:0236c4c2:::benchmark_data_smithi159_116570_object91236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e403)", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:13.015438+0000", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.121277088999999, 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.201525232, 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 91237 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015438+0000", 2024-04-15T18:26:32.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015438+0000", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015439+0000", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.389e-06 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015494+0000", 2024-04-15T18:26:32.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4805000000000001e-05 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015506+0000", 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2591e-05 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.015515+0000", 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3420000000000008e-06 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.018245+0000", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027301550000000002 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.018443+0000", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019802200000000001 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.018609+0000", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016649500000000001 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.021711+0000", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031019469999999999 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.021758+0000", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6551000000000002e-05 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.216840+0000", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19508221100000001 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.216901+0000", 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1029000000000001e-05 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.216963+0000", 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1694999999999998e-05 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "rep_scrubmap(5.0 e358 from shard 4)", 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:13.024061+0000", 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.112653301000002, 2024-04-15T18:26:32.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.225767254, 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.024061+0000", 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.024061+0000", 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.024062+0000", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.024097+0000", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4529000000000001e-05 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.249635+0000", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22553888799999999 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.249643+0000", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4669999999999998e-06 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.249687+0000", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.392e-05 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.249695+0000", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0180000000000001e-06 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:13.249829+0000", 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013377899999999999 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93133 5.0 5:056668e2:::benchmark_data_smithi159_116570_object93132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.446858+0000", 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.689856328000001, 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.83473832400000003, 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.185 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93133 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446858+0000", 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446858+0000", 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446860+0000", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2449999999999999e-06 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446883+0000", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3204999999999999e-05 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446897+0000", 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4498000000000001e-05 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446906+0000", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4689999999999997e-06 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.446964+0000", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7839e-05 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.447088+0000", 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012443999999999999 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.447221+0000", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000133356 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.450268+0000", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030468539999999999 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.450287+0000", 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8785000000000001e-05 2024-04-15T18:26:32.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281477+0000", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.83119015399999996 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281546+0000", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9059000000000004e-05 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281597+0000", 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0420000000000002e-05 2024-04-15T18:26:32.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:93133 5.0 e404/358 ondisk, result = 0)", 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.451132+0000", 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.685582712999999, 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.83047086599999997, 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.451132+0000", 2024-04-15T18:26:32.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.451132+0000", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.451133+0000", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.61e-07 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.451137+0000", 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6059999999999996e-06 2024-04-15T18:26:32.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281386+0000", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.83024864200000004 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281396+0000", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0385e-05 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281459+0000", 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2312999999999998e-05 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281470+0000", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1625e-05 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.281603+0000", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013263399999999999 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93112 5.4 5:26f9a09c:::benchmark_data_smithi159_116570_object93111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:26:32.194 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.704992150000001, 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44799008099999998, 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93112 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431724+0000", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.703e-06 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431750+0000", 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5607000000000001e-05 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431765+0000", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4961e-05 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431776+0000", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1491000000000001e-05 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432594+0000", 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081732599999999995 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432605+0000", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1847e-05 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.869983+0000", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43737783200000002 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.870128+0000", 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014457399999999999 2024-04-15T18:26:32.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.870245+0000", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000117124 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877470+0000", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072249510000000003 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877516+0000", 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6137999999999998e-05 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877527+0000", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0603e-05 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877566+0000", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9755999999999997e-05 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879713+0000", 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021461679999999999 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.200 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93117 5.4 5:2c3e2625:::benchmark_data_smithi159_116570_object93116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.704915361999999, 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45760223, 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93117 2024-04-15T18:26:32.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431800+0000", 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:26:32.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431817+0000", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.768e-05 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431826+0000", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6899999999999998e-06 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431832+0000", 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9429999999999997e-06 2024-04-15T18:26:32.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432666+0000", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083379299999999999 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432675+0000", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0019999999999995e-06 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.874919+0000", 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44224381400000001 2024-04-15T18:26:32.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.875032+0000", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113428 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.875146+0000", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879881+0000", 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047348360000000001 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.885930+0000", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0060485570000000004 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.885992+0000", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1898999999999998e-05 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.886063+0000", 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1125999999999996e-05 2024-04-15T18:26:32.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889402+0000", 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033388329999999998 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93154 5.4 5:2e09de61:::benchmark_data_smithi159_116570_object93153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.704874343, 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45964630499999998, 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.207 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93154 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431841+0000", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3799999999999998e-07 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431859+0000", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8561000000000001e-05 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431867+0000", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2030000000000003e-06 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431871+0000", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9189999999999998e-06 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432711+0000", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083922700000000001 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432716+0000", 2024-04-15T18:26:32.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1889999999999998e-06 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879919+0000", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44720337100000002 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.880100+0000", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018037500000000001 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.880257+0000", 2024-04-15T18:26:32.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015770700000000001 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889476+0000", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092190499999999995 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889494+0000", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.806e-05 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889604+0000", 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109131 2024-04-15T18:26:32.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889667+0000", 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3540999999999994e-05 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.891487+0000", 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018195329999999999 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.213 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93157 5.4 5:26fe9593:::benchmark_data_smithi159_116570_object93156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e406)", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.70483658, 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46360710100000002, 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93157 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431879+0000", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.89e-07 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431898+0000", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9908999999999999e-05 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431906+0000", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431911+0000", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1789999999999999e-06 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432738+0000", 2024-04-15T18:26:32.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00082660600000000002 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432742+0000", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6190000000000002e-06 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883136+0000", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45039331599999999 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883297+0000", 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000161521 2024-04-15T18:26:32.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883486+0000", 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018915499999999999 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889557+0000", 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006070565 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.891498+0000", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001940893 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.894242+0000", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002744607 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.894283+0000", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0658999999999998e-05 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.895485+0000", 2024-04-15T18:26:32.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012021950000000001 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:99204 5.0 5:05660a74:::benchmark_data_smithi159_116570_object6044:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:26.530922+0000", 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.6057928600000002, 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23594352900000001, 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 99204 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530922+0000", 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530922+0000", 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530923+0000", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.451e-06 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530929+0000", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2279999999999998e-06 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530938+0000", 2024-04-15T18:26:32.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9219999999999992e-06 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530947+0000", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0709999999999995e-06 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.530994+0000", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7364999999999998e-05 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.531088+0000", 2024-04-15T18:26:32.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3945999999999998e-05 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.531163+0000", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5451000000000007e-05 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533481+0000", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002317632 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533609+0000", 2024-04-15T18:26:32.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000128429 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766805+0000", 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23319594900000001 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766839+0000", 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4051000000000003e-05 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766865+0000", 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6033999999999998e-05 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:99204 5.0 e409/358 ondisk, result = 0)", 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:26.533657+0000", 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.6030577600000004, 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23321193500000001, 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:26:32.226 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533657+0000", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533657+0000", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533657+0000", 2024-04-15T18:26:32.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.533661+0000", 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0450000000000002e-06 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766752+0000", 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23309032599999999 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766757+0000", 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1939999999999997e-06 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766797+0000", 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9622000000000001e-05 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766802+0000", 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9320000000000004e-06 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766869+0000", 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7319999999999999e-05 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:99273 5.0 5:090fb35e:::benchmark_data_smithi159_116570_object6113:head [delete] snapc 0=[] ondisk+write+known_if_redirected e409)", 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:26.762992+0000", 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.3737229739999997, 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.212526728, 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.230 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 99273 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.762992+0000", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.762992+0000", 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.762993+0000", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1260000000000001e-06 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.762998+0000", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2979999999999999e-06 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.763009+0000", 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1011e-05 2024-04-15T18:26:32.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.763016+0000", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9659999999999999e-06 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.763060+0000", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4187999999999998e-05 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.763152+0000", 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1561e-05 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.763228+0000", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5971999999999997e-05 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766016+0000", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027886059999999999 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766026+0000", 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9790000000000005e-06 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975435+0000", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20940847200000001 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975472+0000", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6940999999999997e-05 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975518+0000", 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6607999999999998e-05 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:99273 5.0 e409/358 ondisk, result = 0)", 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:26.766733+0000", 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.3699817139999997, 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20879213199999999, 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766733+0000", 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766733+0000", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766734+0000", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.766738+0000", 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0539999999999996e-06 2024-04-15T18:26:32.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975402+0000", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20866411000000001 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975405+0000", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3979999999999999e-06 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975428+0000", 2024-04-15T18:26:32.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3113e-05 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975432+0000", 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.456e-06 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:26.975525+0000", 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3330000000000003e-05 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.239 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:26:32.240 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:26:32.240 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:26:32.403 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:32.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:32.510 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:32.521 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:32.523 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:32 smithi159 ceph-mon[45336]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 535 op/s; 14/259281 objects degraded (0.005%); 29125/259281 objects misplaced (11.233%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:26:32.524 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3785499990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.524 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.524 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 14/259281 objects degraded (0.005%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:26:32.544 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:32.620 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:32.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:32.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:32 smithi049 ceph-mon[40087]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 535 op/s; 14/259281 objects degraded (0.005%); 29125/259281 objects misplaced (11.233%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:26:32.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3785499990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 14/259281 objects degraded (0.005%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:32.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:32.755 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:32.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:32 smithi082 ceph-mon[44206]: pgmap v1101: 17 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 535 op/s; 14/259281 objects degraded (0.005%); 29125/259281 objects misplaced (11.233%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:26:32.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3785499990' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:32.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 14/259281 objects degraded (0.005%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:33.100 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:26:33.108 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:33.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:33.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:26:33.263 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:33.295 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:26:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:33.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:26:33.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:33.436 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:33.467 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:33.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:33.474 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:33.571 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:33.602 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:33.671 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:33.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:33.775 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:33 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:26:33.775 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:33.775 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:33 smithi159 ceph-mon[45336]: osdmap e411: 12 total, 12 up, 12 in 2024-04-15T18:26:33.775 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2781260172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:33.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:26:33.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:33 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:26:33.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:33.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:33 smithi049 ceph-mon[40087]: osdmap e411: 12 total, 12 up, 12 in 2024-04-15T18:26:33.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2781260172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:33.817 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:33.841 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:26:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:33.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:26:33.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:34.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:33 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:26:34.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:34.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:33 smithi082 ceph-mon[44206]: osdmap e411: 12 total, 12 up, 12 in 2024-04-15T18:26:34.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2781260172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:34.060 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:34.750 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: pgmap v1103: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 39/251739 objects degraded (0.015%); 28186/251739 objects misplaced (11.197%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:34.751 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:26:34.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:26:34.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:26:34.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:26:34.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: pgmap v1103: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 39/251739 objects degraded (0.015%); 28186/251739 objects misplaced (11.197%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:26:34.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:34.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:34.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:35.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: pgmap v1103: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 39/251739 objects degraded (0.015%); 28186/251739 objects misplaced (11.197%); 16 MiB/s, 250 objects/s recovering 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:35.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:35.407 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:26:35.408 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:26:35.533 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:35.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:35.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:35.680 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:35.728 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:35.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:35.768 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.768 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:35 smithi159 ceph-mon[45336]: osdmap e412: 12 total, 12 up, 12 in 2024-04-15T18:26:35.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:35.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:35 smithi049 ceph-mon[40087]: osdmap e412: 12 total, 12 up, 12 in 2024-04-15T18:26:35.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:35.871 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:36.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:36.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:35 smithi082 ceph-mon[44206]: osdmap e412: 12 total, 12 up, 12 in 2024-04-15T18:26:36.073 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:36.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:36.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:26:36.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:26:36.236 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:26:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:36.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:26:36.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:36.421 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:36.448 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:36.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:36.492 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:36.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:26:36.603 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:36.636 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:26:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:36.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:26:36.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:36.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:36.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: pgmap v1105: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 39/250509 objects degraded (0.016%); 28186/250509 objects misplaced (11.251%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: osdmap e413: 12 total, 12 up, 12 in 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:36.763 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:36.779 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:36.807 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:36.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:36.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: pgmap v1105: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 39/250509 objects degraded (0.016%); 28186/250509 objects misplaced (11.251%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: osdmap e413: 12 total, 12 up, 12 in 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:36.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:36.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:36.956 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:37.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:37.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: pgmap v1105: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 39/250509 objects degraded (0.016%); 28186/250509 objects misplaced (11.251%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: osdmap e413: 12 total, 12 up, 12 in 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:37.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:37.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:26:37.814 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:37.819 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:37 smithi159 ceph-mon[45336]: osdmap e414: 12 total, 12 up, 12 in 2024-04-15T18:26:37.819 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 39/250509 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:37.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:37.921 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:37.953 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:37 smithi049 ceph-mon[40087]: osdmap e414: 12 total, 12 up, 12 in 2024-04-15T18:26:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 39/250509 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:38.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:37 smithi082 ceph-mon[44206]: osdmap e414: 12 total, 12 up, 12 in 2024-04-15T18:26:38.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 39/250509 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:38.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:26:38.458 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:39.259 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 0 (802) 2024-04-15T18:26:39.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:39 smithi082 ceph-mon[44206]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 486 op/s; 39/247218 objects degraded (0.016%); 28186/247218 objects misplaced (11.401%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:26:39.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3964074818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:39 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:39 smithi159 ceph-mon[45336]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 486 op/s; 39/247218 objects degraded (0.016%); 28186/247218 objects misplaced (11.401%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:26:39.631 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3964074818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.631 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:39 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.709 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:39.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:39 smithi049 ceph-mon[40087]: pgmap v1108: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 486 op/s; 39/247218 objects degraded (0.016%); 28186/247218 objects misplaced (11.401%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:26:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3964074818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:39 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-04-15T18:26:39.813 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:39.850 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:39.871 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:39.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:39.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:26:40.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:40.041 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:26:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:40.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:26:40.077 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:26:40.077 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:26:40.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:40.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:40.218 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:40.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:40.268 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:40 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-15T18:26:40.268 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:40 smithi082 ceph-mon[44206]: osdmap e415: 12 total, 12 up, 12 in 2024-04-15T18:26:40.268 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:40 smithi082 ceph-mon[44206]: osdmap e416: 12 total, 12 up, 12 in 2024-04-15T18:26:40.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:40.351 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:40.374 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:40.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:40.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:40 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-15T18:26:40.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:40 smithi049 ceph-mon[40087]: osdmap e415: 12 total, 12 up, 12 in 2024-04-15T18:26:40.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:40 smithi049 ceph-mon[40087]: osdmap e416: 12 total, 12 up, 12 in 2024-04-15T18:26:40.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:26:40.511 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:40.532 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:26:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:40.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:26:40.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:40.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:40 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-04-15T18:26:40.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:40 smithi159 ceph-mon[45336]: osdmap e415: 12 total, 12 up, 12 in 2024-04-15T18:26:40.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:40 smithi159 ceph-mon[45336]: osdmap e416: 12 total, 12 up, 12 in 2024-04-15T18:26:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:40.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:26:41.112 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:26:41.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:41 smithi082 ceph-mon[44206]: pgmap v1110: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 47/240969 objects degraded (0.020%); 27191/240969 objects misplaced (11.284%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:26:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:41 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:41 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:41 smithi082 ceph-mon[44206]: 5.3 scrub starts 2024-04-15T18:26:41.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:41 smithi082 ceph-mon[44206]: osdmap e417: 12 total, 12 up, 12 in 2024-04-15T18:26:41.623 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:41 smithi159 ceph-mon[45336]: pgmap v1110: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 47/240969 objects degraded (0.020%); 27191/240969 objects misplaced (11.284%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:26:41.623 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:41 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:41.623 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:41 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:41.623 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:41 smithi159 ceph-mon[45336]: 5.3 scrub starts 2024-04-15T18:26:41.623 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:41 smithi159 ceph-mon[45336]: osdmap e417: 12 total, 12 up, 12 in 2024-04-15T18:26:41.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:41 smithi049 ceph-mon[40087]: pgmap v1110: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 756 op/s; 47/240969 objects degraded (0.020%); 27191/240969 objects misplaced (11.284%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:26:41.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:41 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:41.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:41 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:41.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:41 smithi049 ceph-mon[40087]: 5.3 scrub starts 2024-04-15T18:26:41.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:41 smithi049 ceph-mon[40087]: osdmap e417: 12 total, 12 up, 12 in 2024-04-15T18:26:41.735 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:41.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:41.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:42.324 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:42.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:42 smithi159 ceph-mon[45336]: 5.5 scrub starts 2024-04-15T18:26:42.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:42 smithi159 ceph-mon[45336]: 5.5 scrub ok 2024-04-15T18:26:42.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/240969 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:42.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:42 smithi159 ceph-mon[45336]: osdmap e418: 12 total, 12 up, 12 in 2024-04-15T18:26:42.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:42 smithi082 ceph-mon[44206]: 5.5 scrub starts 2024-04-15T18:26:42.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:42 smithi082 ceph-mon[44206]: 5.5 scrub ok 2024-04-15T18:26:42.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/240969 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:42.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:42 smithi082 ceph-mon[44206]: osdmap e418: 12 total, 12 up, 12 in 2024-04-15T18:26:42.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:42 smithi049 ceph-mon[40087]: 5.5 scrub starts 2024-04-15T18:26:42.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:42 smithi049 ceph-mon[40087]: 5.5 scrub ok 2024-04-15T18:26:42.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/240969 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:42.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:42 smithi049 ceph-mon[40087]: osdmap e418: 12 total, 12 up, 12 in 2024-04-15T18:26:42.960 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:26:42.960 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:26:43.521 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:43.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:43 smithi082 ceph-mon[44206]: pgmap v1113: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 47/238590 objects degraded (0.020%); 27191/238590 objects misplaced (11.397%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:26:43.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:43 smithi082 ceph-mon[44206]: 5.3 scrub ok 2024-04-15T18:26:43.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:43 smithi082 ceph-mon[44206]: 5.2 scrub starts 2024-04-15T18:26:43.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:43.638 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:43 smithi159 ceph-mon[45336]: pgmap v1113: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 47/238590 objects degraded (0.020%); 27191/238590 objects misplaced (11.397%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:26:43.638 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:43 smithi159 ceph-mon[45336]: 5.3 scrub ok 2024-04-15T18:26:43.639 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:43 smithi159 ceph-mon[45336]: 5.2 scrub starts 2024-04-15T18:26:43.655 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:43.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:43 smithi049 ceph-mon[40087]: pgmap v1113: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 47/238590 objects degraded (0.020%); 27191/238590 objects misplaced (11.397%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:26:43.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:43 smithi049 ceph-mon[40087]: 5.3 scrub ok 2024-04-15T18:26:43.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:43 smithi049 ceph-mon[40087]: 5.2 scrub starts 2024-04-15T18:26:43.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:26:43.674 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:43.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:43.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:43.806 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.057 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:44.101 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:44.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:26:44.188 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.208 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:26:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:26:44.283 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:26:44.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:26:44.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:26:44.342 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.364 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:26:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:26:44.468 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:44.495 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.514 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:44.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:26:44.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:26:44.646 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:26:44.667 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:26:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:26:44.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:26:44.798 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.816 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:26:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:26:44.864 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:26:44.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:26:44.911 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:26:44.921 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:44.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:26:44.946 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:44.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:26:44.970 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:44.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:44.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:26:45.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:26:45.035 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:26:45.058 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:26:45.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:45.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:26:45.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: pgmap v1115: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 382 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: 5.2 scrub ok 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: osdmap e419: 12 total, 12 up, 12 in 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3750304558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:45.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:45.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: pgmap v1115: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 382 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: 5.2 scrub ok 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: osdmap e419: 12 total, 12 up, 12 in 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3750304558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:45.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:45.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: pgmap v1115: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 382 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:26:45.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: 5.2 scrub ok 2024-04-15T18:26:45.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:45.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3843550685' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:26:45.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: osdmap e419: 12 total, 12 up, 12 in 2024-04-15T18:26:45.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3750304558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:45.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:45.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.054 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:46.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:46.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:46.189 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:46.289 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:46.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:46.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:26:46.425 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:46.448 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:26:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:26:46.552 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.579 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:46.598 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:46.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:46.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:46.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:26:46.758 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:26:46.758 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-15T18:26:46.758 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:26:46.758 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:26:46.759 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:46.885 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:46.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:46.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:47.016 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:47.128 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:26:47.465 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:47.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:47.570 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:26:47.596 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:47 smithi159 ceph-mon[45336]: pgmap v1117: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:26:47.596 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/236322 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:47.596 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:47 smithi159 ceph-mon[45336]: osdmap e420: 12 total, 12 up, 12 in 2024-04-15T18:26:47.598 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:47.619 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:26:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:47.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:26:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:47 smithi049 ceph-mon[40087]: pgmap v1117: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:26:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/236322 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:47 smithi049 ceph-mon[40087]: osdmap e420: 12 total, 12 up, 12 in 2024-04-15T18:26:47.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:47.759 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:47 smithi082 ceph-mon[44206]: pgmap v1117: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 47/236322 objects degraded (0.020%); 26235/236322 objects misplaced (11.101%); 12 MiB/s, 189 objects/s recovering 2024-04-15T18:26:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/236322 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:47 smithi082 ceph-mon[44206]: osdmap e420: 12 total, 12 up, 12 in 2024-04-15T18:26:47.783 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:47.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:47.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:26:47.918 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:47.993 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:26:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:47.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:26:48.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:26:48.041 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:26:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:26:48.041 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:26:48.041 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:26:48.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:48.126 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:48.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:48 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:48.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:48 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:48.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:48 smithi082 ceph-mon[44206]: pgmap v1119: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 47/236310 objects degraded (0.020%); 26235/236310 objects misplaced (11.102%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:48.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:48 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:26:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:48 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:48.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:48 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:48.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:48 smithi159 ceph-mon[45336]: pgmap v1119: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 47/236310 objects degraded (0.020%); 26235/236310 objects misplaced (11.102%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:48.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:48 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:26:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:48 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:48 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:48 smithi049 ceph-mon[40087]: pgmap v1119: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 47/236310 objects degraded (0.020%); 26235/236310 objects misplaced (11.102%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:26:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:48 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:26:49.180 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:49.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:26:49.490 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:26:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:26:49.494 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:26:50.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:50 smithi159 ceph-mon[45336]: pgmap v1120: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 47/236310 objects degraded (0.020%); 25289/236310 objects misplaced (10.702%); 17 MiB/s, 269 objects/s recovering 2024-04-15T18:26:50.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:50 smithi049 ceph-mon[40087]: pgmap v1120: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 47/236310 objects degraded (0.020%); 25289/236310 objects misplaced (10.702%); 17 MiB/s, 269 objects/s recovering 2024-04-15T18:26:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:50 smithi082 ceph-mon[44206]: pgmap v1120: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 47/236310 objects degraded (0.020%); 25289/236310 objects misplaced (10.702%); 17 MiB/s, 269 objects/s recovering 2024-04-15T18:26:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:51.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:51.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:51.664 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:51.664 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:51.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:51.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:51.831 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:52.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:52 smithi159 ceph-mon[45336]: pgmap v1121: 17 pgs: 1 active+recovering+remapped, 1 active+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 12287/236076 objects degraded (5.205%); 12875/236076 objects misplaced (5.454%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:26:52.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:52 smithi159 ceph-mon[45336]: osdmap e421: 12 total, 12 up, 12 in 2024-04-15T18:26:52.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12287/236076 objects degraded (5.205%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:26:52.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:52 smithi159 ceph-mon[45336]: osdmap e422: 12 total, 12 up, 12 in 2024-04-15T18:26:52.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:52 smithi049 ceph-mon[40087]: pgmap v1121: 17 pgs: 1 active+recovering+remapped, 1 active+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 12287/236076 objects degraded (5.205%); 12875/236076 objects misplaced (5.454%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:26:52.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:52 smithi049 ceph-mon[40087]: osdmap e421: 12 total, 12 up, 12 in 2024-04-15T18:26:52.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12287/236076 objects degraded (5.205%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:26:52.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:52 smithi049 ceph-mon[40087]: osdmap e422: 12 total, 12 up, 12 in 2024-04-15T18:26:52.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:52 smithi082 ceph-mon[44206]: pgmap v1121: 17 pgs: 1 active+recovering+remapped, 1 active+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 12287/236076 objects degraded (5.205%); 12875/236076 objects misplaced (5.454%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:26:52.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:52 smithi082 ceph-mon[44206]: osdmap e421: 12 total, 12 up, 12 in 2024-04-15T18:26:52.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12287/236076 objects degraded (5.205%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:26:52.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:52 smithi082 ceph-mon[44206]: osdmap e422: 12 total, 12 up, 12 in 2024-04-15T18:26:52.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:26:54.111 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:54.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:54 smithi049 ceph-mon[40087]: pgmap v1124: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+degraded, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 12287/235341 objects degraded (5.221%); 11990/235341 objects misplaced (5.095%); 19 MiB/s, 311 objects/s recovering 2024-04-15T18:26:54.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2206127232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:54 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:54 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.372 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:26:54.616 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:54.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:54 smithi082 ceph-mon[44206]: pgmap v1124: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+degraded, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 12287/235341 objects degraded (5.221%); 11990/235341 objects misplaced (5.095%); 19 MiB/s, 311 objects/s recovering 2024-04-15T18:26:54.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2206127232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:54 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.618 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:54 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:54.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:54.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:54 smithi159 ceph-mon[45336]: pgmap v1124: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+degraded, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 12287/235341 objects degraded (5.221%); 11990/235341 objects misplaced (5.095%); 19 MiB/s, 311 objects/s recovering 2024-04-15T18:26:54.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2206127232' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:54 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:26:54.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:54 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:54.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:26:54.750 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:55 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:55.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:55 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:55.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:55 smithi159 ceph-mon[45336]: osdmap e423: 12 total, 12 up, 12 in 2024-04-15T18:26:55.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:55 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:55.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:55 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:55.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:55 smithi049 ceph-mon[40087]: osdmap e423: 12 total, 12 up, 12 in 2024-04-15T18:26:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:55 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:26:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:55 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:26:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:55 smithi082 ceph-mon[44206]: osdmap e423: 12 total, 12 up, 12 in 2024-04-15T18:26:56.402 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:26:56.403 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:26:56.403 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:26:56.403 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:26:56.403 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:26:56.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:56 smithi159 ceph-mon[45336]: pgmap v1126: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 47/235041 objects degraded (0.020%); 24124/235041 objects misplaced (10.264%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:26:56.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:26:56.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:26:56.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:56 smithi049 ceph-mon[40087]: pgmap v1126: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 47/235041 objects degraded (0.020%); 24124/235041 objects misplaced (10.264%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:26:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:26:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:26:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:56.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:56 smithi082 ceph-mon[44206]: pgmap v1126: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 70 op/s; 47/235041 objects degraded (0.020%); 24124/235041 objects misplaced (10.264%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:26:56.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:26:56.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:26:56.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:26:56.900 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:26:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:56.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:26:57.005 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:26:57.036 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:57.058 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:26:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:57.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:26:57.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:26:57.200 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:57.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:26:57.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/235041 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:57.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:57 smithi159 ceph-mon[45336]: osdmap e424: 12 total, 12 up, 12 in 2024-04-15T18:26:57.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/235041 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:57.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:57 smithi049 ceph-mon[40087]: osdmap e424: 12 total, 12 up, 12 in 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:26:57.500+0000 7fbd8eaf1700 0 --2- 172.21.15.49:0/1910194390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd88102370 0x7fbd8810e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:26:57.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:26:57.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:26:57.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/235041 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:26:57.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:57 smithi082 ceph-mon[44206]: osdmap e424: 12 total, 12 up, 12 in 2024-04-15T18:26:57.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:26:57.751 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:26:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:57.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:26:57.765 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:26:57.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:26:57.814 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:26:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:26:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:26:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:26:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:26:57.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:26:57.885 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:57.938 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:26:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:57.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:26:58.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:26:58.071 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:58.092 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:26:58.093 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:26:58.093 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:26:58.093 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:26:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:58.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:26:58.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:26:58.224 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:58.243 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:26:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:58.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:26:58.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: pgmap v1128: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/234489 objects degraded (0.020%); 24124/234489 objects misplaced (10.288%); 9.6 MiB/s, 154 objects/s recovering 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/476435786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:58.376 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:58.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:26:58.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:26:58.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.377 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.378 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:58.402 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:26:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:58.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:26:58.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:26:58.538 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:58.560 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:26:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:58.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:26:58.605 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: pgmap v1128: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/234489 objects degraded (0.020%); 24124/234489 objects misplaced (10.288%); 9.6 MiB/s, 154 objects/s recovering 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/476435786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:26:58.692 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:58.718 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:26:58.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: pgmap v1128: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/234489 objects degraded (0.020%); 24124/234489 objects misplaced (10.288%); 9.6 MiB/s, 154 objects/s recovering 2024-04-15T18:26:58.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/476435786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:26:58.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:26:58.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:26:58.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:26:58.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:26:58.851 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:26:59.466 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:26:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:26:59 smithi159 ceph-mon[45336]: osdmap e425: 12 total, 12 up, 12 in 2024-04-15T18:26:59.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:26:59 smithi049 ceph-mon[40087]: osdmap e425: 12 total, 12 up, 12 in 2024-04-15T18:26:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:26:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:26:59 smithi082 ceph-mon[44206]: osdmap e425: 12 total, 12 up, 12 in 2024-04-15T18:27:00.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:27:00.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:00 smithi049 ceph-mon[40087]: pgmap v1130: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 86 op/s; 47/233775 objects degraded (0.020%); 23222/233775 objects misplaced (9.933%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:27:00.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:00 smithi159 ceph-mon[45336]: pgmap v1130: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 86 op/s; 47/233775 objects degraded (0.020%); 23222/233775 objects misplaced (9.933%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:27:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:00 smithi082 ceph-mon[44206]: pgmap v1130: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 86 op/s; 47/233775 objects degraded (0.020%); 23222/233775 objects misplaced (9.933%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:27:01.861 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:27:01.861 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:27:02.521 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:02.638 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:02 smithi082 ceph-mon[44206]: pgmap v1131: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 47/233187 objects degraded (0.020%); 23120/233187 objects misplaced (9.915%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:27:02.638 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/233187 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:02.638 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:02 smithi082 ceph-mon[44206]: osdmap e426: 12 total, 12 up, 12 in 2024-04-15T18:27:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:02 smithi159 ceph-mon[45336]: pgmap v1131: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 47/233187 objects degraded (0.020%); 23120/233187 objects misplaced (9.915%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:27:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/233187 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:02 smithi159 ceph-mon[45336]: osdmap e426: 12 total, 12 up, 12 in 2024-04-15T18:27:02.712 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:02.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:02.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:02 smithi049 ceph-mon[40087]: pgmap v1131: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 47/233187 objects degraded (0.020%); 23120/233187 objects misplaced (9.915%); 8.7 MiB/s, 139 objects/s recovering 2024-04-15T18:27:02.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/233187 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:02.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:02 smithi049 ceph-mon[40087]: osdmap e426: 12 total, 12 up, 12 in 2024-04-15T18:27:02.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:02.845 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:02.866 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:02.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:02.970 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:02.998 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:03.024 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:03.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:03.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:03.162 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:03.400 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:27:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3436659329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:03.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:03 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:03.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3436659329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:03.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:03 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:03.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3436659329' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:03.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:03 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:04.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:27:04.514 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:04.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:04 smithi049 ceph-mon[40087]: pgmap v1133: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 47/232254 objects degraded (0.020%); 22126/232254 objects misplaced (9.527%); 20 MiB/s, 318 objects/s recovering 2024-04-15T18:27:04.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:04 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:04.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:04 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:04.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:04 smithi049 ceph-mon[40087]: osdmap e427: 12 total, 12 up, 12 in 2024-04-15T18:27:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:04.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:04.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:04.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:04 smithi159 ceph-mon[45336]: pgmap v1133: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 47/232254 objects degraded (0.020%); 22126/232254 objects misplaced (9.527%); 20 MiB/s, 318 objects/s recovering 2024-04-15T18:27:04.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:04 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:04.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:04 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:04.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:04 smithi159 ceph-mon[45336]: osdmap e427: 12 total, 12 up, 12 in 2024-04-15T18:27:04.647 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:04.669 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:04.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:04.773 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:04.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:04 smithi082 ceph-mon[44206]: pgmap v1133: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 47/232254 objects degraded (0.020%); 22126/232254 objects misplaced (9.527%); 20 MiB/s, 318 objects/s recovering 2024-04-15T18:27:04.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:04 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:04.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:04 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:04.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:04 smithi082 ceph-mon[44206]: osdmap e427: 12 total, 12 up, 12 in 2024-04-15T18:27:04.804 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:04.826 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:04.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:04.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:04.956 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:04.978 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:04.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:05.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:05.109 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:05.659 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:05.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:05.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:05.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:05.824 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:05.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:05.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:05.951 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:05.958 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:05.980 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:05.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:05.991 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:27:06.018 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:27:06.040 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:27:06.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:27:06.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:06.093 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:27:06.110 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:27:06.131 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:06.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:06.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:27:06.175 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:27:06.201 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:27:06.232 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:27:06.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:06.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:27:06.269 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:06.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:27:06.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:27:06.289 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:06.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:06.396 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:27:06.423 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:06.446 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:27:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:06.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:27:06.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:06 smithi049 ceph-mon[40087]: pgmap v1135: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/231927 objects degraded (0.020%); 22018/231927 objects misplaced (9.494%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:27:06.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2750645369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:06.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:06.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:06.583 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:06.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:06 smithi159 ceph-mon[45336]: pgmap v1135: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/231927 objects degraded (0.020%); 22018/231927 objects misplaced (9.494%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:27:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2750645369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:06.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:06.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:06 smithi082 ceph-mon[44206]: pgmap v1135: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 88 op/s; 47/231927 objects degraded (0.020%); 22018/231927 objects misplaced (9.494%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:27:06.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2750645369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:06.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.435 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:07.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:07.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:27:07.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:07.570 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:07.590 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:07.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:07.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/231927 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:07.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:07 smithi049 ceph-mon[40087]: osdmap e428: 12 total, 12 up, 12 in 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.725 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/231927 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:07.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:07 smithi082 ceph-mon[44206]: osdmap e428: 12 total, 12 up, 12 in 2024-04-15T18:27:07.726 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:07.746 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:07.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:07.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:07.881 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/231927 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:07.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:07 smithi159 ceph-mon[45336]: osdmap e428: 12 total, 12 up, 12 in 2024-04-15T18:27:07.901 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:07.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:08.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:08.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:08.037 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:27:08.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 0 1 2024-04-15T18:27:08.054 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:08.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:08.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:08.185 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:08.204 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:08.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:08.239 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:08.308 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:08.335 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:08.354 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:08.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:08.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:08.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:08.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:08.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 428/405 [PushOp(5:66ebf55a:::benchmark_data_smithi159_116570_object45641:head, version: 380'2838, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:66ebf55a:::benchmark_data_smithi159_116570_object45641:head@380'2838, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:08.539826+0000", 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00064497299999999997, 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00069723300000000001, 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:08.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539826+0000", 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539826+0000", 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539827+0000", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.426e-06 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539858+0000", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.046e-05 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539887+0000", 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9292e-05 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539898+0000", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0882e-05 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539977+0000", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8646000000000006e-05 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.539987+0000", 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9760000000000007e-06 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:08.537 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.4 428/405 [PushOp(5:2bdb99a3:::benchmark_data_smithi159_116570_object45091:head, version: 379'2833, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:2bdb99a3:::benchmark_data_smithi159_116570_object45091:head@379'2833, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:08.540470+0000", 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.3629999999999999e-06, 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000146212, 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540470+0000", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540470+0000", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540470+0000", 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.06e-07 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540553+0000", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2465000000000004e-05 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540575+0000", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2739999999999999e-05 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:08.540584+0000", 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0210000000000005e-06 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:08.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:08.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:08.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:08.541 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:27:08.541 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-04-15T18:27:08.541 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:27:08.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:08.643 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:08.665 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:08.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:08.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:08.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:09.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:27:09.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:09 smithi049 ceph-mon[40087]: pgmap v1137: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 47/231441 objects degraded (0.020%); 22018/231441 objects misplaced (9.513%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:27:09.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:27:09.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:09 smithi082 ceph-mon[44206]: pgmap v1137: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 47/231441 objects degraded (0.020%); 22018/231441 objects misplaced (9.513%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:27:09.553 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:09.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:09.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:09 smithi159 ceph-mon[45336]: pgmap v1137: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 47/231441 objects degraded (0.020%); 22018/231441 objects misplaced (9.513%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:27:09.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:09.694 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:09.723 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:09.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:09.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:09.858 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:09.877 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:09.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:09.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:10.011 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:10.041 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:10.041 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:10.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:10.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:27:10.174 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:10.194 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:27:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:10.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:27:10.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:10.334 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:11.387 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:11.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:11.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:11.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:11 smithi049 ceph-mon[40087]: pgmap v1138: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 47/230421 objects degraded (0.020%); 21058/230421 objects misplaced (9.139%); 18 MiB/s, 282 objects/s recovering 2024-04-15T18:27:11.528 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:11.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:11 smithi082 ceph-mon[44206]: pgmap v1138: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 47/230421 objects degraded (0.020%); 21058/230421 objects misplaced (9.139%); 18 MiB/s, 282 objects/s recovering 2024-04-15T18:27:11.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:11 smithi159 ceph-mon[45336]: pgmap v1138: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 130 op/s; 47/230421 objects degraded (0.020%); 21058/230421 objects misplaced (9.139%); 18 MiB/s, 282 objects/s recovering 2024-04-15T18:27:11.781 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:11.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:11.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:11.912 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:11.932 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:11.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:12.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:12.066 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:12.193 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:12.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:12.263 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/230421 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:12.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:12.328 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:12.348 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:12.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:12.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/230421 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:12.483 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/230421 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:12.626 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:12.640 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:12.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:12.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:12.772 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:12.794 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:12.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:12.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:12.927 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:12.968 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:12.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:13.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:13.106 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:13.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:13.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:13.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:13.390 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:13.411 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:13.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:13.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:13 smithi049 ceph-mon[40087]: pgmap v1139: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 105 op/s; 47/229725 objects degraded (0.020%); 20950/229725 objects misplaced (9.120%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:27:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:13.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:13 smithi082 ceph-mon[44206]: pgmap v1139: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 105 op/s; 47/229725 objects degraded (0.020%); 20950/229725 objects misplaced (9.120%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:27:13.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:13.548 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:13 smithi159 ceph-mon[45336]: pgmap v1139: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 105 op/s; 47/229725 objects degraded (0.020%); 20950/229725 objects misplaced (9.120%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:27:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:14.204 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:14.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:14.273 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-04-15T18:27:14.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:14.336 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:14.357 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:14.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:14.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:14 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:14 smithi049 ceph-mon[40087]: osdmap e429: 12 total, 12 up, 12 in 2024-04-15T18:27:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:27:14.488 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:14.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:14.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:14 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:14 smithi082 ceph-mon[44206]: osdmap e429: 12 total, 12 up, 12 in 2024-04-15T18:27:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:27:14.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:14.641 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:14 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3527567384' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:14 smithi159 ceph-mon[45336]: osdmap e429: 12 total, 12 up, 12 in 2024-04-15T18:27:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:27:14.662 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:14.663 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:27:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:14.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:14.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:14.797 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:14.872 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:14.872 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:14.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:14.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:15.008 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:15.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:15.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:15.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:15.165 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:15.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:15 smithi049 ceph-mon[40087]: pgmap v1141: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 146 op/s; 47/228405 objects degraded (0.021%); 19999/228405 objects misplaced (8.756%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:27:15.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:27:15.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:15 smithi049 ceph-mon[40087]: osdmap e430: 12 total, 12 up, 12 in 2024-04-15T18:27:15.515 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:15.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:15.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:15 smithi082 ceph-mon[44206]: pgmap v1141: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 146 op/s; 47/228405 objects degraded (0.021%); 19999/228405 objects misplaced (8.756%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:27:15.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:27:15.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:15 smithi082 ceph-mon[44206]: osdmap e430: 12 total, 12 up, 12 in 2024-04-15T18:27:15.620 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:15.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:15 smithi159 ceph-mon[45336]: pgmap v1141: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 146 op/s; 47/228405 objects degraded (0.021%); 19999/228405 objects misplaced (8.756%); 15 MiB/s, 243 objects/s recovering 2024-04-15T18:27:15.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/773933880' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:27:15.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:15 smithi159 ceph-mon[45336]: osdmap e430: 12 total, 12 up, 12 in 2024-04-15T18:27:15.647 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:15.668 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:15.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:15.772 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:15.804 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:15.824 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:15.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:15.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:15.959 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:16.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:16.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:16.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:16 smithi049 ceph-mon[40087]: pgmap v1143: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 47/228093 objects degraded (0.021%); 19888/228093 objects misplaced (8.719%); 15 MiB/s, 247 objects/s recovering 2024-04-15T18:27:16.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:16 smithi049 ceph-mon[40087]: osdmap e431: 12 total, 12 up, 12 in 2024-04-15T18:27:16.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 1 1 2024-04-15T18:27:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:16 smithi159 ceph-mon[45336]: pgmap v1143: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 47/228093 objects degraded (0.021%); 19888/228093 objects misplaced (8.719%); 15 MiB/s, 247 objects/s recovering 2024-04-15T18:27:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:16 smithi159 ceph-mon[45336]: osdmap e431: 12 total, 12 up, 12 in 2024-04-15T18:27:16.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:16.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:16.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:16 smithi082 ceph-mon[44206]: pgmap v1143: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 47/228093 objects degraded (0.021%); 19888/228093 objects misplaced (8.719%); 15 MiB/s, 247 objects/s recovering 2024-04-15T18:27:16.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:16 smithi082 ceph-mon[44206]: osdmap e431: 12 total, 12 up, 12 in 2024-04-15T18:27:16.809 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:16.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:16.919 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:16.946 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:16.969 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:16.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:17.072 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:17.101 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.122 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:17.228 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:17.255 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.283 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:17.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2312365183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/228093 objects degraded (0.021%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:17 smithi159 ceph-mon[45336]: osdmap e432: 12 total, 12 up, 12 in 2024-04-15T18:27:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:27:17.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:27:17.464 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:27:17.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:27:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:17.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:27:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:17.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:27:17.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2312365183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:17.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/228093 objects degraded (0.021%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:17.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:17 smithi082 ceph-mon[44206]: osdmap e432: 12 total, 12 up, 12 in 2024-04-15T18:27:17.540 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:27:17.568 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:27:17.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:17.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.612 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:27:17.631 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:17.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:27:17.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:27:17.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:27:17.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2312365183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:17.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/228093 objects degraded (0.021%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:27:17.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:17 smithi049 ceph-mon[40087]: osdmap e432: 12 total, 12 up, 12 in 2024-04-15T18:27:17.735 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:17.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:27:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:27:17.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:27:17.764 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.787 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:17.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:17.923 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:17.977 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:17.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:18.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:18.113 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:18.215 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:18.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:18.322 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:18.351 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:18.371 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:18.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: pgmap v1146: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/227403 objects degraded (0.021%); 19888/227403 objects misplaced (8.746%); 539 KiB/s, 8 objects/s recovering 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:18.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: pgmap v1146: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/227403 objects degraded (0.021%); 19888/227403 objects misplaced (8.746%); 539 KiB/s, 8 objects/s recovering 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: pgmap v1146: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/227403 objects degraded (0.021%); 19888/227403 objects misplaced (8.746%); 539 KiB/s, 8 objects/s recovering 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:18.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:19.858 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:19.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:19.919 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:19.963 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:19.992 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:20.011 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:20.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:20.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:20.143 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:20.162 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:20.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:20.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:20.296 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:20.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:20 smithi159 ceph-mon[45336]: pgmap v1147: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/226914 objects degraded (0.021%); 18423/226914 objects misplaced (8.119%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:27:20.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:20 smithi049 ceph-mon[40087]: pgmap v1147: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/226914 objects degraded (0.021%); 18423/226914 objects misplaced (8.119%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:27:20.747 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:20.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:20.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:20 smithi082 ceph-mon[44206]: pgmap v1147: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 82 op/s; 47/226914 objects degraded (0.021%); 18423/226914 objects misplaced (8.119%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:27:20.854 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:20.880 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:20.901 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:20.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:21.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:21.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:27:21.069 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:21.173 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:21.200 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.219 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:21.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:21.355 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.427 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:21.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:21.551 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:27:21.552 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:27:21.558 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.578 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:21.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:21.709 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:21.836 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:21.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:21.887 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:21.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:21.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:22.020 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:22.273 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:22.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:22.341 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-04-15T18:27:22.376 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:27:22.404 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:22.424 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:27:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:22.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:27:22.528 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:22.554 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:22.574 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:22.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:22.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:22 smithi159 ceph-mon[45336]: pgmap v1148: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 47/226281 objects degraded (0.021%); 18317/226281 objects misplaced (8.095%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:27:22.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:27:22.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/226281 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:22.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:22.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:22 smithi049 ceph-mon[40087]: pgmap v1148: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 47/226281 objects degraded (0.021%); 18317/226281 objects misplaced (8.095%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:27:22.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:27:22.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/226281 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:22.706 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:22.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:22 smithi082 ceph-mon[44206]: pgmap v1148: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 47/226281 objects degraded (0.021%); 18317/226281 objects misplaced (8.095%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:27:22.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:27:22.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/226281 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:23.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 4 1 2024-04-15T18:27:23.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:27:23.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:23 smithi049 ceph-mon[40087]: osdmap e433: 12 total, 12 up, 12 in 2024-04-15T18:27:23.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:27:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:27:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:23 smithi159 ceph-mon[45336]: osdmap e433: 12 total, 12 up, 12 in 2024-04-15T18:27:23.656 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:23.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:23.761 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:23.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1121185530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:27:23.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:23 smithi082 ceph-mon[44206]: osdmap e433: 12 total, 12 up, 12 in 2024-04-15T18:27:23.792 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:23.820 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:23.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:23.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:23.958 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:23.984 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:23.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:24.039 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:24.059 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:24.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:24.115 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:24.136 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:24.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:24.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:24.269 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:24.325 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:24.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:24.432 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:24.459 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:24.481 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:24.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:24.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:24 smithi082 ceph-mon[44206]: pgmap v1150: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/225021 objects degraded (0.021%); 16994/225021 objects misplaced (7.552%); 22 MiB/s, 353 objects/s recovering 2024-04-15T18:27:24.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:24 smithi049 ceph-mon[40087]: pgmap v1150: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/225021 objects degraded (0.021%); 16994/225021 objects misplaced (7.552%); 22 MiB/s, 353 objects/s recovering 2024-04-15T18:27:24.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:24.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:24.633 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:24.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:24.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:24 smithi159 ceph-mon[45336]: pgmap v1150: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/225021 objects degraded (0.021%); 16994/225021 objects misplaced (7.552%); 22 MiB/s, 353 objects/s recovering 2024-04-15T18:27:24.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:24.766 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:24.788 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:24.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:24.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:24.927 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:25.388 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:26 smithi159 ceph-mon[45336]: pgmap v1151: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/224604 objects degraded (0.021%); 16884/224604 objects misplaced (7.517%); 22 MiB/s, 355 objects/s recovering 2024-04-15T18:27:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:26 smithi159 ceph-mon[45336]: 5.c scrub starts 2024-04-15T18:27:26.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:26 smithi049 ceph-mon[40087]: pgmap v1151: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/224604 objects degraded (0.021%); 16884/224604 objects misplaced (7.517%); 22 MiB/s, 355 objects/s recovering 2024-04-15T18:27:26.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:26 smithi049 ceph-mon[40087]: 5.c scrub starts 2024-04-15T18:27:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:26 smithi082 ceph-mon[44206]: pgmap v1151: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 47/224604 objects degraded (0.021%); 16884/224604 objects misplaced (7.517%); 22 MiB/s, 355 objects/s recovering 2024-04-15T18:27:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:26 smithi082 ceph-mon[44206]: 5.c scrub starts 2024-04-15T18:27:27.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/224604 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:27.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:27 smithi159 ceph-mon[45336]: osdmap e434: 12 total, 12 up, 12 in 2024-04-15T18:27:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/224604 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:27 smithi049 ceph-mon[40087]: osdmap e434: 12 total, 12 up, 12 in 2024-04-15T18:27:27.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/224604 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:27.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:27 smithi082 ceph-mon[44206]: osdmap e434: 12 total, 12 up, 12 in 2024-04-15T18:27:28.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:28 smithi159 ceph-mon[45336]: 5.c scrub ok 2024-04-15T18:27:28.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:28 smithi159 ceph-mon[45336]: pgmap v1153: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 47/223545 objects degraded (0.021%); 16884/223545 objects misplaced (7.553%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:27:28.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:28 smithi049 ceph-mon[40087]: 5.c scrub ok 2024-04-15T18:27:28.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:28 smithi049 ceph-mon[40087]: pgmap v1153: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 47/223545 objects degraded (0.021%); 16884/223545 objects misplaced (7.553%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:27:28.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:28 smithi082 ceph-mon[44206]: 5.c scrub ok 2024-04-15T18:27:28.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:28 smithi082 ceph-mon[44206]: pgmap v1153: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 47/223545 objects degraded (0.021%); 16884/223545 objects misplaced (7.553%); 11 MiB/s, 174 objects/s recovering 2024-04-15T18:27:29.360 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:27:29.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:27:29.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:27:29.776 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:29.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:29.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:27:29.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:29.910 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:29.931 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:29.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:30.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:30.064 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:30.089 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:30.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:30.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:30.227 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:30.364 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-04-15T18:27:30.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: pgmap v1154: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 47/222606 objects degraded (0.021%); 15441/222606 objects misplaced (6.936%); 22 MiB/s, 350 objects/s recovering 2024-04-15T18:27:30.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:30.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:27:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: osdmap e435: 12 total, 12 up, 12 in 2024-04-15T18:27:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3197833821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:30.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:30 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: pgmap v1154: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 47/222606 objects degraded (0.021%); 15441/222606 objects misplaced (6.936%); 22 MiB/s, 350 objects/s recovering 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: osdmap e435: 12 total, 12 up, 12 in 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3197833821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:30 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: pgmap v1154: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 47/222606 objects degraded (0.021%); 15441/222606 objects misplaced (6.936%); 22 MiB/s, 350 objects/s recovering 2024-04-15T18:27:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/853049203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:27:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: osdmap e435: 12 total, 12 up, 12 in 2024-04-15T18:27:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3197833821' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:27:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:30.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:30 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:31.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 5 1 2024-04-15T18:27:31.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:31 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:27:31.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:31 smithi049 ceph-mon[40087]: osdmap e436: 12 total, 12 up, 12 in 2024-04-15T18:27:31.585 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:31.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:31.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:31 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:27:31.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:31 smithi159 ceph-mon[45336]: osdmap e436: 12 total, 12 up, 12 in 2024-04-15T18:27:31.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:31.714 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:31 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:27:31.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:31 smithi082 ceph-mon[44206]: osdmap e436: 12 total, 12 up, 12 in 2024-04-15T18:27:31.717 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:31.736 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:31.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:31.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:31.870 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:31.889 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:31.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:31.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:32.029 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:32.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:27:32.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:27:32.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:27:32.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:27:32.361 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:27:32.392 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:27:32.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:27:32.447 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93111 5.6 5.6002d106 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:27:32.461 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.855259016999995, 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79303184699999996, 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93111 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609128+0000", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.81e-06 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609258+0000", 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012971 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609273+0000", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4647999999999999e-05 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609284+0000", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1462e-05 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609344+0000", 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0343000000000001e-05 2024-04-15T18:27:32.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609361+0000", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6680999999999998e-05 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402147+0000", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79278600300000002 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402158+0000", 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.119e-05 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.465 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93119 5.6 5.5151b6f6 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.85507681, 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79293819799999998, 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93119 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3000000000000001e-07 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609425+0000", 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011591600000000001 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609436+0000", 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0509999999999999e-05 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609443+0000", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9890000000000003e-06 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609482+0000", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9857999999999997e-05 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609489+0000", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.455e-06 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402240+0000", 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79275137100000004 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402247+0000", 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5690000000000001e-06 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93121 5.a 5.a03eb09a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854928993000001, 2024-04-15T18:27:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79295962099999995, 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93121 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609457+0000", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7e-07 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609524+0000", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6706999999999997e-05 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609533+0000", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6309999999999998e-06 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609542+0000", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7679999999999997e-06 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609584+0000", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1829000000000002e-05 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609588+0000", 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1520000000000002e-06 2024-04-15T18:27:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402404+0000", 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79281564900000001 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402416+0000", 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2415000000000001e-05 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93122 5.a 5.80d4a5fa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854828085999998, 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79291208199999996, 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93122 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609558+0000", 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6699999999999999e-07 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609617+0000", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.906e-05 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609626+0000", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3100000000000006e-06 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609632+0000", 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9889999999999997e-06 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609659+0000", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6908999999999999e-05 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609662+0000", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2320000000000001e-06 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402457+0000", 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79279496599999999 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402469+0000", 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2149e-05 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93124 5.a 5.1f3061a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854740899999996, 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79288053300000005, 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93124 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609645+0000", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6199999999999998e-07 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609704+0000", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8850000000000001e-05 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609714+0000", 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020999999999999e-05 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609720+0000", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8359999999999997e-06 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609760+0000", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0207000000000003e-05 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609763+0000", 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402513+0000", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79274942500000001 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402525+0000", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2283000000000001e-05 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93126 5.a 5.2939364a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854651933, 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.792824317, 2024-04-15T18:27:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93126 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609734+0000", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7399999999999998e-07 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609798+0000", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3664999999999994e-05 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609806+0000", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5010000000000004e-06 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609810+0000", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3789999999999999e-06 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609863+0000", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2661000000000003e-05 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609872+0000", 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7450000000000001e-06 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402548+0000", 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79267563200000002 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402558+0000", 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.026e-05 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93129 5.a 5.c296a3da (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854565643000001, 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79277050400000004, 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93129 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.03e-07 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609865+0000", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4506999999999999e-05 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609869+0000", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7280000000000003e-06 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609873+0000", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6770000000000001e-06 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609915+0000", 2024-04-15T18:27:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1841999999999997e-05 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9530000000000005e-06 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402580+0000", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79265583299999998 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402590+0000", 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0661e-05 2024-04-15T18:27:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93139 5.a 5.5b8b557a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854504155000001, 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79273795199999997, 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93139 2024-04-15T18:27:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609882+0000", 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8900000000000001e-07 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609916+0000", 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4980000000000001e-05 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609921+0000", 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5889999999999996e-06 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1930000000000002e-06 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609972+0000", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7296000000000003e-05 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3039999999999994e-06 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402609+0000", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262850200000001 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402619+0000", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7990000000000005e-06 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93143 5.a 5.524373aa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:27:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.854446789999997, 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79271180799999996, 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93143 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6100000000000002e-07 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609976+0000", 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6807999999999997e-05 2024-04-15T18:27:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4809999999999997e-06 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609985+0000", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0899999999999998e-06 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610009+0000", 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3479e-05 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610012+0000", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6289999999999999e-06 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402641+0000", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262867999999997 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402650+0000", 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.38e-06 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93146 5.6 5.74a5d016 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.848915735999995, 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78680531300000001, 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93146 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615471+0000", 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0860000000000001e-06 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615527+0000", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6440000000000002e-05 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615541+0000", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3718e-05 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615555+0000", 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.367e-05 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615601+0000", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6830999999999999e-05 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615607+0000", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4790000000000004e-06 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402271+0000", 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78666379799999997 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402275+0000", 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2910000000000001e-06 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93150 5.6 5.5c92e826 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:16.026138+0000", 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 76.438247168000004, 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37615737599999999, 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:27:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93150 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026138+0000", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026138+0000", 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026140+0000", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0099999999999998e-06 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026212+0000", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1344e-05 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026231+0000", 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8950999999999999e-05 2024-04-15T18:27:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026240+0000", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9830000000000002e-06 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026297+0000", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7133999999999999e-05 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.026305+0000", 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2919999999999996e-06 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402291+0000", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.375986241 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402296+0000", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4209999999999997e-06 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 419/405 [PushOp(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head, version: 370'1523, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head@370'1523, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:27:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.449562086, 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48793246000000001, 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014824+0000", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014867+0000", 2024-04-15T18:27:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3164999999999998e-05 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.500977+0000", 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48611008500000003 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501005+0000", 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7334e-05 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501108+0000", 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103587 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501119+0000", 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1297e-05 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.502756+0000", 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016362589999999999 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 420/405 [PushOp(5:652b943a:::benchmark_data_smithi159_116570_object24315:head, version: 370'1552, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:652b943a:::benchmark_data_smithi159_116570_object24315:head@370'1552, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:48.044125+0000", 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 44.420259936999997, 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31819323599999999, 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.044125+0000", 2024-04-15T18:27:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.044125+0000", 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.044127+0000", 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.091e-06 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.044165+0000", 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8899999999999997e-05 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.360788+0000", 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31662272299999999 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.360798+0000", 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4129999999999995e-06 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.360864+0000", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6898999999999998e-05 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.360874+0000", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1169999999999993e-06 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.362319+0000", 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014450929999999999 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.4 420/405 [PushOp(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head, version: 370'1593, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head@370'1593, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 43.878243753, 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.622201475, 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586143+0000", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.108e-06 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586181+0000", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8618000000000002e-05 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205809+0000", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61962728300000003 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205830+0000", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1491000000000001e-05 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205930+0000", 2024-04-15T18:27:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9763e-05 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205941+0000", 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0776e-05 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.208343+0000", 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002402436 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 420/405 [PushOp(5:6269e789:::benchmark_data_smithi159_116570_object26106:head, version: 371'1651, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6269e789:::benchmark_data_smithi159_116570_object26106:head@371'1651, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:49.401780+0000", 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 43.062605646999998, 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.338290172, 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.401780+0000", 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.401780+0000", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.401781+0000", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2580000000000001e-06 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.401850+0000", 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9134999999999995e-05 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.737475+0000", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33562449599999999 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.737485+0000", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.028e-05 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.737586+0000", 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010128699999999999 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.737604+0000", 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7442999999999999e-05 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.740070+0000", 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024662730000000002 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 422/405 [PushOp(5:693270da:::benchmark_data_smithi159_116570_object31578:head, version: 372'1976, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:693270da:::benchmark_data_smithi159_116570_object31578:head@372'1976, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 38.620709238000003, 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54885523899999999, 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843678+0000", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6700000000000001e-06 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843793+0000", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115654 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389971+0000", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54617790200000005 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389987+0000", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5892e-05 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390342+0000", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035477700000000002 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390368+0000", 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6444999999999998e-05 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.392531+0000", 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021628989999999998 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 423/405 [PushOp(5:6d5ac67e:::benchmark_data_smithi159_116570_object32999:head, version: 372'2074, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6d5ac67e:::benchmark_data_smithi159_116570_object32999:head@372'2074, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:55.881328+0000", 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 36.583057590000003, 2024-04-15T18:27:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42723437800000003, 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:55.881328+0000", 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:55.881328+0000", 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:55.881329+0000", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.356e-06 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:55.881386+0000", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6709000000000002e-05 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.306930+0000", 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42554428 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.306944+0000", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4104000000000001e-05 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.307036+0000", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1463000000000002e-05 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.307047+0000", 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0934999999999999e-05 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.308562+0000", 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015155310000000001 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 423/405 [PushOp(5:66afd034:::benchmark_data_smithi159_116570_object33003:head, version: 372'2075, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:66afd034:::benchmark_data_smithi159_116570_object33003:head@372'2075, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:56.315555+0000", 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 36.148830009999998, 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.46995759399999998, 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.315555+0000", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.315555+0000", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.315556+0000", 2024-04-15T18:27:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.315643+0000", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6329000000000006e-05 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.783787+0000", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.46814434399999999 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.783810+0000", 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3558e-05 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.783910+0000", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9559e-05 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.783924+0000", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4413000000000001e-05 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.785513+0000", 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015885529999999999 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:108414 5.6 e423/405 5:68014cff:::benchmark_data_smithi159_116570_object15254:head v 423'6704, mlcod=372'2074)", 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:56.382385+0000", 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 36.081999940999999, 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41392604399999999, 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.382385+0000", 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.382385+0000", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.382386+0000", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.78e-07 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.382393+0000", 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9399999999999996e-06 2024-04-15T18:27:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.792947+0000", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41055364100000002 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.792953+0000", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1e-06 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.792999+0000", 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5565999999999997e-05 2024-04-15T18:27:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.793021+0000", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2268000000000001e-05 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.796282+0000", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032606169999999999 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:56.796311+0000", 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9933999999999998e-05 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.4 431/405 [PushOp(5:26823791:::benchmark_data_smithi159_116570_object53820:head, version: 383'3354, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:26823791:::benchmark_data_smithi159_116570_object53820:head@383'3354, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:16.221842+0000", 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.242543855000001, 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.465538075, 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.221842+0000", 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.221842+0000", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.221842+0000", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5499999999999998e-07 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.221908+0000", 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5753000000000006e-05 2024-04-15T18:27:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.685793+0000", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.463885194 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.685805+0000", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1744e-05 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.685879+0000", 2024-04-15T18:27:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3699e-05 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.685888+0000", 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5419999999999994e-06 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:16.687380+0000", 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001491488 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:27:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:27:32.545 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:27:32.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:27:32.546 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:27:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:32.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:32.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:27:32.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:27:32.564 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:27:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:27:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:27:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:27:32.648 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:32.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:32 smithi159 ceph-mon[45336]: pgmap v1157: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 47/221790 objects degraded (0.021%); 15326/221790 objects misplaced (6.910%); 15 MiB/s, 246 objects/s recovering 2024-04-15T18:27:32.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1106959561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:32.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/221790 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:32.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:32 smithi082 ceph-mon[44206]: pgmap v1157: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 47/221790 objects degraded (0.021%); 15326/221790 objects misplaced (6.910%); 15 MiB/s, 246 objects/s recovering 2024-04-15T18:27:32.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1106959561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:32.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/221790 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:32.675 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:32.694 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:32.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:32.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:32 smithi049 ceph-mon[40087]: pgmap v1157: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 47/221790 objects degraded (0.021%); 15326/221790 objects misplaced (6.910%); 15 MiB/s, 246 objects/s recovering 2024-04-15T18:27:32.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1106959561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:32.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/221790 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:32.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:32.824 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:32.842 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:32.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:32.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:32.980 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:33.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:27:33.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:33.834 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:33.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:33.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:33.969 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:33.992 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:33.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:34.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:34.124 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:34.143 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:34.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:34.162 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:27:34.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:34.280 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:34.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:34.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:34 smithi159 ceph-mon[45336]: pgmap v1158: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 47/220545 objects degraded (0.021%); 13563/220545 objects misplaced (6.150%); 34 MiB/s, 536 objects/s recovering 2024-04-15T18:27:34.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:34.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:34 smithi049 ceph-mon[40087]: pgmap v1158: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 47/220545 objects degraded (0.021%); 13563/220545 objects misplaced (6.150%); 34 MiB/s, 536 objects/s recovering 2024-04-15T18:27:34.734 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:34.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:34.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:34.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:34 smithi082 ceph-mon[44206]: pgmap v1158: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 47/220545 objects degraded (0.021%); 13563/220545 objects misplaced (6.150%); 34 MiB/s, 536 objects/s recovering 2024-04-15T18:27:34.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:34.840 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:34.875 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:35.625 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:35.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:35.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:35.764 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:36.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:36 smithi159 ceph-mon[45336]: pgmap v1159: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 47/220149 objects degraded (0.021%); 13447/220149 objects misplaced (6.108%); 26 MiB/s, 410 objects/s recovering 2024-04-15T18:27:36.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:36 smithi049 ceph-mon[40087]: pgmap v1159: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 47/220149 objects degraded (0.021%); 13447/220149 objects misplaced (6.108%); 26 MiB/s, 410 objects/s recovering 2024-04-15T18:27:36.715 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:36.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:36.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:27:36.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:36 smithi082 ceph-mon[44206]: pgmap v1159: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 47/220149 objects degraded (0.021%); 13447/220149 objects misplaced (6.108%); 26 MiB/s, 410 objects/s recovering 2024-04-15T18:27:36.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:36.849 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:36.948 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:36.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:37.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:37.208 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:37.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-04-15T18:27:37.312 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:37.344 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:37 smithi082 ceph-mon[44206]: osdmap e437: 12 total, 12 up, 12 in 2024-04-15T18:27:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:27:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 47/220149 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:27:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:37 smithi082 ceph-mon[44206]: osdmap e438: 12 total, 12 up, 12 in 2024-04-15T18:27:37.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:37 smithi049 ceph-mon[40087]: osdmap e437: 12 total, 12 up, 12 in 2024-04-15T18:27:37.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:27:37.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 47/220149 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:37.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:27:37.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:37 smithi049 ceph-mon[40087]: osdmap e438: 12 total, 12 up, 12 in 2024-04-15T18:27:37.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:37 smithi159 ceph-mon[45336]: osdmap e437: 12 total, 12 up, 12 in 2024-04-15T18:27:37.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:27:37.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 47/220149 objects degraded (0.021%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:27:37.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/375935687' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:27:37.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:37 smithi159 ceph-mon[45336]: osdmap e438: 12 total, 12 up, 12 in 2024-04-15T18:27:38.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 6 1 2024-04-15T18:27:38.279 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:38.503 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:27:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:38 smithi082 ceph-mon[44206]: pgmap v1162: 17 pgs: 1 active, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 12962/219591 objects misplaced (5.903%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:27:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1527663988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:38 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:38.882 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:38 smithi049 ceph-mon[40087]: pgmap v1162: 17 pgs: 1 active, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 12962/219591 objects misplaced (5.903%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:27:38.882 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1527663988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:38.882 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:38 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:38.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:38 smithi159 ceph-mon[45336]: pgmap v1162: 17 pgs: 1 active, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 126 op/s; 12962/219591 objects misplaced (5.903%); 21 MiB/s, 335 objects/s recovering 2024-04-15T18:27:38.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1527663988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:38.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:38 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:39 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:39 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:39 smithi049 ceph-mon[40087]: osdmap e439: 12 total, 12 up, 12 in 2024-04-15T18:27:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:39 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:39 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:39 smithi082 ceph-mon[44206]: osdmap e439: 12 total, 12 up, 12 in 2024-04-15T18:27:39.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:39 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:39.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:39 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:39.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:39 smithi159 ceph-mon[45336]: osdmap e439: 12 total, 12 up, 12 in 2024-04-15T18:27:40.088 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:40.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:40 smithi049 ceph-mon[40087]: pgmap v1164: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11875/218733 objects misplaced (5.429%); 17 MiB/s, 276 objects/s recovering 2024-04-15T18:27:40.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:40 smithi049 ceph-mon[40087]: osdmap e440: 12 total, 12 up, 12 in 2024-04-15T18:27:40.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:40 smithi082 ceph-mon[44206]: pgmap v1164: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11875/218733 objects misplaced (5.429%); 17 MiB/s, 276 objects/s recovering 2024-04-15T18:27:40.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:40 smithi082 ceph-mon[44206]: osdmap e440: 12 total, 12 up, 12 in 2024-04-15T18:27:40.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:40 smithi159 ceph-mon[45336]: pgmap v1164: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11875/218733 objects misplaced (5.429%); 17 MiB/s, 276 objects/s recovering 2024-04-15T18:27:40.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:40 smithi159 ceph-mon[45336]: osdmap e440: 12 total, 12 up, 12 in 2024-04-15T18:27:41.375 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:41.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:41.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:27:41.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:41.508 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:41.703 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:27:41.727 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:27:41.753 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:27:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:27:41.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:41 smithi082 ceph-mon[44206]: osdmap e441: 12 total, 12 up, 12 in 2024-04-15T18:27:41.802 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:27:41.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:27:41.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:27:41.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:27:41.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:41 smithi049 ceph-mon[40087]: osdmap e441: 12 total, 12 up, 12 in 2024-04-15T18:27:41.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:41 smithi159 ceph-mon[45336]: osdmap e441: 12 total, 12 up, 12 in 2024-04-15T18:27:41.901 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:27:41.930 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:27:41.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:27:41.979 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:27:41.980 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:27:41.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:27:42.532 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-04-15T18:27:42.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: pgmap v1167: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 11759/217968 objects misplaced (5.395%); 25 MiB/s, 409 objects/s recovering 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osdmap e442: 12 total, 12 up, 12 in 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1528695355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-04-15T18:27:42.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: pgmap v1167: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 11759/217968 objects misplaced (5.395%); 25 MiB/s, 409 objects/s recovering 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osdmap e442: 12 total, 12 up, 12 in 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1528695355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-04-15T18:27:42.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: pgmap v1167: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active, 1 active+recovering, 13 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 11759/217968 objects misplaced (5.395%); 25 MiB/s, 409 objects/s recovering 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osdmap e442: 12 total, 12 up, 12 in 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1528695355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:42.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-04-15T18:27:42.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:27:43.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:43.664 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:43.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:43.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 7 1 2024-04-15T18:27:43.769 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: osdmap e443: 12 total, 12 up, 12 in 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:43.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:43.798 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:43.818 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:43.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:43.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: osdmap e443: 12 total, 12 up, 12 in 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:43.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:43.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3982796629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: osdmap e443: 12 total, 12 up, 12 in 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:43.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:43.922 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:43.950 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:43.971 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:43.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:44.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:44.101 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:27:44.101 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:27:44.102 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:44.122 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:44.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:44.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:44.253 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:44.351 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:44.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:44.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:44.486 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:44.523 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:44.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:44.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:44.657 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:44.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:44 smithi082 ceph-mon[44206]: pgmap v1170: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 170 op/s; 11142/216681 objects misplaced (5.142%); 9.9 MiB/s, 164 objects/s recovering 2024-04-15T18:27:44.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:44 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-04-15T18:27:44.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:44 smithi082 ceph-mon[44206]: osdmap e444: 12 total, 12 up, 12 in 2024-04-15T18:27:44.779 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:27:44.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:44 smithi049 ceph-mon[40087]: pgmap v1170: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 170 op/s; 11142/216681 objects misplaced (5.142%); 9.9 MiB/s, 164 objects/s recovering 2024-04-15T18:27:44.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:44 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-04-15T18:27:44.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:44 smithi049 ceph-mon[40087]: osdmap e444: 12 total, 12 up, 12 in 2024-04-15T18:27:44.845 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:44.855 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:44.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:44.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:27:44.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:44 smithi159 ceph-mon[45336]: pgmap v1170: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 170 op/s; 11142/216681 objects misplaced (5.142%); 9.9 MiB/s, 164 objects/s recovering 2024-04-15T18:27:44.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:44 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-04-15T18:27:44.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:44 smithi159 ceph-mon[45336]: osdmap e444: 12 total, 12 up, 12 in 2024-04-15T18:27:44.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:44.994 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:45.047 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:27:45.094 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:45.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:45.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:45.228 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:45.781 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:45.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:45.885 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:45.913 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:45.932 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:45.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:46.036 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:46.042 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:46.064 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:46.083 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:46.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:46.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:46.217 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:46.527 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:27:46.603 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:46 smithi082 ceph-mon[44206]: pgmap v1172: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 11019/216276 objects misplaced (5.095%); 8.2 MiB/s, 136 objects/s recovering 2024-04-15T18:27:46.603 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:46.886 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:27:46.887 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:27:46.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:46 smithi159 ceph-mon[45336]: pgmap v1172: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 11019/216276 objects misplaced (5.095%); 8.2 MiB/s, 136 objects/s recovering 2024-04-15T18:27:46.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:46.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:46 smithi049 ceph-mon[40087]: pgmap v1172: 17 pgs: 1 creating+peering, 1 active+recovering, 15 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 11019/216276 objects misplaced (5.095%); 8.2 MiB/s, 136 objects/s recovering 2024-04-15T18:27:46.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:27:47.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:27:47.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: osdmap e445: 12 total, 12 up, 12 in 2024-04-15T18:27:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: osdmap e446: 12 total, 12 up, 12 in 2024-04-15T18:27:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:47.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:47.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: osdmap e445: 12 total, 12 up, 12 in 2024-04-15T18:27:47.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: osdmap e446: 12 total, 12 up, 12 in 2024-04-15T18:27:47.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:47.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:47.868 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:47.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:47.886 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3634896004' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: osdmap e445: 12 total, 12 up, 12 in 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: osdmap e446: 12 total, 12 up, 12 in 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:47.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:47.973 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:47.998 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:48.139 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:48.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:48.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:48.276 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:48.286 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-04-15T18:27:48.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:48.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:48.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:48.575 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:48.596 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:48.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:48.699 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:48.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:48.752 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:48.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:48.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:48 smithi082 ceph-mon[44206]: pgmap v1175: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11019/215460 objects misplaced (5.114%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:27:48.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:48 smithi082 ceph-mon[44206]: 5.f scrub starts 2024-04-15T18:27:48.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:27:48.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:27:48.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:48 smithi082 ceph-mon[44206]: osdmap e447: 12 total, 12 up, 12 in 2024-04-15T18:27:48.856 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:48.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:48 smithi159 ceph-mon[45336]: pgmap v1175: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11019/215460 objects misplaced (5.114%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:27:48.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:48 smithi159 ceph-mon[45336]: 5.f scrub starts 2024-04-15T18:27:48.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:27:48.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:27:48.881 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:48 smithi159 ceph-mon[45336]: osdmap e447: 12 total, 12 up, 12 in 2024-04-15T18:27:48.883 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:48.903 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:48.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:48.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:48 smithi049 ceph-mon[40087]: pgmap v1175: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 11019/215460 objects misplaced (5.114%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:27:48.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:48 smithi049 ceph-mon[40087]: 5.f scrub starts 2024-04-15T18:27:48.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:27:48.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2592358627' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:27:48.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:48 smithi049 ceph-mon[40087]: osdmap e447: 12 total, 12 up, 12 in 2024-04-15T18:27:49.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:49.034 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:49.053 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:49.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:49.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 8 1 2024-04-15T18:27:49.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:49.184 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:49.211 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:49.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:49.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:49.346 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:49.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:49.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:49.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:49 smithi049 ceph-mon[40087]: 5.7 scrub starts 2024-04-15T18:27:49.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:49 smithi049 ceph-mon[40087]: 5.f scrub ok 2024-04-15T18:27:49.650 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:49 smithi049 ceph-mon[40087]: 5.6 scrub starts 2024-04-15T18:27:49.696 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:49.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:49.724 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:49.778 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:27:49.779 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:27:49.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:49.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:49.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:49 smithi082 ceph-mon[44206]: 5.7 scrub starts 2024-04-15T18:27:49.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:49 smithi082 ceph-mon[44206]: 5.f scrub ok 2024-04-15T18:27:49.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:49 smithi082 ceph-mon[44206]: 5.6 scrub starts 2024-04-15T18:27:49.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:49.826 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:49.826 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:49.826 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:49 smithi159 ceph-mon[45336]: 5.7 scrub starts 2024-04-15T18:27:49.826 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:49 smithi159 ceph-mon[45336]: 5.f scrub ok 2024-04-15T18:27:49.827 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:49 smithi159 ceph-mon[45336]: 5.6 scrub starts 2024-04-15T18:27:49.829 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:49.848 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:49.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:49.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:49.982 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:50.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:27:50.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:27:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:50 smithi049 ceph-mon[40087]: pgmap v1177: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 92 op/s; 11019/215076 objects misplaced (5.123%); 442 KiB/s, 6 objects/s recovering 2024-04-15T18:27:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:50 smithi049 ceph-mon[40087]: 5.a scrub starts 2024-04-15T18:27:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:50 smithi049 ceph-mon[40087]: 5.6 scrub ok 2024-04-15T18:27:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:50 smithi049 ceph-mon[40087]: 5.7 scrub ok 2024-04-15T18:27:50.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3207463038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:50 smithi082 ceph-mon[44206]: pgmap v1177: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 92 op/s; 11019/215076 objects misplaced (5.123%); 442 KiB/s, 6 objects/s recovering 2024-04-15T18:27:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:50 smithi082 ceph-mon[44206]: 5.a scrub starts 2024-04-15T18:27:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:50 smithi082 ceph-mon[44206]: 5.6 scrub ok 2024-04-15T18:27:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:50 smithi082 ceph-mon[44206]: 5.7 scrub ok 2024-04-15T18:27:50.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3207463038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:50.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:50 smithi159 ceph-mon[45336]: pgmap v1177: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 92 op/s; 11019/215076 objects misplaced (5.123%); 442 KiB/s, 6 objects/s recovering 2024-04-15T18:27:50.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:50 smithi159 ceph-mon[45336]: 5.a scrub starts 2024-04-15T18:27:50.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:50 smithi159 ceph-mon[45336]: 5.6 scrub ok 2024-04-15T18:27:50.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:50 smithi159 ceph-mon[45336]: 5.7 scrub ok 2024-04-15T18:27:50.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3207463038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:51.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.597 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:27:52.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:27:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: 5.a scrub ok 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: pgmap v1178: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 97 op/s; 10894/214515 objects misplaced (5.078%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: 5.5 scrub starts 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:52.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:52 smithi159 ceph-mon[45336]: osdmap e448: 12 total, 12 up, 12 in 2024-04-15T18:27:52.956 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: 5.a scrub ok 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: pgmap v1178: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 97 op/s; 10894/214515 objects misplaced (5.078%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: 5.5 scrub starts 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:52.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:52 smithi049 ceph-mon[40087]: osdmap e448: 12 total, 12 up, 12 in 2024-04-15T18:27:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:52.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: 5.a scrub ok 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: pgmap v1178: 17 pgs: 1 active+recovering, 16 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 97 op/s; 10894/214515 objects misplaced (5.078%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: 5.5 scrub starts 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:53.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:52 smithi082 ceph-mon[44206]: osdmap e448: 12 total, 12 up, 12 in 2024-04-15T18:27:53.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:27:53.091 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:53.110 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:27:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:53.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:27:53.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:27:53.242 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:53.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:27:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:53.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:27:53.366 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:53.393 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:53.411 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:53.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:53.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:53.546 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:53.619 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:53.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:53.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:53.750 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:53.772 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:53.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:53.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:53 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:27:53.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:53 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:27:53.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:53 smithi082 ceph-mon[44206]: 5.5 scrub ok 2024-04-15T18:27:53.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:27:53.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:27:53.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:53.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:53 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:27:53.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:53 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:27:53.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:53 smithi159 ceph-mon[45336]: 5.5 scrub ok 2024-04-15T18:27:53.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:27:53.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:27:53.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:53 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:27:53.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:53 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:27:53.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:53 smithi049 ceph-mon[40087]: 5.5 scrub ok 2024-04-15T18:27:53.901 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:27:53.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:27:53.904 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:54.102 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:54.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:54.126 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:54.126 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:54.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:54.233 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:54.251 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:54.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:54.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:54.386 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:54.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:54 smithi049 ceph-mon[40087]: pgmap v1180: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 10894/213480 objects misplaced (5.103%); 444 KiB/s, 6 objects/s recovering 2024-04-15T18:27:54.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:54 smithi049 ceph-mon[40087]: osdmap e449: 12 total, 12 up, 12 in 2024-04-15T18:27:54.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:54 smithi159 ceph-mon[45336]: pgmap v1180: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 10894/213480 objects misplaced (5.103%); 444 KiB/s, 6 objects/s recovering 2024-04-15T18:27:54.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:54 smithi159 ceph-mon[45336]: osdmap e449: 12 total, 12 up, 12 in 2024-04-15T18:27:55.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:54 smithi082 ceph-mon[44206]: pgmap v1180: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 10894/213480 objects misplaced (5.103%); 444 KiB/s, 6 objects/s recovering 2024-04-15T18:27:55.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:54 smithi082 ceph-mon[44206]: osdmap e449: 12 total, 12 up, 12 in 2024-04-15T18:27:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:55.144 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:55.170 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.190 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:55.294 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:27:55.320 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.340 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:27:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:27:55.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:55.470 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.488 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:55.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:55.619 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:27:55.637 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:55 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:27:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:55 smithi159 ceph-mon[45336]: 5.b scrub starts 2024-04-15T18:27:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:55 smithi159 ceph-mon[45336]: 5.b scrub ok 2024-04-15T18:27:55.741 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:55.765 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:55 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:27:55.765 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:55 smithi082 ceph-mon[44206]: 5.b scrub starts 2024-04-15T18:27:55.765 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:55 smithi082 ceph-mon[44206]: 5.b scrub ok 2024-04-15T18:27:55.767 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:55.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:27:55.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:55 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:27:55.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:55 smithi049 ceph-mon[40087]: 5.b scrub starts 2024-04-15T18:27:55.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:55 smithi049 ceph-mon[40087]: 5.b scrub ok 2024-04-15T18:27:55.918 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:55.972 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:27:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:55.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:27:56.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:27:56.103 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:56.122 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:27:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:56.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:27:56.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:27:56.252 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:56.270 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:27:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:56.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:27:56.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:27:56.409 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:56 smithi049 ceph-mon[40087]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 91 op/s; 10770/213138 objects misplaced (5.053%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:27:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:27:56.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:56 smithi159 ceph-mon[45336]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 91 op/s; 10770/213138 objects misplaced (5.053%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:27:56.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:27:57.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:56 smithi082 ceph-mon[44206]: pgmap v1182: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 91 op/s; 10770/213138 objects misplaced (5.053%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:27:57.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:27:57.059 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:27:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:27:57.162 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:57.188 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:57.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:57.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:27:57.345 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:57.377 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:27:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:27:57.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:57.512 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:57.613 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:57.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:27:57.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:57.765 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:27:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:27:57.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:57.899 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:57.923 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:27:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:57.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:27:58.028 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:27:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:58.078 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:27:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:27:58.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:27:58.154 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:27:58.158 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:27:58.158 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:27:58.158 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:27:58.158 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:27:58.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:27:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:27:58.283 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-04-15T18:27:58.290 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: 5.e scrub starts 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: osdmap e450: 12 total, 12 up, 12 in 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2620067110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:27:58.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: 5.e scrub starts 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: osdmap e450: 12 total, 12 up, 12 in 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2620067110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:27:58.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: 5.e scrub starts 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: osdmap e450: 12 total, 12 up, 12 in 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2620067110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:27:58.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 10770/212109 objects misplaced (5.078%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: 5.e scrub ok 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:27:59.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:27:59 smithi082 ceph-mon[44206]: osdmap e451: 12 total, 12 up, 12 in 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 10770/212109 objects misplaced (5.078%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: 5.e scrub ok 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:27:59.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:27:59 smithi049 ceph-mon[40087]: osdmap e451: 12 total, 12 up, 12 in 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: pgmap v1184: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 10770/212109 objects misplaced (5.078%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: 5.e scrub ok 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1169255865' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:27:59.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:27:59 smithi159 ceph-mon[45336]: osdmap e451: 12 total, 12 up, 12 in 2024-04-15T18:27:59.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 9 1 2024-04-15T18:28:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:00.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:00.293 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:00.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:00.420 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:00.446 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:00.464 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:00 smithi082 ceph-mon[44206]: 5.4 deep-scrub starts 2024-04-15T18:28:00.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:00.592 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:00 smithi159 ceph-mon[45336]: 5.4 deep-scrub starts 2024-04-15T18:28:00.594 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:00.618 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:00.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:00 smithi049 ceph-mon[40087]: 5.4 deep-scrub starts 2024-04-15T18:28:00.721 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:00.754 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:00.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:00.877 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:00.904 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:00.923 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:00.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:01.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:28:01.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:01.091 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:01.488 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:01.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:01 smithi082 ceph-mon[44206]: pgmap v1186: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 10770/211746 objects misplaced (5.086%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:28:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:01 smithi159 ceph-mon[45336]: pgmap v1186: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 10770/211746 objects misplaced (5.086%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:28:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:01 smithi049 ceph-mon[40087]: pgmap v1186: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 96 op/s; 10770/211746 objects misplaced (5.086%); 427 KiB/s, 6 objects/s recovering 2024-04-15T18:28:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:01.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:01.847 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:01.875 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:01.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:01.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:02.004 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:02.032 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:02.054 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:02.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:02.157 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:02.185 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:02.204 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:02.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:02.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:02.333 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:02.352 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:02.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:02.457 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:02.482 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:02.502 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:02.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:02.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:02.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:02.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:02 smithi082 ceph-mon[44206]: osdmap e452: 12 total, 12 up, 12 in 2024-04-15T18:28:02.531 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:02.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:02.633 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:02.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:02.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:02.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:02 smithi049 ceph-mon[40087]: osdmap e452: 12 total, 12 up, 12 in 2024-04-15T18:28:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:02 smithi159 ceph-mon[45336]: osdmap e452: 12 total, 12 up, 12 in 2024-04-15T18:28:02.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:02.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:02.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:28:02.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:02.787 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:03.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:03.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:03.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:03.239 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:03.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:03.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:03.372 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:03.392 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:03.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:03.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: pgmap v1187: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10646/211059 objects misplaced (5.044%); 683 KiB/s, 10 objects/s recovering 2024-04-15T18:28:03.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:03.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:03.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3753934195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:03.523 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: pgmap v1187: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10646/211059 objects misplaced (5.044%); 683 KiB/s, 10 objects/s recovering 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3753934195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.541 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:03.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:03.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: pgmap v1187: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 103 op/s; 10646/211059 objects misplaced (5.044%); 683 KiB/s, 10 objects/s recovering 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3753934195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:03.671 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:03.689 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:03.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:03.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:03.824 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:04.024 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:04.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:04.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:04.155 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:04.174 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:04.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:04.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:04.286 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-04-15T18:28:04.288 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.288 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.288 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:28:04.311 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:04.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:28:04.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:04.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:28:04.664 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:04.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:04.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:05.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 10 1 2024-04-15T18:28:05.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:05 smithi049 ceph-mon[40087]: pgmap v1189: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10646/209982 objects misplaced (5.070%); 455 KiB/s, 7 objects/s recovering 2024-04-15T18:28:05.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:28:05.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:05 smithi049 ceph-mon[40087]: osdmap e453: 12 total, 12 up, 12 in 2024-04-15T18:28:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:05 smithi159 ceph-mon[45336]: pgmap v1189: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10646/209982 objects misplaced (5.070%); 455 KiB/s, 7 objects/s recovering 2024-04-15T18:28:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:28:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:05 smithi159 ceph-mon[45336]: osdmap e453: 12 total, 12 up, 12 in 2024-04-15T18:28:05.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:05 smithi082 ceph-mon[44206]: pgmap v1189: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 155 op/s; 10646/209982 objects misplaced (5.070%); 455 KiB/s, 7 objects/s recovering 2024-04-15T18:28:05.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1206664138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:28:05.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:05 smithi082 ceph-mon[44206]: osdmap e453: 12 total, 12 up, 12 in 2024-04-15T18:28:05.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:06.157 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:06.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:06.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:06.288 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:06.306 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:06.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:06.375 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:06.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:06.419 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:28:06.419 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:28:06.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:06.472 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:06.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:06.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:06.609 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:06.636 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:06.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:06.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:06.772 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:07.422 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:07.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:07.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:07.553 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:07 smithi159 ceph-mon[45336]: pgmap v1191: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 10646/209982 objects misplaced (5.070%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:07.553 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:07 smithi159 ceph-mon[45336]: osdmap e454: 12 total, 12 up, 12 in 2024-04-15T18:28:07.555 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:07.576 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:07.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:07.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:07 smithi049 ceph-mon[40087]: pgmap v1191: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 10646/209982 objects misplaced (5.070%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:07.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:07 smithi049 ceph-mon[40087]: osdmap e454: 12 total, 12 up, 12 in 2024-04-15T18:28:07.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:07.709 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:07.739 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:07.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:07 smithi082 ceph-mon[44206]: pgmap v1191: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 83 op/s; 10646/209982 objects misplaced (5.070%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:07 smithi082 ceph-mon[44206]: osdmap e454: 12 total, 12 up, 12 in 2024-04-15T18:28:07.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:28:07.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:07.881 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:07.900 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:07.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:08.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:08.033 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:08.057 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:08.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:08.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:08.190 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:08.214 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:08.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:08.318 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:08.347 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:08.372 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:08.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:08.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:08.506 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:08 smithi159 ceph-mon[45336]: pgmap v1193: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10521/208740 objects misplaced (5.040%); 437 KiB/s, 6 objects/s recovering 2024-04-15T18:28:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:08 smithi049 ceph-mon[40087]: pgmap v1193: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10521/208740 objects misplaced (5.040%); 437 KiB/s, 6 objects/s recovering 2024-04-15T18:28:08.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:08 smithi082 ceph-mon[44206]: pgmap v1193: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 10521/208740 objects misplaced (5.040%); 437 KiB/s, 6 objects/s recovering 2024-04-15T18:28:08.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:28:09.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:09.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:09.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:09.392 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:09.413 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:09.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:09.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:09.525 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:09.547 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:09.568 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:09.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:09.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:09.700 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:09.724 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:09.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:09.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:09.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:09.859 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:09.960 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:09.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:10.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:10.100 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:10.378 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-04-15T18:28:10.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:10 smithi159 ceph-mon[45336]: pgmap v1194: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 10521/208320 objects misplaced (5.050%); 372 KiB/s, 5 objects/s recovering 2024-04-15T18:28:10.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/601468581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:10 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:10 smithi049 ceph-mon[40087]: pgmap v1194: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 10521/208320 objects misplaced (5.050%); 372 KiB/s, 5 objects/s recovering 2024-04-15T18:28:10.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/601468581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:10 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:10 smithi082 ceph-mon[44206]: pgmap v1194: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 10521/208320 objects misplaced (5.050%); 372 KiB/s, 5 objects/s recovering 2024-04-15T18:28:10.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/601468581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:10 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:28:10.955 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:10.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:11.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:11.389 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:28:11.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:11 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:28:11.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:11 smithi159 ceph-mon[45336]: osdmap e455: 12 total, 12 up, 12 in 2024-04-15T18:28:11.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/522398979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:11.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:11 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:11 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:28:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:11 smithi049 ceph-mon[40087]: osdmap e455: 12 total, 12 up, 12 in 2024-04-15T18:28:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/522398979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:11 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:11.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:11 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:28:11.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:11 smithi082 ceph-mon[44206]: osdmap e455: 12 total, 12 up, 12 in 2024-04-15T18:28:11.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/522398979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:11.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:11 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:12.171 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:12.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:12.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:12.291 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:12 smithi049 ceph-mon[40087]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 10396/207546 objects misplaced (5.009%); 755 KiB/s, 11 objects/s recovering 2024-04-15T18:28:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:12 smithi049 ceph-mon[40087]: 5.0 scrub starts 2024-04-15T18:28:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:12 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:12.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:12 smithi049 ceph-mon[40087]: osdmap e456: 12 total, 12 up, 12 in 2024-04-15T18:28:12.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:12 smithi159 ceph-mon[45336]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 10396/207546 objects misplaced (5.009%); 755 KiB/s, 11 objects/s recovering 2024-04-15T18:28:12.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:12 smithi159 ceph-mon[45336]: 5.0 scrub starts 2024-04-15T18:28:12.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:12 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:12.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:12.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:12 smithi159 ceph-mon[45336]: osdmap e456: 12 total, 12 up, 12 in 2024-04-15T18:28:12.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 11 1 2024-04-15T18:28:12.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:12 smithi082 ceph-mon[44206]: pgmap v1196: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 115 op/s; 10396/207546 objects misplaced (5.009%); 755 KiB/s, 11 objects/s recovering 2024-04-15T18:28:12.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:12 smithi082 ceph-mon[44206]: 5.0 scrub starts 2024-04-15T18:28:12.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:12 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:12.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:12.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:12 smithi082 ceph-mon[44206]: osdmap e456: 12 total, 12 up, 12 in 2024-04-15T18:28:13.166 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:13.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:13.273 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:13.299 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:13.317 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:13.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:13.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:13.446 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:13.446 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:13.448 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:13.697 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:13.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:13.710 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:13.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:13.827 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:13.848 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:13.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:13.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:13.986 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:14.059 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:14.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:14.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:14.197 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:14 smithi159 ceph-mon[45336]: pgmap v1198: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 10396/206220 objects misplaced (5.041%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:14 smithi159 ceph-mon[45336]: osdmap e457: 12 total, 12 up, 12 in 2024-04-15T18:28:14.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:14 smithi049 ceph-mon[40087]: pgmap v1198: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 10396/206220 objects misplaced (5.041%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:14.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:14 smithi049 ceph-mon[40087]: osdmap e457: 12 total, 12 up, 12 in 2024-04-15T18:28:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:14 smithi082 ceph-mon[44206]: pgmap v1198: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 10396/206220 objects misplaced (5.041%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:14 smithi082 ceph-mon[44206]: osdmap e457: 12 total, 12 up, 12 in 2024-04-15T18:28:15.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:28:15.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:15.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:15.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:15 smithi159 ceph-mon[45336]: osdmap e458: 12 total, 12 up, 12 in 2024-04-15T18:28:15.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:15.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:15 smithi049 ceph-mon[40087]: osdmap e458: 12 total, 12 up, 12 in 2024-04-15T18:28:15.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:15.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:15.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:15 smithi082 ceph-mon[44206]: osdmap e458: 12 total, 12 up, 12 in 2024-04-15T18:28:16.332 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:28:16.332 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:28:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:16 smithi159 ceph-mon[45336]: pgmap v1201: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 124 op/s; 10396/206220 objects misplaced (5.041%); 543 KiB/s, 8 objects/s recovering 2024-04-15T18:28:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:16 smithi049 ceph-mon[40087]: pgmap v1201: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 124 op/s; 10396/206220 objects misplaced (5.041%); 543 KiB/s, 8 objects/s recovering 2024-04-15T18:28:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:16.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:16.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:16 smithi082 ceph-mon[44206]: pgmap v1201: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 124 op/s; 10396/206220 objects misplaced (5.041%); 543 KiB/s, 8 objects/s recovering 2024-04-15T18:28:16.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:16.883 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:16.904 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:16.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:17.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:17.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.053 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:17.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:17.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:28:17.184 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.204 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:17.308 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:17.334 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.352 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:17.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:17.488 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.519 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:17.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:17.650 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.671 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:17.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:17.801 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.820 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:17.924 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:17.952 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:17.973 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:17.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:17.977 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:17.996 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:18.010 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:18.034 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:18.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:18.069 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:18.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:18.094 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:18.103 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:18.121 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:18.126 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:18.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:18.151 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:18.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:18.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:18.228 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:18.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:18.257 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:18.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:18.276 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:18.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:18.286 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-04-15T18:28:18.297 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:18.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:18.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:18.307 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:18.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:18.430 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:18.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:28:18.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2883036599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:18.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:18.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:28:18.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2883036599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:18.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:18.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:28:18.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2883036599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:18.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:18.889 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:18.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:18.993 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:19.022 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:19.041 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:19.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:19.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:19.171 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:19.193 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:19.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:19.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:19.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:19.342 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:19.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:19.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:19.474 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: pgmap v1202: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 10271/204774 objects misplaced (5.016%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:28:19.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osdmap e459: 12 total, 12 up, 12 in 2024-04-15T18:28:19.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: pgmap v1202: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 10271/204774 objects misplaced (5.016%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:28:19.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osdmap e459: 12 total, 12 up, 12 in 2024-04-15T18:28:19.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: pgmap v1202: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 10271/204774 objects misplaced (5.016%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2615952283' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:28:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osdmap e459: 12 total, 12 up, 12 in 2024-04-15T18:28:19.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:20.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:20.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:20.157 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:20.178 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:20.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:20.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:20.312 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:20.663 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:20.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:20.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 3 1 2024-04-15T18:28:20.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:20.795 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:20.999 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:21.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:21.131 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.150 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:21.254 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:21.281 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.301 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:21.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:21 smithi159 ceph-mon[45336]: pgmap v1204: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10271/204276 objects misplaced (5.028%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:21.404 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:21.430 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.451 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:21.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:21.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:21 smithi049 ceph-mon[40087]: pgmap v1204: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10271/204276 objects misplaced (5.028%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:21.579 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:21 smithi082 ceph-mon[44206]: pgmap v1204: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 10271/204276 objects misplaced (5.028%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:21.581 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.601 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:21.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:28:21.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:21.732 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.751 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:21.855 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:21.883 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:21.903 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:21.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:21.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:22.009 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:22.037 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:22.062 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:22.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:22.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:22.198 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:23.287 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:28:23.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:23 smithi082 ceph-mon[44206]: pgmap v1205: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 10142/203568 objects misplaced (4.982%); 686 KiB/s, 10 objects/s recovering 2024-04-15T18:28:23.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:23 smithi082 ceph-mon[44206]: osdmap e460: 12 total, 12 up, 12 in 2024-04-15T18:28:23.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:23.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:23 smithi049 ceph-mon[40087]: pgmap v1205: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 10142/203568 objects misplaced (4.982%); 686 KiB/s, 10 objects/s recovering 2024-04-15T18:28:23.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:23 smithi049 ceph-mon[40087]: osdmap e460: 12 total, 12 up, 12 in 2024-04-15T18:28:23.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:23.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:23 smithi159 ceph-mon[45336]: pgmap v1205: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 111 op/s; 10142/203568 objects misplaced (4.982%); 686 KiB/s, 10 objects/s recovering 2024-04-15T18:28:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:23 smithi159 ceph-mon[45336]: osdmap e460: 12 total, 12 up, 12 in 2024-04-15T18:28:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:24.181 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:24.286 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:119071 5.5 5:a5b21e94:::benchmark_data_smithi159_116570_object25911:head [delete] snapc 0=[] ondisk+write+known_if_redirected e460)", 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:28:22.713801+0000", 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.091802172000000001, 2024-04-15T18:28:24.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.091828707999999995, 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 119071 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713801+0000", 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:24.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713801+0000", 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713803+0000", 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6020000000000001e-06 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:24.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713811+0000", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3769999999999996e-06 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713830+0000", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9055999999999998e-05 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713838+0000", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2139999999999996e-06 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:24.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713890+0000", 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1823999999999997e-05 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.713995+0000", 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104764 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.714106+0000", 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111056 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.717576+0000", 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034696390000000001 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:28:22.717668+0000", 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2559000000000006e-05 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-04-15T18:28:24.301 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:28:24.313 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:24.332 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:24.435 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:24.462 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:24.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:24 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:24.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:24.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:24 smithi049 ceph-mon[40087]: osdmap e461: 12 total, 12 up, 12 in 2024-04-15T18:28:24.481 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:24.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:24 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:24.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:24.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:24 smithi082 ceph-mon[44206]: osdmap e461: 12 total, 12 up, 12 in 2024-04-15T18:28:24.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:24.609 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:24 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:24.609 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3827160349' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:24.609 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:24 smithi159 ceph-mon[45336]: osdmap e461: 12 total, 12 up, 12 in 2024-04-15T18:28:24.611 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:24.630 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:24.734 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:24.760 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:24.778 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:24.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:24.908 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:24.928 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:24.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:25.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:25.062 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:25.083 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:25.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:25.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:25.217 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:25.241 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:25.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:25.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:25.371 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:25.390 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:25.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:25.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:25 smithi049 ceph-mon[40087]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 10142/202251 objects misplaced (5.015%); 459 KiB/s, 7 objects/s recovering 2024-04-15T18:28:25.494 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:25.520 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:25.538 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:25.540 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:25 smithi082 ceph-mon[44206]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 10142/202251 objects misplaced (5.015%); 459 KiB/s, 7 objects/s recovering 2024-04-15T18:28:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:25.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:25.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:28:25.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:25.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:25 smithi159 ceph-mon[45336]: pgmap v1208: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 10142/202251 objects misplaced (5.015%); 459 KiB/s, 7 objects/s recovering 2024-04-15T18:28:25.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:25.749 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:25.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:25.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:25.887 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.082 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:26.085 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:26.190 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:26.216 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.235 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:26.340 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:26.365 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:26.488 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:26.515 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.534 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:26.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:26.667 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.693 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:26.796 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:26.824 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.844 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:26.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:26.974 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:26.992 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:26.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:27.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:27.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:28:27.128 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:27.279 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-04-15T18:28:27.282 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:27.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:27.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:27.417 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:27.436 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:27.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:27.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:27 smithi049 ceph-mon[40087]: pgmap v1209: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 10142/202251 objects misplaced (5.015%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:27.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:28:27.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:28:27.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:27 smithi049 ceph-mon[40087]: osdmap e462: 12 total, 12 up, 12 in 2024-04-15T18:28:27.540 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:27.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:27 smithi159 ceph-mon[45336]: pgmap v1209: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 10142/202251 objects misplaced (5.015%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:27.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:28:27.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:28:27.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:27 smithi159 ceph-mon[45336]: osdmap e462: 12 total, 12 up, 12 in 2024-04-15T18:28:27.567 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:27.587 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:27.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:27.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:27.714 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:27 smithi082 ceph-mon[44206]: pgmap v1209: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 10142/202251 objects misplaced (5.015%); 391 KiB/s, 6 objects/s recovering 2024-04-15T18:28:27.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:28:27.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4224401842' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:28:27.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:27 smithi082 ceph-mon[44206]: osdmap e462: 12 total, 12 up, 12 in 2024-04-15T18:28:27.716 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:27.735 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:27.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:27.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:27.900 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:27.923 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:27.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:28.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:28.055 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:28.076 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:28.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:28.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:28.207 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:28.226 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:28.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:28.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:28.359 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:28.458 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:28.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:28.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 2 1 2024-04-15T18:28:28.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:28.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:28.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:28 smithi082 ceph-mon[44206]: osdmap e463: 12 total, 12 up, 12 in 2024-04-15T18:28:28.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:28.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:28.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:28 smithi049 ceph-mon[40087]: osdmap e463: 12 total, 12 up, 12 in 2024-04-15T18:28:28.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:28.594 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:28.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:28.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:28.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:28 smithi159 ceph-mon[45336]: osdmap e463: 12 total, 12 up, 12 in 2024-04-15T18:28:28.695 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:28.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:28.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:28.827 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:28.976 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:28.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:28.983 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:28.984 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:29.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:29.043 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:29.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:29.081 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:29.091 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:29.108 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:29.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:29.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:29.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:29.145 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:29.172 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:29.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:29.224 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:29.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:29.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:29.264 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:29.283 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:29.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:29.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:29.308 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:29.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:29.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:29.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:29.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:29.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:29 smithi159 ceph-mon[45336]: pgmap v1211: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 10013/200742 objects misplaced (4.988%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:29.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1359924471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:29.417 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:29.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:29 smithi049 ceph-mon[40087]: pgmap v1211: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 10013/200742 objects misplaced (4.988%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:29.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1359924471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:29.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:29 smithi082 ceph-mon[44206]: pgmap v1211: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 10013/200742 objects misplaced (4.988%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:28:29.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1359924471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:30.583 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:30 smithi159 ceph-mon[45336]: pgmap v1213: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 10013/200187 objects misplaced (5.002%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:30.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:30 smithi049 ceph-mon[40087]: pgmap v1213: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 10013/200187 objects misplaced (5.002%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:30.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:30 smithi082 ceph-mon[44206]: pgmap v1213: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 113 op/s; 10013/200187 objects misplaced (5.002%); 445 KiB/s, 6 objects/s recovering 2024-04-15T18:28:31.773 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:31.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:31.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:31.904 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:31.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:31.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:32.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:32.058 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:32.209 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:32.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:32.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:32.343 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:32.441 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:32.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:32.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:32.576 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:32.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:32.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:32.699 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:32.726 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:32.746 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:32.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:32.850 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:32.876 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:32.894 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:32.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:28:32.977 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:28:32.998 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:33.024 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.043 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:33.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:28:33.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:33.174 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.196 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:33.300 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:33.304 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:28:33.327 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.349 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:33.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:33.479 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.497 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:33.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:33 smithi082 ceph-mon[44206]: pgmap v1214: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9879/199320 objects misplaced (4.956%); 704 KiB/s, 10 objects/s recovering 2024-04-15T18:28:33.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:33 smithi082 ceph-mon[44206]: osdmap e464: 12 total, 12 up, 12 in 2024-04-15T18:28:33.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:28:33.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.647 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:33.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:33 smithi159 ceph-mon[45336]: pgmap v1214: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9879/199320 objects misplaced (4.956%); 704 KiB/s, 10 objects/s recovering 2024-04-15T18:28:33.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:33 smithi159 ceph-mon[45336]: osdmap e464: 12 total, 12 up, 12 in 2024-04-15T18:28:33.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:28:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:33 smithi049 ceph-mon[40087]: pgmap v1214: 17 pgs: 1 active+recovering, 16 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 122 op/s; 9879/199320 objects misplaced (4.956%); 704 KiB/s, 10 objects/s recovering 2024-04-15T18:28:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:33 smithi049 ceph-mon[40087]: osdmap e464: 12 total, 12 up, 12 in 2024-04-15T18:28:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:28:33.751 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:33.777 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.796 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:33.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:28:33.899 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:33.925 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:33.944 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:33.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:34.048 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:34.075 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.097 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:34.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:34.230 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.255 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:34.359 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:34.386 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.405 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:34.511 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:34.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:34 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:34.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:28:34.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:34 smithi082 ceph-mon[44206]: osdmap e465: 12 total, 12 up, 12 in 2024-04-15T18:28:34.537 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.554 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:34 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:34.554 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:28:34.554 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:34 smithi049 ceph-mon[40087]: osdmap e465: 12 total, 12 up, 12 in 2024-04-15T18:28:34.556 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:34.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:34 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:34.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/398435673' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:28:34.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:34 smithi159 ceph-mon[45336]: osdmap e465: 12 total, 12 up, 12 in 2024-04-15T18:28:34.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:34.686 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.705 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:34.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:34.836 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:34.876 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:34.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:34.980 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:35.007 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:35.026 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:35.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:35.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:35.158 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:35.306 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:35.307 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:35.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:35.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:35.443 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:35.464 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:35.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:35.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:35 smithi049 ceph-mon[40087]: pgmap v1217: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 9879/197922 objects misplaced (4.991%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:35.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:35.604 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:35.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:35 smithi159 ceph-mon[45336]: pgmap v1217: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 9879/197922 objects misplaced (4.991%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:35.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:35 smithi082 ceph-mon[44206]: pgmap v1217: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 156 op/s; 9879/197922 objects misplaced (4.991%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:35.804 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:35.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:35.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:35.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:36.897 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:36.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:36.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:28:37.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:37.029 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:37.050 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:37.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:37.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:37.180 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:37.199 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:37.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:37.294 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:37 smithi159 ceph-mon[45336]: pgmap v1218: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9879/197922 objects misplaced (4.991%); 418 KiB/s, 6 objects/s recovering 2024-04-15T18:28:37.294 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:37 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:28:37.294 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:37 smithi159 ceph-mon[45336]: osdmap e466: 12 total, 12 up, 12 in 2024-04-15T18:28:37.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:37.334 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:37.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:37 smithi049 ceph-mon[40087]: pgmap v1218: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9879/197922 objects misplaced (4.991%); 418 KiB/s, 6 objects/s recovering 2024-04-15T18:28:37.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:37 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:28:37.518 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:37 smithi049 ceph-mon[40087]: osdmap e466: 12 total, 12 up, 12 in 2024-04-15T18:28:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:37 smithi082 ceph-mon[44206]: pgmap v1218: 17 pgs: 1 active+recovering, 16 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 107 op/s; 9879/197922 objects misplaced (4.991%); 418 KiB/s, 6 objects/s recovering 2024-04-15T18:28:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:37 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:28:37.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:37 smithi082 ceph-mon[44206]: osdmap e466: 12 total, 12 up, 12 in 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:38.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:38.300 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-04-15T18:28:38.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1363234321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: 5.4 scrub starts 2024-04-15T18:28:38.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4013131773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:38.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.585 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:38.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1363234321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: 5.4 scrub starts 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4013131773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.692 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:38.721 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:38.741 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:38.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1363234321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:28:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: 5.4 scrub starts 2024-04-15T18:28:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4013131773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:38.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:38.874 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: pgmap v1220: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 9742/196116 objects misplaced (4.967%); 480 KiB/s, 7 objects/s recovering 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:28:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: osdmap e467: 12 total, 12 up, 12 in 2024-04-15T18:28:39.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:39 smithi159 ceph-mon[45336]: 5.c scrub starts 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: pgmap v1220: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 9742/196116 objects misplaced (4.967%); 480 KiB/s, 7 objects/s recovering 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: osdmap e467: 12 total, 12 up, 12 in 2024-04-15T18:28:39.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:39 smithi049 ceph-mon[40087]: 5.c scrub starts 2024-04-15T18:28:39.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: pgmap v1220: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 9742/196116 objects misplaced (4.967%); 480 KiB/s, 7 objects/s recovering 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: osdmap e467: 12 total, 12 up, 12 in 2024-04-15T18:28:39.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:39 smithi082 ceph-mon[44206]: 5.c scrub starts 2024-04-15T18:28:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:40.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:40.155 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.180 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:40.298 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:40.327 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.349 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:40.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:40.480 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.499 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:40.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:40.626 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:40 smithi159 ceph-mon[45336]: 5.c scrub ok 2024-04-15T18:28:40.626 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:40 smithi159 ceph-mon[45336]: 5.4 scrub ok 2024-04-15T18:28:40.627 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:40 smithi159 ceph-mon[45336]: pgmap v1222: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9742/195522 objects misplaced (4.983%); 477 KiB/s, 7 objects/s recovering 2024-04-15T18:28:40.629 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.649 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:40.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:40 smithi049 ceph-mon[40087]: 5.c scrub ok 2024-04-15T18:28:40.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:40 smithi049 ceph-mon[40087]: 5.4 scrub ok 2024-04-15T18:28:40.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:40 smithi049 ceph-mon[40087]: pgmap v1222: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9742/195522 objects misplaced (4.983%); 477 KiB/s, 7 objects/s recovering 2024-04-15T18:28:40.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:40 smithi082 ceph-mon[44206]: 5.c scrub ok 2024-04-15T18:28:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:40 smithi082 ceph-mon[44206]: 5.4 scrub ok 2024-04-15T18:28:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:40 smithi082 ceph-mon[44206]: pgmap v1222: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 132 op/s; 9742/195522 objects misplaced (4.983%); 477 KiB/s, 7 objects/s recovering 2024-04-15T18:28:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.801 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:40.862 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:40.895 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-15T18:28:40.895 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-15T18:28:40.895 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:28:40.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:28:40.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:40.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:40.953 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:40.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:41.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:41.084 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:41.111 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:41.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:41.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:41.370 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:41.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:41.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:41.502 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:41.521 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:41.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:41.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:41.651 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:41.659 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:28:41.659 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:28:41.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71431 4.4 4:2243c596:::benchmark_data_smithi159_102124_object71430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.820758+0000", 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.84340670699999, 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42225933799999998, 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71431 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.660 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820758+0000", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820758+0000", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820759+0000", 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.23e-07 2024-04-15T18:28:41.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820842+0000", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3051000000000002e-05 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820851+0000", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0429999999999996e-06 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820856+0000", 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7250000000000002e-06 2024-04-15T18:28:41.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820917+0000", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0723000000000001e-05 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821023+0000", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105534 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821116+0000", 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3844999999999993e-05 2024-04-15T18:28:41.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.825348+0000", 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042313029999999996 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.242843+0000", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41749507299999999 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.242906+0000", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2924999999999998e-05 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.243017+0000", 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011129300000000001 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.665 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71434 4.f 4:fb21d3aa:::benchmark_data_smithi159_102124_object71433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.820895+0000", 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.84326986299999, 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42206355299999998, 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71434 2024-04-15T18:28:41.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820895+0000", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820895+0000", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.820895+0000", 2024-04-15T18:28:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0200000000000001e-07 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821077+0000", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000181855 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821085+0000", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.028e-06 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821093+0000", 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0900000000000005e-06 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821139+0000", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5917000000000001e-05 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821299+0000", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015943700000000001 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.821457+0000", 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000158455 2024-04-15T18:28:41.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.823877+0000", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002420307 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.826862+0000", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029842380000000002 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.242857+0000", 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41599506600000002 2024-04-15T18:28:41.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.242902+0000", 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5785000000000003e-05 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.242958+0000", 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5673e-05 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71447 4.f 4:fad1ebb5:::benchmark_data_smithi159_102124_object71446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.671 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.830353+0000", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.83381116499999, 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.419934539, 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71447 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830353+0000", 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830353+0000", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830355+0000", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.252e-06 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830414+0000", 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9664999999999999e-05 2024-04-15T18:28:41.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830425+0000", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0543000000000001e-05 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830433+0000", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9130000000000003e-06 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830494+0000", 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1128000000000002e-05 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830648+0000", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000154122 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.830814+0000", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000166019 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.833164+0000", 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023503360000000002 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.834659+0000", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001494401 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250159+0000", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41550052199999998 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250226+0000", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6984999999999996e-05 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250288+0000", 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1653000000000001e-05 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71456 4.f 4:fe79e8ce:::benchmark_data_smithi159_102124_object71455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.838444+0000", 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.82572053500002, 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.4119989, 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.677 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71456 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838444+0000", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838444+0000", 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838445+0000", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.559e-06 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838491+0000", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5432000000000001e-05 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838501+0000", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7149999999999993e-06 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838506+0000", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6019999999999999e-06 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838554+0000", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7395000000000003e-05 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838718+0000", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016425500000000001 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.838884+0000", 2024-04-15T18:28:41.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016664300000000001 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.841469+0000", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025848690000000001 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.841801+0000", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00033137200000000003 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250358+0000", 2024-04-15T18:28:41.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40855733700000002 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250400+0000", 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1909000000000001e-05 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250443+0000", 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2812000000000003e-05 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.682 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71460 4.4 4:27aed11a:::benchmark_data_smithi159_102124_object71459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.839000+0000", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.82516414999998, 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41191076599999998, 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71460 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839000+0000", 2024-04-15T18:28:41.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839000+0000", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839001+0000", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9599999999999998e-07 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839102+0000", 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010076 2024-04-15T18:28:41.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839114+0000", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1868000000000001e-05 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839122+0000", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0330000000000008e-06 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839212+0000", 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0645999999999999e-05 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839344+0000", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013177600000000001 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.839487+0000", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142879 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.841838+0000", 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023510889999999998 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250740+0000", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40890235800000002 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250821+0000", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0437000000000003e-05 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250911+0000", 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0124000000000007e-05 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71463 4.f 4:fc4c3910:::benchmark_data_smithi159_102124_object71462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.842320+0000", 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.82184416500002, 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40826453000000001, 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71463 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842320+0000", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842320+0000", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842322+0000", 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.393e-06 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842395+0000", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3223e-05 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842408+0000", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3540000000000001e-05 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842416+0000", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4189999999999996e-06 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842458+0000", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.193e-05 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842560+0000", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102172 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.842662+0000", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010199100000000001 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.844619+0000", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019565989999999998 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845180+0000", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056153300000000002 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250493+0000", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40531335400000001 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250555+0000", 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1740000000000002e-05 2024-04-15T18:28:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250585+0000", 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9635999999999999e-05 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71466 4.4 4:25683a13:::benchmark_data_smithi159_102124_object71465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.845229+0000", 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.81893522199999, 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40732321500000002, 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.693 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71466 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845229+0000", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845229+0000", 2024-04-15T18:28:41.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845230+0000", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845279+0000", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9465999999999997e-05 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845288+0000", 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6400000000000003e-06 2024-04-15T18:28:41.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845293+0000", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7360000000000001e-06 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845325+0000", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1819000000000001e-05 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845417+0000", 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2467999999999996e-05 2024-04-15T18:28:41.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.845550+0000", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013272199999999999 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.847503+0000", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001953593 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.252387+0000", 2024-04-15T18:28:41.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40488313999999997 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.252456+0000", 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9345999999999996e-05 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.252552+0000", 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6525000000000002e-05 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71475 4.4 4:2ba07cf7:::benchmark_data_smithi159_102124_object71474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.856094+0000", 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.80807073099999, 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39922540499999998, 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71475 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.699 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856094+0000", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856094+0000", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856095+0000", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.269e-06 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.700 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856154+0000", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8836999999999999e-05 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856165+0000", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0842000000000001e-05 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856173+0000", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0669999999999992e-06 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856216+0000", 2024-04-15T18:28:41.701 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3260000000000003e-05 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856332+0000", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116378 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.856433+0000", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100488 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.858256+0000", 2024-04-15T18:28:41.702 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018229990000000001 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.255136+0000", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39688057900000001 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.255206+0000", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9735999999999998e-05 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.255319+0000", 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011294999999999999 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.703 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:71483 4.f 4:f024d482:::benchmark_data_smithi159_102124_object71482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:49.870476+0000", 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.79368878999998, 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38025572699999999, 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71483 2024-04-15T18:28:41.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870476+0000", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870476+0000", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870478+0000", 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1830000000000001e-06 2024-04-15T18:28:41.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870526+0000", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7883000000000002e-05 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870555+0000", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9725000000000001e-05 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870561+0000", 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3149999999999999e-06 2024-04-15T18:28:41.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870618+0000", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7484000000000001e-05 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870732+0000", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113723 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.870838+0000", 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106258 2024-04-15T18:28:41.707 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.872901+0000", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020628090000000001 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:49.873113+0000", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021154400000000001 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250626+0000", 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37751381000000001 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.708 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250692+0000", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5998000000000001e-05 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:50.250731+0000", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8995000000000002e-05 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:89350 4.7 4:eb1182e1:::benchmark_data_smithi159_102124_object89349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:11.431766+0000", 2024-04-15T18:28:41.709 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 390.23239822900001, 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.481191547, 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 89350 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431766+0000", 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431766+0000", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431774+0000", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7649999999999993e-06 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431839+0000", 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4803999999999998e-05 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431847+0000", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7250000000000003e-06 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431855+0000", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1080000000000004e-06 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.431907+0000", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2392999999999998e-05 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.432121+0000", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021374400000000001 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.432264+0000", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142824 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.435350+0000", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030859630000000002 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.435894+0000", 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054427100000000003 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912874+0000", 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47698014799999999 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912921+0000", 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7193e-05 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912958+0000", 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6609000000000001e-05 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25126.0:89350 4.7 e322/284 ondisk, result = 0)", 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:11.436121+0000", 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 390.228042985, 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.476845148, 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:28:41.715 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.436121+0000", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.436121+0000", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.436122+0000", 2024-04-15T18:28:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7800000000000001e-07 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.436127+0000", 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5210000000000004e-06 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912790+0000", 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47666324399999999 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912798+0000", 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7530000000000001e-06 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912862+0000", 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4117999999999998e-05 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912870+0000", 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9249999999999995e-06 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.912967+0000", 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6909000000000003e-05 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:89352 4.d e322/284 4:b8615e71:::benchmark_last_metadata:head v 322'5532, mlcod=322'5530)", 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:12.450577+0000", 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 389.21358770699999, 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84563364600000002, 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.450577+0000", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.450577+0000", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.450578+0000", 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.114e-06 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.450585+0000", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6669999999999996e-06 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.293787+0000", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84320144799999996 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.293793+0000", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5220000000000004e-06 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.721 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.293848+0000", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5016999999999999e-05 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.293867+0000", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8088999999999999e-05 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.296150+0000", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022832030000000001 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:13.296210+0000", 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0585999999999999e-05 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(5.7 393/377 [PushReplyOp(5:e81ba4c9:::benchmark_data_smithi159_116570_object41077:head)])", 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:51.776892+0000", 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 169.88727235100001, 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48804616200000001, 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.776892+0000", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.776892+0000", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.776893+0000", 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8699999999999995e-07 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.776897+0000", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5329999999999996e-06 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.264532+0000", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48763472899999999 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.264539+0000", 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9469999999999997e-06 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.264601+0000", 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2461999999999999e-05 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.264938+0000", 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00033690400000000001 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:75711 5.7 5:e005d75d:::benchmark_data_smithi159_116570_object75710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e394)", 2024-04-15T18:28:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:52.435849+0000", 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 169.22831574700001, 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.82724697400000002, 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 75711 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435849+0000", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435849+0000", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435850+0000", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435894+0000", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4660999999999998e-05 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435905+0000", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0431000000000001e-05 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435910+0000", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2610000000000001e-06 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.435955+0000", 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4573999999999997e-05 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.436067+0000", 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112347 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.436175+0000", 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108242 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.438263+0000", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020881020000000001 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.440823+0000", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002560119 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262808+0000", 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.82198497199999998 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262890+0000", 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1363e-05 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.263096+0000", 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020593400000000001 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25447.0:75711 5.7 e394/377 ondisk, result = 0)", 2024-04-15T18:28:41.732 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:52.438708+0000", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 169.22545673100001, 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.82440576200000004, 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.438708+0000", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.438708+0000", 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.438708+0000", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5799999999999999e-07 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.438712+0000", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9489999999999998e-06 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262381+0000", 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.82366856200000005 2024-04-15T18:28:41.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262389+0000", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2819999999999996e-06 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262784+0000", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039458700000000001 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.262797+0000", 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3691e-05 2024-04-15T18:28:41.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.263113+0000", 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00031603300000000001 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(5.7 394/377 [PushReplyOp(5:ecfedb63:::benchmark_data_smithi159_116570_object42730:head)])", 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:52.705480+0000", 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 168.95868451499999, 2024-04-15T18:28:41.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53924537100000003, 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.705480+0000", 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.705480+0000", 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.705480+0000", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3600000000000004e-07 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:52.705484+0000", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9090000000000002e-06 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.244231+0000", 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53874633199999999 2024-04-15T18:28:41.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.244241+0000", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8530000000000004e-06 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.244369+0000", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012829399999999999 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:53.244725+0000", 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035644700000000002 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.739 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:108048 5.7 e422/396 5:e3958808:::benchmark_data_smithi159_116570_object14888:head v 422'6807, mlcod=422'6804)", 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:52.638283+0000", 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 109.02588188199999, 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50904598400000001, 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:52.638283+0000", 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:52.638283+0000", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:52.638283+0000", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5799999999999998e-07 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:52.638291+0000", 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3699999999999997e-06 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.143820+0000", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50552893600000004 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.143835+0000", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4856e-05 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.143911+0000", 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6223000000000005e-05 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.143948+0000", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7579000000000001e-05 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.147271+0000", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033222519999999999 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.147329+0000", 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.791e-05 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:110536 5.5 5:a6311603:::benchmark_data_smithi159_116570_object17376:head [delete] snapc 0=[] ondisk+write+known_if_redirected e430)", 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:14.574709+0000", 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.089455701000006, 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42036389600000001, 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 110536 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574709+0000", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574709+0000", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574711+0000", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2699999999999999e-06 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574722+0000", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447000000000001e-05 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574752+0000", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9352e-05 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574788+0000", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6396000000000002e-05 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.574900+0000", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011229400000000001 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.575052+0000", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151352 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.575124+0000", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2448999999999997e-05 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.577685+0000", 2024-04-15T18:28:41.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025610870000000001 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.578220+0000", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00053482599999999996 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.994952+0000", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41673173600000002 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.995018+0000", 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6209000000000003e-05 2024-04-15T18:28:41.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.995073+0000", 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4478000000000003e-05 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25447.0:110536 5.5 e430/415 ondisk, result = 0)", 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:14.578094+0000", 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.086070741, 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41698436799999999, 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.749 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.578094+0000", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.578094+0000", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.578094+0000", 2024-04-15T18:28:41.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-07 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.578099+0000", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7040000000000002e-06 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.994874+0000", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.416774963 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.994884+0000", 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9240000000000002e-06 2024-04-15T18:28:41.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.994936+0000", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1719999999999999e-05 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.994942+0000", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1140000000000002e-06 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:14.995078+0000", 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136193 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:113218 5.0 e436/358 5:07ee6a04:::benchmark_data_smithi159_116570_object20058:head v 436'7155, mlcod=436'7153)", 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:34.893468+0000", 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 66.770696420999997, 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.44808916700000001, 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:34.893468+0000", 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:34.893468+0000", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:34.893469+0000", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8999999999999996e-07 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:34.893474+0000", 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6400000000000002e-06 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.338730+0000", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.44525559300000001 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.338738+0000", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0360000000000006e-06 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.338799+0000", 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1048000000000002e-05 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.338818+0000", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8899e-05 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.341526+0000", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027082550000000001 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.341557+0000", 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1106000000000003e-05 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.757 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:28:41.757 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:28:41.757 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:28:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:41.758 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:41.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:41.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:41.886 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:41.905 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:41.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:42.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:28:42.009 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:42.038 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.061 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:42.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:42.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.268 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:42.372 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:42.398 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.419 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:42.523 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:42.549 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.570 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:42.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:42.703 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.725 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:42.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:42.842 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:28:42.856 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:42.876 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:42.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:42.980 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:43.007 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:43.029 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:43.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:43.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:43.138 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:43.163 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:43.183 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:43.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:43.212 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:43.253 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:43.287 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:43.314 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:43.333 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:43.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:43.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:43 smithi082 ceph-mon[44206]: pgmap v1223: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9601/194706 objects misplaced (4.931%); 736 KiB/s, 11 objects/s recovering 2024-04-15T18:28:43.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:43 smithi082 ceph-mon[44206]: osdmap e468: 12 total, 12 up, 12 in 2024-04-15T18:28:43.437 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:43.437 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:43.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:43.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:43 smithi049 ceph-mon[40087]: pgmap v1223: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9601/194706 objects misplaced (4.931%); 736 KiB/s, 11 objects/s recovering 2024-04-15T18:28:43.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:43 smithi049 ceph-mon[40087]: osdmap e468: 12 total, 12 up, 12 in 2024-04-15T18:28:43.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:43.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:43.467 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:43.489 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:43.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:43.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:43.622 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:43 smithi159 ceph-mon[45336]: pgmap v1223: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 133 op/s; 9601/194706 objects misplaced (4.931%); 736 KiB/s, 11 objects/s recovering 2024-04-15T18:28:43.622 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:43 smithi159 ceph-mon[45336]: osdmap e468: 12 total, 12 up, 12 in 2024-04-15T18:28:43.622 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:43.622 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:43.625 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:43.646 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:43.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:43.750 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:43.780 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:44.030 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:44.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:44.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:44.163 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:44.216 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:44.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:44.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:44.349 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:44.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:44 smithi049 ceph-mon[40087]: osdmap e469: 12 total, 12 up, 12 in 2024-04-15T18:28:44.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:44 smithi049 ceph-mon[40087]: 5.0 scrub starts 2024-04-15T18:28:44.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:44 smithi159 ceph-mon[45336]: osdmap e469: 12 total, 12 up, 12 in 2024-04-15T18:28:44.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:44 smithi159 ceph-mon[45336]: 5.0 scrub starts 2024-04-15T18:28:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:44 smithi082 ceph-mon[44206]: osdmap e469: 12 total, 12 up, 12 in 2024-04-15T18:28:44.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:44 smithi082 ceph-mon[44206]: 5.0 scrub starts 2024-04-15T18:28:45.398 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:45.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:45.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:45 smithi049 ceph-mon[40087]: pgmap v1226: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:45.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:45 smithi049 ceph-mon[40087]: 5.0 scrub ok 2024-04-15T18:28:45.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:45.504 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:45.530 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:45.550 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:45.552 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:45 smithi159 ceph-mon[45336]: pgmap v1226: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:45.552 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:45 smithi159 ceph-mon[45336]: 5.0 scrub ok 2024-04-15T18:28:45.552 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:45.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:45.621 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:28:45.640 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:45 smithi082 ceph-mon[44206]: pgmap v1226: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:45.640 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:45 smithi082 ceph-mon[44206]: 5.0 scrub ok 2024-04-15T18:28:45.640 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:45.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:45.684 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:45.832 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:45.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:45.936 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:45.942 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:28:45.942 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:28:45.943 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:28:45.943 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:28:45.964 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:45.986 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:45.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:46.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:46.118 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:46.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:46 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:28:46.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:46 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:28:46.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:46.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:46 smithi049 ceph-mon[40087]: osdmap e470: 12 total, 12 up, 12 in 2024-04-15T18:28:46.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:46 smithi049 ceph-mon[40087]: pgmap v1228: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:46.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:46 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:28:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:46 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:28:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:46 smithi159 ceph-mon[45336]: osdmap e470: 12 total, 12 up, 12 in 2024-04-15T18:28:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:46 smithi159 ceph-mon[45336]: pgmap v1228: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:46.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:46 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:28:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:46 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:28:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2792009271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:46 smithi082 ceph-mon[44206]: osdmap e470: 12 total, 12 up, 12 in 2024-04-15T18:28:46.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:46 smithi082 ceph-mon[44206]: pgmap v1228: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 120 op/s; 9601/193344 objects misplaced (4.966%); 501 KiB/s, 7 objects/s recovering 2024-04-15T18:28:46.699 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:46.701 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:28:46.826 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:46.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:46.931 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:46.957 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:46.977 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:46.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:46.999 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:47.081 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:47.108 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:47.129 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:47.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:47.235 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:47.266 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:47.289 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:47.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:47.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:47.428 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:47.494 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:47 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:47 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:47 smithi082 ceph-mon[44206]: osdmap e471: 12 total, 12 up, 12 in 2024-04-15T18:28:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:47 smithi082 ceph-mon[44206]: osdmap e472: 12 total, 12 up, 12 in 2024-04-15T18:28:47.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3797439086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:47.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:47.529 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:47.531 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:47.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:47.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:47.618 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:47.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:47.644 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:47 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:47 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:47 smithi159 ceph-mon[45336]: osdmap e471: 12 total, 12 up, 12 in 2024-04-15T18:28:47.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:47 smithi159 ceph-mon[45336]: osdmap e472: 12 total, 12 up, 12 in 2024-04-15T18:28:47.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3797439086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:47.660 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:47.672 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:47.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:47.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:47.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:47.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:47.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:47.682 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:47.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:47.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:47 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:47.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:47 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:47.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:47 smithi049 ceph-mon[40087]: osdmap e471: 12 total, 12 up, 12 in 2024-04-15T18:28:47.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:47 smithi049 ceph-mon[40087]: osdmap e472: 12 total, 12 up, 12 in 2024-04-15T18:28:47.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3797439086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:47.698 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:28:47.699 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":472,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:28:47.289267+0000","last_up_change":"2024-04-15T18:25:13.233296+0000","last_in_change":"2024-04-15T18:26:15.392127+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":10,"score_stable":10,"optimal_score":0.30000001192092896,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}},{"pool":5,"pool_name":"unique_pool_3","create_time":"2024-04-15T18:24:28.921306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"366","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":1.1000000238418579,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":400,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":415,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":307,"up_from":361,"up_thru":442,"down_at":308,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6819","nonce":2410603388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6821","nonce":2410603388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6825","nonce":2410603388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6823","nonce":2410603388}]},"public_addr":"172.21.15.49:6819/2410603388","cluster_addr":"172.21.15.49:6821/2410603388","heartbeat_back_addr":"172.21.15.49:6825/2410603388","heartbeat_front_addr":"172.21.15.49:6823/2410603388","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":26,"last_clean_end":177,"up_from":226,"up_thru":415,"down_at":178,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6827","nonce":1036848877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6829","nonce":1036848877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6833","nonce":1036848877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6831","nonce":1036848877}]},"public_addr":"172.21.15.49:6827/1036848877","cluster_addr":"172.21.15.49:6829/1036848877","heartbeat_back_addr":"172.21.15.49:6833/1036848877","heartbeat_front_addr":"172.21.15.49:6831/1036848877","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":441,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":363,"up_from":376,"up_thru":436,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6809","nonce":1630882481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6811","nonce":1630882481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6815","nonce":1630882481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6813","nonce":1630882481}]},"public_addr":"172.21.15.82:6809/1630882481","cluster_addr":"172.21.15.82:6811/1630882481","heartbeat_back_addr":"172.21.15.82:6815/1630882481","heartbeat_front_addr":"172.21.15.82:6813/1630882481","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":415,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6817","nonce":1896143039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6819","nonce":1896143039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6823","nonce":1896143039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1896143039},{"type":"v1","addr":"172.21.15.82:6821","nonce":1896143039}]},"public_addr":"172.21.15.82:6817/1896143039","cluster_addr":"172.21.15.82:6819/1896143039","heartbeat_back_addr":"172.21.15.82:6823/1896143039","heartbeat_front_addr":"172.21.15.82:6821/1896143039","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":393,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":358,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":409,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":440,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":416,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:14.005416+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:27:48.045207+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:27:52.579470+0000","dead_epoch":307},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:42.495637+0000","dead_epoch":177},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:27.284363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:27:47.434383+0000","dead_epoch":363},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:02.232578+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:01.439868+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:27:52.054455+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:27:43.402382+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:46.044243+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:28:12.642775+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.1","osds":[0,11,6]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:28:47.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:47.815 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:48.267 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:48.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:48.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:28:48.370 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:48.395 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:48.415 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:48.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:48.519 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:28:48.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: pgmap v1231: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9462/191589 objects misplaced (4.939%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2337409938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:28:48.551 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:48.578 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:28:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:48.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: pgmap v1231: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9462/191589 objects misplaced (4.939%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2337409938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:28:48.685 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:48.711 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: pgmap v1231: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9462/191589 objects misplaced (4.939%); 746 KiB/s, 11 objects/s recovering 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:48.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2337409938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:28:48.730 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:48.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:48.745 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:28:48.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:48.835 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:28:48.835 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:28:48.859 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:48.884 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:48.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:48.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:49.017 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:49.115 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:49.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:49.126 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:49.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:49.248 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:49.271 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:49.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:49.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:49.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:49 smithi082 ceph-mon[44206]: osdmap e473: 12 total, 12 up, 12 in 2024-04-15T18:28:49.403 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:49.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:28:49.600 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:49.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:49.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:49 smithi049 ceph-mon[40087]: osdmap e473: 12 total, 12 up, 12 in 2024-04-15T18:28:49.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:49.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:49 smithi159 ceph-mon[45336]: osdmap e473: 12 total, 12 up, 12 in 2024-04-15T18:28:49.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:49.734 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:49.872 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:49.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:50.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:50.070 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:50.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:50 smithi082 ceph-mon[44206]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9462/190929 objects misplaced (4.956%); 747 KiB/s, 11 objects/s recovering 2024-04-15T18:28:50.552 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:50.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:50 smithi159 ceph-mon[45336]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9462/190929 objects misplaced (4.956%); 747 KiB/s, 11 objects/s recovering 2024-04-15T18:28:50.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:50.705 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:50.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:50.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:50 smithi049 ceph-mon[40087]: pgmap v1233: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9462/190929 objects misplaced (4.956%); 747 KiB/s, 11 objects/s recovering 2024-04-15T18:28:50.809 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:50.836 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:50.855 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:50.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:50.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:28:50.987 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:51.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:28:51.236 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:28:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:51.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:28:51.338 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:51.340 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:51.366 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:51.387 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:51.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:51.491 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:51.519 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:51.542 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:51.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:51.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:51.675 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:51.798 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:51.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:51.804 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:28:51.804 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:28:51.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:51.929 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:51.954 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:51.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:52.058 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:52.085 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:52.105 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:52.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:52.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:52.241 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:52.327 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:28:52.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:52.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:52.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:52 smithi082 ceph-mon[44206]: osdmap e474: 12 total, 12 up, 12 in 2024-04-15T18:28:52.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:52 smithi082 ceph-mon[44206]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 9320/189981 objects misplaced (4.906%); 586 KiB/s, 9 objects/s recovering 2024-04-15T18:28:52.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:52.516 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:28:52.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:52 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:52.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:52 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:52.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:52 smithi049 ceph-mon[40087]: osdmap e474: 12 total, 12 up, 12 in 2024-04-15T18:28:52.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:52 smithi049 ceph-mon[40087]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 9320/189981 objects misplaced (4.906%); 586 KiB/s, 9 objects/s recovering 2024-04-15T18:28:52.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:52.591 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:52.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:52.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:52.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:52.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:52 smithi159 ceph-mon[45336]: osdmap e474: 12 total, 12 up, 12 in 2024-04-15T18:28:52.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:52 smithi159 ceph-mon[45336]: pgmap v1235: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s; 9320/189981 objects misplaced (4.906%); 586 KiB/s, 9 objects/s recovering 2024-04-15T18:28:52.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:28:52.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:52.725 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:52.823 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:52.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:52.832 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:52.927 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:52.953 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:52.973 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:52.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:53.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:53.107 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:53.128 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:53.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:53.259 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:53.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:53.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:53.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:53.539 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:53.557 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:53.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:53.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:53 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:53.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:53.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:53 smithi159 ceph-mon[45336]: osdmap e475: 12 total, 12 up, 12 in 2024-04-15T18:28:53.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:53.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:53 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:53.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:53.686 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:53 smithi082 ceph-mon[44206]: osdmap e475: 12 total, 12 up, 12 in 2024-04-15T18:28:53.688 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:53.709 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:53.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:53 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:53.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/799732227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:28:53.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:53 smithi049 ceph-mon[40087]: osdmap e475: 12 total, 12 up, 12 in 2024-04-15T18:28:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:53.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:53.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:53.844 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:53.865 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:53.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:53.971 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:53.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:28:54.000 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.010 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:28:54.025 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:54.034 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:28:54.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:28:54.086 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:28:54.112 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:28:54.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:54.158 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.178 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:54.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:54.309 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.330 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:54.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:28:54.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:28:54.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:28:54.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:28:54.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:54.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:28:54.460 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: pgmap v1237: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: osdmap e476: 12 total, 12 up, 12 in 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/659625941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:54.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:54.483 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:54.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:54.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:28:54.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:28:54.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:28:54.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:28:54.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: pgmap v1237: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:54.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: osdmap e476: 12 total, 12 up, 12 in 2024-04-15T18:28:54.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/659625941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:54.613 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:54.615 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.620 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:28:54.620 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:28:54.621 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:28:54.621 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:28:54.637 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:54.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:28:54.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:28:54.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: pgmap v1237: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 172 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:54.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: osdmap e476: 12 total, 12 up, 12 in 2024-04-15T18:28:54.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/659625941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:28:54.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:54.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:54.770 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.789 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:54.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:54.923 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:54.944 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:54.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:55.048 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:55.076 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:55.095 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:55.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:55.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:28:55.230 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:55.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:28:55.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:28:55.479 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:28:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:55.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:28:55.578 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:55.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.611 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:55.633 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:55.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.712 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:55.737 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.763 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:55.765 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:55.818 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:28:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:55.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:28:55.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:28:55.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:55.976 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:28:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:55.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:28:56.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:28:56.110 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:56.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:56.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:56 smithi049 ceph-mon[40087]: pgmap v1239: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:56 smithi159 ceph-mon[45336]: pgmap v1239: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:56.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:28:56.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:56 smithi082 ceph-mon[44206]: pgmap v1239: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 9320/188478 objects misplaced (4.945%); 491 KiB/s, 7 objects/s recovering 2024-04-15T18:28:56.859 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:28:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:56.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:28:56.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:56.992 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:57.191 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:57.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:57.295 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:28:57.324 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:57.344 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:28:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:57.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:28:57.419 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:28:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:28:57.419 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:28:57.420 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:28:57.420 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:28:57.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:57.478 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:57.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:28:57.499 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:57.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:57.603 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:57.628 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:57.648 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:57.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:57.753 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:57.776 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:28:57.781 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:57.800 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:57.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:28:57.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:57.934 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:58.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:58.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:58.147 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:28:58.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:58.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:28:58.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:58.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:28:58.696 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:58.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:58.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:28:58.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:28:58.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:28:58.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:28:58.834 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:58.866 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:28:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:58.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:28:58.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:28:59.001 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:28:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:59.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:28:59.554 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:28:59.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:59 smithi159 ceph-mon[45336]: pgmap v1240: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 9177/186885 objects misplaced (4.911%); 512 KiB/s, 7 objects/s recovering 2024-04-15T18:28:59.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:59 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:59.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:59.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:28:59 smithi159 ceph-mon[45336]: osdmap e477: 12 total, 12 up, 12 in 2024-04-15T18:28:59.580 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:59.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:28:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:59.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:28:59.691 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:28:59.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:28:59.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:59 smithi049 ceph-mon[40087]: pgmap v1240: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 9177/186885 objects misplaced (4.911%); 512 KiB/s, 7 objects/s recovering 2024-04-15T18:28:59.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:59 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:59.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:59.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:28:59 smithi049 ceph-mon[40087]: osdmap e477: 12 total, 12 up, 12 in 2024-04-15T18:28:59.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:59.754 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:28:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:59.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:28:59.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:59 smithi082 ceph-mon[44206]: pgmap v1240: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 171 op/s; 9177/186885 objects misplaced (4.911%); 512 KiB/s, 7 objects/s recovering 2024-04-15T18:28:59.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:59 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:28:59.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2600197722' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:28:59.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:28:59 smithi082 ceph-mon[44206]: osdmap e477: 12 total, 12 up, 12 in 2024-04-15T18:28:59.858 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:28:59.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:28:59.908 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:28:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:28:59.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:00.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:00.047 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:00.068 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:00.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:00.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:00.200 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:00.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:00.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:00.335 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:00.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:00.386 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:00.404 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:00.411 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:00.433 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:00.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:00.462 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:00.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:00.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2140228514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:00.534 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:00.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:00.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:00.570 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:00.594 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:00.596 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:00.597 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:00.597 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:00.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2140228514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:00.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2140228514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:00.672 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:01.356 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: pgmap v1242: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9177/186315 objects misplaced (4.926%); 439 KiB/s, 6 objects/s recovering 2024-04-15T18:29:01.356 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.356 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.356 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:01.357 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.424 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:01.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:01.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:01.561 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:01.634 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:01.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: pgmap v1242: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9177/186315 objects misplaced (4.926%); 439 KiB/s, 6 objects/s recovering 2024-04-15T18:29:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:01.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:01.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.662 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:01.663 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: pgmap v1242: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 9177/186315 objects misplaced (4.926%); 439 KiB/s, 6 objects/s recovering 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:01.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:01.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:01.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:29:01.789 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:01.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:01.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:01.923 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:02.058 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:02.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:29:02.376 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:02.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:02.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:02.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:02 smithi159 ceph-mon[45336]: osdmap e478: 12 total, 12 up, 12 in 2024-04-15T18:29:02.690 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:29:02.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:02.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:02.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:02 smithi049 ceph-mon[40087]: osdmap e478: 12 total, 12 up, 12 in 2024-04-15T18:29:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:02 smithi082 ceph-mon[44206]: osdmap e478: 12 total, 12 up, 12 in 2024-04-15T18:29:03.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:29:03.591 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:03 smithi049 ceph-mon[40087]: pgmap v1243: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9026/185367 objects misplaced (4.869%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:03.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:03 smithi159 ceph-mon[45336]: pgmap v1243: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9026/185367 objects misplaced (4.869%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:03.657 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:03 smithi082 ceph-mon[44206]: pgmap v1243: 17 pgs: 1 active+recovering, 16 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 9026/185367 objects misplaced (4.869%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:03.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:03.878 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:03.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:03.995 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:04.017 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:04.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:04.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:04.149 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:04.282 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:04.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:04.311 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:29:04.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:04.414 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:04.433 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:04.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:04.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:04 smithi049 ceph-mon[40087]: pgmap v1245: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9026/183693 objects misplaced (4.914%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:04.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:04.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:04.570 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:04.575 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:04.575 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:04.589 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:04.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:04.595 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:04 smithi082 ceph-mon[44206]: pgmap v1245: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9026/183693 objects misplaced (4.914%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:04.595 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:04.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:04 smithi159 ceph-mon[45336]: pgmap v1245: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 9026/183693 objects misplaced (4.914%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:04.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:04.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:04.720 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:04.741 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:04.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:04.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:04.875 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:05.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:29:05.325 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:05.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:05.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:05.458 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:05.479 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:05.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:05.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: osdmap e479: 12 total, 12 up, 12 in 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:05.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:05.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:05.598 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:29:05.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:05.631 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:05.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:05.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: osdmap e479: 12 total, 12 up, 12 in 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:05.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/635535608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: osdmap e479: 12 total, 12 up, 12 in 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:29:05.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:29:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:29:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:29:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:29:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:29:05.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:29:05.735 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:05.761 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:05.781 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:05.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:05.885 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:05.915 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:05.939 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:05.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:06.043 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:06.077 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:06.098 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:06.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:06.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:06.239 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:06.340 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:06.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:06.380 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:06.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:06.434 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:06.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:06.461 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:06.472 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:06.486 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:06.501 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:06.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:06.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:06.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:06.570 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:06.595 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:06.606 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:06.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:06.632 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:06 smithi159 ceph-mon[45336]: 5.a scrub starts 2024-04-15T18:29:06.632 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:06 smithi159 ceph-mon[45336]: osdmap e480: 12 total, 12 up, 12 in 2024-04-15T18:29:06.632 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:06 smithi159 ceph-mon[45336]: pgmap v1248: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9026/183693 objects misplaced (4.914%); 544 KiB/s, 8 objects/s recovering 2024-04-15T18:29:06.632 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:06 smithi159 ceph-mon[45336]: 5.a scrub ok 2024-04-15T18:29:06.634 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:06.645 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:06.654 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:06.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:06.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:06.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:06.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:06.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:06.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:06.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:06.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:06.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:06.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:06 smithi049 ceph-mon[40087]: 5.a scrub starts 2024-04-15T18:29:06.699 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:06 smithi049 ceph-mon[40087]: osdmap e480: 12 total, 12 up, 12 in 2024-04-15T18:29:06.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:06 smithi049 ceph-mon[40087]: pgmap v1248: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9026/183693 objects misplaced (4.914%); 544 KiB/s, 8 objects/s recovering 2024-04-15T18:29:06.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:06 smithi049 ceph-mon[40087]: 5.a scrub ok 2024-04-15T18:29:06.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:06 smithi082 ceph-mon[44206]: 5.a scrub starts 2024-04-15T18:29:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:06 smithi082 ceph-mon[44206]: osdmap e480: 12 total, 12 up, 12 in 2024-04-15T18:29:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:06 smithi082 ceph-mon[44206]: pgmap v1248: 17 pgs: 1 active+recovering, 16 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 9026/183693 objects misplaced (4.914%); 544 KiB/s, 8 objects/s recovering 2024-04-15T18:29:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:06 smithi082 ceph-mon[44206]: 5.a scrub ok 2024-04-15T18:29:06.784 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:06.803 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:06.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:06.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:06.941 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: 5.7 scrub starts 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: 5.7 scrub ok 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3817651193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: 5.e scrub starts 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.493 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:07.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:07.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: 5.7 scrub starts 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: 5.7 scrub ok 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3817651193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: 5.e scrub starts 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.624 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.625 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.625 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.625 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:07.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: 5.7 scrub starts 2024-04-15T18:29:07.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: 5.7 scrub ok 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3817651193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: 5.e scrub starts 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:07.650 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:07.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:07.754 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:07.782 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:07.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:07.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:29:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:07.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:07.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:07.935 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:07.957 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:07.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:08.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:08.091 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:08.189 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:08.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:08.216 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:29:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:29:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:29:08.216 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:29:08.216 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:29:08.293 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:08.320 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:08.339 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:08.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:08.443 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:08.471 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:08.490 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:08.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:08.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: 5.6 scrub starts 2024-04-15T18:29:08.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: 5.e scrub ok 2024-04-15T18:29:08.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: pgmap v1249: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 8879/181971 objects misplaced (4.879%); 523 KiB/s, 8 objects/s recovering 2024-04-15T18:29:08.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:08 smithi082 ceph-mon[44206]: 5.6 scrub ok 2024-04-15T18:29:08.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:08.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: 5.6 scrub starts 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: 5.e scrub ok 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: pgmap v1249: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 8879/181971 objects misplaced (4.879%); 523 KiB/s, 8 objects/s recovering 2024-04-15T18:29:08.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:08 smithi049 ceph-mon[40087]: 5.6 scrub ok 2024-04-15T18:29:08.623 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:08.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: 5.6 scrub starts 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: 5.e scrub ok 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: pgmap v1249: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 188 op/s; 8879/181971 objects misplaced (4.879%); 523 KiB/s, 8 objects/s recovering 2024-04-15T18:29:08.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:08 smithi159 ceph-mon[45336]: 5.6 scrub ok 2024-04-15T18:29:08.746 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:29:08.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:08.872 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:08.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:29:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:08.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:08.978 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:09.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:09.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:09.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:09.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:09.159 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:09.214 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:29:09.487 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:09.510 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:09.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:09.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:29:09.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:10.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:10 smithi082 ceph-mon[44206]: 5.f scrub starts 2024-04-15T18:29:10.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:10 smithi082 ceph-mon[44206]: 5.5 scrub starts 2024-04-15T18:29:10.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:10 smithi082 ceph-mon[44206]: 5.f scrub ok 2024-04-15T18:29:10.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:10 smithi082 ceph-mon[44206]: pgmap v1250: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 8879/181368 objects misplaced (4.896%); 409 KiB/s, 6 objects/s recovering 2024-04-15T18:29:10.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:10 smithi082 ceph-mon[44206]: 5.5 scrub ok 2024-04-15T18:29:10.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:10 smithi049 ceph-mon[40087]: 5.f scrub starts 2024-04-15T18:29:10.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:10 smithi049 ceph-mon[40087]: 5.5 scrub starts 2024-04-15T18:29:10.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:10 smithi049 ceph-mon[40087]: 5.f scrub ok 2024-04-15T18:29:10.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:10 smithi049 ceph-mon[40087]: pgmap v1250: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 8879/181368 objects misplaced (4.896%); 409 KiB/s, 6 objects/s recovering 2024-04-15T18:29:10.646 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:10 smithi049 ceph-mon[40087]: 5.5 scrub ok 2024-04-15T18:29:10.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:10 smithi159 ceph-mon[45336]: 5.f scrub starts 2024-04-15T18:29:10.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:10 smithi159 ceph-mon[45336]: 5.5 scrub starts 2024-04-15T18:29:10.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:10 smithi159 ceph-mon[45336]: 5.f scrub ok 2024-04-15T18:29:10.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:10 smithi159 ceph-mon[45336]: pgmap v1250: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 174 op/s; 8879/181368 objects misplaced (4.896%); 409 KiB/s, 6 objects/s recovering 2024-04-15T18:29:10.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:10 smithi159 ceph-mon[45336]: 5.5 scrub ok 2024-04-15T18:29:10.653 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:10.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:10.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:10.786 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:10.805 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:10.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:10.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:10.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:10.961 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:10.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:11.064 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:11.091 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:11.114 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:11.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.24793.0:212946 3.d 3:b1715a3b:::benchmark_data_smithi159_91621_object106470:head [delete] snapc 0=[] ondisk+write+known_if_redirected e275)", 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:29:11.174 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 536.17568167499996, 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29758224900000002, 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.24793", 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2917891321", 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 212946 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003043+0000", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003044+0000", 2024-04-15T18:29:11.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.19e-06 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003050+0000", 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7379999999999999e-06 2024-04-15T18:29:11.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003063+0000", 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2772000000000001e-05 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003072+0000", 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6980000000000004e-06 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003128+0000", 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6187000000000003e-05 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003250+0000", 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122142 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,9", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.003346+0000", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5867999999999998e-05 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008011+0000", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046649550000000001 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.008401+0000", 2024-04-15T18:29:11.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039006300000000001 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300548+0000", 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29214726699999999 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300587+0000", 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9243999999999999e-05 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:20:15.300625+0000", 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8124999999999998e-05 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25126.0:89299 4.0 e322/284 4:078bede3:::benchmark_data_smithi159_102124_object89298:head v 322'5533, mlcod=322'5530)", 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 419.81437878100002, 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53392096499999997, 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364346+0000", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364347+0000", 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.004e-06 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.364416+0000", 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8975999999999998e-05 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895471+0000", 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53105475999999996 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895477+0000", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1619999999999996e-06 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895526+0000", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9008999999999998e-05 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.895549+0000", 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3104000000000001e-05 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898235+0000", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026861400000000001 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:11.898267+0000", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1810000000000002e-05 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(4.6 322/318 [PushOp(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head, version: 292'910, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:61d2f6eb:::benchmark_data_smithi159_102124_object14797:head@292'910, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:29:11.186 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 418.465545835, 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47774007400000001, 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713179+0000", 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713180+0000", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8599999999999997e-07 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:12.713250+0000", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0133000000000001e-05 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189486+0000", 2024-04-15T18:29:11.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47623641300000003 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189505+0000", 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8240000000000002e-05 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189590+0000", 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5097000000000002e-05 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.189600+0000", 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0101e-05 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:22:13.190919+0000", 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001319204 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43697 5.6 5:62c05016:::benchmark_data_smithi159_116570_object43696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.190 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.92014994799999, 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27956367700000001, 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43697 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:29:11.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258575+0000", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258577+0000", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.866e-06 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258720+0000", 2024-04-15T18:29:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014362700000000001 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258737+0000", 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6290000000000002e-05 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258749+0000", 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2758999999999999e-05 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259906+0000", 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011563929999999999 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259918+0000", 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2048000000000001e-05 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.538024+0000", 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27810570899999998 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.538139+0000", 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000114985 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43706 5.6 5:6ce21a97:::benchmark_data_smithi159_116570_object43705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.91995702700001, 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.281325461, 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:29:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43706 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258768+0000", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.13e-07 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258809+0000", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0833e-05 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258818+0000", 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7460000000000006e-06 2024-04-15T18:29:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258825+0000", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0489999999999998e-06 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259943+0000", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011180140000000001 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259950+0000", 2024-04-15T18:29:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6649999999999997e-06 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.539990+0000", 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28003989200000001 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540093+0000", 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103749 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43717 5.6 5:6ddc9ba4:::benchmark_data_smithi159_116570_object43716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.91989028699999, 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29288951800000002, 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43717 2024-04-15T18:29:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258835+0000", 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6899999999999998e-07 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258875+0000", 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9944000000000002e-05 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258887+0000", 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1799e-05 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258894+0000", 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8159999999999996e-06 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259972+0000", 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001078213 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259978+0000", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3330000000000002e-06 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540122+0000", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28014370599999999 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540281+0000", 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000158703 2024-04-15T18:29:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.540483+0000", 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020204700000000001 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545583+0000", 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051007320000000002 2024-04-15T18:29:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545631+0000", 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7271000000000002e-05 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549706+0000", 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040756949999999998 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549764+0000", 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8183999999999997e-05 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551724+0000", 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001959606 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43723 5.6 5:6cb1f38e:::benchmark_data_smithi159_116570_object43722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.91980389400001, 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29697446900000002, 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43723 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258921+0000", 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2600000000000002e-07 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.258993+0000", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1836999999999997e-05 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259010+0000", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6433e-05 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259017+0000", 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.238e-06 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259999+0000", 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098238199999999996 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260007+0000", 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1509999999999998e-06 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542068+0000", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28206113700000002 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542207+0000", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000139204 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.542355+0000", 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014846099999999999 2024-04-15T18:29:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549548+0000", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071922150000000001 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551744+0000", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021962489999999999 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.553782+0000", 2024-04-15T18:29:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002037905 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.553869+0000", 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7721999999999998e-05 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.555895+0000", 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020260090000000001 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43730 5.6 5:6b90e375:::benchmark_data_smithi159_116570_object43729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.919687587, 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29838256400000002, 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43730 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259037+0000", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259038+0000", 2024-04-15T18:29:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259073+0000", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4836999999999999e-05 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259081+0000", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5979999999999997e-06 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259088+0000", 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6329999999999999e-06 2024-04-15T18:29:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260032+0000", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00094387699999999998 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260039+0000", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4540000000000001e-06 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.543916+0000", 2024-04-15T18:29:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28387716299999999 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.544033+0000", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116709 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.544155+0000", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000121719 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549618+0000", 2024-04-15T18:29:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0054628669999999997 2024-04-15T18:29:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551754+0000", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021362159999999998 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.555955+0000", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004200713 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.556023+0000", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8528000000000005e-05 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.557420+0000", 2024-04-15T18:29:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001396771 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:43734 5.6 5:69603c05:::benchmark_data_smithi159_116570_object43733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e378)", 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 235.91962843799999, 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30329634900000002, 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 43734 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259096+0000", 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259097+0000", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2099999999999997e-07 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259124+0000", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7418000000000001e-05 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259132+0000", 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.182e-06 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.259138+0000", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2979999999999997e-06 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260130+0000", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0009926379999999999 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.260141+0000", 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0523000000000001e-05 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545663+0000", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28552172300000001 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545815+0000", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000152566 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.545995+0000", 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000179326 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.549688+0000", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036929290000000002 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.551762+0000", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002074762 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.560176+0000", 2024-04-15T18:29:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0084137169999999994 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.560236+0000", 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9589000000000002e-05 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:15.562393+0000", 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021571569999999998 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:109493 5.9 e427/358 ondisk, result = 0)", 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:06.230448+0000", 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 124.948276728, 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32015722400000002, 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.230448+0000", 2024-04-15T18:29:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.230448+0000", 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.230471+0000", 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3285000000000002e-05 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.230478+0000", 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2670000000000003e-06 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.550463+0000", 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31998520499999999 2024-04-15T18:29:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.550468+0000", 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1089999999999997e-06 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.550509+0000", 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1047999999999997e-05 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.550516+0000", 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8279999999999997e-06 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:06.550605+0000", 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9481999999999995e-05 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:110394 5.9 5:9370e102:::benchmark_data_smithi159_116570_object17234:head [delete] snapc 0=[] ondisk+write+known_if_redirected e428)", 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:13.256916+0000", 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 117.921809075, 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.412149981, 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110394 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256916+0000", 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256916+0000", 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256917+0000", 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.212e-06 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256922+0000", 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4609999999999999e-06 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256935+0000", 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2442999999999999e-05 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256942+0000", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9779999999999999e-06 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.256999+0000", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7365999999999999e-05 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.257137+0000", 2024-04-15T18:29:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013736099999999999 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.257262+0000", 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125077 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.260772+0000", 2024-04-15T18:29:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003510258 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.260944+0000", 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000171628 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.668983+0000", 2024-04-15T18:29:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.408039396 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.669022+0000", 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9446999999999997e-05 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.669066+0000", 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3353999999999999e-05 2024-04-15T18:29:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:110394 5.9 e428/358 ondisk, result = 0)", 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:13.327929+0000", 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 117.850795985, 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34114274999999999, 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.327929+0000", 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.327929+0000", 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.327930+0000", 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.31e-06 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.327939+0000", 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9959999999999999e-06 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.668911+0000", 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34097165499999998 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.668918+0000", 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0990000000000001e-06 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.668960+0000", 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1650999999999998e-05 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.668979+0000", 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8989000000000001e-05 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:13.669072+0000", 2024-04-15T18:29:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3049999999999998e-05 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:110695 5.9 5:9ada869a:::benchmark_data_smithi159_116570_object17535:head [delete] snapc 0=[] ondisk+write+known_if_redirected e431)", 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:16.328677+0000", 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 114.85004810300001, 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28788481100000002, 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110695 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328677+0000", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328677+0000", 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328678+0000", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0189999999999999e-06 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328682+0000", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6829999999999999e-06 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328693+0000", 2024-04-15T18:29:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0560999999999999e-05 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328699+0000", 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7699999999999998e-06 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328739+0000", 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0327000000000002e-05 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328831+0000", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1949999999999999e-05 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.328907+0000", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6054000000000001e-05 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.332067+0000", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003159926 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.332724+0000", 2024-04-15T18:29:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00065693900000000005 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.616489+0000", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28376521999999998 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.616528+0000", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8971999999999999e-05 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.616562+0000", 2024-04-15T18:29:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3389999999999997e-05 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119297 5.9 5:9de7ad19:::benchmark_data_smithi159_116570_object26137:head [delete] snapc 0=[] ondisk+write+known_if_redirected e461)", 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:24.505559+0000", 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.673165763999997, 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41063875999999999, 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119297 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505559+0000", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505559+0000", 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505561+0000", 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5910000000000001e-06 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505568+0000", 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3629999999999996e-06 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505585+0000", 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6575e-05 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505592+0000", 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6850000000000001e-06 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505641+0000", 2024-04-15T18:29:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8476999999999997e-05 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505744+0000", 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103111 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.505822+0000", 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8537999999999999e-05 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.508761+0000", 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029381780000000001 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.508939+0000", 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017828000000000001 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916120+0000", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.40718118599999997 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916159+0000", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8649000000000002e-05 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916198+0000", 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9127e-05 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119299 5.9 5:9bab56a6:::benchmark_data_smithi159_116570_object26139:head [delete] snapc 0=[] ondisk+write+known_if_redirected e461)", 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:24.616691+0000", 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 46.562033673000002, 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29961352899999999, 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119299 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616691+0000", 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616691+0000", 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616692+0000", 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.15e-06 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616698+0000", 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8850000000000004e-06 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616711+0000", 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2638e-05 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616718+0000", 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7499999999999997e-06 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616758+0000", 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9944999999999998e-05 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616846+0000", 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8494999999999999e-05 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.616918+0000", 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2199000000000004e-05 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.625442+0000", 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0085240300000000001 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.625576+0000", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013371600000000001 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916232+0000", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29065578600000003 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916267+0000", 2024-04-15T18:29:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5203999999999997e-05 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:24.916305+0000", 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7731000000000002e-05 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119557 5.9 5:9526d67b:::benchmark_data_smithi159_116570_object26397:head [delete] snapc 0=[] ondisk+write+known_if_redirected e461)", 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.552860486, 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49196453200000001, 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.270 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119557 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625866+0000", 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6190000000000001e-06 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625873+0000", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3540000000000002e-06 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625891+0000", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7382000000000001e-05 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625899+0000", 2024-04-15T18:29:11.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9570000000000002e-06 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625962+0000", 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3751999999999995e-05 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.626087+0000", 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000124306 2024-04-15T18:29:11.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.626200+0000", 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113007 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629266+0000", 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030667070000000001 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.663677+0000", 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.034410145000000003 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117667+0000", 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45399008600000001 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117757+0000", 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0679000000000004e-05 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117829+0000", 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1538000000000005e-05 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:119557 5.9 e461/358 ondisk, result = 0)", 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:26.629237+0000", 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.549487495000001, 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48860060100000002, 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629237+0000", 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629237+0000", 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629238+0000", 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6499999999999998e-07 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629243+0000", 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1030000000000001e-06 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117487+0000", 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48824375599999997 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117501+0000", 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3463e-05 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117644+0000", 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000142974 2024-04-15T18:29:11.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117657+0000", 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3730000000000001e-05 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117838+0000", 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018071 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.282 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119932 5.9 5:9a8210e3:::benchmark_data_smithi159_116570_object26772:head [delete] snapc 0=[] ondisk+write+known_if_redirected e463)", 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:28.757568+0000", 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 42.421156834999998, 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28380547099999998, 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.283 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119932 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757568+0000", 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757568+0000", 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757570+0000", 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5549999999999999e-06 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757575+0000", 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4879999999999998e-06 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757590+0000", 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5133000000000001e-05 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757598+0000", 2024-04-15T18:29:11.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5170000000000002e-06 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757664+0000", 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6230999999999997e-05 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757755+0000", 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1174000000000005e-05 2024-04-15T18:29:11.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.757829+0000", 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3821999999999999e-05 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.760775+0000", 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002946405 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:28.760967+0000", 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019134800000000001 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:29.041293+0000", 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28032613000000001 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:29.041336+0000", 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3577e-05 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:29.041373+0000", 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7091000000000001e-05 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:121945 5.9 5:902d7b06:::benchmark_data_smithi159_116570_object28785:head [delete] snapc 0=[] ondisk+write+known_if_redirected e467)", 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:42.092881+0000", 2024-04-15T18:29:11.290 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 29.085843602000001, 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32409882600000001, 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 121945 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092881+0000", 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092881+0000", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092883+0000", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3489999999999999e-06 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092888+0000", 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8479999999999997e-06 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092902+0000", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3249e-05 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092925+0000", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3373000000000001e-05 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.092975+0000", 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0392000000000001e-05 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.093072+0000", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6453000000000006e-05 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.093153+0000", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1306999999999999e-05 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.096093+0000", 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029399370000000001 2024-04-15T18:29:11.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.096276+0000", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000182456 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.416866+0000", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32059022799999998 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.416925+0000", 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9413999999999997e-05 2024-04-15T18:29:11.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:42.416980+0000", 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.482e-05 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:124434 5.6 e476/437 5:6baf5e4d:::benchmark_data_smithi159_116570_object31274:head v 476'7676, mlcod=476'7673)", 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:58.084653+0000", 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.094071678000001, 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28069772399999998, 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:29:11.296 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.084653+0000", 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.084653+0000", 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:29:11.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.084654+0000", 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.015e-06 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.084663+0000", 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3669999999999997e-06 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.362510+0000", 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27784759599999997 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.362523+0000", 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2466000000000001e-05 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:29:11.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.362619+0000", 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6718000000000005e-05 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.362636+0000", 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6392000000000002e-05 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.365315+0000", 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026788720000000001 2024-04-15T18:29:11.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:58.365351+0000", 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6297999999999997e-05 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:29:11.301 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:29:11.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:11.336 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:11.411 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:11.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:11.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:11.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:11 smithi082 ceph-mon[44206]: 5.b scrub starts 2024-04-15T18:29:11.546 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:11.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:11 smithi159 ceph-mon[45336]: 5.b scrub starts 2024-04-15T18:29:11.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:11 smithi049 ceph-mon[40087]: 5.b scrub starts 2024-04-15T18:29:11.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:29:12.164 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:12.403 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:12.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:12.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:12.562 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:12.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:12.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:12.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:12.706 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:12.713 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:12.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:12.817 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:12.848 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:12.919 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:12.972 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:12.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:13.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:13.106 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:13.294 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: pgmap v1251: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8732/180405 objects misplaced (4.840%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: 5.b scrub ok 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=925513265,config=1059521453,health=1997216375,kv=3881527133}) 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2252278410,logm=4226673239}) 2024-04-15T18:29:13.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805586092}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187974793}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421665515}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308082797}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959187854}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568865316}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388619096}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=524954056}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552327891}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371843793}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2967335004}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588457275}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663628420}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231271659}) 2024-04-15T18:29:13.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2775363298}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946772674}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1255134971}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1446015197}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1713842623}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156514231}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46346791}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340666597}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1895112490}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150876828}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478519442}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282013600}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690032199}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4150618600}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1312694355}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935781058}) 2024-04-15T18:29:13.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595401620}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095860229}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730800151}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2661695526}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545080005}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178979272}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243433096}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1820722614}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407623897}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573689316}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274794176}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4220306987}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520613523}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572776405}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152272076}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495947204}) 2024-04-15T18:29:13.652 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2147711309}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525517439}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21480493}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003867767}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295644523}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3207614590}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555405956}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998114306}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856503294}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11} crc {logm=234505476,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2985177618}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=3963785185,mon_config_key=3253926076}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4124260236}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=3133270369,monmap=1029477071,osd_metadata=3904996289}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=1367162189,osd_pg_creating=4180552595,osdmap=4082341607}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2269748377}) 2024-04-15T18:29:13.653 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=3613183228}) 2024-04-15T18:29:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: pgmap v1251: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8732/180405 objects misplaced (4.840%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: 5.b scrub ok 2024-04-15T18:29:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:13.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=925513265,config=1059521453,health=1997216375,kv=3881527133}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2252278410,logm=4226673239}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805586092}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187974793}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421665515}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308082797}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959187854}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568865316}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388619096}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=524954056}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552327891}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371843793}) 2024-04-15T18:29:13.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2967335004}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588457275}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663628420}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231271659}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2775363298}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946772674}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1255134971}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1446015197}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1713842623}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156514231}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46346791}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340666597}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1895112490}) 2024-04-15T18:29:13.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150876828}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478519442}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282013600}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690032199}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4150618600}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1312694355}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935781058}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595401620}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095860229}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730800151}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2661695526}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545080005}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178979272}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243433096}) 2024-04-15T18:29:13.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1820722614}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407623897}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573689316}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274794176}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4220306987}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520613523}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572776405}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152272076}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495947204}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2147711309}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525517439}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21480493}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003867767}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295644523}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3207614590}) 2024-04-15T18:29:13.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555405956}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998114306}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856503294}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11} crc {logm=234505476,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2985177618}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=3963785185,mon_config_key=3253926076}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4124260236}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=3133270369,monmap=1029477071,osd_metadata=3904996289}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=1367162189,osd_pg_creating=4180552595,osdmap=4082341607}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2269748377}) 2024-04-15T18:29:13.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=3613183228}) 2024-04-15T18:29:13.756 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:13.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: pgmap v1251: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 136 op/s; 8732/180405 objects misplaced (4.840%); 792 KiB/s, 12 objects/s recovering 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: 5.b scrub ok 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=925513265,config=1059521453,health=1997216375,kv=3881527133}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2252278410,logm=4226673239}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805586092}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187974793}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421665515}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308082797}) 2024-04-15T18:29:13.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959187854}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568865316}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388619096}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=524954056}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552327891}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371843793}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2967335004}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588457275}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663628420}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1231271659}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2775363298}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946772674}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1255134971}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1446015197}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1713842623}) 2024-04-15T18:29:13.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156514231}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=46346791}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3340666597}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1895112490}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3150876828}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478519442}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=282013600}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690032199}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4150618600}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1312694355}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2935781058}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595401620}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095860229}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=730800151}) 2024-04-15T18:29:13.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2661695526}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3545080005}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=178979272}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243433096}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1820722614}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=407623897}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573689316}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274794176}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4220306987}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520613523}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572776405}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152272076}) 2024-04-15T18:29:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495947204}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2147711309}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3525517439}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21480493}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2003867767}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295644523}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3207614590}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3555405956}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998114306}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856503294}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=65,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11} crc {logm=234505476,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=2985177618}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=5,mon_config_key=95} crc {mgrstat=3963785185,mon_config_key=3253926076}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4124260236}) 2024-04-15T18:29:13.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=3133270369,monmap=1029477071,osd_metadata=3904996289}) 2024-04-15T18:29:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=1367162189,osd_pg_creating=4180552595,osdmap=4082341607}) 2024-04-15T18:29:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2269748377}) 2024-04-15T18:29:13.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=19} crc {osdmap=3613183228}) 2024-04-15T18:29:13.860 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:13.894 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:14.021 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:14.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:14.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:14.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:14.158 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:14.158 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:29:14.158 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:29:14.158 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:14.159 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:14.179 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1251,"stamp":"2024-04-15T18:29:11.334160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3941401144,"num_objects":60135,"num_object_clones":0,"num_object_copies":180405,"num_objects_missing_on_primary":2906,"num_objects_missing":2906,"num_objects_degraded":0,"num_objects_misplaced":8732,"num_objects_unfound":0,"num_objects_dirty":60135,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":126364,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38769,"num_bytes_recovered":2537839264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128443,"ondisk_log_size":128443,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15668992,"kb_used_data":12097220,"kb_used_omap":177,"kb_used_meta":3571470,"kb_avail":1109272832,"statfs":{"total":1151940427776,"available":1135895379968,"internally_reserved":0,"allocated":12387553280,"data_stored":12380988240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3657185732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-71827456,"num_objects":-1096,"num_object_clones":0,"num_object_copies":-3288,"num_objects_missing_on_primary":-96,"num_objects_missing":-96,"num_objects_degraded":0,"num_objects_misplaced":-294,"num_objects_unfound":0,"num_objects_dirty":-1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001377"},"pg_stats":[{"pgid":"5.f","version":"480'8005","reported_seq":13934,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:06.900571+0000","last_change":"2024-04-15T18:27:48.754996+0000","last_active":"2024-04-15T18:29:06.900571+0000","last_peered":"2024-04-15T18:29:06.900571+0000","last_clean":"2024-04-15T18:29:06.900571+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:06.900571+0000","last_undegraded":"2024-04-15T18:29:06.900571+0000","last_fullsized":"2024-04-15T18:29:06.900571+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'7291","last_scrub_stamp":"2024-04-15T18:27:48.754858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:27:48.754858+0000","objects_scrubbed":4532,"log_size":8005,"log_dups_size":0,"ondisk_log_size":8005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:07.702052+0000","scrub_duration":1045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250150912,"num_objects":3817,"num_object_clones":0,"num_object_copies":11451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8005,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"480'7919","reported_seq":8332,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:07.248554+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:07.248554+0000","last_peered":"2024-04-15T18:29:07.248554+0000","last_clean":"2024-04-15T18:29:07.248554+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:07.248554+0000","last_undegraded":"2024-04-15T18:29:07.248554+0000","last_fullsized":"2024-04-15T18:29:07.248554+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":7919,"log_dups_size":0,"ondisk_log_size":7919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":11319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7919,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"480'7891","reported_seq":13656,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:11.129891+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:11.129891+0000","last_peered":"2024-04-15T18:29:11.129891+0000","last_clean":"2024-04-15T18:29:11.129891+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:11.129891+0000","last_undegraded":"2024-04-15T18:29:11.129891+0000","last_fullsized":"2024-04-15T18:29:11.129891+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243400728,"num_objects":3715,"num_object_clones":0,"num_object_copies":11145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7891,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"480'7852","reported_seq":8277,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:07.204235+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:07.204235+0000","last_peered":"2024-04-15T18:29:07.204235+0000","last_clean":"2024-04-15T18:29:07.204235+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:07.204235+0000","last_undegraded":"2024-04-15T18:29:07.204235+0000","last_fullsized":"2024-04-15T18:29:07.204235+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7852,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"480'7972","reported_seq":13855,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:06.962455+0000","last_change":"2024-04-15T18:27:55.486438+0000","last_active":"2024-04-15T18:29:06.962455+0000","last_peered":"2024-04-15T18:29:06.962455+0000","last_clean":"2024-04-15T18:29:06.962455+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:06.962455+0000","last_undegraded":"2024-04-15T18:29:06.962455+0000","last_fullsized":"2024-04-15T18:29:06.962455+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7304","last_scrub_stamp":"2024-04-15T18:27:55.486283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:27:55.486283+0000","objects_scrubbed":4518,"log_size":7972,"log_dups_size":0,"ondisk_log_size":7972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:09.292052+0000","scrub_duration":643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"480'7879","reported_seq":19934,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:06.764101+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:06.764101+0000","last_peered":"2024-04-15T18:29:06.764101+0000","last_clean":"2024-04-15T18:29:06.764101+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:06.764101+0000","last_undegraded":"2024-04-15T18:29:06.764101+0000","last_fullsized":"2024-04-15T18:29:06.764101+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":7879,"log_dups_size":0,"ondisk_log_size":7879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246874112,"num_objects":3767,"num_object_clones":0,"num_object_copies":11301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7879,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"480'7881","reported_seq":8222,"reported_epoch":480,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:29:10.871658+0000","last_change":"2024-04-15T18:29:05.964020+0000","last_active":"2024-04-15T18:29:10.871658+0000","last_peered":"2024-04-15T18:29:10.871658+0000","last_clean":"2024-04-15T18:29:10.871658+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:10.871658+0000","last_undegraded":"2024-04-15T18:29:10.871658+0000","last_fullsized":"2024-04-15T18:29:10.871658+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":199,"log_size":7881,"log_dups_size":0,"ondisk_log_size":7881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 5s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"480'7952","reported_seq":8207,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:11.139930+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:11.139930+0000","last_peered":"2024-04-15T18:29:11.139930+0000","last_clean":"2024-04-15T18:29:11.139930+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:11.139930+0000","last_undegraded":"2024-04-15T18:29:11.139930+0000","last_fullsized":"2024-04-15T18:29:11.139930+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"480'7935","reported_seq":13902,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:07.205641+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:07.205641+0000","last_peered":"2024-04-15T18:29:07.205641+0000","last_clean":"2024-04-15T18:29:07.205641+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:07.205641+0000","last_undegraded":"2024-04-15T18:29:07.205641+0000","last_fullsized":"2024-04-15T18:29:07.205641+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242548736,"num_objects":3701,"num_object_clones":0,"num_object_copies":11103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"480'7778","reported_seq":24965,"reported_epoch":480,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:29:07.300253+0000","last_change":"2024-04-15T18:29:07.098007+0000","last_active":"2024-04-15T18:29:07.300253+0000","last_peered":"2024-04-15T18:29:07.300253+0000","last_clean":"2024-04-15T18:29:07.300253+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:07.300253+0000","last_undegraded":"2024-04-15T18:29:07.300253+0000","last_fullsized":"2024-04-15T18:29:07.300253+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"447'7112","last_scrub_stamp":"2024-04-15T18:27:49.959117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:27:49.959117+0000","objects_scrubbed":1899,"log_size":7778,"log_dups_size":0,"ondisk_log_size":7778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 0s","scrub_duration":1006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239730688,"num_objects":3658,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"480'7677","reported_seq":8045,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:06.511549+0000","last_change":"2024-04-15T18:27:52.575112+0000","last_active":"2024-04-15T18:29:06.511549+0000","last_peered":"2024-04-15T18:29:06.511549+0000","last_clean":"2024-04-15T18:29:06.511549+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:06.511549+0000","last_undegraded":"2024-04-15T18:29:06.511549+0000","last_fullsized":"2024-04-15T18:29:06.511549+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'7015","last_scrub_stamp":"2024-04-15T18:27:52.575023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:27:52.575023+0000","objects_scrubbed":4301,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354432,"num_objects":3637,"num_object_clones":0,"num_object_copies":10911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"480'8044","reported_seq":13979,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:10.284976+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:10.284976+0000","last_peered":"2024-04-15T18:29:10.284976+0000","last_clean":"2024-04-15T18:29:10.284976+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:10.284976+0000","last_undegraded":"2024-04-15T18:29:10.284976+0000","last_fullsized":"2024-04-15T18:29:10.284976+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8044,"log_dups_size":0,"ondisk_log_size":8044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250347520,"num_objects":3820,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"480'7797","reported_seq":8110,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:10.337744+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:10.337744+0000","last_peered":"2024-04-15T18:29:10.337744+0000","last_clean":"2024-04-15T18:29:10.337744+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:10.337744+0000","last_undegraded":"2024-04-15T18:29:10.337744+0000","last_fullsized":"2024-04-15T18:29:10.337744+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":7797,"log_dups_size":0,"ondisk_log_size":7797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248053760,"num_objects":3785,"num_object_clones":0,"num_object_copies":11355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7797,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"480'7730","reported_seq":16219,"reported_epoch":480,"state":"active+recovering","last_fresh":"2024-04-15T18:29:11.183215+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:11.183215+0000","last_peered":"2024-04-15T18:29:11.183215+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:11.183215+0000","last_undegraded":"2024-04-15T18:29:11.183215+0000","last_fullsized":"2024-04-15T18:29:11.183215+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237502464,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":2906,"num_objects_missing":2906,"num_objects_degraded":0,"num_objects_misplaced":8732,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":134610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":2906},{"shards":"3,4,7,11","objects":7}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"480'7991","reported_seq":8366,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:07.767416+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:07.767416+0000","last_peered":"2024-04-15T18:29:07.767416+0000","last_clean":"2024-04-15T18:29:07.767416+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:07.767416+0000","last_undegraded":"2024-04-15T18:29:07.767416+0000","last_fullsized":"2024-04-15T18:29:07.767416+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":7991,"log_dups_size":0,"ondisk_log_size":7991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:44.509185+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":11463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"480'11704","reported_seq":29283,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:07.955058+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:07.955058+0000","last_peered":"2024-04-15T18:29:07.955058+0000","last_clean":"2024-04-15T18:29:07.955058+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:07.955058+0000","last_undegraded":"2024-04-15T18:29:07.955058+0000","last_fullsized":"2024-04-15T18:29:07.955058+0000","mapping_epoch":440,"log_start":"371'1701","ondisk_log_start":"371'1701","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245628928,"num_objects":3748,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7880,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1198,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:05.348908+0000","last_change":"2024-04-15T18:27:51.655252+0000","last_active":"2024-04-15T18:29:05.348908+0000","last_peered":"2024-04-15T18:29:05.348908+0000","last_clean":"2024-04-15T18:29:05.348908+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:05.348908+0000","last_undegraded":"2024-04-15T18:29:05.348908+0000","last_fullsized":"2024-04-15T18:29:05.348908+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:27:51.655108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:27:51.655108+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:11.099173+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3940810776,"num_objects":60133,"num_object_clones":0,"num_object_copies":180399,"num_objects_missing_on_primary":2906,"num_objects_missing":2906,"num_objects_degraded":0,"num_objects_misplaced":8732,"num_objects_unfound":0,"num_objects_dirty":60133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126183,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38759,"num_bytes_recovered":2535194624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12349943808,"data_stored":12349931592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128306,"ondisk_log_size":128306,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926064,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419324,"kb_used_data":1013300,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92325828,"statfs":{"total":95995035648,"available":94541647872,"internally_reserved":0,"allocated":1037619200,"data_stored":1037066377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187700,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496048,"kb_used_data":1216360,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92249104,"statfs":{"total":95995035648,"available":94463082496,"internally_reserved":0,"allocated":1245552640,"data_stored":1245007461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514745,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055860,"kb_used_data":727592,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92689292,"statfs":{"total":95995035648,"available":94913835008,"internally_reserved":0,"allocated":745054208,"data_stored":744509029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642672,"kb_used_data":481000,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93102480,"statfs":{"total":95995035648,"available":95336939520,"internally_reserved":0,"allocated":492544000,"data_stored":491998821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359612,"kb_used_data":1031348,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92385540,"statfs":{"total":95995035648,"available":94602792960,"internally_reserved":0,"allocated":1056100352,"data_stored":1055547549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463177,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1132664,"kb_used_data":777968,"kb_used_omap":1,"kb_used_meta":354686,"kb_avail":92612488,"statfs":{"total":95995035648,"available":94835187712,"internally_reserved":0,"allocated":796639232,"data_stored":796090501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":363198918},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336248,"kb_used_data":962152,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92408904,"statfs":{"total":95995035648,"available":94626717696,"internally_reserved":0,"allocated":985243648,"data_stored":984698469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2121272,"kb_used_data":1756904,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91623880,"statfs":{"total":95995035648,"available":93822853120,"internally_reserved":0,"allocated":1799069696,"data_stored":1798524517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825908,"kb_used_data":717544,"kb_used_omap":114,"kb_used_meta":108237,"kb_avail":92919244,"statfs":{"total":95995035648,"available":95149305856,"internally_reserved":0,"allocated":734765056,"data_stored":734219877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":110834929},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609043,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653556,"kb_used_data":308008,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091596,"statfs":{"total":95995035648,"available":95325794304,"internally_reserved":0,"allocated":315400192,"data_stored":314855013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986449,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1765488,"kb_used_data":1413608,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91979664,"statfs":{"total":95995035648,"available":94187175936,"internally_reserved":0,"allocated":1447534592,"data_stored":1446989413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703344,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1860340,"kb_used_data":1691436,"kb_used_omap":4,"kb_used_meta":168891,"kb_avail":91884812,"statfs":{"total":95995035648,"available":94090047488,"internally_reserved":0,"allocated":1732030464,"data_stored":1731481213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":172945089},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1792606208,"data_stored":1792606208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":983498752,"data_stored":983498752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":733216768,"data_stored":733216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1442906112,"data_stored":1442906112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1728909312,"data_stored":1728905240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":793051136,"data_stored":793051136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1054085120,"data_stored":1054081048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":489750528,"data_stored":489750528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":741998592,"data_stored":741998592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1241972736,"data_stored":1241972736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1034031104,"data_stored":1034027032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:14.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:14.358 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:14.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:14.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:14 smithi049 ceph-mon[40087]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:14 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:14 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:29:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:14 smithi049 ceph-mon[40087]: osdmap e481: 12 total, 12 up, 12 in 2024-04-15T18:29:14.489 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:14.501 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:14 smithi159 ceph-mon[45336]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:14.501 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:14 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:14.501 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:14 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:29:14.502 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:14.502 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:14 smithi159 ceph-mon[45336]: osdmap e481: 12 total, 12 up, 12 in 2024-04-15T18:29:14.510 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:14.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:14.613 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:14.645 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:14 smithi082 ceph-mon[44206]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:14 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:14 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:29:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2944413787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:14 smithi082 ceph-mon[44206]: osdmap e481: 12 total, 12 up, 12 in 2024-04-15T18:29:14.816 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:14.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:14.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:29:14.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:15.177 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:15.498 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:15.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:15.535 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:15.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:15.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:15 smithi049 ceph-mon[40087]: pgmap v1253: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 209 op/s; 8732/178644 objects misplaced (4.888%); 790 KiB/s, 12 objects/s recovering 2024-04-15T18:29:15.632 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:15.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:15 smithi159 ceph-mon[45336]: pgmap v1253: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 209 op/s; 8732/178644 objects misplaced (4.888%); 790 KiB/s, 12 objects/s recovering 2024-04-15T18:29:15.733 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:15.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:15.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:15 smithi082 ceph-mon[44206]: pgmap v1253: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 209 op/s; 8732/178644 objects misplaced (4.888%); 790 KiB/s, 12 objects/s recovering 2024-04-15T18:29:15.840 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:15.873 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:15.950 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:15.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:16.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:16.080 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:16.099 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:16.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:16.154 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:16.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:16.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:16.207 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:16.230 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:16.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:16.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:16.289 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:16.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:16.347 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:16.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:16.380 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:16.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:16.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:16.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:16.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:16.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:16.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:16.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:16.424 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:16.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:16 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:29:16.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:16 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:29:16.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:16 smithi049 ceph-mon[40087]: pgmap v1254: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 8732/178644 objects misplaced (4.888%); 633 KiB/s, 9 objects/s recovering 2024-04-15T18:29:16.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2615213057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:16.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:16.521 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:16.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:16 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:29:16.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:16 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:29:16.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:16 smithi082 ceph-mon[44206]: pgmap v1254: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 8732/178644 objects misplaced (4.888%); 633 KiB/s, 9 objects/s recovering 2024-04-15T18:29:16.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2615213057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:16 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:29:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:16 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:29:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:16 smithi159 ceph-mon[45336]: pgmap v1254: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 8732/178644 objects misplaced (4.888%); 633 KiB/s, 9 objects/s recovering 2024-04-15T18:29:16.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2615213057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:17.050 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:17.050 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:17.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:29:17.273 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:17.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:17.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:17.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:17 smithi159 ceph-mon[45336]: osdmap e482: 12 total, 12 up, 12 in 2024-04-15T18:29:17.408 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:17 smithi049 ceph-mon[40087]: osdmap e482: 12 total, 12 up, 12 in 2024-04-15T18:29:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:17.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:17 smithi082 ceph-mon[44206]: osdmap e482: 12 total, 12 up, 12 in 2024-04-15T18:29:17.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:17.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:17.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:29:17.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:17.913 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:17.933 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:17.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:18.037 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:18.065 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:18.084 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:18.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:18.124 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:18.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:18.222 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:18.241 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:18.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:18.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:18.372 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:18.391 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:18.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:18.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:18.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:18 smithi049 ceph-mon[40087]: pgmap v1256: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 8586/176712 objects misplaced (4.859%); 776 KiB/s, 12 objects/s recovering 2024-04-15T18:29:18.494 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:18.495 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:18.522 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:18.542 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:18.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:18.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:18.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:18 smithi159 ceph-mon[45336]: pgmap v1256: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 8586/176712 objects misplaced (4.859%); 776 KiB/s, 12 objects/s recovering 2024-04-15T18:29:18.672 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:18.672 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:18 smithi082 ceph-mon[44206]: pgmap v1256: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 8586/176712 objects misplaced (4.859%); 776 KiB/s, 12 objects/s recovering 2024-04-15T18:29:18.675 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:18.695 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:18.697 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:18.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:18.700 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:18.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:18.829 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:19.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:19 smithi049 ceph-mon[40087]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:19.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:19.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:19.622 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:19.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:19 smithi159 ceph-mon[45336]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:19.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:19 smithi082 ceph-mon[44206]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:19.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:29:20.025 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:20.025 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:20.073 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:20.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:20.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:20.209 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:20.388 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:20.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:20.428 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1256,"stamp":"2024-04-15T18:29:17.335594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3860726328,"num_objects":58904,"num_object_clones":0,"num_object_copies":176712,"num_objects_missing_on_primary":2854,"num_objects_missing":2854,"num_objects_degraded":0,"num_objects_misplaced":8586,"num_objects_unfound":0,"num_objects_dirty":58904,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":127595,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38816,"num_bytes_recovered":2540919456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129674,"ondisk_log_size":129674,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15418768,"kb_used_data":11863716,"kb_used_omap":177,"kb_used_meta":3554766,"kb_avail":1109523056,"statfs":{"total":1151940427776,"available":1136151609344,"internally_reserved":0,"allocated":12148445184,"data_stored":12141844024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3640080836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-101711872,"num_objects":-1552,"num_object_clones":0,"num_object_copies":-4656,"num_objects_missing_on_primary":-100,"num_objects_missing":-100,"num_objects_degraded":0,"num_objects_misplaced":-293,"num_objects_unfound":0,"num_objects_dirty":-1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6356992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001762"},"pg_stats":[{"pgid":"5.f","version":"480'8072","reported_seq":14047,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:12.334639+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:29:12.334639+0000","last_peered":"2024-04-15T18:29:12.334639+0000","last_clean":"2024-04-15T18:29:12.334639+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:12.334639+0000","last_undegraded":"2024-04-15T18:29:12.334639+0000","last_fullsized":"2024-04-15T18:29:12.334639+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760000,"num_objects":3750,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"481'8030","reported_seq":8445,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:17.071490+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:17.071490+0000","last_peered":"2024-04-15T18:29:17.071490+0000","last_clean":"2024-04-15T18:29:17.071490+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:17.071490+0000","last_undegraded":"2024-04-15T18:29:17.071490+0000","last_fullsized":"2024-04-15T18:29:17.071490+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239992832,"num_objects":3662,"num_object_clones":0,"num_object_copies":10986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8030,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"481'7950","reported_seq":13717,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:15.742960+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:15.742960+0000","last_peered":"2024-04-15T18:29:15.742960+0000","last_clean":"2024-04-15T18:29:15.742960+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:15.742960+0000","last_undegraded":"2024-04-15T18:29:15.742960+0000","last_fullsized":"2024-04-15T18:29:15.742960+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7950,"log_dups_size":0,"ondisk_log_size":7950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534104,"num_objects":3656,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7950,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"481'7965","reported_seq":8392,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:16.876804+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:16.876804+0000","last_peered":"2024-04-15T18:29:16.876804+0000","last_clean":"2024-04-15T18:29:16.876804+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:16.876804+0000","last_undegraded":"2024-04-15T18:29:16.876804+0000","last_fullsized":"2024-04-15T18:29:16.876804+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":7965,"log_dups_size":0,"ondisk_log_size":7965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244383744,"num_objects":3729,"num_object_clones":0,"num_object_copies":11187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"481'8085","reported_seq":14016,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:17.118156+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:29:17.118156+0000","last_peered":"2024-04-15T18:29:17.118156+0000","last_clean":"2024-04-15T18:29:17.118156+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:17.118156+0000","last_undegraded":"2024-04-15T18:29:17.118156+0000","last_fullsized":"2024-04-15T18:29:17.118156+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8085,"log_dups_size":0,"ondisk_log_size":8085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244776960,"num_objects":3735,"num_object_clones":0,"num_object_copies":11205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"481'7985","reported_seq":20042,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:16.864549+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:16.864549+0000","last_peered":"2024-04-15T18:29:16.864549+0000","last_clean":"2024-04-15T18:29:16.864549+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:16.864549+0000","last_undegraded":"2024-04-15T18:29:16.864549+0000","last_fullsized":"2024-04-15T18:29:16.864549+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239927296,"num_objects":3661,"num_object_clones":0,"num_object_copies":10983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"481'7931","reported_seq":8276,"reported_epoch":481,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:29:15.754458+0000","last_change":"2024-04-15T18:29:05.964020+0000","last_active":"2024-04-15T18:29:15.754458+0000","last_peered":"2024-04-15T18:29:15.754458+0000","last_clean":"2024-04-15T18:29:15.754458+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:15.754458+0000","last_undegraded":"2024-04-15T18:29:15.754458+0000","last_fullsized":"2024-04-15T18:29:15.754458+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":299,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 9s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244514816,"num_objects":3731,"num_object_clones":0,"num_object_copies":11193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"481'8015","reported_seq":8272,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:15.889571+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:15.889571+0000","last_peered":"2024-04-15T18:29:15.889571+0000","last_clean":"2024-04-15T18:29:15.889571+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:15.889571+0000","last_undegraded":"2024-04-15T18:29:15.889571+0000","last_fullsized":"2024-04-15T18:29:15.889571+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244645888,"num_objects":3733,"num_object_clones":0,"num_object_copies":11199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"480'7998","reported_seq":13965,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:12.241936+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:12.241936+0000","last_peered":"2024-04-15T18:29:12.241936+0000","last_clean":"2024-04-15T18:29:12.241936+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:12.241936+0000","last_undegraded":"2024-04-15T18:29:12.241936+0000","last_fullsized":"2024-04-15T18:29:12.241936+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"481'7878","reported_seq":25089,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:16.845090+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:29:16.845090+0000","last_peered":"2024-04-15T18:29:16.845090+0000","last_clean":"2024-04-15T18:29:16.845090+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:16.845090+0000","last_undegraded":"2024-04-15T18:29:16.845090+0000","last_fullsized":"2024-04-15T18:29:16.845090+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"481'7773","reported_seq":8187,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:17.037862+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:29:17.037862+0000","last_peered":"2024-04-15T18:29:17.037862+0000","last_clean":"2024-04-15T18:29:17.037862+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:17.037862+0000","last_undegraded":"2024-04-15T18:29:17.037862+0000","last_fullsized":"2024-04-15T18:29:17.037862+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":7773,"log_dups_size":0,"ondisk_log_size":7773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232062976,"num_objects":3541,"num_object_clones":0,"num_object_copies":10623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7773,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"481'8109","reported_seq":14046,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:15.178766+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:15.178766+0000","last_peered":"2024-04-15T18:29:15.178766+0000","last_clean":"2024-04-15T18:29:15.178766+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:15.178766+0000","last_undegraded":"2024-04-15T18:29:15.178766+0000","last_fullsized":"2024-04-15T18:29:15.178766+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246087680,"num_objects":3755,"num_object_clones":0,"num_object_copies":11265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8109,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"481'7856","reported_seq":8171,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:15.216645+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:15.216645+0000","last_peered":"2024-04-15T18:29:15.216645+0000","last_clean":"2024-04-15T18:29:15.216645+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:15.216645+0000","last_undegraded":"2024-04-15T18:29:15.216645+0000","last_fullsized":"2024-04-15T18:29:15.216645+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7856,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"481'7778","reported_seq":16462,"reported_epoch":481,"state":"active+recovering","last_fresh":"2024-04-15T18:29:16.243231+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:16.243231+0000","last_peered":"2024-04-15T18:29:16.243231+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:16.243231+0000","last_undegraded":"2024-04-15T18:29:16.243231+0000","last_fullsized":"2024-04-15T18:29:16.243231+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7778,"log_dups_size":0,"ondisk_log_size":7778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234356736,"num_objects":3576,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":2854,"num_objects_missing":2854,"num_objects_degraded":0,"num_objects_misplaced":8586,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2103,"num_bytes_recovered":137691136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2854},{"shards":"3,4,7,11","objects":12}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"480'8046","reported_seq":8421,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:12.934097+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:12.934097+0000","last_peered":"2024-04-15T18:29:12.934097+0000","last_clean":"2024-04-15T18:29:12.934097+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:12.934097+0000","last_undegraded":"2024-04-15T18:29:12.934097+0000","last_fullsized":"2024-04-15T18:29:12.934097+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:44.509185+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246808576,"num_objects":3766,"num_object_clones":0,"num_object_copies":11298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"480'11767","reported_seq":29347,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-15T18:29:12.970686+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:12.970686+0000","last_peered":"2024-04-15T18:29:12.970686+0000","last_clean":"2024-04-15T18:29:12.970686+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:12.970686+0000","last_undegraded":"2024-04-15T18:29:12.970686+0000","last_fullsized":"2024-04-15T18:29:12.970686+0000","mapping_epoch":440,"log_start":"371'1701","ondisk_log_start":"371'1701","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241500160,"num_objects":3685,"num_object_clones":0,"num_object_copies":11055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1208,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-15T18:29:15.185306+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:29:15.185306+0000","last_peered":"2024-04-15T18:29:15.185306+0000","last_clean":"2024-04-15T18:29:15.185306+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:15.185306+0000","last_undegraded":"2024-04-15T18:29:15.185306+0000","last_fullsized":"2024-04-15T18:29:15.185306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3860135960,"num_objects":58902,"num_object_clones":0,"num_object_copies":176706,"num_objects_missing_on_primary":2854,"num_objects_missing":2854,"num_objects_degraded":0,"num_objects_misplaced":8586,"num_objects_unfound":0,"num_objects_dirty":58902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127414,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38806,"num_bytes_recovered":2538274816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12113489920,"data_stored":12113477704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129537,"ondisk_log_size":129537,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926065,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1403468,"kb_used_data":997440,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92341684,"statfs":{"total":95995035648,"available":94557884416,"internally_reserved":0,"allocated":1021378560,"data_stored":1020821222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187701,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482480,"kb_used_data":1198632,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92262672,"statfs":{"total":95995035648,"available":94476976128,"internally_reserved":0,"allocated":1227399168,"data_stored":1226853989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514747,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037820,"kb_used_data":705396,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92707332,"statfs":{"total":95995035648,"available":94932307968,"internally_reserved":0,"allocated":722325504,"data_stored":721775810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":340392384},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":634684,"kb_used_data":473012,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93110468,"statfs":{"total":95995035648,"available":95345119232,"internally_reserved":0,"allocated":484364288,"data_stored":483814594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168836,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350280,"kb_used_data":1022016,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92394872,"statfs":{"total":95995035648,"available":94612348928,"internally_reserved":0,"allocated":1046544384,"data_stored":1045987066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463178,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099972,"kb_used_data":766140,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92645180,"statfs":{"total":95995035648,"available":94868664320,"internally_reserved":0,"allocated":784527360,"data_stored":783974114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1313340,"kb_used_data":935092,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92431812,"statfs":{"total":95995035648,"available":94650175488,"internally_reserved":0,"allocated":957534208,"data_stored":956984514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738664,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057472,"kb_used_data":1705588,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91687680,"statfs":{"total":95995035648,"available":93888184320,"internally_reserved":0,"allocated":1746522112,"data_stored":1745972418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193915,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":805180,"kb_used_data":696820,"kb_used_omap":114,"kb_used_meta":108237,"kb_avail":92939972,"statfs":{"total":95995035648,"available":95170531328,"internally_reserved":0,"allocated":713543680,"data_stored":712993986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":110834929},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609044,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653556,"kb_used_data":308008,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091596,"statfs":{"total":95995035648,"available":95325794304,"internally_reserved":0,"allocated":315400192,"data_stored":314855013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986450,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1745584,"kb_used_data":1389544,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91999568,"statfs":{"total":95995035648,"available":94207557632,"internally_reserved":0,"allocated":1422893056,"data_stored":1422347877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703345,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1834932,"kb_used_data":1666028,"kb_used_omap":4,"kb_used_meta":168891,"kb_avail":91910220,"statfs":{"total":95995035648,"available":94116065280,"internally_reserved":0,"allocated":1706012672,"data_stored":1705463421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":172945089},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1742733312,"data_stored":1742733312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":955580416,"data_stored":955580416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":711983104,"data_stored":711983104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1419116544,"data_stored":1419116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1700925440,"data_stored":1700921368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":781582336,"data_stored":781582336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1043206144,"data_stored":1043202072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":481820672,"data_stored":481820672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":719519744,"data_stored":719519744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1224867840,"data_stored":1224867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1018236928,"data_stored":1018232856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:20.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:20.502 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:20.529 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:20.547 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:20.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:20.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:20 smithi049 ceph-mon[40087]: pgmap v1257: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 8586/176109 objects misplaced (4.875%); 376 KiB/s, 5 objects/s recovering 2024-04-15T18:29:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:20 smithi159 ceph-mon[45336]: pgmap v1257: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 8586/176109 objects misplaced (4.875%); 376 KiB/s, 5 objects/s recovering 2024-04-15T18:29:20.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:20.680 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:20.703 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:20.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:20.741 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:29:20.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:20 smithi082 ceph-mon[44206]: pgmap v1257: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 178 op/s; 8586/176109 objects misplaced (4.875%); 376 KiB/s, 5 objects/s recovering 2024-04-15T18:29:20.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:20.840 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:20.909 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:20.915 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:20.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:20.951 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:21.022 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:21.059 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:21.083 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:21.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:21.112 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:21.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:21.215 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:21.334 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:29:21.364 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:21.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:21.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/208760588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:21.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:21.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:21.497 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:21.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/208760588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:21.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:21.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/208760588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:21.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:22.147 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:22.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:22.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:22.278 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:22.300 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:22.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:22.403 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:22.430 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:22.449 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:22.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:22.457 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:22 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:22.457 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:22 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:29:22.457 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:22 smithi159 ceph-mon[45336]: osdmap e483: 12 total, 12 up, 12 in 2024-04-15T18:29:22.457 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:22 smithi159 ceph-mon[45336]: pgmap v1259: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 8437/175140 objects misplaced (4.817%); 768 KiB/s, 11 objects/s recovering 2024-04-15T18:29:22.457 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:22 smithi159 ceph-mon[45336]: osdmap e484: 12 total, 12 up, 12 in 2024-04-15T18:29:22.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:22 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:22 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:29:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:22 smithi049 ceph-mon[40087]: osdmap e483: 12 total, 12 up, 12 in 2024-04-15T18:29:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:22 smithi049 ceph-mon[40087]: pgmap v1259: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 8437/175140 objects misplaced (4.817%); 768 KiB/s, 11 objects/s recovering 2024-04-15T18:29:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:22 smithi049 ceph-mon[40087]: osdmap e484: 12 total, 12 up, 12 in 2024-04-15T18:29:22.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:22 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:22.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:22 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:29:22.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:22 smithi082 ceph-mon[44206]: osdmap e483: 12 total, 12 up, 12 in 2024-04-15T18:29:22.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:22 smithi082 ceph-mon[44206]: pgmap v1259: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 145 op/s; 8437/175140 objects misplaced (4.817%); 768 KiB/s, 11 objects/s recovering 2024-04-15T18:29:22.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:22 smithi082 ceph-mon[44206]: osdmap e484: 12 total, 12 up, 12 in 2024-04-15T18:29:22.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:22.582 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:22.710 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:22.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:22.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:22.847 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:23.047 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:23.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:23.055 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:23.055 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:23.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:23.183 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:23.308 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:23.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:23.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:23.438 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:23.457 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:23.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:23.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:23.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:23.560 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:23.588 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:23.588 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:23.590 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:23.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:23.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:23.788 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:23.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:23.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:29:23.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:24.119 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:24.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:24.138 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:24.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:24.252 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:24.400 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:24.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:24.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:24.533 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:24.554 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:24.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:24.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:29:24.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:24 smithi049 ceph-mon[40087]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 8437/173340 objects misplaced (4.867%); 519 KiB/s, 8 objects/s recovering 2024-04-15T18:29:24.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:24 smithi159 ceph-mon[45336]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 8437/173340 objects misplaced (4.867%); 519 KiB/s, 8 objects/s recovering 2024-04-15T18:29:24.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:24.686 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:24 smithi082 ceph-mon[44206]: pgmap v1261: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 218 op/s; 8437/173340 objects misplaced (4.867%); 519 KiB/s, 8 objects/s recovering 2024-04-15T18:29:24.938 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:24.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:25.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:25.076 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:25.226 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:25.228 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:25.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:25.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:25.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:25.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:25.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:25.434 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:25.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:25.440 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:25.539 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:25.565 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:25.583 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:25.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:25.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2035414412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:25.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:25.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:25 smithi159 ceph-mon[45336]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:25.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:25.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2035414412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:25.711 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:25.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:25 smithi082 ceph-mon[44206]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:25.713 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:25.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2035414412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:25.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:25.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:25 smithi049 ceph-mon[40087]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:25.731 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:25.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:25.836 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:25.861 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:25.881 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:25.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:25.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:26.013 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:26.021 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:26.021 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:29:26.021 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:29:26.021 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:29:26.021 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:26.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 8437/173340 objects misplaced (4.867%); 392 KiB/s, 6 objects/s recovering 2024-04-15T18:29:26.342 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: osdmap e485: 12 total, 12 up, 12 in 2024-04-15T18:29:26.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:26.343 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:26.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 8437/173340 objects misplaced (4.867%); 392 KiB/s, 6 objects/s recovering 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: osdmap e485: 12 total, 12 up, 12 in 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:26.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:26.663 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:26.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:26.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:29:26.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: pgmap v1262: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 8437/173340 objects misplaced (4.867%); 392 KiB/s, 6 objects/s recovering 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: osdmap e485: 12 total, 12 up, 12 in 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:26.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:26.795 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:27.044 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:27.144 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:27.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:27.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:27.276 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:27.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:27 smithi049 ceph-mon[40087]: osdmap e486: 12 total, 12 up, 12 in 2024-04-15T18:29:27.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:27 smithi159 ceph-mon[45336]: osdmap e486: 12 total, 12 up, 12 in 2024-04-15T18:29:27.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:27 smithi082 ceph-mon[44206]: osdmap e486: 12 total, 12 up, 12 in 2024-04-15T18:29:28.304 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1261,"stamp":"2024-04-15T18:29:23.336598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3787063864,"num_objects":57780,"num_object_clones":0,"num_object_copies":173340,"num_objects_missing_on_primary":2803,"num_objects_missing":2803,"num_objects_degraded":0,"num_objects_misplaced":8437,"num_objects_unfound":0,"num_objects_dirty":57780,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":128719,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38865,"num_bytes_recovered":2544130720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130698,"ondisk_log_size":130698,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15187836,"kb_used_data":11632772,"kb_used_omap":177,"kb_used_meta":3554766,"kb_avail":1109753988,"statfs":{"total":1151940427776,"available":1136388083712,"internally_reserved":0,"allocated":11911958528,"data_stored":11905249058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3640080836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-86441984,"num_objects":-1319,"num_object_clones":0,"num_object_copies":-3957,"num_objects_missing_on_primary":-51,"num_objects_missing":-51,"num_objects_degraded":0,"num_objects_misplaced":-149,"num_objects_unfound":0,"num_objects_dirty":-1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":3211264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.038045"},"pg_stats":[{"pgid":"5.f","version":"483'8186","reported_seq":14169,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.301925+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:29:22.301925+0000","last_peered":"2024-04-15T18:29:22.301925+0000","last_clean":"2024-04-15T18:29:22.301925+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:22.301925+0000","last_undegraded":"2024-04-15T18:29:22.301925+0000","last_fullsized":"2024-04-15T18:29:22.301925+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8186,"log_dups_size":0,"ondisk_log_size":8186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238288896,"num_objects":3636,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"483'8092","reported_seq":8511,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.072719+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:22.072719+0000","last_peered":"2024-04-15T18:29:22.072719+0000","last_clean":"2024-04-15T18:29:22.072719+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:22.072719+0000","last_undegraded":"2024-04-15T18:29:22.072719+0000","last_fullsized":"2024-04-15T18:29:22.072719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235929600,"num_objects":3600,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"482'8012","reported_seq":13781,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-15T18:29:21.165125+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:21.165125+0000","last_peered":"2024-04-15T18:29:21.165125+0000","last_clean":"2024-04-15T18:29:21.165125+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:21.165125+0000","last_undegraded":"2024-04-15T18:29:21.165125+0000","last_fullsized":"2024-04-15T18:29:21.165125+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8012,"log_dups_size":0,"ondisk_log_size":8012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235470872,"num_objects":3594,"num_object_clones":0,"num_object_copies":10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8012,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"483'8037","reported_seq":8468,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.188499+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:22.188499+0000","last_peered":"2024-04-15T18:29:22.188499+0000","last_clean":"2024-04-15T18:29:22.188499+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:22.188499+0000","last_undegraded":"2024-04-15T18:29:22.188499+0000","last_fullsized":"2024-04-15T18:29:22.188499+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":8037,"log_dups_size":0,"ondisk_log_size":8037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239665152,"num_objects":3657,"num_object_clones":0,"num_object_copies":10971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"483'8140","reported_seq":14075,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.096718+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:29:22.096718+0000","last_peered":"2024-04-15T18:29:22.096718+0000","last_clean":"2024-04-15T18:29:22.096718+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:22.096718+0000","last_undegraded":"2024-04-15T18:29:22.096718+0000","last_fullsized":"2024-04-15T18:29:22.096718+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8140,"log_dups_size":0,"ondisk_log_size":8140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8140,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"483'8042","reported_seq":20105,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.300141+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:22.300141+0000","last_peered":"2024-04-15T18:29:22.300141+0000","last_clean":"2024-04-15T18:29:22.300141+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:22.300141+0000","last_undegraded":"2024-04-15T18:29:22.300141+0000","last_fullsized":"2024-04-15T18:29:22.300141+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8042,"log_dups_size":0,"ondisk_log_size":8042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236191744,"num_objects":3604,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8042,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"482'7985","reported_seq":8334,"reported_epoch":482,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:29:20.784579+0000","last_change":"2024-04-15T18:29:05.964020+0000","last_active":"2024-04-15T18:29:20.784579+0000","last_peered":"2024-04-15T18:29:20.784579+0000","last_clean":"2024-04-15T18:29:20.784579+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:20.784579+0000","last_undegraded":"2024-04-15T18:29:20.784579+0000","last_fullsized":"2024-04-15T18:29:20.784579+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":424,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 14s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240975872,"num_objects":3677,"num_object_clones":0,"num_object_copies":11031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"482'8070","reported_seq":8329,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-15T18:29:21.254404+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:21.254404+0000","last_peered":"2024-04-15T18:29:21.254404+0000","last_clean":"2024-04-15T18:29:21.254404+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:21.254404+0000","last_undegraded":"2024-04-15T18:29:21.254404+0000","last_fullsized":"2024-04-15T18:29:21.254404+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8070,"log_dups_size":0,"ondisk_log_size":8070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241041408,"num_objects":3678,"num_object_clones":0,"num_object_copies":11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"483'8110","reported_seq":14085,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.301812+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:22.301812+0000","last_peered":"2024-04-15T18:29:22.301812+0000","last_clean":"2024-04-15T18:29:22.301812+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:22.301812+0000","last_undegraded":"2024-04-15T18:29:22.301812+0000","last_fullsized":"2024-04-15T18:29:22.301812+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231079936,"num_objects":3526,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8110,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"483'7941","reported_seq":25158,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.300141+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:29:22.300141+0000","last_peered":"2024-04-15T18:29:22.300141+0000","last_clean":"2024-04-15T18:29:22.300141+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:22.300141+0000","last_undegraded":"2024-04-15T18:29:22.300141+0000","last_fullsized":"2024-04-15T18:29:22.300141+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229048320,"num_objects":3495,"num_object_clones":0,"num_object_copies":10485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"483'7826","reported_seq":8244,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-04-15T18:29:22.031788+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:29:22.031788+0000","last_peered":"2024-04-15T18:29:22.031788+0000","last_clean":"2024-04-15T18:29:22.031788+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:22.031788+0000","last_undegraded":"2024-04-15T18:29:22.031788+0000","last_fullsized":"2024-04-15T18:29:22.031788+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228589568,"num_objects":3488,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"482'8158","reported_seq":14097,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-15T18:29:20.350629+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:20.350629+0000","last_peered":"2024-04-15T18:29:20.350629+0000","last_clean":"2024-04-15T18:29:20.350629+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:20.350629+0000","last_undegraded":"2024-04-15T18:29:20.350629+0000","last_fullsized":"2024-04-15T18:29:20.350629+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"482'7910","reported_seq":8227,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-15T18:29:20.260538+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:20.260538+0000","last_peered":"2024-04-15T18:29:20.260538+0000","last_clean":"2024-04-15T18:29:20.260538+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:20.260538+0000","last_undegraded":"2024-04-15T18:29:20.260538+0000","last_fullsized":"2024-04-15T18:29:20.260538+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240648192,"num_objects":3672,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7910,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"482'7827","reported_seq":16711,"reported_epoch":482,"state":"active+recovering","last_fresh":"2024-04-15T18:29:21.272682+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:21.272682+0000","last_peered":"2024-04-15T18:29:21.272682+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:21.272682+0000","last_undegraded":"2024-04-15T18:29:21.272682+0000","last_fullsized":"2024-04-15T18:29:21.272682+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231145472,"num_objects":3527,"num_object_clones":0,"num_object_copies":10581,"num_objects_missing_on_primary":2803,"num_objects_missing":2803,"num_objects_degraded":0,"num_objects_misplaced":8437,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7827,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2152,"num_bytes_recovered":140902400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2803},{"shards":"3,4,7,11","objects":14}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"484'8158","reported_seq":8541,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:23.012044+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:23.012044+0000","last_peered":"2024-04-15T18:29:23.012044+0000","last_clean":"2024-04-15T18:29:23.012044+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:23.012044+0000","last_undegraded":"2024-04-15T18:29:23.012044+0000","last_fullsized":"2024-04-15T18:29:23.012044+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:44.509185+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239468544,"num_objects":3654,"num_object_clones":0,"num_object_copies":10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"484'11868","reported_seq":29456,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-15T18:29:23.023843+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:23.023843+0000","last_peered":"2024-04-15T18:29:23.023843+0000","last_clean":"2024-04-15T18:29:23.023843+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:23.023843+0000","last_undegraded":"2024-04-15T18:29:23.023843+0000","last_fullsized":"2024-04-15T18:29:23.023843+0000","mapping_epoch":440,"log_start":"371'1801","ondisk_log_start":"371'1801","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234881024,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1210,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-15T18:29:17.299724+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:29:17.299724+0000","last_peered":"2024-04-15T18:29:17.299724+0000","last_clean":"2024-04-15T18:29:17.299724+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:17.299724+0000","last_undegraded":"2024-04-15T18:29:17.299724+0000","last_fullsized":"2024-04-15T18:29:17.299724+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3786473496,"num_objects":57778,"num_object_clones":0,"num_object_copies":173334,"num_objects_missing_on_primary":2803,"num_objects_missing":2803,"num_objects_degraded":0,"num_objects_misplaced":8437,"num_objects_unfound":0,"num_objects_dirty":57778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128538,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38855,"num_bytes_recovered":2541486080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11870220288,"data_stored":11870208072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130561,"ondisk_log_size":130561,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926066,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1393812,"kb_used_data":983628,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92351340,"statfs":{"total":95995035648,"available":94567772160,"internally_reserved":0,"allocated":1007235072,"data_stored":1006673219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187703,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1448868,"kb_used_data":1165016,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92296284,"statfs":{"total":95995035648,"available":94511394816,"internally_reserved":0,"allocated":1192976384,"data_stored":1192413195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009752,"kb_used_data":693964,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92735400,"statfs":{"total":95995035648,"available":94961049600,"internally_reserved":0,"allocated":710619136,"data_stored":710060412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":633544,"kb_used_data":467712,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93111608,"statfs":{"total":95995035648,"available":95346286592,"internally_reserved":0,"allocated":478937088,"data_stored":478382879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168837,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1341908,"kb_used_data":1013644,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92403244,"statfs":{"total":95995035648,"available":94620921856,"internally_reserved":0,"allocated":1037971456,"data_stored":1037409623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463179,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090832,"kb_used_data":757000,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92654320,"statfs":{"total":95995035648,"available":94878023680,"internally_reserved":0,"allocated":775168000,"data_stored":774610239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575144,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1298452,"kb_used_data":920204,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92446700,"statfs":{"total":95995035648,"available":94665420800,"internally_reserved":0,"allocated":942288896,"data_stored":941730172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2035208,"kb_used_data":1679168,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91709944,"statfs":{"total":95995035648,"available":93910982656,"internally_reserved":0,"allocated":1719468032,"data_stored":1718913823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193916,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794840,"kb_used_data":686476,"kb_used_omap":114,"kb_used_meta":108237,"kb_avail":92950312,"statfs":{"total":95995035648,"available":95181119488,"internally_reserved":0,"allocated":702951424,"data_stored":702392700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":110834929},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653592,"kb_used_data":308044,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091560,"statfs":{"total":95995035648,"available":95325757440,"internally_reserved":0,"allocated":315437056,"data_stored":314878332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986452,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698136,"kb_used_data":1342092,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92047016,"statfs":{"total":95995035648,"available":94256144384,"internally_reserved":0,"allocated":1374302208,"data_stored":1373743484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703347,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788892,"kb_used_data":1615824,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":91956260,"statfs":{"total":95995035648,"available":94163210240,"internally_reserved":0,"allocated":1654603776,"data_stored":1654040980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1715404800,"data_stored":1715404800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":939851776,"data_stored":939851776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":701038592,"data_stored":701038592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1369505792,"data_stored":1369505792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1651380224,"data_stored":1651376152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":771620864,"data_stored":771620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1032065024,"data_stored":1032060952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":476184576,"data_stored":476184576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":708182016,"data_stored":708182016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1187774464,"data_stored":1187774464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1003294720,"data_stored":1003290648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:28.306 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.9 is not active+clean 2024-04-15T18:29:28.306 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.9', 'version': "482'7985", 'reported_seq': 8334, 'reported_epoch': 482, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-15T18:29:20.784579+0000', 'last_change': '2024-04-15T18:29:05.964020+0000', 'last_active': '2024-04-15T18:29:20.784579+0000', 'last_peered': '2024-04-15T18:29:20.784579+0000', 'last_clean': '2024-04-15T18:29:20.784579+0000', 'last_became_active': '2024-04-15T18:24:30.247530+0000', 'last_became_peered': '2024-04-15T18:24:30.247530+0000', 'last_unstale': '2024-04-15T18:29:20.784579+0000', 'last_undegraded': '2024-04-15T18:29:20.784579+0000', 'last_fullsized': '2024-04-15T18:29:20.784579+0000', 'mapping_epoch': 358, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "403'5575", 'last_scrub_stamp': '2024-04-15T18:26:10.256241+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:26:10.256241+0000', 'objects_scrubbed': 424, 'log_size': 7985, 'log_dups_size': 0, 'ondisk_log_size': 7985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'scrubbing for 14s', 'scrub_duration': 1829, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 240975872, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 11031, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7985, 'num_write_kb': 373184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6, 11], 'acting': [7, 6, 11], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T18:29:28.306 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.1 is not active+clean 2024-04-15T18:29:28.306 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.1', 'version': "482'7827", 'reported_seq': 16711, 'reported_epoch': 482, 'state': 'active+recovering', 'last_fresh': '2024-04-15T18:29:21.272682+0000', 'last_change': '2024-04-15T18:26:49.574312+0000', 'last_active': '2024-04-15T18:29:21.272682+0000', 'last_peered': '2024-04-15T18:29:21.272682+0000', 'last_clean': '2024-04-15T18:26:00.259169+0000', 'last_became_active': '2024-04-15T18:26:48.920249+0000', 'last_became_peered': '2024-04-15T18:26:48.920249+0000', 'last_unstale': '2024-04-15T18:29:21.272682+0000', 'last_undegraded': '2024-04-15T18:29:21.272682+0000', 'last_fullsized': '2024-04-15T18:29:21.272682+0000', 'mapping_epoch': 416, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'objects_scrubbed': 0, 'log_size': 7827, 'log_dups_size': 0, 'ondisk_log_size': 7827, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 231145472, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 10581, 'num_objects_missing_on_primary': 2803, 'num_objects_missing': 2803, 'num_objects_degraded': 0, 'num_objects_misplaced': 8437, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7827, 'num_write_kb': 363328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2152, 'num_bytes_recovered': 140902400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0, 6], 'acting': [11, 0, 6], 'avail_no_missing': ['3', '4', '7'], 'object_location_counts': [{'shards': '0,6,11', 'objects': 710}, {'shards': '3,4,7', 'objects': 2803}, {'shards': '3,4,7,11', 'objects': 14}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-15T18:29:28.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.24793.0:212948 3.2 e275/235 3:409c861b:::benchmark_data_smithi159_91621_object106472:head v 275'13490, mlcod=242'8024)", 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 554.02410373800001, 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56367495999999995, 2024-04-15T18:29:29.025 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005998+0000", 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.005999+0000", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3680000000000001e-06 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.006010+0000", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0786e-05 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565792+0000", 2024-04-15T18:29:29.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55978132000000003 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565799+0000", 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.255e-06 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565863+0000", 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4552000000000001e-05 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.565893+0000", 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9329999999999999e-05 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569633+0000", 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037404080000000002 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:20:15.569673+0000", 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9941000000000003e-05 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:71486 4.2 4:46c4a592:::benchmark_data_smithi159_102124_object71485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e315)", 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 459.15388598700002, 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.308684452, 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:29:29.030 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 71486 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876216+0000", 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876217+0000", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3880000000000001e-06 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876262+0000", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4391000000000002e-05 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876277+0000", 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5481e-05 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876284+0000", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.374e-06 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876338+0000", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3653000000000003e-05 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876457+0000", 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118548 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.876557+0000", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010054 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878524+0000", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019665110000000002 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:49.878860+0000", 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00033609900000000002 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184840+0000", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30597998300000001 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184876+0000", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5997000000000001e-05 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:21:50.184900+0000", 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4487000000000001e-05 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.035 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25126.0:89342 4.2 4:4b07b843:::benchmark_data_smithi159_102124_object89341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 437.60867077099999, 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36570210600000003, 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25126", 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 89342 2024-04-15T18:29:29.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421431+0000", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421433+0000", 2024-04-15T18:29:29.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.609e-06 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421458+0000", 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5007999999999998e-05 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421472+0000", 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4518999999999999e-05 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421481+0000", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.507e-06 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421547+0000", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5950000000000004e-05 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421686+0000", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013925 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:29:29.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.421818+0000", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013244400000000001 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.428345+0000", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065266689999999997 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.435961+0000", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076156009999999996 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787063+0000", 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.351102101 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787102+0000", 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9351000000000003e-05 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:22:11.787133+0000", 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1096999999999997e-05 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43719 5.b 5.2edd20bb (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 255.74916698499999, 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96247389699999997, 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43719 2024-04-15T18:29:29.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280937+0000", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7749999999999999e-06 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280959+0000", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2347000000000001e-05 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280972+0000", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3123e-05 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280981+0000", 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1179999999999998e-06 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281041+0000", 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9335e-05 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281049+0000", 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7730000000000005e-06 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243354+0000", 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96230429100000003 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243409+0000", 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5135e-05 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43724 5.b 5.c9c3aa8b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:29:29.046 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 255.72497875600001, 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93864302600000005, 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43724 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305125+0000", 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9e-06 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305150+0000", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4640000000000001e-05 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305167+0000", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.694e-05 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305176+0000", 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6220000000000004e-06 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305242+0000", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5919000000000004e-05 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305251+0000", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9779999999999994e-06 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243757+0000", 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93850576200000002 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243766+0000", 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2650000000000002e-06 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43728 5.b 5.b2d2527b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 255.54204487000001, 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75574586300000002, 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.051 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43728 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:29:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488059+0000", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8610000000000001e-06 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488103+0000", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4598000000000002e-05 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488118+0000", 2024-04-15T18:29:29.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4708e-05 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488125+0000", 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7630000000000003e-06 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488178+0000", 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3477999999999999e-05 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:29:29.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488188+0000", 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2890000000000003e-06 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243796+0000", 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75560810199999995 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243803+0000", 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0639999999999996e-06 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43732 5.b 5.9c14c54b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 255.348937651, 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56280661600000004, 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43732 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.056 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681165+0000", 2024-04-15T18:29:29.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.243e-06 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681206+0000", 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0428e-05 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681218+0000", 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2269999999999999e-05 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681225+0000", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4529999999999999e-06 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681272+0000", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7775999999999998e-05 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681280+0000", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.892e-06 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243939+0000", 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56265829300000003 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243971+0000", 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2261000000000002e-05 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 400/377 [PushReplyOp(5:d68de874:::benchmark_data_smithi159_116570_object10690:head)])", 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 207.830398446, 2024-04-15T18:29:29.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37573558899999998, 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199704+0000", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4300000000000002e-07 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199711+0000", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2540000000000003e-06 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574171+0000", 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.374460026 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574179+0000", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1080000000000004e-06 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574267+0000", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8676000000000003e-05 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.575439+0000", 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011717819999999999 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.063 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:82071 5.e 5:768b069e:::benchmark_data_smithi159_116570_object82070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e401)", 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 207.46644851400001, 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84584237600000001, 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 82071 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.064 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563655+0000", 2024-04-15T18:29:29.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0899999999999999e-06 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563684+0000", 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8977999999999999e-05 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563704+0000", 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9874999999999999e-05 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563715+0000", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0266e-05 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563777+0000", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2502999999999994e-05 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563782+0000", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6979999999999997e-06 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565036+0000", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012537850000000001 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565179+0000", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014387199999999999 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565329+0000", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000150004 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568089+0000", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027595150000000001 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.569096+0000", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010070610000000001 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409426+0000", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84032972100000003 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409466+0000", 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0374999999999999e-05 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409496+0000", 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9632999999999999e-05 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:82071 5.e e401/358 ondisk, result = 0)", 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 207.46207118500001, 2024-04-15T18:29:29.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84147125499999997, 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568032+0000", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3790000000000001e-06 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568044+0000", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1654999999999999e-05 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409368+0000", 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.841324552 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409372+0000", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.048e-06 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409416+0000", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3933000000000001e-05 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409422+0000", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6860000000000003e-06 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409502+0000", 2024-04-15T18:29:29.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0001999999999997e-05 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 404/377 [PushReplyOp(5:dd0ee7d7:::benchmark_data_smithi159_116570_object27502:head)])", 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 193.56081210900001, 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39222512599999998, 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.074 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:29:29.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469295+0000", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6469999999999997e-06 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860402+0000", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39110686099999997 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860410+0000", 2024-04-15T18:29:29.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860475+0000", 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5375000000000003e-05 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.861515+0000", 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010396470000000001 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 405/377 [PushReplyOp(5:d997bfb1:::benchmark_data_smithi159_116570_object30349:head)])", 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 192.273014848, 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43079569699999998, 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757088+0000", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7000000000000005e-07 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757092+0000", 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6079999999999998e-06 2024-04-15T18:29:29.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186476+0000", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42938425299999999 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186493+0000", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6646e-05 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186666+0000", 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017301299999999999 2024-04-15T18:29:29.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.187883+0000", 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001216607 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 406/377 [PushReplyOp(5:da19a5f9:::benchmark_data_smithi159_116570_object32780:head)])", 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 191.145333239, 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.322450403, 2024-04-15T18:29:29.081 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884769+0000", 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884770+0000", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.884778+0000", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9130000000000003e-06 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206398+0000", 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32162084699999999 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206407+0000", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1780000000000003e-06 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.206476+0000", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8959000000000002e-05 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:18.207219+0000", 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074334900000000003 2024-04-15T18:29:29.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:108335 5.e 5:78a9cf03:::benchmark_data_smithi159_116570_object15175:head [delete] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:55.493540+0000", 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 153.53656153399999, 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32044329599999999, 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 108335 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493540+0000", 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493540+0000", 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493542+0000", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.469e-06 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493549+0000", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7400000000000004e-06 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493563+0000", 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3818999999999999e-05 2024-04-15T18:29:29.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493571+0000", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8560000000000007e-06 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493641+0000", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9691999999999996e-05 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493790+0000", 2024-04-15T18:29:29.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014906399999999999 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.493925+0000", 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013539199999999999 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.497452+0000", 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035270319999999998 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.497487+0000", 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4255999999999999e-05 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.813879+0000", 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.31639261899999999 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.813937+0000", 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7908999999999997e-05 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:55.813984+0000", 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6448e-05 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118576 5.e 5:782c6d1f:::benchmark_data_smithi159_116570_object25416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.321287108000007, 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.461995184, 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.091 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118576 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708816+0000", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.75e-06 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708824+0000", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1359999999999999e-06 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708842+0000", 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8142e-05 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708852+0000", 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0181e-05 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708919+0000", 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7094000000000005e-05 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709064+0000", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014527 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709192+0000", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012819900000000001 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712786+0000", 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00359377 2024-04-15T18:29:29.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712964+0000", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000177651 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170740+0000", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45777645 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170776+0000", 2024-04-15T18:29:29.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5828000000000003e-05 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170810+0000", 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3713000000000001e-05 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:118576 5.e e459/358 ondisk, result = 0)", 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.317023702, 2024-04-15T18:29:29.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45773745599999999, 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713079+0000", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000001e-07 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713085+0000", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170680+0000", 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45759575699999999 2024-04-15T18:29:29.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170687+0000", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3509999999999998e-06 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170726+0000", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9140000000000001e-05 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170736+0000", 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0000000000000001e-05 2024-04-15T18:29:29.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170816+0000", 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9723999999999998e-05 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118591 5.e 5:79283452:::benchmark_data_smithi159_116570_object25431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.203166233000005, 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.451595888, 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.101 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118591 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826937+0000", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.375e-06 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826944+0000", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2350000000000002e-06 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826959+0000", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5013e-05 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826969+0000", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0023e-05 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827033+0000", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3962999999999997e-05 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827123+0000", 2024-04-15T18:29:29.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9382999999999994e-05 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827201+0000", 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8164000000000006e-05 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830060+0000", 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028591860000000001 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830383+0000", 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032335000000000003 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278451+0000", 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44806768699999999 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278489+0000", 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7829e-05 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278531+0000", 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2679999999999998e-05 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118617 5.e 5:7a2bd32b:::benchmark_data_smithi159_116570_object25457:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.167317+0000", 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.862784703000003, 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33612408700000002, 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.107 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118617 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167317+0000", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167317+0000", 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167318+0000", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.282e-06 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167326+0000", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167342+0000", 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.605e-05 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167350+0000", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4759999999999998e-06 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167413+0000", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2368999999999999e-05 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167505+0000", 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2070000000000004e-05 2024-04-15T18:29:29.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167572+0000", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7677999999999998e-05 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170625+0000", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030528930000000001 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170843+0000", 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00021769700000000001 2024-04-15T18:29:29.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.503369+0000", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33252625600000002 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.503401+0000", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2011000000000002e-05 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.503441+0000", 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9848000000000002e-05 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.112 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:119911 5.e 5:78ed0cf0:::benchmark_data_smithi159_116570_object26751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e463)", 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.401509003999998, 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42595297199999999, 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 119911 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.113 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628594+0000", 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.556e-06 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628601+0000", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628614+0000", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2962999999999999e-05 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628622+0000", 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1240000000000005e-06 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628689+0000", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6351000000000002e-05 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628833+0000", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014472500000000001 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628965+0000", 2024-04-15T18:29:29.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013210300000000001 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632755+0000", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037895329999999999 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632768+0000", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2626000000000001e-05 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054455+0000", 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42168709399999998 2024-04-15T18:29:29.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054508+0000", 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3328999999999997e-05 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054546+0000", 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7734999999999997e-05 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:119911 5.e e463/358 ondisk, result = 0)", 2024-04-15T18:29:29.118 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.398082115999998, 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42253153399999999, 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:29:29.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632021+0000", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632027+0000", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9689999999999999e-06 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054377+0000", 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42235067500000001 2024-04-15T18:29:29.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054386+0000", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4190000000000002e-06 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054439+0000", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3771999999999999e-05 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054448+0000", 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4149999999999999e-06 2024-04-15T18:29:29.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054551+0000", 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103403 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:29.122 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:29.123 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:29.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:29.124 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:29.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:29.255 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:29.280 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:29.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:29.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:29.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:29 smithi159 ceph-mon[45336]: pgmap v1265: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 8283/171612 objects misplaced (4.827%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:29.418 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:29.441 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:29.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:29.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:29 smithi049 ceph-mon[40087]: pgmap v1265: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 8283/171612 objects misplaced (4.827%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:29.556 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:29.583 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:29.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:29:29.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:29 smithi082 ceph-mon[44206]: pgmap v1265: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 8283/171612 objects misplaced (4.827%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:29.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:29.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:29.817 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:29:29.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:29.914 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:29.940 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:29.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:30.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:30.075 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:30.100 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:30.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:30.167 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:30.205 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:30.250 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:30.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:30.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:30.380 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:30.399 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:30.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:30.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:30.530 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:30.555 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:30.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:30.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:30.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:30.690 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:31.221 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:29:31.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:31 smithi159 ceph-mon[45336]: pgmap v1266: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 8283/171021 objects misplaced (4.843%); 473 KiB/s, 7 objects/s recovering 2024-04-15T18:29:31.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:31.439 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:31.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:31.526 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:31.526 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1518,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":487,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":57007,"data_bytes":3736404536,"bytes_used":15404048384,"bytes_avail":1136536379392,"bytes_total":1151940427776,"misplaced_objects":8283,"misplaced_total":171021,"misplaced_ratio":0.048432648622099041,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":484193,"recovering_keys_per_sec":0,"num_objects_recovered":52,"num_bytes_recovered":3407872,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":35,"modified":"2024-04-15T18:27:37.316853+0000","services":{}},"progress_events":{}} 2024-04-15T18:29:31.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:31.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:31 smithi049 ceph-mon[40087]: pgmap v1266: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 8283/171021 objects misplaced (4.843%); 473 KiB/s, 7 objects/s recovering 2024-04-15T18:29:31.556 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:31.577 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:31.607 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:31.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:31.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:31.741 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:31.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:31 smithi082 ceph-mon[44206]: pgmap v1266: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 177 op/s; 8283/171021 objects misplaced (4.843%); 473 KiB/s, 7 objects/s recovering 2024-04-15T18:29:31.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:31.865 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:31.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:31.969 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:32.002 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:130115 5.0 5:003838ad:::benchmark_data_smithi159_116570_object36955:head [delete] snapc 0=[] ondisk+write+known_if_redirected e487)", 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.61922327899999996, 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61926337899999995, 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 130115 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391783+0000", 2024-04-15T18:29:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7489999999999999e-06 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391788+0000", 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1619999999999999e-06 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391803+0000", 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4533e-05 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391812+0000", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2709999999999998e-06 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391866+0000", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391977+0000", 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000110729 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.392083+0000", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106679 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.396079+0000", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039953519999999998 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.396202+0000", 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012359 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:29:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 1 2024-04-15T18:29:32.009 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:32.025 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:32.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:32.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:32.161 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:32.375 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:29:32.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:32.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:32 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:32.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:32.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:32 smithi049 ceph-mon[40087]: osdmap e487: 12 total, 12 up, 12 in 2024-04-15T18:29:32.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1060096336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:29:32.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:32 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:32.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:32.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:32 smithi159 ceph-mon[45336]: osdmap e487: 12 total, 12 up, 12 in 2024-04-15T18:29:32.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1060096336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:29:32.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:29:32.710 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:32.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:32 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2592765897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:32 smithi082 ceph-mon[44206]: osdmap e487: 12 total, 12 up, 12 in 2024-04-15T18:29:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1060096336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:29:32.813 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:32.840 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:32.858 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:32.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:32.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:32.988 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:33.006 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:33.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:33.045 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:33.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:33.137 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:33.341 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:33.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:33.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:33 smithi159 ceph-mon[45336]: pgmap v1268: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 8283/170574 objects misplaced (4.856%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:33.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:33 smithi159 ceph-mon[45336]: osdmap e488: 12 total, 12 up, 12 in 2024-04-15T18:29:33.428 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:33.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:33.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:33 smithi049 ceph-mon[40087]: pgmap v1268: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 8283/170574 objects misplaced (4.856%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:33.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:33 smithi049 ceph-mon[40087]: osdmap e488: 12 total, 12 up, 12 in 2024-04-15T18:29:33.482 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:33.682 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:33.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:33 smithi082 ceph-mon[44206]: pgmap v1268: 17 pgs: 1 active+recovering, 16 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 153 op/s; 8283/170574 objects misplaced (4.856%); 555 KiB/s, 8 objects/s recovering 2024-04-15T18:29:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:33 smithi082 ceph-mon[44206]: osdmap e488: 12 total, 12 up, 12 in 2024-04-15T18:29:33.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:33.817 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:34.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:34 smithi159 ceph-mon[45336]: pgmap v1270: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 8130/168504 objects misplaced (4.825%); 486 KiB/s, 7 objects/s recovering 2024-04-15T18:29:34.667 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:34.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:34 smithi049 ceph-mon[40087]: pgmap v1270: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 8130/168504 objects misplaced (4.825%); 486 KiB/s, 7 objects/s recovering 2024-04-15T18:29:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:34.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:34.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:34 smithi082 ceph-mon[44206]: pgmap v1270: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 186 op/s; 8130/168504 objects misplaced (4.825%); 486 KiB/s, 7 objects/s recovering 2024-04-15T18:29:34.800 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:34.855 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:34.855 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:29:34.855 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:29:34.855 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:29:34.855 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:35.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:29:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.136 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:35.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:35.237 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:35.263 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:35.281 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:35.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:35.414 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:35.437 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:35.536 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:29:35.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:35.608 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:35.635 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:35.739 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:35.767 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:35.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:35.833 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:35.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:35.927 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:35.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:36.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:36.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:36.068 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:36.090 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:36.100 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:36.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:36.123 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:36.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:36.143 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:36.144 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:36.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:36.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:36.216 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:36.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:36.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:36.229 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:36.257 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:36.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:36 smithi049 ceph-mon[40087]: pgmap v1271: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8130/168504 objects misplaced (4.825%); 424 KiB/s, 6 objects/s recovering 2024-04-15T18:29:36.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1310527754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:36.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:36 smithi049 ceph-mon[40087]: from='client.16158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:36.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:36 smithi159 ceph-mon[45336]: pgmap v1271: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8130/168504 objects misplaced (4.825%); 424 KiB/s, 6 objects/s recovering 2024-04-15T18:29:36.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1310527754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:36.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:36 smithi159 ceph-mon[45336]: from='client.16158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:36 smithi082 ceph-mon[44206]: pgmap v1271: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 129 op/s; 8130/168504 objects misplaced (4.825%); 424 KiB/s, 6 objects/s recovering 2024-04-15T18:29:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1310527754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:36 smithi082 ceph-mon[44206]: from='client.16158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:37.108 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:37.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:37.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:37.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.684 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:37.684 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:37.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:38.409 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:29:38.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:38 smithi049 ceph-mon[40087]: pgmap v1272: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 7968/166620 objects misplaced (4.782%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:38.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:38.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:38 smithi159 ceph-mon[45336]: pgmap v1272: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 7968/166620 objects misplaced (4.782%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:38.706 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:38.741 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1271,"stamp":"2024-04-15T18:29:35.338706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3681419832,"num_objects":56168,"num_object_clones":0,"num_object_copies":168504,"num_objects_missing_on_primary":2706,"num_objects_missing":2706,"num_objects_degraded":0,"num_objects_misplaced":8130,"num_objects_unfound":0,"num_objects_dirty":56168,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":130331,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38970,"num_bytes_recovered":2551012000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132210,"ondisk_log_size":132210,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14902824,"kb_used_data":11313064,"kb_used_omap":177,"kb_used_meta":3589454,"kb_avail":1110039000,"statfs":{"total":1151940427776,"available":1136679936000,"internally_reserved":0,"allocated":11584577536,"data_stored":11577639551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3675601348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1,2,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67895296,"num_objects":-1036,"num_object_clones":0,"num_object_copies":-3108,"num_objects_missing_on_primary":-47,"num_objects_missing":-47,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":-1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":3473408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001428"},"pg_stats":[{"pgid":"5.f","version":"488'8303","reported_seq":14293,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.349483+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:29:32.349483+0000","last_peered":"2024-04-15T18:29:32.349483+0000","last_clean":"2024-04-15T18:29:32.349483+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:32.349483+0000","last_undegraded":"2024-04-15T18:29:32.349483+0000","last_fullsized":"2024-04-15T18:29:32.349483+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8303,"log_dups_size":0,"ondisk_log_size":8303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230621184,"num_objects":3519,"num_object_clones":0,"num_object_copies":10557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"487'8184","reported_seq":8611,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.239002+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:32.239002+0000","last_peered":"2024-04-15T18:29:32.239002+0000","last_clean":"2024-04-15T18:29:32.239002+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:32.239002+0000","last_undegraded":"2024-04-15T18:29:32.239002+0000","last_fullsized":"2024-04-15T18:29:32.239002+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8184,"log_dups_size":0,"ondisk_log_size":8184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229900288,"num_objects":3508,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8184,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"486'8100","reported_seq":13879,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:31.317242+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:31.317242+0000","last_peered":"2024-04-15T18:29:31.317242+0000","last_clean":"2024-04-15T18:29:31.317242+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:31.317242+0000","last_undegraded":"2024-04-15T18:29:31.317242+0000","last_fullsized":"2024-04-15T18:29:31.317242+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8100,"log_dups_size":0,"ondisk_log_size":8100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229703704,"num_objects":3506,"num_object_clones":0,"num_object_copies":10518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8100,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"487'8132","reported_seq":8570,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.226505+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:32.226505+0000","last_peered":"2024-04-15T18:29:32.226505+0000","last_clean":"2024-04-15T18:29:32.226505+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:32.226505+0000","last_undegraded":"2024-04-15T18:29:32.226505+0000","last_fullsized":"2024-04-15T18:29:32.226505+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233439232,"num_objects":3562,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"487'8245","reported_seq":14188,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.231020+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:29:32.231020+0000","last_peered":"2024-04-15T18:29:32.231020+0000","last_clean":"2024-04-15T18:29:32.231020+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:32.231020+0000","last_undegraded":"2024-04-15T18:29:32.231020+0000","last_fullsized":"2024-04-15T18:29:32.231020+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234291200,"num_objects":3575,"num_object_clones":0,"num_object_copies":10725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"487'8145","reported_seq":20214,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.253336+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:32.253336+0000","last_peered":"2024-04-15T18:29:32.253336+0000","last_clean":"2024-04-15T18:29:32.253336+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:32.253336+0000","last_undegraded":"2024-04-15T18:29:32.253336+0000","last_fullsized":"2024-04-15T18:29:32.253336+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229441536,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8145,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"486'8083","reported_seq":8440,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-15T18:29:30.887757+0000","last_change":"2024-04-15T18:29:23.869859+0000","last_active":"2024-04-15T18:29:30.887757+0000","last_peered":"2024-04-15T18:29:30.887757+0000","last_clean":"2024-04-15T18:29:30.887757+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:30.887757+0000","last_undegraded":"2024-04-15T18:29:30.887757+0000","last_fullsized":"2024-04-15T18:29:30.887757+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":424,"log_size":8083,"log_dups_size":0,"ondisk_log_size":8083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234553344,"num_objects":3579,"num_object_clones":0,"num_object_copies":10737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8083,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"486'8188","reported_seq":8457,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:31.315544+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:31.315544+0000","last_peered":"2024-04-15T18:29:31.315544+0000","last_clean":"2024-04-15T18:29:31.315544+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:31.315544+0000","last_undegraded":"2024-04-15T18:29:31.315544+0000","last_fullsized":"2024-04-15T18:29:31.315544+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8188,"log_dups_size":0,"ondisk_log_size":8188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233308160,"num_objects":3560,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"487'8197","reported_seq":14181,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.308803+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:32.308803+0000","last_peered":"2024-04-15T18:29:32.308803+0000","last_clean":"2024-04-15T18:29:32.308803+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:32.308803+0000","last_undegraded":"2024-04-15T18:29:32.308803+0000","last_fullsized":"2024-04-15T18:29:32.308803+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225378304,"num_objects":3439,"num_object_clones":0,"num_object_copies":10317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"487'8038","reported_seq":25261,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.033637+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:29:32.033637+0000","last_peered":"2024-04-15T18:29:32.033637+0000","last_clean":"2024-04-15T18:29:32.033637+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:32.033637+0000","last_undegraded":"2024-04-15T18:29:32.033637+0000","last_fullsized":"2024-04-15T18:29:32.033637+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"487'7917","reported_seq":8343,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.206244+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:29:32.206244+0000","last_peered":"2024-04-15T18:29:32.206244+0000","last_clean":"2024-04-15T18:29:32.206244+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:32.206244+0000","last_undegraded":"2024-04-15T18:29:32.206244+0000","last_fullsized":"2024-04-15T18:29:32.206244+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222625792,"num_objects":3397,"num_object_clones":0,"num_object_copies":10191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"486'8253","reported_seq":14200,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-15T18:29:30.005780+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:30.005780+0000","last_peered":"2024-04-15T18:29:30.005780+0000","last_clean":"2024-04-15T18:29:30.005780+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:30.005780+0000","last_undegraded":"2024-04-15T18:29:30.005780+0000","last_fullsized":"2024-04-15T18:29:30.005780+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8253,"log_dups_size":0,"ondisk_log_size":8253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"486'8028","reported_seq":8353,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-15T18:29:30.055759+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:30.055759+0000","last_peered":"2024-04-15T18:29:30.055759+0000","last_clean":"2024-04-15T18:29:30.055759+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:30.055759+0000","last_undegraded":"2024-04-15T18:29:30.055759+0000","last_fullsized":"2024-04-15T18:29:30.055759+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232914944,"num_objects":3554,"num_object_clones":0,"num_object_copies":10662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"486'7932","reported_seq":17238,"reported_epoch":487,"state":"active+recovering","last_fresh":"2024-04-15T18:29:31.331950+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:31.331950+0000","last_peered":"2024-04-15T18:29:31.331950+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:31.331950+0000","last_undegraded":"2024-04-15T18:29:31.331950+0000","last_fullsized":"2024-04-15T18:29:31.331950+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224264192,"num_objects":3422,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":2706,"num_objects_missing":2706,"num_objects_degraded":0,"num_objects_misplaced":8130,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7932,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2257,"num_bytes_recovered":147783680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2706},{"shards":"3,4,7,11","objects":6}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"488'8251","reported_seq":8642,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.907694+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:32.907694+0000","last_peered":"2024-04-15T18:29:32.907694+0000","last_clean":"2024-04-15T18:29:32.907694+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:32.907694+0000","last_undegraded":"2024-04-15T18:29:32.907694+0000","last_fullsized":"2024-04-15T18:29:32.907694+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":8251,"log_dups_size":0,"ondisk_log_size":8251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:44.509185+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233373696,"num_objects":3561,"num_object_clones":0,"num_object_copies":10683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"488'11978","reported_seq":29574,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-04-15T18:29:32.902293+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:32.902293+0000","last_peered":"2024-04-15T18:29:32.902293+0000","last_clean":"2024-04-15T18:29:32.902293+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:32.902293+0000","last_undegraded":"2024-04-15T18:29:32.902293+0000","last_fullsized":"2024-04-15T18:29:32.902293+0000","mapping_epoch":440,"log_start":"372'1901","ondisk_log_start":"372'1901","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227672064,"num_objects":3474,"num_object_clones":0,"num_object_copies":10422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1218,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-04-15T18:29:26.395839+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:29:26.395839+0000","last_peered":"2024-04-15T18:29:26.395839+0000","last_clean":"2024-04-15T18:29:26.395839+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:26.395839+0000","last_undegraded":"2024-04-15T18:29:26.395839+0000","last_fullsized":"2024-04-15T18:29:26.395839+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3680829464,"num_objects":56166,"num_object_clones":0,"num_object_copies":168498,"num_objects_missing_on_primary":2706,"num_objects_missing":2706,"num_objects_degraded":0,"num_objects_misplaced":8130,"num_objects_unfound":0,"num_objects_dirty":56166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130150,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38960,"num_bytes_recovered":2548367360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11545751552,"data_stored":11545739336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132073,"ondisk_log_size":132073,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926068,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367236,"kb_used_data":957052,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92377916,"statfs":{"total":95995035648,"available":94594985984,"internally_reserved":0,"allocated":980021248,"data_stored":979441485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187705,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1435992,"kb_used_data":1129928,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92309160,"statfs":{"total":95995035648,"available":94524579840,"internally_reserved":0,"allocated":1157046272,"data_stored":1156465123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514750,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991624,"kb_used_data":675836,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92753528,"statfs":{"total":95995035648,"available":94979612672,"internally_reserved":0,"allocated":692056064,"data_stored":691479430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611984,"kb_used_data":446152,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93133168,"statfs":{"total":95995035648,"available":95368364032,"internally_reserved":0,"allocated":456859648,"data_stored":456278499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168839,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1325764,"kb_used_data":993340,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92419388,"statfs":{"total":95995035648,"available":94637453312,"internally_reserved":0,"allocated":1017180160,"data_stored":1016600417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":340392394},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463181,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074240,"kb_used_data":736248,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92670912,"statfs":{"total":95995035648,"available":94895013888,"internally_reserved":0,"allocated":753917952,"data_stored":753342281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575146,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1274824,"kb_used_data":896572,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92470328,"statfs":{"total":95995035648,"available":94689615872,"internally_reserved":0,"allocated":918089728,"data_stored":917513094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1990584,"kb_used_data":1634544,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91754568,"statfs":{"total":95995035648,"available":93956677632,"internally_reserved":0,"allocated":1673773056,"data_stored":1673200937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779140,"kb_used_data":666620,"kb_used_omap":114,"kb_used_meta":112397,"kb_avail":92966012,"statfs":{"total":95995035648,"available":95197196288,"internally_reserved":0,"allocated":682618880,"data_stored":682042246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":115094769},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609048,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653640,"kb_used_data":308092,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091512,"statfs":{"total":95995035648,"available":95325708288,"internally_reserved":0,"allocated":315486208,"data_stored":314909574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986454,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1653016,"kb_used_data":1296968,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92092136,"statfs":{"total":95995035648,"available":94302347264,"internally_reserved":0,"allocated":1328095232,"data_stored":1327514083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703349,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744780,"kb_used_data":1571712,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":92000372,"statfs":{"total":95995035648,"available":94208380928,"internally_reserved":0,"allocated":1609433088,"data_stored":1608852382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1669398528,"data_stored":1669398528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":915341312,"data_stored":915341312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":680263680,"data_stored":680263680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1324023808,"data_stored":1324023808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1605111808,"data_stored":1605107736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":751042560,"data_stored":751042560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1013387264,"data_stored":1013383192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":454754304,"data_stored":454754304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":689569792,"data_stored":689569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1153761280,"data_stored":1153761280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":975179776,"data_stored":975175704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:38 smithi082 ceph-mon[44206]: pgmap v1272: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 183 op/s; 7968/166620 objects misplaced (4.782%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:38.795 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:38.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:38.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:38.928 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:38.947 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:38.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:39.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:39.079 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:39.177 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:39.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:39.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:39.314 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:39.334 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:39.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:39.438 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:39.468 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:39 smithi159 ceph-mon[45336]: osdmap e489: 12 total, 12 up, 12 in 2024-04-15T18:29:39.471 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:39.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:29:39.497 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:39.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:39.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:39.627 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:39.645 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:39.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:39.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:39 smithi049 ceph-mon[40087]: osdmap e489: 12 total, 12 up, 12 in 2024-04-15T18:29:39.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:39.767 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:39.778 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:39.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:39 smithi082 ceph-mon[44206]: osdmap e489: 12 total, 12 up, 12 in 2024-04-15T18:29:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:39.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:39.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:39.935 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:40.035 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:40.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:40.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:40.170 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:40.224 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:40.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:40.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:40.355 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:40.374 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:40.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:40.407 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:29:40.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:40.511 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:40.522 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:40.522 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:40.546 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:40 smithi159 ceph-mon[45336]: pgmap v1274: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 7968/166074 objects misplaced (4.798%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:40.546 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:40.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:40 smithi049 ceph-mon[40087]: pgmap v1274: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 7968/166074 objects misplaced (4.798%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:40.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:40.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:40 smithi082 ceph-mon[44206]: pgmap v1274: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 7968/166074 objects misplaced (4.798%); 840 KiB/s, 13 objects/s recovering 2024-04-15T18:29:40.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:29:41.255 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:29:41.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:41.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:41.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:41.428 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:41.451 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:41.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:41.549 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:41.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:41.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:41 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:41.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:41.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:41 smithi049 ceph-mon[40087]: osdmap e490: 12 total, 12 up, 12 in 2024-04-15T18:29:41.584 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:41.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:41 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:41.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:41.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:41 smithi159 ceph-mon[45336]: osdmap e490: 12 total, 12 up, 12 in 2024-04-15T18:29:41.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:41.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:41 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:29:41.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4286860302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:29:41.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:41 smithi082 ceph-mon[44206]: osdmap e490: 12 total, 12 up, 12 in 2024-04-15T18:29:41.783 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:41.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:41.888 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:41.914 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:41.932 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:41.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:42.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:42.064 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:42.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:42.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:42.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:42.226 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:42.350 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:42.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:42.395 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:42.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:42.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:42.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:42.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:42.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:42.483 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:42.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:42.527 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:42.581 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:42.607 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:42.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:42.668 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:43.131 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:43.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:43.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:43.262 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:43.280 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:43.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:43.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:29:43.378 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: pgmap v1276: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7968/165597 objects misplaced (4.812%); 416 KiB/s, 6 objects/s recovering 2024-04-15T18:29:43.378 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3574290556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:43.378 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.379 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:43.414 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:43.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:43.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:43.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: pgmap v1276: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7968/165597 objects misplaced (4.812%); 416 KiB/s, 6 objects/s recovering 2024-04-15T18:29:43.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3574290556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:43.566 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:43.584 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:43.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:43.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: pgmap v1276: 17 pgs: 1 active+recovering, 16 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7968/165597 objects misplaced (4.812%); 416 KiB/s, 6 objects/s recovering 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3574290556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:43.676 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:43.676 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:29:43.676 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:29:43.677 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:29:43.677 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:43.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:43.717 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:44.367 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:44.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:44.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:29:44.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:44.514 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:44.537 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:44.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:44.643 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:44.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.674 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:44.700 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:44.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:44.743 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:44.745 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:44.748 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:29:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:44.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:45.116 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:45.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:45 smithi049 ceph-mon[40087]: pgmap v1277: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 7804/163713 objects misplaced (4.767%); 848 KiB/s, 13 objects/s recovering 2024-04-15T18:29:45.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:45 smithi049 ceph-mon[40087]: from='client.16176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:45.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:45 smithi159 ceph-mon[45336]: pgmap v1277: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 7804/163713 objects misplaced (4.767%); 848 KiB/s, 13 objects/s recovering 2024-04-15T18:29:45.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:45 smithi159 ceph-mon[45336]: from='client.16176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:45.780 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1277,"stamp":"2024-04-15T18:29:43.340170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3576758840,"num_objects":54571,"num_object_clones":0,"num_object_copies":163713,"num_objects_missing_on_primary":2593,"num_objects_missing":2593,"num_objects_degraded":0,"num_objects_misplaced":7804,"num_objects_unfound":0,"num_objects_dirty":54571,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":131928,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39076,"num_bytes_recovered":2557958816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133707,"ondisk_log_size":133707,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14594560,"kb_used_data":11017280,"kb_used_omap":177,"kb_used_meta":3576974,"kb_avail":1110347264,"statfs":{"total":1151940427776,"available":1136995598336,"internally_reserved":0,"allocated":11281694720,"data_stored":11274603825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3662821828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-104660992,"num_objects":-1597,"num_object_clones":0,"num_object_copies":-4791,"num_objects_missing_on_primary":-113,"num_objects_missing":-113,"num_objects_degraded":0,"num_objects_misplaced":-326,"num_objects_unfound":0,"num_objects_dirty":-1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1597,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":6946816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001464"},"pg_stats":[{"pgid":"5.f","version":"490'8401","reported_seq":14397,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.214777+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:29:42.214777+0000","last_peered":"2024-04-15T18:29:42.214777+0000","last_clean":"2024-04-15T18:29:42.214777+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:42.214777+0000","last_undegraded":"2024-04-15T18:29:42.214777+0000","last_fullsized":"2024-04-15T18:29:42.214777+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224198656,"num_objects":3421,"num_object_clones":0,"num_object_copies":10263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"490'8280","reported_seq":8713,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.193638+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:42.193638+0000","last_peered":"2024-04-15T18:29:42.193638+0000","last_clean":"2024-04-15T18:29:42.193638+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:42.193638+0000","last_undegraded":"2024-04-15T18:29:42.193638+0000","last_fullsized":"2024-04-15T18:29:42.193638+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223608832,"num_objects":3412,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"490'8204","reported_seq":13989,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:41.060335+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:41.060335+0000","last_peered":"2024-04-15T18:29:41.060335+0000","last_clean":"2024-04-15T18:29:41.060335+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:41.060335+0000","last_undegraded":"2024-04-15T18:29:41.060335+0000","last_fullsized":"2024-04-15T18:29:41.060335+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8204,"log_dups_size":0,"ondisk_log_size":8204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222887960,"num_objects":3402,"num_object_clones":0,"num_object_copies":10206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8204,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"490'8254","reported_seq":8699,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.104463+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:42.104463+0000","last_peered":"2024-04-15T18:29:42.104463+0000","last_clean":"2024-04-15T18:29:42.104463+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:42.104463+0000","last_undegraded":"2024-04-15T18:29:42.104463+0000","last_fullsized":"2024-04-15T18:29:42.104463+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":8254,"log_dups_size":0,"ondisk_log_size":8254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225443840,"num_objects":3440,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8254,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"490'8337","reported_seq":14286,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.292536+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:29:42.292536+0000","last_peered":"2024-04-15T18:29:42.292536+0000","last_clean":"2024-04-15T18:29:42.292536+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:42.292536+0000","last_undegraded":"2024-04-15T18:29:42.292536+0000","last_fullsized":"2024-04-15T18:29:42.292536+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8337,"log_dups_size":0,"ondisk_log_size":8337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228261888,"num_objects":3483,"num_object_clones":0,"num_object_copies":10449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8337,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"490'8253","reported_seq":20328,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.160110+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:42.160110+0000","last_peered":"2024-04-15T18:29:42.160110+0000","last_clean":"2024-04-15T18:29:42.160110+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:42.160110+0000","last_undegraded":"2024-04-15T18:29:42.160110+0000","last_fullsized":"2024-04-15T18:29:42.160110+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8253,"log_dups_size":0,"ondisk_log_size":8253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222363648,"num_objects":3393,"num_object_clones":0,"num_object_copies":10179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"490'8184","reported_seq":8549,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:40.658221+0000","last_change":"2024-04-15T18:29:23.869859+0000","last_active":"2024-04-15T18:29:40.658221+0000","last_peered":"2024-04-15T18:29:40.658221+0000","last_clean":"2024-04-15T18:29:40.658221+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:40.658221+0000","last_undegraded":"2024-04-15T18:29:40.658221+0000","last_fullsized":"2024-04-15T18:29:40.658221+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":424,"log_size":8184,"log_dups_size":0,"ondisk_log_size":8184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:42.057959+0000","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227934208,"num_objects":3478,"num_object_clones":0,"num_object_copies":10434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8184,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"490'8272","reported_seq":8547,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:41.035698+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:41.035698+0000","last_peered":"2024-04-15T18:29:41.035698+0000","last_clean":"2024-04-15T18:29:41.035698+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:41.035698+0000","last_undegraded":"2024-04-15T18:29:41.035698+0000","last_fullsized":"2024-04-15T18:29:41.035698+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8272,"log_dups_size":0,"ondisk_log_size":8272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"490'8304","reported_seq":14292,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.195355+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:42.195355+0000","last_peered":"2024-04-15T18:29:42.195355+0000","last_clean":"2024-04-15T18:29:42.195355+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:42.195355+0000","last_undegraded":"2024-04-15T18:29:42.195355+0000","last_fullsized":"2024-04-15T18:29:42.195355+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"490'8132","reported_seq":25361,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.200692+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:29:42.200692+0000","last_peered":"2024-04-15T18:29:42.200692+0000","last_clean":"2024-04-15T18:29:42.200692+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:42.200692+0000","last_undegraded":"2024-04-15T18:29:42.200692+0000","last_fullsized":"2024-04-15T18:29:42.200692+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216530944,"num_objects":3304,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"490'8011","reported_seq":8443,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.115913+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:29:42.115913+0000","last_peered":"2024-04-15T18:29:42.115913+0000","last_clean":"2024-04-15T18:29:42.115913+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:42.115913+0000","last_undegraded":"2024-04-15T18:29:42.115913+0000","last_fullsized":"2024-04-15T18:29:42.115913+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216465408,"num_objects":3303,"num_object_clones":0,"num_object_copies":9909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"489'8366","reported_seq":14319,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-04-15T18:29:40.230339+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:40.230339+0000","last_peered":"2024-04-15T18:29:40.230339+0000","last_clean":"2024-04-15T18:29:40.230339+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:40.230339+0000","last_undegraded":"2024-04-15T18:29:40.230339+0000","last_fullsized":"2024-04-15T18:29:40.230339+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8366,"log_dups_size":0,"ondisk_log_size":8366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229244928,"num_objects":3498,"num_object_clones":0,"num_object_copies":10494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8366,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"489'8126","reported_seq":8457,"reported_epoch":489,"state":"active+clean","last_fresh":"2024-04-15T18:29:40.219583+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:40.219583+0000","last_peered":"2024-04-15T18:29:40.219583+0000","last_clean":"2024-04-15T18:29:40.219583+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:40.219583+0000","last_undegraded":"2024-04-15T18:29:40.219583+0000","last_fullsized":"2024-04-15T18:29:40.219583+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"490'8037","reported_seq":17781,"reported_epoch":490,"state":"active+recovering","last_fresh":"2024-04-15T18:29:41.408398+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:41.408398+0000","last_peered":"2024-04-15T18:29:41.408398+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:41.408398+0000","last_undegraded":"2024-04-15T18:29:41.408398+0000","last_fullsized":"2024-04-15T18:29:41.408398+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8037,"log_dups_size":0,"ondisk_log_size":8037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":2593,"num_objects_missing":2593,"num_objects_degraded":0,"num_objects_misplaced":7804,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2363,"num_bytes_recovered":154730496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,3,4,7,11","objects":1},{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":2593},{"shards":"3,4,7,11","objects":12}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"490'8342","reported_seq":8737,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.616011+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:42.616011+0000","last_peered":"2024-04-15T18:29:42.616011+0000","last_clean":"2024-04-15T18:29:42.616011+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:42.616011+0000","last_undegraded":"2024-04-15T18:29:42.616011+0000","last_fullsized":"2024-04-15T18:29:42.616011+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":8342,"log_dups_size":0,"ondisk_log_size":8342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:44.509185+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227409920,"num_objects":3470,"num_object_clones":0,"num_object_copies":10410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"490'12068","reported_seq":29668,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:42.603644+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:42.603644+0000","last_peered":"2024-04-15T18:29:42.603644+0000","last_clean":"2024-04-15T18:29:42.603644+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:42.603644+0000","last_undegraded":"2024-04-15T18:29:42.603644+0000","last_fullsized":"2024-04-15T18:29:42.603644+0000","mapping_epoch":440,"log_start":"372'2001","ondisk_log_start":"372'2001","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221773824,"num_objects":3384,"num_object_clones":0,"num_object_copies":10152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1226,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:40.412474+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:29:40.412474+0000","last_peered":"2024-04-15T18:29:40.412474+0000","last_clean":"2024-04-15T18:29:40.412474+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:40.412474+0000","last_undegraded":"2024-04-15T18:29:40.412474+0000","last_fullsized":"2024-04-15T18:29:40.412474+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3576168472,"num_objects":54569,"num_object_clones":0,"num_object_copies":163707,"num_objects_missing_on_primary":2593,"num_objects_missing":2593,"num_objects_degraded":0,"num_objects_misplaced":7804,"num_objects_unfound":0,"num_objects_dirty":54569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131747,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39066,"num_bytes_recovered":2555314176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11239698432,"data_stored":11239686216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133570,"ondisk_log_size":133570,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345592,"kb_used_data":931244,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92399560,"statfs":{"total":95995035648,"available":94617149440,"internally_reserved":0,"allocated":953593856,"data_stored":952996083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187707,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1405548,"kb_used_data":1099488,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92339604,"statfs":{"total":95995035648,"available":94555754496,"internally_reserved":0,"allocated":1125875712,"data_stored":1125285583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514752,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974120,"kb_used_data":658336,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92771032,"statfs":{"total":95995035648,"available":94997536768,"internally_reserved":0,"allocated":674136064,"data_stored":673545935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605916,"kb_used_data":440084,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93139236,"statfs":{"total":95995035648,"available":95374577664,"internally_reserved":0,"allocated":450646016,"data_stored":450060402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168841,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289388,"kb_used_data":973600,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92455764,"statfs":{"total":95995035648,"available":94674702336,"internally_reserved":0,"allocated":996966400,"data_stored":996373162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463183,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055588,"kb_used_data":717596,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92689564,"statfs":{"total":95995035648,"available":94914113536,"internally_reserved":0,"allocated":734818304,"data_stored":734229138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252588,"kb_used_data":870176,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92492564,"statfs":{"total":95995035648,"available":94712385536,"internally_reserved":0,"allocated":891060224,"data_stored":890470095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1950764,"kb_used_data":1590560,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91794388,"statfs":{"total":95995035648,"available":93997453312,"internally_reserved":0,"allocated":1628733440,"data_stored":1628143311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193920,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760360,"kb_used_data":647840,"kb_used_omap":114,"kb_used_meta":112397,"kb_avail":92984792,"statfs":{"total":95995035648,"available":95216427008,"internally_reserved":0,"allocated":663388160,"data_stored":662798031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":115094769},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609050,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653680,"kb_used_data":308128,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091472,"statfs":{"total":95995035648,"available":95325667328,"internally_reserved":0,"allocated":315523072,"data_stored":314932943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611944,"kb_used_data":1251744,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92133208,"statfs":{"total":95995035648,"available":94344404992,"internally_reserved":0,"allocated":1281785856,"data_stored":1281195727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703351,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689072,"kb_used_data":1528484,"kb_used_omap":4,"kb_used_meta":160571,"kb_avail":92056080,"statfs":{"total":95995035648,"available":94265425920,"internally_reserved":0,"allocated":1565167616,"data_stored":1564573415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":164425409},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1625096192,"data_stored":1625096192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":887947264,"data_stored":887947264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":661127168,"data_stored":661127168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1276903424,"data_stored":1276903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1559171072,"data_stored":1559167000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":730791936,"data_stored":730791936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":993726464,"data_stored":993722392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":447807488,"data_stored":447807488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":670302208,"data_stored":670302208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1122435072,"data_stored":1122435072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":950472704,"data_stored":950468632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:45.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:29:45.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:45 smithi082 ceph-mon[44206]: pgmap v1277: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 7804/163713 objects misplaced (4.767%); 848 KiB/s, 13 objects/s recovering 2024-04-15T18:29:45.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:45 smithi082 ceph-mon[44206]: from='client.16176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:45.792 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:45.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:45.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:45.926 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:45.948 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:45.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:46.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:46.083 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:46.280 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:46.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:46.385 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:46.411 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:46.430 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:46.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:46.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:46 smithi049 ceph-mon[40087]: pgmap v1278: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7804/163713 objects misplaced (4.767%); 432 KiB/s, 6 objects/s recovering 2024-04-15T18:29:46.534 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:46.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:46 smithi159 ceph-mon[45336]: pgmap v1278: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7804/163713 objects misplaced (4.767%); 432 KiB/s, 6 objects/s recovering 2024-04-15T18:29:46.568 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:46.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:46.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:46.630 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:43707 5.f e375/364)", 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 273.39833952100003, 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.94185888299999998, 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237680+0000", 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6700000000000003e-07 2024-04-15T18:29:46.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237786+0000", 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010644100000000001 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179372+0000", 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.94158575 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179378+0000", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6400000000000002e-06 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179521+0000", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014270600000000001 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179538+0000", 2024-04-15T18:29:46.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7578999999999999e-05 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:75530 5.d 5:b48816ed:::benchmark_data_smithi159_116570_object75529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e393)", 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:51.560160+0000", 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 235.07585856200001, 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53966530199999996, 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:46.635 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 75530 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560160+0000", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560160+0000", 2024-04-15T18:29:46.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560161+0000", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.78e-07 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560179+0000", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7844000000000001e-05 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560189+0000", 2024-04-15T18:29:46.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8700000000000004e-06 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560194+0000", 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5180000000000004e-06 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560232+0000", 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7449e-05 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560337+0000", 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010555100000000001 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.560439+0000", 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010153 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.562486+0000", 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020470520000000002 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:51.562931+0000", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044480799999999999 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.099744+0000", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53681334599999997 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.099780+0000", 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5961999999999999e-05 2024-04-15T18:29:46.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.099826+0000", 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5393999999999999e-05 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 422/416 [PushOp(5:8704fa45:::benchmark_data_smithi159_116570_object14951:head, version: 369'921, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8704fa45:::benchmark_data_smithi159_116570_object14951:head@369'921, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 173.58140452699999, 2024-04-15T18:29:46.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.79326008699999995, 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:29:46.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054615+0000", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054649+0000", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3844999999999999e-05 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.792384+0000", 2024-04-15T18:29:46.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.73773477899999995 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.792405+0000", 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0985999999999999e-05 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.822826+0000", 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.030421790000000001 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.823399+0000", 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057227600000000003 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.847874+0000", 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024475635999999999 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109624 5.1 5:84c547a6:::benchmark_data_smithi159_116570_object16464:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:29:46.645 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 157.74849135599999, 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61741415600000005, 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109624 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887529+0000", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.172e-06 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887534+0000", 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9250000000000003e-06 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887545+0000", 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1678999999999999e-05 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887552+0000", 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8530000000000003e-06 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.939213+0000", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.051660436999999997 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.944118+0000", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049053029999999997 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.952640+0000", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085221399999999992 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.002593+0000", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.049952866999999998 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.004827+0000", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022339119999999998 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.489082+0000", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48425561 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:46.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.492309+0000", 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032266809999999999 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.504942+0000", 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012632577000000001 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109639 5.1 5:8cec3473:::benchmark_data_smithi159_116570_object16479:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:29:46.651 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 157.709617904, 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59574422100000002, 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109639 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926402+0000", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.54e-07 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926408+0000", 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.981e-06 2024-04-15T18:29:46.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926420+0000", 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2374999999999999e-05 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926429+0000", 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2429999999999994e-06 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.959548+0000", 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.033118455999999998 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.964288+0000", 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047403590000000004 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.972993+0000", 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087051390000000006 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.006957+0000", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.033963659 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.009091+0000", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021345159999999999 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.505524+0000", 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.49643292 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.508285+0000", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002760523 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.522145+0000", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013860196 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109646 5.1 5:847282b1:::benchmark_data_smithi159_116570_object16486:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:29:46.657 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 157.70480789699999, 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61196233099999997, 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109646 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931212+0000", 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931216+0000", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4039999999999997e-06 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931227+0000", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1061e-05 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931233+0000", 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.010244+0000", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.079011203000000002 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.015127+0000", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048829340000000002 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.023722+0000", 2024-04-15T18:29:46.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085948350000000003 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.032963+0000", 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092412020000000004 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.035115+0000", 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021520290000000002 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:46.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.524421+0000", 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48930604100000002 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.528026+0000", 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036048389999999999 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.543173+0000", 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015147287000000001 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109740 5.1 5:85fc6f90:::benchmark_data_smithi159_116570_object16580:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 157.14840476500001, 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.57482975400000003, 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:29:46.664 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109740 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487615+0000", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1480000000000001e-06 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487619+0000", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0810000000000004e-06 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487630+0000", 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1104000000000001e-05 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487637+0000", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8140000000000003e-06 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.628337+0000", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.140700198 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.633780+0000", 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054425510000000003 2024-04-15T18:29:46.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.642947+0000", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091669679999999993 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.652867+0000", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099201830000000008 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.673347+0000", 2024-04-15T18:29:46.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020479687 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.045461+0000", 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37211393300000001 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.048790+0000", 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033297650000000002 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.062444+0000", 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013653322000000001 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 428/416 [PushOp(5:8992b35b:::benchmark_data_smithi159_116570_object16825:head, version: 369'1040, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8992b35b:::benchmark_data_smithi159_116570_object16825:head@369'1040, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 156.434939525, 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.78917612000000004, 2024-04-15T18:29:46.670 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201080+0000", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.051e-06 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201140+0000", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9311000000000002e-05 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201171+0000", 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1680000000000002e-05 2024-04-15T18:29:46.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201186+0000", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4775e-05 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.965149+0000", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.76396304299999995 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.965718+0000", 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056873799999999995 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.990255+0000", 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024537521999999999 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 428/416 [PushReplyOp(5:8b327da9:::benchmark_data_smithi159_116570_object16904:head)])", 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 154.35991784300001, 2024-04-15T18:29:46.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64978421600000003, 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:29:46.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276102+0000", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8700000000000004e-07 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276109+0000", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6830000000000002e-06 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276140+0000", 2024-04-15T18:29:46.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1109000000000003e-05 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276151+0000", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1572e-05 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.899505+0000", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62335336200000002 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.925885+0000", 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026380503 2024-04-15T18:29:46.677 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 428/416 [PushReplyOp(5:8b327da9:::benchmark_data_smithi159_116570_object16904:head)])", 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 154.358118437, 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67448548900000005, 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277901+0000", 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1099999999999995e-07 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277909+0000", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5320000000000001e-06 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277937+0000", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8476999999999999e-05 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277945+0000", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7039999999999994e-06 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.926536+0000", 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64859101600000002 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.952386+0000", 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025850148999999999 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 430/416 [PushReplyOp(5:86484eaf:::benchmark_data_smithi159_116570_object17263:head)])", 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:29:46.681 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 151.268832304, 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63286371799999996, 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:29:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367187+0000", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4900000000000001e-07 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367194+0000", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.153e-06 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367213+0000", 2024-04-15T18:29:46.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8885999999999998e-05 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367237+0000", 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3858000000000001e-05 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.972237+0000", 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60500030100000002 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:16.000050+0000", 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.027813071000000002 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 449/416 [PushReplyOp(5:8d33612f:::benchmark_data_smithi159_116570_object22274:head)])", 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 112.065815089, 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61168342600000003, 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1099999999999996e-07 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570208+0000", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8800000000000001e-06 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570231+0000", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2594000000000001e-05 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570237+0000", 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5470000000000002e-06 2024-04-15T18:29:46.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.168270+0000", 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59803278500000001 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.181887+0000", 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013617109000000001 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 449/416 [PushReplyOp(5:8d33612f:::benchmark_data_smithi159_116570_object22274:head)])", 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 112.06547895, 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.673407639, 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570543+0000", 2024-04-15T18:29:46.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1990000000000002e-06 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570560+0000", 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6484e-05 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570583+0000", 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3203000000000001e-05 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.216682+0000", 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64609930500000001 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.243947+0000", 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.027265007000000001 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 450/416 [PushOp(5:824abbce:::benchmark_data_smithi159_116570_object22686:head, version: 370'1404, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:824abbce:::benchmark_data_smithi159_116570_object22686:head@370'1404, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:57.521535+0000", 2024-04-15T18:29:46.692 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 109.114483398, 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.54695932800000002, 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521535+0000", 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521535+0000", 2024-04-15T18:29:46.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521536+0000", 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3799999999999996e-07 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521595+0000", 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8501000000000001e-05 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521625+0000", 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0540000000000002e-05 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.521643+0000", 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7920000000000001e-05 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:58.047332+0000", 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.52568846000000002 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:58.047884+0000", 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055231799999999999 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:58.068495+0000", 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020610651000000001 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 459/416 [PushReplyOp(5:83564489:::benchmark_data_smithi159_116570_object25233:head)])", 2024-04-15T18:29:46.696 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.532655958000007, 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.78063833000000005, 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.75e-07 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103370+0000", 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.939e-06 2024-04-15T18:29:46.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103401+0000", 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0516999999999999e-05 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103408+0000", 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.272e-06 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.860295+0000", 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75688664000000005 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.884001+0000", 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023706386999999999 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 459/416 [PushOp(5:8aa070ee:::benchmark_data_smithi159_116570_object25481:head, version: 370'1582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8aa070ee:::benchmark_data_smithi159_116570_object25481:head@370'1582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:29:46.700 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.421250447999995, 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55792225500000003, 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:29:46.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214769+0000", 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1900000000000003e-07 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214830+0000", 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0820999999999999e-05 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214857+0000", 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7039e-05 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214875+0000", 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8512000000000002e-05 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.752069+0000", 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53719326000000001 2024-04-15T18:29:46.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.752683+0000", 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00061441800000000004 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.772691+0000", 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020007686 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 459/416 [PushOp(5:88c22e6a:::benchmark_data_smithi159_116570_object25781:head, version: 371'1606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:88c22e6a:::benchmark_data_smithi159_116570_object25781:head@371'1606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 84.661025324999997, 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56908714999999999, 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974995+0000", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.026e-06 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.975056+0000", 2024-04-15T18:29:46.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1749999999999997e-05 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.452398+0000", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.47734127900000001 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.452409+0000", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1143e-05 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.523093+0000", 2024-04-15T18:29:46.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.070684223000000004 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.523564+0000", 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00047106200000000001 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.544081+0000", 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020516666999999999 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 476/416 [PushReplyOp(5:8fb5237a:::benchmark_data_smithi159_116570_object31080:head)])", 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.723947365000001, 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62137753200000001, 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912072+0000", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8899999999999999e-07 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912076+0000", 2024-04-15T18:29:46.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7009999999999998e-06 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912098+0000", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2235999999999999e-05 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912103+0000", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1819999999999997e-06 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:58.511060+0000", 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59895651100000002 2024-04-15T18:29:46.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:58.533449+0000", 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022389313000000001 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 480/416 [PushOp(5:801faef5:::benchmark_data_smithi159_116570_object33041:head, version: 372'2043, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:801faef5:::benchmark_data_smithi159_116570_object33041:head@372'2043, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.712 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.835772364999997, 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.72250066199999996, 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800247+0000", 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800287+0000", 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9996000000000001e-05 2024-04-15T18:29:46.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.336390+0000", 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53610254300000004 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.336406+0000", 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6167e-05 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.503579+0000", 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16717265100000001 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.504081+0000", 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050283800000000003 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.522747+0000", 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018665698000000001 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 490/416 [PushOp(5:864bd29b:::benchmark_data_smithi159_116570_object38897:head, version: 374'2403, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:864bd29b:::benchmark_data_smithi159_116570_object38897:head@374'2403, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:44.061386+0000", 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.5746324029999998, 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.52764603899999996, 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:29:46.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061386+0000", 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061386+0000", 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061387+0000", 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7899999999999998e-07 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061545+0000", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015765700000000001 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061570+0000", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5686000000000001e-05 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.061588+0000", 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7850999999999999e-05 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.573103+0000", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51151506800000002 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.573468+0000", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00036487899999999997 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:44.589032+0000", 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015564219000000001 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.720 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:29:46.721 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:29:46.721 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:29:46.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:46.750 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:46 smithi082 ceph-mon[44206]: pgmap v1278: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 7804/163713 objects misplaced (4.767%); 432 KiB/s, 6 objects/s recovering 2024-04-15T18:29:46.752 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:46.773 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:46.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:46.876 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:29:46.902 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:46.921 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:29:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:46.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:29:47.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:47.055 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:47.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:29:47.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:47 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:29:47.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:29:47.403 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:47.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:47.507 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:47.537 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:47.556 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:47.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:47 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:29:47.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:47.683 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:47 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:29:47.685 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:47.704 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:47.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:47.808 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:47.834 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:47.852 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:47.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:47.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:47.985 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:48.011 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:48.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:48.115 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:48.142 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:48.162 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:48.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:48.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:48.296 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:48.318 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:29:48.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:48 smithi159 ceph-mon[45336]: pgmap v1279: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7639/161919 objects misplaced (4.718%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:48.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:48.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:48 smithi049 ceph-mon[40087]: pgmap v1279: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7639/161919 objects misplaced (4.718%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:48.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:48.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:48 smithi082 ceph-mon[44206]: pgmap v1279: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 154 op/s; 7639/161919 objects misplaced (4.718%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:48.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:29:49.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:49 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:49.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:49 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:29:49.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:49.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:49 smithi049 ceph-mon[40087]: osdmap e491: 12 total, 12 up, 12 in 2024-04-15T18:29:49.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:49 smithi049 ceph-mon[40087]: 5.0 scrub starts 2024-04-15T18:29:49.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:49 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:49.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:49 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:29:49.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:49.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:49 smithi159 ceph-mon[45336]: osdmap e491: 12 total, 12 up, 12 in 2024-04-15T18:29:49.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:49 smithi159 ceph-mon[45336]: 5.0 scrub starts 2024-04-15T18:29:49.650 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:49.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:49.758 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:49 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:29:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:49 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:29:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/233484385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:29:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:49 smithi082 ceph-mon[44206]: osdmap e491: 12 total, 12 up, 12 in 2024-04-15T18:29:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:49 smithi082 ceph-mon[44206]: 5.0 scrub starts 2024-04-15T18:29:49.787 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:49.806 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:49.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:49.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:49.936 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:49.955 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:49.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:50.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:50.091 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:50.110 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:50.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:50.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:50.248 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:50.274 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:50.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:50.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:50 smithi049 ceph-mon[40087]: pgmap v1281: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 7639/161244 objects misplaced (4.738%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:50.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:50.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:50.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:50 smithi049 ceph-mon[40087]: 5.0 scrub ok 2024-04-15T18:29:50.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:50.413 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:50 smithi159 ceph-mon[45336]: pgmap v1281: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 7639/161244 objects misplaced (4.738%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:50 smithi159 ceph-mon[45336]: 5.0 scrub ok 2024-04-15T18:29:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:50.662 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:50.666 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:50.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:50.772 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:50 smithi082 ceph-mon[44206]: pgmap v1281: 17 pgs: 1 active+recovering, 16 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 168 op/s; 7639/161244 objects misplaced (4.738%); 788 KiB/s, 12 objects/s recovering 2024-04-15T18:29:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:50 smithi082 ceph-mon[44206]: 5.0 scrub ok 2024-04-15T18:29:50.799 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:50.821 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:50.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:29:50.911 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:29:50.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:50.952 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:50.973 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:50.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:51.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:51.104 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:51.126 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:51.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:51.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:51.256 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:51.277 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:51.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:51.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:51.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:51 smithi159 ceph-mon[45336]: osdmap e492: 12 total, 12 up, 12 in 2024-04-15T18:29:51.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:51 smithi159 ceph-mon[45336]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:51.415 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:51.434 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:51.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:51.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:51 smithi049 ceph-mon[40087]: osdmap e492: 12 total, 12 up, 12 in 2024-04-15T18:29:51.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:51 smithi049 ceph-mon[40087]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:51.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:51 smithi082 ceph-mon[44206]: osdmap e492: 12 total, 12 up, 12 in 2024-04-15T18:29:51.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:51 smithi082 ceph-mon[44206]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:29:51.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:51.566 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:51.585 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:51.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:51.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:51.721 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:52.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:29:52.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:52 smithi159 ceph-mon[45336]: pgmap v1283: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 7639/160653 objects misplaced (4.755%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:29:52.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:52 smithi049 ceph-mon[40087]: pgmap v1283: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 7639/160653 objects misplaced (4.755%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:29:52.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:52 smithi082 ceph-mon[44206]: pgmap v1283: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 7639/160653 objects misplaced (4.755%); 448 KiB/s, 6 objects/s recovering 2024-04-15T18:29:53.172 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:53.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:53.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:53.309 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:53.330 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:53.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:53.435 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:53.461 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:53.480 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:53.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:53.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:29:53.611 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:53.631 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:29:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:53.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:29:53.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:53.765 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:54.082 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1282,"stamp":"2024-04-15T18:29:50.322686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3522822712,"num_objects":53748,"num_object_clones":0,"num_object_copies":161244,"num_objects_missing_on_primary":2539,"num_objects_missing":2539,"num_objects_degraded":0,"num_objects_misplaced":7639,"num_objects_unfound":0,"num_objects_dirty":53748,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":132751,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39132,"num_bytes_recovered":2561628832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134430,"ondisk_log_size":134430,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14449116,"kb_used_data":10855152,"kb_used_omap":177,"kb_used_meta":3593614,"kb_avail":1110492708,"statfs":{"total":1151940427776,"available":1137144532992,"internally_reserved":0,"allocated":11115675648,"data_stored":11108566693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3679861188},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95092736,"num_objects":-1451,"num_object_clones":0,"num_object_copies":-4353,"num_objects_missing_on_primary":-109,"num_objects_missing":-109,"num_objects_degraded":0,"num_objects_misplaced":-329,"num_objects_unfound":0,"num_objects_dirty":-1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":7208960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.982846"},"pg_stats":[{"pgid":"5.f","version":"490'8447","reported_seq":14443,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.147366+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:29:47.147366+0000","last_peered":"2024-04-15T18:29:47.147366+0000","last_clean":"2024-04-15T18:29:47.147366+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:29:47.147366+0000","last_undegraded":"2024-04-15T18:29:47.147366+0000","last_fullsized":"2024-04-15T18:29:47.147366+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8447,"log_dups_size":0,"ondisk_log_size":8447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221184000,"num_objects":3375,"num_object_clones":0,"num_object_copies":10125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8447,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"490'8327","reported_seq":8760,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.279905+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:29:47.279905+0000","last_peered":"2024-04-15T18:29:47.279905+0000","last_clean":"2024-04-15T18:29:47.279905+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:29:47.279905+0000","last_undegraded":"2024-04-15T18:29:47.279905+0000","last_fullsized":"2024-04-15T18:29:47.279905+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8327,"log_dups_size":0,"ondisk_log_size":8327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220528640,"num_objects":3365,"num_object_clones":0,"num_object_copies":10095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8327,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"490'8256","reported_seq":14041,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:45.931915+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:29:45.931915+0000","last_peered":"2024-04-15T18:29:45.931915+0000","last_clean":"2024-04-15T18:29:45.931915+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:29:45.931915+0000","last_undegraded":"2024-04-15T18:29:45.931915+0000","last_fullsized":"2024-04-15T18:29:45.931915+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8256,"log_dups_size":0,"ondisk_log_size":8256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219480088,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8256,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"490'8300","reported_seq":8745,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.146195+0000","last_change":"2024-04-15T18:28:38.810827+0000","last_active":"2024-04-15T18:29:47.146195+0000","last_peered":"2024-04-15T18:29:47.146195+0000","last_clean":"2024-04-15T18:29:47.146195+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:29:47.146195+0000","last_undegraded":"2024-04-15T18:29:47.146195+0000","last_fullsized":"2024-04-15T18:29:47.146195+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'7570","last_scrub_stamp":"2024-04-15T18:28:38.810655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:38.810655+0000","objects_scrubbed":4127,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:56.394076+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222429184,"num_objects":3394,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"490'8403","reported_seq":14352,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.232459+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:29:47.232459+0000","last_peered":"2024-04-15T18:29:47.232459+0000","last_clean":"2024-04-15T18:29:47.232459+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:29:47.232459+0000","last_undegraded":"2024-04-15T18:29:47.232459+0000","last_fullsized":"2024-04-15T18:29:47.232459+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8403,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"490'8303","reported_seq":20378,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.240358+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:29:47.240358+0000","last_peered":"2024-04-15T18:29:47.240358+0000","last_clean":"2024-04-15T18:29:47.240358+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:29:47.240358+0000","last_undegraded":"2024-04-15T18:29:47.240358+0000","last_fullsized":"2024-04-15T18:29:47.240358+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8303,"log_dups_size":0,"ondisk_log_size":8303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219086848,"num_objects":3343,"num_object_clones":0,"num_object_copies":10029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"490'8225","reported_seq":8590,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:45.774530+0000","last_change":"2024-04-15T18:29:23.869859+0000","last_active":"2024-04-15T18:29:45.774530+0000","last_peered":"2024-04-15T18:29:45.774530+0000","last_clean":"2024-04-15T18:29:45.774530+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:29:45.774530+0000","last_undegraded":"2024-04-15T18:29:45.774530+0000","last_fullsized":"2024-04-15T18:29:45.774530+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":424,"log_size":8225,"log_dups_size":0,"ondisk_log_size":8225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225247232,"num_objects":3437,"num_object_clones":0,"num_object_copies":10311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8225,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"490'8317","reported_seq":8592,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:46.218994+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:29:46.218994+0000","last_peered":"2024-04-15T18:29:46.218994+0000","last_clean":"2024-04-15T18:29:46.218994+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:29:46.218994+0000","last_undegraded":"2024-04-15T18:29:46.218994+0000","last_fullsized":"2024-04-15T18:29:46.218994+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8317,"log_dups_size":0,"ondisk_log_size":8317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224854016,"num_objects":3431,"num_object_clones":0,"num_object_copies":10293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"490'8360","reported_seq":14348,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.347201+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:29:47.347201+0000","last_peered":"2024-04-15T18:29:47.347201+0000","last_clean":"2024-04-15T18:29:47.347201+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:29:47.347201+0000","last_undegraded":"2024-04-15T18:29:47.347201+0000","last_fullsized":"2024-04-15T18:29:47.347201+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"490'8177","reported_seq":25406,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.254585+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:29:47.254585+0000","last_peered":"2024-04-15T18:29:47.254585+0000","last_clean":"2024-04-15T18:29:47.254585+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:29:47.254585+0000","last_undegraded":"2024-04-15T18:29:47.254585+0000","last_fullsized":"2024-04-15T18:29:47.254585+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8177,"log_dups_size":0,"ondisk_log_size":8177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213581824,"num_objects":3259,"num_object_clones":0,"num_object_copies":9777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"490'8057","reported_seq":8489,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.155237+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:29:47.155237+0000","last_peered":"2024-04-15T18:29:47.155237+0000","last_clean":"2024-04-15T18:29:47.155237+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:29:47.155237+0000","last_undegraded":"2024-04-15T18:29:47.155237+0000","last_fullsized":"2024-04-15T18:29:47.155237+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":8057,"log_dups_size":0,"ondisk_log_size":8057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213450752,"num_objects":3257,"num_object_clones":0,"num_object_copies":9771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8057,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"490'8419","reported_seq":14374,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:45.288870+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:29:45.288870+0000","last_peered":"2024-04-15T18:29:45.288870+0000","last_clean":"2024-04-15T18:29:45.288870+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:29:45.288870+0000","last_undegraded":"2024-04-15T18:29:45.288870+0000","last_fullsized":"2024-04-15T18:29:45.288870+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8419,"log_dups_size":0,"ondisk_log_size":8419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225771520,"num_objects":3445,"num_object_clones":0,"num_object_copies":10335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"490'8176","reported_seq":8509,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:45.311503+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:29:45.311503+0000","last_peered":"2024-04-15T18:29:45.311503+0000","last_clean":"2024-04-15T18:29:45.311503+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:29:45.311503+0000","last_undegraded":"2024-04-15T18:29:45.311503+0000","last_fullsized":"2024-04-15T18:29:45.311503+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223215616,"num_objects":3406,"num_object_clones":0,"num_object_copies":10218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8176,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"490'8094","reported_seq":18059,"reported_epoch":490,"state":"active+recovering","last_fresh":"2024-04-15T18:29:46.421351+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:29:46.421351+0000","last_peered":"2024-04-15T18:29:46.421351+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:29:46.421351+0000","last_undegraded":"2024-04-15T18:29:46.421351+0000","last_fullsized":"2024-04-15T18:29:46.421351+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213647360,"num_objects":3260,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":2539,"num_objects_missing":2539,"num_objects_degraded":0,"num_objects_misplaced":7639,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2419,"num_bytes_recovered":158400512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2539},{"shards":"3,4,7,11","objects":11}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"490'8397","reported_seq":8792,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.997558+0000","last_change":"2024-04-15T18:28:44.111297+0000","last_active":"2024-04-15T18:29:47.997558+0000","last_peered":"2024-04-15T18:29:47.997558+0000","last_clean":"2024-04-15T18:29:47.997558+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:29:47.997558+0000","last_undegraded":"2024-04-15T18:29:47.997558+0000","last_fullsized":"2024-04-15T18:29:47.997558+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'7744","last_scrub_stamp":"2024-04-15T18:28:44.111158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:44.111158+0000","objects_scrubbed":4069,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223805440,"num_objects":3415,"num_object_clones":0,"num_object_copies":10245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"490'12136","reported_seq":29736,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:47.981371+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:29:47.981371+0000","last_peered":"2024-04-15T18:29:47.981371+0000","last_clean":"2024-04-15T18:29:47.981371+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:29:47.981371+0000","last_undegraded":"2024-04-15T18:29:47.981371+0000","last_fullsized":"2024-04-15T18:29:47.981371+0000","mapping_epoch":440,"log_start":"372'2101","ondisk_log_start":"372'2101","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:29:57.294144+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1226,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-04-15T18:29:40.412474+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:29:40.412474+0000","last_peered":"2024-04-15T18:29:40.412474+0000","last_clean":"2024-04-15T18:29:40.412474+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:29:40.412474+0000","last_undegraded":"2024-04-15T18:29:40.412474+0000","last_fullsized":"2024-04-15T18:29:40.412474+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3522232344,"num_objects":53746,"num_object_clones":0,"num_object_copies":161238,"num_objects_missing_on_primary":2539,"num_objects_missing":2539,"num_objects_degraded":0,"num_objects_misplaced":7639,"num_objects_unfound":0,"num_objects_dirty":53746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132570,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39122,"num_bytes_recovered":2558984192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11074351104,"data_stored":11074338888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134293,"ondisk_log_size":134293,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1334324,"kb_used_data":919980,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92410828,"statfs":{"total":95995035648,"available":94628687872,"internally_reserved":0,"allocated":942059520,"data_stored":941461747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187708,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1389224,"kb_used_data":1083168,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92355928,"statfs":{"total":95995035648,"available":94572470272,"internally_reserved":0,"allocated":1109164032,"data_stored":1108573903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514753,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966952,"kb_used_data":647008,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92778200,"statfs":{"total":95995035648,"available":95004876800,"internally_reserved":0,"allocated":662536192,"data_stored":661946063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592952,"kb_used_data":427116,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93152200,"statfs":{"total":95995035648,"available":95387852800,"internally_reserved":0,"allocated":437366784,"data_stored":436772140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168842,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1281268,"kb_used_data":965484,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92463884,"statfs":{"total":95995035648,"available":94683017216,"internally_reserved":0,"allocated":988655616,"data_stored":988057863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463184,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049712,"kb_used_data":707560,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92695440,"statfs":{"total":95995035648,"available":94920130560,"internally_reserved":0,"allocated":724541440,"data_stored":723947759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239528,"kb_used_data":857120,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92505624,"statfs":{"total":95995035648,"available":94725758976,"internally_reserved":0,"allocated":877690880,"data_stored":877100751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738670,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1931688,"kb_used_data":1567328,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91813464,"statfs":{"total":95995035648,"available":94016987136,"internally_reserved":0,"allocated":1604943872,"data_stored":1604353743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193921,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750376,"kb_used_data":637856,"kb_used_omap":114,"kb_used_meta":112397,"kb_avail":92994776,"statfs":{"total":95995035648,"available":95226650624,"internally_reserved":0,"allocated":653164544,"data_stored":652574415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":115094769},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609051,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653680,"kb_used_data":308128,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091472,"statfs":{"total":95995035648,"available":95325667328,"internally_reserved":0,"allocated":315523072,"data_stored":314932943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1590504,"kb_used_data":1230304,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92154648,"statfs":{"total":95995035648,"available":94366359552,"internally_reserved":0,"allocated":1259831296,"data_stored":1259241167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703352,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1668908,"kb_used_data":1504100,"kb_used_omap":4,"kb_used_meta":164731,"kb_avail":92076244,"statfs":{"total":95995035648,"available":94286073856,"internally_reserved":0,"allocated":1540198400,"data_stored":1539604199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":168685249},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1601634304,"data_stored":1601634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":875233280,"data_stored":875233280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":650903552,"data_stored":650903552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1255342080,"data_stored":1255342080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1534857216,"data_stored":1534853144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":721158144,"data_stored":721158144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":984813568,"data_stored":984809496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":434569216,"data_stored":434569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":659292160,"data_stored":659292160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1104150528,"data_stored":1104150528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":938479616,"data_stored":938475544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:29:54.083 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.1 is not active+clean 2024-04-15T18:29:54.084 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.1', 'version': "490'8094", 'reported_seq': 18059, 'reported_epoch': 490, 'state': 'active+recovering', 'last_fresh': '2024-04-15T18:29:46.421351+0000', 'last_change': '2024-04-15T18:26:49.574312+0000', 'last_active': '2024-04-15T18:29:46.421351+0000', 'last_peered': '2024-04-15T18:29:46.421351+0000', 'last_clean': '2024-04-15T18:26:00.259169+0000', 'last_became_active': '2024-04-15T18:26:48.920249+0000', 'last_became_peered': '2024-04-15T18:26:48.920249+0000', 'last_unstale': '2024-04-15T18:29:46.421351+0000', 'last_undegraded': '2024-04-15T18:29:46.421351+0000', 'last_fullsized': '2024-04-15T18:29:46.421351+0000', 'mapping_epoch': 416, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'objects_scrubbed': 0, 'log_size': 8094, 'log_dups_size': 0, 'ondisk_log_size': 8094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 213647360, 'num_objects': 3260, 'num_object_clones': 0, 'num_object_copies': 9780, 'num_objects_missing_on_primary': 2539, 'num_objects_missing': 2539, 'num_objects_degraded': 0, 'num_objects_misplaced': 7639, 'num_objects_unfound': 0, 'num_objects_dirty': 3260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8094, 'num_write_kb': 363328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2419, 'num_bytes_recovered': 158400512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0, 6], 'acting': [11, 0, 6], 'avail_no_missing': ['3', '4', '7'], 'object_location_counts': [{'shards': '0,6,11', 'objects': 710}, {'shards': '3,4,7', 'objects': 2539}, {'shards': '3,4,7,11', 'objects': 11}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-15T18:29:54.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:29:54.315 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:54.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:54.356 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:29:54.393 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:29:54.397 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:54.419 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:29:54.421 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:54.446 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:29:54.449 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:54.468 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:54.470 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:29:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:54.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:54.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:29:54.523 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:29:54.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:29:54.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:54.574 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:29:54.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:29:54.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:54 smithi049 ceph-mon[40087]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 222 op/s; 7468/158364 objects misplaced (4.716%); 904 KiB/s, 14 objects/s recovering 2024-04-15T18:29:54.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1958813970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:54.602 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:54.621 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:54.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:54.629 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:29:54.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:54 smithi159 ceph-mon[45336]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 222 op/s; 7468/158364 objects misplaced (4.716%); 904 KiB/s, 14 objects/s recovering 2024-04-15T18:29:54.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1958813970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:54.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:29:54.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:29:54.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:29:54.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:29:54.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:29:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:29:54.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:54.753 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:54.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:54.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:54 smithi082 ceph-mon[44206]: pgmap v1284: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 222 op/s; 7468/158364 objects misplaced (4.716%); 904 KiB/s, 14 objects/s recovering 2024-04-15T18:29:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1958813970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:29:54.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:54.903 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:54.924 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:54.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:55.027 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:55.055 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:55.078 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:55.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:55.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:55.217 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:55.227 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:29:55.227 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:55.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:55.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:55.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:56 smithi159 ceph-mon[45336]: pgmap v1285: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 7468/158364 objects misplaced (4.716%); 456 KiB/s, 7 objects/s recovering 2024-04-15T18:29:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:56 smithi159 ceph-mon[45336]: osdmap e493: 12 total, 12 up, 12 in 2024-04-15T18:29:56.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:56 smithi049 ceph-mon[40087]: pgmap v1285: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 7468/158364 objects misplaced (4.716%); 456 KiB/s, 7 objects/s recovering 2024-04-15T18:29:56.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:56 smithi049 ceph-mon[40087]: osdmap e493: 12 total, 12 up, 12 in 2024-04-15T18:29:56.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:29:56.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:56 smithi082 ceph-mon[44206]: pgmap v1285: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 148 op/s; 7468/158364 objects misplaced (4.716%); 456 KiB/s, 7 objects/s recovering 2024-04-15T18:29:56.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:56 smithi082 ceph-mon[44206]: osdmap e493: 12 total, 12 up, 12 in 2024-04-15T18:29:56.975 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:56.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:57.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:57.107 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:57.126 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:57.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:57.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:29:57.261 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:57.316 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:29:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:57.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:29:57.422 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:29:57.448 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:57.466 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:29:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:57.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:29:57.570 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:29:57.596 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:57.614 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:29:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:57.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:29:57.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:29:57.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:29:57.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:57 smithi049 ceph-mon[40087]: osdmap e494: 12 total, 12 up, 12 in 2024-04-15T18:29:57.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:57.749 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:57.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:29:57.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:29:57.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:57 smithi082 ceph-mon[44206]: osdmap e494: 12 total, 12 up, 12 in 2024-04-15T18:29:57.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:29:57.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:29:57.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:57 smithi159 ceph-mon[45336]: osdmap e494: 12 total, 12 up, 12 in 2024-04-15T18:29:58.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:29:58.198 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:29:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:58.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:29:58.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:58.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:58 smithi049 ceph-mon[40087]: 5.c scrub starts 2024-04-15T18:29:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:58 smithi049 ceph-mon[40087]: pgmap v1288: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 7295/155934 objects misplaced (4.678%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:29:58.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:58 smithi049 ceph-mon[40087]: 5.c scrub ok 2024-04-15T18:29:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:58.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:58.562 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:58.593 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:58.618 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:58.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:58.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:29:58.763 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:58.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:58 smithi082 ceph-mon[44206]: 5.c scrub starts 2024-04-15T18:29:58.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:58 smithi082 ceph-mon[44206]: pgmap v1288: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 7295/155934 objects misplaced (4.678%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:29:58.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:58 smithi082 ceph-mon[44206]: 5.c scrub ok 2024-04-15T18:29:58.785 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:29:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:58.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:29:58.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:29:58.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:58 smithi159 ceph-mon[45336]: 5.c scrub starts 2024-04-15T18:29:58.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:58 smithi159 ceph-mon[45336]: pgmap v1288: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 252 op/s; 7295/155934 objects misplaced (4.678%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:29:58.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:58 smithi159 ceph-mon[45336]: 5.c scrub ok 2024-04-15T18:29:58.915 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:58.935 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:29:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:58.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:29:59.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:29:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:59.239 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:29:59.273 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:29:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:59.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:29:59.378 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:29:59.408 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:59.410 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:29:59.429 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:29:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:59.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:29:59.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:29:59.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:59.580 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:29:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:59.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:29:59.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:29:59.708 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:29:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:59.710 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:59.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:29:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:59.729 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:29:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:29:59.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:29:59.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:29:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:29:59.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:29:59.860 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:29:59.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:30:00.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:00.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:00.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:00.344 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:00.364 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:00.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: pgmap v1289: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 7295/155175 objects misplaced (4.701%); 912 KiB/s, 14 objects/s recovering 2024-04-15T18:30:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:00.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:00.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: osdmap e495: 12 total, 12 up, 12 in 2024-04-15T18:30:00.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-15T18:30:00.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:00 smithi049 ceph-mon[40087]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-15T18:30:00.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:00.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:00.763 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:00.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: pgmap v1289: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 7295/155175 objects misplaced (4.701%); 912 KiB/s, 14 objects/s recovering 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: osdmap e495: 12 total, 12 up, 12 in 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-15T18:30:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:00 smithi082 ceph-mon[44206]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-15T18:30:00.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:00.894 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: pgmap v1289: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 7295/155175 objects misplaced (4.701%); 912 KiB/s, 14 objects/s recovering 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2928341360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: osdmap e495: 12 total, 12 up, 12 in 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-15T18:30:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:00 smithi159 ceph-mon[45336]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-15T18:30:00.913 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:00.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:01.017 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:01.047 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.072 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:01.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:01.205 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.278 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:01.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:01.409 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.430 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:01.537 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:01.566 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.590 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:01.695 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:01.723 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.737 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:01.737 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1549,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":495,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":17,"num_pools":2,"num_objects":51512,"data_bytes":3376284216,"bytes_used":14372749312,"bytes_avail":1137567678464,"bytes_total":1151940427776,"misplaced_objects":7295,"misplaced_total":154536,"misplaced_ratio":0.047205829062483821,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":622476,"recovering_keys_per_sec":0,"num_objects_recovered":57,"num_bytes_recovered":3735552,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":35,"modified":"2024-04-15T18:27:37.316853+0000","services":{}},"progress_events":{}} 2024-04-15T18:30:01.742 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:01.846 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:01.873 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:01.894 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:01.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:01.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:02.026 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.045 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:02.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:02.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:02.176 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.199 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:02.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:02.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:02.330 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.349 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:02.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:02.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:02.483 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.506 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:02.506 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:02.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:02.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:02.636 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:02 smithi159 ceph-mon[45336]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 212 op/s; 7295/154536 objects misplaced (4.721%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:02.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1541509579' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:02.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:02 smithi159 ceph-mon[45336]: osdmap e496: 12 total, 12 up, 12 in 2024-04-15T18:30:02.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.660 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:02.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:02.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:02 smithi049 ceph-mon[40087]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 212 op/s; 7295/154536 objects misplaced (4.721%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:02.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1541509579' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:02.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:02 smithi049 ceph-mon[40087]: osdmap e496: 12 total, 12 up, 12 in 2024-04-15T18:30:02.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:02 smithi082 ceph-mon[44206]: pgmap v1291: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 212 op/s; 7295/154536 objects misplaced (4.721%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1541509579' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:02.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:02 smithi082 ceph-mon[44206]: osdmap e496: 12 total, 12 up, 12 in 2024-04-15T18:30:02.804 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:02.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:30:03.206 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:30:03.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:03.455 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:03.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:03.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:03.590 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:03.721 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:03.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:03.744 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:30:03.744 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:30:03.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:03.855 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:03.879 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:03.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:03.984 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:04.010 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:04.028 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:04.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:04.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:04.158 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:04.179 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:04.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:04.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:04.310 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:04.330 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:04.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:04.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:04.466 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:04.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:04 smithi159 ceph-mon[45336]: pgmap v1293: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 7113/151998 objects misplaced (4.680%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:04.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:04 smithi049 ceph-mon[40087]: pgmap v1293: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 7113/151998 objects misplaced (4.680%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:04.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:04 smithi082 ceph-mon[44206]: pgmap v1293: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 7113/151998 objects misplaced (4.680%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:04.918 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:04.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:05.024 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:05.051 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:05.071 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:05.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:05.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:05.203 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:05.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:05 smithi049 ceph-mon[40087]: osdmap e497: 12 total, 12 up, 12 in 2024-04-15T18:30:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:30:05.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:05.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:05.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:05 smithi082 ceph-mon[44206]: osdmap e497: 12 total, 12 up, 12 in 2024-04-15T18:30:05.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:30:05.820 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:05.820 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:05.821 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:05 smithi159 ceph-mon[45336]: osdmap e497: 12 total, 12 up, 12 in 2024-04-15T18:30:05.821 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:30:06.181 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:06.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:30:06.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:06 smithi049 ceph-mon[40087]: pgmap v1295: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 7113/151998 objects misplaced (4.680%); 651 KiB/s, 10 objects/s recovering 2024-04-15T18:30:06.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:06 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:06.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:06 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:06.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:06.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:06 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:06 smithi082 ceph-mon[44206]: pgmap v1295: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 7113/151998 objects misplaced (4.680%); 651 KiB/s, 10 objects/s recovering 2024-04-15T18:30:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:06 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:06 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:06.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:06 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:06 smithi159 ceph-mon[45336]: pgmap v1295: 17 pgs: 1 active+recovering, 16 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 176 op/s; 7113/151998 objects misplaced (4.680%); 651 KiB/s, 10 objects/s recovering 2024-04-15T18:30:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:06 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:06 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:06 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:07.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:30:07.130 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:30:07.138 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:07.146 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:30:07.173 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:30:07.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:30:07.224 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:30:07.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:30:07.275 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:30:07.299 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:30:07.324 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:30:07.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:30:07.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:30:07.376 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:30:07.377 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:30:07.377 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:30:07.654 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:07.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: osdmap e498: 12 total, 12 up, 12 in 2024-04-15T18:30:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:30:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:30:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:07.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1316676371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:07.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:07.758 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: osdmap e498: 12 total, 12 up, 12 in 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1316676371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:07.785 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:07.804 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:07.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:07.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: osdmap e498: 12 total, 12 up, 12 in 2024-04-15T18:30:07.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:30:07.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:30:07.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:30:07.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1316676371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:07.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:07.908 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:07.936 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:07.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:07.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:08.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:08.086 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:08.106 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:08.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:08.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:08.235 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:08.254 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:08.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:08.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:08.388 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:08.637 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:08.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:08.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: pgmap v1297: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 258 op/s; 6936/149880 objects misplaced (4.628%); 1.2 MiB/s, 19 objects/s recovering 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:08.771 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: pgmap v1297: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 258 op/s; 6936/149880 objects misplaced (4.628%); 1.2 MiB/s, 19 objects/s recovering 2024-04-15T18:30:08.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: pgmap v1297: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 258 op/s; 6936/149880 objects misplaced (4.628%); 1.2 MiB/s, 19 objects/s recovering 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:08.927 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:08.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:09.031 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:09.060 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:09.082 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:09.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:09.186 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:09.201 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:09.215 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:09.234 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:09.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:09.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:09.368 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:09.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:09.511 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:09.541 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:09.562 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:09.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:09.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:09.700 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:10.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:30:10.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:10 smithi049 ceph-mon[40087]: pgmap v1298: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 6936/149154 objects misplaced (4.650%); 537 KiB/s, 8 objects/s recovering 2024-04-15T18:30:10.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:10 smithi049 ceph-mon[40087]: from='client.26021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:10.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:10 smithi082 ceph-mon[44206]: pgmap v1298: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 6936/149154 objects misplaced (4.650%); 537 KiB/s, 8 objects/s recovering 2024-04-15T18:30:10.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:10 smithi082 ceph-mon[44206]: from='client.26021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:10.851 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:10.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:10.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:10 smithi159 ceph-mon[45336]: pgmap v1298: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 6936/149154 objects misplaced (4.650%); 537 KiB/s, 8 objects/s recovering 2024-04-15T18:30:10.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:10 smithi159 ceph-mon[45336]: from='client.26021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:10.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:10.986 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.008 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:11.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:11.140 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.159 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:11.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:11.291 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.310 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:11.400 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1298,"stamp":"2024-04-15T18:30:09.345005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3258712632,"num_objects":49718,"num_object_clones":0,"num_object_copies":149154,"num_objects_missing_on_primary":2305,"num_objects_missing":2305,"num_objects_degraded":0,"num_objects_misplaced":6936,"num_objects_unfound":0,"num_objects_dirty":49718,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":136781,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39366,"num_bytes_recovered":2576964256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138260,"ondisk_log_size":138260,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13702716,"kb_used_data":10087976,"kb_used_omap":177,"kb_used_meta":3614414,"kb_avail":1111239108,"statfs":{"total":1151940427776,"available":1137908846592,"internally_reserved":0,"allocated":10330087424,"data_stored":10322574869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3701160388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-95682560,"num_objects":-1460,"num_object_clones":0,"num_object_copies":-4380,"num_objects_missing_on_primary":-58,"num_objects_missing":-58,"num_objects_degraded":0,"num_objects_misplaced":-177,"num_objects_unfound":0,"num_objects_dirty":-1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":3866624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.035247"},"pg_stats":[{"pgid":"5.f","version":"498'8715","reported_seq":14727,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.094132+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:30:07.094132+0000","last_peered":"2024-04-15T18:30:07.094132+0000","last_clean":"2024-04-15T18:30:07.094132+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:07.094132+0000","last_undegraded":"2024-04-15T18:30:07.094132+0000","last_fullsized":"2024-04-15T18:30:07.094132+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8715,"log_dups_size":0,"ondisk_log_size":8715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"498'8555","reported_seq":9004,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:06.860593+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:30:06.860593+0000","last_peered":"2024-04-15T18:30:06.860593+0000","last_clean":"2024-04-15T18:30:06.860593+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:06.860593+0000","last_undegraded":"2024-04-15T18:30:06.860593+0000","last_fullsized":"2024-04-15T18:30:06.860593+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8555,"log_dups_size":0,"ondisk_log_size":8555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205586432,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"497'8529","reported_seq":14330,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:06.449121+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:06.449121+0000","last_peered":"2024-04-15T18:30:06.449121+0000","last_clean":"2024-04-15T18:30:06.449121+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:06.449121+0000","last_undegraded":"2024-04-15T18:30:06.449121+0000","last_fullsized":"2024-04-15T18:30:06.449121+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588760,"num_objects":3077,"num_object_clones":0,"num_object_copies":9231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8529,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"498'8560","reported_seq":9063,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.211073+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:07.211073+0000","last_peered":"2024-04-15T18:30:07.211073+0000","last_clean":"2024-04-15T18:30:07.211073+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:07.211073+0000","last_undegraded":"2024-04-15T18:30:07.211073+0000","last_fullsized":"2024-04-15T18:30:07.211073+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205389824,"num_objects":3134,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"498'8647","reported_seq":14612,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.047771+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:30:07.047771+0000","last_peered":"2024-04-15T18:30:07.047771+0000","last_clean":"2024-04-15T18:30:07.047771+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:07.047771+0000","last_undegraded":"2024-04-15T18:30:07.047771+0000","last_fullsized":"2024-04-15T18:30:07.047771+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8647,"log_dups_size":0,"ondisk_log_size":8647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207945728,"num_objects":3173,"num_object_clones":0,"num_object_copies":9519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"498'8567","reported_seq":20659,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.056855+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:30:07.056855+0000","last_peered":"2024-04-15T18:30:07.056855+0000","last_clean":"2024-04-15T18:30:07.056855+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:07.056855+0000","last_undegraded":"2024-04-15T18:30:07.056855+0000","last_fullsized":"2024-04-15T18:30:07.056855+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8567,"log_dups_size":0,"ondisk_log_size":8567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201785344,"num_objects":3079,"num_object_clones":0,"num_object_copies":9237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8567,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"497'8507","reported_seq":8896,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-15T18:30:05.841509+0000","last_change":"2024-04-15T18:30:00.163141+0000","last_active":"2024-04-15T18:30:05.841509+0000","last_peered":"2024-04-15T18:30:05.841509+0000","last_clean":"2024-04-15T18:30:05.841509+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:05.841509+0000","last_undegraded":"2024-04-15T18:30:05.841509+0000","last_fullsized":"2024-04-15T18:30:05.841509+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":549,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206766080,"num_objects":3155,"num_object_clones":0,"num_object_copies":9465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8507,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"498'8560","reported_seq":8851,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:06.449075+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:06.449075+0000","last_peered":"2024-04-15T18:30:06.449075+0000","last_clean":"2024-04-15T18:30:06.449075+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:06.449075+0000","last_undegraded":"2024-04-15T18:30:06.449075+0000","last_fullsized":"2024-04-15T18:30:06.449075+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208928768,"num_objects":3188,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"498'8617","reported_seq":14621,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.061590+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:30:07.061590+0000","last_peered":"2024-04-15T18:30:07.061590+0000","last_clean":"2024-04-15T18:30:07.061590+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:07.061590+0000","last_undegraded":"2024-04-15T18:30:07.061590+0000","last_fullsized":"2024-04-15T18:30:07.061590+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:09.514603+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197853184,"num_objects":3019,"num_object_clones":0,"num_object_copies":9057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8617,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"498'8409","reported_seq":25654,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.089966+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:30:07.089966+0000","last_peered":"2024-04-15T18:30:07.089966+0000","last_clean":"2024-04-15T18:30:07.089966+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:07.089966+0000","last_undegraded":"2024-04-15T18:30:07.089966+0000","last_fullsized":"2024-04-15T18:30:07.089966+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:16.116671+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":9081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8409,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"498'8285","reported_seq":8734,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.052506+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:30:07.052506+0000","last_peered":"2024-04-15T18:30:07.052506+0000","last_clean":"2024-04-15T18:30:07.052506+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:07.052506+0000","last_undegraded":"2024-04-15T18:30:07.052506+0000","last_fullsized":"2024-04-15T18:30:07.052506+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.463522+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198508544,"num_objects":3029,"num_object_clones":0,"num_object_copies":9087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8285,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"497'8696","reported_seq":14665,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-15T18:30:04.803324+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:04.803324+0000","last_peered":"2024-04-15T18:30:04.803324+0000","last_clean":"2024-04-15T18:30:04.803324+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:04.803324+0000","last_undegraded":"2024-04-15T18:30:04.803324+0000","last_fullsized":"2024-04-15T18:30:04.803324+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8696,"log_dups_size":0,"ondisk_log_size":8696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207618048,"num_objects":3168,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"497'8436","reported_seq":8783,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-15T18:30:04.648181+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:04.648181+0000","last_peered":"2024-04-15T18:30:04.648181+0000","last_clean":"2024-04-15T18:30:04.648181+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:04.648181+0000","last_undegraded":"2024-04-15T18:30:04.648181+0000","last_fullsized":"2024-04-15T18:30:04.648181+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206176256,"num_objects":3146,"num_object_clones":0,"num_object_copies":9438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"498'8328","reported_seq":19246,"reported_epoch":498,"state":"active+recovering","last_fresh":"2024-04-15T18:30:06.583304+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:06.583304+0000","last_peered":"2024-04-15T18:30:06.583304+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:06.583304+0000","last_undegraded":"2024-04-15T18:30:06.583304+0000","last_fullsized":"2024-04-15T18:30:06.583304+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":2305,"num_objects_missing":2305,"num_objects_degraded":0,"num_objects_misplaced":6936,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2653,"num_bytes_recovered":173735936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,3,4,7,11","objects":1},{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2305},{"shards":"3,4,7,11","objects":10}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"498'8641","reported_seq":9094,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.720172+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:07.720172+0000","last_peered":"2024-04-15T18:30:07.720172+0000","last_clean":"2024-04-15T18:30:07.720172+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:07.720172+0000","last_undegraded":"2024-04-15T18:30:07.720172+0000","last_fullsized":"2024-04-15T18:30:07.720172+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":8641,"log_dups_size":0,"ondisk_log_size":8641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207814656,"num_objects":3171,"num_object_clones":0,"num_object_copies":9513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"498'12372","reported_seq":29988,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:07.724400+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:30:07.724400+0000","last_peered":"2024-04-15T18:30:07.724400+0000","last_clean":"2024-04-15T18:30:07.724400+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:07.724400+0000","last_undegraded":"2024-04-15T18:30:07.724400+0000","last_fullsized":"2024-04-15T18:30:07.724400+0000","mapping_epoch":440,"log_start":"373'2301","ondisk_log_start":"373'2301","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201850880,"num_objects":3080,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1241,"reported_epoch":497,"state":"active+clean","last_fresh":"2024-04-15T18:30:04.445458+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:30:04.445458+0000","last_peered":"2024-04-15T18:30:04.445458+0000","last_clean":"2024-04-15T18:30:04.445458+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:04.445458+0000","last_undegraded":"2024-04-15T18:30:04.445458+0000","last_fullsized":"2024-04-15T18:30:04.445458+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3258122264,"num_objects":49716,"num_object_clones":0,"num_object_copies":149148,"num_objects_missing_on_primary":2305,"num_objects_missing":2305,"num_objects_degraded":0,"num_objects_misplaced":6936,"num_objects_unfound":0,"num_objects_dirty":49716,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136600,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39356,"num_bytes_recovered":2574319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10295128064,"data_stored":10295115848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138123,"ondisk_log_size":138123,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926075,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1272924,"kb_used_data":850252,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92472228,"statfs":{"total":95995035648,"available":94691561472,"internally_reserved":0,"allocated":870658048,"data_stored":870024405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":432798148},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187712,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1318408,"kb_used_data":1004032,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92426744,"statfs":{"total":95995035648,"available":94644985856,"internally_reserved":0,"allocated":1028128768,"data_stored":1027502769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":321911232},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919740,"kb_used_data":599796,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92825412,"statfs":{"total":95995035648,"available":95053221888,"internally_reserved":0,"allocated":614191104,"data_stored":613569570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874505,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571840,"kb_used_data":401844,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":93173312,"statfs":{"total":95995035648,"available":95409471488,"internally_reserved":0,"allocated":411488256,"data_stored":410866722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168846,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234640,"kb_used_data":914688,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92510512,"statfs":{"total":95995035648,"available":94730764288,"internally_reserved":0,"allocated":936640512,"data_stored":936011354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463188,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000456,"kb_used_data":654140,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92744696,"statfs":{"total":95995035648,"available":94970568704,"internally_reserved":0,"allocated":669839360,"data_stored":669214274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163408,"kb_used_data":793472,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92581744,"statfs":{"total":95995035648,"available":94803705856,"internally_reserved":0,"allocated":812515328,"data_stored":811889329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810496,"kb_used_data":1454452,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91934656,"statfs":{"total":95995035648,"available":94141087744,"internally_reserved":0,"allocated":1489358848,"data_stored":1488737314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193925,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707212,"kb_used_data":590528,"kb_used_omap":114,"kb_used_meta":116557,"kb_avail":93037940,"statfs":{"total":95995035648,"available":95270850560,"internally_reserved":0,"allocated":604700672,"data_stored":604074673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":119354609},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653772,"kb_used_data":308224,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091380,"statfs":{"total":95995035648,"available":95325573120,"internally_reserved":0,"allocated":315621376,"data_stored":314995377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986461,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1492172,"kb_used_data":1127808,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92252980,"statfs":{"total":95995035648,"available":94467051520,"internally_reserved":0,"allocated":1154875392,"data_stored":1154249393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703356,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557648,"kb_used_data":1388740,"kb_used_omap":4,"kb_used_meta":168891,"kb_avail":92187504,"statfs":{"total":95995035648,"available":94400004096,"internally_reserved":0,"allocated":1422069760,"data_stored":1421439689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":172945089},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1486880768,"data_stored":1486880768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":810090496,"data_stored":810090496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":602537984,"data_stored":602537984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1150287872,"data_stored":1150287872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1417547776,"data_stored":1417543704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":667287552,"data_stored":667287552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":933236736,"data_stored":933232664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409468928,"data_stored":409468928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":611385344,"data_stored":611385344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1024065536,"data_stored":1024065536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":868421632,"data_stored":868417560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:11.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:11.443 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.467 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:11.571 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:11.599 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.618 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:11.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:11.749 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.768 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:11.874 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:11.902 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:11.925 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:11.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:12.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:12.059 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:12.613 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:12.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:12.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:12 smithi049 ceph-mon[40087]: pgmap v1299: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 6936/148659 objects misplaced (4.666%); 472 KiB/s, 7 objects/s recovering 2024-04-15T18:30:12.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:12.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:12 smithi082 ceph-mon[44206]: pgmap v1299: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 6936/148659 objects misplaced (4.666%); 472 KiB/s, 7 objects/s recovering 2024-04-15T18:30:12.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:12.762 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:12.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:12.865 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:12.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:12.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:12 smithi159 ceph-mon[45336]: pgmap v1299: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 6936/148659 objects misplaced (4.666%); 472 KiB/s, 7 objects/s recovering 2024-04-15T18:30:12.893 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.2 e275,235 3:4c80762f:::benchmark_data_smithi159_91621_object20580:head,242'8026)", 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 597.891223642, 2024-04-15T18:30:12.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37528735600000002, 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011470+0000", 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011471+0000", 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.011475+0000", 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6940000000000001e-06 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386618+0000", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37514348800000002 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386630+0000", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1559999999999999e-05 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386699+0000", 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9227000000000007e-05 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386705+0000", 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4790000000000001e-06 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.386758+0000", 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2429000000000002e-05 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.2 e275,235 3:4aa76ad9:::benchmark_data_smithi159_91621_object21018:head,242'8068)", 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 597.43332026300004, 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59152146100000003, 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469374+0000", 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469375+0000", 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8899999999999999e-07 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.469380+0000", 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9849999999999997e-06 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060797+0000", 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59141723800000001 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060803+0000", 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5479999999999998e-06 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060850+0000", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6220999999999997e-05 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060855+0000", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8109999999999999e-06 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.060895+0000", 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9969000000000003e-05 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e275,235 3:af3b238d:::benchmark_data_smithi159_91621_object23467:head,243'8240)", 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 597.11722862500005, 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29041672899999998, 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785465+0000", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785466+0000", 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2799999999999996e-07 2024-04-15T18:30:12.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:15.785470+0000", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2039999999999999e-06 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075781+0000", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29031098599999999 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075787+0000", 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3229999999999997e-06 2024-04-15T18:30:12.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075835+0000", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8937999999999998e-05 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075842+0000", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9310000000000002e-06 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.075882+0000", 2024-04-15T18:30:12.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9818999999999999e-05 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e276,235 3:a6b57d98:::benchmark_data_smithi159_91621_object30030:head,247'8679)", 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 596.11587578199999, 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33609018000000002, 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.910 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786818+0000", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786819+0000", 2024-04-15T18:30:12.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3599999999999999e-07 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:16.786822+0000", 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3239999999999999e-06 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122780+0000", 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.335958219 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122786+0000", 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4299999999999997e-06 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122844+0000", 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7929000000000001e-05 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122853+0000", 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3780000000000007e-06 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.122908+0000", 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5464000000000003e-05 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e277,235 3:ad41a5e8:::benchmark_data_smithi159_91621_object45130:head,253'9622)", 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 595.00502616999995, 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24596616600000001, 2024-04-15T18:30:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897668+0000", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2800000000000002e-07 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:17.897672+0000", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3890000000000001e-06 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143032+0000", 2024-04-15T18:30:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24536002900000001 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143038+0000", 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0299999999999999e-06 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143562+0000", 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052384700000000003 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143572+0000", 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0637e-05 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.143634+0000", 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1805999999999997e-05 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e278,235 3:a6cd20bc:::benchmark_data_smithi159_91621_object76133:head,266'11576)", 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 594.08593779499995, 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38370323899999997, 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816756+0000", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816757+0000", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4000000000000002e-07 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:18.816760+0000", 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3629999999999998e-06 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200349+0000", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.38358931899999998 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200355+0000", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5310000000000001e-06 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200415+0000", 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9953e-05 2024-04-15T18:30:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200422+0000", 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0849999999999999e-06 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.200460+0000", 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7548e-05 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(3.5 e279,235 3:ac24a022:::benchmark_data_smithi159_91621_object76992:head,266'11636)", 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 593.65802325599998, 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227295887, 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244671+0000", 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244672+0000", 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.244681+0000", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4129999999999995e-06 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471801+0000", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227119823 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471816+0000", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4686e-05 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471897+0000", 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1538000000000004e-05 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471905+0000", 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8220000000000007e-06 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:20:19.471967+0000", 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1418e-05 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:39801 4.c e301/284 4:3ea5c1e4:::benchmark_data_smithi159_102124_object39800:head v 301'2488, mlcod=301'2484)", 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 530.80576839100002, 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21772696599999999, 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096926+0000", 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8100000000000003e-07 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.096981+0000", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4803999999999999e-05 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313237+0000", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21625582600000001 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313245+0000", 2024-04-15T18:30:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1610000000000003e-06 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313366+0000", 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120587 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.313392+0000", 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6222000000000001e-05 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.314621+0000", 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001229401 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:21:22.314653+0000", 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1483999999999999e-05 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25126.0:89299 4.0 4:078bede3:::benchmark_data_smithi159_102124_object89298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e322)", 2024-04-15T18:30:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:11.363845+0000", 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 481.53884930499999, 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77776312199999997, 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25126", 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 89299 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363845+0000", 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363845+0000", 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363846+0000", 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363902+0000", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5572000000000003e-05 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363913+0000", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1762e-05 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363920+0000", 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.489e-06 2024-04-15T18:30:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.363983+0000", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2774000000000006e-05 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.364131+0000", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014858800000000001 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.364275+0000", 2024-04-15T18:30:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143566 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.367253+0000", 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002978404 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.898584+0000", 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53133045400000001 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.141523+0000", 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24293953800000001 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.141567+0000", 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3522000000000001e-05 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:12.141608+0000", 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1254999999999997e-05 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:89341 4.c e322/284 4:33e79400:::benchmark_data_smithi159_102124_object89340:head v 322'5657, mlcod=322'5653)", 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:11.421235+0000", 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 481.48145943200001, 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26474382499999999, 2024-04-15T18:30:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.421235+0000", 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.421235+0000", 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.421235+0000", 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5600000000000005e-07 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.421275+0000", 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9561000000000003e-05 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.682821+0000", 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26154650800000001 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.682838+0000", 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6079e-05 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.682937+0000", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9784000000000006e-05 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.682957+0000", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9233999999999999e-05 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.685922+0000", 2024-04-15T18:30:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002965358 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:11.685978+0000", 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6545000000000002e-05 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:129174 4.c e334/284 4:3ab79f80:::benchmark_data_smithi159_102124_object39820:head v 334'8147, mlcod=334'8144)", 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:22:58.735006+0000", 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.16768859199999, 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21561704500000001, 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.735006+0000", 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.735006+0000", 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.735006+0000", 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3699999999999999e-07 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.735015+0000", 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2730000000000002e-06 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.946226+0000", 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21121177599999999 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.946235+0000", 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4030000000000007e-06 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.946299+0000", 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4535000000000005e-05 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.946329+0000", 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9462e-05 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.950578+0000", 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042492169999999996 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:22:58.950623+0000", 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4542000000000001e-05 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:134801 4.4 e335/318 4:264779a6:::benchmark_data_smithi159_102124_object45447:head v 335'8574, mlcod=309'3953)", 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:23:05.604796+0000", 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 427.29789818500001, 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.234263624, 2024-04-15T18:30:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.604796+0000", 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.604796+0000", 2024-04-15T18:30:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.604797+0000", 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3900000000000004e-07 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.604803+0000", 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4289999999999998e-06 2024-04-15T18:30:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.835375+0000", 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23057211999999999 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.835385+0000", 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7799999999999995e-06 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.835450+0000", 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5271000000000006e-05 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.835481+0000", 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0719000000000002e-05 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.838996+0000", 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035148190000000002 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.839060+0000", 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3847e-05 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25126.0:134860 4.f e335/284 4:faae49a3:::benchmark_data_smithi159_102124_object45506:head v 335'8484, mlcod=335'8482)", 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:23:05.864744+0000", 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 427.03795013000001, 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.242350913, 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.864744+0000", 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.864744+0000", 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.864745+0000", 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.850000000000001e-07 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:05.864753+0000", 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7519999999999996e-06 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.103451+0000", 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23869786700000001 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.103461+0000", 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0277e-05 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.103523+0000", 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2106000000000005e-05 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.103541+0000", 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7972999999999999e-05 2024-04-15T18:30:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.107044+0000", 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035028160000000002 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:23:06.107095+0000", 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1137000000000002e-05 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81983 5.1 5:8c9a41f5:::benchmark_data_smithi159_116570_object81982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 250.43132318900001, 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.203960954, 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:12.960 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81983 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:30:12.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9400000000000001e-07 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471416+0000", 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5123000000000001e-05 2024-04-15T18:30:12.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471423+0000", 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.058e-06 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471427+0000", 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7910000000000002e-06 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472653+0000", 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001225711 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472657+0000", 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9720000000000003e-06 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624096+0000", 2024-04-15T18:30:12.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15143899299999999 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624272+0000", 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017645900000000001 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624432+0000", 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160011 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639480+0000", 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015047414 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.671266+0000", 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031786552000000003 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.671312+0000", 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5293000000000002e-05 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673194+0000", 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018826660000000001 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673258+0000", 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3279000000000002e-05 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675332+0000", 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020742379999999999 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.968 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81985 5.1 5:8cf02e22:::benchmark_data_smithi159_116570_object81984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 250.43126152299999, 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20675183599999999, 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81985 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.969 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9299999999999999e-07 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471475+0000", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2219999999999999e-05 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471481+0000", 2024-04-15T18:30:12.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2659999999999998e-06 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471485+0000", 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3610000000000001e-06 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472676+0000", 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001191613 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472680+0000", 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.861e-06 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.626705+0000", 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15402469899999999 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.626866+0000", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160929 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.627037+0000", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017111900000000001 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639490+0000", 2024-04-15T18:30:12.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012453331999999999 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673060+0000", 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033569977000000001 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673139+0000", 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9054999999999994e-05 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675506+0000", 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002366983 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675584+0000", 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7785000000000001e-05 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678184+0000", 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026002429999999999 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82009 5.1 5:894d6e9c:::benchmark_data_smithi159_116570_object82008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 250.43120435899999, 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.209574175, 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.977 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82009 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3799999999999999e-07 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471527+0000", 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6906000000000002e-05 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471533+0000", 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0700000000000003e-06 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471536+0000", 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3440000000000001e-06 2024-04-15T18:30:12.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472699+0000", 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011631460000000001 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472703+0000", 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9389999999999999e-06 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629581+0000", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15687794899999999 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629719+0000", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013762600000000001 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629866+0000", 2024-04-15T18:30:12.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014707300000000001 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639639+0000", 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097726110000000005 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675401+0000", 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.035762032999999999 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675457+0000", 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6598999999999998e-05 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678355+0000", 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028978160000000001 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678424+0000", 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9103999999999994e-05 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681064+0000", 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026397209999999998 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82035 5.1 5:83347078:::benchmark_data_smithi159_116570_object82034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 250.431153284, 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21252658299999999, 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:12.986 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82035 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4299999999999999e-07 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471594+0000", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3236000000000003e-05 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471604+0000", 2024-04-15T18:30:12.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0118999999999999e-05 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471608+0000", 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9720000000000003e-06 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472725+0000", 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011170900000000001 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472729+0000", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9280000000000003e-06 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632068+0000", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15933883600000001 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632211+0000", 2024-04-15T18:30:12.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142848 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632315+0000", 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103732 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639649+0000", 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073340530000000001 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678251+0000", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.038601997999999998 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678302+0000", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0932e-05 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681246+0000", 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029446229999999999 2024-04-15T18:30:12.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681321+0000", 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4324000000000002e-05 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684067+0000", 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027466489999999999 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:12.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82070 5.1 5:8955e426:::benchmark_data_smithi159_116570_object82069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 250.431079582, 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21553371599999999, 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:12.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82070 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:12.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7000000000000001e-07 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471677+0000", 2024-04-15T18:30:12.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2382e-05 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471684+0000", 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2139999999999999e-06 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471688+0000", 2024-04-15T18:30:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7280000000000001e-06 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472743+0000", 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010547969999999999 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472747+0000", 2024-04-15T18:30:12.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0280000000000002e-06 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.634900+0000", 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16215306800000001 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.635056+0000", 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015597000000000001 2024-04-15T18:30:12.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.635243+0000", 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018687900000000001 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639659+0000", 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044159179999999996 2024-04-15T18:30:13.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681129+0000", 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.041470411999999998 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681188+0000", 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8548000000000003e-05 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684230+0000", 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030423719999999998 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684300+0000", 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9839999999999995e-05 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:13.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.687148+0000", 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028482899999999998 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPull(5.1 e428/416 cost 1049576)", 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 180.029852444, 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39438863499999999, 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:13.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872843+0000", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.048e-06 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872856+0000", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3015e-05 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266896+0000", 2024-04-15T18:30:13.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39404034100000002 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266905+0000", 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2920000000000001e-06 2024-04-15T18:30:13.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266985+0000", 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9593999999999997e-05 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.267230+0000", 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000245345 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPull(5.1 e477/416 cost 1049576)", 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:29:00.051018+0000", 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 72.851676019999999, 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00065439700000000003, 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:13.008 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051018+0000", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051018+0000", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051019+0000", 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.359e-06 2024-04-15T18:30:13.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051039+0000", 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9704e-05 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051087+0000", 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7862999999999999e-05 2024-04-15T18:30:13.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051099+0000", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2052000000000001e-05 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051171+0000", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1693000000000006e-05 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:00.051672+0000", 2024-04-15T18:30:13.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00050172599999999997 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:13.012 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:30:13.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:13.013 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:13.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:13.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:13.140 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:13.158 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:13.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:13.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:13.290 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:13.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:13.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:13.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:13.448 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:14.099 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:14.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:14.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:14.234 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:14.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:14.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:14.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:14.461 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:14.481 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:14.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:14.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:14.614 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:14.634 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:14.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:14.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:14 smithi049 ceph-mon[40087]: pgmap v1300: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 6748/146001 objects misplaced (4.622%); 881 KiB/s, 13 objects/s recovering 2024-04-15T18:30:14.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:30:14.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:14.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:14 smithi082 ceph-mon[44206]: pgmap v1300: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 6748/146001 objects misplaced (4.622%); 881 KiB/s, 13 objects/s recovering 2024-04-15T18:30:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:14 smithi159 ceph-mon[45336]: pgmap v1300: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 223 op/s; 6748/146001 objects misplaced (4.622%); 881 KiB/s, 13 objects/s recovering 2024-04-15T18:30:15.184 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:15.222 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:15.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:15.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:15.356 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:15.375 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:15.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:30:15.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:15.506 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:15.527 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:15.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:15.631 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:15.655 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3012029993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.655 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.657 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:15.675 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:15.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:15.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3012029993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:15.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:15.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3012029993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:15.960 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:15.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:16.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:16.094 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:16.115 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:16.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:16.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:16.248 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:16.538 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:16.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:16 smithi049 ceph-mon[40087]: pgmap v1301: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6748/146001 objects misplaced (4.622%); 787 KiB/s, 12 objects/s recovering 2024-04-15T18:30:16.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:16 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:16.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:16 smithi049 ceph-mon[40087]: osdmap e499: 12 total, 12 up, 12 in 2024-04-15T18:30:16.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:30:16.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:30:16.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:30:16.773 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:30:16.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:16 smithi082 ceph-mon[44206]: pgmap v1301: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6748/146001 objects misplaced (4.622%); 787 KiB/s, 12 objects/s recovering 2024-04-15T18:30:16.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:16 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:16.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:16.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:16 smithi082 ceph-mon[44206]: osdmap e499: 12 total, 12 up, 12 in 2024-04-15T18:30:16.800 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:30:16.825 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:30:16.850 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:30:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:16 smithi159 ceph-mon[45336]: pgmap v1301: 17 pgs: 1 active+recovering, 16 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 6748/146001 objects misplaced (4.622%); 787 KiB/s, 12 objects/s recovering 2024-04-15T18:30:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:16 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:16 smithi159 ceph-mon[45336]: osdmap e499: 12 total, 12 up, 12 in 2024-04-15T18:30:16.972 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:30:16.997 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:30:17.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:30:17.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:30:17.072 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:30:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:30:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:30:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:30:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:30:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:30:17.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:30:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3122894067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:17 smithi049 ceph-mon[40087]: osdmap e500: 12 total, 12 up, 12 in 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3122894067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:17 smithi082 ceph-mon[44206]: osdmap e500: 12 total, 12 up, 12 in 2024-04-15T18:30:17.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3122894067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:17.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:17.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:17 smithi159 ceph-mon[45336]: osdmap e500: 12 total, 12 up, 12 in 2024-04-15T18:30:18.099 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:18.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:18.234 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:18.259 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:18.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:18.395 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:18.413 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:18.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:18.544 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:18.562 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:18 smithi159 ceph-mon[45336]: pgmap v1304: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 6553/143673 objects misplaced (4.561%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:18.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:18 smithi082 ceph-mon[44206]: pgmap v1304: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 6553/143673 objects misplaced (4.561%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:18.693 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:18.715 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:18.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:18 smithi049 ceph-mon[40087]: pgmap v1304: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 228 op/s; 6553/143673 objects misplaced (4.561%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:18.729 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:18.730 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:18.820 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:18.846 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:18.864 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:18.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:18.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:18.995 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.015 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:19.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:19.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.163 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:19.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:19.293 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.311 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:19.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:30:19.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:19.441 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.463 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:19.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:19.594 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.613 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:19.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:19 smithi049 ceph-mon[40087]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:19.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:19 smithi159 ceph-mon[45336]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:19.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:19 smithi082 ceph-mon[44206]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:19.818 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:19.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:19.951 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:19.969 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:19.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:20.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:20.101 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.121 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:20.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:20.252 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.271 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:20.353 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1304,"stamp":"2024-04-15T18:30:17.346473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3138978360,"num_objects":47891,"num_object_clones":0,"num_object_copies":143673,"num_objects_missing_on_primary":2179,"num_objects_missing":2179,"num_objects_degraded":0,"num_objects_misplaced":6553,"num_objects_unfound":0,"num_objects_dirty":47891,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":138608,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39495,"num_bytes_recovered":2585418400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139987,"ondisk_log_size":139987,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13354508,"kb_used_data":9777248,"kb_used_omap":177,"kb_used_meta":3576974,"kb_avail":1111587316,"statfs":{"total":1151940427776,"available":1138265411584,"internally_reserved":0,"allocated":10011901952,"data_stored":10004344497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3662821828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-119734272,"num_objects":-1827,"num_object_clones":0,"num_object_copies":-5481,"num_objects_missing_on_primary":-126,"num_objects_missing":-126,"num_objects_degraded":0,"num_objects_misplaced":-383,"num_objects_unfound":0,"num_objects_dirty":-1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001468"},"pg_stats":[{"pgid":"5.f","version":"498'8785","reported_seq":14797,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:12.197264+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:30:12.197264+0000","last_peered":"2024-04-15T18:30:12.197264+0000","last_clean":"2024-04-15T18:30:12.197264+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:12.197264+0000","last_undegraded":"2024-04-15T18:30:12.197264+0000","last_fullsized":"2024-04-15T18:30:12.197264+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8785,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"499'8690","reported_seq":9144,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:17.318384+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:30:17.318384+0000","last_peered":"2024-04-15T18:30:17.318384+0000","last_clean":"2024-04-15T18:30:17.318384+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:17.318384+0000","last_undegraded":"2024-04-15T18:30:17.318384+0000","last_fullsized":"2024-04-15T18:30:17.318384+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8690,"log_dups_size":0,"ondisk_log_size":8690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196739072,"num_objects":3002,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"499'8662","reported_seq":14463,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:16.613387+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:16.613387+0000","last_peered":"2024-04-15T18:30:16.613387+0000","last_clean":"2024-04-15T18:30:16.613387+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:16.613387+0000","last_undegraded":"2024-04-15T18:30:16.613387+0000","last_fullsized":"2024-04-15T18:30:16.613387+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8662,"log_dups_size":0,"ondisk_log_size":8662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192872472,"num_objects":2944,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8662,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"499'8684","reported_seq":9189,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:17.164270+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:17.164270+0000","last_peered":"2024-04-15T18:30:17.164270+0000","last_clean":"2024-04-15T18:30:17.164270+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:17.164270+0000","last_undegraded":"2024-04-15T18:30:17.164270+0000","last_fullsized":"2024-04-15T18:30:17.164270+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":8684,"log_dups_size":0,"ondisk_log_size":8684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"499'8783","reported_seq":14751,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:17.318384+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:30:17.318384+0000","last_peered":"2024-04-15T18:30:17.318384+0000","last_clean":"2024-04-15T18:30:17.318384+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:17.318384+0000","last_undegraded":"2024-04-15T18:30:17.318384+0000","last_fullsized":"2024-04-15T18:30:17.318384+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8783,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"499'8689","reported_seq":20782,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:16.920961+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:30:16.920961+0000","last_peered":"2024-04-15T18:30:16.920961+0000","last_clean":"2024-04-15T18:30:16.920961+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:16.920961+0000","last_undegraded":"2024-04-15T18:30:16.920961+0000","last_fullsized":"2024-04-15T18:30:16.920961+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:17.753685+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193789952,"num_objects":2957,"num_object_clones":0,"num_object_copies":8871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"499'8659","reported_seq":9052,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:15.785549+0000","last_change":"2024-04-15T18:30:00.163141+0000","last_active":"2024-04-15T18:30:15.785549+0000","last_peered":"2024-04-15T18:30:15.785549+0000","last_clean":"2024-04-15T18:30:15.785549+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:15.785549+0000","last_undegraded":"2024-04-15T18:30:15.785549+0000","last_fullsized":"2024-04-15T18:30:15.785549+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":549,"log_size":8659,"log_dups_size":0,"ondisk_log_size":8659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:19.001820+0000","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196804608,"num_objects":3003,"num_object_clones":0,"num_object_copies":9009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8659,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"499'8682","reported_seq":8975,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:16.448851+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:16.448851+0000","last_peered":"2024-04-15T18:30:16.448851+0000","last_clean":"2024-04-15T18:30:16.448851+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:16.448851+0000","last_undegraded":"2024-04-15T18:30:16.448851+0000","last_fullsized":"2024-04-15T18:30:16.448851+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200933376,"num_objects":3066,"num_object_clones":0,"num_object_copies":9198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8682,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"498'8680","reported_seq":14684,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:12.322201+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:30:12.322201+0000","last_peered":"2024-04-15T18:30:12.322201+0000","last_clean":"2024-04-15T18:30:12.322201+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:12.322201+0000","last_undegraded":"2024-04-15T18:30:12.322201+0000","last_fullsized":"2024-04-15T18:30:12.322201+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193724416,"num_objects":2956,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8680,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"499'8550","reported_seq":25797,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:17.212673+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:30:17.212673+0000","last_peered":"2024-04-15T18:30:17.212673+0000","last_clean":"2024-04-15T18:30:17.212673+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:17.212673+0000","last_undegraded":"2024-04-15T18:30:17.212673+0000","last_fullsized":"2024-04-15T18:30:17.212673+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8550,"log_dups_size":0,"ondisk_log_size":8550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":8658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8550,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"499'8407","reported_seq":8858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:16.807014+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:30:16.807014+0000","last_peered":"2024-04-15T18:30:16.807014+0000","last_clean":"2024-04-15T18:30:16.807014+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:16.807014+0000","last_undegraded":"2024-04-15T18:30:16.807014+0000","last_fullsized":"2024-04-15T18:30:16.807014+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":8407,"log_dups_size":0,"ondisk_log_size":8407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:21.931162+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"498'8817","reported_seq":14788,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:15.305712+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:15.305712+0000","last_peered":"2024-04-15T18:30:15.305712+0000","last_clean":"2024-04-15T18:30:15.305712+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:15.305712+0000","last_undegraded":"2024-04-15T18:30:15.305712+0000","last_fullsized":"2024-04-15T18:30:15.305712+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199688192,"num_objects":3047,"num_object_clones":0,"num_object_copies":9141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"498'8554","reported_seq":8903,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:15.301461+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:15.301461+0000","last_peered":"2024-04-15T18:30:15.301461+0000","last_clean":"2024-04-15T18:30:15.301461+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:15.301461+0000","last_undegraded":"2024-04-15T18:30:15.301461+0000","last_fullsized":"2024-04-15T18:30:15.301461+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8554,"log_dups_size":0,"ondisk_log_size":8554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"499'8456","reported_seq":19888,"reported_epoch":499,"state":"active+recovering","last_fresh":"2024-04-15T18:30:16.727242+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:16.727242+0000","last_peered":"2024-04-15T18:30:16.727242+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:16.727242+0000","last_undegraded":"2024-04-15T18:30:16.727242+0000","last_fullsized":"2024-04-15T18:30:16.727242+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189923328,"num_objects":2898,"num_object_clones":0,"num_object_copies":8694,"num_objects_missing_on_primary":2179,"num_objects_missing":2179,"num_objects_degraded":0,"num_objects_misplaced":6553,"num_objects_unfound":0,"num_objects_dirty":2898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2782,"num_bytes_recovered":182190080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":2179},{"shards":"3,4,7,11","objects":8}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"498'8707","reported_seq":9161,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:12.863205+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:12.863205+0000","last_peered":"2024-04-15T18:30:12.863205+0000","last_clean":"2024-04-15T18:30:12.863205+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:12.863205+0000","last_undegraded":"2024-04-15T18:30:12.863205+0000","last_fullsized":"2024-04-15T18:30:12.863205+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203489280,"num_objects":3105,"num_object_clones":0,"num_object_copies":9315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"498'12446","reported_seq":30062,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-04-15T18:30:12.790073+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:30:12.790073+0000","last_peered":"2024-04-15T18:30:12.790073+0000","last_clean":"2024-04-15T18:30:12.790073+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:12.790073+0000","last_undegraded":"2024-04-15T18:30:12.790073+0000","last_fullsized":"2024-04-15T18:30:12.790073+0000","mapping_epoch":440,"log_start":"374'2401","ondisk_log_start":"374'2401","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:13.214211+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197001216,"num_objects":3006,"num_object_clones":0,"num_object_copies":9018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1245,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-04-15T18:30:15.456848+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:30:15.456848+0000","last_peered":"2024-04-15T18:30:15.456848+0000","last_clean":"2024-04-15T18:30:15.456848+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:15.456848+0000","last_undegraded":"2024-04-15T18:30:15.456848+0000","last_fullsized":"2024-04-15T18:30:15.456848+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3138387992,"num_objects":47889,"num_object_clones":0,"num_object_copies":143667,"num_objects_missing_on_primary":2179,"num_objects_missing":2179,"num_objects_degraded":0,"num_objects_misplaced":6553,"num_objects_unfound":0,"num_objects_dirty":47889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138427,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39485,"num_bytes_recovered":2582773760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9966989312,"data_stored":9966977096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139850,"ondisk_log_size":139850,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926077,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1222368,"kb_used_data":816344,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92522784,"statfs":{"total":95995035648,"available":94743330816,"internally_reserved":0,"allocated":835936256,"data_stored":835298098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187713,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280776,"kb_used_data":983040,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92464376,"statfs":{"total":95995035648,"available":94683521024,"internally_reserved":0,"allocated":1006632960,"data_stored":1006006961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":304871872},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514759,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899608,"kb_used_data":575500,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92845544,"statfs":{"total":95995035648,"available":95073837056,"internally_reserved":0,"allocated":589312000,"data_stored":588681486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":540552,"kb_used_data":387200,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93204600,"statfs":{"total":95995035648,"available":95441510400,"internally_reserved":0,"allocated":396492800,"data_stored":395866801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168848,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209108,"kb_used_data":889164,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92536044,"statfs":{"total":95995035648,"available":94756909056,"internally_reserved":0,"allocated":910503936,"data_stored":909870313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463190,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977104,"kb_used_data":630792,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92768048,"statfs":{"total":95995035648,"available":94994481152,"internally_reserved":0,"allocated":645931008,"data_stored":645301457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1130900,"kb_used_data":760972,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92614252,"statfs":{"total":95995035648,"available":94836994048,"internally_reserved":0,"allocated":779235328,"data_stored":778604814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760920,"kb_used_data":1400716,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91984232,"statfs":{"total":95995035648,"available":94191853568,"internally_reserved":0,"allocated":1434333184,"data_stored":1433702670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681492,"kb_used_data":564812,"kb_used_omap":114,"kb_used_meta":116557,"kb_avail":93063660,"statfs":{"total":95995035648,"available":95297187840,"internally_reserved":0,"allocated":578367488,"data_stored":577736974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":119354609},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609056,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653772,"kb_used_data":308224,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091380,"statfs":{"total":95995035648,"available":95325573120,"internally_reserved":0,"allocated":315621376,"data_stored":314995377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986462,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465992,"kb_used_data":1101632,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92279160,"statfs":{"total":95995035648,"available":94493859840,"internally_reserved":0,"allocated":1128071168,"data_stored":1127445169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703357,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531916,"kb_used_data":1358852,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":92213236,"statfs":{"total":95995035648,"available":94426353664,"internally_reserved":0,"allocated":1391464448,"data_stored":1390834377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1431502848,"data_stored":1431502848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":776994816,"data_stored":776994816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":575930368,"data_stored":575930368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1123024896,"data_stored":1123024896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1384648704,"data_stored":1384644632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":641925120,"data_stored":641925120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":905777152,"data_stored":905773080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":393674752,"data_stored":393674752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":585236480,"data_stored":585236480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1001914368,"data_stored":1001914368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":832442368,"data_stored":832438296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:20.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:20.374 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:20.400 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.419 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:20.523 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:20.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:20 smithi082 ceph-mon[44206]: pgmap v1305: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 6553/142833 objects misplaced (4.588%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:20.548 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.567 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:20.671 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:20.697 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:20 smithi049 ceph-mon[40087]: pgmap v1305: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 6553/142833 objects misplaced (4.588%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:20.716 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:20.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:20.864 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:20.892 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:20.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:20 smithi159 ceph-mon[45336]: pgmap v1305: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 242 op/s; 6553/142833 objects misplaced (4.588%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:20.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:20.899 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:20.915 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:21.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:21.034 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:21.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:21.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:21.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:21.274 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:21.294 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:21.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:21.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:21.434 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:21.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:30:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:22 smithi049 ceph-mon[40087]: pgmap v1306: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 6553/142221 objects misplaced (4.608%); 520 KiB/s, 8 objects/s recovering 2024-04-15T18:30:22.773 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:30:22.773 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:30:22.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:22 smithi082 ceph-mon[44206]: pgmap v1306: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 6553/142221 objects misplaced (4.608%); 520 KiB/s, 8 objects/s recovering 2024-04-15T18:30:22.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:22 smithi159 ceph-mon[45336]: pgmap v1306: 17 pgs: 1 active+recovering, 16 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 6553/142221 objects misplaced (4.608%); 520 KiB/s, 8 objects/s recovering 2024-04-15T18:30:23.084 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:23.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:23.219 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:23.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:23.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:23.369 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:23.389 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:23.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:23.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:23.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:23.520 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:23.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:23.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:23.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:23.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:23.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:23.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:23.825 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:23.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:30:23.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:23.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:23.975 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:23.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:24.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:24.106 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:24.127 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:24.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:24.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:24.265 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:24.440 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:24.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:24 smithi049 ceph-mon[40087]: pgmap v1307: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6359/139314 objects misplaced (4.565%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:24.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:24 smithi049 ceph-mon[40087]: osdmap e501: 12 total, 12 up, 12 in 2024-04-15T18:30:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:24.578 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:24 smithi082 ceph-mon[44206]: pgmap v1307: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6359/139314 objects misplaced (4.565%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:24 smithi082 ceph-mon[44206]: osdmap e501: 12 total, 12 up, 12 in 2024-04-15T18:30:24.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:24 smithi159 ceph-mon[45336]: pgmap v1307: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6359/139314 objects misplaced (4.565%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:24.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:24 smithi159 ceph-mon[45336]: osdmap e501: 12 total, 12 up, 12 in 2024-04-15T18:30:25.520 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:25.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:25.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1308,"stamp":"2024-04-15T18:30:23.455956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3043754552,"num_objects":46438,"num_object_clones":0,"num_object_copies":139314,"num_objects_missing_on_primary":2113,"num_objects_missing":2113,"num_objects_degraded":0,"num_objects_misplaced":6359,"num_objects_unfound":0,"num_objects_dirty":46438,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":140061,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39559,"num_bytes_recovered":2589612704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141340,"ondisk_log_size":141340,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13053680,"kb_used_data":9497220,"kb_used_omap":177,"kb_used_meta":3556174,"kb_avail":1111888144,"statfs":{"total":1151940427776,"available":1138573459456,"internally_reserved":0,"allocated":9725153280,"data_stored":9717510040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3641522628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-136970240,"num_objects":-2090,"num_object_clones":0,"num_object_copies":-6270,"num_objects_missing_on_primary":-131,"num_objects_missing":-131,"num_objects_degraded":0,"num_objects_misplaced":-389,"num_objects_unfound":0,"num_objects_dirty":-2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.015752"},"pg_stats":[{"pgid":"5.f","version":"500'8932","reported_seq":14948,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.207728+0000","last_change":"2024-04-15T18:29:09.178348+0000","last_active":"2024-04-15T18:30:22.207728+0000","last_peered":"2024-04-15T18:30:22.207728+0000","last_clean":"2024-04-15T18:30:22.207728+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:22.207728+0000","last_undegraded":"2024-04-15T18:30:22.207728+0000","last_fullsized":"2024-04-15T18:30:22.207728+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8020","last_scrub_stamp":"2024-04-15T18:29:09.178248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.178248+0000","objects_scrubbed":3803,"log_size":8932,"log_dups_size":0,"ondisk_log_size":8932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:36.553621+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"500'8791","reported_seq":9245,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.252396+0000","last_change":"2024-04-15T18:29:07.248658+0000","last_active":"2024-04-15T18:30:22.252396+0000","last_peered":"2024-04-15T18:30:22.252396+0000","last_clean":"2024-04-15T18:30:22.252396+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:22.252396+0000","last_undegraded":"2024-04-15T18:30:22.252396+0000","last_fullsized":"2024-04-15T18:30:22.252396+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7919","last_scrub_stamp":"2024-04-15T18:29:07.248472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.248472+0000","objects_scrubbed":3774,"log_size":8791,"log_dups_size":0,"ondisk_log_size":8791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:31.587530+0000","scrub_duration":908,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190119936,"num_objects":2901,"num_object_clones":0,"num_object_copies":8703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"500'8733","reported_seq":14538,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:21.737729+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:21.737729+0000","last_peered":"2024-04-15T18:30:21.737729+0000","last_clean":"2024-04-15T18:30:21.737729+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:21.737729+0000","last_undegraded":"2024-04-15T18:30:21.737729+0000","last_fullsized":"2024-04-15T18:30:21.737729+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188219416,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8733,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"500'8756","reported_seq":9263,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.266041+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:22.266041+0000","last_peered":"2024-04-15T18:30:22.266041+0000","last_clean":"2024-04-15T18:30:22.266041+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:22.266041+0000","last_undegraded":"2024-04-15T18:30:22.266041+0000","last_fullsized":"2024-04-15T18:30:22.266041+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192544768,"num_objects":2938,"num_object_clones":0,"num_object_copies":8814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"500'8844","reported_seq":14813,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.194622+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:30:22.194622+0000","last_peered":"2024-04-15T18:30:22.194622+0000","last_clean":"2024-04-15T18:30:22.194622+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:22.194622+0000","last_undegraded":"2024-04-15T18:30:22.194622+0000","last_fullsized":"2024-04-15T18:30:22.194622+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:41.104044+0000","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195035136,"num_objects":2976,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"500'8761","reported_seq":20857,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.282636+0000","last_change":"2024-04-15T18:29:05.624510+0000","last_active":"2024-04-15T18:30:22.282636+0000","last_peered":"2024-04-15T18:30:22.282636+0000","last_clean":"2024-04-15T18:30:22.282636+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:22.282636+0000","last_undegraded":"2024-04-15T18:30:22.282636+0000","last_fullsized":"2024-04-15T18:30:22.282636+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7860","last_scrub_stamp":"2024-04-15T18:29:05.624362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:05.624362+0000","objects_scrubbed":3788,"log_size":8761,"log_dups_size":0,"ondisk_log_size":8761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:22.840984+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"500'8711","reported_seq":9105,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:20.917965+0000","last_change":"2024-04-15T18:30:00.163141+0000","last_active":"2024-04-15T18:30:20.917965+0000","last_peered":"2024-04-15T18:30:20.917965+0000","last_clean":"2024-04-15T18:30:20.917965+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:20.917965+0000","last_undegraded":"2024-04-15T18:30:20.917965+0000","last_fullsized":"2024-04-15T18:30:20.917965+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":549,"log_size":8711,"log_dups_size":0,"ondisk_log_size":8711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:24.996957+0000","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193396736,"num_objects":2951,"num_object_clones":0,"num_object_copies":8853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8711,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"500'8765","reported_seq":9060,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:21.707242+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:21.707242+0000","last_peered":"2024-04-15T18:30:21.707242+0000","last_clean":"2024-04-15T18:30:21.707242+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:21.707242+0000","last_undegraded":"2024-04-15T18:30:21.707242+0000","last_fullsized":"2024-04-15T18:30:21.707242+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8765,"log_dups_size":0,"ondisk_log_size":8765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195493888,"num_objects":2983,"num_object_clones":0,"num_object_copies":8949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"500'8821","reported_seq":14829,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.268942+0000","last_change":"2024-04-15T18:29:06.134163+0000","last_active":"2024-04-15T18:30:22.268942+0000","last_peered":"2024-04-15T18:30:22.268942+0000","last_clean":"2024-04-15T18:30:22.268942+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:22.268942+0000","last_undegraded":"2024-04-15T18:30:22.268942+0000","last_fullsized":"2024-04-15T18:30:22.268942+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7920","last_scrub_stamp":"2024-04-15T18:29:06.134013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:06.134013+0000","objects_scrubbed":3721,"log_size":8821,"log_dups_size":0,"ondisk_log_size":8821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:25.345327+0000","scrub_duration":436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":8445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"500'8622","reported_seq":25871,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.154180+0000","last_change":"2024-04-15T18:29:07.517905+0000","last_active":"2024-04-15T18:30:22.154180+0000","last_peered":"2024-04-15T18:30:22.154180+0000","last_clean":"2024-04-15T18:30:22.154180+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:22.154180+0000","last_undegraded":"2024-04-15T18:30:22.154180+0000","last_fullsized":"2024-04-15T18:30:22.154180+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7782","last_scrub_stamp":"2024-04-15T18:29:07.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:07.517730+0000","objects_scrubbed":3656,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:22.840942+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184418304,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8622,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"500'8480","reported_seq":8933,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.247505+0000","last_change":"2024-04-15T18:29:09.647406+0000","last_active":"2024-04-15T18:30:22.247505+0000","last_peered":"2024-04-15T18:30:22.247505+0000","last_clean":"2024-04-15T18:30:22.247505+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:22.247505+0000","last_undegraded":"2024-04-15T18:30:22.247505+0000","last_fullsized":"2024-04-15T18:30:22.247505+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'7696","last_scrub_stamp":"2024-04-15T18:29:09.647299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:09.647299+0000","objects_scrubbed":3622,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"500'8888","reported_seq":14863,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:20.022090+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:20.022090+0000","last_peered":"2024-04-15T18:30:20.022090+0000","last_clean":"2024-04-15T18:30:20.022090+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:20.022090+0000","last_undegraded":"2024-04-15T18:30:20.022090+0000","last_fullsized":"2024-04-15T18:30:20.022090+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195035136,"num_objects":2976,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"500'8635","reported_seq":8988,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:20.150682+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:20.150682+0000","last_peered":"2024-04-15T18:30:20.150682+0000","last_clean":"2024-04-15T18:30:20.150682+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:20.150682+0000","last_undegraded":"2024-04-15T18:30:20.150682+0000","last_fullsized":"2024-04-15T18:30:20.150682+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193134592,"num_objects":2947,"num_object_clones":0,"num_object_copies":8841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"500'8521","reported_seq":20213,"reported_epoch":500,"state":"active+recovering","last_fresh":"2024-04-15T18:30:21.741104+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:21.741104+0000","last_peered":"2024-04-15T18:30:21.741104+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:21.741104+0000","last_undegraded":"2024-04-15T18:30:21.741104+0000","last_fullsized":"2024-04-15T18:30:21.741104+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8521,"log_dups_size":0,"ondisk_log_size":8521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185663488,"num_objects":2833,"num_object_clones":0,"num_object_copies":8499,"num_objects_missing_on_primary":2113,"num_objects_missing":2113,"num_objects_degraded":0,"num_objects_misplaced":6359,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8521,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2846,"num_bytes_recovered":186384384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":2113},{"shards":"3,4,7,11","objects":10}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"500'8855","reported_seq":9312,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.861393+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:22.861393+0000","last_peered":"2024-04-15T18:30:22.861393+0000","last_clean":"2024-04-15T18:30:22.861393+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:22.861393+0000","last_undegraded":"2024-04-15T18:30:22.861393+0000","last_fullsized":"2024-04-15T18:30:22.861393+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193789952,"num_objects":2957,"num_object_clones":0,"num_object_copies":8871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8855,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"500'12589","reported_seq":30209,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:22.912960+0000","last_change":"2024-04-15T18:28:39.059152+0000","last_active":"2024-04-15T18:30:22.912960+0000","last_peered":"2024-04-15T18:30:22.912960+0000","last_clean":"2024-04-15T18:30:22.912960+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:22.912960+0000","last_undegraded":"2024-04-15T18:30:22.912960+0000","last_fullsized":"2024-04-15T18:30:22.912960+0000","mapping_epoch":440,"log_start":"374'2501","ondisk_log_start":"374'2501","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'11409","last_scrub_stamp":"2024-04-15T18:28:39.058941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:28:39.058941+0000","objects_scrubbed":4046,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187629568,"num_objects":2863,"num_object_clones":0,"num_object_copies":8589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1247,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-04-15T18:30:17.317919+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:30:17.317919+0000","last_peered":"2024-04-15T18:30:17.317919+0000","last_clean":"2024-04-15T18:30:17.317919+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:17.317919+0000","last_undegraded":"2024-04-15T18:30:17.317919+0000","last_fullsized":"2024-04-15T18:30:17.317919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3043164184,"num_objects":46436,"num_object_clones":0,"num_object_copies":139308,"num_objects_missing_on_primary":2113,"num_objects_missing":2113,"num_objects_degraded":0,"num_objects_misplaced":6359,"num_objects_unfound":0,"num_objects_dirty":46436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139880,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39549,"num_bytes_recovered":2586968064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9680728064,"data_stored":9680715848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141203,"ondisk_log_size":141203,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926078,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207916,"kb_used_data":797732,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92537236,"statfs":{"total":95995035648,"available":94758129664,"internally_reserved":0,"allocated":816877568,"data_stored":816234895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187715,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239200,"kb_used_data":937304,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92505952,"statfs":{"total":95995035648,"available":94726094848,"internally_reserved":0,"allocated":959799296,"data_stored":959164267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":309131712},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514760,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885408,"kb_used_data":561304,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92859744,"statfs":{"total":95995035648,"available":95088377856,"internally_reserved":0,"allocated":574775296,"data_stored":574140267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":530976,"kb_used_data":377624,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93214176,"statfs":{"total":95995035648,"available":95451316224,"internally_reserved":0,"allocated":386686976,"data_stored":386051947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168849,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197040,"kb_used_data":877092,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92548112,"statfs":{"total":95995035648,"available":94769266688,"internally_reserved":0,"allocated":898142208,"data_stored":897499555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463191,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951464,"kb_used_data":617632,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92793688,"statfs":{"total":95995035648,"available":95020736512,"internally_reserved":0,"allocated":632455168,"data_stored":631816587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575156,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117408,"kb_used_data":743320,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92627744,"statfs":{"total":95995035648,"available":94850809856,"internally_reserved":0,"allocated":761159680,"data_stored":760524651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730784,"kb_used_data":1366424,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92014368,"statfs":{"total":95995035648,"available":94222712832,"internally_reserved":0,"allocated":1399218176,"data_stored":1398583147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193928,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668832,"kb_used_data":552152,"kb_used_omap":114,"kb_used_meta":116557,"kb_avail":93076320,"statfs":{"total":95995035648,"available":95310151680,"internally_reserved":0,"allocated":565403648,"data_stored":564768619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":119354609},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609058,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653800,"kb_used_data":308248,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091352,"statfs":{"total":95995035648,"available":95325544448,"internally_reserved":0,"allocated":315645952,"data_stored":315010923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986464,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417120,"kb_used_data":1065240,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92328032,"statfs":{"total":95995035648,"available":94543904768,"internally_reserved":0,"allocated":1090805760,"data_stored":1090170731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1453732,"kb_used_data":1293148,"kb_used_omap":4,"kb_used_meta":160571,"kb_avail":92291420,"statfs":{"total":95995035648,"available":94506414080,"internally_reserved":0,"allocated":1324183552,"data_stored":1323544451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":164425409},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1395982336,"data_stored":1395982336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":757202944,"data_stored":757202944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":561971200,"data_stored":561971200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1086390272,"data_stored":1086390272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1322455040,"data_stored":1322450968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":628817920,"data_stored":628817920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":893063168,"data_stored":893059096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":383778816,"data_stored":383778816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":569638912,"data_stored":569638912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":953810944,"data_stored":953810944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":813699072,"data_stored":813695000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:25.548 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.1 is not active+clean 2024-04-15T18:30:25.548 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.1', 'version': "500'8521", 'reported_seq': 20213, 'reported_epoch': 500, 'state': 'active+recovering', 'last_fresh': '2024-04-15T18:30:21.741104+0000', 'last_change': '2024-04-15T18:26:49.574312+0000', 'last_active': '2024-04-15T18:30:21.741104+0000', 'last_peered': '2024-04-15T18:30:21.741104+0000', 'last_clean': '2024-04-15T18:26:00.259169+0000', 'last_became_active': '2024-04-15T18:26:48.920249+0000', 'last_became_peered': '2024-04-15T18:26:48.920249+0000', 'last_unstale': '2024-04-15T18:30:21.741104+0000', 'last_undegraded': '2024-04-15T18:30:21.741104+0000', 'last_fullsized': '2024-04-15T18:30:21.741104+0000', 'mapping_epoch': 416, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'objects_scrubbed': 0, 'log_size': 8521, 'log_dups_size': 0, 'ondisk_log_size': 8521, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 185663488, 'num_objects': 2833, 'num_object_clones': 0, 'num_object_copies': 8499, 'num_objects_missing_on_primary': 2113, 'num_objects_missing': 2113, 'num_objects_degraded': 0, 'num_objects_misplaced': 6359, 'num_objects_unfound': 0, 'num_objects_dirty': 2833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8521, 'num_write_kb': 363328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2846, 'num_bytes_recovered': 186384384, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0, 6], 'acting': [11, 0, 6], 'avail_no_missing': ['3', '4', '7'], 'object_location_counts': [{'shards': '0,6,11', 'objects': 710}, {'shards': '3,4,7', 'objects': 2113}, {'shards': '3,4,7,11', 'objects': 10}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-15T18:30:25.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:30:25.624 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:25.654 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:25.672 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:25.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:25.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:25 smithi049 ceph-mon[40087]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:25.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:25 smithi082 ceph-mon[44206]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:25.802 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:25.821 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:25.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:25.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:25 smithi159 ceph-mon[45336]: from='client.16239 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:25.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:25.954 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:26.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:26 smithi049 ceph-mon[40087]: pgmap v1309: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 6359/139314 objects misplaced (4.565%); 510 KiB/s, 7 objects/s recovering 2024-04-15T18:30:26.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:26 smithi082 ceph-mon[44206]: pgmap v1309: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 6359/139314 objects misplaced (4.565%); 510 KiB/s, 7 objects/s recovering 2024-04-15T18:30:26.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:26 smithi159 ceph-mon[45336]: pgmap v1309: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 6359/139314 objects misplaced (4.565%); 510 KiB/s, 7 objects/s recovering 2024-04-15T18:30:27.475 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:30:27.690 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:27.707 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:27.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:30:27.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:27 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:27.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:30:27.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:27 smithi049 ceph-mon[40087]: osdmap e502: 12 total, 12 up, 12 in 2024-04-15T18:30:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:27.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:27.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:30:27.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:27 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:27.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:30:27.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:27 smithi082 ceph-mon[44206]: osdmap e502: 12 total, 12 up, 12 in 2024-04-15T18:30:27.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:27.838 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:27.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:27.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:30:27.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:27 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:27.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/660221349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:30:27.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:27 smithi159 ceph-mon[45336]: osdmap e502: 12 total, 12 up, 12 in 2024-04-15T18:30:27.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:27.988 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:28.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:28.158 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:28.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:28.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.343 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:28.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:28.475 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.495 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:28.598 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:28.625 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.645 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:28 smithi049 ceph-mon[40087]: 5.7 scrub starts 2024-04-15T18:30:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:28 smithi049 ceph-mon[40087]: pgmap v1311: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 6157/136797 objects misplaced (4.501%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:28.750 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:28.774 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:28 smithi082 ceph-mon[44206]: 5.7 scrub starts 2024-04-15T18:30:28.774 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:28 smithi082 ceph-mon[44206]: pgmap v1311: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 6157/136797 objects misplaced (4.501%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:28.776 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:28.794 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:28.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:28.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:28 smithi159 ceph-mon[45336]: 5.7 scrub starts 2024-04-15T18:30:28.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:28 smithi159 ceph-mon[45336]: pgmap v1311: 17 pgs: 1 active+recovering, 16 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 6157/136797 objects misplaced (4.501%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:28.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:28.934 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:29.011 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:29.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:29.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:29.146 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:29.175 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:29.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:29.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:29.316 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:29.514 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:29.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:29.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:29.645 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:29.665 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:29.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:29.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:29 smithi049 ceph-mon[40087]: 5.7 scrub ok 2024-04-15T18:30:29.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:29 smithi049 ceph-mon[40087]: 5.6 scrub starts 2024-04-15T18:30:29.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:29.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:29 smithi082 ceph-mon[44206]: 5.7 scrub ok 2024-04-15T18:30:29.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:29 smithi082 ceph-mon[44206]: 5.6 scrub starts 2024-04-15T18:30:29.798 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:29.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:29 smithi159 ceph-mon[45336]: 5.7 scrub ok 2024-04-15T18:30:29.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:29 smithi159 ceph-mon[45336]: 5.6 scrub starts 2024-04-15T18:30:29.927 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:29.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:29.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:30.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:30.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:30.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:30.319 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.338 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:30.442 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:30.468 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.486 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:30.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:30.622 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.654 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:30.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:30 smithi049 ceph-mon[40087]: 5.6 scrub ok 2024-04-15T18:30:30.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:30 smithi049 ceph-mon[40087]: pgmap v1312: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 266 op/s; 6157/135825 objects misplaced (4.533%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:30.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:30 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:30.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:30 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:30.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:30 smithi049 ceph-mon[40087]: 5.a scrub starts 2024-04-15T18:30:30.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:30.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:30 smithi082 ceph-mon[44206]: 5.6 scrub ok 2024-04-15T18:30:30.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:30 smithi082 ceph-mon[44206]: pgmap v1312: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 266 op/s; 6157/135825 objects misplaced (4.533%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:30.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:30 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:30.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:30 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:30.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:30 smithi082 ceph-mon[44206]: 5.a scrub starts 2024-04-15T18:30:30.782 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:30 smithi159 ceph-mon[45336]: 5.6 scrub ok 2024-04-15T18:30:30.782 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:30 smithi159 ceph-mon[45336]: pgmap v1312: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 266 op/s; 6157/135825 objects misplaced (4.533%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:30:30.782 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:30 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:30.783 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:30 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:30.783 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:30 smithi159 ceph-mon[45336]: 5.a scrub starts 2024-04-15T18:30:30.784 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.806 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:30.909 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:30.937 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:30.959 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:30.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:31.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:30:31.063 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:31.091 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.111 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:31.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:31.160 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:30:31.160 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:30:31.215 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:31.241 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.259 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:31.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:31.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:31.389 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.410 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:31.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:31.465 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:31.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:31.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:31 smithi082 ceph-mon[44206]: 5.a scrub ok 2024-04-15T18:30:31.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:31 smithi082 ceph-mon[44206]: osdmap e503: 12 total, 12 up, 12 in 2024-04-15T18:30:31.539 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.561 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:31.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:31.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:31.692 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.713 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:31.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:31.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:31 smithi049 ceph-mon[40087]: 5.a scrub ok 2024-04-15T18:30:31.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:31 smithi049 ceph-mon[40087]: osdmap e503: 12 total, 12 up, 12 in 2024-04-15T18:30:31.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:31.849 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:31.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:31 smithi159 ceph-mon[45336]: 5.a scrub ok 2024-04-15T18:30:31.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:31 smithi159 ceph-mon[45336]: osdmap e503: 12 total, 12 up, 12 in 2024-04-15T18:30:32.502 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:32.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:32.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:32.634 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:32 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:32 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:32 smithi082 ceph-mon[44206]: pgmap v1314: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 6157/135168 objects misplaced (4.555%); 551 KiB/s, 8 objects/s recovering 2024-04-15T18:30:32.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:32 smithi082 ceph-mon[44206]: osdmap e504: 12 total, 12 up, 12 in 2024-04-15T18:30:32.785 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:32.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:32 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:32.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:32 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:32.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:32 smithi049 ceph-mon[40087]: pgmap v1314: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 6157/135168 objects misplaced (4.555%); 551 KiB/s, 8 objects/s recovering 2024-04-15T18:30:32.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:32 smithi049 ceph-mon[40087]: osdmap e504: 12 total, 12 up, 12 in 2024-04-15T18:30:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:32.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:32.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:32.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:32 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:32.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:32 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:32.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:32 smithi159 ceph-mon[45336]: pgmap v1314: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 175 op/s; 6157/135168 objects misplaced (4.555%); 551 KiB/s, 8 objects/s recovering 2024-04-15T18:30:32.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:32 smithi159 ceph-mon[45336]: osdmap e504: 12 total, 12 up, 12 in 2024-04-15T18:30:32.916 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:32.936 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:32.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:33.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:33.071 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:33.625 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:33.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:33.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: 5.4 scrub starts 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: 5.4 scrub ok 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3438166470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.764 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.767 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: 5.4 scrub starts 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: 5.4 scrub ok 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3438166470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: 5.4 scrub starts 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: 5.4 scrub ok 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3438166470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:30:34.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:30:34.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:30:34.150 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:30:34.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: pgmap v1316: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 5955/132900 objects misplaced (4.481%); 711 KiB/s, 11 objects/s recovering 2024-04-15T18:30:34.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: pgmap v1316: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 5955/132900 objects misplaced (4.481%); 711 KiB/s, 11 objects/s recovering 2024-04-15T18:30:34.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: pgmap v1316: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s; 5955/132900 objects misplaced (4.481%); 711 KiB/s, 11 objects/s recovering 2024-04-15T18:30:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:35.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:35.128 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:35.154 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:35.173 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:35.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:35.305 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:35.429 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:35.486 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:35 smithi159 ceph-mon[45336]: 5.e scrub starts 2024-04-15T18:30:35.486 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:35 smithi159 ceph-mon[45336]: 5.e scrub ok 2024-04-15T18:30:35.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:35.561 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:35.584 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:35.647 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:35 smithi049 ceph-mon[40087]: 5.e scrub starts 2024-04-15T18:30:35.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:35 smithi049 ceph-mon[40087]: 5.e scrub ok 2024-04-15T18:30:35.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:35.715 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:35.734 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:35.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:35 smithi082 ceph-mon[44206]: 5.e scrub starts 2024-04-15T18:30:35.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:35 smithi082 ceph-mon[44206]: 5.e scrub ok 2024-04-15T18:30:35.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:35.863 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:35.882 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:35.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:35.985 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:36.019 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:36.041 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:36.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:36.145 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:36.172 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:36.199 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:36.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:36.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:36.333 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:36.433 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:36.433 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1583,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":505,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":44300,"data_bytes":2903638584,"bytes_used":13017636864,"bytes_avail":1138922790912,"bytes_total":1151940427776,"misplaced_objects":5955,"misplaced_total":132900,"misplaced_ratio":0.044808126410835217,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":548757,"recovering_keys_per_sec":0,"num_objects_recovered":67,"num_bytes_recovered":4390912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":35,"modified":"2024-04-15T18:27:37.316853+0000","services":{}},"progress_events":{}} 2024-04-15T18:30:36.681 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:36.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:30:36.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:36.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:36.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: pgmap v1317: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 5955/132900 objects misplaced (4.481%); 536 KiB/s, 8 objects/s recovering 2024-04-15T18:30:36.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: osdmap e505: 12 total, 12 up, 12 in 2024-04-15T18:30:36.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:36 smithi049 ceph-mon[40087]: from='client.16254 172.21.15.49:0/3606477154' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:36.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: pgmap v1317: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 5955/132900 objects misplaced (4.481%); 536 KiB/s, 8 objects/s recovering 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: osdmap e505: 12 total, 12 up, 12 in 2024-04-15T18:30:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:36 smithi082 ceph-mon[44206]: from='client.16254 172.21.15.49:0/3606477154' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:36.785 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:36.813 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:30:36.813 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:36.813 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:36.813 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: pgmap v1317: 17 pgs: 1 active+recovering, 16 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 162 op/s; 5955/132900 objects misplaced (4.481%); 536 KiB/s, 8 objects/s recovering 2024-04-15T18:30:36.813 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: osdmap e505: 12 total, 12 up, 12 in 2024-04-15T18:30:36.814 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:36 smithi159 ceph-mon[45336]: from='client.16254 172.21.15.49:0/3606477154' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:30:36.816 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:36.836 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:36.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:36.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:37.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:37.143 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.163 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:37.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:37.293 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:37.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:37.447 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:30:37.468 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:37.545 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:37.575 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:37.588 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:30:37.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:37.607 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.641 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:37.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:37.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:37 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:37.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:37 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:37.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:37 smithi049 ceph-mon[40087]: osdmap e506: 12 total, 12 up, 12 in 2024-04-15T18:30:37.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:37 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:37 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:37 smithi082 ceph-mon[44206]: osdmap e506: 12 total, 12 up, 12 in 2024-04-15T18:30:37.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:37.781 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:37.863 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:37.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:37.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:37 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:37.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:37 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:37.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:37 smithi159 ceph-mon[45336]: osdmap e506: 12 total, 12 up, 12 in 2024-04-15T18:30:37.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:37.970 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:38.001 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:38.024 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:38.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:38.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:38.154 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:38.172 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:38.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:38.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:38.306 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:30:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:38.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:38.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5744/130536 objects misplaced (4.400%); 1.4 MiB/s, 22 objects/s recovering 2024-04-15T18:30:38.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:30:38.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:30:38.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:38.767 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: osdmap e507: 12 total, 12 up, 12 in 2024-04-15T18:30:38.767 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:38 smithi049 ceph-mon[40087]: 5.f scrub starts 2024-04-15T18:30:38.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:38.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5744/130536 objects misplaced (4.400%); 1.4 MiB/s, 22 objects/s recovering 2024-04-15T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: osdmap e507: 12 total, 12 up, 12 in 2024-04-15T18:30:38.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:38 smithi082 ceph-mon[44206]: 5.f scrub starts 2024-04-15T18:30:38.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: pgmap v1320: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 257 op/s; 5744/130536 objects misplaced (4.400%); 1.4 MiB/s, 22 objects/s recovering 2024-04-15T18:30:38.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:30:38.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:30:38.791 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1244048438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:30:38.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: osdmap e507: 12 total, 12 up, 12 in 2024-04-15T18:30:38.792 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:38 smithi159 ceph-mon[45336]: 5.f scrub starts 2024-04-15T18:30:38.793 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:38.812 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:38.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:38.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:38.945 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:38.966 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:38.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:39.070 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:39.097 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:39.115 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:39.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:39.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:39.247 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:39.256 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:39.257 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:39.267 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:39.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:39.371 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:39.398 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:39.418 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:39.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:39.521 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:39.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:39 smithi082 ceph-mon[44206]: 5.5 scrub starts 2024-04-15T18:30:39.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:39 smithi082 ceph-mon[44206]: 5.5 scrub ok 2024-04-15T18:30:39.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:39 smithi082 ceph-mon[44206]: 5.f scrub ok 2024-04-15T18:30:39.548 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:39.567 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:39.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:39.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:39 smithi159 ceph-mon[45336]: 5.5 scrub starts 2024-04-15T18:30:39.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:39 smithi159 ceph-mon[45336]: 5.5 scrub ok 2024-04-15T18:30:39.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:39 smithi159 ceph-mon[45336]: 5.f scrub ok 2024-04-15T18:30:39.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:39.700 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:39 smithi049 ceph-mon[40087]: 5.5 scrub starts 2024-04-15T18:30:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:39 smithi049 ceph-mon[40087]: 5.5 scrub ok 2024-04-15T18:30:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:39 smithi049 ceph-mon[40087]: 5.f scrub ok 2024-04-15T18:30:40.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:40 smithi049 ceph-mon[40087]: pgmap v1322: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 5744/129675 objects misplaced (4.430%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:40 smithi082 ceph-mon[44206]: pgmap v1322: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 5744/129675 objects misplaced (4.430%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:40.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:40.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:40.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:40.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:40 smithi159 ceph-mon[45336]: pgmap v1322: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 5744/129675 objects misplaced (4.430%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:40.924 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:40.942 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:40.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:41.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:41.075 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:41.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:41.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:41.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:30:41.315 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:41.344 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:41.366 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:41.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:41.470 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:41.498 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:41.531 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:41.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:41.665 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:41.764 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:41.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:41.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:41.896 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:41.918 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:41.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:42.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:42.051 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:42.129 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:42.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:42.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:42.262 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:42.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:42 smithi049 ceph-mon[40087]: pgmap v1323: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 5744/128910 objects misplaced (4.456%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:42.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:42 smithi049 ceph-mon[40087]: osdmap e508: 12 total, 12 up, 12 in 2024-04-15T18:30:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:42 smithi082 ceph-mon[44206]: pgmap v1323: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 5744/128910 objects misplaced (4.456%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:42 smithi082 ceph-mon[44206]: osdmap e508: 12 total, 12 up, 12 in 2024-04-15T18:30:42.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:42 smithi159 ceph-mon[45336]: pgmap v1323: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 5744/128910 objects misplaced (4.456%); 757 KiB/s, 11 objects/s recovering 2024-04-15T18:30:42.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:42 smithi159 ceph-mon[45336]: osdmap e508: 12 total, 12 up, 12 in 2024-04-15T18:30:42.911 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:42.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:43.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:43.048 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:43.174 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:43.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93111 5.6 5.6002d106 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57754957600002, 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79303184699999996, 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.181 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93111 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609128+0000", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.81e-06 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609258+0000", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012971 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609273+0000", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4647999999999999e-05 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609284+0000", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1462e-05 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609344+0000", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0343000000000001e-05 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609361+0000", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6680999999999998e-05 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402147+0000", 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79278600300000002 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402158+0000", 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.119e-05 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93119 5.6 5.5151b6f6 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.577367369, 2024-04-15T18:30:43.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79293819799999998, 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93119 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:30:43.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3000000000000001e-07 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609425+0000", 2024-04-15T18:30:43.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011591600000000001 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609436+0000", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0509999999999999e-05 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609443+0000", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9890000000000003e-06 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609482+0000", 2024-04-15T18:30:43.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9857999999999997e-05 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609489+0000", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.455e-06 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402240+0000", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79275137100000004 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402247+0000", 2024-04-15T18:30:43.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5690000000000001e-06 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93121 5.a 5.a03eb09a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57721955199997, 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79295962099999995, 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.190 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93121 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609457+0000", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7e-07 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609524+0000", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6706999999999997e-05 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609533+0000", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6309999999999998e-06 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609542+0000", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7679999999999997e-06 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609584+0000", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1829000000000002e-05 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609588+0000", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1520000000000002e-06 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402404+0000", 2024-04-15T18:30:43.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79281564900000001 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402416+0000", 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2415000000000001e-05 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93122 5.a 5.80d4a5fa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57711864499998, 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79291208199999996, 2024-04-15T18:30:43.194 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93122 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609558+0000", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6699999999999999e-07 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609617+0000", 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.906e-05 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609626+0000", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3100000000000006e-06 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609632+0000", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9889999999999997e-06 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609659+0000", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6908999999999999e-05 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609662+0000", 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2320000000000001e-06 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402457+0000", 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79279496599999999 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402469+0000", 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2149e-05 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93124 5.a 5.1f3061a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57703145900001, 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79288053300000005, 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93124 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609645+0000", 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6199999999999998e-07 2024-04-15T18:30:43.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609704+0000", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8850000000000001e-05 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609714+0000", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020999999999999e-05 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609720+0000", 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8359999999999997e-06 2024-04-15T18:30:43.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609760+0000", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0207000000000003e-05 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609763+0000", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402513+0000", 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79274942500000001 2024-04-15T18:30:43.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402525+0000", 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2283000000000001e-05 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93126 5.a 5.2939364a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.576942492, 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.792824317, 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93126 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609734+0000", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7399999999999998e-07 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609798+0000", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3664999999999994e-05 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609806+0000", 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5010000000000004e-06 2024-04-15T18:30:43.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609810+0000", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3789999999999999e-06 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609863+0000", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2661000000000003e-05 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609872+0000", 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7450000000000001e-06 2024-04-15T18:30:43.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402548+0000", 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79267563200000002 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402558+0000", 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.026e-05 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93129 5.a 5.c296a3da (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57685620199999, 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79277050400000004, 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93129 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.03e-07 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609865+0000", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4506999999999999e-05 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609869+0000", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7280000000000003e-06 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609873+0000", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6770000000000001e-06 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609915+0000", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1841999999999997e-05 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9530000000000005e-06 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402580+0000", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79265583299999998 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402590+0000", 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0661e-05 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93139 5.a 5.5b8b557a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57679471400002, 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79273795199999997, 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.212 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93139 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609882+0000", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8900000000000001e-07 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609916+0000", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4980000000000001e-05 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609921+0000", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5889999999999996e-06 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1930000000000002e-06 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609972+0000", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7296000000000003e-05 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3039999999999994e-06 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402609+0000", 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262850200000001 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402619+0000", 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7990000000000005e-06 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93143 5.a 5.524373aa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:30:43.216 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57673734899998, 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79271180799999996, 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93143 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6100000000000002e-07 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609976+0000", 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6807999999999997e-05 2024-04-15T18:30:43.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4809999999999997e-06 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609985+0000", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0899999999999998e-06 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610009+0000", 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3479e-05 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610012+0000", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6289999999999999e-06 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402641+0000", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262867999999997 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402650+0000", 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.38e-06 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.220 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93146 5.6 5.74a5d016 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 267.57120629500002, 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78680531300000001, 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93146 2024-04-15T18:30:43.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615471+0000", 2024-04-15T18:30:43.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0860000000000001e-06 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615527+0000", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6440000000000002e-05 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615541+0000", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3718e-05 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615555+0000", 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.367e-05 2024-04-15T18:30:43.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615601+0000", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6830999999999999e-05 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615607+0000", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4790000000000004e-06 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402271+0000", 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78666379799999997 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402275+0000", 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2910000000000001e-06 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 419/405 [PushOp(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head, version: 370'1523, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head@370'1523, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 236.171852645, 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48793246000000001, 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.225 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014824+0000", 2024-04-15T18:30:43.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014867+0000", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3164999999999998e-05 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.500977+0000", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48611008500000003 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501005+0000", 2024-04-15T18:30:43.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7334e-05 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501108+0000", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103587 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501119+0000", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1297e-05 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.502756+0000", 2024-04-15T18:30:43.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016362589999999999 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.4 420/405 [PushOp(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head, version: 370'1593, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head@370'1593, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 234.60053431200001, 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.622201475, 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586143+0000", 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.108e-06 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586181+0000", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8618000000000002e-05 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205809+0000", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61962728300000003 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205830+0000", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1491000000000001e-05 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205930+0000", 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9763e-05 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205941+0000", 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0776e-05 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.208343+0000", 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002402436 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 422/405 [PushOp(5:693270da:::benchmark_data_smithi159_116570_object31578:head, version: 372'1976, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:693270da:::benchmark_data_smithi159_116570_object31578:head@372'1976, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 229.342999797, 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54885523899999999, 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843678+0000", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6700000000000001e-06 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843793+0000", 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115654 2024-04-15T18:30:43.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389971+0000", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54617790200000005 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389987+0000", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5892e-05 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390342+0000", 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035477700000000002 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390368+0000", 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6444999999999998e-05 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.392531+0000", 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021628989999999998 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head, version: 402'5230, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head@402'5230, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 191.09336041, 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66078538499999995, 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093367+0000", 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0549e-05 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752462+0000", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.65909545999999997 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752479+0000", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6592e-05 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752556+0000", 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6841000000000005e-05 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752564+0000", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6589999999999996e-06 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.754101+0000", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015365310000000001 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.a 436/405 [PushOp(5:54042545:::benchmark_data_smithi159_116570_object78839:head, version: 397'4917, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:54042545:::benchmark_data_smithi159_116570_object78839:head@397'4917, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:30:43.240 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 190.46213549300001, 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50450683799999996, 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724541+0000", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7700000000000004e-07 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724582+0000", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1350999999999998e-05 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227396+0000", 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.502813128 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227405+0000", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9379999999999998e-06 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227470+0000", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5637999999999997e-05 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227479+0000", 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4700000000000002e-06 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.229047+0000", 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001568636 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:6189473b:::benchmark_data_smithi159_116570_object88044:head, version: 403'5400, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6189473b:::benchmark_data_smithi159_116570_object88044:head@403'5400, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 189.681120225, 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48358990099999999, 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.244 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505557+0000", 2024-04-15T18:30:43.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.55e-06 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505641+0000", 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3617999999999998e-05 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987415+0000", 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48177457400000001 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987426+0000", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0428000000000001e-05 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987508+0000", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2164999999999997e-05 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987517+0000", 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5420000000000005e-06 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.989146+0000", 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001629024 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:62260e07:::benchmark_data_smithi159_116570_object92807:head, version: 403'5695, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:62260e07:::benchmark_data_smithi159_116570_object92807:head@403'5695, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 188.04389385499999, 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49781294999999998, 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142783+0000", 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142843+0000", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0627000000000001e-05 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638705+0000", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49586129000000001 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638724+0000", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9579999999999999e-05 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638840+0000", 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011596599999999999 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638857+0000", 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6864e-05 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.640595+0000", 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00173799 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e447)", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 171.44917861100001, 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55780170699999998, 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737499+0000", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2920000000000001e-06 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737515+0000", 2024-04-15T18:30:43.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737531+0000", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737542+0000", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0835999999999999e-05 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737608+0000", 2024-04-15T18:30:43.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6070999999999998e-05 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737764+0000", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156816 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737886+0000", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121797 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.740403+0000", 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002517039 2024-04-15T18:30:43.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.741178+0000", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077443900000000003 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295224+0000", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55404577600000005 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295259+0000", 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5099999999999999e-05 2024-04-15T18:30:43.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295299+0000", 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0435000000000002e-05 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e447)", 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 171.399767876, 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50835932699999997, 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.257 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786909+0000", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786912+0000", 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7390000000000001e-06 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786921+0000", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2730000000000002e-06 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786928+0000", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1709999999999996e-06 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786991+0000", 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.321e-05 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for ondisk", 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.787066+0000", 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4807000000000004e-05 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295267+0000", 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50820147699999996 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:126776 5.a 5:5a87c32d:::benchmark_data_smithi159_116570_object33616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-04-15T18:30:43.261 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 91.479435967000001, 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.477931355, 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 126776 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707242+0000", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.39e-06 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707255+0000", 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2709e-05 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707287+0000", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2403000000000001e-05 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707306+0000", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8539e-05 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707479+0000", 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017332900000000001 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707663+0000", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018317 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707820+0000", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015760200000000001 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.709796+0000", 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019761599999999998 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.711733+0000", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019369490000000001 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185060+0000", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47332694600000003 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185114+0000", 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4221999999999997e-05 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:30:43.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185171+0000", 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6935999999999998e-05 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:30:43.267 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:30:43.280 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:43.308 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:43.333 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:43.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:43.439 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:43.471 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:43.492 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:43.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:43.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:43.635 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:43.658 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:43.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:43.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:43.793 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:43.815 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:43.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:43.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:43.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:44.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:30:44.095+0000 7fea337fe700 0 --2- 172.21.15.49:0/320731024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea34102b70 0x7fea3410e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:30:44.326 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:44.329 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:44 smithi049 ceph-mon[40087]: pgmap v1325: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 5532/125841 objects misplaced (4.396%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:30:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1484723918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:44 smithi082 ceph-mon[44206]: pgmap v1325: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 5532/125841 objects misplaced (4.396%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:30:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1484723918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:44.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:30:44.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:30:44.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:30:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:30:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:30:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:30:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:30:44.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:30:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:44 smithi159 ceph-mon[45336]: pgmap v1325: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 5532/125841 objects misplaced (4.396%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:30:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1484723918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:45.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:45.317 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:45.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:45.444 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:45.464 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:45.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:45.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:45.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.597 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:45 smithi049 ceph-mon[40087]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:45.599 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:45.624 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:45.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:45.730 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.754 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.755 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:45 smithi082 ceph-mon[44206]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:45.755 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:45.774 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:45.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:45.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:45.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:45 smithi159 ceph-mon[45336]: from='client.16272 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:45.908 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:45.931 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:45.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:46.036 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:46.061 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:46.080 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:46.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:46.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:46.215 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:46.256 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:46.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:46.359 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:46.387 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:46.406 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:46.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:46.510 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:46.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:46.561 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:46.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:46 smithi159 ceph-mon[45336]: pgmap v1326: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 5532/125841 objects misplaced (4.396%); 552 KiB/s, 8 objects/s recovering 2024-04-15T18:30:46.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:46 smithi159 ceph-mon[45336]: osdmap e509: 12 total, 12 up, 12 in 2024-04-15T18:30:46.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:46.698 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:46.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:30:46.716 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:46.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:46 smithi082 ceph-mon[44206]: pgmap v1326: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 5532/125841 objects misplaced (4.396%); 552 KiB/s, 8 objects/s recovering 2024-04-15T18:30:46.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:46 smithi082 ceph-mon[44206]: osdmap e509: 12 total, 12 up, 12 in 2024-04-15T18:30:46.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:46.833 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:46.834 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:46.834 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:46 smithi049 ceph-mon[40087]: pgmap v1326: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 5532/125841 objects misplaced (4.396%); 552 KiB/s, 8 objects/s recovering 2024-04-15T18:30:46.834 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:46 smithi049 ceph-mon[40087]: osdmap e509: 12 total, 12 up, 12 in 2024-04-15T18:30:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:47.015 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:30:47.183 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1325,"stamp":"2024-04-15T18:30:43.351332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2749432376,"num_objects":41947,"num_object_clones":0,"num_object_copies":125841,"num_objects_missing_on_primary":1837,"num_objects_missing":1837,"num_objects_degraded":0,"num_objects_misplaced":5532,"num_objects_unfound":0,"num_objects_dirty":41947,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":144552,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39834,"num_bytes_recovered":2607635104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145531,"ondisk_log_size":145531,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12369016,"kb_used_data":8684876,"kb_used_omap":177,"kb_used_meta":3683854,"kb_avail":1112572808,"statfs":{"total":1151940427776,"available":1139274555392,"internally_reserved":0,"allocated":8893313024,"data_stored":8885284494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3772266948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-142344192,"num_objects":-2172,"num_object_clones":0,"num_object_copies":-6516,"num_objects_missing_on_primary":-142,"num_objects_missing":-142,"num_objects_degraded":0,"num_objects_misplaced":-423,"num_objects_unfound":0,"num_objects_dirty":-2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":9175040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.857187"},"pg_stats":[{"pgid":"5.f","version":"507'9199","reported_seq":15264,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.112476+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:30:42.112476+0000","last_peered":"2024-04-15T18:30:42.112476+0000","last_clean":"2024-04-15T18:30:42.112476+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:42.112476+0000","last_undegraded":"2024-04-15T18:30:42.112476+0000","last_fullsized":"2024-04-15T18:30:42.112476+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":9199,"log_dups_size":0,"ondisk_log_size":9199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171900928,"num_objects":2623,"num_object_clones":0,"num_object_copies":7869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"507'9085","reported_seq":9587,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.298211+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:30:42.298211+0000","last_peered":"2024-04-15T18:30:42.298211+0000","last_clean":"2024-04-15T18:30:42.298211+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:42.298211+0000","last_undegraded":"2024-04-15T18:30:42.298211+0000","last_fullsized":"2024-04-15T18:30:42.298211+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":9085,"log_dups_size":0,"ondisk_log_size":9085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9085,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"507'8991","reported_seq":14810,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:41.665693+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:41.665693+0000","last_peered":"2024-04-15T18:30:41.665693+0000","last_clean":"2024-04-15T18:30:41.665693+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:41.665693+0000","last_undegraded":"2024-04-15T18:30:41.665693+0000","last_fullsized":"2024-04-15T18:30:41.665693+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171311128,"num_objects":2615,"num_object_clones":0,"num_object_copies":7845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8991,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"507'9045","reported_seq":9566,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.068890+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:42.068890+0000","last_peered":"2024-04-15T18:30:42.068890+0000","last_clean":"2024-04-15T18:30:42.068890+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:42.068890+0000","last_undegraded":"2024-04-15T18:30:42.068890+0000","last_fullsized":"2024-04-15T18:30:42.068890+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":7947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"507'9140","reported_seq":15123,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.112444+0000","last_change":"2024-04-15T18:29:12.025040+0000","last_active":"2024-04-15T18:30:42.112444+0000","last_peered":"2024-04-15T18:30:42.112444+0000","last_clean":"2024-04-15T18:30:42.112444+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:42.112444+0000","last_undegraded":"2024-04-15T18:30:42.112444+0000","last_fullsized":"2024-04-15T18:30:42.112444+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'8033","last_scrub_stamp":"2024-04-15T18:29:12.024829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:12.024829+0000","objects_scrubbed":3798,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636480,"num_objects":2680,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"507'9013","reported_seq":21158,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.103840+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:30:42.103840+0000","last_peered":"2024-04-15T18:30:42.103840+0000","last_clean":"2024-04-15T18:30:42.103840+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:42.103840+0000","last_undegraded":"2024-04-15T18:30:42.103840+0000","last_fullsized":"2024-04-15T18:30:42.103840+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"507'9010","reported_seq":9425,"reported_epoch":507,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:30:40.868821+0000","last_change":"2024-04-15T18:30:35.019946+0000","last_active":"2024-04-15T18:30:40.868821+0000","last_peered":"2024-04-15T18:30:40.868821+0000","last_clean":"2024-04-15T18:30:40.868821+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:40.868821+0000","last_undegraded":"2024-04-15T18:30:40.868821+0000","last_fullsized":"2024-04-15T18:30:40.868821+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":149,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 5s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173801472,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9010,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"507'9066","reported_seq":9375,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:41.801319+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:41.801319+0000","last_peered":"2024-04-15T18:30:41.801319+0000","last_clean":"2024-04-15T18:30:41.801319+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:41.801319+0000","last_undegraded":"2024-04-15T18:30:41.801319+0000","last_fullsized":"2024-04-15T18:30:41.801319+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9066,"log_dups_size":0,"ondisk_log_size":9066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175767552,"num_objects":2682,"num_object_clones":0,"num_object_copies":8046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"507'9096","reported_seq":15154,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.200378+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:30:42.200378+0000","last_peered":"2024-04-15T18:30:42.200378+0000","last_clean":"2024-04-15T18:30:42.200378+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:42.200378+0000","last_undegraded":"2024-04-15T18:30:42.200378+0000","last_fullsized":"2024-04-15T18:30:42.200378+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":9096,"log_dups_size":0,"ondisk_log_size":9096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"507'8888","reported_seq":26186,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.107579+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:30:42.107579+0000","last_peered":"2024-04-15T18:30:42.107579+0000","last_clean":"2024-04-15T18:30:42.107579+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:42.107579+0000","last_undegraded":"2024-04-15T18:30:42.107579+0000","last_fullsized":"2024-04-15T18:30:42.107579+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"507'8748","reported_seq":9249,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:42.161342+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:30:42.161342+0000","last_peered":"2024-04-15T18:30:42.161342+0000","last_clean":"2024-04-15T18:30:42.161342+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:42.161342+0000","last_undegraded":"2024-04-15T18:30:42.161342+0000","last_fullsized":"2024-04-15T18:30:42.161342+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":8748,"log_dups_size":0,"ondisk_log_size":8748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168165376,"num_objects":2566,"num_object_clones":0,"num_object_copies":7698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"507'9174","reported_seq":15163,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:40.312753+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:40.312753+0000","last_peered":"2024-04-15T18:30:40.312753+0000","last_clean":"2024-04-15T18:30:40.312753+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:40.312753+0000","last_undegraded":"2024-04-15T18:30:40.312753+0000","last_fullsized":"2024-04-15T18:30:40.312753+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"507'8937","reported_seq":9304,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:40.331920+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:40.331920+0000","last_peered":"2024-04-15T18:30:40.331920+0000","last_clean":"2024-04-15T18:30:40.331920+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:40.331920+0000","last_undegraded":"2024-04-15T18:30:40.331920+0000","last_fullsized":"2024-04-15T18:30:40.331920+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173342720,"num_objects":2645,"num_object_clones":0,"num_object_copies":7935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"507'8796","reported_seq":21604,"reported_epoch":507,"state":"active+recovering","last_fresh":"2024-04-15T18:30:41.880400+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:41.880400+0000","last_peered":"2024-04-15T18:30:41.880400+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:41.880400+0000","last_undegraded":"2024-04-15T18:30:41.880400+0000","last_fullsized":"2024-04-15T18:30:41.880400+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8796,"log_dups_size":0,"ondisk_log_size":8796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167641088,"num_objects":2558,"num_object_clones":0,"num_object_copies":7674,"num_objects_missing_on_primary":1837,"num_objects_missing":1837,"num_objects_degraded":0,"num_objects_misplaced":5532,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3121,"num_bytes_recovered":204406784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,3,4,7,11","objects":1},{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":1837},{"shards":"3,4,7,11","objects":10}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"508'9145","reported_seq":9619,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-15T18:30:43.025019+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:43.025019+0000","last_peered":"2024-04-15T18:30:43.025019+0000","last_clean":"2024-04-15T18:30:43.025019+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:43.025019+0000","last_undegraded":"2024-04-15T18:30:43.025019+0000","last_fullsized":"2024-04-15T18:30:43.025019+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174784512,"num_objects":2667,"num_object_clones":0,"num_object_copies":8001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"508'12862","reported_seq":30534,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-15T18:30:43.105653+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:30:43.105653+0000","last_peered":"2024-04-15T18:30:43.105653+0000","last_clean":"2024-04-15T18:30:43.105653+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:43.105653+0000","last_undegraded":"2024-04-15T18:30:43.105653+0000","last_fullsized":"2024-04-15T18:30:43.105653+0000","mapping_epoch":440,"log_start":"379'2801","ondisk_log_start":"379'2801","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9038,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":1261,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-15T18:30:38.342244+0000","last_change":"2024-04-15T18:29:15.185340+0000","last_active":"2024-04-15T18:30:38.342244+0000","last_peered":"2024-04-15T18:30:38.342244+0000","last_clean":"2024-04-15T18:30:38.342244+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:38.342244+0000","last_undegraded":"2024-04-15T18:30:38.342244+0000","last_fullsized":"2024-04-15T18:30:38.342244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'137","last_scrub_stamp":"2024-04-15T18:29:15.185244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:05:43.609679+0000","last_clean_scrub_stamp":"2024-04-15T18:29:15.185244+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:45.112330+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2748842008,"num_objects":41945,"num_object_clones":0,"num_object_copies":125835,"num_objects_missing_on_primary":1837,"num_objects_missing":1837,"num_objects_degraded":0,"num_objects_misplaced":5532,"num_objects_unfound":0,"num_objects_dirty":41945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144371,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39824,"num_bytes_recovered":2604990464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8853139456,"data_stored":8853127240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145394,"ondisk_log_size":145394,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926082,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139840,"kb_used_data":725496,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92605312,"statfs":{"total":95995035648,"available":94827839488,"internally_reserved":0,"allocated":742907904,"data_stored":742233876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187719,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151748,"kb_used_data":845688,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92593404,"statfs":{"total":95995035648,"available":94815645696,"internally_reserved":0,"allocated":865984512,"data_stored":865313613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514764,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850740,"kb_used_data":504428,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92894412,"statfs":{"total":95995035648,"available":95123877888,"internally_reserved":0,"allocated":516534272,"data_stored":515867888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500276,"kb_used_data":342764,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93244876,"statfs":{"total":95995035648,"available":95482753024,"internally_reserved":0,"allocated":350990336,"data_stored":350323952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168068,"kb_used_data":825912,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92577084,"statfs":{"total":95995035648,"available":94798934016,"internally_reserved":0,"allocated":845733888,"data_stored":845059880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463195,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898940,"kb_used_data":560948,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92846212,"statfs":{"total":95995035648,"available":95074521088,"internally_reserved":0,"allocated":574410752,"data_stored":573740816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575160,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1048884,"kb_used_data":670636,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92696268,"statfs":{"total":95995035648,"available":94920978432,"internally_reserved":0,"allocated":686731264,"data_stored":686064880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629620,"kb_used_data":1237484,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92115532,"statfs":{"total":95995035648,"available":94326304768,"internally_reserved":0,"allocated":1267183616,"data_stored":1266517232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622516,"kb_used_data":501676,"kb_used_omap":114,"kb_used_meta":120717,"kb_avail":93122636,"statfs":{"total":95995035648,"available":95357579264,"internally_reserved":0,"allocated":513716224,"data_stored":513049840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":123614449},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609062,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653892,"kb_used_data":308344,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091260,"statfs":{"total":95995035648,"available":95325450240,"internally_reserved":0,"allocated":315744256,"data_stored":315073357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986468,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366324,"kb_used_data":992236,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92378828,"statfs":{"total":95995035648,"available":94595919872,"internally_reserved":0,"allocated":1016049664,"data_stored":1015383280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703363,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338168,"kb_used_data":1169264,"kb_used_omap":4,"kb_used_meta":168891,"kb_avail":92406984,"statfs":{"total":95995035648,"available":94624751616,"internally_reserved":0,"allocated":1197326336,"data_stored":1196655880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":172945089},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1262878720,"data_stored":1262878720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":683212800,"data_stored":683212800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":509607936,"data_stored":509607936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1013841920,"data_stored":1013841920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1195511808,"data_stored":1195507736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":570687488,"data_stored":570687488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":840437760,"data_stored":840433688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":347602944,"data_stored":347602944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":512950272,"data_stored":512950272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":863633408,"data_stored":863633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":738856960,"data_stored":738852888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:47.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:47 smithi082 ceph-mon[44206]: 1.0 deep-scrub starts 2024-04-15T18:30:47.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:47 smithi082 ceph-mon[44206]: 1.0 deep-scrub ok 2024-04-15T18:30:47.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:47 smithi082 ceph-mon[44206]: osdmap e510: 12 total, 12 up, 12 in 2024-04-15T18:30:47.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:47 smithi049 ceph-mon[40087]: 1.0 deep-scrub starts 2024-04-15T18:30:47.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:47 smithi049 ceph-mon[40087]: 1.0 deep-scrub ok 2024-04-15T18:30:47.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:47 smithi049 ceph-mon[40087]: osdmap e510: 12 total, 12 up, 12 in 2024-04-15T18:30:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:47 smithi159 ceph-mon[45336]: 1.0 deep-scrub starts 2024-04-15T18:30:47.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:47 smithi159 ceph-mon[45336]: 1.0 deep-scrub ok 2024-04-15T18:30:47.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:47 smithi159 ceph-mon[45336]: osdmap e510: 12 total, 12 up, 12 in 2024-04-15T18:30:47.904 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:47.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:48.008 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:48.036 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:48.061 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:48.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:48.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:48.195 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:48 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:48 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:48 smithi082 ceph-mon[44206]: pgmap v1329: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 5302/123099 objects misplaced (4.307%); 1.5 MiB/s, 24 objects/s recovering 2024-04-15T18:30:48.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:48 smithi082 ceph-mon[44206]: osdmap e511: 12 total, 12 up, 12 in 2024-04-15T18:30:48.745 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:48.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:48.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:48.881 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:48.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:48 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:30:48.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:48 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:30:48.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:48 smithi159 ceph-mon[45336]: pgmap v1329: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 5302/123099 objects misplaced (4.307%); 1.5 MiB/s, 24 objects/s recovering 2024-04-15T18:30:48.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:48 smithi159 ceph-mon[45336]: osdmap e511: 12 total, 12 up, 12 in 2024-04-15T18:30:48.938 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:48.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:48.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:48 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:30:48.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:48 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:30:48.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:48 smithi049 ceph-mon[40087]: pgmap v1329: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 5302/123099 objects misplaced (4.307%); 1.5 MiB/s, 24 objects/s recovering 2024-04-15T18:30:48.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:48 smithi049 ceph-mon[40087]: osdmap e511: 12 total, 12 up, 12 in 2024-04-15T18:30:49.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:49.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:49.093 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:49.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:49.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:30:49.199 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:30:49.199 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:30:49.231 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:49.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:30:49.514 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:30:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:49 smithi082 ceph-mon[44206]: 5.b scrub starts 2024-04-15T18:30:49.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:30:49.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:30:49.887 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:30:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:49.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:30:49.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:49 smithi159 ceph-mon[45336]: 5.b scrub starts 2024-04-15T18:30:49.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:30:49.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:49 smithi049 ceph-mon[40087]: 5.b scrub starts 2024-04-15T18:30:49.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:30:49.990 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:50.020 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.045 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:50.150 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:50.180 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:50.192 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:30:50.202 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:50.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:50.314 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:50.334 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.354 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:50.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:50.485 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.504 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:50.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:50.635 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:50 smithi159 ceph-mon[45336]: 5.b scrub ok 2024-04-15T18:30:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:50 smithi159 ceph-mon[45336]: pgmap v1331: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 5302/122235 objects misplaced (4.338%); 832 KiB/s, 12 objects/s recovering 2024-04-15T18:30:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:50 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:50 smithi159 ceph-mon[45336]: osdmap e512: 12 total, 12 up, 12 in 2024-04-15T18:30:50.657 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:50.761 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:50 smithi082 ceph-mon[44206]: 5.b scrub ok 2024-04-15T18:30:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:50 smithi082 ceph-mon[44206]: pgmap v1331: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 5302/122235 objects misplaced (4.338%); 832 KiB/s, 12 objects/s recovering 2024-04-15T18:30:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:50 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:50 smithi082 ceph-mon[44206]: osdmap e512: 12 total, 12 up, 12 in 2024-04-15T18:30:50.787 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:50.808 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:50.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:50.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:50 smithi049 ceph-mon[40087]: 5.b scrub ok 2024-04-15T18:30:50.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:50 smithi049 ceph-mon[40087]: pgmap v1331: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 200 op/s; 5302/122235 objects misplaced (4.338%); 832 KiB/s, 12 objects/s recovering 2024-04-15T18:30:50.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:50 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:30:50.820 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2407464944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:30:50.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:50 smithi049 ceph-mon[40087]: osdmap e512: 12 total, 12 up, 12 in 2024-04-15T18:30:50.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:50.942 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:51.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:51.148 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:51.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:30:51.271 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:51.291 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:30:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:51.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:30:51.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:30:51.428 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:51.915 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:30:51.915 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:30:52.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:30:52.649 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:52 smithi082 ceph-mon[44206]: pgmap v1333: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 5302/121506 objects misplaced (4.364%); 855 KiB/s, 13 objects/s recovering 2024-04-15T18:30:52.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:52 smithi049 ceph-mon[40087]: pgmap v1333: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 5302/121506 objects misplaced (4.364%); 855 KiB/s, 13 objects/s recovering 2024-04-15T18:30:52.678 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:30:52.684 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:30:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:52.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:30:52.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:52.821 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:52.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:52 smithi159 ceph-mon[45336]: pgmap v1333: 17 pgs: 1 active+recovering, 16 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 5302/121506 objects misplaced (4.364%); 855 KiB/s, 13 objects/s recovering 2024-04-15T18:30:52.924 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:52.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:52.983 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:30:53.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:30:53.062 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:53.587 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:30:53.587 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:30:53.587 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:30:53.591 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:30:53.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:30:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:53.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:30:53.596 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:53.619 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:30:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:53.627 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:53.644 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:30:53.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:30:53.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:53.698 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:30:53.732 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:53.732 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:30:53.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:30:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/704653030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:53.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:30:53.812 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:30:53.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:30:53.816 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:30:53.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:30:53.836 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/704653030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:53.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/704653030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:30:54.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: pgmap v1334: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 5069/118104 objects misplaced (4.292%); 807 KiB/s, 12 objects/s recovering 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:54.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:54.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: pgmap v1334: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 5069/118104 objects misplaced (4.292%); 807 KiB/s, 12 objects/s recovering 2024-04-15T18:30:54.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.864 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.891 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: pgmap v1334: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 302 op/s; 5069/118104 objects misplaced (4.292%); 807 KiB/s, 12 objects/s recovering 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: from='client.16284 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:54.919 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:54.950 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:54.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:55.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:55.085 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.187 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:55.226 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1334,"stamp":"2024-04-15T18:30:53.352864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2580415032,"num_objects":39368,"num_object_clones":0,"num_object_copies":118104,"num_objects_missing_on_primary":1682,"num_objects_missing":1682,"num_objects_degraded":0,"num_objects_misplaced":5069,"num_objects_unfound":0,"num_objects_dirty":39368,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":147131,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39988,"num_bytes_recovered":2617727648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147912,"ondisk_log_size":147912,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11950164,"kb_used_data":8228484,"kb_used_omap":177,"kb_used_meta":3721294,"kb_avail":1112991660,"statfs":{"total":1151940427776,"available":1139703459840,"internally_reserved":0,"allocated":8425967616,"data_stored":8417678416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3810605508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-119734272,"num_objects":-1827,"num_object_clones":0,"num_object_copies":-5481,"num_objects_missing_on_primary":-81,"num_objects_missing":-81,"num_objects_degraded":0,"num_objects_misplaced":-233,"num_objects_unfound":0,"num_objects_dirty":-1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1827,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.029929"},"pg_stats":[{"pgid":"5.f","version":"512'9358","reported_seq":15433,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.215492+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:30:52.215492+0000","last_peered":"2024-04-15T18:30:52.215492+0000","last_clean":"2024-04-15T18:30:52.215492+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:52.215492+0000","last_undegraded":"2024-04-15T18:30:52.215492+0000","last_fullsized":"2024-04-15T18:30:52.215492+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9358,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"512'9253","reported_seq":9766,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:51.710431+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:30:51.710431+0000","last_peered":"2024-04-15T18:30:51.710431+0000","last_clean":"2024-04-15T18:30:51.710431+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:51.710431+0000","last_undegraded":"2024-04-15T18:30:51.710431+0000","last_fullsized":"2024-04-15T18:30:51.710431+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"512'9140","reported_seq":14969,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:51.878569+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:51.878569+0000","last_peered":"2024-04-15T18:30:51.878569+0000","last_clean":"2024-04-15T18:30:51.878569+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:51.878569+0000","last_undegraded":"2024-04-15T18:30:51.878569+0000","last_fullsized":"2024-04-15T18:30:51.878569+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161546264,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9140,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"512'9227","reported_seq":9758,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.235213+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:52.235213+0000","last_peered":"2024-04-15T18:30:52.235213+0000","last_clean":"2024-04-15T18:30:52.235213+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:52.235213+0000","last_undegraded":"2024-04-15T18:30:52.235213+0000","last_fullsized":"2024-04-15T18:30:52.235213+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161677312,"num_objects":2467,"num_object_clones":0,"num_object_copies":7401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9227,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"512'9323","reported_seq":15350,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.225190+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:30:52.225190+0000","last_peered":"2024-04-15T18:30:52.225190+0000","last_clean":"2024-04-15T18:30:52.225190+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:52.225190+0000","last_undegraded":"2024-04-15T18:30:52.225190+0000","last_fullsized":"2024-04-15T18:30:52.225190+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":9323,"log_dups_size":0,"ondisk_log_size":9323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163643392,"num_objects":2497,"num_object_clones":0,"num_object_copies":7491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9323,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"512'9162","reported_seq":21317,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:51.868308+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:30:51.868308+0000","last_peered":"2024-04-15T18:30:51.868308+0000","last_clean":"2024-04-15T18:30:51.868308+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:51.868308+0000","last_undegraded":"2024-04-15T18:30:51.868308+0000","last_fullsized":"2024-04-15T18:30:51.868308+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162791424,"num_objects":2484,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9162,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"512'9176","reported_seq":9604,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:50.862658+0000","last_change":"2024-04-15T18:30:49.739135+0000","last_active":"2024-04-15T18:30:50.862658+0000","last_peered":"2024-04-15T18:30:50.862658+0000","last_clean":"2024-04-15T18:30:50.862658+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:50.862658+0000","last_undegraded":"2024-04-15T18:30:50.862658+0000","last_fullsized":"2024-04-15T18:30:50.862658+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":249,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:30:54.731457+0000","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9176,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"512'9217","reported_seq":9536,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:51.925090+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:51.925090+0000","last_peered":"2024-04-15T18:30:51.925090+0000","last_clean":"2024-04-15T18:30:51.925090+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:51.925090+0000","last_undegraded":"2024-04-15T18:30:51.925090+0000","last_fullsized":"2024-04-15T18:30:51.925090+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9217,"log_dups_size":0,"ondisk_log_size":9217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165871616,"num_objects":2531,"num_object_clones":0,"num_object_copies":7593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"512'9262","reported_seq":15329,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.268795+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:30:52.268795+0000","last_peered":"2024-04-15T18:30:52.268795+0000","last_clean":"2024-04-15T18:30:52.268795+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:52.268795+0000","last_undegraded":"2024-04-15T18:30:52.268795+0000","last_fullsized":"2024-04-15T18:30:52.268795+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"512'9044","reported_seq":26352,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.255899+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:30:52.255899+0000","last_peered":"2024-04-15T18:30:52.255899+0000","last_clean":"2024-04-15T18:30:52.255899+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:52.255899+0000","last_undegraded":"2024-04-15T18:30:52.255899+0000","last_fullsized":"2024-04-15T18:30:52.255899+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156762112,"num_objects":2392,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"512'8909","reported_seq":9419,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.199711+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:30:52.199711+0000","last_peered":"2024-04-15T18:30:52.199711+0000","last_clean":"2024-04-15T18:30:52.199711+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:52.199711+0000","last_undegraded":"2024-04-15T18:30:52.199711+0000","last_fullsized":"2024-04-15T18:30:52.199711+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157614080,"num_objects":2405,"num_object_clones":0,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8909,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"512'9356","reported_seq":15356,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:50.334045+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:50.334045+0000","last_peered":"2024-04-15T18:30:50.334045+0000","last_clean":"2024-04-15T18:30:50.334045+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:50.334045+0000","last_undegraded":"2024-04-15T18:30:50.334045+0000","last_fullsized":"2024-04-15T18:30:50.334045+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"512'9076","reported_seq":9454,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:50.246336+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:50.246336+0000","last_peered":"2024-04-15T18:30:50.246336+0000","last_clean":"2024-04-15T18:30:50.246336+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:50.246336+0000","last_undegraded":"2024-04-15T18:30:50.246336+0000","last_fullsized":"2024-04-15T18:30:50.246336+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164233216,"num_objects":2506,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9076,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"512'8950","reported_seq":22386,"reported_epoch":512,"state":"active+recovering","last_fresh":"2024-04-15T18:30:51.963180+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:51.963180+0000","last_peered":"2024-04-15T18:30:51.963180+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:51.963180+0000","last_undegraded":"2024-04-15T18:30:51.963180+0000","last_fullsized":"2024-04-15T18:30:51.963180+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":8950,"log_dups_size":0,"ondisk_log_size":8950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157548544,"num_objects":2404,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":1682,"num_objects_missing":1682,"num_objects_degraded":0,"num_objects_misplaced":5069,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":214499328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,3,4,7,11","objects":1},{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":1682},{"shards":"3,4,7,11","objects":11}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"512'9308","reported_seq":9790,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:53.018294+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:53.018294+0000","last_peered":"2024-04-15T18:30:53.018294+0000","last_clean":"2024-04-15T18:30:53.018294+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:53.018294+0000","last_undegraded":"2024-04-15T18:30:53.018294+0000","last_fullsized":"2024-04-15T18:30:53.018294+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":9308,"log_dups_size":0,"ondisk_log_size":9308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"512'13013","reported_seq":30693,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:53.033176+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:30:53.033176+0000","last_peered":"2024-04-15T18:30:53.033176+0000","last_clean":"2024-04-15T18:30:53.033176+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:53.033176+0000","last_undegraded":"2024-04-15T18:30:53.033176+0000","last_fullsized":"2024-04-15T18:30:53.033176+0000","mapping_epoch":440,"log_start":"380'3001","ondisk_log_start":"380'3001","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1281,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:49.519839+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:30:49.519839+0000","last_peered":"2024-04-15T18:30:49.519839+0000","last_clean":"2024-04-15T18:30:49.519839+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:49.519839+0000","last_undegraded":"2024-04-15T18:30:49.519839+0000","last_fullsized":"2024-04-15T18:30:49.519839+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2579824664,"num_objects":39366,"num_object_clones":0,"num_object_copies":118098,"num_objects_missing_on_primary":1682,"num_objects_missing":1682,"num_objects_degraded":0,"num_objects_misplaced":5069,"num_objects_unfound":0,"num_objects_dirty":39366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146950,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39978,"num_bytes_recovered":2615083008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8376365056,"data_stored":8376352840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147773,"ondisk_log_size":147773,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926084,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104000,"kb_used_data":685492,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92641152,"statfs":{"total":95995035648,"available":94864539648,"internally_reserved":0,"allocated":701943808,"data_stored":701247305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187721,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105776,"kb_used_data":795560,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92639376,"statfs":{"total":95995035648,"available":94862721024,"internally_reserved":0,"allocated":814653440,"data_stored":813964581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514766,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817328,"kb_used_data":471016,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92927824,"statfs":{"total":95995035648,"available":95158091776,"internally_reserved":0,"allocated":482320384,"data_stored":481631525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479672,"kb_used_data":322152,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93265480,"statfs":{"total":95995035648,"available":95503851520,"internally_reserved":0,"allocated":329883648,"data_stored":329194789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1141768,"kb_used_data":795444,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92603384,"statfs":{"total":95995035648,"available":94825865216,"internally_reserved":0,"allocated":814534656,"data_stored":813838173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463197,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872320,"kb_used_data":530160,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92872832,"statfs":{"total":95995035648,"available":95101779968,"internally_reserved":0,"allocated":542883840,"data_stored":542191429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575162,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006516,"kb_used_data":628200,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92738636,"statfs":{"total":95995035648,"available":94964363264,"internally_reserved":0,"allocated":643276800,"data_stored":642587941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564468,"kb_used_data":1164008,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92180684,"statfs":{"total":95995035648,"available":94393020416,"internally_reserved":0,"allocated":1191944192,"data_stored":1191255333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193934,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597424,"kb_used_data":472424,"kb_used_omap":114,"kb_used_meta":124877,"kb_avail":93147728,"statfs":{"total":95995035648,"available":95383273472,"internally_reserved":0,"allocated":483762176,"data_stored":483073317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":127874289},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653944,"kb_used_data":308392,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091208,"statfs":{"total":95995035648,"available":95325396992,"internally_reserved":0,"allocated":315793408,"data_stored":315104549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986470,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328688,"kb_used_data":950440,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92416464,"statfs":{"total":95995035648,"available":94634459136,"internally_reserved":0,"allocated":973250560,"data_stored":972561701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703365,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278260,"kb_used_data":1105196,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":92466892,"statfs":{"total":95995035648,"available":94686097408,"internally_reserved":0,"allocated":1131720704,"data_stored":1131027773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1187053568,"data_stored":1187053568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":640024576,"data_stored":640024576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":480509952,"data_stored":480509952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":969998336,"data_stored":969998336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1121456128,"data_stored":1121452056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":538640384,"data_stored":538640384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":810029056,"data_stored":810024984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":326107136,"data_stored":326107136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":479068160,"data_stored":479068160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":811794432,"data_stored":811794432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":697765888,"data_stored":697761816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:55.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:30:55.292 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:55.319 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.350 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:55.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:30:55.485 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.512 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:30:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:30:55.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:55.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.647 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.671 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:55.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:55.804 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.827 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:55.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:30:55.935 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:55.970 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:55.991 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:55.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:56.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:30:56.123 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:56.177 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:30:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:56.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:30:56.203 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:75530 5.d e393/392 5:b48816ed:::benchmark_data_smithi159_116570_object75529:head v 393'4673, mlcod=393'4671)", 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:51.560517+0000", 2024-04-15T18:30:56.270 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 304.71549253699999, 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53894305799999997, 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.560517+0000", 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.560517+0000", 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.560518+0000", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5199999999999995e-07 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.560592+0000", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4355000000000002e-05 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.097542+0000", 2024-04-15T18:30:56.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53695032399999998 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.097555+0000", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2680000000000001e-05 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.097695+0000", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014027399999999999 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.097736+0000", 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1258999999999999e-05 2024-04-15T18:30:56.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.099417+0000", 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016805940000000001 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.099460+0000", 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2719999999999998e-05 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.274 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(5.b 393/377 [PushOp(5:d4563bd6:::benchmark_data_smithi159_116570_object1342:head, version: 367'91, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:d4563bd6:::benchmark_data_smithi159_116570_object1342:head@367'91, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:51.768973+0000", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 304.507036476, 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.497365958, 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.768973+0000", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.768973+0000", 2024-04-15T18:30:56.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.768973+0000", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3299999999999998e-07 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.769037+0000", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4028000000000004e-05 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.264405+0000", 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49536828500000002 2024-04-15T18:30:56.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.264421+0000", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5418999999999999e-05 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.264550+0000", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000128935 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.264563+0000", 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3104e-05 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:52.266339+0000", 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017758540000000001 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:93153 5.d e404/392 5:bc735d28:::benchmark_data_smithi159_116570_object93152:head v 404'5801, mlcod=404'5799)", 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:16.282524+0000", 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 279.99348560999999, 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46176208400000002, 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.282524+0000", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.282524+0000", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.282525+0000", 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6139999999999999e-06 2024-04-15T18:30:56.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.282625+0000", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100182 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.742106+0000", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45948030600000001 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.742118+0000", 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.252e-05 2024-04-15T18:30:56.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.742276+0000", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000157894 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.742342+0000", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6285000000000007e-05 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.744248+0000", 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019056489999999999 2024-04-15T18:30:56.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.744286+0000", 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7633999999999999e-05 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(5.b 405/377 [PushOp(5:d46d2fd5:::benchmark_data_smithi159_116570_object30401:head, version: 372'1908, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:d46d2fd5:::benchmark_data_smithi159_116570_object30401:head@372'1908, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:16.755607+0000", 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 279.52040246399997, 2024-04-15T18:30:56.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43737457099999999, 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.755607+0000", 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.755607+0000", 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.755607+0000", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2900000000000003e-07 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.755686+0000", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8473000000000006e-05 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.190907+0000", 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43522075799999999 2024-04-15T18:30:56.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.190918+0000", 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1606e-05 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.191051+0000", 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013279899999999999 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.285 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.191066+0000", 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5270000000000001e-05 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.192981+0000", 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001915036 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93111 5.6 5:608b4006:::benchmark_data_smithi159_116570_object93110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:30:56.286 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.433442+0000", 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.84256735899999, 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44469789900000001, 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93111 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433442+0000", 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433442+0000", 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433443+0000", 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.344e-06 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433577+0000", 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133363 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433591+0000", 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4671000000000001e-05 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433599+0000", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8900000000000007e-06 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435068+0000", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001468507 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435075+0000", 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3200000000000002e-06 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.869090+0000", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43401500599999998 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.869207+0000", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116606 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.869319+0000", 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112508 2024-04-15T18:30:56.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.875933+0000", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0066140879999999997 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.875997+0000", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4157000000000003e-05 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.876016+0000", 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8457e-05 2024-04-15T18:30:56.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.876074+0000", 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8356999999999997e-05 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878140+0000", 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020656250000000002 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.293 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93119 5.6 5:6f6d8a8a:::benchmark_data_smithi159_116570_object93118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.433613+0000", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.84239577699998, 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45854285500000003, 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93119 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433613+0000", 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433613+0000", 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433614+0000", 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0499999999999999e-07 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433641+0000", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6888000000000001e-05 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433647+0000", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.365e-06 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433651+0000", 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4150000000000001e-06 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435102+0000", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014503610000000001 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435109+0000", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3780000000000003e-06 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.873251+0000", 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43814201699999999 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.873361+0000", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000109584 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.873480+0000", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118748 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878249+0000", 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0047695749999999999 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878260+0000", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0366999999999999e-05 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.890630+0000", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012370908999999999 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.890678+0000", 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7574999999999998e-05 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.892156+0000", 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001478268 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93146 5.6 5:680ba52e:::benchmark_data_smithi159_116570_object93145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.433660+0000", 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.842349033, 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45982710300000001, 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.300 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93146 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433660+0000", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433660+0000", 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433661+0000", 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4299999999999998e-07 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433700+0000", 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9400000000000002e-05 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433711+0000", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0781000000000001e-05 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433717+0000", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1890000000000004e-06 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435144+0000", 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014265809999999999 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435149+0000", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.215e-06 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878286+0000", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44313679099999997 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878381+0000", 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5267999999999997e-05 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.878497+0000", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116255 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.882616+0000", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004118694 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.883818+0000", 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001202123 2024-04-15T18:30:56.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.892215+0000", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0083971789999999994 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.892283+0000", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7829000000000004e-05 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.893487+0000", 2024-04-15T18:30:56.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012043550000000001 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93150 5.6 5:6417493a:::benchmark_data_smithi159_116570_object93149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.433725+0000", 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.84228381899999, 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46186581999999998, 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.307 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93150 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433725+0000", 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433725+0000", 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433726+0000", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5600000000000001e-07 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433761+0000", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5306999999999999e-05 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433770+0000", 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6449999999999994e-06 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433777+0000", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7220000000000002e-06 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435165+0000", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013881799999999999 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.435169+0000", 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3340000000000003e-06 2024-04-15T18:30:56.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.880293+0000", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.445124099 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.880389+0000", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6074000000000002e-05 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.880505+0000", 2024-04-15T18:30:56.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116066 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.884651+0000", 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041459510000000002 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.885501+0000", 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084977700000000002 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.894510+0000", 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0090086119999999992 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.894549+0000", 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9373999999999999e-05 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.895591+0000", 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010422230000000001 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.6 419/405 [PushReplyOp(5:6192f1aa:::benchmark_data_smithi159_116570_object21191:head)])", 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:44.801216+0000", 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 251.47479336000001, 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52531230399999995, 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:44.801216+0000", 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:44.801216+0000", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:44.801217+0000", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.207e-06 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:44.801224+0000", 2024-04-15T18:30:56.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3030000000000002e-06 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.323866+0000", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52264211100000002 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.323875+0000", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4130000000000006e-06 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.323968+0000", 2024-04-15T18:30:56.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3541000000000004e-05 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.326528+0000", 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025597290000000002 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:108048 5.7 5:e3958808:::benchmark_data_smithi159_116570_object14888:head [delete] snapc 0=[] ondisk+write+known_if_redirected e422)", 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:52.637861+0000", 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 243.63814793099999, 2024-04-15T18:30:56.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50971672700000004, 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 108048 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637861+0000", 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637861+0000", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637862+0000", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7400000000000002e-07 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637866+0000", 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1579999999999998e-06 2024-04-15T18:30:56.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637881+0000", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4637999999999999e-05 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637889+0000", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4600000000000003e-06 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.637940+0000", 2024-04-15T18:30:56.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0327000000000001e-05 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.638063+0000", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012359 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.638174+0000", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110297 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.641111+0000", 2024-04-15T18:30:56.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029379190000000002 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.641489+0000", 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00037795199999999998 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.147490+0000", 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50600028699999999 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.147534+0000", 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4038000000000001e-05 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.147578+0000", 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4286999999999998e-05 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.6 431/405 [PushReplyOp(5:68eb31bf:::benchmark_data_smithi159_116570_object57511:head)])", 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:16.317161+0000", 2024-04-15T18:30:56.323 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 219.95884838399999, 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48305716700000001, 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.317161+0000", 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.317161+0000", 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.317162+0000", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.95e-07 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.317168+0000", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.913e-06 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.798490+0000", 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.481321946 2024-04-15T18:30:56.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.798498+0000", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0909999999999993e-06 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.798555+0000", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6762999999999998e-05 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:16.800218+0000", 2024-04-15T18:30:56.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016634589999999999 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:113653 5.f 5:fa10d0de:::benchmark_data_smithi159_116570_object20493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e438)", 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 198.03107105800001, 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44737796299999999, 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.327 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113653 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244940+0000", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5770000000000001e-06 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244948+0000", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1540000000000002e-06 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244966+0000", 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7779999999999999e-05 2024-04-15T18:30:56.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244974+0000", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4440000000000008e-06 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245031+0000", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245129+0000", 2024-04-15T18:30:56.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8061999999999997e-05 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245207+0000", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7355999999999996e-05 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.247755+0000", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025486670000000001 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.248470+0000", 2024-04-15T18:30:56.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071462499999999998 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692202+0000", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44373232600000001 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692262+0000", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9703000000000001e-05 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692316+0000", 2024-04-15T18:30:56.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4274999999999998e-05 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:118580 5.3 e459/415 5:c1d7d9b0:::benchmark_data_smithi159_116570_object25420:head v 459'7541, mlcod=459'7538)", 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 157.558254369, 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48049194000000001, 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717756+0000", 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3499999999999995e-07 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717761+0000", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5629999999999999e-06 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193762+0000", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47600057200000001 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193767+0000", 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0919999999999997e-06 2024-04-15T18:30:56.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193812+0000", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5049999999999997e-05 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193833+0000", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0786000000000001e-05 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198189+0000", 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043559719999999996 2024-04-15T18:30:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198247+0000", 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119061 5.7 5:e4a9cef3:::benchmark_data_smithi159_116570_object25901:head [delete] snapc 0=[] ondisk+write+known_if_redirected e460)", 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 153.66485560300001, 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52903251299999998, 2024-04-15T18:30:56.337 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119061 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9599999999999998e-07 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611159+0000", 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4669999999999998e-06 2024-04-15T18:30:56.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611170+0000", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0614e-05 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611176+0000", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0249999999999999e-06 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611214+0000", 2024-04-15T18:30:56.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8059999999999998e-05 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611297+0000", 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3330000000000003e-05 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611364+0000", 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6617000000000003e-05 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.613743+0000", 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002379459 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614653+0000", 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091005100000000005 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140066+0000", 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52541264300000001 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140152+0000", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6450000000000001e-05 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140186+0000", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3900999999999999e-05 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:119061 5.7 e460/396 ondisk, result = 0)", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:30:56.343 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 153.66141420700001, 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52559624999999999, 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:30:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6899999999999999e-07 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614598+0000", 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3099999999999999e-06 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.139929+0000", 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52533111200000004 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.139940+0000", 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1453e-05 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140035+0000", 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5049000000000004e-05 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140048+0000", 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3239e-05 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140191+0000", 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014281799999999999 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:127441 5.f 5:f29729c2:::benchmark_data_smithi159_116570_object34281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e481)", 2024-04-15T18:30:56.347 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.090168551, 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.509987313, 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 127441 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:30:56.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185842+0000", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.088e-06 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185846+0000", 2024-04-15T18:30:56.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5889999999999996e-06 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185857+0000", 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0915e-05 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185863+0000", 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.981e-06 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185908+0000", 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4372999999999997e-05 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185994+0000", 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6490999999999996e-05 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186065+0000", 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0646000000000001e-05 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.189580+0000", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035155519999999999 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.476339+0000", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286758654 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695753+0000", 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.219414315 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695794+0000", 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0277000000000001e-05 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695828+0000", 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4431999999999998e-05 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.353 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:129236 5.3 e486/415 5:ce6b8309:::benchmark_data_smithi159_116570_object36076:head v 486'8209, mlcod=484'8206)", 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 89.876327021999998, 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44633631600000001, 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8e-07 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399686+0000", 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5980000000000001e-06 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843210+0000", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44352421400000003 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843216+0000", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.767e-06 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843268+0000", 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2008000000000001e-05 2024-04-15T18:30:56.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843284+0000", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5656000000000001e-05 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.845981+0000", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026974550000000001 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.846018+0000", 2024-04-15T18:30:56.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7438000000000003e-05 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:129784 5.f 5:fae35ede:::benchmark_data_smithi159_116570_object36624:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.975766938999996, 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57517218699999995, 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:30:56.358 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 129784 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300244+0000", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.266e-06 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300249+0000", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.096e-06 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300266+0000", 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7793999999999999e-05 2024-04-15T18:30:56.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300275+0000", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.602e-06 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300326+0000", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1465000000000003e-05 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300428+0000", 2024-04-15T18:30:56.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101933 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300510+0000", 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1563999999999994e-05 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304070+0000", 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035599609999999999 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304862+0000", 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00079191600000000004 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875350+0000", 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.570488459 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875384+0000", 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4087000000000001e-05 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875414+0000", 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0043999999999999e-05 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:129784 5.f e486/393 ondisk, result = 0)", 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 86.971348734000003, 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57075858000000002, 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.364 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304661+0000", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5899999999999995e-07 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304666+0000", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2459999999999997e-06 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875294+0000", 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57062857700000003 2024-04-15T18:30:56.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875300+0000", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2090000000000002e-06 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875342+0000", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1758000000000002e-05 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875347+0000", 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8409999999999999e-06 2024-04-15T18:30:56.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875419+0000", 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2089999999999996e-05 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:144515 5.6 e508/437 5:63eb13b9:::benchmark_data_smithi159_116570_object51355:head v 508'8892, mlcod=507'8889)", 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 13.676816690999999, 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.474313188, 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:30:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7999999999999995e-07 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599198+0000", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6739999999999996e-06 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070635+0000", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47143691900000001 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070639+0000", 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.729e-06 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070689+0000", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9104000000000003e-05 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070706+0000", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7028e-05 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.073477+0000", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027715420000000001 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.073506+0000", 2024-04-15T18:30:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8612e-05 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:30:56.372 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:30:56.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:30:56.406 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:56.544 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:30:56.546 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:56 smithi082 ceph-mon[44206]: pgmap v1335: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 5069/118104 objects misplaced (4.292%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:56.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:30:56.648 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:30:56.676 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:56.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:56 smithi159 ceph-mon[45336]: pgmap v1335: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 5069/118104 objects misplaced (4.292%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:56.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:56 smithi049 ceph-mon[40087]: pgmap v1335: 17 pgs: 1 active+recovering, 16 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 5069/118104 objects misplaced (4.292%); 608 KiB/s, 9 objects/s recovering 2024-04-15T18:30:57.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:30:57.129 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:30:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:57.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:30:57.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:57.519 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:57.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:57.626 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:57.653 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:57.675 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:57.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:57.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:30:57.810 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:57.836 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:30:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:57.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:30:57.940 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:30:57.971 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:57.993 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:30:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:57.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:30:58.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:58.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:30:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:58.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:30:58.418 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:30:58.436 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:30:58.465 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:58.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:58 smithi049 ceph-mon[40087]: pgmap v1336: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4824/114687 objects misplaced (4.206%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:58.485 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:30:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:58.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:30:58.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:30:58.591 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:30:58.592 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:30:58.621 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:30:58.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:58 smithi159 ceph-mon[45336]: pgmap v1336: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4824/114687 objects misplaced (4.206%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:58.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:58 smithi082 ceph-mon[44206]: pgmap v1336: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 4824/114687 objects misplaced (4.206%); 1.1 MiB/s, 17 objects/s recovering 2024-04-15T18:30:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:30:59.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:59.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:30:59 smithi159 ceph-mon[45336]: from='client.16302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:59.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:59.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:30:59 smithi049 ceph-mon[40087]: from='client.16302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:59.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:30:59.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:30:59 smithi082 ceph-mon[44206]: from='client.16302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:30:59.913 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1336,"stamp":"2024-04-15T18:30:57.353520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2505769528,"num_objects":38229,"num_object_clones":0,"num_object_copies":114687,"num_objects_missing_on_primary":1600,"num_objects_missing":1600,"num_objects_degraded":0,"num_objects_misplaced":4824,"num_objects_unfound":0,"num_objects_dirty":38229,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":148270,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40070,"num_bytes_recovered":2623101600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149051,"ondisk_log_size":149051,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11759468,"kb_used_data":8054532,"kb_used_omap":177,"kb_used_meta":3704654,"kb_avail":1113182356,"statfs":{"total":1151940427776,"available":1139898732544,"internally_reserved":0,"allocated":8247840768,"data_stored":8239551568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3793566148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-164888576,"num_objects":-2516,"num_object_clones":0,"num_object_copies":-7548,"num_objects_missing_on_primary":-163,"num_objects_missing":-163,"num_objects_degraded":0,"num_objects_misplaced":-478,"num_objects_unfound":0,"num_objects_dirty":-2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":10354688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.025800"},"pg_stats":[{"pgid":"5.f","version":"512'9358","reported_seq":15433,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.215492+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:30:52.215492+0000","last_peered":"2024-04-15T18:30:52.215492+0000","last_clean":"2024-04-15T18:30:52.215492+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:30:52.215492+0000","last_undegraded":"2024-04-15T18:30:52.215492+0000","last_fullsized":"2024-04-15T18:30:52.215492+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":9358,"log_dups_size":0,"ondisk_log_size":9358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9358,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"512'9343","reported_seq":9855,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.254989+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:30:57.254989+0000","last_peered":"2024-04-15T18:30:57.254989+0000","last_clean":"2024-04-15T18:30:57.254989+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:30:57.254989+0000","last_undegraded":"2024-04-15T18:30:57.254989+0000","last_fullsized":"2024-04-15T18:30:57.254989+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153944064,"num_objects":2349,"num_object_clones":0,"num_object_copies":7047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"512'9236","reported_seq":15065,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:56.825294+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:30:56.825294+0000","last_peered":"2024-04-15T18:30:56.825294+0000","last_clean":"2024-04-15T18:30:56.825294+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:30:56.825294+0000","last_undegraded":"2024-04-15T18:30:56.825294+0000","last_fullsized":"2024-04-15T18:30:56.825294+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254808,"num_objects":2370,"num_object_clones":0,"num_object_copies":7110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9236,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"512'9329","reported_seq":9860,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.259697+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:30:57.259697+0000","last_peered":"2024-04-15T18:30:57.259697+0000","last_clean":"2024-04-15T18:30:57.259697+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:30:57.259697+0000","last_undegraded":"2024-04-15T18:30:57.259697+0000","last_fullsized":"2024-04-15T18:30:57.259697+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154992640,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"512'9424","reported_seq":15451,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.171774+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:30:57.171774+0000","last_peered":"2024-04-15T18:30:57.171774+0000","last_clean":"2024-04-15T18:30:57.171774+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:30:57.171774+0000","last_undegraded":"2024-04-15T18:30:57.171774+0000","last_fullsized":"2024-04-15T18:30:57.171774+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157024256,"num_objects":2396,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"512'9248","reported_seq":21403,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.078249+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:30:57.078249+0000","last_peered":"2024-04-15T18:30:57.078249+0000","last_clean":"2024-04-15T18:30:57.078249+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:30:57.078249+0000","last_undegraded":"2024-04-15T18:30:57.078249+0000","last_fullsized":"2024-04-15T18:30:57.078249+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9248,"log_dups_size":0,"ondisk_log_size":9248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157155328,"num_objects":2398,"num_object_clones":0,"num_object_copies":7194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"512'9273","reported_seq":9701,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:55.957952+0000","last_change":"2024-04-15T18:30:49.739135+0000","last_active":"2024-04-15T18:30:55.957952+0000","last_peered":"2024-04-15T18:30:55.957952+0000","last_clean":"2024-04-15T18:30:55.957952+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:30:55.957952+0000","last_undegraded":"2024-04-15T18:30:55.957952+0000","last_fullsized":"2024-04-15T18:30:55.957952+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":249,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9273,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"512'9297","reported_seq":9616,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:56.833822+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:30:56.833822+0000","last_peered":"2024-04-15T18:30:56.833822+0000","last_clean":"2024-04-15T18:30:56.833822+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:30:56.833822+0000","last_undegraded":"2024-04-15T18:30:56.833822+0000","last_fullsized":"2024-04-15T18:30:56.833822+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9297,"log_dups_size":0,"ondisk_log_size":9297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9297,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"512'9262","reported_seq":15329,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:52.268795+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:30:52.268795+0000","last_peered":"2024-04-15T18:30:52.268795+0000","last_clean":"2024-04-15T18:30:52.268795+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:30:52.268795+0000","last_undegraded":"2024-04-15T18:30:52.268795+0000","last_fullsized":"2024-04-15T18:30:52.268795+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"512'9141","reported_seq":26449,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.277613+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:30:57.277613+0000","last_peered":"2024-04-15T18:30:57.277613+0000","last_clean":"2024-04-15T18:30:57.277613+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:30:57.277613+0000","last_undegraded":"2024-04-15T18:30:57.277613+0000","last_fullsized":"2024-04-15T18:30:57.277613+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"512'9016","reported_seq":9527,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:57.270376+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:30:57.270376+0000","last_peered":"2024-04-15T18:30:57.270376+0000","last_clean":"2024-04-15T18:30:57.270376+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:30:57.270376+0000","last_undegraded":"2024-04-15T18:30:57.270376+0000","last_fullsized":"2024-04-15T18:30:57.270376+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":9016,"log_dups_size":0,"ondisk_log_size":9016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150601728,"num_objects":2298,"num_object_clones":0,"num_object_copies":6894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9016,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"512'9464","reported_seq":15464,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:55.385812+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:30:55.385812+0000","last_peered":"2024-04-15T18:30:55.385812+0000","last_clean":"2024-04-15T18:30:55.385812+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:30:55.385812+0000","last_undegraded":"2024-04-15T18:30:55.385812+0000","last_fullsized":"2024-04-15T18:30:55.385812+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157286400,"num_objects":2400,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"512'9170","reported_seq":9548,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:55.187613+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:30:55.187613+0000","last_peered":"2024-04-15T18:30:55.187613+0000","last_clean":"2024-04-15T18:30:55.187613+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:30:55.187613+0000","last_undegraded":"2024-04-15T18:30:55.187613+0000","last_fullsized":"2024-04-15T18:30:55.187613+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"512'9031","reported_seq":22794,"reported_epoch":512,"state":"active+recovering","last_fresh":"2024-04-15T18:30:56.991579+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:30:56.991579+0000","last_peered":"2024-04-15T18:30:56.991579+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:30:56.991579+0000","last_undegraded":"2024-04-15T18:30:56.991579+0000","last_fullsized":"2024-04-15T18:30:56.991579+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9031,"log_dups_size":0,"ondisk_log_size":9031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":6969,"num_objects_missing_on_primary":1600,"num_objects_missing":1600,"num_objects_degraded":0,"num_objects_misplaced":4824,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9031,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3357,"num_bytes_recovered":219873280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":1600},{"shards":"3,4,7,11","objects":12}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"512'9308","reported_seq":9790,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:53.018294+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:30:53.018294+0000","last_peered":"2024-04-15T18:30:53.018294+0000","last_clean":"2024-04-15T18:30:53.018294+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:30:53.018294+0000","last_undegraded":"2024-04-15T18:30:53.018294+0000","last_fullsized":"2024-04-15T18:30:53.018294+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":9308,"log_dups_size":0,"ondisk_log_size":9308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:11.828078+0000","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"512'13013","reported_seq":30693,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:53.033176+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:30:53.033176+0000","last_peered":"2024-04-15T18:30:53.033176+0000","last_clean":"2024-04-15T18:30:53.033176+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:30:53.033176+0000","last_undegraded":"2024-04-15T18:30:53.033176+0000","last_fullsized":"2024-04-15T18:30:53.033176+0000","mapping_epoch":440,"log_start":"380'3001","ondisk_log_start":"380'3001","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1281,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-04-15T18:30:49.519839+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:30:49.519839+0000","last_peered":"2024-04-15T18:30:49.519839+0000","last_clean":"2024-04-15T18:30:49.519839+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:30:49.519839+0000","last_undegraded":"2024-04-15T18:30:49.519839+0000","last_fullsized":"2024-04-15T18:30:49.519839+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2505179160,"num_objects":38227,"num_object_clones":0,"num_object_copies":114681,"num_objects_missing_on_primary":1600,"num_objects_missing":1600,"num_objects_degraded":0,"num_objects_misplaced":4824,"num_objects_unfound":0,"num_objects_dirty":38227,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148089,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40060,"num_bytes_recovered":2620456960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192405504,"data_stored":8192393288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148912,"ondisk_log_size":148912,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926085,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067132,"kb_used_data":661108,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92678020,"statfs":{"total":95995035648,"available":94902292480,"internally_reserved":0,"allocated":676974592,"data_stored":676278089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187721,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105776,"kb_used_data":795560,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92639376,"statfs":{"total":95995035648,"available":94862721024,"internally_reserved":0,"allocated":814653440,"data_stored":813964581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514767,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804592,"kb_used_data":454120,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":92940560,"statfs":{"total":95995035648,"available":95171133440,"internally_reserved":0,"allocated":465018880,"data_stored":464330021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":358873536},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469936,"kb_used_data":312424,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93275216,"statfs":{"total":95995035648,"available":95513821184,"internally_reserved":0,"allocated":319922176,"data_stored":319233317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124668,"kb_used_data":778356,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92620484,"statfs":{"total":95995035648,"available":94843375616,"internally_reserved":0,"allocated":797036544,"data_stored":796340061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463198,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854008,"kb_used_data":511856,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92891144,"statfs":{"total":95995035648,"available":95120531456,"internally_reserved":0,"allocated":524140544,"data_stored":523448133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575163,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":985776,"kb_used_data":603368,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92759376,"statfs":{"total":95995035648,"available":94985601024,"internally_reserved":0,"allocated":617848832,"data_stored":617159973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1523056,"kb_used_data":1118440,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":92222096,"statfs":{"total":95995035648,"available":94435426304,"internally_reserved":0,"allocated":1145282560,"data_stored":1144593701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563632,"kb_used_data":455272,"kb_used_omap":114,"kb_used_meta":108237,"kb_avail":93181520,"statfs":{"total":95995035648,"available":95417876480,"internally_reserved":0,"allocated":466198528,"data_stored":465509669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":110834929},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653944,"kb_used_data":308392,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091208,"statfs":{"total":95995035648,"available":95325396992,"internally_reserved":0,"allocated":315793408,"data_stored":315104549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986470,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328688,"kb_used_data":950440,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92416464,"statfs":{"total":95995035648,"available":94634459136,"internally_reserved":0,"allocated":973250560,"data_stored":972561701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703365,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278260,"kb_used_data":1105196,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":92466892,"statfs":{"total":95995035648,"available":94686097408,"internally_reserved":0,"allocated":1131720704,"data_stored":1131027773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1140523008,"data_stored":1140523008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":613679104,"data_stored":613679104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462028800,"data_stored":462028800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":969998336,"data_stored":969998336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1121456128,"data_stored":1121452056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":519176192,"data_stored":519176192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":790827008,"data_stored":790822936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":316145664,"data_stored":316145664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":460193792,"data_stored":460193792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":811794432,"data_stored":811794432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":672665600,"data_stored":672661528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:30:59.914 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.1 is not active+clean 2024-04-15T18:30:59.914 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.1', 'version': "512'9031", 'reported_seq': 22794, 'reported_epoch': 512, 'state': 'active+recovering', 'last_fresh': '2024-04-15T18:30:56.991579+0000', 'last_change': '2024-04-15T18:26:49.574312+0000', 'last_active': '2024-04-15T18:30:56.991579+0000', 'last_peered': '2024-04-15T18:30:56.991579+0000', 'last_clean': '2024-04-15T18:26:00.259169+0000', 'last_became_active': '2024-04-15T18:26:48.920249+0000', 'last_became_peered': '2024-04-15T18:26:48.920249+0000', 'last_unstale': '2024-04-15T18:30:56.991579+0000', 'last_undegraded': '2024-04-15T18:30:56.991579+0000', 'last_fullsized': '2024-04-15T18:30:56.991579+0000', 'mapping_epoch': 416, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'objects_scrubbed': 0, 'log_size': 9031, 'log_dups_size': 0, 'ondisk_log_size': 9031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 152240128, 'num_objects': 2323, 'num_object_clones': 0, 'num_object_copies': 6969, 'num_objects_missing_on_primary': 1600, 'num_objects_missing': 1600, 'num_objects_degraded': 0, 'num_objects_misplaced': 4824, 'num_objects_unfound': 0, 'num_objects_dirty': 2323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9031, 'num_write_kb': 363328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3357, 'num_bytes_recovered': 219873280, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0, 6], 'acting': [11, 0, 6], 'avail_no_missing': ['3', '4', '7'], 'object_location_counts': [{'shards': '0,6,11', 'objects': 711}, {'shards': '3,4,7', 'objects': 1600}, {'shards': '3,4,7,11', 'objects': 12}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-15T18:30:59.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:30:59.975 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:30:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:30:59.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:00.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:00.106 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.127 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:00.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:00.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:00.426 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.445 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:00.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:00.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: osdmap e513: 12 total, 12 up, 12 in 2024-04-15T18:31:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: pgmap v1338: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 297 op/s; 4824/113451 objects misplaced (4.252%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:31:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:00.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:00.548 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:00.575 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.593 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:00.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: osdmap e513: 12 total, 12 up, 12 in 2024-04-15T18:31:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: pgmap v1338: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 297 op/s; 4824/113451 objects misplaced (4.252%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:31:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:00.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:00.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:00.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4240486456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:00.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: osdmap e513: 12 total, 12 up, 12 in 2024-04-15T18:31:00.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: pgmap v1338: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 297 op/s; 4824/113451 objects misplaced (4.252%); 1.0 MiB/s, 16 objects/s recovering 2024-04-15T18:31:00.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:00.690 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:00.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:00.724 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.747 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:00.854 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:00.880 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:00.901 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:00.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:01.006 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:01.033 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.053 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:01.159 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:01.189 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.211 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:01.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:01.343 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.376 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:01.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:01.514 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:01 smithi159 ceph-mon[45336]: osdmap e514: 12 total, 12 up, 12 in 2024-04-15T18:31:01.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:01 smithi049 ceph-mon[40087]: osdmap e514: 12 total, 12 up, 12 in 2024-04-15T18:31:01.654 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:01.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:01 smithi082 ceph-mon[44206]: osdmap e514: 12 total, 12 up, 12 in 2024-04-15T18:31:01.800 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.825 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:01.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:150055 5.5 e515/415 5:aeb04ea2:::benchmark_data_smithi159_116570_object56895:head v 515'9118, mlcod=515'9116)", 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:31:01.950838+0000", 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0028886430000000002, 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029139819999999999, 2024-04-15T18:31:01.946 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950838+0000", 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950838+0000", 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950840+0000", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5030000000000001e-06 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950851+0000", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1358e-05 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950903+0000", 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1758999999999997e-05 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950914+0000", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0645000000000001e-05 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.950981+0000", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7589000000000006e-05 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:01.951008+0000", 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7180000000000001e-05 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:01.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:31:01.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:01.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:01.950 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:31:01.950 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-04-15T18:31:01.950 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:31:01.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:01.963 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:01.977 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:01.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:02.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:02.113 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:02.216 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:02.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:02.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:02.349 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:02.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:02.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:02.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:02.500 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:02.520 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:02.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:02.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:02.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:02 smithi159 ceph-mon[45336]: osdmap e515: 12 total, 12 up, 12 in 2024-04-15T18:31:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:02 smithi159 ceph-mon[45336]: pgmap v1341: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 4824/112602 objects misplaced (4.284%); 875 KiB/s, 13 objects/s recovering 2024-04-15T18:31:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:02 smithi159 ceph-mon[45336]: osdmap e516: 12 total, 12 up, 12 in 2024-04-15T18:31:02.650 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:02.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:02.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:02 smithi082 ceph-mon[44206]: osdmap e515: 12 total, 12 up, 12 in 2024-04-15T18:31:02.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:02 smithi082 ceph-mon[44206]: pgmap v1341: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 4824/112602 objects misplaced (4.284%); 875 KiB/s, 13 objects/s recovering 2024-04-15T18:31:02.651 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:02 smithi082 ceph-mon[44206]: osdmap e516: 12 total, 12 up, 12 in 2024-04-15T18:31:02.653 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:02.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:02.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:02.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:02 smithi049 ceph-mon[40087]: osdmap e515: 12 total, 12 up, 12 in 2024-04-15T18:31:02.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:02 smithi049 ceph-mon[40087]: pgmap v1341: 17 pgs: 1 active+recovering, 16 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 4824/112602 objects misplaced (4.284%); 875 KiB/s, 13 objects/s recovering 2024-04-15T18:31:02.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:02 smithi049 ceph-mon[40087]: osdmap e516: 12 total, 12 up, 12 in 2024-04-15T18:31:02.672 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:02.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:02.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:02.802 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:02.821 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:02.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:02.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:02.955 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:03.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:31:04.107 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:04.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:04.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:04.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:31:04.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:04.245 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:04.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:04 smithi049 ceph-mon[40087]: pgmap v1343: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4570/108876 objects misplaced (4.197%); 1.3 MiB/s, 21 objects/s recovering 2024-04-15T18:31:04.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:04 smithi082 ceph-mon[44206]: pgmap v1343: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4570/108876 objects misplaced (4.197%); 1.3 MiB/s, 21 objects/s recovering 2024-04-15T18:31:04.897 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:04.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:04 smithi159 ceph-mon[45336]: pgmap v1343: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4570/108876 objects misplaced (4.197%); 1.3 MiB/s, 21 objects/s recovering 2024-04-15T18:31:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:04.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:04.946 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:05.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:05.031 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:05.178 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:05.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:05.286 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:05.313 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:05.335 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:05.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:05.439 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:05.472 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:05.493 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:05.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:05.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:05.636 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:05.899 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:05.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:06.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:06.041 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:06.073 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:06.073 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1613,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":516,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":36292,"data_bytes":2378826296,"bytes_used":11623989248,"bytes_avail":1140316438528,"bytes_total":1151940427776,"misplaced_objects":4570,"misplaced_total":108876,"misplaced_ratio":0.041974356148278782,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":939187,"recovering_keys_per_sec":0,"num_objects_recovered":86,"num_bytes_recovered":5636096,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":36,"modified":"2024-04-15T18:31:01.354261+0000","services":{}},"progress_events":{}} 2024-04-15T18:31:06.121 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:06.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:06.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:06.258 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:06.279 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:06.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:06.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:06.414 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:06.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:06.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:06.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:06 smithi082 ceph-mon[44206]: pgmap v1344: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4570/108876 objects misplaced (4.197%); 917 KiB/s, 14 objects/s recovering 2024-04-15T18:31:06.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/156491843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:06.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:06.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2827947507' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:06.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:06.570 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:06.632 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:31:06.663 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:31:06.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:31:06.712 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:31:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:06 smithi049 ceph-mon[40087]: pgmap v1344: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4570/108876 objects misplaced (4.197%); 917 KiB/s, 14 objects/s recovering 2024-04-15T18:31:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/156491843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:06.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2827947507' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:06.736 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:31:06.765 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:31:06.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:31:06.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:31:06.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:31:06.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:31:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:06 smithi159 ceph-mon[45336]: pgmap v1344: 17 pgs: 1 active+recovering, 16 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 4570/108876 objects misplaced (4.197%); 917 KiB/s, 14 objects/s recovering 2024-04-15T18:31:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/156491843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:06.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2827947507' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:06.905 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:31:06.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:31:06.931 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:31:07.526 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:07.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:07.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:07.666 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.666 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.667 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.667 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.667 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.667 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:07 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:31:07.674 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:07.697 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:07.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:07 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:07.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:07 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:31:07.800 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:07.828 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:07.850 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:07.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:07.954 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:08.004 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:08.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:08.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:08.138 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: pgmap v1345: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4305/105462 objects misplaced (4.082%); 1.6 MiB/s, 24 objects/s recovering 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:31:08.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:31:08.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: pgmap v1345: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4305/105462 objects misplaced (4.082%); 1.6 MiB/s, 24 objects/s recovering 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:31:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:31:08.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: pgmap v1345: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 4305/105462 objects misplaced (4.082%); 1.6 MiB/s, 24 objects/s recovering 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:31:08.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:31:09.390 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:09.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:09.411 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:31:09.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:31:09.480 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:31:09.480 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:31:09.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:09.528 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:09.679 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:09.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:09.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:09.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:09 smithi049 ceph-mon[40087]: osdmap e517: 12 total, 12 up, 12 in 2024-04-15T18:31:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:09.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:09 smithi082 ceph-mon[44206]: osdmap e517: 12 total, 12 up, 12 in 2024-04-15T18:31:09.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:09.811 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:09.831 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:09.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:09.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:09.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:09.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:09 smithi159 ceph-mon[45336]: osdmap e517: 12 total, 12 up, 12 in 2024-04-15T18:31:09.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:09.965 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:10.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:31:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:10.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:10.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:10.460 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:10.660 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:10.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:10.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:10.810 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:10.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:10.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:10.947 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:10 smithi049 ceph-mon[40087]: pgmap v1347: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 344 op/s; 4305/104322 objects misplaced (4.127%); 1.4 MiB/s, 21 objects/s recovering 2024-04-15T18:31:10.972 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:10.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:11.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:10 smithi082 ceph-mon[44206]: pgmap v1347: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 344 op/s; 4305/104322 objects misplaced (4.127%); 1.4 MiB/s, 21 objects/s recovering 2024-04-15T18:31:11.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:11.104 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:11.129 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:11.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:11.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:10 smithi159 ceph-mon[45336]: pgmap v1347: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 344 op/s; 4305/104322 objects misplaced (4.127%); 1.4 MiB/s, 21 objects/s recovering 2024-04-15T18:31:11.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:11.266 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:11.292 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:11.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:11.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:11.427 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:11.450 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:11.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:11.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:11.589 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:11.838 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:11.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:11.845 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:11.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:11.972 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:11.992 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:11.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:12.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:12.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:12.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:12.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:12.120 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:12.122 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:12.123 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:12.125 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.147 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:12.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:12.279 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.299 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:12.403 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:12.429 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.447 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:12.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:12.580 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.601 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:12.706 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:12.732 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.751 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:12.854 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:12.882 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:12.901 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:12.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:12 smithi049 ceph-mon[40087]: pgmap v1348: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 4305/103371 objects misplaced (4.165%); 624 KiB/s, 9 objects/s recovering 2024-04-15T18:31:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:12 smithi049 ceph-mon[40087]: osdmap e518: 12 total, 12 up, 12 in 2024-04-15T18:31:13.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:13.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:12 smithi082 ceph-mon[44206]: pgmap v1348: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 4305/103371 objects misplaced (4.165%); 624 KiB/s, 9 objects/s recovering 2024-04-15T18:31:13.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:12 smithi082 ceph-mon[44206]: osdmap e518: 12 total, 12 up, 12 in 2024-04-15T18:31:13.038 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:13.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:12 smithi159 ceph-mon[45336]: pgmap v1348: 17 pgs: 1 active+recovering, 16 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 268 op/s; 4305/103371 objects misplaced (4.165%); 624 KiB/s, 9 objects/s recovering 2024-04-15T18:31:13.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:12 smithi159 ceph-mon[45336]: osdmap e518: 12 total, 12 up, 12 in 2024-04-15T18:31:13.776 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:31:13.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:31:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:13 smithi082 ceph-mon[44206]: osdmap e519: 12 total, 12 up, 12 in 2024-04-15T18:31:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:13 smithi159 ceph-mon[45336]: osdmap e519: 12 total, 12 up, 12 in 2024-04-15T18:31:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:13 smithi049 ceph-mon[40087]: osdmap e519: 12 total, 12 up, 12 in 2024-04-15T18:31:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:14.494 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:14.496 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:31:15.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:14 smithi082 ceph-mon[44206]: pgmap v1351: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 327 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:15.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:14 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:15.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:15.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:14 smithi082 ceph-mon[44206]: osdmap e520: 12 total, 12 up, 12 in 2024-04-15T18:31:15.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:14 smithi049 ceph-mon[40087]: pgmap v1351: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 327 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:15.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:14 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:15.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:15.080 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:14 smithi049 ceph-mon[40087]: osdmap e520: 12 total, 12 up, 12 in 2024-04-15T18:31:15.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:14 smithi159 ceph-mon[45336]: pgmap v1351: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 327 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:15.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:14 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:15.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/852899568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:15.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:14 smithi159 ceph-mon[45336]: osdmap e520: 12 total, 12 up, 12 in 2024-04-15T18:31:15.513 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1352,"stamp":"2024-04-15T18:31:13.778403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2175337016,"num_objects":33187,"num_object_clones":0,"num_object_copies":99561,"num_objects_missing_on_primary":1337,"num_objects_missing":1337,"num_objects_degraded":0,"num_objects_misplaced":4027,"num_objects_unfound":0,"num_objects_dirty":33187,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":153312,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40337,"num_bytes_recovered":2640599712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153693,"ondisk_log_size":153693,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10779356,"kb_used_data":7110428,"kb_used_omap":177,"kb_used_meta":3668622,"kb_avail":1114162468,"statfs":{"total":1151940427776,"available":1140902367232,"internally_reserved":0,"allocated":7281078272,"data_stored":7272493182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3756669380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-104005632,"num_objects":-1587,"num_object_clones":0,"num_object_copies":-4761,"num_objects_missing_on_primary":-92,"num_objects_missing":-92,"num_objects_degraded":0,"num_objects_misplaced":-278,"num_objects_unfound":0,"num_objects_dirty":-1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":6094848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.018374"},"pg_stats":[{"pgid":"5.f","version":"518'9769","reported_seq":15856,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.352841+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:31:12.352841+0000","last_peered":"2024-04-15T18:31:12.352841+0000","last_clean":"2024-04-15T18:31:12.352841+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:31:12.352841+0000","last_undegraded":"2024-04-15T18:31:12.352841+0000","last_fullsized":"2024-04-15T18:31:12.352841+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134545408,"num_objects":2053,"num_object_clones":0,"num_object_copies":6159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"518'9616","reported_seq":10141,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.253370+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:31:12.253370+0000","last_peered":"2024-04-15T18:31:12.253370+0000","last_clean":"2024-04-15T18:31:12.253370+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:31:12.253370+0000","last_undegraded":"2024-04-15T18:31:12.253370+0000","last_fullsized":"2024-04-15T18:31:12.253370+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":9616,"log_dups_size":0,"ondisk_log_size":9616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"518'9516","reported_seq":15357,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.013614+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:31:12.013614+0000","last_peered":"2024-04-15T18:31:12.013614+0000","last_clean":"2024-04-15T18:31:12.013614+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:31:12.013614+0000","last_undegraded":"2024-04-15T18:31:12.013614+0000","last_fullsized":"2024-04-15T18:31:12.013614+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136904728,"num_objects":2090,"num_object_clones":0,"num_object_copies":6270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9516,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"518'9621","reported_seq":10164,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.220066+0000","last_change":"2024-04-15T18:29:57.398780+0000","last_active":"2024-04-15T18:31:12.220066+0000","last_peered":"2024-04-15T18:31:12.220066+0000","last_clean":"2024-04-15T18:31:12.220066+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:31:12.220066+0000","last_undegraded":"2024-04-15T18:31:12.220066+0000","last_fullsized":"2024-04-15T18:31:12.220066+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8439","last_scrub_stamp":"2024-04-15T18:29:57.398670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:57.398670+0000","objects_scrubbed":3257,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:17.280228+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":6219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9621,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"518'9707","reported_seq":15746,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.327578+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:31:12.327578+0000","last_peered":"2024-04-15T18:31:12.327578+0000","last_clean":"2024-04-15T18:31:12.327578+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:31:12.327578+0000","last_undegraded":"2024-04-15T18:31:12.327578+0000","last_fullsized":"2024-04-15T18:31:12.327578+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138477568,"num_objects":2113,"num_object_clones":0,"num_object_copies":6339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"518'9520","reported_seq":21687,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.275436+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:31:12.275436+0000","last_peered":"2024-04-15T18:31:12.275436+0000","last_clean":"2024-04-15T18:31:12.275436+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:31:12.275436+0000","last_undegraded":"2024-04-15T18:31:12.275436+0000","last_fullsized":"2024-04-15T18:31:12.275436+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139329536,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9520,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"517'9557","reported_seq":9995,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-04-15T18:31:10.931035+0000","last_change":"2024-04-15T18:30:49.739135+0000","last_active":"2024-04-15T18:31:10.931035+0000","last_peered":"2024-04-15T18:31:10.931035+0000","last_clean":"2024-04-15T18:31:10.931035+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:31:10.931035+0000","last_undegraded":"2024-04-15T18:31:10.931035+0000","last_fullsized":"2024-04-15T18:31:10.931035+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":249,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:12.191201+0000","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137953280,"num_objects":2105,"num_object_clones":0,"num_object_copies":6315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"518'9615","reported_seq":9947,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.034709+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:31:12.034709+0000","last_peered":"2024-04-15T18:31:12.034709+0000","last_clean":"2024-04-15T18:31:12.034709+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:31:12.034709+0000","last_undegraded":"2024-04-15T18:31:12.034709+0000","last_fullsized":"2024-04-15T18:31:12.034709+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9615,"log_dups_size":0,"ondisk_log_size":9615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139788288,"num_objects":2133,"num_object_clones":0,"num_object_copies":6399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9615,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"518'9650","reported_seq":15730,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.254358+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:31:12.254358+0000","last_peered":"2024-04-15T18:31:12.254358+0000","last_clean":"2024-04-15T18:31:12.254358+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:31:12.254358+0000","last_undegraded":"2024-04-15T18:31:12.254358+0000","last_fullsized":"2024-04-15T18:31:12.254358+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":1986,"num_object_clones":0,"num_object_copies":5958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"518'9429","reported_seq":26749,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.292860+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:31:12.292860+0000","last_peered":"2024-04-15T18:31:12.292860+0000","last_clean":"2024-04-15T18:31:12.292860+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:31:12.292860+0000","last_undegraded":"2024-04-15T18:31:12.292860+0000","last_fullsized":"2024-04-15T18:31:12.292860+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131530752,"num_objects":2007,"num_object_clones":0,"num_object_copies":6021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9429,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"518'9324","reported_seq":9847,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.252449+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:31:12.252449+0000","last_peered":"2024-04-15T18:31:12.252449+0000","last_clean":"2024-04-15T18:31:12.252449+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:31:12.252449+0000","last_undegraded":"2024-04-15T18:31:12.252449+0000","last_fullsized":"2024-04-15T18:31:12.252449+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":9324,"log_dups_size":0,"ondisk_log_size":9324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"517'9755","reported_seq":15764,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-04-15T18:31:10.390534+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:31:10.390534+0000","last_peered":"2024-04-15T18:31:10.390534+0000","last_clean":"2024-04-15T18:31:10.390534+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:31:10.390534+0000","last_undegraded":"2024-04-15T18:31:10.390534+0000","last_fullsized":"2024-04-15T18:31:10.390534+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138215424,"num_objects":2109,"num_object_clones":0,"num_object_copies":6327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"517'9467","reported_seq":9855,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-04-15T18:31:10.313758+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:31:10.313758+0000","last_peered":"2024-04-15T18:31:10.313758+0000","last_clean":"2024-04-15T18:31:10.313758+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:31:10.313758+0000","last_undegraded":"2024-04-15T18:31:10.313758+0000","last_fullsized":"2024-04-15T18:31:10.313758+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":9467,"log_dups_size":0,"ondisk_log_size":9467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9467,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"518'9298","reported_seq":24137,"reported_epoch":518,"state":"active+recovering","last_fresh":"2024-04-15T18:31:12.044987+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:31:12.044987+0000","last_peered":"2024-04-15T18:31:12.044987+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:31:12.044987+0000","last_undegraded":"2024-04-15T18:31:12.044987+0000","last_fullsized":"2024-04-15T18:31:12.044987+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134742016,"num_objects":2056,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":1337,"num_objects_missing":1337,"num_objects_degraded":0,"num_objects_misplaced":4027,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3624,"num_bytes_recovered":237371392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":1337},{"shards":"3,4,7,11","objects":8}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"519'9707","reported_seq":10203,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.993034+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:31:12.993034+0000","last_peered":"2024-04-15T18:31:12.993034+0000","last_clean":"2024-04-15T18:31:12.993034+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:31:12.993034+0000","last_undegraded":"2024-04-15T18:31:12.993034+0000","last_fullsized":"2024-04-15T18:31:12.993034+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137953280,"num_objects":2105,"num_object_clones":0,"num_object_copies":6315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"519'13404","reported_seq":31098,"reported_epoch":519,"state":"active+clean","last_fresh":"2024-04-15T18:31:12.949448+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:31:12.949448+0000","last_peered":"2024-04-15T18:31:12.949448+0000","last_clean":"2024-04-15T18:31:12.949448+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:31:12.949448+0000","last_undegraded":"2024-04-15T18:31:12.949448+0000","last_fullsized":"2024-04-15T18:31:12.949448+0000","mapping_epoch":440,"log_start":"384'3401","ondisk_log_start":"384'3401","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":2048,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9580,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1291,"reported_epoch":517,"state":"active+clean","last_fresh":"2024-04-15T18:31:08.765878+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:31:08.765878+0000","last_peered":"2024-04-15T18:31:08.765878+0000","last_clean":"2024-04-15T18:31:08.765878+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:31:08.765878+0000","last_undegraded":"2024-04-15T18:31:08.765878+0000","last_fullsized":"2024-04-15T18:31:08.765878+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2174746648,"num_objects":33185,"num_object_clones":0,"num_object_copies":99555,"num_objects_missing_on_primary":1337,"num_objects_missing":1337,"num_objects_degraded":0,"num_objects_misplaced":4027,"num_objects_unfound":0,"num_objects_dirty":33185,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153131,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40327,"num_bytes_recovered":2637955072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7227453440,"data_stored":7227441224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153554,"ondisk_log_size":153554,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926088,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000632,"kb_used_data":586288,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92744520,"statfs":{"total":95995035648,"available":94970388480,"internally_reserved":0,"allocated":600358912,"data_stored":599639984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187725,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":990924,"kb_used_data":670972,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92754228,"statfs":{"total":95995035648,"available":94980329472,"internally_reserved":0,"allocated":687075328,"data_stored":686355114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514770,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733420,"kb_used_data":399588,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93011732,"statfs":{"total":95995035648,"available":95244013568,"internally_reserved":0,"allocated":409178112,"data_stored":408466828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":341834176},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438832,"kb_used_data":277156,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93306320,"statfs":{"total":95995035648,"available":95545671680,"internally_reserved":0,"allocated":283807744,"data_stored":283096460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057404,"kb_used_data":723568,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92687748,"statfs":{"total":95995035648,"available":94912253952,"internally_reserved":0,"allocated":740933632,"data_stored":740214724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":341834186},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463201,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790772,"kb_used_data":456940,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92954380,"statfs":{"total":95995035648,"available":95185285120,"internally_reserved":0,"allocated":467906560,"data_stored":467191724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897708,"kb_used_data":527780,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92847444,"statfs":{"total":95995035648,"available":95075782656,"internally_reserved":0,"allocated":540446720,"data_stored":539735436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385836,"kb_used_data":989540,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92359316,"statfs":{"total":95995035648,"available":94575939584,"internally_reserved":0,"allocated":1013288960,"data_stored":1012577676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193938,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":513020,"kb_used_data":400496,"kb_used_omap":114,"kb_used_meta":112397,"kb_avail":93232132,"statfs":{"total":95995035648,"available":95469703168,"internally_reserved":0,"allocated":410107904,"data_stored":409392155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":115094769},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654012,"kb_used_data":308464,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091140,"statfs":{"total":95995035648,"available":95325327360,"internally_reserved":0,"allocated":315867136,"data_stored":315151387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986474,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1229500,"kb_used_data":847088,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92515652,"statfs":{"total":95995035648,"available":94736027648,"internally_reserved":0,"allocated":867418112,"data_stored":866702363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703369,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087296,"kb_used_data":922548,"kb_used_omap":4,"kb_used_meta":164731,"kb_avail":92657856,"statfs":{"total":95995035648,"available":94881644544,"internally_reserved":0,"allocated":944689152,"data_stored":943969331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":168685249},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1006108672,"data_stored":1006108672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":535363584,"data_stored":535363584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":405929984,"data_stored":405929984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":863371264,"data_stored":863371264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":941953024,"data_stored":941948952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":462225408,"data_stored":462225408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":734793728,"data_stored":734789656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":280625152,"data_stored":280625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":404684800,"data_stored":404684800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":683802624,"data_stored":683802624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":594677760,"data_stored":594673688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:31:15.520 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:15.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:15.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:15.657 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:15.678 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:15.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:15.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:15.816 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:16.015 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:16.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:16.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:15 smithi082 ceph-mon[44206]: from='client.16326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:16.032 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:16.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:16.145 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:15 smithi049 ceph-mon[40087]: from='client.16326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:16.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:15 smithi159 ceph-mon[45336]: from='client.16326 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:16.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:31:16.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:16 smithi049 ceph-mon[40087]: pgmap v1353: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 264 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:16.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:16 smithi049 ceph-mon[40087]: 5.9 scrub starts 2024-04-15T18:31:17.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:16 smithi082 ceph-mon[44206]: pgmap v1353: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 264 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:17.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:16 smithi082 ceph-mon[44206]: 5.9 scrub starts 2024-04-15T18:31:17.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:16 smithi159 ceph-mon[45336]: pgmap v1353: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 264 op/s; 4027/99561 objects misplaced (4.045%); 992 KiB/s, 15 objects/s recovering 2024-04-15T18:31:17.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:16 smithi159 ceph-mon[45336]: 5.9 scrub starts 2024-04-15T18:31:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:17.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:17.507 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:17.542 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:17.565 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:17.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:17.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:17.826 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:17.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:17.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:17.957 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:17.976 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:17.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:18.080 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:18.107 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.127 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:18.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:18.131 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:31:18.131 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:31:18.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:18.258 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.281 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:18.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:18.384 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:18.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:18 smithi159 ceph-mon[45336]: pgmap v1354: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 3729/96468 objects misplaced (3.866%); 2.0 MiB/s, 31 objects/s recovering 2024-04-15T18:31:18.413 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:18 smithi049 ceph-mon[40087]: pgmap v1354: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 3729/96468 objects misplaced (3.866%); 2.0 MiB/s, 31 objects/s recovering 2024-04-15T18:31:18.435 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:18.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:18.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:31:18.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:18 smithi082 ceph-mon[44206]: pgmap v1354: 17 pgs: 1 active+recovering, 16 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 3729/96468 objects misplaced (3.866%); 2.0 MiB/s, 31 objects/s recovering 2024-04-15T18:31:18.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:18.587 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.608 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:18.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:18.712 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:18.737 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.756 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:18.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:18.827 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:18.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:18.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:31:18.895 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:18.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:31:18.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:31:18.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:31:18.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:31:19.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:31:19.053 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:31:19.080 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:31:19.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:19.117 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:31:19.146 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:31:19.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:31:19.201 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:31:19.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:31:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:31:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:31:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:31:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:31:19.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:19.232 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:19.251 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:19.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:19.385 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:19.426 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:19.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3348501197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:19.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.531 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:19.557 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:19.577 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3348501197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:19.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3348501197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:19.715 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:19.756 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:19.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:19.888 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:19.909 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:19.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:20.013 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:20.039 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:20.063 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:20.068 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:20.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:20.196 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:20.226 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:20.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:20.329 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: 5.c scrub starts 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.339 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: 5.c scrub ok 2024-04-15T18:31:20.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.340 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:20 smithi159 ceph-mon[45336]: pgmap v1355: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 3729/95385 objects misplaced (3.909%); 1.6 MiB/s, 25 objects/s recovering 2024-04-15T18:31:20.364 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:20.386 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:20.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: 5.c scrub starts 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: 5.c scrub ok 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:20 smithi049 ceph-mon[40087]: pgmap v1355: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 3729/95385 objects misplaced (3.909%); 1.6 MiB/s, 25 objects/s recovering 2024-04-15T18:31:20.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:20.520 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: 5.c scrub starts 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: 5.c scrub ok 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:20.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:20 smithi082 ceph-mon[44206]: pgmap v1355: 17 pgs: 1 active+recovering, 16 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 350 op/s; 3729/95385 objects misplaced (3.909%); 1.6 MiB/s, 25 objects/s recovering 2024-04-15T18:31:20.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:31:20.869 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:20.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:20.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:21.012 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:21.038 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:21.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:21.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:21.170 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:21.368 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:21.369 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:21.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:21.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:21.504 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:21.523 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:21.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:21.627 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:21.655 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:21.675 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:21.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:21.779 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:21.806 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:21.825 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:21.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:21.934 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:21.982 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:22.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:22.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:22.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:22.136 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:22.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:31:22.586 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:22.586 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:22.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:22.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:22.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:22 smithi049 ceph-mon[40087]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 3729/94503 objects misplaced (3.946%); 738 KiB/s, 11 objects/s recovering 2024-04-15T18:31:22.722 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:22.723 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:22 smithi159 ceph-mon[45336]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 3729/94503 objects misplaced (3.946%); 738 KiB/s, 11 objects/s recovering 2024-04-15T18:31:22.744 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:22.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:22 smithi082 ceph-mon[44206]: pgmap v1356: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 220 op/s; 3729/94503 objects misplaced (3.946%); 738 KiB/s, 11 objects/s recovering 2024-04-15T18:31:22.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:22.875 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:22.895 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:22.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:23.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:23.030 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:23.419 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:31:23.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:23.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:23.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:23.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:23.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:23.955 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:23.977 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:23.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:24.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:24.113 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:24.133 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:24.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:24.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:24.268 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:24.292 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:24.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:24.395 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:24.422 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:24.441 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:24.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:24.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:24 smithi082 ceph-mon[44206]: pgmap v1357: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:24.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:24 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:31:24.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:24 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:31:24.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:24.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:24 smithi082 ceph-mon[44206]: osdmap e521: 12 total, 12 up, 12 in 2024-04-15T18:31:24.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:24.573 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:24.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:24 smithi049 ceph-mon[40087]: pgmap v1357: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:24.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:24 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:31:24.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:24 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:31:24.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:24.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:24 smithi049 ceph-mon[40087]: osdmap e521: 12 total, 12 up, 12 in 2024-04-15T18:31:24.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:24 smithi159 ceph-mon[45336]: pgmap v1357: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:24.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:24 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:31:24.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:24 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:31:24.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/122510938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:24.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:24 smithi159 ceph-mon[45336]: osdmap e521: 12 total, 12 up, 12 in 2024-04-15T18:31:25.625 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:25.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:25.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:25.806 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:25.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:25.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:25.944 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:25.966 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:25.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:26.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:26.108 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:26.191 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:26.194 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:31:26.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:26 smithi049 ceph-mon[40087]: pgmap v1359: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:26.660 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:26 smithi049 ceph-mon[40087]: from='client.16344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:26.664 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:26.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:26.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:26.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:26 smithi082 ceph-mon[44206]: pgmap v1359: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:26.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:26 smithi082 ceph-mon[44206]: from='client.16344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:26.802 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:26.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:26 smithi159 ceph-mon[45336]: pgmap v1359: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 288 op/s; 3408/90900 objects misplaced (3.749%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:26.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:26 smithi159 ceph-mon[45336]: from='client.16344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:26.971 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:27.004 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:27.015 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1359,"stamp":"2024-04-15T18:31:25.365708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1986134584,"num_objects":30300,"num_object_clones":0,"num_object_copies":90900,"num_objects_missing_on_primary":1131,"num_objects_missing":1131,"num_objects_degraded":0,"num_objects_misplaced":3408,"num_objects_unfound":0,"num_objects_dirty":30300,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":156199,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40543,"num_bytes_recovered":2654100128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156480,"ondisk_log_size":156480,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10255868,"kb_used_data":6578640,"kb_used_omap":177,"kb_used_meta":3676942,"kb_avail":1114685956,"statfs":{"total":1151940427776,"available":1141438418944,"internally_reserved":0,"allocated":6736527360,"data_stored":6727803205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3765189060},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,3,1,1,2,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-189202432,"num_objects":-2887,"num_object_clones":0,"num_object_copies":-8661,"num_objects_missing_on_primary":-206,"num_objects_missing":-206,"num_objects_degraded":0,"num_objects_misplaced":-619,"num_objects_unfound":0,"num_objects_dirty":-2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":13500416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.009153"},"pg_stats":[{"pgid":"5.f","version":"520'9944","reported_seq":16035,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.194876+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:31:22.194876+0000","last_peered":"2024-04-15T18:31:22.194876+0000","last_clean":"2024-04-15T18:31:22.194876+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:31:22.194876+0000","last_undegraded":"2024-04-15T18:31:22.194876+0000","last_fullsized":"2024-04-15T18:31:22.194876+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123076608,"num_objects":1878,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9944,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"520'9795","reported_seq":10324,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.203767+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:31:22.203767+0000","last_peered":"2024-04-15T18:31:22.203767+0000","last_clean":"2024-04-15T18:31:22.203767+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:31:22.203767+0000","last_undegraded":"2024-04-15T18:31:22.203767+0000","last_fullsized":"2024-04-15T18:31:22.203767+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":9795,"log_dups_size":0,"ondisk_log_size":9795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124321792,"num_objects":1897,"num_object_clones":0,"num_object_copies":5691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9795,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"520'9671","reported_seq":15517,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.092922+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:31:22.092922+0000","last_peered":"2024-04-15T18:31:22.092922+0000","last_clean":"2024-04-15T18:31:22.092922+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:31:22.092922+0000","last_undegraded":"2024-04-15T18:31:22.092922+0000","last_fullsized":"2024-04-15T18:31:22.092922+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9671,"log_dups_size":0,"ondisk_log_size":9671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126746648,"num_objects":1935,"num_object_clones":0,"num_object_copies":5805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9671,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"520'9812","reported_seq":10386,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.312853+0000","last_change":"2024-04-15T18:31:19.169916+0000","last_active":"2024-04-15T18:31:22.312853+0000","last_peered":"2024-04-15T18:31:22.312853+0000","last_clean":"2024-04-15T18:31:22.312853+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:31:22.312853+0000","last_undegraded":"2024-04-15T18:31:22.312853+0000","last_fullsized":"2024-04-15T18:31:22.312853+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'9752","last_scrub_stamp":"2024-04-15T18:31:19.169721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:19.169721+0000","objects_scrubbed":1942,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:19.608911+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"520'9881","reported_seq":15924,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.225161+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:31:22.225161+0000","last_peered":"2024-04-15T18:31:22.225161+0000","last_clean":"2024-04-15T18:31:22.225161+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:31:22.225161+0000","last_undegraded":"2024-04-15T18:31:22.225161+0000","last_fullsized":"2024-04-15T18:31:22.225161+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074304,"num_objects":1939,"num_object_clones":0,"num_object_copies":5817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9881,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"520'9682","reported_seq":21853,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.190288+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:31:22.190288+0000","last_peered":"2024-04-15T18:31:22.190288+0000","last_clean":"2024-04-15T18:31:22.190288+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:31:22.190288+0000","last_undegraded":"2024-04-15T18:31:22.190288+0000","last_fullsized":"2024-04-15T18:31:22.190288+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128712704,"num_objects":1964,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9682,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"520'9762","reported_seq":10212,"reported_epoch":520,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:31:20.923973+0000","last_change":"2024-04-15T18:31:16.132418+0000","last_active":"2024-04-15T18:31:20.923973+0000","last_peered":"2024-04-15T18:31:20.923973+0000","last_clean":"2024-04-15T18:31:20.923973+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:31:20.923973+0000","last_undegraded":"2024-04-15T18:31:20.923973+0000","last_fullsized":"2024-04-15T18:31:20.923973+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":174,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 4s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124518400,"num_objects":1900,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9762,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"520'9815","reported_seq":10151,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:21.940411+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:31:21.940411+0000","last_peered":"2024-04-15T18:31:21.940411+0000","last_clean":"2024-04-15T18:31:21.940411+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:31:21.940411+0000","last_undegraded":"2024-04-15T18:31:21.940411+0000","last_fullsized":"2024-04-15T18:31:21.940411+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9815,"log_dups_size":0,"ondisk_log_size":9815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126681088,"num_objects":1933,"num_object_clones":0,"num_object_copies":5799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9815,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"520'9825","reported_seq":15907,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.253670+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:31:22.253670+0000","last_peered":"2024-04-15T18:31:22.253670+0000","last_clean":"2024-04-15T18:31:22.253670+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:31:22.253670+0000","last_undegraded":"2024-04-15T18:31:22.253670+0000","last_fullsized":"2024-04-15T18:31:22.253670+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":9825,"log_dups_size":0,"ondisk_log_size":9825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118685696,"num_objects":1811,"num_object_clones":0,"num_object_copies":5433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9825,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"520'9581","reported_seq":26906,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.225463+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:31:22.225463+0000","last_peered":"2024-04-15T18:31:22.225463+0000","last_clean":"2024-04-15T18:31:22.225463+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:31:22.225463+0000","last_undegraded":"2024-04-15T18:31:22.225463+0000","last_fullsized":"2024-04-15T18:31:22.225463+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":9581,"log_dups_size":0,"ondisk_log_size":9581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121569280,"num_objects":1855,"num_object_clones":0,"num_object_copies":5565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"520'9497","reported_seq":10024,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.151276+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:31:22.151276+0000","last_peered":"2024-04-15T18:31:22.151276+0000","last_clean":"2024-04-15T18:31:22.151276+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:31:22.151276+0000","last_undegraded":"2024-04-15T18:31:22.151276+0000","last_fullsized":"2024-04-15T18:31:22.151276+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":9497,"log_dups_size":0,"ondisk_log_size":9497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":5451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"520'9949","reported_seq":15965,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:20.367770+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:31:20.367770+0000","last_peered":"2024-04-15T18:31:20.367770+0000","last_clean":"2024-04-15T18:31:20.367770+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:31:20.367770+0000","last_undegraded":"2024-04-15T18:31:20.367770+0000","last_fullsized":"2024-04-15T18:31:20.367770+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125501440,"num_objects":1915,"num_object_clones":0,"num_object_copies":5745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"520'9656","reported_seq":10050,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:20.292277+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:31:20.292277+0000","last_peered":"2024-04-15T18:31:20.292277+0000","last_clean":"2024-04-15T18:31:20.292277+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:31:20.292277+0000","last_undegraded":"2024-04-15T18:31:20.292277+0000","last_fullsized":"2024-04-15T18:31:20.292277+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":9656,"log_dups_size":0,"ondisk_log_size":9656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126222336,"num_objects":1926,"num_object_clones":0,"num_object_copies":5778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9656,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"520'9504","reported_seq":25172,"reported_epoch":520,"state":"active+recovering","last_fresh":"2024-04-15T18:31:22.092317+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:31:22.092317+0000","last_peered":"2024-04-15T18:31:22.092317+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:31:22.092317+0000","last_undegraded":"2024-04-15T18:31:22.092317+0000","last_fullsized":"2024-04-15T18:31:22.092317+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":1131,"num_objects_missing":1131,"num_objects_degraded":0,"num_objects_misplaced":3408,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9504,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3830,"num_bytes_recovered":250871808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,6,7,11","objects":1},{"shards":"3,4,7","objects":1131},{"shards":"3,4,7,11","objects":7}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"520'9893","reported_seq":10391,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.977529+0000","last_change":"2024-04-15T18:29:49.705087+0000","last_active":"2024-04-15T18:31:22.977529+0000","last_peered":"2024-04-15T18:31:22.977529+0000","last_clean":"2024-04-15T18:31:22.977529+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:31:22.977529+0000","last_undegraded":"2024-04-15T18:31:22.977529+0000","last_fullsized":"2024-04-15T18:31:22.977529+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":3404,"log_size":9893,"log_dups_size":0,"ondisk_log_size":9893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125763584,"num_objects":1919,"num_object_clones":0,"num_object_copies":5757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"520'13575","reported_seq":31271,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:22.983632+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:31:22.983632+0000","last_peered":"2024-04-15T18:31:22.983632+0000","last_clean":"2024-04-15T18:31:22.983632+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:31:22.983632+0000","last_undegraded":"2024-04-15T18:31:22.983632+0000","last_fullsized":"2024-04-15T18:31:22.983632+0000","mapping_epoch":440,"log_start":"384'3501","ondisk_log_start":"384'3501","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123011072,"num_objects":1877,"num_object_clones":0,"num_object_copies":5631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9751,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1297,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-15T18:31:13.779214+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:31:13.779214+0000","last_peered":"2024-04-15T18:31:13.779214+0000","last_clean":"2024-04-15T18:31:13.779214+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:31:13.779214+0000","last_undegraded":"2024-04-15T18:31:13.779214+0000","last_fullsized":"2024-04-15T18:31:13.779214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1985544216,"num_objects":30298,"num_object_clones":0,"num_object_copies":90894,"num_objects_missing_on_primary":1131,"num_objects_missing":1131,"num_objects_degraded":0,"num_objects_misplaced":3408,"num_objects_unfound":0,"num_objects_dirty":30298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156018,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40533,"num_bytes_recovered":2651455488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6685339648,"data_stored":6685327432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156341,"ondisk_log_size":156341,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926090,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957788,"kb_used_data":539284,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92787364,"statfs":{"total":95995035648,"available":95014260736,"internally_reserved":0,"allocated":552226816,"data_stored":551494443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187727,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938064,"kb_used_data":613960,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92807088,"statfs":{"total":95995035648,"available":95034458112,"internally_reserved":0,"allocated":628695040,"data_stored":627970311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514772,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703696,"kb_used_data":365704,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93041456,"statfs":{"total":95995035648,"available":95274450944,"internally_reserved":0,"allocated":374480896,"data_stored":373756167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":402460,"kb_used_data":240788,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93342692,"statfs":{"total":95995035648,"available":95582916608,"internally_reserved":0,"allocated":246566912,"data_stored":245837668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024796,"kb_used_data":690964,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92720356,"statfs":{"total":95995035648,"available":94945644544,"internally_reserved":0,"allocated":707547136,"data_stored":706814783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":341834186},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463204,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":757144,"kb_used_data":419152,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92988008,"statfs":{"total":95995035648,"available":95219720192,"internally_reserved":0,"allocated":429211648,"data_stored":428483367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575168,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854224,"kb_used_data":480136,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92890928,"statfs":{"total":95995035648,"available":95120310272,"internally_reserved":0,"allocated":491659264,"data_stored":490934535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292240,"kb_used_data":904264,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92452912,"statfs":{"total":95995035648,"available":94671781888,"internally_reserved":0,"allocated":925966336,"data_stored":925241607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":481552,"kb_used_data":369032,"kb_used_omap":114,"kb_used_meta":112397,"kb_avail":93263600,"statfs":{"total":95995035648,"available":95501926400,"internally_reserved":0,"allocated":377888768,"data_stored":377164039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":115094769},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654044,"kb_used_data":308488,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091108,"statfs":{"total":95995035648,"available":95325294592,"internally_reserved":0,"allocated":315891712,"data_stored":315166983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986476,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1170448,"kb_used_data":800520,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92574704,"statfs":{"total":95995035648,"available":94796496896,"internally_reserved":0,"allocated":819732480,"data_stored":819007751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019412,"kb_used_data":846348,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":92725740,"statfs":{"total":95995035648,"available":94951157760,"internally_reserved":0,"allocated":866660352,"data_stored":865931551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":920387584,"data_stored":920387584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487456768,"data_stored":487456768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374734848,"data_stored":374734848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":815857664,"data_stored":815857664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":864096256,"data_stored":864092184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":423886848,"data_stored":423886848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":700715008,"data_stored":700710936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":243597312,"data_stored":243597312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":370016256,"data_stored":370016256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":624427008,"data_stored":624427008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":546246656,"data_stored":546242584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:31:27.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:31:27.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:27.170 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:27.232 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:27.336 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:27.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:27.390 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:27.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:31:27.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:27.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:27.545 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:27.650 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:27.678 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:27.704 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:27.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:27.841 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:27.863 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:27.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:27.967 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:27.997 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.023 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:28.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:28.153 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.172 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:158211 5.0 e522/358 5:03182d4c:::benchmark_data_smithi159_116570_object65051:head v 522'10211, mlcod=522'10209)", 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:31:28.212757+0000", 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.018079716999999999, 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.018126890999999999, 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:31:28.223 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.212757+0000", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.212757+0000", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.212759+0000", 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.61e-06 2024-04-15T18:31:28.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.212794+0000", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5241999999999999e-05 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.219997+0000", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072029260000000001 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.220009+0000", 2024-04-15T18:31:28.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2e-05 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.220076+0000", 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6872999999999995e-05 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.220111+0000", 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4501000000000001e-05 2024-04-15T18:31:28.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(osd.10.0:23902 5.0 e522/358 5:01bd3f78:::benchmark_data_smithi159_116570_object84724:head v 522'10212, mlcod=522'10209)", 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:31:28.219953+0000", 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.010883742, 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011011396999999999, 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:31:28.227 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.219953+0000", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.219953+0000", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.219955+0000", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.668e-06 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.219968+0000", 2024-04-15T18:31:28.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2379e-05 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223674+0000", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037064910000000001 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223682+0000", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5970000000000003e-06 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223913+0000", 2024-04-15T18:31:28.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023147599999999999 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223935+0000", 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2257999999999999e-05 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(osd.10.0:23903 5.0 e522/358 5:01bda142:::benchmark_data_smithi159_116570_object91231:head v 522'10213, mlcod=522'10209)", 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:31:28.223640+0000", 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0071975629999999997, 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007394121, 2024-04-15T18:31:28.230 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223640+0000", 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223640+0000", 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:31:28.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223641+0000", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4360000000000001e-06 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.223652+0000", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0771000000000001e-05 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.227822+0000", 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041702570000000001 2024-04-15T18:31:28.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.227830+0000", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.644e-06 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.228043+0000", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000213673 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:31:28.228066+0000", 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2688999999999999e-05 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.233 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 3 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:31:28.234 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:28.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:28.303 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.328 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:28.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:28.458 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.477 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:28.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:28.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:28.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:28.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:28 smithi159 ceph-mon[45336]: 5.0 deep-scrub starts 2024-04-15T18:31:28.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:28 smithi159 ceph-mon[45336]: osdmap e522: 12 total, 12 up, 12 in 2024-04-15T18:31:28.606 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:28 smithi159 ceph-mon[45336]: pgmap v1361: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 357 op/s; 3069/86790 objects misplaced (3.536%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:28.608 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.631 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:28 smithi049 ceph-mon[40087]: 5.0 deep-scrub starts 2024-04-15T18:31:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:28 smithi049 ceph-mon[40087]: osdmap e522: 12 total, 12 up, 12 in 2024-04-15T18:31:28.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:28 smithi049 ceph-mon[40087]: pgmap v1361: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 357 op/s; 3069/86790 objects misplaced (3.536%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:28.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:28.766 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:28 smithi082 ceph-mon[44206]: 5.0 deep-scrub starts 2024-04-15T18:31:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:28 smithi082 ceph-mon[44206]: osdmap e522: 12 total, 12 up, 12 in 2024-04-15T18:31:28.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:28 smithi082 ceph-mon[44206]: pgmap v1361: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering, 15 active+clean; 1.8 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 357 op/s; 3069/86790 objects misplaced (3.536%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:28.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:28.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:28.927 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:28.935 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:28.953 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:28.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:29.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:29.098 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:30.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:31:30.649 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:30.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:30.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:30 smithi049 ceph-mon[40087]: pgmap v1362: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 3069/85350 objects misplaced (3.596%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:30.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:30.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:30 smithi082 ceph-mon[44206]: pgmap v1362: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 3069/85350 objects misplaced (3.596%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:30.790 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:30.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:30 smithi159 ceph-mon[45336]: pgmap v1362: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 3069/85350 objects misplaced (3.596%); 1.7 MiB/s, 27 objects/s recovering 2024-04-15T18:31:31.540 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:31.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:31.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:31.684 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:31.810 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:31.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:31.915 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:31.954 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:31.986 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:31.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:32.091 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:32.122 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:32.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-15T18:31:32.141+0000 7f342f7fe700 0 --2- 172.21.15.49:0/1443058717 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343006b050 0x7f343006b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-15T18:31:32.144 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:32.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:32.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:31:32.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:31:32.235 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:32.248 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:32.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:31:32.278 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:32.295 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:31:32.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:32.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:32.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:31:32.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:31:32.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:31:32.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:31:32.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:32.463 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:31:32.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:32 smithi049 ceph-mon[40087]: pgmap v1363: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 276 op/s; 3069/84258 objects misplaced (3.642%); 920 KiB/s, 14 objects/s recovering 2024-04-15T18:31:32.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2634236085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:32.486 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:32.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:31:32.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:31:32.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:31:32.585 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:31:32.586 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:31:32.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:32 smithi159 ceph-mon[45336]: pgmap v1363: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 276 op/s; 3069/84258 objects misplaced (3.642%); 920 KiB/s, 14 objects/s recovering 2024-04-15T18:31:32.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2634236085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:32 smithi082 ceph-mon[44206]: pgmap v1363: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering, 14 active+clean; 1.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 276 op/s; 3069/84258 objects misplaced (3.642%); 920 KiB/s, 14 objects/s recovering 2024-04-15T18:31:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2634236085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: 5.9 scrub ok 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: 5.9 scrub ok 2024-04-15T18:31:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: 5.9 scrub ok 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:34.427 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:31:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:34 smithi049 ceph-mon[40087]: pgmap v1364: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s; 2714/79797 objects misplaced (3.401%); 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:31:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:31:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:34 smithi082 ceph-mon[44206]: pgmap v1364: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s; 2714/79797 objects misplaced (3.401%); 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:31:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:31:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:34 smithi159 ceph-mon[45336]: pgmap v1364: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s; 2714/79797 objects misplaced (3.401%); 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:31:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:31:35.025 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:35.026 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:31:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:35 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:31:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:35 smithi049 ceph-mon[40087]: osdmap e523: 12 total, 12 up, 12 in 2024-04-15T18:31:35.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:35 smithi049 ceph-mon[40087]: from='client.16359 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:35.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:35 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:35.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:31:35.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:35 smithi082 ceph-mon[44206]: osdmap e523: 12 total, 12 up, 12 in 2024-04-15T18:31:35.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:35 smithi082 ceph-mon[44206]: from='client.16359 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:35.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:35 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:35.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3867937282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:31:35.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:35 smithi159 ceph-mon[45336]: osdmap e523: 12 total, 12 up, 12 in 2024-04-15T18:31:35.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:35 smithi159 ceph-mon[45336]: from='client.16359 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:36.455 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1365,"stamp":"2024-04-15T18:31:34.432813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1743585848,"num_objects":26599,"num_object_clones":0,"num_object_copies":79797,"num_objects_missing_on_primary":904,"num_objects_missing":904,"num_objects_degraded":0,"num_objects_misplaced":2714,"num_objects_unfound":0,"num_objects_dirty":26599,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":159900,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40777,"num_bytes_recovered":2669435552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159534,"ondisk_log_size":159534,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9630224,"kb_used_data":5948772,"kb_used_omap":177,"kb_used_meta":3681102,"kb_avail":1115311600,"statfs":{"total":1151940427776,"available":1142079078400,"internally_reserved":0,"allocated":6091542528,"data_stored":6082715128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3769448900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,8,8,11,6,24],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-242548736,"num_objects":-3701,"num_object_clones":0,"num_object_copies":-11103,"num_objects_missing_on_primary":-227,"num_objects_missing":-227,"num_objects_degraded":0,"num_objects_misplaced":-694,"num_objects_unfound":0,"num_objects_dirty":-3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067104"},"pg_stats":[{"pgid":"5.f","version":"522'10167","reported_seq":16260,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.391035+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:31:32.391035+0000","last_peered":"2024-04-15T18:31:32.391035+0000","last_clean":"2024-04-15T18:31:32.391035+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:31:32.391035+0000","last_undegraded":"2024-04-15T18:31:32.391035+0000","last_fullsized":"2024-04-15T18:31:32.391035+0000","mapping_epoch":393,"log_start":"367'100","ondisk_log_start":"367'100","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"522'10032","reported_seq":10565,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.253282+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:31:32.253282+0000","last_peered":"2024-04-15T18:31:32.253282+0000","last_clean":"2024-04-15T18:31:32.253282+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:31:32.253282+0000","last_undegraded":"2024-04-15T18:31:32.253282+0000","last_fullsized":"2024-04-15T18:31:32.253282+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"522'9906","reported_seq":15755,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.094197+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:31:32.094197+0000","last_peered":"2024-04-15T18:31:32.094197+0000","last_clean":"2024-04-15T18:31:32.094197+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:31:32.094197+0000","last_undegraded":"2024-04-15T18:31:32.094197+0000","last_fullsized":"2024-04-15T18:31:32.094197+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9906,"log_dups_size":0,"ondisk_log_size":9906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111345688,"num_objects":1700,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9906,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"522'10049","reported_seq":10627,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.278844+0000","last_change":"2024-04-15T18:31:19.169916+0000","last_active":"2024-04-15T18:31:32.278844+0000","last_peered":"2024-04-15T18:31:32.278844+0000","last_clean":"2024-04-15T18:31:32.278844+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:31:32.278844+0000","last_undegraded":"2024-04-15T18:31:32.278844+0000","last_fullsized":"2024-04-15T18:31:32.278844+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'9752","last_scrub_stamp":"2024-04-15T18:31:19.169721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:19.169721+0000","objects_scrubbed":1942,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:19.608911+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10049,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"522'10101","reported_seq":16149,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.303868+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:31:32.303868+0000","last_peered":"2024-04-15T18:31:32.303868+0000","last_clean":"2024-04-15T18:31:32.303868+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:31:32.303868+0000","last_undegraded":"2024-04-15T18:31:32.303868+0000","last_fullsized":"2024-04-15T18:31:32.303868+0000","mapping_epoch":409,"log_start":"367'100","ondisk_log_start":"367'100","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":5157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"522'9922","reported_seq":22097,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.309532+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:31:32.309532+0000","last_peered":"2024-04-15T18:31:32.309532+0000","last_clean":"2024-04-15T18:31:32.309532+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:31:32.309532+0000","last_undegraded":"2024-04-15T18:31:32.309532+0000","last_fullsized":"2024-04-15T18:31:32.309532+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.369155+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"522'9986","reported_seq":10453,"reported_epoch":522,"state":"active+clean+scrubbing","last_fresh":"2024-04-15T18:31:30.750741+0000","last_change":"2024-04-15T18:31:16.132418+0000","last_active":"2024-04-15T18:31:30.750741+0000","last_peered":"2024-04-15T18:31:30.750741+0000","last_clean":"2024-04-15T18:31:30.750741+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:31:30.750741+0000","last_undegraded":"2024-04-15T18:31:30.750741+0000","last_fullsized":"2024-04-15T18:31:30.750741+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'5575","last_scrub_stamp":"2024-04-15T18:26:10.256241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:10.256241+0000","objects_scrubbed":949,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"scrubbing for 14s","scrub_duration":1829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109838336,"num_objects":1676,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9986,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"522'10038","reported_seq":10378,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.094246+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:31:32.094246+0000","last_peered":"2024-04-15T18:31:32.094246+0000","last_clean":"2024-04-15T18:31:32.094246+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:31:32.094246+0000","last_undegraded":"2024-04-15T18:31:32.094246+0000","last_fullsized":"2024-04-15T18:31:32.094246+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112066560,"num_objects":1710,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"522'10048","reported_seq":16136,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.366963+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:31:32.366963+0000","last_peered":"2024-04-15T18:31:32.366963+0000","last_clean":"2024-04-15T18:31:32.366963+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:31:32.366963+0000","last_undegraded":"2024-04-15T18:31:32.366963+0000","last_fullsized":"2024-04-15T18:31:32.366963+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.038101+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10048,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"522'9803","reported_seq":27132,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.295973+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:31:32.295973+0000","last_peered":"2024-04-15T18:31:32.295973+0000","last_clean":"2024-04-15T18:31:32.295973+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:31:32.295973+0000","last_undegraded":"2024-04-15T18:31:32.295973+0000","last_fullsized":"2024-04-15T18:31:32.295973+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":9803,"log_dups_size":0,"ondisk_log_size":9803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:46.853247+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":4899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9803,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"522'9721","reported_seq":10252,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:32.337998+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:31:32.337998+0000","last_peered":"2024-04-15T18:31:32.337998+0000","last_clean":"2024-04-15T18:31:32.337998+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:31:32.337998+0000","last_undegraded":"2024-04-15T18:31:32.337998+0000","last_fullsized":"2024-04-15T18:31:32.337998+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104398848,"num_objects":1593,"num_object_clones":0,"num_object_copies":4779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"522'10180","reported_seq":16200,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:30.391078+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:31:30.391078+0000","last_peered":"2024-04-15T18:31:30.391078+0000","last_clean":"2024-04-15T18:31:30.391078+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:31:30.391078+0000","last_undegraded":"2024-04-15T18:31:30.391078+0000","last_fullsized":"2024-04-15T18:31:30.391078+0000","mapping_epoch":415,"log_start":"367'100","ondisk_log_start":"367'100","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10180,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"522'9892","reported_seq":10289,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:30.329531+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:31:30.329531+0000","last_peered":"2024-04-15T18:31:30.329531+0000","last_clean":"2024-04-15T18:31:30.329531+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:31:30.329531+0000","last_undegraded":"2024-04-15T18:31:30.329531+0000","last_fullsized":"2024-04-15T18:31:30.329531+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110755840,"num_objects":1690,"num_object_clones":0,"num_object_copies":5070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9892,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"522'9738","reported_seq":26338,"reported_epoch":522,"state":"active+recovering","last_fresh":"2024-04-15T18:31:32.135934+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:31:32.135934+0000","last_peered":"2024-04-15T18:31:32.135934+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:31:32.135934+0000","last_undegraded":"2024-04-15T18:31:32.135934+0000","last_fullsized":"2024-04-15T18:31:32.135934+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":9738,"log_dups_size":0,"ondisk_log_size":9738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":904,"num_objects_missing":904,"num_objects_degraded":0,"num_objects_misplaced":2714,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4064,"num_bytes_recovered":266207232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":711},{"shards":"3,4,7","objects":904},{"shards":"3,4,7,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"522'11221","reported_seq":11745,"reported_epoch":522,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-15T18:31:33.120985+0000","last_change":"2024-04-15T18:31:26.871380+0000","last_active":"2024-04-15T18:31:33.120985+0000","last_peered":"2024-04-15T18:31:33.120985+0000","last_clean":"2024-04-15T18:31:33.120985+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:31:33.120985+0000","last_undegraded":"2024-04-15T18:31:33.120985+0000","last_fullsized":"2024-04-15T18:31:33.120985+0000","mapping_epoch":358,"log_start":"369'1202","ondisk_log_start":"369'1202","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'8414","last_scrub_stamp":"2024-04-15T18:29:49.704796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:29:49.704796+0000","objects_scrubbed":1055,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":1295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10166,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"522'13794","reported_seq":31494,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:33.007206+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:31:33.007206+0000","last_peered":"2024-04-15T18:31:33.007206+0000","last_clean":"2024-04-15T18:31:33.007206+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:31:33.007206+0000","last_undegraded":"2024-04-15T18:31:33.007206+0000","last_fullsized":"2024-04-15T18:31:33.007206+0000","mapping_epoch":440,"log_start":"385'3701","ondisk_log_start":"385'3701","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:39.022081+0000","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108658688,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9970,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1301,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-04-15T18:31:27.339695+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:31:27.339695+0000","last_peered":"2024-04-15T18:31:27.339695+0000","last_clean":"2024-04-15T18:31:27.339695+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:31:27.339695+0000","last_undegraded":"2024-04-15T18:31:27.339695+0000","last_fullsized":"2024-04-15T18:31:27.339695+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1742995480,"num_objects":26597,"num_object_clones":0,"num_object_copies":79791,"num_objects_missing_on_primary":904,"num_objects_missing":904,"num_objects_degraded":0,"num_objects_misplaced":2714,"num_objects_unfound":0,"num_objects_dirty":26597,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159719,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40767,"num_bytes_recovered":2666790912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6025719808,"data_stored":6025707592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159395,"ondisk_log_size":159395,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926092,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890548,"kb_used_data":484524,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92854604,"statfs":{"total":95995035648,"available":95083114496,"internally_reserved":0,"allocated":496152576,"data_stored":495411223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,2,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187729,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851176,"kb_used_data":535392,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92893976,"statfs":{"total":95995035648,"available":95123431424,"internally_reserved":0,"allocated":548241408,"data_stored":547507699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,9,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514774,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666472,"kb_used_data":324320,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93078680,"statfs":{"total":95995035648,"available":95312568320,"internally_reserved":0,"allocated":332103680,"data_stored":331369971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874522,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388844,"kb_used_data":227168,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93356308,"statfs":{"total":95995035648,"available":95596859392,"internally_reserved":0,"allocated":232620032,"data_stored":231886323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":985268,"kb_used_data":647276,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92759884,"statfs":{"total":95995035648,"available":94986121216,"internally_reserved":0,"allocated":662810624,"data_stored":662069291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463206,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717812,"kb_used_data":375592,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93027340,"statfs":{"total":95995035648,"available":95259996160,"internally_reserved":0,"allocated":384606208,"data_stored":383868947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799720,"kb_used_data":421472,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92945432,"statfs":{"total":95995035648,"available":95176122368,"internally_reserved":0,"allocated":431587328,"data_stored":430853619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203112,"kb_used_data":806816,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92542040,"statfs":{"total":95995035648,"available":94763048960,"internally_reserved":0,"allocated":826179584,"data_stored":825445875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193942,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":443496,"kb_used_data":326816,"kb_used_omap":114,"kb_used_meta":116557,"kb_avail":93301656,"statfs":{"total":95995035648,"available":95540895744,"internally_reserved":0,"allocated":334659584,"data_stored":333925875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":119354609},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609072,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654060,"kb_used_data":308512,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091092,"statfs":{"total":95995035648,"available":95325278208,"internally_reserved":0,"allocated":315916288,"data_stored":315182579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986478,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122152,"kb_used_data":743904,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92623000,"statfs":{"total":95995035648,"available":94845952000,"internally_reserved":0,"allocated":761757696,"data_stored":761023987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703373,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907564,"kb_used_data":746980,"kb_used_omap":4,"kb_used_meta":160571,"kb_avail":92837588,"statfs":{"total":95995035648,"available":95065690112,"internally_reserved":0,"allocated":764907520,"data_stored":764169739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":164425409},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":820510720,"data_stored":820510720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":426639360,"data_stored":426639360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329515008,"data_stored":329515008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752943104,"data_stored":752943104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":760549376,"data_stored":760545304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":377815040,"data_stored":377815040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":657723392,"data_stored":657719320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":228392960,"data_stored":228392960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":325582848,"data_stored":325582848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":544800768,"data_stored":544800768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":487329792,"data_stored":487325720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:31:36.456 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.9 is not active+clean 2024-04-15T18:31:36.457 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.9', 'version': "522'9986", 'reported_seq': 10453, 'reported_epoch': 522, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-04-15T18:31:30.750741+0000', 'last_change': '2024-04-15T18:31:16.132418+0000', 'last_active': '2024-04-15T18:31:30.750741+0000', 'last_peered': '2024-04-15T18:31:30.750741+0000', 'last_clean': '2024-04-15T18:31:30.750741+0000', 'last_became_active': '2024-04-15T18:24:30.247530+0000', 'last_became_peered': '2024-04-15T18:24:30.247530+0000', 'last_unstale': '2024-04-15T18:31:30.750741+0000', 'last_undegraded': '2024-04-15T18:31:30.750741+0000', 'last_fullsized': '2024-04-15T18:31:30.750741+0000', 'mapping_epoch': 358, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "403'5575", 'last_scrub_stamp': '2024-04-15T18:26:10.256241+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:26:10.256241+0000', 'objects_scrubbed': 949, 'log_size': 9986, 'log_dups_size': 0, 'ondisk_log_size': 9986, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'scrubbing for 14s', 'scrub_duration': 1829, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109838336, 'num_objects': 1676, 'num_object_clones': 0, 'num_object_copies': 5028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9986, 'num_write_kb': 373184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6, 11], 'acting': [7, 6, 11], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T18:31:36.457 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.1 is not active+clean 2024-04-15T18:31:36.457 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.1', 'version': "522'9738", 'reported_seq': 26338, 'reported_epoch': 522, 'state': 'active+recovering', 'last_fresh': '2024-04-15T18:31:32.135934+0000', 'last_change': '2024-04-15T18:26:49.574312+0000', 'last_active': '2024-04-15T18:31:32.135934+0000', 'last_peered': '2024-04-15T18:31:32.135934+0000', 'last_clean': '2024-04-15T18:26:00.259169+0000', 'last_became_active': '2024-04-15T18:26:48.920249+0000', 'last_became_peered': '2024-04-15T18:26:48.920249+0000', 'last_unstale': '2024-04-15T18:31:32.135934+0000', 'last_undegraded': '2024-04-15T18:31:32.135934+0000', 'last_fullsized': '2024-04-15T18:31:32.135934+0000', 'mapping_epoch': 416, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'objects_scrubbed': 0, 'log_size': 9738, 'log_dups_size': 0, 'ondisk_log_size': 9738, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 105906176, 'num_objects': 1616, 'num_object_clones': 0, 'num_object_copies': 4848, 'num_objects_missing_on_primary': 904, 'num_objects_missing': 904, 'num_objects_degraded': 0, 'num_objects_misplaced': 2714, 'num_objects_unfound': 0, 'num_objects_dirty': 1616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9738, 'num_write_kb': 363328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4064, 'num_bytes_recovered': 266207232, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 0, 6], 'acting': [11, 0, 6], 'avail_no_missing': ['3', '4', '7'], 'object_location_counts': [{'shards': '0,6,11', 'objects': 711}, {'shards': '3,4,7', 'objects': 904}, {'shards': '3,4,7,11', 'objects': 1}], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-15T18:31:36.457 INFO:tasks.cephadm.ceph_manager.ceph:PG 5.0 is not active+clean 2024-04-15T18:31:36.457 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '5.0', 'version': "522'11221", 'reported_seq': 11745, 'reported_epoch': 522, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-04-15T18:31:33.120985+0000', 'last_change': '2024-04-15T18:31:26.871380+0000', 'last_active': '2024-04-15T18:31:33.120985+0000', 'last_peered': '2024-04-15T18:31:33.120985+0000', 'last_clean': '2024-04-15T18:31:33.120985+0000', 'last_became_active': '2024-04-15T18:24:30.246821+0000', 'last_became_peered': '2024-04-15T18:24:30.246821+0000', 'last_unstale': '2024-04-15T18:31:33.120985+0000', 'last_undegraded': '2024-04-15T18:31:33.120985+0000', 'last_fullsized': '2024-04-15T18:31:33.120985+0000', 'mapping_epoch': 358, 'log_start': "369'1202", 'ondisk_log_start': "369'1202", 'created': 358, 'last_epoch_clean': 359, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "491'8414", 'last_scrub_stamp': '2024-04-15T18:29:49.704796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:24:29.224778+0000', 'last_clean_scrub_stamp': '2024-04-15T18:29:49.704796+0000', 'objects_scrubbed': 1055, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'deep scrubbing for 6s', 'scrub_duration': 1295, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 107872256, 'num_objects': 1646, 'num_object_clones': 0, 'num_object_copies': 4938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10166, 'num_write_kb': 377984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 1, 4], 'acting': [10, 1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-15T18:31:36.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:31:36.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:36 smithi049 ceph-mon[40087]: pgmap v1366: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 2714/79797 objects misplaced (3.401%); 948 KiB/s, 14 objects/s recovering 2024-04-15T18:31:36.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:36 smithi082 ceph-mon[44206]: pgmap v1366: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 2714/79797 objects misplaced (3.401%); 948 KiB/s, 14 objects/s recovering 2024-04-15T18:31:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:36 smithi159 ceph-mon[45336]: pgmap v1366: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 2714/79797 objects misplaced (3.401%); 948 KiB/s, 14 objects/s recovering 2024-04-15T18:31:37.154 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:37.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:37.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:37.294 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:37.319 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:37.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:37.423 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:37.455 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:37.476 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:37.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:37.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:37.609 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:37.630 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:37.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:37.733 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:37.768 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:37.796 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:37.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:37.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:37.938 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.039 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:38.142 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:38.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:38.169 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.188 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:38.194 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:31:38.195 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:31:38.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:31:38.330 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.412 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:31:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:31:38.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:38 smithi049 ceph-mon[40087]: osdmap e524: 12 total, 12 up, 12 in 2024-04-15T18:31:38.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:38 smithi049 ceph-mon[40087]: pgmap v1368: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 2334/74529 objects misplaced (3.132%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:38.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:38 smithi049 ceph-mon[40087]: 5.0 deep-scrub ok 2024-04-15T18:31:38.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:38.549 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.581 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:38.584 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:38 smithi082 ceph-mon[44206]: osdmap e524: 12 total, 12 up, 12 in 2024-04-15T18:31:38.584 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:38 smithi082 ceph-mon[44206]: pgmap v1368: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 2334/74529 objects misplaced (3.132%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:38.584 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:38 smithi082 ceph-mon[44206]: 5.0 deep-scrub ok 2024-04-15T18:31:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:38 smithi159 ceph-mon[45336]: osdmap e524: 12 total, 12 up, 12 in 2024-04-15T18:31:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:38 smithi159 ceph-mon[45336]: pgmap v1368: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering, 15 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 2334/74529 objects misplaced (3.132%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:38.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:38 smithi159 ceph-mon[45336]: 5.0 deep-scrub ok 2024-04-15T18:31:38.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:38.722 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.808 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:38.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:38.941 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:38.960 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:38.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:39.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:39.092 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.120 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:39.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:31:39.225 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:39.251 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.269 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:39.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:39.398 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.419 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:39.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:31:39.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:39.554 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.582 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:39.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:31:39.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:39.712 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.733 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:39.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:39.867 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:39.891 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:39.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:39.950 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:39.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:40.028 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:40.049 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:40.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:40.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:40.181 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:40.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:40.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:40.255 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:40.319 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:40.345 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:40.365 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:40.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:40.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:40 smithi049 ceph-mon[40087]: pgmap v1369: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 482 op/s; 2334/72675 objects misplaced (3.212%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:40.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:40.519 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:40.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:40 smithi082 ceph-mon[44206]: pgmap v1369: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 482 op/s; 2334/72675 objects misplaced (3.212%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:40.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:40 smithi159 ceph-mon[45336]: pgmap v1369: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 482 op/s; 2334/72675 objects misplaced (3.212%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:42.274 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:42.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:42.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:42.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:42 smithi049 ceph-mon[40087]: pgmap v1370: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 346 op/s; 2334/71478 objects misplaced (3.265%); 1016 KiB/s, 15 objects/s recovering 2024-04-15T18:31:42.526 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:42.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:42.536 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:42.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:42.667 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:42 smithi082 ceph-mon[44206]: pgmap v1370: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 346 op/s; 2334/71478 objects misplaced (3.265%); 1016 KiB/s, 15 objects/s recovering 2024-04-15T18:31:42.797 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:42.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:42.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:42 smithi159 ceph-mon[45336]: pgmap v1370: 17 pgs: 1 active+recovering, 16 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 346 op/s; 2334/71478 objects misplaced (3.265%); 1016 KiB/s, 15 objects/s recovering 2024-04-15T18:31:42.904 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:42.933 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:42.933 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1650,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":524,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering","count":1}],"num_pgs":17,"num_pools":2,"num_objects":23826,"data_bytes":1561854520,"bytes_used":9387065344,"bytes_avail":1142553362432,"bytes_total":1151940427776,"misplaced_objects":2334,"misplaced_total":71478,"misplaced_ratio":0.032653403844539582,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":1040204,"recovering_keys_per_sec":0,"num_objects_recovered":127,"num_bytes_recovered":8323072,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":36,"modified":"2024-04-15T18:31:01.354261+0000","services":{}},"progress_events":{}} 2024-04-15T18:31:42.933 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:42.958 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:42.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:43.066 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:43.095 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:43.121 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:43.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:43.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:43.263 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:43.621 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:43.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:43 smithi049 ceph-mon[40087]: from='client.16365 172.21.15.49:0/4102046759' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:43.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:43.758 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:43 smithi159 ceph-mon[45336]: from='client.16365 172.21.15.49:0/4102046759' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:43.760 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:43.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:43 smithi082 ceph-mon[44206]: from='client.16365 172.21.15.49:0/4102046759' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:31:43.785 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:43.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:43.894 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:43.930 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:43.951 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:43.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:44.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:44.087 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:44.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:44 smithi049 ceph-mon[40087]: pgmap v1371: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 1893/65730 objects misplaced (2.880%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:44.435 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:31:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2533314753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:44 smithi082 ceph-mon[44206]: pgmap v1371: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 1893/65730 objects misplaced (2.880%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2533314753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:44 smithi159 ceph-mon[45336]: pgmap v1371: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 1893/65730 objects misplaced (2.880%); 1.9 MiB/s, 30 objects/s recovering 2024-04-15T18:31:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2533314753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:31:44.938 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:44.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:45.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:45.098 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:45 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:45.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:45 smithi049 ceph-mon[40087]: osdmap e525: 12 total, 12 up, 12 in 2024-04-15T18:31:45.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:45 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:45.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:45.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:45 smithi082 ceph-mon[44206]: osdmap e525: 12 total, 12 up, 12 in 2024-04-15T18:31:45.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:45 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:45.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:31:45.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:45 smithi159 ceph-mon[45336]: osdmap e525: 12 total, 12 up, 12 in 2024-04-15T18:31:46.128 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:31:46.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:31:46.375 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:46.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:46.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:46.519 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:46.551 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:46.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:46.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:46.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:46 smithi049 ceph-mon[40087]: pgmap v1373: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 1893/65730 objects misplaced (2.880%); 1.1 MiB/s, 18 objects/s recovering 2024-04-15T18:31:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:46 smithi082 ceph-mon[44206]: pgmap v1373: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 1893/65730 objects misplaced (2.880%); 1.1 MiB/s, 18 objects/s recovering 2024-04-15T18:31:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:46 smithi159 ceph-mon[45336]: pgmap v1373: 17 pgs: 1 active+recovering, 16 active+clean; 1.3 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 1893/65730 objects misplaced (2.880%); 1.1 MiB/s, 18 objects/s recovering 2024-04-15T18:31:47.128 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:31:47.129 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:31:47.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:31:48.114 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:31:48.382 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:48.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:48.486 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:48.521 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:48.567 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:48.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:48.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:48 smithi159 ceph-mon[45336]: osdmap e526: 12 total, 12 up, 12 in 2024-04-15T18:31:48.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:48 smithi159 ceph-mon[45336]: pgmap v1375: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 1403/59253 objects misplaced (2.368%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:48.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:48.706 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:48.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:48 smithi049 ceph-mon[40087]: osdmap e526: 12 total, 12 up, 12 in 2024-04-15T18:31:48.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:48 smithi049 ceph-mon[40087]: pgmap v1375: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 1403/59253 objects misplaced (2.368%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:48.733 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:48.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:48.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:48 smithi082 ceph-mon[44206]: osdmap e526: 12 total, 12 up, 12 in 2024-04-15T18:31:48.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:48 smithi082 ceph-mon[44206]: pgmap v1375: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 1403/59253 objects misplaced (2.368%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:48.838 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:48.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:31:48.874 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:48.875 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:48.908 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:31:48.943 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:31:48.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:31:48.997 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:31:49.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:31:49.050 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:31:49.081 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:31:49.091 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:49.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:49.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:31:49.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:31:49.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:31:49.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:49.228 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:31:49.237 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:31:49.240 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:49.276 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:49.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:49.380 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:49.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2021248166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:49.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2021248166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:49.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.416 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:49.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2021248166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:49.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:49.681 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:49.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:49.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:49.820 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:49.954 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:49.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:50.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:50.105 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:50.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:31:50.421 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:31:50.421 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:31:50.421 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:31:50.421 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:31:50.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:50 smithi049 ceph-mon[40087]: pgmap v1376: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 1403/56841 objects misplaced (2.468%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:50.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:50 smithi159 ceph-mon[45336]: pgmap v1376: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 1403/56841 objects misplaced (2.468%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:50 smithi082 ceph-mon[44206]: pgmap v1376: 17 pgs: 1 active+recovering, 16 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 1403/56841 objects misplaced (2.468%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:31:50.957 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:50.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:51.064 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:51.103 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:51.128 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:51.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:31:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:51.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:51.234 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:51.282 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:51.308 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:51.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:51.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:51.444 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:51.486 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:51.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:51.547 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:31:51.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:31:51.596 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:51.645 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:51.855 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:31:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:51.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:31:51.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:51.941 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:31:51.960 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:51.992 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:52.020 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:52.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:52.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:52.163 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:52.193 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:52.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:52.297 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:52.333 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:52.368 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:52.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:52.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:52 smithi049 ceph-mon[40087]: pgmap v1377: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 1403/55041 objects misplaced (2.549%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:52.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:52 smithi049 ceph-mon[40087]: from='client.16377 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:52.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:52.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:52.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:31:52.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:52 smithi159 ceph-mon[45336]: pgmap v1377: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 1403/55041 objects misplaced (2.549%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:52.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:52 smithi159 ceph-mon[45336]: from='client.16377 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:52.711 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:52.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:52.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1377,"stamp":"2024-04-15T18:31:51.370430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1202782776,"num_objects":18347,"num_object_clones":0,"num_object_copies":55041,"num_objects_missing_on_primary":467,"num_objects_missing":467,"num_objects_degraded":0,"num_objects_misplaced":1403,"num_objects_unfound":0,"num_objects_dirty":18347,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":168152,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41214,"num_bytes_recovered":2698074784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8206268,"kb_used_data":4479044,"kb_used_omap":177,"kb_used_meta":3726862,"kb_avail":1116735556,"statfs":{"total":1151940427776,"available":1143537209344,"internally_reserved":0,"allocated":4586541056,"data_stored":4577533056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3816307140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-233504768,"num_objects":-3563,"num_object_clones":0,"num_object_copies":-10689,"num_objects_missing_on_primary":-162,"num_objects_missing":-162,"num_objects_degraded":0,"num_objects_misplaced":-490,"num_objects_unfound":0,"num_objects_dirty":-3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3563,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":10747904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001599"},"pg_stats":[{"pgid":"5.f","version":"526'10642","reported_seq":16745,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.377631+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:31:47.377631+0000","last_peered":"2024-04-15T18:31:47.377631+0000","last_clean":"2024-04-15T18:31:47.377631+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:31:47.377631+0000","last_undegraded":"2024-04-15T18:31:47.377631+0000","last_fullsized":"2024-04-15T18:31:47.377631+0000","mapping_epoch":393,"log_start":"367'600","ondisk_log_start":"367'600","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:48.005464+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10642,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"525'10532","reported_seq":11071,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.291468+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:31:47.291468+0000","last_peered":"2024-04-15T18:31:47.291468+0000","last_clean":"2024-04-15T18:31:47.291468+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:31:47.291468+0000","last_undegraded":"2024-04-15T18:31:47.291468+0000","last_fullsized":"2024-04-15T18:31:47.291468+0000","mapping_epoch":358,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.610159+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"525'10406","reported_seq":16262,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.132914+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:31:47.132914+0000","last_peered":"2024-04-15T18:31:47.132914+0000","last_clean":"2024-04-15T18:31:47.132914+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:31:47.132914+0000","last_undegraded":"2024-04-15T18:31:47.132914+0000","last_fullsized":"2024-04-15T18:31:47.132914+0000","mapping_epoch":392,"log_start":"367'400","ondisk_log_start":"367'400","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577688,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10406,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"525'10527","reported_seq":11111,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.277523+0000","last_change":"2024-04-15T18:31:19.169916+0000","last_active":"2024-04-15T18:31:47.277523+0000","last_peered":"2024-04-15T18:31:47.277523+0000","last_clean":"2024-04-15T18:31:47.277523+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:31:47.277523+0000","last_undegraded":"2024-04-15T18:31:47.277523+0000","last_fullsized":"2024-04-15T18:31:47.277523+0000","mapping_epoch":358,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'9752","last_scrub_stamp":"2024-04-15T18:31:19.169721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:19.169721+0000","objects_scrubbed":1942,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:19.608911+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76480512,"num_objects":1167,"num_object_clones":0,"num_object_copies":3501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"525'10586","reported_seq":16640,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.307718+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:31:47.307718+0000","last_peered":"2024-04-15T18:31:47.307718+0000","last_clean":"2024-04-15T18:31:47.307718+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:31:47.307718+0000","last_undegraded":"2024-04-15T18:31:47.307718+0000","last_fullsized":"2024-04-15T18:31:47.307718+0000","mapping_epoch":409,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:52.528515+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80871424,"num_objects":1234,"num_object_clones":0,"num_object_copies":3702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"525'10450","reported_seq":22630,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.263650+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:31:47.263650+0000","last_peered":"2024-04-15T18:31:47.263650+0000","last_clean":"2024-04-15T18:31:47.263650+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:31:47.263650+0000","last_undegraded":"2024-04-15T18:31:47.263650+0000","last_fullsized":"2024-04-15T18:31:47.263650+0000","mapping_epoch":442,"log_start":"367'400","ondisk_log_start":"367'400","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:47.740708+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"526'10638","reported_seq":11125,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:50.934021+0000","last_change":"2024-04-15T18:31:32.316720+0000","last_active":"2024-04-15T18:31:50.934021+0000","last_peered":"2024-04-15T18:31:50.934021+0000","last_clean":"2024-04-15T18:31:50.934021+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:31:50.934021+0000","last_undegraded":"2024-04-15T18:31:50.934021+0000","last_fullsized":"2024-04-15T18:31:50.934021+0000","mapping_epoch":358,"log_start":"367'600","ondisk_log_start":"367'600","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"522'10017","last_scrub_stamp":"2024-04-15T18:31:32.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:32.316527+0000","objects_scrubbed":1723,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:01.321630+0000","scrub_duration":16183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":1024,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10638,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"525'10520","reported_seq":10865,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.173680+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:31:47.173680+0000","last_peered":"2024-04-15T18:31:47.173680+0000","last_clean":"2024-04-15T18:31:47.173680+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:31:47.173680+0000","last_undegraded":"2024-04-15T18:31:47.173680+0000","last_fullsized":"2024-04-15T18:31:47.173680+0000","mapping_epoch":358,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80478208,"num_objects":1228,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10520,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"526'10527","reported_seq":16622,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.377475+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:31:47.377475+0000","last_peered":"2024-04-15T18:31:47.377475+0000","last_clean":"2024-04-15T18:31:47.377475+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:31:47.377475+0000","last_undegraded":"2024-04-15T18:31:47.377475+0000","last_fullsized":"2024-04-15T18:31:47.377475+0000","mapping_epoch":396,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72679424,"num_objects":1109,"num_object_clones":0,"num_object_copies":3327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"525'10262","reported_seq":27594,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.329755+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:31:47.329755+0000","last_peered":"2024-04-15T18:31:47.329755+0000","last_clean":"2024-04-15T18:31:47.329755+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:31:47.329755+0000","last_undegraded":"2024-04-15T18:31:47.329755+0000","last_fullsized":"2024-04-15T18:31:47.329755+0000","mapping_epoch":437,"log_start":"367'200","ondisk_log_start":"367'200","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76939264,"num_objects":1174,"num_object_clones":0,"num_object_copies":3522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10262,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"525'10176","reported_seq":10713,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.322028+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:31:47.322028+0000","last_peered":"2024-04-15T18:31:47.322028+0000","last_clean":"2024-04-15T18:31:47.322028+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:31:47.322028+0000","last_undegraded":"2024-04-15T18:31:47.322028+0000","last_fullsized":"2024-04-15T18:31:47.322028+0000","mapping_epoch":415,"log_start":"367'100","ondisk_log_start":"367'100","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74579968,"num_objects":1138,"num_object_clones":0,"num_object_copies":3414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"526'10824","reported_seq":16851,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:50.416312+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:31:50.416312+0000","last_peered":"2024-04-15T18:31:50.416312+0000","last_clean":"2024-04-15T18:31:50.416312+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:31:50.416312+0000","last_undegraded":"2024-04-15T18:31:50.416312+0000","last_fullsized":"2024-04-15T18:31:50.416312+0000","mapping_epoch":415,"log_start":"368'800","ondisk_log_start":"368'800","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10824,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"526'10519","reported_seq":10925,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:50.403550+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:31:50.403550+0000","last_peered":"2024-04-15T18:31:50.403550+0000","last_clean":"2024-04-15T18:31:50.403550+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:31:50.403550+0000","last_undegraded":"2024-04-15T18:31:50.403550+0000","last_fullsized":"2024-04-15T18:31:50.403550+0000","mapping_epoch":358,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69664768,"num_objects":1063,"num_object_clones":0,"num_object_copies":3189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"525'10176","reported_seq":28530,"reported_epoch":525,"state":"active+recovering","last_fresh":"2024-04-15T18:31:47.175148+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:31:47.175148+0000","last_peered":"2024-04-15T18:31:47.175148+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:31:47.175148+0000","last_undegraded":"2024-04-15T18:31:47.175148+0000","last_fullsized":"2024-04-15T18:31:47.175148+0000","mapping_epoch":416,"log_start":"367'100","ondisk_log_start":"367'100","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77201408,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":467,"num_objects_missing":467,"num_objects_degraded":0,"num_objects_misplaced":1403,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4501,"num_bytes_recovered":294846464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":467},{"shards":"3,4,7,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"526'12380","reported_seq":12968,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:48.206943+0000","last_change":"2024-04-15T18:31:37.379790+0000","last_active":"2024-04-15T18:31:48.206943+0000","last_peered":"2024-04-15T18:31:48.206943+0000","last_clean":"2024-04-15T18:31:48.206943+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:31:48.206943+0000","last_undegraded":"2024-04-15T18:31:48.206943+0000","last_fullsized":"2024-04-15T18:31:48.206943+0000","mapping_epoch":358,"log_start":"373'2303","ondisk_log_start":"373'2303","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'11977","last_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_deep_scrub":"524'11977","last_deep_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_clean_scrub_stamp":"2024-04-15T18:31:37.379575+0000","objects_scrubbed":1688,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:45.594882+0000","scrub_duration":10507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"526'14318","reported_seq":32026,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:48.119898+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:31:48.119898+0000","last_peered":"2024-04-15T18:31:48.119898+0000","last_clean":"2024-04-15T18:31:48.119898+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:31:48.119898+0000","last_undegraded":"2024-04-15T18:31:48.119898+0000","last_fullsized":"2024-04-15T18:31:48.119898+0000","mapping_epoch":440,"log_start":"388'4301","ondisk_log_start":"388'4301","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74317824,"num_objects":1134,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1309,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-15T18:31:47.348779+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:31:47.348779+0000","last_peered":"2024-04-15T18:31:47.348779+0000","last_clean":"2024-04-15T18:31:47.348779+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:31:47.348779+0000","last_undegraded":"2024-04-15T18:31:47.348779+0000","last_fullsized":"2024-04-15T18:31:47.348779+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:49.569892+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1202192408,"num_objects":18345,"num_object_clones":0,"num_object_copies":55035,"num_objects_missing_on_primary":467,"num_objects_missing":467,"num_objects_degraded":0,"num_objects_misplaced":1403,"num_objects_unfound":0,"num_objects_dirty":18345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167971,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41204,"num_bytes_recovered":2695430144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4504825856,"data_stored":4504813640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160679,"ondisk_log_size":160679,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926095,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780120,"kb_used_data":361616,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92965032,"statfs":{"total":95995035648,"available":95196192768,"internally_reserved":0,"allocated":370294784,"data_stored":369539886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187732,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":702812,"kb_used_data":374544,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93042340,"statfs":{"total":95995035648,"available":95275356160,"internally_reserved":0,"allocated":383533056,"data_stored":382781287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514777,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":576464,"kb_used_data":230148,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93168688,"statfs":{"total":95995035648,"available":95404736512,"internally_reserved":0,"allocated":235671552,"data_stored":234924298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310232,"kb_used_data":140240,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":93434920,"statfs":{"total":95995035648,"available":95677358080,"internally_reserved":0,"allocated":143605760,"data_stored":142853991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168867,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872616,"kb_used_data":526300,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92872536,"statfs":{"total":95995035648,"available":95101476864,"internally_reserved":0,"allocated":538931200,"data_stored":538171807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463210,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595168,"kb_used_data":257176,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93149984,"statfs":{"total":95995035648,"available":95385583616,"internally_reserved":0,"allocated":263348224,"data_stored":262592903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671244,"kb_used_data":297156,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93073908,"statfs":{"total":95995035648,"available":95307681792,"internally_reserved":0,"allocated":304287744,"data_stored":303540490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994960,"kb_used_data":598660,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92750192,"statfs":{"total":95995035648,"available":94976196608,"internally_reserved":0,"allocated":613027840,"data_stored":612280586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353936,"kb_used_data":233092,"kb_used_omap":114,"kb_used_meta":120717,"kb_avail":93391216,"statfs":{"total":95995035648,"available":95632605184,"internally_reserved":0,"allocated":238686208,"data_stored":237938954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":123614449},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609075,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654096,"kb_used_data":308548,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091056,"statfs":{"total":95995035648,"available":95325241344,"internally_reserved":0,"allocated":315953152,"data_stored":315205898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986481,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991052,"kb_used_data":621060,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92754100,"statfs":{"total":95995035648,"available":94980198400,"internally_reserved":0,"allocated":635965440,"data_stored":635218186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703376,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703568,"kb_used_data":530504,"kb_used_omap":4,"kb_used_meta":173051,"kb_avail":93041584,"statfs":{"total":95995035648,"available":95274582016,"internally_reserved":0,"allocated":543236096,"data_stored":542484770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":177204929},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":606797824,"data_stored":606797824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299106304,"data_stored":299106304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":231342080,"data_stored":231342080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":627572736,"data_stored":627572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":537464832,"data_stored":537460760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":253100032,"data_stored":253100032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":529534976,"data_stored":529530904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":138608640,"data_stored":138608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":229769216,"data_stored":229769216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":376504320,"data_stored":376504320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":361107456,"data_stored":361103384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:31:52.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:52 smithi082 ceph-mon[44206]: pgmap v1377: 17 pgs: 1 active+recovering, 16 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 1403/55041 objects misplaced (2.549%); 1.3 MiB/s, 20 objects/s recovering 2024-04-15T18:31:52.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:52 smithi082 ceph-mon[44206]: from='client.16377 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:31:52.815 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:52.867 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:52.904 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:52.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:53.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:53.043 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:53.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:53.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:53.328 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:53.351 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:53.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:53.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:53.496 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:53.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:53.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:53.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:53.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:53.706 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:54.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:54 smithi049 ceph-mon[40087]: pgmap v1378: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 772/47283 objects misplaced (1.633%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:31:54.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:54.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:54.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:54 smithi049 ceph-mon[40087]: osdmap e527: 12 total, 12 up, 12 in 2024-04-15T18:31:54.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:54 smithi082 ceph-mon[44206]: pgmap v1378: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 772/47283 objects misplaced (1.633%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:31:54.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:54.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:54.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:54 smithi082 ceph-mon[44206]: osdmap e527: 12 total, 12 up, 12 in 2024-04-15T18:31:54.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:54 smithi159 ceph-mon[45336]: pgmap v1378: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 772/47283 objects misplaced (1.633%); 2.6 MiB/s, 41 objects/s recovering 2024-04-15T18:31:54.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:54.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:54.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:54 smithi159 ceph-mon[45336]: osdmap e527: 12 total, 12 up, 12 in 2024-04-15T18:31:55.363 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:55.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:55.446 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:31:55.472 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:55.508 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: osdmap e528: 12 total, 12 up, 12 in 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:31:55.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:55.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: osdmap e528: 12 total, 12 up, 12 in 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:31:55.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: osdmap e528: 12 total, 12 up, 12 in 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:31:55.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:31:56.078 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:31:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:56.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:31:56.121 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:31:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:31:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:31:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:31:56.121 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:31:56.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:31:56.217 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:56.246 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:31:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:56.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:31:56.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:31:56.395 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:56.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:56 smithi082 ceph-mon[44206]: pgmap v1381: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 772/47283 objects misplaced (1.633%); 1.6 MiB/s, 26 objects/s recovering 2024-04-15T18:31:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:56 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:56 smithi082 ceph-mon[44206]: osdmap e529: 12 total, 12 up, 12 in 2024-04-15T18:31:56.528 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:31:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:56.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:31:56.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:31:56.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:56 smithi159 ceph-mon[45336]: pgmap v1381: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 772/47283 objects misplaced (1.633%); 1.6 MiB/s, 26 objects/s recovering 2024-04-15T18:31:56.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:56 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:56.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:56.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:56 smithi159 ceph-mon[45336]: osdmap e529: 12 total, 12 up, 12 in 2024-04-15T18:31:56.664 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:56.672 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:56.690 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:31:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:56.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:31:56.714 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:31:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:56 smithi049 ceph-mon[40087]: pgmap v1381: 17 pgs: 1 active+recovering, 16 active+clean; 985 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 772/47283 objects misplaced (1.633%); 1.6 MiB/s, 26 objects/s recovering 2024-04-15T18:31:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:56 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:31:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/339609348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:31:56.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:56 smithi049 ceph-mon[40087]: osdmap e529: 12 total, 12 up, 12 in 2024-04-15T18:31:56.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:31:56.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:31:56.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:56.807 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:31:56.840 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:56.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:31:56.844 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:31:56.862 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:56.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:56.871 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:31:56.902 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:31:56.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:31:56.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:31:56.969 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:31:56.993 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:57.015 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:31:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:57.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:31:57.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:31:57.164 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:57.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:31:57.205 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:31:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:57.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:31:57.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:31:57.254 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:31:57.257 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:31:57.258 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:31:57.258 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:31:57.277 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:31:57.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:31:57.356 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:57.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:31:57.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:31:57.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1259987563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:57.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.456 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:57 smithi082 ceph-mon[44206]: osdmap e530: 12 total, 12 up, 12 in 2024-04-15T18:31:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:31:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1259987563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:57 smithi049 ceph-mon[40087]: osdmap e530: 12 total, 12 up, 12 in 2024-04-15T18:31:57.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1259987563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:57.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:57 smithi159 ceph-mon[45336]: osdmap e530: 12 total, 12 up, 12 in 2024-04-15T18:31:58.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:58 smithi049 ceph-mon[40087]: pgmap v1384: 17 pgs: 1 active+recovering, 16 active+clean; 816 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 88/39156 objects misplaced (0.225%); 3.6 MiB/s, 57 objects/s recovering 2024-04-15T18:31:58.717 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:31:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:58.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.724 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:58 smithi082 ceph-mon[44206]: pgmap v1384: 17 pgs: 1 active+recovering, 16 active+clean; 816 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 88/39156 objects misplaced (0.225%); 3.6 MiB/s, 57 objects/s recovering 2024-04-15T18:31:58.800 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:31:58.822 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:31:58.853 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:58 smithi159 ceph-mon[45336]: pgmap v1384: 17 pgs: 1 active+recovering, 16 active+clean; 816 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 88/39156 objects misplaced (0.225%); 3.6 MiB/s, 57 objects/s recovering 2024-04-15T18:31:58.854 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:58.879 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:31:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:58.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:31:58.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:31:59.016 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:59.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:31:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:59.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:31:59.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:31:59.206 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:59.283 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:31:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:31:59.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:31:59.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:31:59.399 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:31:59.399 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:31:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:31:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:59 smithi082 ceph-mon[44206]: 5.3 scrub starts 2024-04-15T18:31:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:31:59 smithi082 ceph-mon[44206]: 5.a scrub starts 2024-04-15T18:31:59.843 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:31:59.849 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:31:59.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:59 smithi049 ceph-mon[40087]: 5.3 scrub starts 2024-04-15T18:31:59.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:31:59 smithi049 ceph-mon[40087]: 5.a scrub starts 2024-04-15T18:31:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:59 smithi159 ceph-mon[45336]: 5.3 scrub starts 2024-04-15T18:31:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:31:59 smithi159 ceph-mon[45336]: 5.a scrub starts 2024-04-15T18:32:00.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:32:00.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.3 scrub ok 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.a scrub ok 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: pgmap v1385: 17 pgs: 1 active+recovering, 16 active+clean; 760 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 88/36465 objects misplaced (0.241%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.7 scrub starts 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.7 scrub ok 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.d scrub starts 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: from='client.16392 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.d scrub ok 2024-04-15T18:32:00.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:00 smithi049 ceph-mon[40087]: 5.6 scrub starts 2024-04-15T18:32:00.597 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:00.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:00.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.3 scrub ok 2024-04-15T18:32:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.a scrub ok 2024-04-15T18:32:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: pgmap v1385: 17 pgs: 1 active+recovering, 16 active+clean; 760 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 88/36465 objects misplaced (0.241%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:00.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.7 scrub starts 2024-04-15T18:32:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.7 scrub ok 2024-04-15T18:32:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.d scrub starts 2024-04-15T18:32:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: from='client.16392 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.d scrub ok 2024-04-15T18:32:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:00 smithi082 ceph-mon[44206]: 5.6 scrub starts 2024-04-15T18:32:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.3 scrub ok 2024-04-15T18:32:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.a scrub ok 2024-04-15T18:32:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: pgmap v1385: 17 pgs: 1 active+recovering, 16 active+clean; 760 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 88/36465 objects misplaced (0.241%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.7 scrub starts 2024-04-15T18:32:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.7 scrub ok 2024-04-15T18:32:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.d scrub starts 2024-04-15T18:32:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: from='client.16392 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.d scrub ok 2024-04-15T18:32:00.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:00 smithi159 ceph-mon[45336]: 5.6 scrub starts 2024-04-15T18:32:00.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:32:01.338 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:01.508 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:01.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:01.528 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1385,"stamp":"2024-04-15T18:31:59.371986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":796983864,"num_objects":12155,"num_object_clones":0,"num_object_copies":36465,"num_objects_missing_on_primary":29,"num_objects_missing":29,"num_objects_degraded":0,"num_objects_misplaced":88,"num_objects_unfound":0,"num_objects_dirty":12155,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":174344,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41652,"num_bytes_recovered":2726779552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161010,"ondisk_log_size":161010,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7154116,"kb_used_data":3378236,"kb_used_omap":177,"kb_used_meta":3775502,"kb_avail":1117787708,"statfs":{"total":1151940427776,"available":1144614612992,"internally_reserved":0,"allocated":3459313664,"data_stored":3450118284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3866114500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,4,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-236322816,"num_objects":-3606,"num_object_clones":0,"num_object_copies":-10818,"num_objects_missing_on_primary":-225,"num_objects_missing":-225,"num_objects_degraded":0,"num_objects_misplaced":-684,"num_objects_unfound":0,"num_objects_dirty":-3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":15007744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.939830"},"pg_stats":[{"pgid":"5.f","version":"530'11057","reported_seq":17166,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.401818+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:31:57.401818+0000","last_peered":"2024-04-15T18:31:57.401818+0000","last_clean":"2024-04-15T18:31:57.401818+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:31:57.401818+0000","last_undegraded":"2024-04-15T18:31:57.401818+0000","last_fullsized":"2024-04-15T18:31:57.401818+0000","mapping_epoch":393,"log_start":"369'1000","ondisk_log_start":"369'1000","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:58.375315+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"529'10943","reported_seq":11490,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.288185+0000","last_change":"2024-04-15T18:30:33.981565+0000","last_active":"2024-04-15T18:31:57.288185+0000","last_peered":"2024-04-15T18:31:57.288185+0000","last_clean":"2024-04-15T18:31:57.288185+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:31:57.288185+0000","last_undegraded":"2024-04-15T18:31:57.288185+0000","last_fullsized":"2024-04-15T18:31:57.288185+0000","mapping_epoch":358,"log_start":"369'900","ondisk_log_start":"369'900","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'8940","last_scrub_stamp":"2024-04-15T18:30:33.981379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:33.981379+0000","objects_scrubbed":2754,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:59.943656+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10943,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"529'10783","reported_seq":16647,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.193244+0000","last_change":"2024-04-15T18:25:51.447911+0000","last_active":"2024-04-15T18:31:57.193244+0000","last_peered":"2024-04-15T18:31:57.193244+0000","last_clean":"2024-04-15T18:31:57.193244+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:31:57.193244+0000","last_undegraded":"2024-04-15T18:31:57.193244+0000","last_fullsized":"2024-04-15T18:31:57.193244+0000","mapping_epoch":392,"log_start":"367'700","ondisk_log_start":"367'700","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53870616,"num_objects":823,"num_object_clones":0,"num_object_copies":2469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10783,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"529'10938","reported_seq":11530,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.263408+0000","last_change":"2024-04-15T18:31:19.169916+0000","last_active":"2024-04-15T18:31:57.263408+0000","last_peered":"2024-04-15T18:31:57.263408+0000","last_clean":"2024-04-15T18:31:57.263408+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:31:57.263408+0000","last_undegraded":"2024-04-15T18:31:57.263408+0000","last_fullsized":"2024-04-15T18:31:57.263408+0000","mapping_epoch":358,"log_start":"369'900","ondisk_log_start":"369'900","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'9752","last_scrub_stamp":"2024-04-15T18:31:19.169721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:19.169721+0000","objects_scrubbed":1942,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:19.608911+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10938,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"529'11033","reported_seq":17095,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.307671+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:31:57.307671+0000","last_peered":"2024-04-15T18:31:57.307671+0000","last_clean":"2024-04-15T18:31:57.307671+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:31:57.307671+0000","last_undegraded":"2024-04-15T18:31:57.307671+0000","last_fullsized":"2024-04-15T18:31:57.307671+0000","mapping_epoch":409,"log_start":"369'1000","ondisk_log_start":"369'1000","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:59.943689+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51576832,"num_objects":787,"num_object_clones":0,"num_object_copies":2361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"529'10885","reported_seq":23074,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.284832+0000","last_change":"2024-04-15T18:30:30.160151+0000","last_active":"2024-04-15T18:31:57.284832+0000","last_peered":"2024-04-15T18:31:57.284832+0000","last_clean":"2024-04-15T18:31:57.284832+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:31:57.284832+0000","last_undegraded":"2024-04-15T18:31:57.284832+0000","last_fullsized":"2024-04-15T18:31:57.284832+0000","mapping_epoch":442,"log_start":"369'800","ondisk_log_start":"369'800","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8858","last_scrub_stamp":"2024-04-15T18:30:30.159996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:30.159996+0000","objects_scrubbed":2789,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:58.893019+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49872896,"num_objects":761,"num_object_clones":0,"num_object_copies":2283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10885,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"529'10869","reported_seq":11363,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:55.994740+0000","last_change":"2024-04-15T18:31:32.316720+0000","last_active":"2024-04-15T18:31:55.994740+0000","last_peered":"2024-04-15T18:31:55.994740+0000","last_clean":"2024-04-15T18:31:55.994740+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:31:55.994740+0000","last_undegraded":"2024-04-15T18:31:55.994740+0000","last_fullsized":"2024-04-15T18:31:55.994740+0000","mapping_epoch":358,"log_start":"369'800","ondisk_log_start":"369'800","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"522'10017","last_scrub_stamp":"2024-04-15T18:31:32.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:32.316527+0000","objects_scrubbed":1723,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:01.321630+0000","scrub_duration":16183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10869,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"529'10980","reported_seq":11334,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.171642+0000","last_change":"2024-04-15T18:24:30.244918+0000","last_active":"2024-04-15T18:31:57.171642+0000","last_peered":"2024-04-15T18:31:57.171642+0000","last_clean":"2024-04-15T18:31:57.171642+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:31:57.171642+0000","last_undegraded":"2024-04-15T18:31:57.171642+0000","last_fullsized":"2024-04-15T18:31:57.171642+0000","mapping_epoch":358,"log_start":"369'900","ondisk_log_start":"369'900","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":768,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10980,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"530'10949","reported_seq":17053,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.401678+0000","last_change":"2024-04-15T18:30:27.654638+0000","last_active":"2024-04-15T18:31:57.401678+0000","last_peered":"2024-04-15T18:31:57.401678+0000","last_clean":"2024-04-15T18:31:57.401678+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:31:57.401678+0000","last_undegraded":"2024-04-15T18:31:57.401678+0000","last_fullsized":"2024-04-15T18:31:57.401678+0000","mapping_epoch":396,"log_start":"369'900","ondisk_log_start":"369'900","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8905","last_scrub_stamp":"2024-04-15T18:30:27.654496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:27.654496+0000","objects_scrubbed":2731,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:58.374062+0000","scrub_duration":321,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"529'10701","reported_seq":28044,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.263936+0000","last_change":"2024-04-15T18:30:29.145457+0000","last_active":"2024-04-15T18:31:57.263936+0000","last_peered":"2024-04-15T18:31:57.263936+0000","last_clean":"2024-04-15T18:31:57.263936+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:31:57.263936+0000","last_undegraded":"2024-04-15T18:31:57.263936+0000","last_fullsized":"2024-04-15T18:31:57.263936+0000","mapping_epoch":437,"log_start":"367'700","ondisk_log_start":"367'700","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'8699","last_scrub_stamp":"2024-04-15T18:30:29.145304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:29.145304+0000","objects_scrubbed":2737,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:31:58.893039+0000","scrub_duration":309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48168960,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"529'10600","reported_seq":11145,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:57.303189+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:31:57.303189+0000","last_peered":"2024-04-15T18:31:57.303189+0000","last_clean":"2024-04-15T18:31:57.303189+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:31:57.303189+0000","last_undegraded":"2024-04-15T18:31:57.303189+0000","last_fullsized":"2024-04-15T18:31:57.303189+0000","mapping_epoch":415,"log_start":"367'500","ondisk_log_start":"367'500","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:06.584916+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":714,"num_object_clones":0,"num_object_copies":2142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"528'11015","reported_seq":17047,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-15T18:31:55.381055+0000","last_change":"2024-04-15T18:26:42.436279+0000","last_active":"2024-04-15T18:31:55.381055+0000","last_peered":"2024-04-15T18:31:55.381055+0000","last_clean":"2024-04-15T18:31:55.381055+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:31:55.381055+0000","last_undegraded":"2024-04-15T18:31:55.381055+0000","last_fullsized":"2024-04-15T18:31:55.381055+0000","mapping_epoch":415,"log_start":"369'1000","ondisk_log_start":"369'1000","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6851","last_scrub_stamp":"2024-04-15T18:26:42.436175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:42.436175+0000","objects_scrubbed":5013,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55640064,"num_objects":849,"num_object_clones":0,"num_object_copies":2547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11015,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"528'10743","reported_seq":11153,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-15T18:31:55.411585+0000","last_change":"2024-04-15T18:26:44.103319+0000","last_active":"2024-04-15T18:31:55.411585+0000","last_peered":"2024-04-15T18:31:55.411585+0000","last_clean":"2024-04-15T18:31:55.411585+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:31:55.411585+0000","last_undegraded":"2024-04-15T18:31:55.411585+0000","last_fullsized":"2024-04-15T18:31:55.411585+0000","mapping_epoch":358,"log_start":"368'700","ondisk_log_start":"368'700","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'6663","last_scrub_stamp":"2024-04-15T18:26:44.103208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:26:44.103208+0000","objects_scrubbed":4919,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54984704,"num_objects":839,"num_object_clones":0,"num_object_copies":2517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"529'10614","reported_seq":30729,"reported_epoch":529,"state":"active+recovering","last_fresh":"2024-04-15T18:31:57.185110+0000","last_change":"2024-04-15T18:26:49.574312+0000","last_active":"2024-04-15T18:31:57.185110+0000","last_peered":"2024-04-15T18:31:57.185110+0000","last_clean":"2024-04-15T18:26:00.259169+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:31:57.185110+0000","last_undegraded":"2024-04-15T18:31:57.185110+0000","last_fullsized":"2024-04-15T18:31:57.185110+0000","mapping_epoch":416,"log_start":"367'600","ondisk_log_start":"367'600","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:24:29.224778+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":29,"num_objects_missing":29,"num_objects_degraded":0,"num_objects_misplaced":88,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10614,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4939,"num_bytes_recovered":323551232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"0,3,4,7,11","objects":1},{"shards":"0,6,11","objects":710},{"shards":"3,4,7","objects":29}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"530'12803","reported_seq":13399,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-15T18:31:58.203209+0000","last_change":"2024-04-15T18:31:37.379790+0000","last_active":"2024-04-15T18:31:58.203209+0000","last_peered":"2024-04-15T18:31:58.203209+0000","last_clean":"2024-04-15T18:31:58.203209+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:31:58.203209+0000","last_undegraded":"2024-04-15T18:31:58.203209+0000","last_fullsized":"2024-04-15T18:31:58.203209+0000","mapping_epoch":358,"log_start":"375'2703","ondisk_log_start":"375'2703","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'11977","last_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_deep_scrub":"524'11977","last_deep_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_clean_scrub_stamp":"2024-04-15T18:31:37.379575+0000","objects_scrubbed":1688,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:45.594882+0000","scrub_duration":10507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45678592,"num_objects":697,"num_object_clones":0,"num_object_copies":2091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"530'14762","reported_seq":32478,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-04-15T18:31:58.207830+0000","last_change":"2024-04-15T18:30:32.507807+0000","last_active":"2024-04-15T18:31:58.207830+0000","last_peered":"2024-04-15T18:31:58.207830+0000","last_clean":"2024-04-15T18:31:58.207830+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:31:58.207830+0000","last_undegraded":"2024-04-15T18:31:58.207830+0000","last_fullsized":"2024-04-15T18:31:58.207830+0000","mapping_epoch":440,"log_start":"391'4701","ondisk_log_start":"391'4701","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"504'12714","last_scrub_stamp":"2024-04-15T18:30:32.507614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:32.507614+0000","objects_scrubbed":2741,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45219840,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10938,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1315,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-04-15T18:31:55.457127+0000","last_change":"2024-04-15T18:30:46.224610+0000","last_active":"2024-04-15T18:31:55.457127+0000","last_peered":"2024-04-15T18:31:55.457127+0000","last_clean":"2024-04-15T18:31:55.457127+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:31:55.457127+0000","last_undegraded":"2024-04-15T18:31:55.457127+0000","last_fullsized":"2024-04-15T18:31:55.457127+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:30:46.224467+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":796393496,"num_objects":12153,"num_object_clones":0,"num_object_copies":36459,"num_objects_missing_on_primary":29,"num_objects_missing":29,"num_objects_degraded":0,"num_objects_misplaced":88,"num_objects_unfound":0,"num_objects_dirty":12153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174163,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41642,"num_bytes_recovered":2724134912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3354603520,"data_stored":3354591304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926097,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697736,"kb_used_data":251456,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93047416,"statfs":{"total":95995035648,"available":95280553984,"internally_reserved":0,"allocated":257490944,"data_stored":256718186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":456980932},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187734,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556944,"kb_used_data":236992,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93188208,"statfs":{"total":95995035648,"available":95424724992,"internally_reserved":0,"allocated":242679808,"data_stored":241910179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514779,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":481020,"kb_used_data":147188,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93264132,"statfs":{"total":95995035648,"available":95502471168,"internally_reserved":0,"allocated":150720512,"data_stored":149955398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":341834176},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":266356,"kb_used_data":113000,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93478796,"statfs":{"total":95995035648,"available":95722287104,"internally_reserved":0,"allocated":115712000,"data_stored":114951351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":832836,"kb_used_data":486452,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92912316,"statfs":{"total":95995035648,"available":95142211584,"internally_reserved":0,"allocated":498126848,"data_stored":497358575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463211,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559228,"kb_used_data":217072,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93185924,"statfs":{"total":95995035648,"available":95422386176,"internally_reserved":0,"allocated":222281728,"data_stored":221517527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575175,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":589440,"kb_used_data":188980,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93155712,"statfs":{"total":95995035648,"available":95391449088,"internally_reserved":0,"allocated":193515520,"data_stored":192750406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798524,"kb_used_data":406388,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92946628,"statfs":{"total":95995035648,"available":95177347072,"internally_reserved":0,"allocated":416141312,"data_stored":415376198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193947,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":279292,"kb_used_data":148724,"kb_used_omap":114,"kb_used_meta":130445,"kb_avail":93465860,"statfs":{"total":95995035648,"available":95709040640,"internally_reserved":0,"allocated":152293376,"data_stored":151528262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":133575921},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609077,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654148,"kb_used_data":308596,"kb_used_omap":44,"kb_used_meta":345491,"kb_avail":93091004,"statfs":{"total":95995035648,"available":95325188096,"internally_reserved":0,"allocated":316002304,"data_stored":315237190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":353783338},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986483,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893116,"kb_used_data":514868,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92852036,"statfs":{"total":95995035648,"available":95080484864,"internally_reserved":0,"allocated":527224832,"data_stored":526459718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703378,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545476,"kb_used_data":358520,"kb_used_omap":4,"kb_used_meta":186939,"kb_avail":93199676,"statfs":{"total":95995035648,"available":95436468224,"internally_reserved":0,"allocated":367124480,"data_stored":366355294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":191426241},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410189824,"data_stored":410189824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190054400,"data_stored":190054400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146472960,"data_stored":146472960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":313917440,"data_stored":313917440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":515506176,"data_stored":515506176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":345051136,"data_stored":345047064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":210960384,"data_stored":210960384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":486739968,"data_stored":486735896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":110559232,"data_stored":110559232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":145817600,"data_stored":145817600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":230096896,"data_stored":230096896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":249237504,"data_stored":249233432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:32:01.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:32:01.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:01.645 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:01.669 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:01.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:01.775 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: 5.e scrub starts 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: 5.e scrub ok 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: 5.6 scrub ok 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: osdmap e531: 12 total, 12 up, 12 in 2024-04-15T18:32:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:01.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: 5.e scrub starts 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: 5.e scrub ok 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: 5.6 scrub ok 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: osdmap e531: 12 total, 12 up, 12 in 2024-04-15T18:32:01.807 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:01.808 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:01.809 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:01.842 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:01.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:01.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: 5.e scrub starts 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: 5.e scrub ok 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: 5.6 scrub ok 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: osdmap e531: 12 total, 12 up, 12 in 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:02.017 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:02.184 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:02.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:02.288 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:02.318 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:02.352 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:02.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:02.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:02.494 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: pgmap v1387: 17 pgs: 1 active+recovering, 16 active+clean; 714 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 88/34242 objects misplaced (0.257%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: osdmap e532: 12 total, 12 up, 12 in 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: 5.8 scrub starts 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: 5.2 scrub starts 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: 5.8 scrub ok 2024-04-15T18:32:02.504 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:02 smithi082 ceph-mon[44206]: 5.2 scrub ok 2024-04-15T18:32:02.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: pgmap v1387: 17 pgs: 1 active+recovering, 16 active+clean; 714 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 88/34242 objects misplaced (0.257%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: osdmap e532: 12 total, 12 up, 12 in 2024-04-15T18:32:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: 5.8 scrub starts 2024-04-15T18:32:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: 5.2 scrub starts 2024-04-15T18:32:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: 5.8 scrub ok 2024-04-15T18:32:02.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:02 smithi159 ceph-mon[45336]: 5.2 scrub ok 2024-04-15T18:32:02.649 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:02.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:02.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: pgmap v1387: 17 pgs: 1 active+recovering, 16 active+clean; 714 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 88/34242 objects misplaced (0.257%); 2.4 MiB/s, 38 objects/s recovering 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: osdmap e532: 12 total, 12 up, 12 in 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: 5.8 scrub starts 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: 5.2 scrub starts 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: 5.8 scrub ok 2024-04-15T18:32:02.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:02 smithi049 ceph-mon[40087]: 5.2 scrub ok 2024-04-15T18:32:02.791 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:03.158 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:03.159 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:03.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:03.181 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:03.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:03.307 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:03.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:03.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:03.449 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:03.449 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:03.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:03.542 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:03.566 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:03.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:03.670 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:03.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:03.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:03.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:03 smithi049 ceph-mon[40087]: osdmap e533: 12 total, 12 up, 12 in 2024-04-15T18:32:03.701 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:03 smithi049 ceph-mon[40087]: 5.f scrub starts 2024-04-15T18:32:03.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:03.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:03.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:03 smithi082 ceph-mon[44206]: osdmap e533: 12 total, 12 up, 12 in 2024-04-15T18:32:03.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:03 smithi082 ceph-mon[44206]: 5.f scrub starts 2024-04-15T18:32:03.726 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:03.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:03.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:03.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:03 smithi159 ceph-mon[45336]: osdmap e533: 12 total, 12 up, 12 in 2024-04-15T18:32:03.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:03 smithi159 ceph-mon[45336]: 5.f scrub starts 2024-04-15T18:32:04.533 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:32:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: 5.4 scrub starts 2024-04-15T18:32:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: 5.4 scrub ok 2024-04-15T18:32:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: pgmap v1390: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:04.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: 5.f scrub ok 2024-04-15T18:32:04.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: 5.1 scrub starts 2024-04-15T18:32:04.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: 5.1 scrub ok 2024-04-15T18:32:04.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:04.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: 5.4 scrub starts 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: 5.4 scrub ok 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: pgmap v1390: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: 5.f scrub ok 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: 5.1 scrub starts 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: 5.1 scrub ok 2024-04-15T18:32:04.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:04.978 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: 5.4 scrub starts 2024-04-15T18:32:04.978 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: 5.4 scrub ok 2024-04-15T18:32:04.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: pgmap v1390: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:04.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: 5.f scrub ok 2024-04-15T18:32:04.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: 5.1 scrub starts 2024-04-15T18:32:04.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: 5.1 scrub ok 2024-04-15T18:32:04.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:04.993 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:05.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:05.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:32:05.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:05.143 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:05.427 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:05.705 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:05.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:05 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:05.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:05 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:32:05.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:05.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:05 smithi049 ceph-mon[40087]: osdmap e534: 12 total, 12 up, 12 in 2024-04-15T18:32:05.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:05.849 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:05 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:05.849 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:05 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:32:05.849 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:05.849 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:05 smithi082 ceph-mon[44206]: osdmap e534: 12 total, 12 up, 12 in 2024-04-15T18:32:05.853 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:05 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:05 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:32:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3306436572' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:05 smithi159 ceph-mon[45336]: osdmap e534: 12 total, 12 up, 12 in 2024-04-15T18:32:05.955 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:05.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:06.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:06.096 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:06.177 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:06.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:06.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:06.363 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:06.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:06.470 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:06.508 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:06.531 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:06.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:06.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:06.670 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:06.691 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:06 smithi082 ceph-mon[44206]: pgmap v1392: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 393 KiB/s, 6 objects/s recovering 2024-04-15T18:32:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:06 smithi159 ceph-mon[45336]: pgmap v1392: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 393 KiB/s, 6 objects/s recovering 2024-04-15T18:32:06.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:06 smithi049 ceph-mon[40087]: pgmap v1392: 17 pgs: 17 active+clean; 514 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 393 KiB/s, 6 objects/s recovering 2024-04-15T18:32:07.434 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:07.434 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:07.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:32:07.831 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:07.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:07.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:07.998 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:08.024 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:08.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:08.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:08.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:32:08.181 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:08.221 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:08.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:08.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:08.374 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:08.551 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:08.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:08.598 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:08.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:08.681 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:32:08.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:32:08.683 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:08.716 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:08.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:08 smithi049 ceph-mon[40087]: pgmap v1393: 17 pgs: 17 active+clean; 331 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:08 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:32:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/191814784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:08.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:08 smithi159 ceph-mon[45336]: pgmap v1393: 17 pgs: 17 active+clean; 331 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:08.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:08 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:32:08.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/191814784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:08.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.852 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.852 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:08.872 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:08.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:08.882 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:08 smithi082 ceph-mon[44206]: pgmap v1393: 17 pgs: 17 active+clean; 331 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 320 KiB/s, 4 objects/s recovering 2024-04-15T18:32:08.882 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:08 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:32:08.882 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/191814784' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:08.882 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.882 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:08.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:09.004 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:09.026 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:09.029 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:09.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:09.120 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:09.121 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:32:09.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:09.177 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:09.549 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:09.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:09.654 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:09.684 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:09.718 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:09.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:09.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: 5.5 deep-scrub starts 2024-04-15T18:32:09.850 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:32:09.851 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:09 smithi159 ceph-mon[45336]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:09.852 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:09.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:09.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: 5.5 deep-scrub starts 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:32:09.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:09 smithi082 ceph-mon[44206]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: 5.5 deep-scrub starts 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:09.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:32:09.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:32:09.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:32:09.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:09 smithi049 ceph-mon[40087]: from='client.26201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:09.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:10.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:10.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:10.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:10.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:10.168 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:10.190 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:10.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:10.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:10.337 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:10.350 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1393,"stamp":"2024-04-15T18:32:07.373405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":347341368,"num_objects":5294,"num_object_clones":0,"num_object_copies":15882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":181205,"num_write_kb":5964987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41682,"num_bytes_recovered":2728745632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5884968,"kb_used_data":2086760,"kb_used_omap":177,"kb_used_meta":3797710,"kb_avail":1119056856,"statfs":{"total":1151940427776,"available":1145914220544,"internally_reserved":0,"allocated":2136842240,"data_stored":2127391355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3888855492},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-401080320,"num_objects":-6120,"num_object_clones":0,"num_object_copies":-18360,"num_objects_missing_on_primary":-29,"num_objects_missing":-29,"num_objects_degraded":0,"num_objects_misplaced":-88,"num_objects_unfound":0,"num_objects_dirty":-6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001018"},"pg_stats":[{"pgid":"5.f","version":"532'11347","reported_seq":17459,"reported_epoch":532,"state":"active+clean","last_fresh":"2024-04-15T18:32:02.408197+0000","last_change":"2024-04-15T18:30:39.049442+0000","last_active":"2024-04-15T18:32:02.408197+0000","last_peered":"2024-04-15T18:32:02.408197+0000","last_clean":"2024-04-15T18:32:02.408197+0000","last_became_active":"2024-04-15T18:25:52.242855+0000","last_became_peered":"2024-04-15T18:25:52.242855+0000","last_unstale":"2024-04-15T18:32:02.408197+0000","last_undegraded":"2024-04-15T18:32:02.408197+0000","last_fullsized":"2024-04-15T18:32:02.408197+0000","mapping_epoch":393,"log_start":"369'1300","ondisk_log_start":"369'1300","created":358,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'9148","last_scrub_stamp":"2024-04-15T18:30:39.049342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:39.049342+0000","objects_scrubbed":2675,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":184221696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"534'11450","reported_seq":12021,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.343729+0000","last_change":"2024-04-15T18:32:00.252040+0000","last_active":"2024-04-15T18:32:07.343729+0000","last_peered":"2024-04-15T18:32:07.343729+0000","last_clean":"2024-04-15T18:32:07.343729+0000","last_became_active":"2024-04-15T18:24:30.244521+0000","last_became_peered":"2024-04-15T18:24:30.244521+0000","last_unstale":"2024-04-15T18:32:07.343729+0000","last_undegraded":"2024-04-15T18:32:07.343729+0000","last_fullsized":"2024-04-15T18:32:07.343729+0000","mapping_epoch":358,"log_start":"370'1400","ondisk_log_start":"370'1400","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'11093","last_scrub_stamp":"2024-04-15T18:32:00.251863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:00.251863+0000","objects_scrubbed":610,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:14.458929+0000","scrub_duration":280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"534'11305","reported_seq":17192,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.195327+0000","last_change":"2024-04-15T18:31:59.931591+0000","last_active":"2024-04-15T18:32:07.195327+0000","last_peered":"2024-04-15T18:32:07.195327+0000","last_clean":"2024-04-15T18:32:07.195327+0000","last_became_active":"2024-04-15T18:25:51.422406+0000","last_became_peered":"2024-04-15T18:25:51.422406+0000","last_unstale":"2024-04-15T18:32:07.195327+0000","last_undegraded":"2024-04-15T18:32:07.195327+0000","last_fullsized":"2024-04-15T18:32:07.195327+0000","mapping_epoch":392,"log_start":"370'1300","ondisk_log_start":"370'1300","created":358,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'10933","last_scrub_stamp":"2024-04-15T18:31:59.931236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:59.931236+0000","objects_scrubbed":679,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:08.584958+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19660824,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11305,"num_write_kb":371329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":180027392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"534'11435","reported_seq":12037,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.259725+0000","last_change":"2024-04-15T18:31:19.169916+0000","last_active":"2024-04-15T18:32:07.259725+0000","last_peered":"2024-04-15T18:32:07.259725+0000","last_clean":"2024-04-15T18:32:07.259725+0000","last_became_active":"2024-04-15T18:24:30.244536+0000","last_became_peered":"2024-04-15T18:24:30.244536+0000","last_unstale":"2024-04-15T18:32:07.259725+0000","last_undegraded":"2024-04-15T18:32:07.259725+0000","last_fullsized":"2024-04-15T18:32:07.259725+0000","mapping_epoch":358,"log_start":"370'1400","ondisk_log_start":"370'1400","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'9752","last_scrub_stamp":"2024-04-15T18:31:19.169721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:19.169721+0000","objects_scrubbed":1942,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:19.608911+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"534'11553","reported_seq":17625,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.329853+0000","last_change":"2024-04-15T18:30:48.705028+0000","last_active":"2024-04-15T18:32:07.329853+0000","last_peered":"2024-04-15T18:32:07.329853+0000","last_clean":"2024-04-15T18:32:07.329853+0000","last_became_active":"2024-04-15T18:26:27.474662+0000","last_became_peered":"2024-04-15T18:26:27.474662+0000","last_unstale":"2024-04-15T18:32:07.329853+0000","last_undegraded":"2024-04-15T18:32:07.329853+0000","last_fullsized":"2024-04-15T18:32:07.329853+0000","mapping_epoch":409,"log_start":"370'1500","ondisk_log_start":"370'1500","created":358,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'9242","last_scrub_stamp":"2024-04-15T18:30:48.704905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:48.704905+0000","objects_scrubbed":2594,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":267,"num_object_clones":0,"num_object_copies":801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":378240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2749,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"534'11383","reported_seq":23598,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.312949+0000","last_change":"2024-04-15T18:31:59.125509+0000","last_active":"2024-04-15T18:32:07.312949+0000","last_peered":"2024-04-15T18:32:07.312949+0000","last_clean":"2024-04-15T18:32:07.312949+0000","last_became_active":"2024-04-15T18:27:42.524149+0000","last_became_peered":"2024-04-15T18:27:42.524149+0000","last_unstale":"2024-04-15T18:32:07.312949+0000","last_undegraded":"2024-04-15T18:32:07.312949+0000","last_fullsized":"2024-04-15T18:32:07.312949+0000","mapping_epoch":442,"log_start":"370'1300","ondisk_log_start":"370'1300","created":358,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'10970","last_scrub_stamp":"2024-04-15T18:31:59.125307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:59.125307+0000","objects_scrubbed":681,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:04.194282+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11383,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5815,"num_bytes_recovered":379387904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"534'11337","reported_seq":11841,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:05.958200+0000","last_change":"2024-04-15T18:31:32.316720+0000","last_active":"2024-04-15T18:32:05.958200+0000","last_peered":"2024-04-15T18:32:05.958200+0000","last_clean":"2024-04-15T18:32:05.958200+0000","last_became_active":"2024-04-15T18:24:30.247530+0000","last_became_peered":"2024-04-15T18:24:30.247530+0000","last_unstale":"2024-04-15T18:32:05.958200+0000","last_undegraded":"2024-04-15T18:32:05.958200+0000","last_fullsized":"2024-04-15T18:32:05.958200+0000","mapping_epoch":358,"log_start":"370'1300","ondisk_log_start":"370'1300","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"522'10017","last_scrub_stamp":"2024-04-15T18:31:32.316527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:32.316527+0000","objects_scrubbed":1723,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:01.321630+0000","scrub_duration":16183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"534'11466","reported_seq":11843,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.149043+0000","last_change":"2024-04-15T18:32:01.790254+0000","last_active":"2024-04-15T18:32:07.149043+0000","last_peered":"2024-04-15T18:32:07.149043+0000","last_clean":"2024-04-15T18:32:07.149043+0000","last_became_active":"2024-04-15T18:24:30.244549+0000","last_became_peered":"2024-04-15T18:24:30.244549+0000","last_unstale":"2024-04-15T18:32:07.149043+0000","last_undegraded":"2024-04-15T18:32:07.149043+0000","last_fullsized":"2024-04-15T18:32:07.149043+0000","mapping_epoch":358,"log_start":"370'1400","ondisk_log_start":"370'1400","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"532'11202","last_scrub_stamp":"2024-04-15T18:32:01.790021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:01.790021+0000","objects_scrubbed":548,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:11.284693+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":375936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.7","version":"532'11178","reported_seq":17298,"reported_epoch":532,"state":"active+clean","last_fresh":"2024-04-15T18:32:02.389874+0000","last_change":"2024-04-15T18:31:59.693382+0000","last_active":"2024-04-15T18:32:02.389874+0000","last_peered":"2024-04-15T18:32:02.389874+0000","last_clean":"2024-04-15T18:32:02.389874+0000","last_became_active":"2024-04-15T18:25:55.298709+0000","last_became_peered":"2024-04-15T18:25:55.298709+0000","last_unstale":"2024-04-15T18:32:02.389874+0000","last_undegraded":"2024-04-15T18:32:02.389874+0000","last_fullsized":"2024-04-15T18:32:02.389874+0000","mapping_epoch":396,"log_start":"369'1100","ondisk_log_start":"369'1100","created":358,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'11063","last_scrub_stamp":"2024-04-15T18:31:59.693108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:59.693108+0000","objects_scrubbed":577,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:13.010276+0000","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2808,"num_bytes_recovered":184025088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"534'11172","reported_seq":28541,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.241087+0000","last_change":"2024-04-15T18:32:00.336591+0000","last_active":"2024-04-15T18:32:07.241087+0000","last_peered":"2024-04-15T18:32:07.241087+0000","last_clean":"2024-04-15T18:32:07.241087+0000","last_became_active":"2024-04-15T18:27:37.274273+0000","last_became_peered":"2024-04-15T18:27:37.274273+0000","last_unstale":"2024-04-15T18:32:07.241087+0000","last_undegraded":"2024-04-15T18:32:07.241087+0000","last_fullsized":"2024-04-15T18:32:07.241087+0000","mapping_epoch":437,"log_start":"369'1100","ondisk_log_start":"369'1100","created":358,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'10853","last_scrub_stamp":"2024-04-15T18:32:00.335493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:00.335493+0000","objects_scrubbed":592,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:04.599894+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":264,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8425,"num_bytes_recovered":549060608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"534'11055","reported_seq":11611,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.262260+0000","last_change":"2024-04-15T18:30:38.158485+0000","last_active":"2024-04-15T18:32:07.262260+0000","last_peered":"2024-04-15T18:32:07.262260+0000","last_clean":"2024-04-15T18:32:07.262260+0000","last_became_active":"2024-04-15T18:26:40.258384+0000","last_became_peered":"2024-04-15T18:26:40.258384+0000","last_unstale":"2024-04-15T18:32:07.262260+0000","last_undegraded":"2024-04-15T18:32:07.262260+0000","last_fullsized":"2024-04-15T18:32:07.262260+0000","mapping_epoch":415,"log_start":"369'1000","ondisk_log_start":"369'1000","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8679","last_scrub_stamp":"2024-04-15T18:30:38.158360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:30:38.158360+0000","objects_scrubbed":2638,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11055,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"534'11501","reported_seq":17557,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:05.295141+0000","last_change":"2024-04-15T18:31:59.124238+0000","last_active":"2024-04-15T18:32:05.295141+0000","last_peered":"2024-04-15T18:32:05.295141+0000","last_clean":"2024-04-15T18:32:05.295141+0000","last_became_active":"2024-04-15T18:26:40.262957+0000","last_became_peered":"2024-04-15T18:26:40.262957+0000","last_unstale":"2024-04-15T18:32:05.295141+0000","last_undegraded":"2024-04-15T18:32:05.295141+0000","last_fullsized":"2024-04-15T18:32:05.295141+0000","mapping_epoch":415,"log_start":"370'1500","ondisk_log_start":"370'1500","created":358,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"530'11203","last_scrub_stamp":"2024-04-15T18:31:59.123935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:31:59.123935+0000","objects_scrubbed":669,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:27.013390+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11501,"num_write_kb":379648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2801,"num_bytes_recovered":183566336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"534'11235","reported_seq":11670,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:05.416901+0000","last_change":"2024-04-15T18:32:01.831601+0000","last_active":"2024-04-15T18:32:05.416901+0000","last_peered":"2024-04-15T18:32:05.416901+0000","last_clean":"2024-04-15T18:32:05.416901+0000","last_became_active":"2024-04-15T18:24:30.245654+0000","last_became_peered":"2024-04-15T18:24:30.245654+0000","last_unstale":"2024-04-15T18:32:05.416901+0000","last_undegraded":"2024-04-15T18:32:05.416901+0000","last_fullsized":"2024-04-15T18:32:05.416901+0000","mapping_epoch":358,"log_start":"369'1200","ondisk_log_start":"369'1200","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"532'11064","last_scrub_stamp":"2024-04-15T18:32:01.831410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:01.831410+0000","objects_scrubbed":522,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:05.475952+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"534'11074","reported_seq":31333,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:07.184868+0000","last_change":"2024-04-15T18:32:03.807098+0000","last_active":"2024-04-15T18:32:07.184868+0000","last_peered":"2024-04-15T18:32:07.184868+0000","last_clean":"2024-04-15T18:32:07.184868+0000","last_became_active":"2024-04-15T18:26:48.920249+0000","last_became_peered":"2024-04-15T18:26:48.920249+0000","last_unstale":"2024-04-15T18:32:07.184868+0000","last_undegraded":"2024-04-15T18:32:07.184868+0000","last_fullsized":"2024-04-15T18:32:07.184868+0000","mapping_epoch":416,"log_start":"369'1000","ondisk_log_start":"369'1000","created":358,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'10915","last_scrub_stamp":"2024-04-15T18:32:03.806902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:03.806902+0000","objects_scrubbed":442,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:04.457205+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":363328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4969,"num_bytes_recovered":325517312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"533'13039","reported_seq":13641,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-04-15T18:32:03.198335+0000","last_change":"2024-04-15T18:31:37.379790+0000","last_active":"2024-04-15T18:32:03.198335+0000","last_peered":"2024-04-15T18:32:03.198335+0000","last_clean":"2024-04-15T18:32:03.198335+0000","last_became_active":"2024-04-15T18:24:30.246821+0000","last_became_peered":"2024-04-15T18:24:30.246821+0000","last_unstale":"2024-04-15T18:32:03.198335+0000","last_undegraded":"2024-04-15T18:32:03.198335+0000","last_fullsized":"2024-04-15T18:32:03.198335+0000","mapping_epoch":358,"log_start":"380'3003","ondisk_log_start":"380'3003","created":358,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'11977","last_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_deep_scrub":"524'11977","last_deep_scrub_stamp":"2024-04-15T18:31:37.379575+0000","last_clean_scrub_stamp":"2024-04-15T18:31:37.379575+0000","objects_scrubbed":1688,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:32:45.594882+0000","scrub_duration":10507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11351,"num_write_kb":377984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"533'15006","reported_seq":32743,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-04-15T18:32:03.218898+0000","last_change":"2024-04-15T18:32:03.146192+0000","last_active":"2024-04-15T18:32:03.218898+0000","last_peered":"2024-04-15T18:32:03.218898+0000","last_clean":"2024-04-15T18:32:03.218898+0000","last_became_active":"2024-04-15T18:27:40.514374+0000","last_became_peered":"2024-04-15T18:27:40.514374+0000","last_unstale":"2024-04-15T18:32:03.218898+0000","last_undegraded":"2024-04-15T18:32:03.218898+0000","last_fullsized":"2024-04-15T18:32:03.218898+0000","mapping_epoch":440,"log_start":"398'5001","ondisk_log_start":"398'5001","created":358,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'15004","last_scrub_stamp":"2024-04-15T18:32:03.145982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:24:29.224778+0000","last_clean_scrub_stamp":"2024-04-15T18:32:03.145982+0000","objects_scrubbed":453,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:29.400279+0000","scrub_duration":393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":560136192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1334,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-04-15T18:32:04.547759+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:32:04.547759+0000","last_peered":"2024-04-15T18:32:04.547759+0000","last_clean":"2024-04-15T18:32:04.547759+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:32:04.547759+0000","last_undegraded":"2024-04-15T18:32:04.547759+0000","last_fullsized":"2024-04-15T18:32:04.547759+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:00.300641+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":346751000,"num_objects":5292,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181024,"num_write_kb":5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41672,"num_bytes_recovered":2726100992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048536576,"data_stored":2048524360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926099,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559044,"kb_used_data":100284,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":93186108,"statfs":{"total":95995035648,"available":95422574592,"internally_reserved":0,"allocated":102690816,"data_stored":101895633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":469760452},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187735,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":478644,"kb_used_data":154468,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93266508,"statfs":{"total":95995035648,"available":95504904192,"internally_reserved":0,"allocated":158175232,"data_stored":157392208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514781,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396472,"kb_used_data":54320,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93348680,"statfs":{"total":95995035648,"available":95589048320,"internally_reserved":0,"allocated":55623680,"data_stored":54836141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206396,"kb_used_data":48880,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93538756,"statfs":{"total":95995035648,"available":95783686144,"internally_reserved":0,"allocated":50053120,"data_stored":49265581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709316,"kb_used_data":371324,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93035836,"statfs":{"total":95995035648,"available":95268696064,"internally_reserved":0,"allocated":380235776,"data_stored":379440613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463213,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":434628,"kb_used_data":100792,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93310524,"statfs":{"total":95995035648,"available":95549976576,"internally_reserved":0,"allocated":103211008,"data_stored":102419917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575177,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455352,"kb_used_data":67376,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93289800,"statfs":{"total":95995035648,"available":95528755200,"internally_reserved":0,"allocated":68993024,"data_stored":68205485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":558840,"kb_used_data":166704,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93186312,"statfs":{"total":95995035648,"available":95422783488,"internally_reserved":0,"allocated":170704896,"data_stored":169917357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193949,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":192120,"kb_used_data":57392,"kb_used_omap":114,"kb_used_meta":134605,"kb_avail":93553032,"statfs":{"total":95995035648,"available":95798304768,"internally_reserved":0,"allocated":58769408,"data_stored":57981869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":137835761},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609079,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641148,"kb_used_data":291440,"kb_used_omap":44,"kb_used_meta":349651,"kb_avail":93104004,"statfs":{"total":95995035648,"available":95338500096,"internally_reserved":0,"allocated":298434560,"data_stored":297647021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":358043178},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986484,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829864,"kb_used_data":447448,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92915288,"statfs":{"total":95995035648,"available":95145254912,"internally_reserved":0,"allocated":458186752,"data_stored":457408193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703379,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423144,"kb_used_data":226332,"kb_used_omap":4,"kb_used_meta":196667,"kb_avail":93322008,"statfs":{"total":95995035648,"available":95561736192,"internally_reserved":0,"allocated":231763968,"data_stored":230981337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":201387713},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163643392,"data_stored":163643392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66191360,"data_stored":66191360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51118080,"data_stored":51118080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294322176,"data_stored":294322176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":442171392,"data_stored":442171392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":227741696,"data_stored":227737624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":93323264,"data_stored":93323264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":374738944,"data_stored":374734872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45744128,"data_stored":45744128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":48955392,"data_stored":48955392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":149553152,"data_stored":149553152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":91033600,"data_stored":91029528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:32:10.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:32:10.696 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:10.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25126.0:134868 4.a 4:5834a9fd:::benchmark_data_smithi159_102124_object45514:head [delete] snapc 0=[] ondisk+write+known_if_redirected e335)", 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:05.884575+0000", 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 544.82765339100001, 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.294412591, 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.707 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25126", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/8238918", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 134868 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884575+0000", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884575+0000", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884576+0000", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0699999999999999e-06 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884581+0000", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3190000000000002e-06 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884592+0000", 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0349999999999999e-05 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884598+0000", 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2199999999999997e-06 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884643+0000", 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4910000000000002e-05 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884737+0000", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4691999999999994e-05 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.884813+0000", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5227000000000003e-05 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.887846+0000", 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030334120000000001 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.888114+0000", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00026813299999999999 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178925+0000", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29081062699999999 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178958+0000", 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3640000000000003e-05 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178987+0000", 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8991000000000001e-05 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25126.0:134868 4.a e335/284 ondisk, result = 0)", 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:23:05.888406+0000", 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 544.82382251700005, 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29058609800000001, 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.888406+0000", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.888406+0000", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.888406+0000", 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.84e-07 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:05.888408+0000", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2419999999999999e-06 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178872+0000", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29046316 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178878+0000", 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7780000000000002e-06 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178915+0000", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6683000000000001e-05 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178921+0000", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3879999999999997e-06 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:23:06.178992+0000", 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0463000000000006e-05 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.716 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:76857 5.2 5:4bcfb460:::benchmark_data_smithi159_116570_object76856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:54.326879+0000", 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 376.38534973600002, 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31175343, 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 76857 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.717 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326879+0000", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326879+0000", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326886+0000", 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6149999999999999e-06 2024-04-15T18:32:10.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326923+0000", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6974000000000002e-05 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326937+0000", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4029000000000001e-05 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326946+0000", 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4710000000000007e-06 2024-04-15T18:32:10.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.326999+0000", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2991999999999997e-05 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.327151+0000", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015244299999999999 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.327285+0000", 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133931 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.329385+0000", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020996930000000001 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.330085+0000", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00070007200000000004 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.638546+0000", 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30846160500000003 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.638592+0000", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5751999999999998e-05 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:54.638632+0000", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9852999999999999e-05 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93144 5.2 5:476f72f4:::benchmark_data_smithi159_116570_object93143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:15.474165+0000", 2024-04-15T18:32:10.722 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 355.23806287799999, 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.92538430699999996, 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93144 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474165+0000", 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474165+0000", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474168+0000", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3300000000000001e-06 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474218+0000", 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9982999999999999e-05 2024-04-15T18:32:10.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474246+0000", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8070000000000001e-05 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474262+0000", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5752000000000001e-05 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474387+0000", 2024-04-15T18:32:10.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012543499999999999 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474555+0000", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016834799999999999 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.474653+0000", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7934000000000001e-05 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.476996+0000", 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002342874 2024-04-15T18:32:10.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:15.477582+0000", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058600600000000003 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.399425+0000", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.92184240200000001 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.399498+0000", 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3557000000000001e-05 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:16.399550+0000", 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1616000000000002e-05 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(5.1 406/401 [PushOp(5:8204bd65:::benchmark_data_smithi159_116570_object6237:head, version: 367'374, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8204bd65:::benchmark_data_smithi159_116570_object6237:head@367'374, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.731034+0000", 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 352.98119450799999, 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.58602213400000003, 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.728 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.731034+0000", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.731034+0000", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.731034+0000", 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0100000000000005e-07 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.731127+0000", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3091999999999996e-05 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:18.315494+0000", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.58436633000000004 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:18.315507+0000", 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3575e-05 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:18.315579+0000", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1740000000000001e-05 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:18.315589+0000", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4690000000000003e-06 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:18.317056+0000", 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001467327 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:108411 5.3 5:ce4e8f69:::benchmark_data_smithi159_116570_object15251:head [delete] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:56.373541+0000", 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 314.33868717000001, 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.282116325, 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 108411 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373541+0000", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373541+0000", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373542+0000", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0189999999999999e-06 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373547+0000", 2024-04-15T18:32:10.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2859999999999999e-06 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373559+0000", 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1280999999999999e-05 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373565+0000", 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0430000000000004e-06 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373612+0000", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7163999999999998e-05 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373694+0000", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1679000000000002e-05 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.373764+0000", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0322999999999997e-05 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.376533+0000", 2024-04-15T18:32:10.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002769484 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.377102+0000", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00056901100000000002 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655572+0000", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27846981199999998 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655616+0000", 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3890000000000002e-05 2024-04-15T18:32:10.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655657+0000", 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1332999999999999e-05 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:108411 5.3 e423/415 ondisk, result = 0)", 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:56.377605+0000", 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 314.334623561, 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27805730200000001, 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.737 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.377605+0000", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.377605+0000", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.377605+0000", 2024-04-15T18:32:10.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.99e-07 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.377610+0000", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5510000000000001e-06 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655468+0000", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27785848200000002 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655475+0000", 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9890000000000003e-06 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655560+0000", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4512000000000007e-05 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655568+0000", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7749999999999993e-06 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.655662+0000", 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4294000000000002e-05 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:110491 5.2 5:42b1158e:::benchmark_data_smithi159_116570_object17331:head [delete] snapc 0=[] ondisk+write+known_if_redirected e429)", 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:14.095272+0000", 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 296.61695610999999, 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44509258400000001, 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110491 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095272+0000", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095272+0000", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095274+0000", 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.658e-06 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095281+0000", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8140000000000003e-06 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095297+0000", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6164999999999999e-05 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095305+0000", 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9140000000000008e-06 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095371+0000", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5895000000000005e-05 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095517+0000", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000146107 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.095648+0000", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013121899999999999 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.098704+0000", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030558289999999999 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.098860+0000", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015615900000000001 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540293+0000", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44143270899999998 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540331+0000", 2024-04-15T18:32:10.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8220999999999999e-05 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540365+0000", 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3893999999999998e-05 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:110491 5.2 e429/358 ondisk, result = 0)", 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:14.099600+0000", 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 296.61262778499997, 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44076924200000001, 2024-04-15T18:32:10.746 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.099600+0000", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.099600+0000", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.099601+0000", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.68e-07 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.099606+0000", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8319999999999996e-06 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540211+0000", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44060451299999998 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540215+0000", 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.775e-06 2024-04-15T18:32:10.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540284+0000", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8707000000000005e-05 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540289+0000", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.101e-06 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:14.540370+0000", 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0646e-05 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.749 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:113274 5.2 5:4361819b:::benchmark_data_smithi159_116570_object20114:head [delete] snapc 0=[] ondisk+write+known_if_redirected e436)", 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:35.661751+0000", 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 275.05047710500003, 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.42335297199999999, 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113274 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661751+0000", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661751+0000", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661752+0000", 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.11e-06 2024-04-15T18:32:10.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661758+0000", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7629999999999997e-06 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661773+0000", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5227e-05 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661780+0000", 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3300000000000004e-06 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661827+0000", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7309999999999999e-05 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661911+0000", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4436000000000003e-05 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.661977+0000", 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5656999999999998e-05 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.664602+0000", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002624805 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.664985+0000", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00038347300000000001 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085043+0000", 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.42005722299999998 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085076+0000", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3865999999999997e-05 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085104+0000", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7772000000000001e-05 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:113274 5.2 e436/358 ondisk, result = 0)", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:35.665606+0000", 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 275.04662197099998, 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41950082, 2024-04-15T18:32:10.755 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.665606+0000", 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.665606+0000", 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.665607+0000", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5199999999999996e-07 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:35.665611+0000", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3579999999999996e-06 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.084991+0000", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41938002299999999 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.084998+0000", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9090000000000003e-06 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085035+0000", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6659000000000003e-05 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085039+0000", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.642e-06 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:36.085107+0000", 2024-04-15T18:32:10.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8577000000000004e-05 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:114037 5.2 5:47780400:::benchmark_data_smithi159_116570_object20877:head [delete] snapc 0=[] ondisk+write+known_if_redirected e441)", 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:40.900311+0000", 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 269.81191737300003, 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36920837000000001, 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.759 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 114037 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900311+0000", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900311+0000", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900312+0000", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5099999999999999e-07 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900315+0000", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5650000000000002e-06 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900324+0000", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3019999999999995e-06 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900331+0000", 2024-04-15T18:32:10.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1910000000000003e-06 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900370+0000", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9088999999999998e-05 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900462+0000", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2571000000000005e-05 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.900525+0000", 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2836000000000006e-05 2024-04-15T18:32:10.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.901905+0000", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013794 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:40.903005+0000", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011002270000000001 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:41.269310+0000", 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36630492799999997 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:41.269397+0000", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7511999999999998e-05 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:41.269519+0000", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012199800000000001 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:115835 5.3 5:ccf50274:::benchmark_data_smithi159_116570_object22675:head [delete] snapc 0=[] ondisk+write+known_if_redirected e450)", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:57.440166+0000", 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 253.27206182, 2024-04-15T18:32:10.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29980196799999997, 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 115835 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440166+0000", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440166+0000", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440168+0000", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.167e-06 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440174+0000", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0789999999999997e-06 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440187+0000", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2836e-05 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440196+0000", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0890000000000006e-06 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440272+0000", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6533999999999996e-05 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440361+0000", 2024-04-15T18:32:10.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8383999999999999e-05 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.440429+0000", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8930000000000006e-05 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.443145+0000", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027157660000000001 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.443209+0000", 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3965000000000002e-05 2024-04-15T18:32:10.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739895+0000", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29668556499999998 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739929+0000", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4031e-05 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739968+0000", 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9622000000000001e-05 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.769 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:115835 5.3 e450/415 ondisk, result = 0)", 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:57.444427+0000", 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 253.26780091000001, 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29554561299999998, 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.444427+0000", 2024-04-15T18:32:10.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.444427+0000", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.444428+0000", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3500000000000005e-07 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.444432+0000", 2024-04-15T18:32:10.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8229999999999997e-06 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739843+0000", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.295410438 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739848+0000", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5300000000000004e-06 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739885+0000", 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6582999999999999e-05 2024-04-15T18:32:10.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739891+0000", 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3149999999999997e-06 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:57.739973+0000", 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1989000000000004e-05 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:118580 5.3 5:c1d7d9b0:::benchmark_data_smithi159_116570_object25420:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:32:10.773 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:18.717271+0000", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 231.99495759199999, 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48125952399999999, 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 118580 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717271+0000", 2024-04-15T18:32:10.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717271+0000", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717272+0000", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4419999999999999e-06 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717280+0000", 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6990000000000003e-06 2024-04-15T18:32:10.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717296+0000", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6022999999999999e-05 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717304+0000", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4849999999999992e-06 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717367+0000", 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2957e-05 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717525+0000", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000157978 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717656+0000", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013097099999999999 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.720756+0000", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030993689999999998 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.720941+0000", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000184897 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198394+0000", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47745376099999998 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198475+0000", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0785999999999995e-05 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198530+0000", 2024-04-15T18:32:10.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5155999999999998e-05 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:126128 5.2 5:429d5b8c:::benchmark_data_smithi159_116570_object32968:head [delete] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:08.156937+0000", 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 182.555291544, 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29333488499999999, 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 126128 2024-04-15T18:32:10.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156937+0000", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156937+0000", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156938+0000", 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7060000000000001e-06 2024-04-15T18:32:10.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156946+0000", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9899999999999997e-06 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156962+0000", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5441e-05 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.156972+0000", 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0335e-05 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.157023+0000", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0827e-05 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.157118+0000", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5200999999999999e-05 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,10", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.157195+0000", 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7008999999999994e-05 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.160233+0000", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030377759999999998 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.160550+0000", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031740300000000001 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.450142+0000", 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.289591392 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.783 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.450201+0000", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9215999999999997e-05 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:08.450272+0000", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0588999999999998e-05 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:126557 5.9 e480/358 5:94639ee4:::benchmark_data_smithi159_116570_object33397:head v 480'7881, mlcod=480'7878)", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:10.863810+0000", 2024-04-15T18:32:10.784 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 179.84841827299999, 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27601089000000001, 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:10.863810+0000", 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:10.863810+0000", 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:10.863812+0000", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.57e-06 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:10.863820+0000", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8580000000000006e-06 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.135818+0000", 2024-04-15T18:32:10.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27199793500000002 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.135827+0000", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9579999999999996e-06 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.135894+0000", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6272000000000005e-05 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.135921+0000", 2024-04-15T18:32:10.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7163000000000001e-05 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.139781+0000", 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003859786 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:11.139821+0000", 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0348000000000001e-05 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:129236 5.3 5:ce6b8309:::benchmark_data_smithi159_116570_object36076:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:26.398597+0000", 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 164.31363081000001, 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.447619288, 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 129236 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398597+0000", 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398597+0000", 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398599+0000", 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6080000000000001e-06 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398606+0000", 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8249999999999999e-06 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398621+0000", 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5505000000000001e-05 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398632+0000", 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0386e-05 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398699+0000", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6878999999999994e-05 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.398704+0000", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.699e-06 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399398+0000", 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00069373299999999998 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.795 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399497+0000", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9180999999999998e-05 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399574+0000", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7146999999999998e-05 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.402654+0000", 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003079371 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.403328+0000", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00067455399999999995 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.846143+0000", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44281440900000002 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.846180+0000", 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7218000000000002e-05 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.846217+0000", 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6773000000000002e-05 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:132875 5.3 5:cec657d8:::benchmark_data_smithi159_116570_object39715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e491)", 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:48.721046+0000", 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 141.99118209700001, 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36387661900000001, 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.798 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 132875 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721046+0000", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721046+0000", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721048+0000", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3689999999999999e-06 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721053+0000", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5759999999999996e-06 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721067+0000", 2024-04-15T18:32:10.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3611999999999999e-05 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721076+0000", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2410000000000001e-06 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721124+0000", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7642000000000002e-05 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721216+0000", 2024-04-15T18:32:10.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2424999999999997e-05 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.721289+0000", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2739999999999998e-05 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.724379+0000", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003090003 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:48.795722+0000", 2024-04-15T18:32:10.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.071343308999999994 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:49.084856+0000", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28913385800000002 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:49.084890+0000", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3905999999999997e-05 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:49.084923+0000", 2024-04-15T18:32:10.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2938000000000002e-05 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:157620 5.3 5:cebefff7:::benchmark_data_smithi159_116570_object64460:head [delete] snapc 0=[] ondisk+write+known_if_redirected e521)", 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:31:26.164988+0000", 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.547240659000003, 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.243586786, 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:32:10.804 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 157620 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.164988+0000", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.164988+0000", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:32:10.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.164989+0000", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1149999999999999e-06 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.164994+0000", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9470000000000002e-06 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.165006+0000", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2670000000000001e-05 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:32:10.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.165031+0000", 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4740999999999999e-05 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.165070+0000", 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9356000000000001e-05 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.165154+0000", 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3883999999999998e-05 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.165229+0000", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4552e-05 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.169775+0000", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045457620000000001 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.169946+0000", 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017146899999999999 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.808 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.408496+0000", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23854998299999999 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.408528+0000", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2005999999999998e-05 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:31:26.408574+0000", 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6301000000000003e-05 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.809 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:32:10.810 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:32:10.810 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:32:10.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:10.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:10 smithi049 ceph-mon[40087]: pgmap v1394: 17 pgs: 17 active+clean; 274 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s; 244 KiB/s, 3 objects/s recovering 2024-04-15T18:32:10.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:10 smithi049 ceph-mon[40087]: osdmap e535: 12 total, 12 up, 12 in 2024-04-15T18:32:10.857 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:10.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:10 smithi159 ceph-mon[45336]: pgmap v1394: 17 pgs: 17 active+clean; 274 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s; 244 KiB/s, 3 objects/s recovering 2024-04-15T18:32:10.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:10 smithi159 ceph-mon[45336]: osdmap e535: 12 total, 12 up, 12 in 2024-04-15T18:32:11.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:10 smithi082 ceph-mon[44206]: pgmap v1394: 17 pgs: 17 active+clean; 274 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 894 op/s; 244 KiB/s, 3 objects/s recovering 2024-04-15T18:32:11.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:10 smithi082 ceph-mon[44206]: osdmap e535: 12 total, 12 up, 12 in 2024-04-15T18:32:11.218 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:11.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:11.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:11.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:11.387 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:11.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:11.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:32:11.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:11.563 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:11.591 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:11.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:11.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:11.730 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:11.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:11 smithi082 ceph-mon[44206]: 5.5 deep-scrub ok 2024-04-15T18:32:11.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:11 smithi082 ceph-mon[44206]: 5.b scrub starts 2024-04-15T18:32:11.735 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:11 smithi082 ceph-mon[44206]: 5.b scrub ok 2024-04-15T18:32:11.736 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:11.762 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:11.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:11.847 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:11.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:11.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:11 smithi159 ceph-mon[45336]: 5.5 deep-scrub ok 2024-04-15T18:32:11.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:11 smithi159 ceph-mon[45336]: 5.b scrub starts 2024-04-15T18:32:11.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:11 smithi159 ceph-mon[45336]: 5.b scrub ok 2024-04-15T18:32:11.904 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:11.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:11 smithi049 ceph-mon[40087]: 5.5 deep-scrub ok 2024-04-15T18:32:11.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:11 smithi049 ceph-mon[40087]: 5.b scrub starts 2024-04-15T18:32:11.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:11 smithi049 ceph-mon[40087]: 5.b scrub ok 2024-04-15T18:32:12.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:32:12.108 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:12.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:12.214 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:12.241 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:12.267 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:12.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:12.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:12.417 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:12.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:12 smithi049 ceph-mon[40087]: pgmap v1396: 17 pgs: 17 active+clean; 231 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s 2024-04-15T18:32:12.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:12 smithi049 ceph-mon[40087]: osdmap e536: 12 total, 12 up, 12 in 2024-04-15T18:32:12.604 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:12 smithi159 ceph-mon[45336]: pgmap v1396: 17 pgs: 17 active+clean; 231 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s 2024-04-15T18:32:12.604 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:12 smithi159 ceph-mon[45336]: osdmap e536: 12 total, 12 up, 12 in 2024-04-15T18:32:12.678 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:12.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:12 smithi082 ceph-mon[44206]: pgmap v1396: 17 pgs: 17 active+clean; 231 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s 2024-04-15T18:32:12.681 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:12 smithi082 ceph-mon[44206]: osdmap e536: 12 total, 12 up, 12 in 2024-04-15T18:32:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:12.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:12.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:12.821 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:13.280 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:13.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:13.384 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:13.411 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:13.429 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:13.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:13.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:13.560 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:13.612 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:13.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:13.617 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:32:13.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3961440877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:13 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:13.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:13.836 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:32:13.836 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:32:13.836 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:32:13.836 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:32:13.855 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3961440877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.855 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:13 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.895 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:13.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3961440877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:13 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:13.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:13.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:14.028 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:14.066 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:14.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:14.094 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 93157 objects 2024-04-15T18:32:14.095 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :355.321 2024-04-15T18:32:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:14.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:14.198 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:14.324 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:14.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:14.406 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:14.406 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1681,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":537,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":817,"data_bytes":53936696,"bytes_used":5073788928,"bytes_avail":1146866638848,"bytes_total":1151940427776,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":924},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":36,"modified":"2024-04-15T18:31:01.354261+0000","services":{}},"progress_events":{}} 2024-04-15T18:32:14.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:14.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:32:14.474 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:14.732 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:14 smithi082 ceph-mon[44206]: pgmap v1398: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-04-15T18:32:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:14 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:14 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:14 smithi082 ceph-mon[44206]: osdmap e537: 12 total, 12 up, 12 in 2024-04-15T18:32:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1969007876' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:14 smithi159 ceph-mon[45336]: pgmap v1398: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-04-15T18:32:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:14 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:14 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:14 smithi159 ceph-mon[45336]: osdmap e537: 12 total, 12 up, 12 in 2024-04-15T18:32:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1969007876' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:14.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:14 smithi049 ceph-mon[40087]: pgmap v1398: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 924 op/s 2024-04-15T18:32:14.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:14 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:14.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:14 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:14.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:14 smithi049 ceph-mon[40087]: osdmap e537: 12 total, 12 up, 12 in 2024-04-15T18:32:14.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1969007876' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:15.360 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:15.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:15.464 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:15.497 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:15.518 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:15.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:15.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:15.652 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:15.672 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:15.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:15.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:15.813 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:15.834 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:15.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:15.939 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:15.967 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:15.987 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:15.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:16.091 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:16.118 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:16.137 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:16.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:16.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:16.313 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:16.342 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-15T18:32:16.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:32:16.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:16.446 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:16.465 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:16.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:16.479 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:32:16.479 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:32:16.479 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:32:16.479 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:32:16.479 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:32:16.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:16.598 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:16.620 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:16.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:16.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:16.752 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:16.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:16 smithi082 ceph-mon[44206]: pgmap v1400: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s 2024-04-15T18:32:16.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:16 smithi159 ceph-mon[45336]: pgmap v1400: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s 2024-04-15T18:32:16.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:16 smithi049 ceph-mon[40087]: pgmap v1400: 17 pgs: 17 active+clean; 51 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s 2024-04-15T18:32:17.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:17.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:17.105 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:17.153 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:32:17.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:17.237 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:17.258 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:17.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:17.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:32:17.361 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:17.390 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:17.410 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:17.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:17.459 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:32:17.514 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:17.541 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:17.664 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:17.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:17.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:17.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:17.816 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:32:17.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:32:17.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:32:17.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:32:17.912 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:32:17.938 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:32:17.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:32:17.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:32:18.012 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:32:18.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:32:18.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:32:18.086 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:32:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:32:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:32:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:32:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:32:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:32:18.203 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-15T18:32:18.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-15T18:32:18.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: osdmap e538: 12 total, 12 up, 12 in 2024-04-15T18:32:18.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: pgmap v1402: 17 pgs: 17 active+clean; 4.2 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s 2024-04-15T18:32:18.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/699642257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:18.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: osdmap e538: 12 total, 12 up, 12 in 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: pgmap v1402: 17 pgs: 17 active+clean; 4.2 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/699642257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.567 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: osdmap e538: 12 total, 12 up, 12 in 2024-04-15T18:32:18.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: pgmap v1402: 17 pgs: 17 active+clean; 4.2 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s 2024-04-15T18:32:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/699642257' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:18.851 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:18.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:18.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:18.986 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:19.134 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:19.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:19.156 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:19.156 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:19.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:19.267 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:19.364 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:19.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:19.447 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:19.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:19.482 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.492 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.497 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:19.746 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:19.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:19.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:32:19.850 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:19.876 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:19.894 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:19.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:19.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:20.025 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:20.025 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:20.043 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:20.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:20.147 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:20.173 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:20.192 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:20.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:20.295 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:20.321 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:20.339 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:20.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:20.443 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:20.472 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:20.492 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:20.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:20.516 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:20 smithi159 ceph-mon[45336]: pgmap v1403: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-04-15T18:32:20.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:20.623 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:20.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:20 smithi049 ceph-mon[40087]: pgmap v1403: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-04-15T18:32:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:20 smithi082 ceph-mon[44206]: pgmap v1403: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-04-15T18:32:20.821 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:20.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:20.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:32:20.924 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:20.951 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:21.299 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:21.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:21.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:21.432 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:21.796 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:21.796 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:21.981 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:21.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:22.085 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:22.117 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:22.140 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:22.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:22.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:22.292 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:22.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:22.396 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:22.428 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:22.440 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:32:22.502 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:22.505 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:22 smithi159 ceph-mon[45336]: pgmap v1404: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 101 op/s 2024-04-15T18:32:22.505 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:32:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:22.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:22.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:32:22.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:22.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:22.660 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:22.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:22.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:22 smithi049 ceph-mon[40087]: pgmap v1404: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 101 op/s 2024-04-15T18:32:22.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:32:22.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:22 smithi082 ceph-mon[44206]: pgmap v1404: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 101 op/s 2024-04-15T18:32:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:32:22.791 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:22.810 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:22.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:22.836 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:22.913 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:22.941 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:23.090 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:23.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:23.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:23.225 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:23.331 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:23.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:23.453 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' removed 2024-04-15T18:32:23.483 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:23.512 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:23.610 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:23.612 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:32:23.612 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:23 smithi159 ceph-mon[45336]: osdmap e539: 12 total, 12 up, 12 in 2024-04-15T18:32:23.612 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:23 smithi159 ceph-mon[45336]: from='client.16428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:23.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:32:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:23 smithi049 ceph-mon[40087]: osdmap e539: 12 total, 12 up, 12 in 2024-04-15T18:32:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:23 smithi049 ceph-mon[40087]: from='client.16428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:23.713 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:23.740 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1541577934' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:32:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:23 smithi082 ceph-mon[44206]: osdmap e539: 12 total, 12 up, 12 in 2024-04-15T18:32:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:23 smithi082 ceph-mon[44206]: from='client.16428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:23.792 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:23.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:23.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:23.934 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:24.006 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:32:24.007 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:32:24.007 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-15T18:32:24.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-04-15T18:32:24.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:24.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:24.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:24.263 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:24.441 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:32:24.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:24 smithi049 ceph-mon[40087]: pgmap v1406: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:24.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:24.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1405,"stamp":"2024-04-15T18:32:22.435129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4712012,"kb_used_data":843200,"kb_used_omap":177,"kb_used_meta":3868494,"kb_avail":1120229812,"statfs":{"total":1151940427776,"available":1147115327488,"internally_reserved":0,"allocated":863436800,"data_stored":853768566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3961338308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-53346328,"num_objects":-815,"num_object_clones":0,"num_object_copies":-2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-815,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-185501,"num_write_kb":-5962049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-41672,"num_bytes_recovered":-2726100992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.818514"},"pg_stats":[{"pgid":"1.0","version":"509'139","reported_seq":1342,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-04-15T18:32:18.134647+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:32:18.134647+0000","last_peered":"2024-04-15T18:32:18.134647+0000","last_clean":"2024-04-15T18:32:18.134647+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:32:18.134647+0000","last_undegraded":"2024-04-15T18:32:18.134647+0000","last_fullsized":"2024-04-15T18:32:18.134647+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:00.300641+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926102,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448948,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93296204,"statfs":{"total":95995035648,"available":95535312896,"internally_reserved":0,"allocated":2727936,"data_stored":1918815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":456980932},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187738,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317920,"kb_used_data":2132,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93427232,"statfs":{"total":95995035648,"available":95669485568,"internally_reserved":0,"allocated":2183168,"data_stored":1382134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514784,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344300,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93400852,"statfs":{"total":95995035648,"available":95642472448,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163820,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93581332,"statfs":{"total":95995035648,"available":95827283968,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618676,"kb_used_data":276520,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93126476,"statfs":{"total":95995035648,"available":95361511424,"internally_reserved":0,"allocated":283156480,"data_stored":282347379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463216,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":358708,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93386444,"statfs":{"total":95995035648,"available":95627718656,"internally_reserved":0,"allocated":2727936,"data_stored":1918835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398444,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93346708,"statfs":{"total":95995035648,"available":95587028992,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438700,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93306452,"statfs":{"total":95995035648,"available":95545806848,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193952,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141036,"kb_used_data":2144,"kb_used_omap":114,"kb_used_meta":138765,"kb_avail":93604116,"statfs":{"total":95995035648,"available":95850614784,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":142095601},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609082,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612396,"kb_used_data":262688,"kb_used_omap":44,"kb_used_meta":349651,"kb_avail":93132756,"statfs":{"total":95995035648,"available":95367942144,"internally_reserved":0,"allocated":268992512,"data_stored":268186963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":358043178},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657756,"kb_used_data":283668,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93087396,"statfs":{"total":95995035648,"available":95321493504,"internally_reserved":0,"allocated":290476032,"data_stored":289674998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703382,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211308,"kb_used_data":2144,"kb_used_omap":4,"kb_used_meta":209147,"kb_avail":93533844,"statfs":{"total":95995035648,"available":95778656256,"internally_reserved":0,"allocated":2195456,"data_stored":1389907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":214167233},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:32:24.637 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:24.637 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:24.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:24 smithi159 ceph-mon[45336]: pgmap v1406: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:24.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:24.713 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:24.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:24.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:24 smithi082 ceph-mon[44206]: pgmap v1406: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:24.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:24.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:24.846 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:25.182 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:25.195 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:25.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:25.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:25.331 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:25.353 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:25.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:25.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:32:25.457 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:25.485 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:25.504 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:25.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:25.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:25.623 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:25.638 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:25 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:25 smithi159 ceph-mon[45336]: osdmap e540: 12 total, 12 up, 12 in 2024-04-15T18:32:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:25 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:25 smithi049 ceph-mon[40087]: osdmap e540: 12 total, 12 up, 12 in 2024-04-15T18:32:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:25 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3739075813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:25 smithi082 ceph-mon[44206]: osdmap e540: 12 total, 12 up, 12 in 2024-04-15T18:32:25.854 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:26.494 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:26.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:26.599 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:26.627 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:26.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:26 smithi049 ceph-mon[40087]: pgmap v1408: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:26.768 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:26 smithi159 ceph-mon[45336]: pgmap v1408: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:26.776 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:26.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:26.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:26 smithi082 ceph-mon[44206]: pgmap v1408: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:26.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:26.912 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:26.931 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:26.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:27.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:27.064 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:27.071 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:32:27.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:32:27.120 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:32:27.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:32:27.145 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:32:27.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:32:27.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:32:27.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:32:27.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:32:27.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:32:27.297 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:32:27.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.323 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:75705 5.a 5:5a5ff8f8:::benchmark_data_smithi159_116570_object75704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e394)", 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:52.419296+0000", 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 394.91072428699999, 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41751247000000002, 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 75705 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.324 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419296+0000", 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419296+0000", 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419298+0000", 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5629999999999999e-06 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419319+0000", 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1642e-05 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419331+0000", 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1248e-05 2024-04-15T18:32:27.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419339+0000", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3790000000000006e-06 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419421+0000", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2250999999999996e-05 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419561+0000", 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000139877 2024-04-15T18:32:27.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,10", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.419712+0000", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015135699999999999 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.421754+0000", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002041583 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.422463+0000", 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070930799999999997 2024-04-15T18:32:27.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.836690+0000", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41422705300000001 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.836759+0000", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8867000000000003e-05 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:52.836809+0000", 2024-04-15T18:32:27.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9342000000000003e-05 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:76866 5.a 5:52d161c6:::benchmark_data_smithi159_116570_object76865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e396)", 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:54.397410+0000", 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 392.93261005800002, 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23467919500000001, 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:27.330 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 76866 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397410+0000", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397410+0000", 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397412+0000", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.612e-06 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397435+0000", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3173e-05 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397453+0000", 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7676000000000002e-05 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397463+0000", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404000000000001e-05 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397519+0000", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5497000000000001e-05 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397629+0000", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000110018 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 4,10", 2024-04-15T18:32:27.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.397735+0000", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010588499999999999 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.399792+0000", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002057077 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.400224+0000", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215000000000001 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:27.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.632009+0000", 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.231785506 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.632047+0000", 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7709000000000001e-05 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:54.632089+0000", 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2487999999999997e-05 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:110006 5.a e428/405 5:5ce26520:::benchmark_data_smithi159_116570_object16846:head v 428'6868, mlcod=380'2945)", 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:10.695834+0000", 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 316.63418652199999, 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25185369699999999, 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.695834+0000", 2024-04-15T18:32:27.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.695834+0000", 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.695835+0000", 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7499999999999998e-07 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.695843+0000", 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5380000000000002e-06 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.944985+0000", 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24914135400000001 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.944994+0000", 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1500000000000005e-06 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.945064+0000", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0199999999999999e-05 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.945093+0000", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8496e-05 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.947640+0000", 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025473179999999998 2024-04-15T18:32:27.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.947688+0000", 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7666000000000001e-05 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:110025 5.a e428/405 5:5f9c4e34:::benchmark_data_smithi159_116570_object16865:head v 428'6870, mlcod=380'2947)", 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:10.956598+0000", 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 316.37342206800002, 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26258671300000003, 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.340 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.956598+0000", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.956598+0000", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.956599+0000", 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.159e-06 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.956614+0000", 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4861999999999999e-05 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.216450+0000", 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25983591 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.216458+0000", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0539999999999995e-06 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.216536+0000", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7731000000000005e-05 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.216555+0000", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9343000000000001e-05 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.219148+0000", 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002592329 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:11.219185+0000", 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7324999999999999e-05 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:115113 5.a e447/442 5:5d0aabf6:::benchmark_data_smithi159_116570_object21953:head v 447'7173, mlcod=447'7171)", 2024-04-15T18:32:27.344 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:51.301380+0000", 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 276.02864005499998, 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22742219, 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.301380+0000", 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.301380+0000", 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.301381+0000", 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.09e-07 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.301391+0000", 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4809999999999995e-06 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525284+0000", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22389331200000001 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525293+0000", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.003e-06 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525353+0000", 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9506999999999998e-05 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525388+0000", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5608999999999997e-05 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.528766+0000", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033778800000000002 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.528802+0000", 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6489000000000002e-05 2024-04-15T18:32:27.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:115115 5.a e447/442 5:53b69bb5:::benchmark_data_smithi159_116570_object21955:head v 447'7174, mlcod=447'7171)", 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:51.525246+0000", 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 275.80477422899997, 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20131807800000001, 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525246+0000", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525246+0000", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525248+0000", 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6190000000000001e-06 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.525259+0000", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0781999999999999e-05 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.723184+0000", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.197925935 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.723194+0000", 2024-04-15T18:32:27.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2159999999999995e-06 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.723256+0000", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2001999999999994e-05 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.723280+0000", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4232000000000001e-05 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.726517+0000", 2024-04-15T18:32:27.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032375250000000002 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.726564+0000", 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6767000000000001e-05 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:115172 5.a e447/442 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head v 447'7176, mlcod=447'7173)", 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:51.737995+0000", 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 275.592025654, 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55704828100000003, 2024-04-15T18:32:27.353 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.737995+0000", 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.737995+0000", 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.737996+0000", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8700000000000004e-07 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:51.738012+0000", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6237999999999999e-05 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.292383+0000", 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55437141400000001 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.292392+0000", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0599999999999997e-06 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.292446+0000", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3863999999999998e-05 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.292465+0000", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8964e-05 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.295014+0000", 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025491350000000001 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:52.295043+0000", 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8619000000000002e-05 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:118591 5.e e459/358 5:79283452:::benchmark_data_smithi159_116570_object25431:head v 459'7445, mlcod=459'7442)", 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.827300+0000", 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 248.50272048400001, 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34332276900000003, 2024-04-15T18:32:27.357 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827300+0000", 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827300+0000", 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827301+0000", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2139999999999999e-06 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827311+0000", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0137e-05 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167693+0000", 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.340381451 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167699+0000", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0430000000000004e-06 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167754+0000", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5340999999999998e-05 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167772+0000", 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7530999999999999e-05 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170570+0000", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027979989999999998 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170623+0000", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3053000000000002e-05 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:118617 5.e e459/358 5:7a2bd32b:::benchmark_data_smithi159_116570_object25457:head v 459'7446, mlcod=459'7442)", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.167664+0000", 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 248.16235634700001, 2024-04-15T18:32:27.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33555991099999999, 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167664+0000", 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167664+0000", 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167665+0000", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8100000000000001e-07 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.167673+0000", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9219999999999997e-06 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.500298+0000", 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33262507800000002 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.500307+0000", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1740000000000006e-06 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.500373+0000", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5769e-05 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.500400+0000", 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6823999999999999e-05 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.503181+0000", 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002780977 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.503224+0000", 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3186000000000003e-05 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:124190 5.e e476/358 5:77ba6795:::benchmark_data_smithi159_116570_object31030:head v 476'7801, mlcod=476'7798)", 2024-04-15T18:32:27.365 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:56.457727+0000", 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 210.87229330400001, 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20060599000000001, 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.457727+0000", 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.457727+0000", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.457728+0000", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.212e-06 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.457739+0000", 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.026e-05 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.655245+0000", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19750601400000001 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.655251+0000", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3169999999999998e-06 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.655304+0000", 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3395e-05 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.655322+0000", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8224e-05 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.658300+0000", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029773220000000001 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.658333+0000", 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3246e-05 2024-04-15T18:32:27.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:124222 5.e e476/358 5:7e8543a6:::benchmark_data_smithi159_116570_object31062:head v 476'7803, mlcod=476'7800)", 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:56.663151+0000", 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 210.666869499, 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25643543699999999, 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.663151+0000", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.663151+0000", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.663152+0000", 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5000000000000001e-07 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.663159+0000", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6040000000000004e-06 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.916556+0000", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25339623500000003 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.916562+0000", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2400000000000004e-06 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.916630+0000", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8133000000000006e-05 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.916655+0000", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5545e-05 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.919547+0000", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028915189999999999 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:56.919586+0000", 2024-04-15T18:32:27.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9310999999999997e-05 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:126671 5.a e480/442 5:58971c7b:::benchmark_data_smithi159_116570_object33511:head v 480'7926, mlcod=480'7923)", 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:11.352635+0000", 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 195.97738528100001, 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20779967499999999, 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.352635+0000", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.352635+0000", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.352636+0000", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5199999999999996e-07 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.352642+0000", 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4969999999999998e-06 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.556291+0000", 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20364828700000001 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.556300+0000", 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0089999999999996e-06 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.556371+0000", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1933000000000004e-05 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.556397+0000", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5797000000000001e-05 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.560388+0000", 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039911299999999999 2024-04-15T18:32:27.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.560435+0000", 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6369999999999998e-05 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:126769 5.a e480/442 5:5eb8079e:::benchmark_data_smithi159_116570_object33609:head v 480'7933, mlcod=480'7930)", 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:11.661025+0000", 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 195.66899488199999, 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30291291799999998, 2024-04-15T18:32:27.378 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.661025+0000", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.661025+0000", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.661027+0000", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4619999999999999e-06 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.661036+0000", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6070000000000008e-06 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959867+0000", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29883187900000002 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959877+0000", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0052e-05 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959948+0000", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0827999999999993e-05 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959975+0000", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6346000000000002e-05 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.963894+0000", 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039190509999999998 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.963938+0000", 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4693000000000001e-05 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:126776 5.a e480/442 5:5a87c32d:::benchmark_data_smithi159_116570_object33616:head v 480'7934, mlcod=480'7931)", 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:11.707956+0000", 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 195.62206419699999, 2024-04-15T18:32:27.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.47685698399999998, 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.707956+0000", 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.707956+0000", 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.707957+0000", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2419999999999999e-06 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.707968+0000", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0288e-05 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.181393+0000", 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.47342503899999999 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.181404+0000", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1421e-05 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.181458+0000", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3689e-05 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.181478+0000", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9942e-05 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.184753+0000", 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032754820000000001 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.184813+0000", 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9880999999999998e-05 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:126795 5.a e480/442 5:53ab6dca:::benchmark_data_smithi159_116570_object33635:head v 480'7935, mlcod=480'7931)", 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:11.959828+0000", 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 195.370192261, 2024-04-15T18:32:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25932401300000002, 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959828+0000", 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959828+0000", 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959829+0000", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2100000000000001e-06 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:11.959841+0000", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1507e-05 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.215189+0000", 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25534779400000002 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.215198+0000", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5689999999999994e-06 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.215265+0000", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6514999999999996e-05 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.215288+0000", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3550999999999999e-05 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.219105+0000", 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038167510000000002 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:12.219152+0000", 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7116000000000001e-05 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:133451 5.e e492/358 5:7b367e68:::benchmark_data_smithi159_116570_object40291:head v 492'8377, mlcod=492'8374)", 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:51.369500+0000", 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.960519917, 2024-04-15T18:32:27.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18147374099999999, 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.369500+0000", 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.369500+0000", 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.369501+0000", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.369509+0000", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6680000000000001e-06 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.547788+0000", 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17827939900000001 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.547795+0000", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5849999999999996e-06 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.547851+0000", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6267000000000002e-05 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.547876+0000", 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4624e-05 2024-04-15T18:32:27.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.550918+0000", 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030424739999999999 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.550974+0000", 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6014999999999998e-05 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.394 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:133566 5.e e492/358 5:7fd548e2:::benchmark_data_smithi159_116570_object40406:head v 492'8384, mlcod=492'8381)", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:51.881129+0000", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.44889169300001, 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20747364300000001, 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.881129+0000", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.881129+0000", 2024-04-15T18:32:27.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.881130+0000", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.161e-06 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:51.881139+0000", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0960000000000007e-06 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.084381+0000", 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20324251099999999 2024-04-15T18:32:27.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.084390+0000", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3340000000000002e-06 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.084453+0000", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2947000000000006e-05 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.084479+0000", 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5987999999999999e-05 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.088557+0000", 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040787059999999997 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:52.088602+0000", 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.49e-05 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.398 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:142922 5.e e505/358 5:78b441c6:::benchmark_data_smithi159_116570_object49762:head v 505'8979, mlcod=505'8976)", 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:30:36.063658+0000", 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 111.26636234599999, 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.189250158, 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.063658+0000", 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.063658+0000", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.063659+0000", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.229e-06 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.063666+0000", 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1250000000000004e-06 2024-04-15T18:32:27.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.248628+0000", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18496214 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.248637+0000", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9770000000000006e-06 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.248696+0000", 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8350999999999997e-05 2024-04-15T18:32:27.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.248720+0000", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4454e-05 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.252867+0000", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041469239999999997 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:36.252908+0000", 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0958000000000003e-05 2024-04-15T18:32:27.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:143723 5.e e507/358 5:750142d7:::benchmark_data_smithi159_116570_object50563:head v 507'9032, mlcod=507'9029)", 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:30:39.413914+0000", 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 107.916106339, 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18784662399999999, 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.413914+0000", 2024-04-15T18:32:27.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.413914+0000", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.413915+0000", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2759999999999999e-06 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.413925+0000", 2024-04-15T18:32:27.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7280000000000007e-06 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.597708+0000", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18378277800000001 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.597715+0000", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9480000000000002e-06 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.597773+0000", 2024-04-15T18:32:27.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8323999999999999e-05 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.597797+0000", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4278999999999999e-05 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.601723+0000", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039254229999999999 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:39.601761+0000", 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7867999999999997e-05 2024-04-15T18:32:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:186306 5.e e536/358 5:78abf885:::benchmark_data_smithi159_116570_object93146:head v 536'11692, mlcod=536'11687)", 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.226028213999999, 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36219522999999998, 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1799999999999999e-07 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.104003+0000", 2024-04-15T18:32:27.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0264e-05 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462410+0000", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35840707900000002 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462420+0000", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0335e-05 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462477+0000", 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7283e-05 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462498+0000", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1155999999999999e-05 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466146+0000", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036480150000000001 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466187+0000", 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0880000000000002e-05 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.410 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:32:27.411 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:32:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:32:27.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:32:27.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2510696317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:27.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:27 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:27.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2510696317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:27.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:27 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:27.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2510696317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:27.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:27 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.114 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:28.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:28.131 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:32:28.152 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:28.155 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:28.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:28.247 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:28.346 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:28.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:28.395 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:28.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:28.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:28.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:28 smithi049 ceph-mon[40087]: pgmap v1409: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:28.640 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:28.659 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:28.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:28.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:28 smithi082 ceph-mon[44206]: pgmap v1409: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:28.788 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:28.789 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:28 smithi159 ceph-mon[45336]: pgmap v1409: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:28.791 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:28.863 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:28.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:28.967 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:28.993 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:29.456 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' created 2024-04-15T18:32:29.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-15T18:32:29.641 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:29.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:29.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-15T18:32:29.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:29.773 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:29.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-15T18:32:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:32:30.164 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:32:30.164 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:30.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:30 smithi049 ceph-mon[40087]: pgmap v1410: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:30.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-15T18:32:30.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:30 smithi049 ceph-mon[40087]: osdmap e541: 12 total, 12 up, 12 in 2024-04-15T18:32:30.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:30 smithi082 ceph-mon[44206]: pgmap v1410: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:30.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-15T18:32:30.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:30 smithi082 ceph-mon[44206]: osdmap e541: 12 total, 12 up, 12 in 2024-04-15T18:32:30.831 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:30 smithi159 ceph-mon[45336]: pgmap v1410: 1 pgs: 1 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:30.831 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2822390427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-15T18:32:30.831 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:30 smithi159 ceph-mon[45336]: osdmap e541: 12 total, 12 up, 12 in 2024-04-15T18:32:30.856 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:32:30.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:32:31.119 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:31.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-15T18:32:31.452 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:31.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:31.556 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:31.588 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:31.597 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:31 smithi159 ceph-mon[45336]: osdmap e542: 12 total, 12 up, 12 in 2024-04-15T18:32:31.608 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:31.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:31.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:31 smithi049 ceph-mon[40087]: osdmap e542: 12 total, 12 up, 12 in 2024-04-15T18:32:31.742 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:31.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:31 smithi082 ceph-mon[44206]: osdmap e542: 12 total, 12 up, 12 in 2024-04-15T18:32:31.841 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:31.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:31.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:32.577 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:32 smithi159 ceph-mon[45336]: pgmap v1413: 17 pgs: 5 active+clean, 2 creating+peering, 10 unknown; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:32.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:32 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:32:32.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:32 smithi159 ceph-mon[45336]: osdmap e543: 12 total, 12 up, 12 in 2024-04-15T18:32:32.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:32 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:32.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:32 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:32.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:32 smithi049 ceph-mon[40087]: pgmap v1413: 17 pgs: 5 active+clean, 2 creating+peering, 10 unknown; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:32.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:32 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:32:32.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:32 smithi049 ceph-mon[40087]: osdmap e543: 12 total, 12 up, 12 in 2024-04-15T18:32:32.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:32 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:32.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:32 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:32 smithi082 ceph-mon[44206]: pgmap v1413: 17 pgs: 5 active+clean, 2 creating+peering, 10 unknown; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:32 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:32:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:32 smithi082 ceph-mon[44206]: osdmap e543: 12 total, 12 up, 12 in 2024-04-15T18:32:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:32 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:32.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:32 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:32.818 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:32.824 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:32.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:32.848 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:32.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:32.957 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:32:32.960 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:33.069 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:33.070 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:32:33.070 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:32:33.070 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:32:33.070 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:33.482 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:32:33.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:33 smithi049 ceph-mon[40087]: osdmap e544: 12 total, 12 up, 12 in 2024-04-15T18:32:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:32:33.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:33 smithi049 ceph-mon[40087]: from='client.16458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:33 smithi082 ceph-mon[44206]: osdmap e544: 12 total, 12 up, 12 in 2024-04-15T18:32:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:32:33.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:33 smithi082 ceph-mon[44206]: from='client.16458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:33 smithi159 ceph-mon[45336]: osdmap e544: 12 total, 12 up, 12 in 2024-04-15T18:32:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:32:33.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:33 smithi159 ceph-mon[45336]: from='client.16458 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:34.210 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:34.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:34.314 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:34.352 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: pgmap v1416: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:32:34.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:34 smithi159 ceph-mon[45336]: osdmap e545: 12 total, 12 up, 12 in 2024-04-15T18:32:34.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:32:34.604 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:34.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:34.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:34.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: pgmap v1416: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:32:34.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:34 smithi049 ceph-mon[40087]: osdmap e545: 12 total, 12 up, 12 in 2024-04-15T18:32:34.740 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: pgmap v1416: 17 pgs: 15 active+clean, 2 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1081810639' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:32:34.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:34 smithi082 ceph-mon[44206]: osdmap e545: 12 total, 12 up, 12 in 2024-04-15T18:32:34.867 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:35.020 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:35.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:35.040 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1415,"stamp":"2024-04-15T18:32:32.479907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":33,"acting":33,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":4725004,"kb_used_data":795152,"kb_used_omap":177,"kb_used_meta":3926798,"kb_avail":1120216820,"statfs":{"total":1151940427776,"available":1147102023680,"internally_reserved":0,"allocated":814235648,"data_stored":805319187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":4021041604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.102807"},"pg_stats":[{"pgid":"6.f","version":"0'0","reported_seq":13,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-15T18:32:31.475529+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:32:31.475529+0000","last_peered":"2024-04-15T18:32:31.475529+0000","last_clean":"2024-04-15T18:32:31.475529+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:32:31.475529+0000","last_undegraded":"2024-04-15T18:32:31.475529+0000","last_fullsized":"2024-04-15T18:32:31.475529+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:47.154458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":541,"state":"creating+peering","last_fresh":"2024-04-15T18:32:29.471258+0000","last_change":"2024-04-15T18:32:29.471258+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.471258+0000","last_undegraded":"2024-04-15T18:32:29.471258+0000","last_fullsized":"2024-04-15T18:32:29.471258+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.475662+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:32:30.475662+0000","last_peered":"2024-04-15T18:32:30.475662+0000","last_clean":"2024-04-15T18:32:30.475662+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:32:30.475662+0000","last_undegraded":"2024-04-15T18:32:30.475662+0000","last_fullsized":"2024-04-15T18:32:30.475662+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:55.899149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":13,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-15T18:32:31.657604+0000","last_change":"2024-04-15T18:32:30.982142+0000","last_active":"2024-04-15T18:32:31.657604+0000","last_peered":"2024-04-15T18:32:31.657604+0000","last_clean":"2024-04-15T18:32:31.657604+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:32:31.657604+0000","last_undegraded":"2024-04-15T18:32:31.657604+0000","last_fullsized":"2024-04-15T18:32:31.657604+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:37.360210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.736560+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:32:30.736560+0000","last_peered":"2024-04-15T18:32:30.736560+0000","last_clean":"2024-04-15T18:32:30.736560+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:32:30.736560+0000","last_undegraded":"2024-04-15T18:32:30.736560+0000","last_fullsized":"2024-04-15T18:32:30.736560+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:45.916025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.480569+0000","last_change":"2024-04-15T18:32:30.480197+0000","last_active":"2024-04-15T18:32:30.480569+0000","last_peered":"2024-04-15T18:32:30.480569+0000","last_clean":"2024-04-15T18:32:30.480569+0000","last_became_active":"2024-04-15T18:32:30.479842+0000","last_became_peered":"2024-04-15T18:32:30.479842+0000","last_unstale":"2024-04-15T18:32:30.480569+0000","last_undegraded":"2024-04-15T18:32:30.480569+0000","last_fullsized":"2024-04-15T18:32:30.480569+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:35.478788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.478260+0000","last_change":"2024-04-15T18:32:30.477765+0000","last_active":"2024-04-15T18:32:30.478260+0000","last_peered":"2024-04-15T18:32:30.478260+0000","last_clean":"2024-04-15T18:32:30.478260+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:32:30.478260+0000","last_undegraded":"2024-04-15T18:32:30.478260+0000","last_fullsized":"2024-04-15T18:32:30.478260+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.817218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.736382+0000","last_change":"2024-04-15T18:32:30.736024+0000","last_active":"2024-04-15T18:32:30.736382+0000","last_peered":"2024-04-15T18:32:30.736382+0000","last_clean":"2024-04-15T18:32:30.736382+0000","last_became_active":"2024-04-15T18:32:30.735671+0000","last_became_peered":"2024-04-15T18:32:30.735671+0000","last_unstale":"2024-04-15T18:32:30.736382+0000","last_undegraded":"2024-04-15T18:32:30.736382+0000","last_fullsized":"2024-04-15T18:32:30.736382+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:49.832518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":541,"state":"creating+peering","last_fresh":"2024-04-15T18:32:29.463818+0000","last_change":"2024-04-15T18:32:29.463818+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.463818+0000","last_undegraded":"2024-04-15T18:32:29.463818+0000","last_fullsized":"2024-04-15T18:32:29.463818+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":13,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-04-15T18:32:31.475471+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:32:31.475471+0000","last_peered":"2024-04-15T18:32:31.475471+0000","last_clean":"2024-04-15T18:32:31.475471+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:32:31.475471+0000","last_undegraded":"2024-04-15T18:32:31.475471+0000","last_fullsized":"2024-04-15T18:32:31.475471+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:33.539021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T18:32:29.457742+0000","last_change":"2024-04-15T18:32:29.457742+0000","last_active":"2024-04-15T18:32:29.457742+0000","last_peered":"2024-04-15T18:32:29.457742+0000","last_clean":"2024-04-15T18:32:29.457742+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T18:32:29.457742+0000","last_undegraded":"2024-04-15T18:32:29.457742+0000","last_fullsized":"2024-04-15T18:32:29.457742+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1350,"reported_epoch":542,"state":"active+clean","last_fresh":"2024-04-15T18:32:30.470699+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:32:30.470699+0000","last_peered":"2024-04-15T18:32:30.470699+0000","last_clean":"2024-04-15T18:32:30.470699+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:32:30.470699+0000","last_undegraded":"2024-04-15T18:32:30.470699+0000","last_fullsized":"2024-04-15T18:32:30.470699+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:00.300641+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":30,"acting":30,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926104,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449004,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93296148,"statfs":{"total":95995035648,"available":95535255552,"internally_reserved":0,"allocated":2412544,"data_stored":1712367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":456980932},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":317960,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93427192,"statfs":{"total":95995035648,"available":95669444608,"internally_reserved":0,"allocated":2224128,"data_stored":1405047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514786,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348532,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93396620,"statfs":{"total":95995035648,"available":95638138880,"internally_reserved":0,"allocated":1708032,"data_stored":1073499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874534,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163860,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93581292,"statfs":{"total":95995035648,"available":95827243008,"internally_reserved":0,"allocated":2052096,"data_stored":1295236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601500,"kb_used_data":259164,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93143652,"statfs":{"total":95995035648,"available":95379099648,"internally_reserved":0,"allocated":265383936,"data_stored":264623524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463218,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376796,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93368356,"statfs":{"total":95995035648,"available":95609196544,"internally_reserved":0,"allocated":2584576,"data_stored":1824164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575182,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416548,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93328604,"statfs":{"total":95995035648,"available":95568490496,"internally_reserved":0,"allocated":1880064,"data_stored":1183459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438756,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93306396,"statfs":{"total":95995035648,"available":95545749504,"internally_reserved":0,"allocated":1880064,"data_stored":1183459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159156,"kb_used_data":1668,"kb_used_omap":114,"kb_used_meta":156813,"kb_avail":93585996,"statfs":{"total":95995035648,"available":95832059904,"internally_reserved":0,"allocated":1708032,"data_stored":1073499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":160576753},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609083,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":602888,"kb_used_data":253180,"kb_used_omap":44,"kb_used_meta":349651,"kb_avail":93142264,"statfs":{"total":95995035648,"available":95377678336,"internally_reserved":0,"allocated":259256320,"data_stored":258437239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":358043178},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986489,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638664,"kb_used_data":264572,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93106488,"statfs":{"total":95995035648,"available":95341043712,"internally_reserved":0,"allocated":270921728,"data_stored":270102647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":211340,"kb_used_data":2172,"kb_used_omap":4,"kb_used_meta":209147,"kb_avail":93533812,"statfs":{"total":95995035648,"available":95778623488,"internally_reserved":0,"allocated":2224128,"data_stored":1405047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":214167233},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:32:35.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:32:35.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:35.152 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:35.171 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:35.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:35.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:35.307 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:35.329 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:35.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:35.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:35.460 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:35.658 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:35.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:35.762 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:35.788 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:35.986 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:35.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:36.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:36.148 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:36.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:36.252 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:36.286 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:36.309 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:36.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:36.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:36.441 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:36.462 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:36.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:36.483 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-15T18:32:36.497 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:36.497 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43719 5.b 5.2edd20bb (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 443.22341422300002, 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96247389699999997, 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43719 2024-04-15T18:32:36.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280935+0000", 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280937+0000", 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7749999999999999e-06 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280959+0000", 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2347000000000001e-05 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280972+0000", 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3123e-05 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.280981+0000", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1179999999999998e-06 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281041+0000", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9335e-05 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.281049+0000", 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7730000000000005e-06 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243354+0000", 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.96230429100000003 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243409+0000", 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5135e-05 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.502 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43724 5.b 5.c9c3aa8b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 443.19922599400002, 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93864302600000005, 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43724 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305123+0000", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305125+0000", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9e-06 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305150+0000", 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4640000000000001e-05 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305167+0000", 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.694e-05 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305176+0000", 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6220000000000004e-06 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305242+0000", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5919000000000004e-05 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.305251+0000", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9779999999999994e-06 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243757+0000", 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.93850576200000002 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243766+0000", 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2650000000000002e-06 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43728 5.b 5.b2d2527b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 443.01629210800002, 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75574586300000002, 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43728 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488057+0000", 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488059+0000", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8610000000000001e-06 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488103+0000", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4598000000000002e-05 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488118+0000", 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4708e-05 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488125+0000", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7630000000000003e-06 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488178+0000", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3477999999999999e-05 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.488188+0000", 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2890000000000003e-06 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243796+0000", 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75560810199999995 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243803+0000", 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0639999999999996e-06 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.511 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:43732 5.b 5.9c14c54b (undecoded) ondisk+write+known_if_redirected e376)", 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 442.823184889, 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56280661600000004, 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 43732 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681164+0000", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681165+0000", 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.243e-06 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681206+0000", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0428e-05 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681218+0000", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2269999999999999e-05 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681225+0000", 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4529999999999999e-06 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681272+0000", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7775999999999998e-05 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.681280+0000", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.892e-06 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243939+0000", 2024-04-15T18:32:36.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56265829300000003 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.243971+0000", 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2261000000000002e-05 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 400/377 [PushReplyOp(5:d68de874:::benchmark_data_smithi159_116570_object10690:head)])", 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 395.30464568399998, 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37573558899999998, 2024-04-15T18:32:36.516 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199703+0000", 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199704+0000", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4300000000000002e-07 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.199711+0000", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2540000000000003e-06 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574171+0000", 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.374460026 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574179+0000", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1080000000000004e-06 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.574267+0000", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8676000000000003e-05 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.575439+0000", 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011717819999999999 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.519 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:82071 5.e 5:768b069e:::benchmark_data_smithi159_116570_object82070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e401)", 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 394.94069575200001, 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84584237600000001, 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.520 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 82071 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563653+0000", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563655+0000", 2024-04-15T18:32:36.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0899999999999999e-06 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563684+0000", 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8977999999999999e-05 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563704+0000", 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9874999999999999e-05 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563715+0000", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0266e-05 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563777+0000", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2502999999999994e-05 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.563782+0000", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6979999999999997e-06 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565036+0000", 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012537850000000001 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565179+0000", 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014387199999999999 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.565329+0000", 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000150004 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568089+0000", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027595150000000001 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.569096+0000", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010070610000000001 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409426+0000", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84032972100000003 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409466+0000", 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0374999999999999e-05 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409496+0000", 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9632999999999999e-05 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:82071 5.e e401/358 ondisk, result = 0)", 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:32:36.526 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 394.93631842299999, 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84147125499999997, 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568031+0000", 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568032+0000", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3790000000000001e-06 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:01.568044+0000", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1654999999999999e-05 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409368+0000", 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.841324552 2024-04-15T18:32:36.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409372+0000", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.048e-06 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409416+0000", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3933000000000001e-05 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409422+0000", 2024-04-15T18:32:36.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6860000000000003e-06 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:02.409502+0000", 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0001999999999997e-05 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 404/377 [PushReplyOp(5:dd0ee7d7:::benchmark_data_smithi159_116570_object27502:head)])", 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 381.03505934700001, 2024-04-15T18:32:36.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39222512599999998, 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469295+0000", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6469999999999997e-06 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860402+0000", 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39110686099999997 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860410+0000", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860475+0000", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5375000000000003e-05 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.861515+0000", 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010396470000000001 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 405/377 [PushReplyOp(5:d997bfb1:::benchmark_data_smithi159_116570_object30349:head)])", 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 379.74726208599998, 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43079569699999998, 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:32:36.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757088+0000", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7000000000000005e-07 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757092+0000", 2024-04-15T18:32:36.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6079999999999998e-06 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186476+0000", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42938425299999999 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186493+0000", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6646e-05 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186666+0000", 2024-04-15T18:32:36.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017301299999999999 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.187883+0000", 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001216607 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118576 5.e 5:782c6d1f:::benchmark_data_smithi159_116570_object25416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 257.79553434600001, 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.461995184, 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.537 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118576 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708816+0000", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.75e-06 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708824+0000", 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1359999999999999e-06 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708842+0000", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8142e-05 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708852+0000", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0181e-05 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708919+0000", 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7094000000000005e-05 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709064+0000", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014527 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709192+0000", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012819900000000001 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712786+0000", 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00359377 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712964+0000", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000177651 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170740+0000", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45777645 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170776+0000", 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5828000000000003e-05 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170810+0000", 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3713000000000001e-05 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:118576 5.e e459/358 ondisk, result = 0)", 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 257.79127094, 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45773745599999999, 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.543 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713079+0000", 2024-04-15T18:32:36.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000001e-07 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713085+0000", 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170680+0000", 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45759575699999999 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170687+0000", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3509999999999998e-06 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170726+0000", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9140000000000001e-05 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170736+0000", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0000000000000001e-05 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170816+0000", 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9723999999999998e-05 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118591 5.e 5:79283452:::benchmark_data_smithi159_116570_object25431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 257.67741347100002, 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.451595888, 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.547 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118591 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826937+0000", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.375e-06 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826944+0000", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2350000000000002e-06 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826959+0000", 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5013e-05 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826969+0000", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0023e-05 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827033+0000", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3962999999999997e-05 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827123+0000", 2024-04-15T18:32:36.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9382999999999994e-05 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827201+0000", 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8164000000000006e-05 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830060+0000", 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028591860000000001 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830383+0000", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032335000000000003 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278451+0000", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44806768699999999 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278489+0000", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7829e-05 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278531+0000", 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2679999999999998e-05 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:119911 5.e 5:78ed0cf0:::benchmark_data_smithi159_116570_object26751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e463)", 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 247.87575624199999, 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42595297199999999, 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.553 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 119911 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628594+0000", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.556e-06 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628601+0000", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628614+0000", 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2962999999999999e-05 2024-04-15T18:32:36.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628622+0000", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1240000000000005e-06 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628689+0000", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6351000000000002e-05 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628833+0000", 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014472500000000001 2024-04-15T18:32:36.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628965+0000", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013210300000000001 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632755+0000", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037895329999999999 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632768+0000", 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2626000000000001e-05 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054455+0000", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42168709399999998 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054508+0000", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3328999999999997e-05 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054546+0000", 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7734999999999997e-05 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.558 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:119911 5.e e463/358 ondisk, result = 0)", 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 247.87232935399999, 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42253153399999999, 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:32:36.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632021+0000", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632027+0000", 2024-04-15T18:32:36.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9689999999999999e-06 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054377+0000", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42235067500000001 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054386+0000", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4190000000000002e-06 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054439+0000", 2024-04-15T18:32:36.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3771999999999999e-05 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054448+0000", 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4149999999999999e-06 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054551+0000", 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103403 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186297 5.e 5:7bc39edb:::benchmark_data_smithi159_116570_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.418803584999999, 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.38087093300000002, 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186297 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085546+0000", 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3399999999999997e-07 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085552+0000", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4530000000000001e-06 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085564+0000", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2542e-05 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085572+0000", 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7610000000000007e-06 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089739+0000", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004166477 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089871+0000", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000132669 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089950+0000", 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8208999999999996e-05 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.094319+0000", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043699189999999999 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.094340+0000", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0687999999999999e-05 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466324+0000", 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37198407 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466368+0000", 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3843e-05 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466416+0000", 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8368000000000002e-05 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186306 5.e 5:78abf885:::benchmark_data_smithi159_116570_object93146:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:32:36.568 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.406204851999998, 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67465055600000001, 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186306 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098146+0000", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098152+0000", 2024-04-15T18:32:36.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0310000000000004e-06 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098163+0000", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1181e-05 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098171+0000", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0239999999999997e-06 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098226+0000", 2024-04-15T18:32:36.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5457000000000001e-05 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098370+0000", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014345300000000001 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098497+0000", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012726299999999999 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103398+0000", 2024-04-15T18:32:36.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049014469999999997 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.108448+0000", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050492990000000001 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772670+0000", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.66422262099999996 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772755+0000", 2024-04-15T18:32:36.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5030000000000004e-05 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772795+0000", 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9406999999999997e-05 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:186297 5.e e536/358 ondisk, result = 0)", 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.396355503999999, 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35842624299999998, 2024-04-15T18:32:36.574 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107995+0000", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.68e-06 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.108002+0000", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6390000000000003e-06 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466258+0000", 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35825611800000001 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466265+0000", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5169999999999996e-06 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466312+0000", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7539e-05 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466320+0000", 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4680000000000003e-06 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466420+0000", 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100282 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186297 5.e 5:7bc39edb:::benchmark_data_smithi159_116570_object93137:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e536)", 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.113138+0000", 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.391210717, 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.353240793, 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:32:36.578 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186297 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113138+0000", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113138+0000", 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113139+0000", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2300000000000002e-07 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113141+0000", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9379999999999999e-06 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113146+0000", 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1649999999999997e-06 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113149+0000", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.596e-06 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113334+0000", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000184857 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for ondisk", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113396+0000", 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2203000000000001e-05 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466379+0000", 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.352982711 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186306 5.e 5:78abf885:::benchmark_data_smithi159_116570_object93146:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e536)", 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.391177604999999, 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.65959372100000002, 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.582 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186306 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:32:36.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113172+0000", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7899999999999999e-07 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113175+0000", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3239999999999999e-06 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113183+0000", 2024-04-15T18:32:36.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5290000000000003e-06 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113189+0000", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1299999999999998e-06 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113434+0000", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024558700000000002 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for ondisk", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113477+0000", 2024-04-15T18:32:36.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3093000000000002e-05 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772765+0000", 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.65928767899999996 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186314 5.b 5:db471f5e:::benchmark_data_smithi159_116570_object93154:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.391114164000001, 2024-04-15T18:32:36.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.99055303699999997, 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186314 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5600000000000001e-07 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113238+0000", 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7130000000000001e-06 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113248+0000", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0137e-05 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113252+0000", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000001e-06 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113371+0000", 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118981 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113479+0000", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010749800000000001 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113615+0000", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013696199999999999 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.117350+0000", 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003734818 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.120268+0000", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029180479999999999 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103702+0000", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.983433848 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103742+0000", 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9328e-05 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:36.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103788+0000", 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6397000000000003e-05 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:36.592 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:36.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:36.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:36 smithi159 ceph-mon[45336]: pgmap v1418: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:36.593 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:32:36.621 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:36.718 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:36.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:36.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:36 smithi082 ceph-mon[44206]: pgmap v1418: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:36.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:32:36.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:36.847 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:36.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:36 smithi049 ceph-mon[40087]: pgmap v1418: 17 pgs: 16 active+clean, 1 creating+peering; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:36.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:32:37.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:32:37.195 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:37.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:37.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:37.325 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:37.343 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:37.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:37.357 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:37.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:37.469 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:37.478 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:37.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:32:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:32:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:37 smithi082 ceph-mon[44206]: osdmap e546: 12 total, 12 up, 12 in 2024-04-15T18:32:37.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:32:37.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:37 smithi049 ceph-mon[40087]: osdmap e546: 12 total, 12 up, 12 in 2024-04-15T18:32:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/847864168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:32:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:37 smithi159 ceph-mon[45336]: osdmap e546: 12 total, 12 up, 12 in 2024-04-15T18:32:37.931 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:37.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:38.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:38.066 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:38.088 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:38.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:38.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:38.220 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:38.420 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:38.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:38.525 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:38.556 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:38.560 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:38 smithi159 ceph-mon[45336]: pgmap v1420: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:38.560 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2374469287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:38.560 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:38 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:32:38.578 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:38.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:38.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:38.711 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:38.732 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:38.733 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:38 smithi082 ceph-mon[44206]: pgmap v1420: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:38.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2374469287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:38.734 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:38 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:32:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:38.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:38.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:38 smithi049 ceph-mon[40087]: pgmap v1420: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:38.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2374469287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:38.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:38 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:32:38.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:38.865 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:38.909 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-15T18:32:38.912 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:32:38.912 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:32:38.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:32:38.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:32:38.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_139342 2024-04-15T18:32:38.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:32:38.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:32:39.029 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:32:39.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:32:39.095 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:32:39.129 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:32:39.132 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:39.132 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:32:39.132 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:32:39.132 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:39.164 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:32:39.192 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:32:39.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:32:39.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:32:39.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:32:39.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:32:39.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:32:39.361 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:32:39.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:32:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.809 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.809 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:39.835 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:32:39.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 1407 1391 86.9333 86.9375 0.00178001 0.0113966 2024-04-15T18:32:40.197 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:40.316 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:40.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:40.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:40.451 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:40.473 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:40.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:40.577 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:40.608 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:40.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:40.702 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:40 smithi159 ceph-mon[45336]: pgmap v1421: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:40.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:40 smithi049 ceph-mon[40087]: pgmap v1421: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:40.765 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:40 smithi082 ceph-mon[44206]: pgmap v1421: 17 pgs: 17 active+clean; 577 KiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:32:40.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:40.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:40.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 2778 2762 86.3066 85.6875 0.00967367 0.0115393 2024-04-15T18:32:40.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:40.997 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:41.016 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:41.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:41.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:41.152 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:41.253 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:41.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:41.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:41.391 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:32:41.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:41.821 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:41.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 4577 4561 95.0137 112.438 0.00328448 0.0104921 2024-04-15T18:32:41.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:32:42.043 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:42.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:42.133 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:42.133 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:42.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:42.184 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:42.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:32:42.386 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:42.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:42.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:42.519 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:42.540 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:42.543 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:42.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:42.644 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:42.676 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:42.700 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:42.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:42 smithi049 ceph-mon[40087]: pgmap v1422: 17 pgs: 17 active+clean; 60 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s wr, 118 op/s 2024-04-15T18:32:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:42 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:42.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:42 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:32:42.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:42.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:42 smithi049 ceph-mon[40087]: osdmap e547: 12 total, 12 up, 12 in 2024-04-15T18:32:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:42 smithi082 ceph-mon[44206]: pgmap v1422: 17 pgs: 17 active+clean; 60 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s wr, 118 op/s 2024-04-15T18:32:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:42 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:42 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:32:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:42.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:42 smithi082 ceph-mon[44206]: osdmap e547: 12 total, 12 up, 12 in 2024-04-15T18:32:42.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:42.835 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:42 smithi159 ceph-mon[45336]: pgmap v1422: 17 pgs: 17 active+clean; 60 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s wr, 118 op/s 2024-04-15T18:32:42.835 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:42 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:32:42.835 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:42 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:32:42.835 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3928386873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:32:42.835 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:42 smithi159 ceph-mon[45336]: osdmap e547: 12 total, 12 up, 12 in 2024-04-15T18:32:42.845 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:42.863 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:32:42.866 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:42.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:42.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 6136 6120 95.6178 97.4375 0.0115652 0.0104288 2024-04-15T18:32:42.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:43.002 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:43.026 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:43.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:43.130 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:43.160 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:43.173 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:43.181 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:43.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:43.285 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:43.314 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:43.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:43.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:43.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:43.558 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:43.561 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:32:43.574 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:43.598 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:43.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:43.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:43.733 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:43.932 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:43.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:43.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 7552 7536 94.1929 88.5 0.181393 0.0102837 2024-04-15T18:32:44.036 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:44.065 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:44.089 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:44.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:44.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:44.224 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:44.375 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:44.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:44.398 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1424,"stamp":"2024-04-15T18:32:43.380137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289473056,"num_objects":4410,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":4589,"num_write_kb":285050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4547,"ondisk_log_size":4547,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5363436,"kb_used_data":1455876,"kb_used_omap":177,"kb_used_meta":3905998,"kb_avail":1119578388,"statfs":{"total":1151940427776,"available":1146448269312,"internally_reserved":0,"allocated":1490817024,"data_stored":1485718944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3999742404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":288882688,"num_objects":4408,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":282112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001572"},"pg_stats":[{"pgid":"6.f","version":"547'305","reported_seq":326,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.335813+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:32:42.335813+0000","last_peered":"2024-04-15T18:32:42.335813+0000","last_clean":"2024-04-15T18:32:42.335813+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:32:42.335813+0000","last_undegraded":"2024-04-15T18:32:42.335813+0000","last_fullsized":"2024-04-15T18:32:42.335813+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.454307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":305,"num_write_kb":19520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"547'394","reported_seq":414,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.686912+0000","last_change":"2024-04-15T18:32:30.477777+0000","last_active":"2024-04-15T18:32:42.686912+0000","last_peered":"2024-04-15T18:32:42.686912+0000","last_clean":"2024-04-15T18:32:42.686912+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:32:42.686912+0000","last_undegraded":"2024-04-15T18:32:42.686912+0000","last_fullsized":"2024-04-15T18:32:42.686912+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.795587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821184,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":394,"num_write_kb":25216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"547'352","reported_seq":373,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.455795+0000","last_change":"2024-04-15T18:32:30.640352+0000","last_active":"2024-04-15T18:32:42.455795+0000","last_peered":"2024-04-15T18:32:42.455795+0000","last_clean":"2024-04-15T18:32:42.455795+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:32:42.455795+0000","last_undegraded":"2024-04-15T18:32:42.455795+0000","last_fullsized":"2024-04-15T18:32:42.455795+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:36.209094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"546'149","reported_seq":168,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.378037+0000","last_change":"2024-04-15T18:32:30.479011+0000","last_active":"2024-04-15T18:32:40.378037+0000","last_peered":"2024-04-15T18:32:40.378037+0000","last_clean":"2024-04-15T18:32:40.378037+0000","last_became_active":"2024-04-15T18:32:30.478523+0000","last_became_peered":"2024-04-15T18:32:30.478523+0000","last_unstale":"2024-04-15T18:32:40.378037+0000","last_undegraded":"2024-04-15T18:32:40.378037+0000","last_fullsized":"2024-04-15T18:32:40.378037+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:57.752075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149,"num_write_kb":9536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"546'173","reported_seq":191,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.977571+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:32:40.977571+0000","last_peered":"2024-04-15T18:32:40.977571+0000","last_clean":"2024-04-15T18:32:40.977571+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:32:40.977571+0000","last_undegraded":"2024-04-15T18:32:40.977571+0000","last_fullsized":"2024-04-15T18:32:40.977571+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.581404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"547'338","reported_seq":360,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.451597+0000","last_change":"2024-04-15T18:32:30.982302+0000","last_active":"2024-04-15T18:32:42.451597+0000","last_peered":"2024-04-15T18:32:42.451597+0000","last_clean":"2024-04-15T18:32:42.451597+0000","last_became_active":"2024-04-15T18:32:30.981964+0000","last_became_peered":"2024-04-15T18:32:30.981964+0000","last_unstale":"2024-04-15T18:32:42.451597+0000","last_undegraded":"2024-04-15T18:32:42.451597+0000","last_fullsized":"2024-04-15T18:32:42.451597+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:34.969438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":338,"num_write_kb":21632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"547'336","reported_seq":358,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.359066+0000","last_change":"2024-04-15T18:32:30.982142+0000","last_active":"2024-04-15T18:32:42.359066+0000","last_peered":"2024-04-15T18:32:42.359066+0000","last_clean":"2024-04-15T18:32:42.359066+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:32:42.359066+0000","last_undegraded":"2024-04-15T18:32:42.359066+0000","last_fullsized":"2024-04-15T18:32:42.359066+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:30.349808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":336,"num_write_kb":21504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"546'181","reported_seq":200,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.980999+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:32:40.980999+0000","last_peered":"2024-04-15T18:32:40.980999+0000","last_clean":"2024-04-15T18:32:40.980999+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:32:40.980999+0000","last_undegraded":"2024-04-15T18:32:40.980999+0000","last_fullsized":"2024-04-15T18:32:40.980999+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:38.538924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"547'345","reported_seq":365,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.704070+0000","last_change":"2024-04-15T18:32:30.480070+0000","last_active":"2024-04-15T18:32:42.704070+0000","last_peered":"2024-04-15T18:32:42.704070+0000","last_clean":"2024-04-15T18:32:42.704070+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:32:42.704070+0000","last_undegraded":"2024-04-15T18:32:42.704070+0000","last_fullsized":"2024-04-15T18:32:42.704070+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:53.238775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":345,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"546'146","reported_seq":166,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.981784+0000","last_change":"2024-04-15T18:32:30.480197+0000","last_active":"2024-04-15T18:32:40.981784+0000","last_peered":"2024-04-15T18:32:40.981784+0000","last_clean":"2024-04-15T18:32:40.981784+0000","last_became_active":"2024-04-15T18:32:30.479842+0000","last_became_peered":"2024-04-15T18:32:30.479842+0000","last_unstale":"2024-04-15T18:32:40.981784+0000","last_undegraded":"2024-04-15T18:32:40.981784+0000","last_fullsized":"2024-04-15T18:32:40.981784+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.405063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":146,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"547'316","reported_seq":338,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.353226+0000","last_change":"2024-04-15T18:32:30.477765+0000","last_active":"2024-04-15T18:32:42.353226+0000","last_peered":"2024-04-15T18:32:42.353226+0000","last_clean":"2024-04-15T18:32:42.353226+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:32:42.353226+0000","last_undegraded":"2024-04-15T18:32:42.353226+0000","last_fullsized":"2024-04-15T18:32:42.353226+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:52.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":20224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"546'177","reported_seq":197,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.974653+0000","last_change":"2024-04-15T18:32:30.736024+0000","last_active":"2024-04-15T18:32:40.974653+0000","last_peered":"2024-04-15T18:32:40.974653+0000","last_clean":"2024-04-15T18:32:40.974653+0000","last_became_active":"2024-04-15T18:32:30.735671+0000","last_became_peered":"2024-04-15T18:32:30.735671+0000","last_unstale":"2024-04-15T18:32:40.974653+0000","last_undegraded":"2024-04-15T18:32:40.974653+0000","last_fullsized":"2024-04-15T18:32:40.974653+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.574745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"547'364","reported_seq":384,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.680530+0000","last_change":"2024-04-15T18:32:30.478009+0000","last_active":"2024-04-15T18:32:42.680530+0000","last_peered":"2024-04-15T18:32:42.680530+0000","last_clean":"2024-04-15T18:32:42.680530+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:32:42.680530+0000","last_undegraded":"2024-04-15T18:32:42.680530+0000","last_fullsized":"2024-04-15T18:32:42.680530+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.474140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23855104,"num_objects":364,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":364,"num_write_kb":23296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"546'122","reported_seq":139,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:40.311442+0000","last_change":"2024-04-15T18:32:30.639803+0000","last_active":"2024-04-15T18:32:40.311442+0000","last_peered":"2024-04-15T18:32:40.311442+0000","last_clean":"2024-04-15T18:32:40.311442+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:32:40.311442+0000","last_undegraded":"2024-04-15T18:32:40.311442+0000","last_fullsized":"2024-04-15T18:32:40.311442+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:47.630738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"547'345","reported_seq":366,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.332116+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:32:42.332116+0000","last_peered":"2024-04-15T18:32:42.332116+0000","last_clean":"2024-04-15T18:32:42.332116+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:32:42.332116+0000","last_undegraded":"2024-04-15T18:32:42.332116+0000","last_fullsized":"2024-04-15T18:32:42.332116+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.855955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":345,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"547'365","reported_seq":386,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-04-15T18:32:42.461127+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:32:42.461127+0000","last_peered":"2024-04-15T18:32:42.461127+0000","last_clean":"2024-04-15T18:32:42.461127+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:32:42.461127+0000","last_undegraded":"2024-04-15T18:32:42.461127+0000","last_fullsized":"2024-04-15T18:32:42.461127+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:45.545736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":23360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1358,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-04-15T18:32:36.490119+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:32:36.490119+0000","last_peered":"2024-04-15T18:32:36.490119+0000","last_clean":"2024-04-15T18:32:36.490119+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:32:36.490119+0000","last_undegraded":"2024-04-15T18:32:36.490119+0000","last_fullsized":"2024-04-15T18:32:36.490119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:00.300641+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":288882688,"num_objects":4408,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":282112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":912261120,"data_stored":912261120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4408,"ondisk_log_size":4408,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926106,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":486380,"kb_used_data":35756,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93258772,"statfs":{"total":95995035648,"available":95496982528,"internally_reserved":0,"allocated":36614144,"data_stored":36224413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":461240772},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187743,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386980,"kb_used_data":66852,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93358172,"statfs":{"total":95995035648,"available":95598768128,"internally_reserved":0,"allocated":68456448,"data_stored":68070289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514788,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422884,"kb_used_data":72228,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":93322268,"statfs":{"total":95995035648,"available":95562002432,"internally_reserved":0,"allocated":73961472,"data_stored":73575313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":358873536},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874536,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174536,"kb_used_data":12864,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93570616,"statfs":{"total":95995035648,"available":95816310784,"internally_reserved":0,"allocated":13172736,"data_stored":12656652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168877,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627100,"kb_used_data":280788,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93118052,"statfs":{"total":95995035648,"available":95352885248,"internally_reserved":0,"allocated":287526912,"data_stored":287074306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":354613706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463220,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":391900,"kb_used_data":17812,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93353252,"statfs":{"total":95995035648,"available":95593730048,"internally_reserved":0,"allocated":18239488,"data_stored":17786882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575184,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500132,"kb_used_data":81444,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93245020,"statfs":{"total":95995035648,"available":95482900480,"internally_reserved":0,"allocated":83398656,"data_stored":83012497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461972,"kb_used_data":37900,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93283180,"statfs":{"total":95995035648,"available":95521976320,"internally_reserved":0,"allocated":38809600,"data_stored":38360546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193956,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219812,"kb_used_data":75172,"kb_used_omap":114,"kb_used_meta":144333,"kb_avail":93525340,"statfs":{"total":95995035648,"available":95769948160,"internally_reserved":0,"allocated":76976128,"data_stored":76589969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":147797233},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609086,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726496,"kb_used_data":385112,"kb_used_omap":44,"kb_used_meta":341331,"kb_avail":93018656,"statfs":{"total":95995035648,"available":95251103744,"internally_reserved":0,"allocated":394354688,"data_stored":393901086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":349523498},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986492,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":684568,"kb_used_data":306136,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93060584,"statfs":{"total":95995035648,"available":95294038016,"internally_reserved":0,"allocated":313483264,"data_stored":313029662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703387,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280676,"kb_used_data":83812,"kb_used_omap":4,"kb_used_meta":196667,"kb_avail":93464476,"statfs":{"total":95995035648,"available":95707623424,"internally_reserved":0,"allocated":85823488,"data_stored":85437329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":201387713},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41877504,"data_stored":41877504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87097344,"data_stored":87097344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86966272,"data_stored":86966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":169410560,"data_stored":169410560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":97845248,"data_stored":97845248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":114556928,"data_stored":114556928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36306944,"data_stored":36306944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":55836672,"data_stored":55836672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15728640,"data_stored":15728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":84410368,"data_stored":84410368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":79691776,"data_stored":79691776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":42532864,"data_stored":42532864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:32:44.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:32:44.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:44.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:32:44.517 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:44.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:44.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:44.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:44.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:44 smithi159 ceph-mon[45336]: pgmap v1424: 17 pgs: 17 active+clean; 276 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 550 op/s 2024-04-15T18:32:44.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:44 smithi159 ceph-mon[45336]: from='client.16491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:44.683 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:44.709 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:44.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:44 smithi049 ceph-mon[40087]: pgmap v1424: 17 pgs: 17 active+clean; 276 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 550 op/s 2024-04-15T18:32:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:44 smithi049 ceph-mon[40087]: from='client.16491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:44 smithi082 ceph-mon[44206]: pgmap v1424: 17 pgs: 17 active+clean; 276 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 550 op/s 2024-04-15T18:32:44.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:44 smithi082 ceph-mon[44206]: from='client.16491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:32:44.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:44.857 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:44.887 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:44.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:44.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:44.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:44.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 9319 9303 96.8991 110.438 0.00349113 0.0103105 2024-04-15T18:32:44.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:45.028 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:45.140 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:32:45.141 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:32:45.158 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:45.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:45.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:45.294 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:45.420 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:45.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:45.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:45.554 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:45.615 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:45.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:45.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:45.750 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:45.770 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:45.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:45.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:45.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:32:45.920 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:45.964 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 10498 10482 93.5823 73.6875 0.0044552 0.0106617 2024-04-15T18:32:46.183 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: pgmap v1425: 17 pgs: 17 active+clean; 308 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 552 op/s 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1518815275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: pgmap v1425: 17 pgs: 17 active+clean; 308 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 552 op/s 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1518815275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 11846 11830 92.4149 84.25 0.00556952 0.0107861 2024-04-15T18:32:46.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: pgmap v1425: 17 pgs: 17 active+clean; 308 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 552 op/s 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1518815275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:46.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:32:47.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:32:47.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:32:47.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:32:47.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:32:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:32:47.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:32:47.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.718 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.718 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.718 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.718 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.718 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:32:47.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 13268 13252 92.0208 88.875 0.00179705 0.0108355 2024-04-15T18:32:47.995 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:47.995 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:32:47.995 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:32:47.995 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:48.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:48 smithi159 ceph-mon[45336]: pgmap v1426: 17 pgs: 17 active+clean; 568 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:32:48.713 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:32:48.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:48 smithi082 ceph-mon[44206]: pgmap v1426: 17 pgs: 17 active+clean; 568 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:32:48.881 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:48.883 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:48 smithi049 ceph-mon[40087]: pgmap v1426: 17 pgs: 17 active+clean; 568 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-04-15T18:32:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:48.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:48.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 14830 14814 92.5792 97.625 0.0860323 0.0107781 2024-04-15T18:32:48.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:49.028 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:49.038 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:49.055 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:49.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:49.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:49.212 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:49.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 16691 16675 94.7359 116.312 0.00207852 0.0105445 2024-04-15T18:32:50.369 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:50.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:50.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:32:50.514 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:50.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:50 smithi049 ceph-mon[40087]: pgmap v1427: 17 pgs: 17 active+clean; 730 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:32:50.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:50.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:50.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:50 smithi082 ceph-mon[44206]: pgmap v1427: 17 pgs: 17 active+clean; 730 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:32:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:50.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:50.854 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:50.854 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:32:50.854 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:32:50.854 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:32:50.854 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:50.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:32:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:50.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:32:50.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:50 smithi159 ceph-mon[45336]: pgmap v1427: 17 pgs: 17 active+clean; 730 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:32:50.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:50.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:50.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 17952 17936 93.4085 78.8125 0.00962181 0.0106954 2024-04-15T18:32:50.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:51.002 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:51.552 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:51.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:51.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:32:51.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:51.687 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:51.708 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:51.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:51.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:51 smithi049 ceph-mon[40087]: osdmap e548: 12 total, 12 up, 12 in 2024-04-15T18:32:51.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:51 smithi082 ceph-mon[44206]: osdmap e548: 12 total, 12 up, 12 in 2024-04-15T18:32:51.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:51.840 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:51 smithi159 ceph-mon[45336]: osdmap e548: 12 total, 12 up, 12 in 2024-04-15T18:32:51.844 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:51.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:51.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:51.930 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:32:51.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 19555 19539 93.9291 100.188 0.0199214 0.0106275 2024-04-15T18:32:51.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:52.001 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:52.020 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:52.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:52.124 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:52.138 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:32:52.138 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1719,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":548,"num_osds":12,"num_up_osds":12,"osd_up_since":1713205513,"num_in_osds":12,"osd_in_since":1713205575,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":14449,"data_bytes":947388960,"bytes_used":7233933312,"bytes_avail":1144706494464,"bytes_total":1151940427776,"write_bytes_sec":89614701,"read_op_per_sec":0,"write_op_per_sec":1367},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":37,"modified":"2024-04-15T18:32:17.375315+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-15T18:32:52.171 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:52.198 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:52.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:52.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:52.343 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:52.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:32:52.424 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:52.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:52.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:32:52.564 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:52.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:52 smithi049 ceph-mon[40087]: pgmap v1429: 17 pgs: 17 active+clean; 904 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-15T18:32:52.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:52.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1353163054' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:52.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:52 smithi082 ceph-mon[44206]: pgmap v1429: 17 pgs: 17 active+clean; 904 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-15T18:32:52.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:52.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1353163054' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:52.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:52 smithi159 ceph-mon[45336]: pgmap v1429: 17 pgs: 17 active+clean; 904 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-04-15T18:32:52.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:32:52.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1353163054' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:32:52.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 21175 21159 94.4515 101.25 0.00655697 0.0105773 2024-04-15T18:32:53.217 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:53.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:53.320 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:53.347 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:53.496 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:53.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:53.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:53.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:53.700 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:53.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:53 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:53.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:53 smithi082 ceph-mon[44206]: osdmap e549: 12 total, 12 up, 12 in 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:43707 5.f e375/364)", 2024-04-15T18:32:53.795 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 460.56390109799997, 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.94185888299999998, 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237679+0000", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237680+0000", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6700000000000003e-07 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:13.237786+0000", 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010644100000000001 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179372+0000", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.94158575 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179378+0000", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6400000000000002e-06 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179521+0000", 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014270600000000001 2024-04-15T18:32:53.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:25:14.179538+0000", 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7578999999999999e-05 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 422/416 [PushOp(5:8704fa45:::benchmark_data_smithi159_116570_object14951:head, version: 369'921, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8704fa45:::benchmark_data_smithi159_116570_object14951:head@369'921, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 360.74696610400002, 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.79326008699999995, 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.799 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054614+0000", 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054615+0000", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.054649+0000", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3844999999999999e-05 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.792384+0000", 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.73773477899999995 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.792405+0000", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0985999999999999e-05 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.822826+0000", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.030421790000000001 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.823399+0000", 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057227600000000003 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:53.847874+0000", 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024475635999999999 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109624 5.1 5:84c547a6:::benchmark_data_smithi159_116570_object16464:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 344.91405293299999, 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61741415600000005, 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.803 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109624 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887527+0000", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887529+0000", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.172e-06 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887534+0000", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9250000000000003e-06 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887545+0000", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1678999999999999e-05 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.887552+0000", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8530000000000003e-06 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.939213+0000", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.051660436999999997 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.944118+0000", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049053029999999997 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.952640+0000", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085221399999999992 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.002593+0000", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.049952866999999998 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.004827+0000", 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022339119999999998 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.489082+0000", 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48425561 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.492309+0000", 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032266809999999999 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.504942+0000", 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012632577000000001 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109639 5.1 5:8cec3473:::benchmark_data_smithi159_116570_object16479:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 344.87517948099998, 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59574422100000002, 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:53.809 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109639 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926401+0000", 2024-04-15T18:32:53.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926402+0000", 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.54e-07 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926408+0000", 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.981e-06 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926420+0000", 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2374999999999999e-05 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.926429+0000", 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2429999999999994e-06 2024-04-15T18:32:53.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.959548+0000", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.033118455999999998 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.964288+0000", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047403590000000004 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.972993+0000", 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087051390000000006 2024-04-15T18:32:53.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.006957+0000", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.033963659 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.009091+0000", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021345159999999999 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.505524+0000", 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.49643292 2024-04-15T18:32:53.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.508285+0000", 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002760523 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.522145+0000", 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013860196 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109646 5.1 5:847282b1:::benchmark_data_smithi159_116570_object16486:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 344.87036947399997, 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61196233099999997, 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109646 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931211+0000", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931212+0000", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931216+0000", 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4039999999999997e-06 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931227+0000", 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1061e-05 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.931233+0000", 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-04-15T18:32:53.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.010244+0000", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.079011203000000002 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.015127+0000", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048829340000000002 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.023722+0000", 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085948350000000003 2024-04-15T18:32:53.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.032963+0000", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092412020000000004 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.035115+0000", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021520290000000002 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.524421+0000", 2024-04-15T18:32:53.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48930604100000002 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.528026+0000", 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036048389999999999 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.543173+0000", 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015147287000000001 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:109740 5.1 5:85fc6f90:::benchmark_data_smithi159_116570_object16580:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e428)", 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 344.31396634200001, 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.57482975400000003, 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:32:53.822 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109740 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487614+0000", 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487615+0000", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1480000000000001e-06 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487619+0000", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0810000000000004e-06 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487630+0000", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1104000000000001e-05 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.487637+0000", 2024-04-15T18:32:53.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8140000000000003e-06 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.628337+0000", 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.140700198 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.633780+0000", 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054425510000000003 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,6", 2024-04-15T18:32:53.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.642947+0000", 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091669679999999993 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.652867+0000", 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099201830000000008 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.673347+0000", 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020479687 2024-04-15T18:32:53.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.045461+0000", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37211393300000001 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.048790+0000", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033297650000000002 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.062444+0000", 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013653322000000001 2024-04-15T18:32:53.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 428/416 [PushOp(5:8992b35b:::benchmark_data_smithi159_116570_object16825:head, version: 369'1040, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8992b35b:::benchmark_data_smithi159_116570_object16825:head@369'1040, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 343.60050110200001, 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.78917612000000004, 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201079+0000", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201080+0000", 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.051e-06 2024-04-15T18:32:53.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201140+0000", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9311000000000002e-05 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201171+0000", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1680000000000002e-05 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.201186+0000", 2024-04-15T18:32:53.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4775e-05 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.965149+0000", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.76396304299999995 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.965718+0000", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056873799999999995 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:10.990255+0000", 2024-04-15T18:32:53.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024537521999999999 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 428/416 [PushReplyOp(5:8b327da9:::benchmark_data_smithi159_116570_object16904:head)])", 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 341.52547942000001, 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64978421600000003, 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276101+0000", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276102+0000", 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8700000000000004e-07 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276109+0000", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6830000000000002e-06 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276140+0000", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1109000000000003e-05 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.276151+0000", 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1572e-05 2024-04-15T18:32:53.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.899505+0000", 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62335336200000002 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.925885+0000", 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026380503 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.835 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 428/416 [PushReplyOp(5:8b327da9:::benchmark_data_smithi159_116570_object16904:head)])", 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 341.52368001399998, 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67448548900000005, 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277900+0000", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277901+0000", 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1099999999999995e-07 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277909+0000", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5320000000000001e-06 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277937+0000", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8476999999999999e-05 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.277945+0000", 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7039999999999994e-06 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.926536+0000", 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64859101600000002 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:12.952386+0000", 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025850148999999999 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 430/416 [PushReplyOp(5:86484eaf:::benchmark_data_smithi159_116570_object17263:head)])", 2024-04-15T18:32:53.839 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 338.43439388100001, 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63286371799999996, 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367186+0000", 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367187+0000", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4900000000000001e-07 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367194+0000", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.153e-06 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367213+0000", 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8885999999999998e-05 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.367237+0000", 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3858000000000001e-05 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:15.972237+0000", 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60500030100000002 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:16.000050+0000", 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.027813071000000002 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 449/416 [PushReplyOp(5:8d33612f:::benchmark_data_smithi159_116570_object22274:head)])", 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 299.23137666600002, 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61168342600000003, 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.843 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570204+0000", 2024-04-15T18:32:53.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1099999999999996e-07 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570208+0000", 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8800000000000001e-06 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570231+0000", 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2594000000000001e-05 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570237+0000", 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5470000000000002e-06 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.168270+0000", 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59803278500000001 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.181887+0000", 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013617109000000001 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 449/416 [PushReplyOp(5:8d33612f:::benchmark_data_smithi159_116570_object22274:head)])", 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 299.231040527, 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.673407639, 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570540+0000", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570543+0000", 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1990000000000002e-06 2024-04-15T18:32:53.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570560+0000", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6484e-05 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:54.570583+0000", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3203000000000001e-05 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.216682+0000", 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64609930500000001 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:55.243947+0000", 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.027265007000000001 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 459/416 [PushReplyOp(5:83564489:::benchmark_data_smithi159_116570_object25233:head)])", 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 274.69821753500003, 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.78063833000000005, 2024-04-15T18:32:53.850 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103363+0000", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.75e-07 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103370+0000", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.939e-06 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103401+0000", 2024-04-15T18:32:53.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0516999999999999e-05 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.103408+0000", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.272e-06 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.860295+0000", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75688664000000005 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.884001+0000", 2024-04-15T18:32:53.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023706386999999999 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 459/416 [PushOp(5:8aa070ee:::benchmark_data_smithi159_116570_object25481:head, version: 370'1582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:8aa070ee:::benchmark_data_smithi159_116570_object25481:head@370'1582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 274.58681202499997, 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55792225500000003, 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.854 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214768+0000", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214769+0000", 2024-04-15T18:32:53.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1900000000000003e-07 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214830+0000", 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0820999999999999e-05 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214857+0000", 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7039e-05 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.214875+0000", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8512000000000002e-05 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.752069+0000", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53719326000000001 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.752683+0000", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00061441800000000004 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.772691+0000", 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020007686 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 459/416 [PushOp(5:88c22e6a:::benchmark_data_smithi159_116570_object25781:head, version: 371'1606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:88c22e6a:::benchmark_data_smithi159_116570_object25781:head@371'1606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 271.82658690199997, 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56908714999999999, 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.858 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974993+0000", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.974995+0000", 2024-04-15T18:32:53.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.026e-06 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:21.975056+0000", 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1749999999999997e-05 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.452398+0000", 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.47734127900000001 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.452409+0000", 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1143e-05 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.523093+0000", 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.070684223000000004 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.523564+0000", 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00047106200000000001 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:22.544081+0000", 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020516666999999999 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 476/416 [PushReplyOp(5:8fb5237a:::benchmark_data_smithi159_116570_object31080:head)])", 2024-04-15T18:32:53.862 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 235.88950894199999, 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62137753200000001, 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912071+0000", 2024-04-15T18:32:53.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912072+0000", 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8899999999999999e-07 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912076+0000", 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7009999999999998e-06 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912098+0000", 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2235999999999999e-05 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:57.912103+0000", 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1819999999999997e-06 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:58.511060+0000", 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59895651100000002 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:58.533449+0000", 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022389313000000001 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 480/416 [PushOp(5:801faef5:::benchmark_data_smithi159_116570_object33041:head, version: 372'2043, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:801faef5:::benchmark_data_smithi159_116570_object33041:head@372'2043, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 225.001333942, 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.72250066199999996, 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.866 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800246+0000", 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800247+0000", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:08.800287+0000", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9996000000000001e-05 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.336390+0000", 2024-04-15T18:32:53.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.53610254300000004 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.336406+0000", 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6167e-05 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.503579+0000", 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16717265100000001 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.504081+0000", 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050283800000000003 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:09.522747+0000", 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018665698000000001 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(5.1 512/416 [PushOp(5:87326a1b:::benchmark_data_smithi159_116570_object55573:head, version: 384'3379, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:87326a1b:::benchmark_data_smithi159_116570_object55573:head@384'3379, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:30:57.680471+0000", 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 116.12110894200001, 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.58172505600000002, 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680471+0000", 2024-04-15T18:32:53.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680471+0000", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680472+0000", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4399999999999999e-07 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680532+0000", 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9951000000000002e-05 2024-04-15T18:32:53.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680564+0000", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1674999999999997e-05 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.680575+0000", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1524e-05 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.250773+0000", 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.57019814899999999 2024-04-15T18:32:53.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.251000+0000", 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022657600000000001 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.262196+0000", 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011196437 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 512/416 [PushReplyOp(5:8c6b2c02:::benchmark_data_smithi159_116570_object55404:head)])", 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:30:57.719072+0000", 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 116.082508115, 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59952648600000003, 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719072+0000", 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719072+0000", 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719073+0000", 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5000000000000002e-07 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719081+0000", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.661e-06 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719110+0000", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9507999999999999e-05 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719120+0000", 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3230000000000004e-06 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.307702+0000", 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.58858200100000002 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.318599+0000", 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010897242999999999 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.1 512/416 [PushReplyOp(5:8c6b2c02:::benchmark_data_smithi159_116570_object55404:head)])", 2024-04-15T18:32:53.878 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:30:57.719605+0000", 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 116.08197510399999, 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.587051561, 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719605+0000", 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:32:53.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719605+0000", 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719606+0000", 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1900000000000001e-07 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719610+0000", 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9219999999999999e-06 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719650+0000", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0015000000000002e-05 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:57.719673+0000", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2611000000000001e-05 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.301914+0000", 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.58224150600000002 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:32:53.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:30:58.306657+0000", 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047426880000000001 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:32:53.882 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:32:53.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:53.883 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:53 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:53.883 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:53.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:53 smithi159 ceph-mon[45336]: osdmap e549: 12 total, 12 up, 12 in 2024-04-15T18:32:53.913 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:53.935 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:53.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:53.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:53 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:32:53.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3830603607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:32:53.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:53 smithi049 ceph-mon[40087]: osdmap e549: 12 total, 12 up, 12 in 2024-04-15T18:32:53.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 22101 22085 92.0127 57.875 0.00262382 0.0108582 2024-04-15T18:32:54.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:54.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:54.325 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-15T18:32:54.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:32:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:54.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:54.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:54.491 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:54.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:32:54.555 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:54.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:54.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:54 smithi049 ceph-mon[40087]: pgmap v1431: 17 pgs: 17 active+clean; 1.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-15T18:32:54.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:54.688 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:54.708 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:54.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:54.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:54 smithi082 ceph-mon[44206]: pgmap v1431: 17 pgs: 17 active+clean; 1.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-15T18:32:54.811 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:32:54.843 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:54.867 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:32:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:54.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:32:54.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:54 smithi159 ceph-mon[45336]: pgmap v1431: 17 pgs: 17 active+clean; 1.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.81k op/s 2024-04-15T18:32:54.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 23307 23291 90.9724 75.375 0.00306398 0.0109846 2024-04-15T18:32:54.972 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:32:55.001 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:55.005 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:55.019 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:55.024 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:32:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:55.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:32:55.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:55.160 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:55.181 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:32:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:55.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:32:55.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:55.329 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:55.360 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:55.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:55.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:32:55.500 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:55.755 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:32:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:55.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:32:55.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:32:55.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:55.904 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:32:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:55.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:32:55.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 24815 24799 91.1644 94.25 0.00220237 0.010906 2024-04-15T18:32:56.008 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:32:56.035 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:56.053 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:32:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:56.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:32:56.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:56.221 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:56.394 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:56.417 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:56.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:56.522 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:32:56.558 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:56.587 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:32:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:56.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:32:56.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:56.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:56.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:56.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:56 smithi049 ceph-mon[40087]: pgmap v1432: 17 pgs: 17 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:32:56.766 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:56 smithi049 ceph-mon[40087]: osdmap e550: 12 total, 12 up, 12 in 2024-04-15T18:32:56.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:56.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:56.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:56.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:56 smithi082 ceph-mon[44206]: pgmap v1432: 17 pgs: 17 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:32:56.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:56 smithi082 ceph-mon[44206]: osdmap e550: 12 total, 12 up, 12 in 2024-04-15T18:32:56.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:56.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:56.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:56 smithi159 ceph-mon[45336]: pgmap v1432: 17 pgs: 17 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:32:56.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:56 smithi159 ceph-mon[45336]: osdmap e550: 12 total, 12 up, 12 in 2024-04-15T18:32:56.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 26034 26018 90.3319 76.1875 0.00475928 0.0110623 2024-04-15T18:32:57.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:32:57.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 26911 26895 88.4622 54.8125 0.189885 0.0111927 2024-04-15T18:32:58.424 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:58.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:58.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:58.570 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:58.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:58 smithi159 ceph-mon[45336]: pgmap v1434: 17 pgs: 17 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-15T18:32:58.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:58 smithi049 ceph-mon[40087]: pgmap v1434: 17 pgs: 17 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-15T18:32:58.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:58 smithi082 ceph-mon[44206]: pgmap v1434: 17 pgs: 17 active+clean; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.67k op/s 2024-04-15T18:32:58.928 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:58.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:58.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:32:58.975372+0000 min lat: 0.00106491 max lat: 0.293279 avg lat: 0.011163 2024-04-15T18:32:58.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:32:58.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 28670 28654 89.5357 109.938 0.0101361 0.011163 2024-04-15T18:32:59.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:32:59.064 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:59.085 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:32:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:59.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:32:59.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:32:59.226 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:59.478 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:32:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:32:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:32:59.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:32:59 smithi049 ceph-mon[40087]: osdmap e551: 12 total, 12 up, 12 in 2024-04-15T18:32:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:59.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:32:59.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:32:59.614 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:59.640 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:32:59.643 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:32:59.643 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:32:59.643 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:32:59 smithi082 ceph-mon[44206]: osdmap e551: 12 total, 12 up, 12 in 2024-04-15T18:32:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:59.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:32:59.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:32:59.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:32:59.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:32:59 smithi159 ceph-mon[45336]: osdmap e551: 12 total, 12 up, 12 in 2024-04-15T18:32:59.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:32:59.790 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:32:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:32:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:32:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:32:59.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:32:59.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:32:59.965 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 30062 30046 89.4146 87 0.0164907 0.0111747 2024-04-15T18:32:59.991 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:00.011 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:00.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:00.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:00.144 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:33:00.148 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:33:00.156 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:00 smithi049 ceph-mon[40087]: pgmap v1436: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:33:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:00.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:00 smithi049 ceph-mon[40087]: from='client.16512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:00.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:00 smithi159 ceph-mon[45336]: pgmap v1436: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:33:00.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:00.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:00.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:00 smithi159 ceph-mon[45336]: from='client.16512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:00.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:00 smithi082 ceph-mon[44206]: pgmap v1436: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:33:00.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:00.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:00.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:00 smithi082 ceph-mon[44206]: from='client.16512 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:00.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 15 31575 31560 89.6511 94.625 0.00225184 0.0111489 2024-04-15T18:33:01.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:01 smithi049 ceph-mon[40087]: osdmap e552: 12 total, 12 up, 12 in 2024-04-15T18:33:01.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:01 smithi159 ceph-mon[45336]: osdmap e552: 12 total, 12 up, 12 in 2024-04-15T18:33:01.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:01 smithi082 ceph-mon[44206]: osdmap e552: 12 total, 12 up, 12 in 2024-04-15T18:33:01.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 33391 33375 90.6849 113.438 0.00356138 0.0110187 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:33680 6.4 e552/541 6:2f1dfcb1:::benchmark_data_smithi159_139342_object33679:head v 552'2030, mlcod=552'2026)", 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.184490+0000", 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.013863614999999999, 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013894215999999999, 2024-04-15T18:33:02.240 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.184490+0000", 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.184490+0000", 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.184491+0000", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.184509+0000", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7989e-05 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.196836+0000", 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012327515000000001 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.196844+0000", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2279999999999998e-06 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.196923+0000", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8799000000000002e-05 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.196943+0000", 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0203e-05 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.198381+0000", 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014378399999999999 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:33697 6.0 e552/541 6:03cea9a4:::benchmark_data_smithi159_139342_object33696:head v 552'2131, mlcod=552'2128)", 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.197141+0000", 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001212402, 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001330934, 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:02.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197141+0000", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197141+0000", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197142+0000", 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.68e-07 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197163+0000", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0738e-05 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197195+0000", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2271000000000003e-05 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197202+0000", 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5459999999999997e-06 2024-04-15T18:33:02.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197242+0000", 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0389000000000002e-05 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.197260+0000", 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8012e-05 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-04-15T18:33:02.247 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:33:02.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:02 smithi049 ceph-mon[40087]: pgmap v1438: 17 pgs: 17 active+clean; 1.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:33:02.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:02 smithi082 ceph-mon[44206]: pgmap v1438: 17 pgs: 17 active+clean; 1.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:33:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:02 smithi159 ceph-mon[45336]: pgmap v1438: 17 pgs: 17 active+clean; 1.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:33:02.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 34128 34112 88.8255 46.0625 0.00428612 0.0111959 2024-04-15T18:33:03.507 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1436,"stamp":"2024-04-15T18:32:59.384096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1692402208,"num_objects":25817,"num_object_clones":0,"num_object_copies":77451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25817,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":25996,"num_write_kb":1655098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25954,"ondisk_log_size":25954,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9390280,"kb_used_data":5492216,"kb_used_omap":177,"kb_used_meta":3897678,"kb_avail":1115551544,"statfs":{"total":1151940427776,"available":1142324781056,"internally_reserved":0,"allocated":5624029184,"data_stored":5620443588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":3991222724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,6,5,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":615579648,"num_objects":9393,"num_object_clones":0,"num_object_copies":28179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":601152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.850799"},"pg_stats":[{"pgid":"6.f","version":"550'1711","reported_seq":1739,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.310594+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:32:57.310594+0000","last_peered":"2024-04-15T18:32:57.310594+0000","last_clean":"2024-04-15T18:32:57.310594+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:32:57.310594+0000","last_undegraded":"2024-04-15T18:32:57.310594+0000","last_fullsized":"2024-04-15T18:32:57.310594+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1711,"log_dups_size":0,"ondisk_log_size":1711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.454307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112132096,"num_objects":1711,"num_object_clones":0,"num_object_copies":5133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":109504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"550'1657","reported_seq":1683,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.723363+0000","last_change":"2024-04-15T18:32:30.477777+0000","last_active":"2024-04-15T18:32:57.723363+0000","last_peered":"2024-04-15T18:32:57.723363+0000","last_clean":"2024-04-15T18:32:57.723363+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:32:57.723363+0000","last_undegraded":"2024-04-15T18:32:57.723363+0000","last_fullsized":"2024-04-15T18:32:57.723363+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1657,"log_dups_size":0,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.795587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108593152,"num_objects":1657,"num_object_clones":0,"num_object_copies":4971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1657,"num_write_kb":106048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"550'1681","reported_seq":1709,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.555685+0000","last_change":"2024-04-15T18:32:30.640352+0000","last_active":"2024-04-15T18:32:57.555685+0000","last_peered":"2024-04-15T18:32:57.555685+0000","last_clean":"2024-04-15T18:32:57.555685+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:32:57.555685+0000","last_undegraded":"2024-04-15T18:32:57.555685+0000","last_fullsized":"2024-04-15T18:32:57.555685+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1681,"log_dups_size":0,"ondisk_log_size":1681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:36.209094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1681,"num_write_kb":107584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"549'1540","reported_seq":1565,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.423102+0000","last_change":"2024-04-15T18:32:30.479011+0000","last_active":"2024-04-15T18:32:55.423102+0000","last_peered":"2024-04-15T18:32:55.423102+0000","last_clean":"2024-04-15T18:32:55.423102+0000","last_became_active":"2024-04-15T18:32:30.478523+0000","last_became_peered":"2024-04-15T18:32:30.478523+0000","last_unstale":"2024-04-15T18:32:55.423102+0000","last_undegraded":"2024-04-15T18:32:55.423102+0000","last_fullsized":"2024-04-15T18:32:55.423102+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:57.752075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100925440,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1540,"num_write_kb":98560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"550'1562","reported_seq":1589,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.911999+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:32:55.911999+0000","last_peered":"2024-04-15T18:32:55.911999+0000","last_clean":"2024-04-15T18:32:55.911999+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:32:55.911999+0000","last_undegraded":"2024-04-15T18:32:55.911999+0000","last_fullsized":"2024-04-15T18:32:55.911999+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.581404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1562,"num_write_kb":99968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"550'1631","reported_seq":1659,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.544721+0000","last_change":"2024-04-15T18:32:30.982302+0000","last_active":"2024-04-15T18:32:57.544721+0000","last_peered":"2024-04-15T18:32:57.544721+0000","last_clean":"2024-04-15T18:32:57.544721+0000","last_became_active":"2024-04-15T18:32:30.981964+0000","last_became_peered":"2024-04-15T18:32:30.981964+0000","last_unstale":"2024-04-15T18:32:57.544721+0000","last_undegraded":"2024-04-15T18:32:57.544721+0000","last_fullsized":"2024-04-15T18:32:57.544721+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:34.969438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1631,"num_write_kb":104384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"550'1612","reported_seq":1640,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.277484+0000","last_change":"2024-04-15T18:32:30.982142+0000","last_active":"2024-04-15T18:32:57.277484+0000","last_peered":"2024-04-15T18:32:57.277484+0000","last_clean":"2024-04-15T18:32:57.277484+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:32:57.277484+0000","last_undegraded":"2024-04-15T18:32:57.277484+0000","last_fullsized":"2024-04-15T18:32:57.277484+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:30.349808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":103168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"550'1527","reported_seq":1555,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.878643+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:32:55.878643+0000","last_peered":"2024-04-15T18:32:55.878643+0000","last_clean":"2024-04-15T18:32:55.878643+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:32:55.878643+0000","last_undegraded":"2024-04-15T18:32:55.878643+0000","last_fullsized":"2024-04-15T18:32:55.878643+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:38.538924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100073472,"num_objects":1527,"num_object_clones":0,"num_object_copies":4581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1527,"num_write_kb":97728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"550'1636","reported_seq":1663,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.709686+0000","last_change":"2024-04-15T18:32:30.480070+0000","last_active":"2024-04-15T18:32:57.709686+0000","last_peered":"2024-04-15T18:32:57.709686+0000","last_clean":"2024-04-15T18:32:57.709686+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:32:57.709686+0000","last_undegraded":"2024-04-15T18:32:57.709686+0000","last_fullsized":"2024-04-15T18:32:57.709686+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:53.238775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":104704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"550'1560","reported_seq":1588,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.829198+0000","last_change":"2024-04-15T18:32:30.480197+0000","last_active":"2024-04-15T18:32:55.829198+0000","last_peered":"2024-04-15T18:32:55.829198+0000","last_clean":"2024-04-15T18:32:55.829198+0000","last_became_active":"2024-04-15T18:32:30.479842+0000","last_became_peered":"2024-04-15T18:32:30.479842+0000","last_unstale":"2024-04-15T18:32:55.829198+0000","last_undegraded":"2024-04-15T18:32:55.829198+0000","last_fullsized":"2024-04-15T18:32:55.829198+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.405063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102236160,"num_objects":1560,"num_object_clones":0,"num_object_copies":4680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1560,"num_write_kb":99840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"550'1591","reported_seq":1618,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.306642+0000","last_change":"2024-04-15T18:32:30.477765+0000","last_active":"2024-04-15T18:32:57.306642+0000","last_peered":"2024-04-15T18:32:57.306642+0000","last_clean":"2024-04-15T18:32:57.306642+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:32:57.306642+0000","last_undegraded":"2024-04-15T18:32:57.306642+0000","last_fullsized":"2024-04-15T18:32:57.306642+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:52.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104267776,"num_objects":1591,"num_object_clones":0,"num_object_copies":4773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1591,"num_write_kb":101824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"550'1584","reported_seq":1612,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.867285+0000","last_change":"2024-04-15T18:32:30.736024+0000","last_active":"2024-04-15T18:32:55.867285+0000","last_peered":"2024-04-15T18:32:55.867285+0000","last_clean":"2024-04-15T18:32:55.867285+0000","last_became_active":"2024-04-15T18:32:30.735671+0000","last_became_peered":"2024-04-15T18:32:30.735671+0000","last_unstale":"2024-04-15T18:32:55.867285+0000","last_undegraded":"2024-04-15T18:32:55.867285+0000","last_fullsized":"2024-04-15T18:32:55.867285+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.574745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"550'1709","reported_seq":1736,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.714121+0000","last_change":"2024-04-15T18:32:30.478009+0000","last_active":"2024-04-15T18:32:57.714121+0000","last_peered":"2024-04-15T18:32:57.714121+0000","last_clean":"2024-04-15T18:32:57.714121+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:32:57.714121+0000","last_undegraded":"2024-04-15T18:32:57.714121+0000","last_fullsized":"2024-04-15T18:32:57.714121+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1709,"log_dups_size":0,"ondisk_log_size":1709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.474140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":5127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1709,"num_write_kb":109376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"549'1493","reported_seq":1517,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.313563+0000","last_change":"2024-04-15T18:32:30.639803+0000","last_active":"2024-04-15T18:32:55.313563+0000","last_peered":"2024-04-15T18:32:55.313563+0000","last_clean":"2024-04-15T18:32:55.313563+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:32:55.313563+0000","last_undegraded":"2024-04-15T18:32:55.313563+0000","last_fullsized":"2024-04-15T18:32:55.313563+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:47.630738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97845248,"num_objects":1493,"num_object_clones":0,"num_object_copies":4479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1493,"num_write_kb":95552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"550'1680","reported_seq":1708,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.290703+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:32:57.290703+0000","last_peered":"2024-04-15T18:32:57.290703+0000","last_clean":"2024-04-15T18:32:57.290703+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:32:57.290703+0000","last_undegraded":"2024-04-15T18:32:57.290703+0000","last_fullsized":"2024-04-15T18:32:57.290703+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1680,"log_dups_size":0,"ondisk_log_size":1680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.855955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":107520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"550'1641","reported_seq":1667,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:57.548219+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:32:57.548219+0000","last_peered":"2024-04-15T18:32:57.548219+0000","last_clean":"2024-04-15T18:32:57.548219+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:32:57.548219+0000","last_undegraded":"2024-04-15T18:32:57.548219+0000","last_fullsized":"2024-04-15T18:32:57.548219+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:45.545736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":4923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1366,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-04-15T18:32:55.546223+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:32:55.546223+0000","last_peered":"2024-04-15T18:32:55.546223+0000","last_clean":"2024-04-15T18:32:55.546223+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:32:55.546223+0000","last_undegraded":"2024-04-15T18:32:55.546223+0000","last_fullsized":"2024-04-15T18:32:55.546223+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:00.300641+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1691811840,"num_objects":25815,"num_object_clones":0,"num_object_copies":77445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25815,"num_write_kb":1652160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5102764032,"data_stored":5102764032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25815,"ondisk_log_size":25815,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650564,"kb_used_data":208444,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93094588,"statfs":{"total":95995035648,"available":95328858112,"internally_reserved":0,"allocated":213446656,"data_stored":213146668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":452721092},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187746,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":643072,"kb_used_data":323124,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93102080,"statfs":{"total":95995035648,"available":95336529920,"internally_reserved":0,"allocated":330878976,"data_stored":330582560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514791,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760956,"kb_used_data":422964,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92984196,"statfs":{"total":95995035648,"available":95215816704,"internally_reserved":0,"allocated":433115136,"data_stored":432818720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344464,"kb_used_data":186952,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93400688,"statfs":{"total":95995035648,"available":95642304512,"internally_reserved":0,"allocated":191438848,"data_stored":191111010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168880,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031288,"kb_used_data":693296,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92713864,"statfs":{"total":95995035648,"available":94938996736,"internally_reserved":0,"allocated":709935104,"data_stored":709639634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463223,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755448,"kb_used_data":373040,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92989704,"statfs":{"total":95995035648,"available":95221456896,"internally_reserved":0,"allocated":381992960,"data_stored":381697490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575187,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822272,"kb_used_data":416244,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92922880,"statfs":{"total":95995035648,"available":95153029120,"internally_reserved":0,"allocated":426233856,"data_stored":425937440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710448,"kb_used_data":286376,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93034704,"statfs":{"total":95995035648,"available":95267536896,"internally_reserved":0,"allocated":293249024,"data_stored":292957106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193959,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":568508,"kb_used_data":419892,"kb_used_omap":114,"kb_used_meta":148493,"kb_avail":93176644,"statfs":{"total":95995035648,"available":95412883456,"internally_reserved":0,"allocated":429969408,"data_stored":429672992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":152057073},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609089,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382140,"kb_used_data":1032436,"kb_used_omap":44,"kb_used_meta":349651,"kb_avail":92363012,"statfs":{"total":95995035648,"available":94579724288,"internally_reserved":0,"allocated":1057214464,"data_stored":1056918048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":358043178},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986495,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999200,"kb_used_data":616692,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92745952,"statfs":{"total":95995035648,"available":94971854848,"internally_reserved":0,"allocated":631492608,"data_stored":631196192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703390,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721920,"kb_used_data":512756,"kb_used_omap":4,"kb_used_meta":209147,"kb_avail":93023232,"statfs":{"total":95995035648,"available":95255789568,"internally_reserved":0,"allocated":525062144,"data_stored":524765728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":214167233},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305332224,"data_stored":305332224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":429195264,"data_stored":429195264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439222272,"data_stored":439222272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":861011968,"data_stored":861011968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":436600832,"data_stored":436600832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":540737536,"data_stored":540737536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":397934592,"data_stored":397934592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":507183104,"data_stored":507183104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":194117632,"data_stored":194117632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":435421184,"data_stored":435421184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":341311488,"data_stored":341311488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":214695936,"data_stored":214695936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:33:03.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 34257 34241 85.5948 8.0625 0.00276601 0.0112502 2024-04-15T18:33:04.118 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:04.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:04.158 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:04.181 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:04.205 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:04.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:04.230 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:04.257 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:04.260 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:04.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:33:04.312 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:04.336 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:04.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:04.366 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:04.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:04.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:33:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:04.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:04.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:04.486 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:04.508 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:04.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:04.644 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:04.665 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:04 smithi049 ceph-mon[40087]: pgmap v1439: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:33:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2216567346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:04.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:04.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:04.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:04 smithi082 ceph-mon[44206]: pgmap v1439: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:33:04.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2216567346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:04.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:04.796 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:04 smithi159 ceph-mon[45336]: pgmap v1439: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:33:04.796 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2216567346' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:04.797 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:04.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:04.819 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:04.923 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:04.954 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:04.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 34275 34259 82.3458 1.125 0.00157482 0.0112682 2024-04-15T18:33:04.976 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:04.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:05.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:33:05.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:05.111 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.131 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:05.235 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:05.266 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.287 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:05.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:05.423 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.445 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.549 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:05.578 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.599 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:05.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.731 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.751 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:05.855 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:05.882 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:05.901 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:05.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:05.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 34275 34259 79.2958 0 - 0.0112682 2024-04-15T18:33:06.005 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:06.030 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:06.049 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:06.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:06.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:06.183 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:06.207 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:06.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:06.228 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:06.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:06.339 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:33:06.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:33:06.538 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:06.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:06.643 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:06.671 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:06.692 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:06 smithi082 ceph-mon[44206]: pgmap v1440: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:33:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:06.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:06.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:06 smithi049 ceph-mon[40087]: pgmap v1440: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:33:06.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:06.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:06.823 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:06.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:06.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:06 smithi159 ceph-mon[45336]: pgmap v1440: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:33:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:06.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:06.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 34275 34259 76.4638 0 - 0.0112682 2024-04-15T18:33:06.987 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:07.008 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:07.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:07.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:07.143 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:07.165 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:07.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:07.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:07.298 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:07.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:07 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:07.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:07.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:07 smithi049 ceph-mon[40087]: osdmap e553: 12 total, 12 up, 12 in 2024-04-15T18:33:07.648 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:07.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:07.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:07.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:07 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:07.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:07.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:07 smithi082 ceph-mon[44206]: osdmap e553: 12 total, 12 up, 12 in 2024-04-15T18:33:07.784 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:07.806 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:07.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:07 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2569744740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:07 smithi159 ceph-mon[45336]: osdmap e553: 12 total, 12 up, 12 in 2024-04-15T18:33:07.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:07.938 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:07.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:33:07.966 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 34275 34259 73.827 0 - 0.0112682 2024-04-15T18:33:08.387 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:08.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:08.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:08.523 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:08.544 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:08.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:08.630 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:08.632 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:08.649 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:08.654 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:33:08.654 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:33:08.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:08 smithi049 ceph-mon[40087]: pgmap v1442: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-15T18:33:08.681 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:08.701 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:08.703 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:08 smithi159 ceph-mon[45336]: pgmap v1442: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-15T18:33:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:08.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:08.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:08 smithi082 ceph-mon[44206]: pgmap v1442: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-04-15T18:33:08.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:08.837 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:08.858 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:08.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:08.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:08.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 34275 34259 71.3661 0 - 0.0112682 2024-04-15T18:33:08.992 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:09.013 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:09.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:09.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:09.147 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:09.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:09.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:09.530 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:09.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:09.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:09.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 34275 34259 69.064 0 - 0.0112682 2024-04-15T18:33:10.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:33:10.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:33:10.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:10 smithi049 ceph-mon[40087]: pgmap v1443: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-04-15T18:33:10.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:10.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:10.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:10.681 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:10.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:10.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:10 smithi082 ceph-mon[44206]: pgmap v1443: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-04-15T18:33:10.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:10.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:10.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:10.785 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:10.816 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:10 smithi159 ceph-mon[45336]: pgmap v1443: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-04-15T18:33:10.816 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:10.816 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:10.816 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:10.818 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:10.840 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:10.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:10.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:10.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 34275 34259 66.9057 0 - 0.0112682 2024-04-15T18:33:10.976 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:10.997 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:11.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:11.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:11.129 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:11.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:11.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:11.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:11.510 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:11.531 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:11.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:11.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:11.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:11.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:11.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:11.666 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:11.678 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:11.686 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:11.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:11.703 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:11.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:11.753 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:33:11.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:11.789 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:11.802 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:11.822 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:11.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:11.843 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:11.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:11.852 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:11.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:11.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:11.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:11.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:11.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:11.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:11.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 34275 34259 64.8782 0 - 0.0112682 2024-04-15T18:33:11.986 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.135 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:12.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:12.271 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.292 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:12.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:12.428 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.450 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:12.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: pgmap v1444: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 589 op/s 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/151890496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:12.583 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.603 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:12.708 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: pgmap v1444: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 589 op/s 2024-04-15T18:33:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/151890496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.737 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.754 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:33:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:33:12.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:12.885 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: pgmap v1444: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 589 op/s 2024-04-15T18:33:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/151890496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:12.904 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:12.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:12.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 34275 34259 62.9699 0 - 0.0112682 2024-04-15T18:33:13.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:13.039 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.060 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:13.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:13.191 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.210 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:13.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:13.344 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.494 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:13.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:13.625 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.645 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:13.748 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:13 smithi159 ceph-mon[45336]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:13.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:13 smithi082 ceph-mon[44206]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:13.780 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.801 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:13.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:13 smithi049 ceph-mon[40087]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:33:13.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:13.932 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:13.952 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:13.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:13.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 34275 34259 61.1708 0 - 0.0112682 2024-04-15T18:33:14.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:14.088 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:14.108 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:14.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:14.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:14.244 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:14.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:14.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:14.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:14 smithi049 ceph-mon[40087]: pgmap v1445: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s wr, 132 op/s 2024-04-15T18:33:14.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:14 smithi049 ceph-mon[40087]: osdmap e554: 12 total, 12 up, 12 in 2024-04-15T18:33:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:14 smithi082 ceph-mon[44206]: pgmap v1445: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s wr, 132 op/s 2024-04-15T18:33:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:14 smithi082 ceph-mon[44206]: osdmap e554: 12 total, 12 up, 12 in 2024-04-15T18:33:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:14 smithi159 ceph-mon[45336]: pgmap v1445: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s wr, 132 op/s 2024-04-15T18:33:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:14 smithi159 ceph-mon[45336]: osdmap e554: 12 total, 12 up, 12 in 2024-04-15T18:33:14.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 34275 34259 59.4715 0 - 0.0112682 2024-04-15T18:33:15.297 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:15.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:15.401 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:15.434 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:15.455 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:15.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:15.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:15.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:15.614 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:15.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:15.715 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1444,"stamp":"2024-04-15T18:33:11.386321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2246836768,"num_objects":34277,"num_object_clones":0,"num_object_copies":102831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34277,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":34456,"num_write_kb":2196538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34414,"ondisk_log_size":34414,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11107512,"kb_used_data":7130428,"kb_used_omap":177,"kb_used_meta":3976782,"kb_avail":1113834312,"statfs":{"total":1151940427776,"available":1140566335488,"internally_reserved":0,"allocated":7301558272,"data_stored":7297850816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":181820,"internal_metadata":4072225220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,14,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":386727936,"num_objects":5901,"num_object_clones":0,"num_object_copies":17703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5901,"num_write_kb":377664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001913"},"pg_stats":[{"pgid":"6.f","version":"552'2206","reported_seq":2240,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.454233+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:33:06.454233+0000","last_peered":"2024-04-15T18:33:06.454233+0000","last_clean":"2024-04-15T18:33:06.454233+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:33:06.454233+0000","last_undegraded":"2024-04-15T18:33:06.454233+0000","last_fullsized":"2024-04-15T18:33:06.454233+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2206,"log_dups_size":0,"ondisk_log_size":2206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.454307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144572416,"num_objects":2206,"num_object_clones":0,"num_object_copies":6618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2206,"num_write_kb":141184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"552'2107","reported_seq":2138,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-04-15T18:33:03.459799+0000","last_change":"2024-04-15T18:32:30.477777+0000","last_active":"2024-04-15T18:33:03.459799+0000","last_peered":"2024-04-15T18:33:03.459799+0000","last_clean":"2024-04-15T18:33:03.459799+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:33:03.459799+0000","last_undegraded":"2024-04-15T18:33:03.459799+0000","last_fullsized":"2024-04-15T18:33:03.459799+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.795587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138084352,"num_objects":2107,"num_object_clones":0,"num_object_copies":6321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2107,"num_write_kb":134848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"552'2183","reported_seq":2217,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.552590+0000","last_change":"2024-04-15T18:32:30.640352+0000","last_active":"2024-04-15T18:33:06.552590+0000","last_peered":"2024-04-15T18:33:06.552590+0000","last_clean":"2024-04-15T18:33:06.552590+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:33:06.552590+0000","last_undegraded":"2024-04-15T18:33:06.552590+0000","last_fullsized":"2024-04-15T18:33:06.552590+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:36.209094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143065088,"num_objects":2183,"num_object_clones":0,"num_object_copies":6549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2183,"num_write_kb":139712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"552'2165","reported_seq":2199,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:08.962426+0000","last_change":"2024-04-15T18:32:30.479011+0000","last_active":"2024-04-15T18:33:08.962426+0000","last_peered":"2024-04-15T18:33:08.962426+0000","last_clean":"2024-04-15T18:33:08.962426+0000","last_became_active":"2024-04-15T18:32:30.478523+0000","last_became_peered":"2024-04-15T18:32:30.478523+0000","last_unstale":"2024-04-15T18:33:08.962426+0000","last_undegraded":"2024-04-15T18:33:08.962426+0000","last_fullsized":"2024-04-15T18:33:08.962426+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:57.752075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141885440,"num_objects":2165,"num_object_clones":0,"num_object_copies":6495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2165,"num_write_kb":138560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"552'2199","reported_seq":2233,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455469+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:33:06.455469+0000","last_peered":"2024-04-15T18:33:06.455469+0000","last_clean":"2024-04-15T18:33:06.455469+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:33:06.455469+0000","last_undegraded":"2024-04-15T18:33:06.455469+0000","last_fullsized":"2024-04-15T18:33:06.455469+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.581404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144113664,"num_objects":2199,"num_object_clones":0,"num_object_copies":6597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2199,"num_write_kb":140736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"552'2108","reported_seq":2142,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.552585+0000","last_change":"2024-04-15T18:32:30.982302+0000","last_active":"2024-04-15T18:33:06.552585+0000","last_peered":"2024-04-15T18:33:06.552585+0000","last_clean":"2024-04-15T18:33:06.552585+0000","last_became_active":"2024-04-15T18:32:30.981964+0000","last_became_peered":"2024-04-15T18:32:30.981964+0000","last_unstale":"2024-04-15T18:33:06.552585+0000","last_undegraded":"2024-04-15T18:33:06.552585+0000","last_fullsized":"2024-04-15T18:33:06.552585+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:34.969438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138149888,"num_objects":2108,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":134912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"552'2130","reported_seq":2164,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.661483+0000","last_change":"2024-04-15T18:32:30.982142+0000","last_active":"2024-04-15T18:33:06.661483+0000","last_peered":"2024-04-15T18:33:06.661483+0000","last_clean":"2024-04-15T18:33:06.661483+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:33:06.661483+0000","last_undegraded":"2024-04-15T18:33:06.661483+0000","last_fullsized":"2024-04-15T18:33:06.661483+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:30.349808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139591680,"num_objects":2130,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2130,"num_write_kb":136320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"552'2115","reported_seq":2149,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455469+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:33:06.455469+0000","last_peered":"2024-04-15T18:33:06.455469+0000","last_clean":"2024-04-15T18:33:06.455469+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:33:06.455469+0000","last_undegraded":"2024-04-15T18:33:06.455469+0000","last_fullsized":"2024-04-15T18:33:06.455469+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:38.538924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":135360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"552'2095","reported_seq":2126,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-04-15T18:33:03.464039+0000","last_change":"2024-04-15T18:32:30.480070+0000","last_active":"2024-04-15T18:33:03.464039+0000","last_peered":"2024-04-15T18:33:03.464039+0000","last_clean":"2024-04-15T18:33:03.464039+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:33:03.464039+0000","last_undegraded":"2024-04-15T18:33:03.464039+0000","last_fullsized":"2024-04-15T18:33:03.464039+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2095,"log_dups_size":0,"ondisk_log_size":2095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:53.238775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"552'2155","reported_seq":2173,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455492+0000","last_change":"2024-04-15T18:32:30.480197+0000","last_active":"2024-04-15T18:33:06.455492+0000","last_peered":"2024-04-15T18:33:06.455492+0000","last_clean":"2024-04-15T18:33:06.455492+0000","last_became_active":"2024-04-15T18:32:30.479842+0000","last_became_peered":"2024-04-15T18:32:30.479842+0000","last_unstale":"2024-04-15T18:33:06.455492+0000","last_undegraded":"2024-04-15T18:33:06.455492+0000","last_fullsized":"2024-04-15T18:33:06.455492+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.405063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":6465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":137920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"552'2070","reported_seq":2104,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455177+0000","last_change":"2024-04-15T18:32:30.477765+0000","last_active":"2024-04-15T18:33:06.455177+0000","last_peered":"2024-04-15T18:33:06.455177+0000","last_clean":"2024-04-15T18:33:06.455177+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:33:06.455177+0000","last_undegraded":"2024-04-15T18:33:06.455177+0000","last_fullsized":"2024-04-15T18:33:06.455177+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2070,"log_dups_size":0,"ondisk_log_size":2070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:52.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135659520,"num_objects":2070,"num_object_clones":0,"num_object_copies":6210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":132480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"552'2163","reported_seq":2197,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455590+0000","last_change":"2024-04-15T18:32:30.736024+0000","last_active":"2024-04-15T18:33:06.455590+0000","last_peered":"2024-04-15T18:33:06.455590+0000","last_clean":"2024-04-15T18:33:06.455590+0000","last_became_active":"2024-04-15T18:32:30.735671+0000","last_became_peered":"2024-04-15T18:32:30.735671+0000","last_unstale":"2024-04-15T18:33:06.455590+0000","last_undegraded":"2024-04-15T18:33:06.455590+0000","last_fullsized":"2024-04-15T18:33:06.455590+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:32.574745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":6489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":138432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"552'2126","reported_seq":2157,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-04-15T18:33:04.271813+0000","last_change":"2024-04-15T18:32:30.478009+0000","last_active":"2024-04-15T18:33:04.271813+0000","last_peered":"2024-04-15T18:33:04.271813+0000","last_clean":"2024-04-15T18:33:04.271813+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:33:04.271813+0000","last_undegraded":"2024-04-15T18:33:04.271813+0000","last_fullsized":"2024-04-15T18:33:04.271813+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:50.474140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139329536,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":136064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"552'2179","reported_seq":2212,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:07.683842+0000","last_change":"2024-04-15T18:32:30.639803+0000","last_active":"2024-04-15T18:33:07.683842+0000","last_peered":"2024-04-15T18:33:07.683842+0000","last_clean":"2024-04-15T18:33:07.683842+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:33:07.683842+0000","last_undegraded":"2024-04-15T18:33:07.683842+0000","last_fullsized":"2024-04-15T18:33:07.683842+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:47.630738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142802944,"num_objects":2179,"num_object_clones":0,"num_object_copies":6537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":139456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"552'2168","reported_seq":2202,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.454244+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:33:06.454244+0000","last_peered":"2024-04-15T18:33:06.454244+0000","last_clean":"2024-04-15T18:33:06.454244+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:33:06.454244+0000","last_undegraded":"2024-04-15T18:33:06.454244+0000","last_fullsized":"2024-04-15T18:33:06.454244+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:40.855955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":138752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"552'2106","reported_seq":2140,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.552588+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:33:06.552588+0000","last_peered":"2024-04-15T18:33:06.552588+0000","last_clean":"2024-04-15T18:33:06.552588+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:33:06.552588+0000","last_undegraded":"2024-04-15T18:33:06.552588+0000","last_fullsized":"2024-04-15T18:33:06.552588+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:45.545736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":134784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1372,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-04-15T18:33:06.455540+0000","last_change":"2024-04-15T18:31:56.204922+0000","last_active":"2024-04-15T18:33:06.455540+0000","last_peered":"2024-04-15T18:33:06.455540+0000","last_clean":"2024-04-15T18:33:06.455540+0000","last_became_active":"2024-04-15T18:09:37.624368+0000","last_became_peered":"2024-04-15T18:09:37.624368+0000","last_unstale":"2024-04-15T18:33:06.455540+0000","last_undegraded":"2024-04-15T18:33:06.455540+0000","last_fullsized":"2024-04-15T18:33:06.455540+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:31:56.204785+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:31:56.204785+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:33:06.471471+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2246246400,"num_objects":34275,"num_object_clones":0,"num_object_copies":102825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34275,"num_write_kb":2193600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6737690624,"data_stored":6737690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34275,"ondisk_log_size":34275,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926111,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721372,"kb_used_data":275092,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93023780,"statfs":{"total":95995035648,"available":95256350720,"internally_reserved":0,"allocated":281694208,"data_stored":281385224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":456980932},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187748,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767444,"kb_used_data":421132,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92977708,"statfs":{"total":95995035648,"available":95209172992,"internally_reserved":0,"allocated":431239168,"data_stored":430933756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514793,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900964,"kb_used_data":554648,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92844188,"statfs":{"total":95995035648,"available":95072448512,"internally_reserved":0,"allocated":567959552,"data_stored":567649592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438244,"kb_used_data":276568,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93306908,"statfs":{"total":95995035648,"available":95546273792,"internally_reserved":0,"allocated":283205632,"data_stored":282895672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168883,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201516,"kb_used_data":867680,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92543636,"statfs":{"total":95995035648,"available":94764683264,"internally_reserved":0,"allocated":888504320,"data_stored":888190808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":341834186},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,14,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929388,"kb_used_data":551136,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92815764,"statfs":{"total":95995035648,"available":95043342336,"internally_reserved":0,"allocated":564363264,"data_stored":564049752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575189,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962916,"kb_used_data":548568,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92782236,"statfs":{"total":95995035648,"available":95009009664,"internally_reserved":0,"allocated":561733632,"data_stored":561423672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738710,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843156,"kb_used_data":410764,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92901996,"statfs":{"total":95995035648,"available":95131643904,"internally_reserved":0,"allocated":420622336,"data_stored":420316924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193961,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708580,"kb_used_data":551640,"kb_used_omap":114,"kb_used_meta":156813,"kb_avail":93036572,"statfs":{"total":95995035648,"available":95269449728,"internally_reserved":0,"allocated":564879360,"data_stored":564569400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":160576753},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609091,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1633556,"kb_used_data":1265804,"kb_used_omap":44,"kb_used_meta":367699,"kb_avail":92111596,"statfs":{"total":95995035648,"available":94322274304,"internally_reserved":0,"allocated":1296183296,"data_stored":1295877884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":376524330},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986497,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1118292,"kb_used_data":726156,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92626860,"statfs":{"total":95995035648,"available":94849904640,"internally_reserved":0,"allocated":743583744,"data_stored":743278332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703392,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882084,"kb_used_data":681240,"kb_used_omap":4,"kb_used_meta":200827,"kb_avail":92863068,"statfs":{"total":95995035648,"available":95091781632,"internally_reserved":0,"allocated":697589760,"data_stored":697279800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":205647553},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419823616,"data_stored":419823616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":560922624,"data_stored":560922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":564068352,"data_stored":564068352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1116995584,"data_stored":1116995584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":552730624,"data_stored":552730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":696778752,"data_stored":696778752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":563019776,"data_stored":563019776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":703004672,"data_stored":703004672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":282394624,"data_stored":282394624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":567148544,"data_stored":567148544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":430440448,"data_stored":430440448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":280363008,"data_stored":280363008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:33:15.716 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:33:15.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:15.749 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:15.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:15.967 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 34275 34259 57.864 0 - 0.0112682 2024-04-15T18:33:15.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:16.004 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:16.155 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:16.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:16.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:16.291 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:16.312 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:16.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:16.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:16.448 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:16.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:16 smithi049 ceph-mon[40087]: pgmap v1447: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 21 op/s 2024-04-15T18:33:16.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:16 smithi082 ceph-mon[44206]: pgmap v1447: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 21 op/s 2024-04-15T18:33:16.861 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:33:16.861 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:33:16.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81983 5.1 5:8c9a41f5:::benchmark_data_smithi159_116570_object81982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.39701923199999, 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.203960954, 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:16.862 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81983 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471371+0000", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9400000000000001e-07 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471416+0000", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5123000000000001e-05 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471423+0000", 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.058e-06 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471427+0000", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7910000000000002e-06 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472653+0000", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001225711 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472657+0000", 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9720000000000003e-06 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624096+0000", 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15143899299999999 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624272+0000", 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017645900000000001 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:33:16.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.624432+0000", 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160011 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639480+0000", 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015047414 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.671266+0000", 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031786552000000003 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.671312+0000", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5293000000000002e-05 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673194+0000", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018826660000000001 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673258+0000", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3279000000000002e-05 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675332+0000", 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020742379999999999 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81985 5.1 5:8cf02e22:::benchmark_data_smithi159_116570_object81984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.39695756600003, 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20675183599999999, 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.869 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81985 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471433+0000", 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9299999999999999e-07 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471475+0000", 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2219999999999999e-05 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471481+0000", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2659999999999998e-06 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471485+0000", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3610000000000001e-06 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472676+0000", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001191613 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472680+0000", 2024-04-15T18:33:16.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.861e-06 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.626705+0000", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15402469899999999 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.626866+0000", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160929 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.627037+0000", 2024-04-15T18:33:16.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017111900000000001 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639490+0000", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012453331999999999 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673060+0000", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033569977000000001 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.673139+0000", 2024-04-15T18:33:16.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9054999999999994e-05 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675506+0000", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002366983 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675584+0000", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7785000000000001e-05 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678184+0000", 2024-04-15T18:33:16.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026002429999999999 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82009 5.1 5:894d6e9c:::benchmark_data_smithi159_116570_object82008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.39690040200003, 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.209574175, 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:16.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82009 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471490+0000", 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3799999999999999e-07 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471527+0000", 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6906000000000002e-05 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471533+0000", 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0700000000000003e-06 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471536+0000", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3440000000000001e-06 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472699+0000", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011631460000000001 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472703+0000", 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9389999999999999e-06 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629581+0000", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15687794899999999 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629719+0000", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013762600000000001 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.629866+0000", 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014707300000000001 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639639+0000", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097726110000000005 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675401+0000", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.035762032999999999 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.675457+0000", 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6598999999999998e-05 2024-04-15T18:33:16.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678355+0000", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028978160000000001 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678424+0000", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9103999999999994e-05 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681064+0000", 2024-04-15T18:33:16.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026397209999999998 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82035 5.1 5:83347078:::benchmark_data_smithi159_116570_object82034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.39684932699998, 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21252658299999999, 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:16.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82035 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471541+0000", 2024-04-15T18:33:16.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4299999999999999e-07 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471594+0000", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3236000000000003e-05 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471604+0000", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0118999999999999e-05 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471608+0000", 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9720000000000003e-06 2024-04-15T18:33:16.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472725+0000", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011170900000000001 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472729+0000", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9280000000000003e-06 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632068+0000", 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15933883600000001 2024-04-15T18:33:16.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632211+0000", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142848 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.632315+0000", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103732 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639649+0000", 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073340530000000001 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678251+0000", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.038601997999999998 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.678302+0000", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0932e-05 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681246+0000", 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029446229999999999 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681321+0000", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4324000000000002e-05 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684067+0000", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027466489999999999 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:82070 5.1 5:8955e426:::benchmark_data_smithi159_116570_object82069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e402)", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:33:16.889 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 434.39677562499998, 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21553371599999999, 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 82070 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:33:16.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471615+0000", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7000000000000001e-07 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471677+0000", 2024-04-15T18:33:16.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2382e-05 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471684+0000", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2139999999999999e-06 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.471688+0000", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7280000000000001e-06 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472743+0000", 2024-04-15T18:33:16.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010547969999999999 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.472747+0000", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0280000000000002e-06 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.634900+0000", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16215306800000001 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.635056+0000", 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015597000000000001 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,6,11", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.635243+0000", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018687900000000001 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.639659+0000", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044159179999999996 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681129+0000", 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.041470411999999998 2024-04-15T18:33:16.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.681188+0000", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8548000000000003e-05 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684230+0000", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030423719999999998 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.684300+0000", 2024-04-15T18:33:16.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9839999999999995e-05 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:02.687148+0000", 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028482899999999998 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPull(5.1 e428/416 cost 1049576)", 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 363.99554848700001, 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39438863499999999, 2024-04-15T18:33:16.896 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872842+0000", 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872843+0000", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.048e-06 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:12.872856+0000", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3015e-05 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266896+0000", 2024-04-15T18:33:16.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39404034100000002 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266905+0000", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2920000000000001e-06 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.266985+0000", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9593999999999997e-05 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:13.267230+0000", 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000245345 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:10294 6.4 e547/541 6:2f2a2346:::benchmark_data_smithi159_139342_object10293:head v 547'637, mlcod=547'633)", 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:45.759826+0000", 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.108563958000001, 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15048423399999999, 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.759826+0000", 2024-04-15T18:33:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.759826+0000", 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.759827+0000", 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4000000000000001e-07 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.759847+0000", 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.969e-05 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.909671+0000", 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149824759 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.909676+0000", 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0760000000000002e-06 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.909724+0000", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7939000000000003e-05 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.909740+0000", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5851000000000001e-05 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.910283+0000", 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054274900000000003 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:45.910310+0000", 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7529999999999999e-05 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:17341 6.0 e547/541 6:0e82fc9f:::benchmark_data_smithi159_139342_object17340:head v 547'1089, mlcod=547'1086)", 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:50.455374+0000", 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.413015875999999, 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.200475917, 2024-04-15T18:33:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.455374+0000", 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.455374+0000", 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.455375+0000", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1500000000000004e-07 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.455391+0000", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6081000000000001e-05 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.654748+0000", 2024-04-15T18:33:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.199356544 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.654752+0000", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9540000000000003e-06 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.654835+0000", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2633999999999995e-05 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.654861+0000", 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5466e-05 2024-04-15T18:33:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.655807+0000", 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00094676999999999999 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.655850+0000", 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2852999999999998e-05 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:17958 6.c e548/541 6:38fc1ff2:::benchmark_data_smithi159_139342_object17957:head v 548'1139, mlcod=548'1136)", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:50.977313+0000", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 25.891077243000002, 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20861687800000001, 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.977313+0000", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.977313+0000", 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.977313+0000", 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7999999999999995e-07 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:50.977383+0000", 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9935e-05 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185004+0000", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.207620215 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185011+0000", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0230000000000004e-06 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185069+0000", 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8522e-05 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185095+0000", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5633e-05 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185898+0000", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080298999999999995 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.185930+0000", 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1980000000000002e-05 2024-04-15T18:33:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:19550 6.c e548/541 6:3adcc4d9:::benchmark_data_smithi159_139342_object19549:head v 548'1247, mlcod=548'1242)", 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:51.959426+0000", 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 24.908964042000001, 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.123942553, 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.959426+0000", 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.959426+0000", 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.959427+0000", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9599999999999999e-07 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:51.959494+0000", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7609999999999998e-05 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.082621+0000", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12312680199999999 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.082626+0000", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.082664+0000", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7620999999999998e-05 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.082677+0000", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3703000000000001e-05 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.083342+0000", 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066476100000000004 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:52.083369+0000", 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6420000000000001e-05 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:26879 6.2 e550/541 6:4ba4b07f:::benchmark_data_smithi159_139342_object26878:head v 550'1710, mlcod=550'1708)", 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:33:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.116831315999999, 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22823505999999999, 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751560+0000", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9199999999999999e-07 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751592+0000", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2632000000000002e-05 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978825+0000", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22723285100000001 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978831+0000", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.57e-06 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978884+0000", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3667e-05 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978900+0000", 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5863e-05 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.979741+0000", 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000840907 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.979794+0000", 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2677999999999999e-05 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:26903 6.4 e550/541 6:20efec16:::benchmark_data_smithi159_139342_object26902:head v 550'1619, mlcod=550'1616)", 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:57.784980+0000", 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.083410345000001, 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18867650599999999, 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.784980+0000", 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.784980+0000", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.784981+0000", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4399999999999999e-07 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.785002+0000", 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1393999999999999e-05 2024-04-15T18:33:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.972591+0000", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18758952100000001 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.972598+0000", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2979999999999997e-06 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.972650+0000", 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2017e-05 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.972669+0000", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8921999999999999e-05 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.973617+0000", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000948361 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.973656+0000", 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9249000000000003e-05 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:30320 6.c e551/541 6:38828b0e:::benchmark_data_smithi159_139342_object30319:head v 551'1920, mlcod=551'1916)", 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:00.158791+0000", 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.709599523000001, 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13309099499999999, 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.158791+0000", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.158791+0000", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.158791+0000", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.13e-07 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.158839+0000", 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7494000000000003e-05 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.290787+0000", 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13194874600000001 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.290795+0000", 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.106e-06 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.290838+0000", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2212000000000002e-05 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.290853+0000", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5014000000000001e-05 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.291848+0000", 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00099507999999999997 2024-04-15T18:33:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.291882+0000", 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.383e-05 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:30332 6.c e551/541 6:3052e48b:::benchmark_data_smithi159_139342_object30331:head v 551'1921, mlcod=551'1917)", 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:00.167954+0000", 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.700436580000002, 2024-04-15T18:33:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160846671, 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.167954+0000", 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.167954+0000", 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.167955+0000", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.034e-06 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.168000+0000", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5074000000000002e-05 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.327785+0000", 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.159785451 2024-04-15T18:33:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.327791+0000", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4079999999999997e-06 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.327830+0000", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9475999999999999e-05 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.327843+0000", 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3195e-05 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.328764+0000", 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00092054099999999998 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:00.328800+0000", 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492000000000002e-05 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:34040 6.4 e552/541 6:2e3e46ae:::benchmark_data_smithi159_139342_object34039:head v 552'2056, mlcod=552'2053)", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.558214+0000", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 14.310176099, 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15577693500000001, 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.558214+0000", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.558214+0000", 2024-04-15T18:33:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.558215+0000", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4600000000000004e-07 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.558235+0000", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0560999999999998e-05 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.712855+0000", 2024-04-15T18:33:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15461930700000001 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.712860+0000", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2120000000000002e-06 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.712917+0000", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6954999999999999e-05 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.712938+0000", 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1679999999999999e-05 2024-04-15T18:33:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.713956+0000", 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010179099999999999 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.713991+0000", 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4663999999999999e-05 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:34090 6.7 e552/541 6:ebf6abfd:::benchmark_data_smithi159_139342_object34089:head v 552'2092, mlcod=552'2090)", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.743270+0000", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 14.12511973, 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19587025399999999, 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.743270+0000", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.743270+0000", 2024-04-15T18:33:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.743272+0000", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5e-06 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.743321+0000", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9082000000000002e-05 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.937978+0000", 2024-04-15T18:33:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.194657148 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.937986+0000", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.52e-06 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.938036+0000", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0781000000000001e-05 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.938068+0000", 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1251000000000003e-05 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.939093+0000", 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001024912 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.939141+0000", 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8059999999999997e-05 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:34121 6.4 e552/541 6:26243d18:::benchmark_data_smithi159_139342_object34120:head v 552'2060, mlcod=552'2057)", 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 13.931589008, 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.259943007, 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936802+0000", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936821+0000", 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9222999999999999e-05 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195530+0000", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25870855500000001 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195535+0000", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.57e-06 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195587+0000", 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1397000000000003e-05 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195603+0000", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6858999999999999e-05 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.196691+0000", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010880200000000001 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.196744+0000", 2024-04-15T18:33:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2673999999999997e-05 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:34146 6.7 e552/541 6:e12c820b:::benchmark_data_smithi159_139342_object34145:head v 552'2094, mlcod=552'2091)", 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:03.045169+0000", 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 13.823220940000001, 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149048659, 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.045169+0000", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.045169+0000", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.045170+0000", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7199999999999998e-07 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.045216+0000", 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6242999999999998e-05 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.192910+0000", 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14769343800000001 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.192925+0000", 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5362999999999998e-05 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.193045+0000", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001197 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.193082+0000", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7335000000000001e-05 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.194182+0000", 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010997719999999999 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.194218+0000", 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6136000000000001e-05 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:16.951 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:33:16.952 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:16 smithi159 ceph-mon[45336]: pgmap v1447: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s wr, 21 op/s 2024-04-15T18:33:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:16.952 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:16.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:16.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 34275 34259 56.3413 0 - 0.0112682 2024-04-15T18:33:17.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:17.083 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:17.102 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:17.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:17.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:33:17.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:17.238 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:17.259 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:17.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:17.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:17.393 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:33:17.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:33:17.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:33:17.805 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:33:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:33:17.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 35037 35021 56.1177 3.66346 0.00601804 0.0178136 2024-04-15T18:33:18.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:18.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:18.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:18.520 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:18.547 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:18.587 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:18.626 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:18.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:18 smithi049 ceph-mon[40087]: pgmap v1448: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 10 op/s 2024-04-15T18:33:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:18 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:33:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:18 smithi049 ceph-mon[40087]: osdmap e555: 12 total, 12 up, 12 in 2024-04-15T18:33:18.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:18.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:18 smithi159 ceph-mon[45336]: pgmap v1448: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 10 op/s 2024-04-15T18:33:18.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:18 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:18.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:33:18.762 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:18 smithi159 ceph-mon[45336]: osdmap e555: 12 total, 12 up, 12 in 2024-04-15T18:33:18.765 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:18.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:18 smithi082 ceph-mon[44206]: pgmap v1448: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s wr, 10 op/s 2024-04-15T18:33:18.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:18 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:18.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2760351568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:33:18.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:18 smithi082 ceph-mon[44206]: osdmap e555: 12 total, 12 up, 12 in 2024-04-15T18:33:18.789 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:18.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:18.892 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:18.920 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:18.939 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:18.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:18.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:33:18.977633+0000 min lat: 0.00106491 max lat: 15.0148 avg lat: 0.0174533 2024-04-15T18:33:18.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:33:18.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 36608 36592 57.1692 98.1875 0.0024129 0.0174533 2024-04-15T18:33:19.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:19.072 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:19.152 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:19.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:19.257 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:19.288 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:19.311 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:19.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:19.416 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:19.445 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:19.467 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:19.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:19.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:19.615 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:19 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:33:19.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:19 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:33:19.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:19 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:33:19.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:19 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:33:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:19 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:33:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:19 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:33:19.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 38229 38213 58.2456 101.312 0.00238361 0.0171634 2024-04-15T18:33:19.972 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:19.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:20.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:20.122 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:20.152 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:20.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:20.257 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:20.295 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:20.322 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:20.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:20.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:20.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:33:20.456 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:20.479 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:20.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:20.583 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:20.612 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:20.632 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:20.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:20.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:20.763 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:20 smithi049 ceph-mon[40087]: pgmap v1450: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s wr, 8 op/s 2024-04-15T18:33:20.765 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:20 smithi082 ceph-mon[44206]: pgmap v1450: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s wr, 8 op/s 2024-04-15T18:33:20.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:20 smithi159 ceph-mon[45336]: pgmap v1450: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s wr, 8 op/s 2024-04-15T18:33:20.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 39512 39496 58.7678 80.1875 0.0637454 0.0169522 2024-04-15T18:33:21.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 40837 40821 59.3268 82.8125 0.00458817 0.0168504 2024-04-15T18:33:22.044 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:22.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:22.171 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:22.197 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:22.217 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:22.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:22.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:22.358 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:22.382 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:22.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:22.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:22.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:22 smithi082 ceph-mon[44206]: pgmap v1451: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 253 op/s 2024-04-15T18:33:22.488 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:22.489 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:22.489 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:22 smithi082 ceph-mon[44206]: osdmap e556: 12 total, 12 up, 12 in 2024-04-15T18:33:22.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:22.552 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:22.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:22.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:22 smithi049 ceph-mon[40087]: pgmap v1451: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 253 op/s 2024-04-15T18:33:22.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:22.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:22.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:22 smithi049 ceph-mon[40087]: osdmap e556: 12 total, 12 up, 12 in 2024-04-15T18:33:22.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:22.698 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:22.723 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:22.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:22.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:22.872 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:22.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:22 smithi159 ceph-mon[45336]: pgmap v1451: 17 pgs: 17 active+clean; 2.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 253 op/s 2024-04-15T18:33:22.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:22.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:22.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:22 smithi159 ceph-mon[45336]: osdmap e556: 12 total, 12 up, 12 in 2024-04-15T18:33:22.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 41915 41899 59.5096 67.375 0.00459742 0.016795 2024-04-15T18:33:23.127 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26224.0:41828 6.f 6:f005c57e:::benchmark_data_smithi159_139342_object41827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:22.905776+0000", 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.020560897000000002, 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020595025999999999, 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26224", 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 41828 2024-04-15T18:33:23.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905776+0000", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905776+0000", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905778+0000", 2024-04-15T18:33:23.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.781e-06 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905876+0000", 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7657000000000004e-05 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905889+0000", 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3791000000000001e-05 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:23.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905897+0000", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0229999999999992e-06 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.905958+0000", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0695000000000001e-05 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.906064+0000", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106124 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:33:23.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.906158+0000", 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4168999999999999e-05 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.908491+0000", 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023327479999999999 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26224.0:41847 6.0 6:0e8edf8c:::benchmark_data_smithi159_139342_object41846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:22.923071+0000", 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0032657989999999998, 2024-04-15T18:33:23.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033921469999999999, 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26224", 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 41847 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923071+0000", 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923071+0000", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923072+0000", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.127e-06 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923130+0000", 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8035999999999998e-05 2024-04-15T18:33:23.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923142+0000", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1195999999999999e-05 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923148+0000", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1040000000000003e-06 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923200+0000", 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2428e-05 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923312+0000", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111826 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.923411+0000", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8618999999999999e-05 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.924750+0000", 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001339104 2024-04-15T18:33:23.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925414+0000", 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066374299999999995 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26224.0:41849 6.f 6:f2c21169:::benchmark_data_smithi159_139342_object41848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e556)", 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:22.925531+0000", 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00080662699999999997, 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010225869999999999, 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:23.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26224", 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 41849 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925531+0000", 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925531+0000", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925531+0000", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0400000000000005e-07 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925590+0000", 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8492999999999997e-05 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925599+0000", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.901e-06 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925603+0000", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.651e-06 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925652+0000", 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8213000000000001e-05 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925764+0000", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112229 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.925883+0000", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011930900000000001 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26224.0:41847 6.0 e556/541 ondisk, result = 0)", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:22.926497+0000", 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 4294967295.9998398, 2024-04-15T18:33:23.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013287999999999999, 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926497+0000", 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926497+0000", 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:23.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926498+0000", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8599999999999997e-07 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926503+0000", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5990000000000004e-06 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926531+0000", 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8164999999999999e-05 2024-04-15T18:33:23.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926537+0000", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8939999999999998e-06 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926586+0000", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9549999999999998e-05 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:22.926594+0000", 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3780000000000003e-06 2024-04-15T18:33:23.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 4 2024-04-15T18:33:23.145 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:33:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.146 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:23.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:23.247 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:23.278 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:23.301 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:23.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:23.435 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:23.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:23.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:23.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:23.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:23.457 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:23.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:23.591 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:23.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4247942799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:23.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.668 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:23.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4247942799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:23.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:23.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4247942799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:23.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:23.803 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:23.824 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:23.928 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:23.959 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:23.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 43435 43419 60.2981 95 0.00458661 0.0165778 2024-04-15T18:33:23.985 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:23.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:24.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:24.130 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:24.586 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:24.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:24.692 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:24.721 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:24.741 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:24.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: pgmap v1453: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:33:24.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: pgmap v1453: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:33:24.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.876 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:24.896 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:24.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:24.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: pgmap v1453: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s 2024-04-15T18:33:24.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:24.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 44987 44971 61.0958 97 0.00580334 0.0163617 2024-04-15T18:33:25.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:25.031 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:25.052 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:25.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:25.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:25.183 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:25.203 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:25.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:25.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:25.335 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:25.357 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:25.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:25.394 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:25.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:33:25.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:25.506 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:25.641 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:25.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:25.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:25.773 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:25.873 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:25.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:25.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 45827 45811 60.9128 52.5 0.00577757 0.0164101 2024-04-15T18:33:25.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:26.017 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:26.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:33:26.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:26 smithi049 ceph-mon[40087]: pgmap v1454: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:33:26.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:26.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:26.671 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:26.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:26.679 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:26.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:26.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:26 smithi082 ceph-mon[44206]: pgmap v1454: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:33:26.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:26.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:26.805 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:26.834 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:26.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:26.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:26 smithi159 ceph-mon[45336]: pgmap v1454: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:33:26.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:26.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:26.935 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:26.965 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:26.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 46840 46824 60.9627 63.3125 0.0152434 0.0163929 2024-04-15T18:33:26.989 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:26.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:27.095 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:27.143 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:27.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:27.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:27.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:27.309 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:27.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:27.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:27.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:27.467 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:27.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:27.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:27.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:27.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:27.607 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:27 smithi082 ceph-mon[44206]: osdmap e557: 12 total, 12 up, 12 in 2024-04-15T18:33:27.608 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:27 smithi082 ceph-mon[44206]: osdmap e558: 12 total, 12 up, 12 in 2024-04-15T18:33:27.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:27.635 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:27.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:27.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:27.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:27.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:27.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:27 smithi049 ceph-mon[40087]: osdmap e557: 12 total, 12 up, 12 in 2024-04-15T18:33:27.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:27 smithi049 ceph-mon[40087]: osdmap e558: 12 total, 12 up, 12 in 2024-04-15T18:33:27.799 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:27.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:27.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:27.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:27 smithi159 ceph-mon[45336]: osdmap e557: 12 total, 12 up, 12 in 2024-04-15T18:33:27.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:27 smithi159 ceph-mon[45336]: osdmap e558: 12 total, 12 up, 12 in 2024-04-15T18:33:27.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 48074 48058 61.2924 77.125 0.00451578 0.0162543 2024-04-15T18:33:28.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:28.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:28.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:33:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:28.539 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:33:28.541 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:33:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:28.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:33:28.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:28.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:28 smithi049 ceph-mon[40087]: pgmap v1457: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:33:28.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1441670453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:28 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.685 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:28 smithi082 ceph-mon[44206]: pgmap v1457: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:33:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1441670453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:28 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:28 smithi159 ceph-mon[45336]: pgmap v1457: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:33:28.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1441670453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:28 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:28.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 48782 48766 60.9514 44.25 0.00259477 0.0163624 2024-04-15T18:33:29.239 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:29.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:29.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:29.371 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:29.394 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:29.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:29.499 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:29.526 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:29.547 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:29.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:29.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:29 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:33:29.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:29 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:33:29.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:29 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:29.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:29 smithi159 ceph-mon[45336]: osdmap e559: 12 total, 12 up, 12 in 2024-04-15T18:33:29.651 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:29.677 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:29.696 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:29.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:29.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:29 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:33:29.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:29 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:33:29.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:29 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:29.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:29 smithi082 ceph-mon[44206]: osdmap e559: 12 total, 12 up, 12 in 2024-04-15T18:33:29.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:29.831 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:29.853 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:29.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:29.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:29 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:33:29.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:29 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:33:29.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:29 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:29.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:29 smithi049 ceph-mon[40087]: osdmap e559: 12 total, 12 up, 12 in 2024-04-15T18:33:29.920 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:29.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:29.968 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 49803 49787 61.0074 63.8125 0.00907721 0.0163862 2024-04-15T18:33:29.986 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:30.007 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:30.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:30.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:30.139 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:30.162 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:30.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:30.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:30.318 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:30.381 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:30.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:30.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:30.531 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:30 smithi049 ceph-mon[40087]: pgmap v1459: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-15T18:33:30.536 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:30.590 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:33:30.590 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:33:30.601 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:30.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:30.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:30.737 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:30.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:30 smithi082 ceph-mon[44206]: pgmap v1459: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-15T18:33:30.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:30 smithi159 ceph-mon[45336]: pgmap v1459: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-04-15T18:33:30.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 51152 51136 61.4554 84.3125 0.0176559 0.0162619 2024-04-15T18:33:31.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:31.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:31.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:31 smithi159 ceph-mon[45336]: osdmap e560: 12 total, 12 up, 12 in 2024-04-15T18:33:31.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:33:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:31 smithi049 ceph-mon[40087]: osdmap e560: 12 total, 12 up, 12 in 2024-04-15T18:33:31.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 52215 52199 61.5493 66.4375 0.00569075 0.016241 2024-04-15T18:33:32.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:32.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:32.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:31 smithi082 ceph-mon[44206]: osdmap e560: 12 total, 12 up, 12 in 2024-04-15T18:33:32.372 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:32.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:32.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:32.532 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:32.558 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:32.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:32.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:32.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:32 smithi049 ceph-mon[40087]: pgmap v1461: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:33:32.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:32 smithi049 ceph-mon[40087]: osdmap e561: 12 total, 12 up, 12 in 2024-04-15T18:33:32.702 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:32.731 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:32.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:32.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:32.880 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:32 smithi159 ceph-mon[45336]: pgmap v1461: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:33:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:32 smithi159 ceph-mon[45336]: osdmap e561: 12 total, 12 up, 12 in 2024-04-15T18:33:32.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 53611 53595 62.0251 87.25 0.00963618 0.0161173 2024-04-15T18:33:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:32.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:33.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:32 smithi082 ceph-mon[44206]: pgmap v1461: 17 pgs: 17 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:33:33.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:32 smithi082 ceph-mon[44206]: osdmap e561: 12 total, 12 up, 12 in 2024-04-15T18:33:33.066 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:33.091 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:33.101 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:33.119 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:33.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:33.142 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:33.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:33.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:33.203 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:33.242 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:33.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:33.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:33:33.280 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:33.308 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:33.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:33.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:33.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:33.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:33.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:33.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/5881038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:33.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:33.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/5881038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:33.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:33.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 54822 54806 62.2734 75.6875 0.0122799 0.0160528 2024-04-15T18:33:34.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/5881038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:34.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.135 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:34.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:34.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:33:34.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:34.273 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:34.295 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:34.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:34.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:34.429 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:34.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: pgmap v1463: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:34 smithi049 ceph-mon[40087]: 6.9 scrub starts 2024-04-15T18:33:34.724 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:34.782 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:34.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:34.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: pgmap v1463: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:34 smithi159 ceph-mon[45336]: 6.9 scrub starts 2024-04-15T18:33:34.937 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:34.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 55866 55850 62.3264 65.25 0.0088749 0.0160399 2024-04-15T18:33:34.986 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:34.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: pgmap v1463: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:33:35.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:35.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:34 smithi082 ceph-mon[44206]: 6.9 scrub starts 2024-04-15T18:33:35.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:35.136 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:35.224 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:35.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:35.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:35.355 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:35.509 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:35.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:35 smithi049 ceph-mon[40087]: 6.9 scrub ok 2024-04-15T18:33:35.717 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-15T18:33:35.718 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 4, 6, 7, 8, 9, 10, 11, 3, 2, 5] 2024-04-15T18:33:35.718 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-15T18:33:35.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6 2024-04-15T18:33:35.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:35 smithi159 ceph-mon[45336]: 6.9 scrub ok 2024-04-15T18:33:35.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 56902 56886 62.3688 64.75 0.0058608 0.0160278 2024-04-15T18:33:36.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:35 smithi082 ceph-mon[44206]: 6.9 scrub ok 2024-04-15T18:33:36.030 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:33:35 smithi082 systemd[1]: Stopping Ceph osd.6 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:33:36.528 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[65336]: 2024-04-15T18:33:36.143+0000 7fcd71113700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:33:36.529 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[65336]: 2024-04-15T18:33:36.143+0000 7fcd71113700 -1 osd.6 561 *** Got signal Terminated *** 2024-04-15T18:33:36.529 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[65336]: 2024-04-15T18:33:36.143+0000 7fcd71113700 -1 osd.6 561 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:33:36.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:33:36.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:36 smithi049 ceph-mon[40087]: pgmap v1464: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:33:36.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:36 smithi049 ceph-mon[40087]: 6.a deep-scrub starts 2024-04-15T18:33:36.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:36 smithi049 ceph-mon[40087]: osd.6 marked itself down and dead 2024-04-15T18:33:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:36 smithi159 ceph-mon[45336]: pgmap v1464: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:33:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:36 smithi159 ceph-mon[45336]: 6.a deep-scrub starts 2024-04-15T18:33:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:36 smithi159 ceph-mon[45336]: osd.6 marked itself down and dead 2024-04-15T18:33:36.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 57393 57377 61.8226 30.6875 0.175114 0.0160681 2024-04-15T18:33:37.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-mon[44206]: pgmap v1464: 17 pgs: 17 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-04-15T18:33:37.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-mon[44206]: 6.a deep-scrub starts 2024-04-15T18:33:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:36 smithi082 ceph-mon[44206]: osd.6 marked itself down and dead 2024-04-15T18:33:37.105 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:37.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:37.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:37.690 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:37.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:37.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:33:37 smithi082 bash[145198]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6 2024-04-15T18:33:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:37 smithi082 ceph-mon[44206]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:33:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:37 smithi082 ceph-mon[44206]: osdmap e562: 12 total, 11 up, 12 in 2024-04-15T18:33:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:37 smithi082 ceph-mon[44206]: 6.d scrub starts 2024-04-15T18:33:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:37 smithi082 ceph-mon[44206]: 6.d scrub ok 2024-04-15T18:33:37.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:37 smithi082 ceph-mon[44206]: osdmap e563: 12 total, 11 up, 12 in 2024-04-15T18:33:37.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:37.837 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:37 smithi159 ceph-mon[45336]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:33:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:37 smithi159 ceph-mon[45336]: osdmap e562: 12 total, 11 up, 12 in 2024-04-15T18:33:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:37 smithi159 ceph-mon[45336]: 6.d scrub starts 2024-04-15T18:33:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:37 smithi159 ceph-mon[45336]: 6.d scrub ok 2024-04-15T18:33:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:37 smithi159 ceph-mon[45336]: osdmap e563: 12 total, 11 up, 12 in 2024-04-15T18:33:37.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:37 smithi049 ceph-mon[40087]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:33:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:37 smithi049 ceph-mon[40087]: osdmap e562: 12 total, 11 up, 12 in 2024-04-15T18:33:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:37 smithi049 ceph-mon[40087]: 6.d scrub starts 2024-04-15T18:33:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:37 smithi049 ceph-mon[40087]: 6.d scrub ok 2024-04-15T18:33:37.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:37 smithi049 ceph-mon[40087]: osdmap e563: 12 total, 11 up, 12 in 2024-04-15T18:33:37.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 58378 58362 61.8181 61.5625 0.0066521 0.0161716 2024-04-15T18:33:38.056 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:38.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:38.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:38.559 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:33:38.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:38 smithi159 ceph-mon[45336]: pgmap v1467: 17 pgs: 1 stale+active+clean, 16 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-15T18:33:38.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:33:38.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:38 smithi049 ceph-mon[40087]: pgmap v1467: 17 pgs: 1 stale+active+clean, 16 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-15T18:33:38.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:33:38.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:33:38.979444+0000 min lat: 0.00106491 max lat: 15.0148 avg lat: 0.0161141 2024-04-15T18:33:38.969 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:33:38.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 59574 59558 62.0335 74.75 0.00665346 0.0161141 2024-04-15T18:33:39.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:38 smithi082 ceph-mon[44206]: pgmap v1467: 17 pgs: 1 stale+active+clean, 16 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-04-15T18:33:39.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:33:39.660 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:39.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:39 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:33:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:39 smithi049 ceph-mon[40087]: osdmap e564: 12 total, 11 up, 12 in 2024-04-15T18:33:39.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:39 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:33:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:39 smithi082 ceph-mon[44206]: osdmap e564: 12 total, 11 up, 12 in 2024-04-15T18:33:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:39.827 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:39.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:39.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:39 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:39.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2018944867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:33:39.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:39 smithi159 ceph-mon[45336]: osdmap e564: 12 total, 11 up, 12 in 2024-04-15T18:33:39.927 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:39.932 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:39.968 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:39.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 60605 60589 62.0728 64.4375 0.0252823 0.0161022 2024-04-15T18:33:39.996 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:33:39.996 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93111 5.6 5.6002d106 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.391761863, 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79303184699999996, 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:39.997 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93111 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609126+0000", 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:39.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609128+0000", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.81e-06 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609258+0000", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012971 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609273+0000", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4647999999999999e-05 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:39.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609284+0000", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1462e-05 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609344+0000", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0343000000000001e-05 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609361+0000", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6680999999999998e-05 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402147+0000", 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79278600300000002 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402158+0000", 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.119e-05 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93119 5.6 5.5151b6f6 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39157965599998, 2024-04-15T18:33:40.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79293819799999998, 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93119 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609309+0000", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3000000000000001e-07 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609425+0000", 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011591600000000001 2024-04-15T18:33:40.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609436+0000", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0509999999999999e-05 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609443+0000", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9890000000000003e-06 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609482+0000", 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9857999999999997e-05 2024-04-15T18:33:40.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609489+0000", 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.455e-06 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402240+0000", 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79275137100000004 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402247+0000", 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5690000000000001e-06 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93121 5.a 5.a03eb09a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39143183900001, 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79295962099999995, 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93121 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609456+0000", 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609457+0000", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7e-07 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609524+0000", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6706999999999997e-05 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609533+0000", 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6309999999999998e-06 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609542+0000", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7679999999999997e-06 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609584+0000", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1829000000000002e-05 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609588+0000", 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1520000000000002e-06 2024-04-15T18:33:40.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402404+0000", 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79281564900000001 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402416+0000", 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2415000000000001e-05 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93122 5.a 5.80d4a5fa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39133093200002, 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79291208199999996, 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93122 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609557+0000", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609558+0000", 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6699999999999999e-07 2024-04-15T18:33:40.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609617+0000", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.906e-05 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609626+0000", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3100000000000006e-06 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609632+0000", 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9889999999999997e-06 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609659+0000", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6908999999999999e-05 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609662+0000", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2320000000000001e-06 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402457+0000", 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79279496599999999 2024-04-15T18:33:40.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402469+0000", 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2149e-05 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93124 5.a 5.1f3061a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39124374599999, 2024-04-15T18:33:40.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79288053300000005, 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93124 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609644+0000", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609645+0000", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6199999999999998e-07 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609704+0000", 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8850000000000001e-05 2024-04-15T18:33:40.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609714+0000", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020999999999999e-05 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609720+0000", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8359999999999997e-06 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609760+0000", 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0207000000000003e-05 2024-04-15T18:33:40.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609763+0000", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402513+0000", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79274942500000001 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402525+0000", 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2283000000000001e-05 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.019 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93126 5.a 5.2939364a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39115477899998, 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.792824317, 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93126 2024-04-15T18:33:40.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609733+0000", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609734+0000", 2024-04-15T18:33:40.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7399999999999998e-07 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609798+0000", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3664999999999994e-05 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609806+0000", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5010000000000004e-06 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609810+0000", 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3789999999999999e-06 2024-04-15T18:33:40.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609863+0000", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2661000000000003e-05 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609872+0000", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7450000000000001e-06 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402548+0000", 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79267563200000002 2024-04-15T18:33:40.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402558+0000", 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.026e-05 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93129 5.a 5.c296a3da (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39106848900002, 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79277050400000004, 2024-04-15T18:33:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93129 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609820+0000", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.03e-07 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609865+0000", 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4506999999999999e-05 2024-04-15T18:33:40.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609869+0000", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7280000000000003e-06 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609873+0000", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6770000000000001e-06 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609915+0000", 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1841999999999997e-05 2024-04-15T18:33:40.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9530000000000005e-06 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402580+0000", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79265583299999998 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402590+0000", 2024-04-15T18:33:40.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0661e-05 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93139 5.a 5.5b8b557a (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39100700099999, 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79273795199999997, 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.029 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93139 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609881+0000", 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609882+0000", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8900000000000001e-07 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609916+0000", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4980000000000001e-05 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609921+0000", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5889999999999996e-06 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609924+0000", 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1930000000000002e-06 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609972+0000", 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7296000000000003e-05 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3039999999999994e-06 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402609+0000", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262850200000001 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402619+0000", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7990000000000005e-06 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93143 5.a 5.524373aa (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:33:40.033 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.39094963600002, 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79271180799999996, 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93143 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609939+0000", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6100000000000002e-07 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609976+0000", 2024-04-15T18:33:40.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6807999999999997e-05 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609981+0000", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4809999999999997e-06 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.609985+0000", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0899999999999998e-06 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610009+0000", 2024-04-15T18:33:40.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3479e-05 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.610012+0000", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6289999999999999e-06 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402641+0000", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.79262867999999997 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402650+0000", 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.38e-06 2024-04-15T18:33:40.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93146 5.6 5.74a5d016 (undecoded) ondisk+write+known_if_redirected e404)", 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 444.385418582, 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78680531300000001, 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.038 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93146 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615470+0000", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615471+0000", 2024-04-15T18:33:40.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0860000000000001e-06 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615527+0000", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6440000000000002e-05 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615541+0000", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3718e-05 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615555+0000", 2024-04-15T18:33:40.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.367e-05 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615601+0000", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6830999999999999e-05 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.615607+0000", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4790000000000004e-06 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402271+0000", 2024-04-15T18:33:40.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78666379799999997 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.402275+0000", 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2910000000000001e-06 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 419/405 [PushOp(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head, version: 370'1523, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:670a5e00:::benchmark_data_smithi159_116570_object23707:head@370'1523, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 412.98606493199998, 2024-04-15T18:33:40.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48793246000000001, 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014823+0000", 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014824+0000", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3300000000000001e-07 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.014867+0000", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3164999999999998e-05 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.500977+0000", 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48611008500000003 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501005+0000", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7334e-05 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501108+0000", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103587 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.501119+0000", 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1297e-05 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:47.502756+0000", 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016362589999999999 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.4 420/405 [PushOp(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head, version: 370'1593, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:23243fb3:::benchmark_data_smithi159_116570_object25609:head@370'1593, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 411.41474659900001, 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.622201475, 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.046 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586142+0000", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586143+0000", 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.108e-06 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:48.586181+0000", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8618000000000002e-05 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205809+0000", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61962728300000003 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205830+0000", 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1491000000000001e-05 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205930+0000", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9763e-05 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.205941+0000", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0776e-05 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:49.208343+0000", 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002402436 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 422/405 [PushOp(5:693270da:::benchmark_data_smithi159_116570_object31578:head, version: 372'1976, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:693270da:::benchmark_data_smithi159_116570_object31578:head@372'1976, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 406.15721208399998, 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54885523899999999, 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843676+0000", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843678+0000", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6700000000000001e-06 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:53.843793+0000", 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115654 2024-04-15T18:33:40.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389971+0000", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54617790200000005 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.389987+0000", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5892e-05 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390342+0000", 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035477700000000002 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.390368+0000", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6444999999999998e-05 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:54.392531+0000", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021628989999999998 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head, version: 402'5230, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head@402'5230, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:33:40.053 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 367.90757269699998, 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66078538499999995, 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093367+0000", 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0549e-05 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752462+0000", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.65909545999999997 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752479+0000", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6592e-05 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752556+0000", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6841000000000005e-05 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752564+0000", 2024-04-15T18:33:40.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6589999999999996e-06 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.754101+0000", 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015365310000000001 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.a 436/405 [PushOp(5:54042545:::benchmark_data_smithi159_116570_object78839:head, version: 397'4917, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:54042545:::benchmark_data_smithi159_116570_object78839:head@397'4917, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 367.27634777999998, 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50450683799999996, 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724541+0000", 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7700000000000004e-07 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724582+0000", 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1350999999999998e-05 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227396+0000", 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.502813128 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227405+0000", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9379999999999998e-06 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227470+0000", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5637999999999997e-05 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227479+0000", 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4700000000000002e-06 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.229047+0000", 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001568636 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:6189473b:::benchmark_data_smithi159_116570_object88044:head, version: 403'5400, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6189473b:::benchmark_data_smithi159_116570_object88044:head@403'5400, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.495332512, 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48358990099999999, 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505557+0000", 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.55e-06 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505641+0000", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3617999999999998e-05 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987415+0000", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48177457400000001 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987426+0000", 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0428000000000001e-05 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987508+0000", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2164999999999997e-05 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987517+0000", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5420000000000005e-06 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.989146+0000", 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001629024 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:62260e07:::benchmark_data_smithi159_116570_object92807:head, version: 403'5695, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:62260e07:::benchmark_data_smithi159_116570_object92807:head@403'5695, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 364.858106142, 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49781294999999998, 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142783+0000", 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142843+0000", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0627000000000001e-05 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638705+0000", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49586129000000001 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638724+0000", 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9579999999999999e-05 2024-04-15T18:33:40.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638840+0000", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011596599999999999 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638857+0000", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6864e-05 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.640595+0000", 2024-04-15T18:33:40.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00173799 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e447)", 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 348.26339089800001, 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55780170699999998, 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737499+0000", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2920000000000001e-06 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737515+0000", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737531+0000", 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737542+0000", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0835999999999999e-05 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737608+0000", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6070999999999998e-05 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737764+0000", 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156816 2024-04-15T18:33:40.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737886+0000", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121797 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.740403+0000", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002517039 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.741178+0000", 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077443900000000003 2024-04-15T18:33:40.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295224+0000", 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55404577600000005 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295259+0000", 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5099999999999999e-05 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295299+0000", 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0435000000000002e-05 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e447)", 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 348.21398016299997, 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50835932699999997, 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:33:40.075 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786909+0000", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786912+0000", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7390000000000001e-06 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786921+0000", 2024-04-15T18:33:40.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2730000000000002e-06 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786928+0000", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1709999999999996e-06 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786991+0000", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.321e-05 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for ondisk", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.787066+0000", 2024-04-15T18:33:40.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4807000000000004e-05 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295267+0000", 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50820147699999996 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:126776 5.a 5:5a87c32d:::benchmark_data_smithi159_116570_object33616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 268.293648254, 2024-04-15T18:33:40.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.477931355, 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 126776 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707242+0000", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.39e-06 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707255+0000", 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2709e-05 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707287+0000", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2403000000000001e-05 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707306+0000", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8539e-05 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707479+0000", 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017332900000000001 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707663+0000", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018317 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707820+0000", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015760200000000001 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.709796+0000", 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019761599999999998 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.711733+0000", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019369490000000001 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185060+0000", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47332694600000003 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185114+0000", 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4221999999999997e-05 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:33:40.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185171+0000", 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6935999999999998e-05 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:33:40.085 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:33:40.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6.service' 2024-04-15T18:33:40.171 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:40.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:40.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:33:40.216 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-15T18:33:40.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd down 6 2024-04-15T18:33:40.277 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:40.305 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:40.328 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:40.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:40.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:40.471 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:40.495 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:40.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:40.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:40.635 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:40 smithi159 ceph-mon[45336]: pgmap v1469: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 7233/172218 objects degraded (4.200%) 2024-04-15T18:33:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:40 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 7233/172218 objects degraded (4.200%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:40.657 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:40.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:40.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:40 smithi082 ceph-mon[44206]: pgmap v1469: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 7233/172218 objects degraded (4.200%) 2024-04-15T18:33:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:40 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 7233/172218 objects degraded (4.200%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:40.798 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:40.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:40 smithi049 ceph-mon[40087]: pgmap v1469: 17 pgs: 2 active+undersized+degraded, 15 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 7233/172218 objects degraded (4.200%) 2024-04-15T18:33:40.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:40 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 7233/172218 objects degraded (4.200%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:40.956 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:40.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:40.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 61754 61738 62.2298 71.8125 0.00614635 0.0160635 2024-04-15T18:33:40.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:33:41.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:41.092 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:41.115 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:41.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:41.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:41.249 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:41.272 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:41.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:41.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:41.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:33:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:41 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:41 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:41 smithi049 ceph-mon[40087]: osdmap e565: 12 total, 11 up, 12 in 2024-04-15T18:33:41.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:41 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:41.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:41 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:41.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:41 smithi159 ceph-mon[45336]: osdmap e565: 12 total, 11 up, 12 in 2024-04-15T18:33:41.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 62849 62833 62.3283 68.4375 0.00347566 0.0160375 2024-04-15T18:33:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:41 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:41 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:41 smithi082 ceph-mon[44206]: osdmap e565: 12 total, 11 up, 12 in 2024-04-15T18:33:42.682 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:42.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:42.779 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:42.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:42.820 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:42.846 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:42.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:42 smithi049 ceph-mon[40087]: pgmap v1471: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s; 15010/176874 objects degraded (8.486%) 2024-04-15T18:33:42.853 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:42.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:42 smithi159 ceph-mon[45336]: pgmap v1471: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s; 15010/176874 objects degraded (8.486%) 2024-04-15T18:33:42.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 63932 63916 62.4119 67.6875 0.00364954 0.0160181 2024-04-15T18:33:43.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:42 smithi082 ceph-mon[44206]: pgmap v1471: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s; 15010/176874 objects degraded (8.486%) 2024-04-15T18:33:43.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:43.596 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:43.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:43.658 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:43.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:43.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:43.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:43.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:43.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:43.830 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:43.869 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:43.890 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1228433549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1228433549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:43.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 64901 64885 62.3834 60.5625 0.00486974 0.0160251 2024-04-15T18:33:44.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1228433549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: pgmap v1472: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s; 15684/188589 objects degraded (8.316%) 2024-04-15T18:33:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.811 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:44.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:44.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: pgmap v1472: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s; 15684/188589 objects degraded (8.316%) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:44.919 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:44.948 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:44.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 66416 66400 62.8727 94.6875 0.00280876 0.0158998 2024-04-15T18:33:44.970 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:44.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:45.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: pgmap v1472: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s; 15684/188589 objects degraded (8.316%) 2024-04-15T18:33:45.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:45.108 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.136 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:45.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:45.277 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.293 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:45.397 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:45.432 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:45.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:45.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:45.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:45.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:45.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:45.600 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.627 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:45.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:45.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:45.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:45.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:45.767 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:45.769 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:45.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:45.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:45.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:45.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:45.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:45.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:45.936 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:45.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:45.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:45.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 67765 67749 63.1926 84.3125 0.00682479 0.0158205 2024-04-15T18:33:46.061 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:46.093 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:46.115 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:46.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:46.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:46.263 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:33:46.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:33:46.539 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:46.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:46.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:46.677 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:46.702 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:46.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:46.806 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:46.839 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:46.861 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:46.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:46.966 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:46.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 69061 69045 63.4544 81 0.00895688 0.0157535 2024-04-15T18:33:46.997 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:47.021 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:47.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:47.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:47.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.166 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:47.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.323 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:47.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:47.432 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:47.461 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:47.483 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:47.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:47.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:47.619 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:47.640 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:47.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:47.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:47.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 15684/188589 objects degraded (8.316%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:47.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:47.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 70198 70182 63.5645 71.0625 0.00326577 0.0157275 2024-04-15T18:33:48.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 15684/188589 objects degraded (8.316%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:48.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:48.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:48.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 15684/188589 objects degraded (8.316%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:48.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:48.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:33:48.762 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:48.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:48.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:48.902 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:48.926 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:48.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:48.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 71391 71375 63.7216 74.5625 0.00584882 0.0156882 2024-04-15T18:33:49.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:48 smithi049 ceph-mon[40087]: pgmap v1474: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 16442/199041 objects degraded (8.261%) 2024-04-15T18:33:49.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:33:49.065 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:49.091 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:33:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:49.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:33:49.094 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:48 smithi082 ceph-mon[44206]: pgmap v1474: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 16442/199041 objects degraded (8.261%) 2024-04-15T18:33:49.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:48 smithi159 ceph-mon[45336]: pgmap v1474: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 16442/199041 objects degraded (8.261%) 2024-04-15T18:33:49.194 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:33:49.225 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:49.251 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:33:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:49.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:33:49.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:49.387 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:49.410 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:49.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:49.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:49.574 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:49.786 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:49.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:49.866 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:33:49.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:49.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:49.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 72769 72753 64.037 86.125 0.00760691 0.01561 2024-04-15T18:33:50.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3791797484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3791797484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3791797484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:33:50.461 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6 is already down. 2024-04-15T18:33:50.501 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:33:50.501 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:33:50.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:33:50.464+0000 7f8015e7e700 -1 mon.a@0(leader).osd e566 definitely_dead 0 2024-04-15T18:33:50.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 15 73869 73854 64.1033 68.8125 0.0878173 0.0155841 2024-04-15T18:33:51.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:50 smithi159 ceph-mon[45336]: pgmap v1475: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 17218/207264 objects degraded (8.307%) 2024-04-15T18:33:51.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:50 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:51.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:51.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:50 smithi159 ceph-mon[45336]: osdmap e566: 12 total, 11 up, 12 in 2024-04-15T18:33:51.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3952889933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-15T18:33:51.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-mon[40087]: pgmap v1475: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 17218/207264 objects degraded (8.307%) 2024-04-15T18:33:51.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:51.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:51.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-mon[40087]: osdmap e566: 12 total, 11 up, 12 in 2024-04-15T18:33:51.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3952889933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-15T18:33:51.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:50 smithi082 ceph-mon[44206]: pgmap v1475: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 17218/207264 objects degraded (8.307%) 2024-04-15T18:33:51.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:50 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:33:51.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:33:51.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:50 smithi082 ceph-mon[44206]: osdmap e566: 12 total, 11 up, 12 in 2024-04-15T18:33:51.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3952889933' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["6"]}]: dispatch 2024-04-15T18:33:51.495 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:51.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:51.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:51.628 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:51.926 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:51.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:51.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:33:51.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 74763 74747 63.9896 55.8125 0.00786141 0.0156223 2024-04-15T18:33:52.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:33:52.067 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:52.420 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:33:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:52.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:33:52.465 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 3, 2] 2024-04-15T18:33:52.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd out 6 2024-04-15T18:33:52.527 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:52.551 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:52.554 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:52.568 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:52.578 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:52.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:52.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:52.715 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:52.740 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:52.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:52.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:33:52.906 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:52.970 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 15 75825 75810 64.0226 66.4375 0.00978707 0.0156139 2024-04-15T18:33:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:52 smithi159 ceph-mon[45336]: pgmap v1477: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 17998/212901 objects degraded (8.454%) 2024-04-15T18:33:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 17998/212901 objects degraded (8.454%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:52 smithi159 ceph-mon[45336]: osdmap e567: 12 total, 11 up, 12 in 2024-04-15T18:33:53.153 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:52 smithi049 ceph-mon[40087]: pgmap v1477: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 17998/212901 objects degraded (8.454%) 2024-04-15T18:33:53.153 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 17998/212901 objects degraded (8.454%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:53.153 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:52 smithi049 ceph-mon[40087]: osdmap e567: 12 total, 11 up, 12 in 2024-04-15T18:33:53.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:52 smithi082 ceph-mon[44206]: pgmap v1477: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 17998/212901 objects degraded (8.454%) 2024-04-15T18:33:53.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 17998/212901 objects degraded (8.454%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:53.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:52 smithi082 ceph-mon[44206]: osdmap e567: 12 total, 11 up, 12 in 2024-04-15T18:33:53.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:33:53.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 76830 76814 64.0056 62.75 0.0107645 0.0156167 2024-04-15T18:33:53.986 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:33:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:53.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:33:54.093 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:33:54.125 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:54.145 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:33:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:54.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:33:54.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:54.301 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:54.433 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:33:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:54.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:33:54.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:33:54.568 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:54.590 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:33:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:54.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:33:54.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:54.726 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:54.753 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:54.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:54.858 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:54.886 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:54.907 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:54.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:54.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 78058 78042 64.1732 76.75 0.0287341 0.0155784 2024-04-15T18:33:55.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:33:55.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:54 smithi082 ceph-mon[44206]: pgmap v1479: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 18704/224574 objects degraded (8.329%) 2024-04-15T18:33:55.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:54 smithi049 ceph-mon[40087]: pgmap v1479: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 18704/224574 objects degraded (8.329%) 2024-04-15T18:33:55.056 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:55.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:54 smithi159 ceph-mon[45336]: pgmap v1479: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 18704/224574 objects degraded (8.329%) 2024-04-15T18:33:55.606 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:33:55.606 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:33:55.789 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:55.790 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:33:55.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 79019 79003 64.1198 60.0625 0.00319678 0.0155897 2024-04-15T18:33:56.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:33:56.423 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:33:56.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:56.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:33:56.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:33:56.561 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:56.584 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:33:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:56.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:33:56.688 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:33:56.717 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:56.740 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:33:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:56.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:33:56.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:33:56.863 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:56 smithi159 ceph-mon[45336]: pgmap v1480: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 18704/225504 objects degraded (8.294%) 2024-04-15T18:33:56.881 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:56.909 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:33:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:33:56.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:33:56.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:56 smithi049 ceph-mon[40087]: pgmap v1480: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 18704/225504 objects degraded (8.294%) 2024-04-15T18:33:56.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 80157 80141 64.2095 71.125 0.0120719 0.0155694 2024-04-15T18:33:57.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:33:57.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:56 smithi082 ceph-mon[44206]: pgmap v1480: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 18704/225504 objects degraded (8.294%) 2024-04-15T18:33:57.062 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:33:57.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 81413 81397 64.3903 78.5 0.00450901 0.0155256 2024-04-15T18:33:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osdmap e568: 12 total, 11 up, 12 in 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 18704/225504 objects degraded (8.294%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osdmap e569: 12 total, 11 up, 12 in 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2768266717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:58.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:58.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osdmap e568: 12 total, 11 up, 12 in 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 18704/225504 objects degraded (8.294%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osdmap e569: 12 total, 11 up, 12 in 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2768266717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:58.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:58.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osdmap e568: 12 total, 11 up, 12 in 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 18704/225504 objects degraded (8.294%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osdmap e569: 12 total, 11 up, 12 in 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2768266717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:58.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:33:58.981090+0000 min lat: 0.00106491 max lat: 15.0148 avg lat: 0.0154952 2024-04-15T18:33:58.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:33:58.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 82600 82584 64.5127 74.1875 0.00832565 0.0154952 2024-04-15T18:33:59.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:33:59.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:33:59.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:33:59.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:33:59.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:33:59.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:33:59.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:33:59.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:33:59.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:33:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:33:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:33:59.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 83884 83868 64.7069 80.25 0.00602792 0.0154498 2024-04-15T18:34:00.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:00.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:00.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:33:59 smithi159 ceph-mon[45336]: osdmap e570: 12 total, 11 up, 12 in 2024-04-15T18:34:00.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:00.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:00.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:33:59 smithi049 ceph-mon[40087]: osdmap e570: 12 total, 11 up, 12 in 2024-04-15T18:34:00.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:00.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:00.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:33:59 smithi082 ceph-mon[44206]: osdmap e570: 12 total, 11 up, 12 in 2024-04-15T18:34:00.447 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:00.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:00.551 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:34:00.580 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:00.603 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:34:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:00.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:34:00.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:00.906 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:00.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:00.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 84876 84860 64.6738 62 0.0144143 0.0154571 2024-04-15T18:34:01.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: pgmap v1485: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 20031/240858 objects degraded (8.317%) 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: osdmap e571: 12 total, 11 up, 12 in 2024-04-15T18:34:01.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:01.043 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:01.045 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:01.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: pgmap v1485: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 20031/240858 objects degraded (8.317%) 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: osdmap e571: 12 total, 11 up, 12 in 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:01.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: pgmap v1485: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s; 20031/240858 objects degraded (8.317%) 2024-04-15T18:34:01.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:01.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:01.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: osdmap e571: 12 total, 11 up, 12 in 2024-04-15T18:34:01.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:01.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:01.395 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:01.493 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:01.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 85933 85917 64.6905 66.0625 0.00777831 0.0154527 2024-04-15T18:34:02.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:01 smithi049 ceph-mon[40087]: osdmap e572: 12 total, 11 up, 12 in 2024-04-15T18:34:02.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:01 smithi082 ceph-mon[44206]: osdmap e572: 12 total, 11 up, 12 in 2024-04-15T18:34:02.300 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:02.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:02.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:01 smithi159 ceph-mon[45336]: osdmap e572: 12 total, 11 up, 12 in 2024-04-15T18:34:02.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:02.461 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:02.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:02.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:02.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:34:02.544+0000 7f8015e7e700 -1 mon.a@0(leader).osd e573 definitely_dead 0 2024-04-15T18:34:02.652 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:02.684 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:02.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:02.790 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:02.819 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:02.843 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:02.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:02.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:34:02.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 86918 86902 64.6532 61.5625 0.0104377 0.0154618 2024-04-15T18:34:02.980 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:03.004 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:34:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:03.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:34:03.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:03.146 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: pgmap v1488: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 20757/246192 objects degraded (8.431%) 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 20757/246192 objects degraded (8.431%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: osdmap e573: 12 total, 11 up, 12 in 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-15T18:34:03.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:02 smithi049 ceph-mon[40087]: Client client.admin marked osd.6 out, after it was down for 25 seconds 2024-04-15T18:34:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: pgmap v1488: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 20757/246192 objects degraded (8.431%) 2024-04-15T18:34:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 20757/246192 objects degraded (8.431%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: osdmap e573: 12 total, 11 up, 12 in 2024-04-15T18:34:03.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:03.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-15T18:34:03.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:02 smithi082 ceph-mon[44206]: Client client.admin marked osd.6 out, after it was down for 25 seconds 2024-04-15T18:34:03.394 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.6. 2024-04-15T18:34:03.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: pgmap v1488: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 20757/246192 objects degraded (8.431%) 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 20757/246192 objects degraded (8.431%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: osdmap e573: 12 total, 11 up, 12 in 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-15T18:34:03.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:02 smithi159 ceph-mon[45336]: Client client.admin marked osd.6 out, after it was down for 25 seconds 2024-04-15T18:34:03.398 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:34:03.803 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:03.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:03.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:03.935 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:03.954 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:03.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:03.971 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 87373 87357 64.2271 28.4375 0.291541 0.0154969 2024-04-15T18:34:04.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:04.089 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:04.111 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:04.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:03 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:04.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:03 smithi049 ceph-mon[40087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:34:04.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:04.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-15T18:34:04.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:03 smithi049 ceph-mon[40087]: osdmap e574: 12 total, 11 up, 11 in 2024-04-15T18:34:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:04.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:04.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:04.242 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:03 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:03 smithi082 ceph-mon[44206]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:34:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:04.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-15T18:34:04.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:03 smithi082 ceph-mon[44206]: osdmap e574: 12 total, 11 up, 11 in 2024-04-15T18:34:04.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:03 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:04.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:03 smithi159 ceph-mon[45336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:34:04.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/910834430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:04.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1238089777' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-15T18:34:04.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:03 smithi159 ceph-mon[45336]: osdmap e574: 12 total, 11 up, 11 in 2024-04-15T18:34:04.972 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 87377 87361 63.4831 0.25 0.0040086 0.0155025 2024-04-15T18:34:05.137 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:05.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:05.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:04 smithi049 ceph-mon[40087]: pgmap v1491: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 21486/257751 objects degraded (8.336%) 2024-04-15T18:34:05.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:04 smithi049 ceph-mon[40087]: osdmap e575: 12 total, 11 up, 11 in 2024-04-15T18:34:05.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:05.271 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:05.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:04 smithi082 ceph-mon[44206]: pgmap v1491: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 21486/257751 objects degraded (8.336%) 2024-04-15T18:34:05.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:04 smithi082 ceph-mon[44206]: osdmap e575: 12 total, 11 up, 11 in 2024-04-15T18:34:05.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:04 smithi159 ceph-mon[45336]: pgmap v1491: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s; 21486/257751 objects degraded (8.336%) 2024-04-15T18:34:05.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:04 smithi159 ceph-mon[45336]: osdmap e575: 12 total, 11 up, 11 in 2024-04-15T18:34:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:05.472 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-04-15T18:34:05.472 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi082.front.sepia.ceph.com 2024-04-15T18:34:05.473 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.6 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-04-15T18:34:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:05.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:05.574 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:05.602 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:05.627 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:05.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:05.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:05.743 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.6/config 2024-04-15T18:34:05.772 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:05.793 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:05.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:05.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:05.926 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:05 smithi049 ceph-mon[40087]: 6.1 scrub starts 2024-04-15T18:34:05.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:05 smithi049 ceph-mon[40087]: osdmap e576: 12 total, 11 up, 11 in 2024-04-15T18:34:05.938 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:05.938 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:05.969 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:05.972 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 87620 87604 62.9279 15.1875 0.0107492 0.0158853 2024-04-15T18:34:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:05.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:06.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:06.108 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:06.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:05 smithi159 ceph-mon[45336]: 6.1 scrub starts 2024-04-15T18:34:06.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:05 smithi159 ceph-mon[45336]: osdmap e576: 12 total, 11 up, 11 in 2024-04-15T18:34:06.273 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:06.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:06.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:05 smithi082 ceph-mon[44206]: 6.1 scrub starts 2024-04-15T18:34:06.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:05 smithi082 ceph-mon[44206]: osdmap e576: 12 total, 11 up, 11 in 2024-04-15T18:34:06.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:06.410 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:06.490 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:06.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:06.597 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:06.629 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:06.657 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:06.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:06.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:06.826 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:06.859 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:06.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:06.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:06.972 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 88303 88287 62.6979 42.6875 0.031369 0.0159429 2024-04-15T18:34:07.011 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:07.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:34:07.200 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:06 smithi049 ceph-mon[40087]: pgmap v1493: 17 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 21486/258429 objects degraded (8.314%) 2024-04-15T18:34:07.200 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:06 smithi049 ceph-mon[40087]: osdmap e577: 12 total, 11 up, 11 in 2024-04-15T18:34:07.225 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:07.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:07.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:06 smithi082 ceph-mon[44206]: pgmap v1493: 17 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 21486/258429 objects degraded (8.314%) 2024-04-15T18:34:07.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:06 smithi082 ceph-mon[44206]: osdmap e577: 12 total, 11 up, 11 in 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:75421 5.3 5:cb1d90dc:::benchmark_data_smithi159_116570_object75420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:50.899734+0000", 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 496.39713704500002, 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54096360499999996, 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.291 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 75421 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899734+0000", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899734+0000", 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899735+0000", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0100000000000001e-06 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899758+0000", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2819e-05 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899777+0000", 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8978999999999999e-05 2024-04-15T18:34:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899783+0000", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9599999999999997e-06 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899829+0000", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5908000000000002e-05 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.899927+0000", 2024-04-15T18:34:07.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8733999999999994e-05 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.900036+0000", 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108986 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.902110+0000", 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020739690000000002 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.905455+0000", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033446859999999999 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440488+0000", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53503283199999996 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440531+0000", 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3339e-05 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440698+0000", 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016638300000000001 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25447.0:75421 5.3 e392/377 ondisk, result = 0)", 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:25:50.902636+0000", 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 496.39423489199999, 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53806738899999995, 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:34:07.297 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.902636+0000", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.902636+0000", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.902637+0000", 2024-04-15T18:34:07.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5600000000000005e-07 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:50.902641+0000", 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0110000000000002e-06 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440427+0000", 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.537786023 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440431+0000", 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1899999999999997e-06 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440476+0000", 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5225000000000002e-05 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440484+0000", 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.593e-06 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:25:51.440703+0000", 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021969099999999999 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81669 5.1 5.6c5f6781 (undecoded) ondisk+retry+write+known_if_redirected e400)", 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.572283+0000", 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.72458800700002, 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.765194924, 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81669 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572283+0000", 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572283+0000", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572284+0000", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2999999999999999e-07 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572406+0000", 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012239600000000001 2024-04-15T18:34:07.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572424+0000", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7648000000000001e-05 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572432+0000", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3829999999999992e-06 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572501+0000", 2024-04-15T18:34:07.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9196000000000006e-05 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572507+0000", 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2109999999999997e-06 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337468+0000", 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76496124200000004 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337478+0000", 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0018e-05 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81772 5.1 5.f7bc3dd1 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.572473+0000", 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.72439796499998, 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76506320999999999, 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.306 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81772 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572473+0000", 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572473+0000", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572474+0000", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3499999999999996e-07 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572507+0000", 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3226999999999999e-05 2024-04-15T18:34:07.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572516+0000", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0129999999999999e-06 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572522+0000", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6560000000000003e-06 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572561+0000", 2024-04-15T18:34:07.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8457000000000001e-05 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572564+0000", 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.196e-06 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337524+0000", 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76495953400000005 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337536+0000", 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2592000000000001e-05 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81784 5.1 5.3ca10291 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.572552+0000", 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.72431904500002, 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76504169600000005, 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81784 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572552+0000", 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572552+0000", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572553+0000", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3399999999999998e-07 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572607+0000", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4685999999999998e-05 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572617+0000", 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4760000000000004e-06 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572624+0000", 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7440000000000001e-06 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572649+0000", 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5432e-05 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.314 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572652+0000", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.943e-06 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337582+0000", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76492978300000003 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337594+0000", 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1898e-05 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81793 5.1 5.9d871fb1 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.572636+0000", 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.72423518900001, 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76499205000000003, 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81793 2024-04-15T18:34:07.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572636+0000", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572636+0000", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572636+0000", 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7300000000000001e-07 2024-04-15T18:34:07.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572687+0000", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0716e-05 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572704+0000", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7133e-05 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572711+0000", 2024-04-15T18:34:07.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4790000000000001e-06 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572756+0000", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5447e-05 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.572761+0000", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8230000000000002e-06 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337621+0000", 2024-04-15T18:34:07.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76485992999999997 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337628+0000", 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0489999999999998e-06 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81850 5.1 5.ab076861 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.584578+0000", 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.712292805, 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.75308249900000002, 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81850 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584578+0000", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584578+0000", 2024-04-15T18:34:07.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584579+0000", 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584621+0000", 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2218000000000001e-05 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584630+0000", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7220000000000005e-06 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584634+0000", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6020000000000002e-06 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584673+0000", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8658000000000002e-05 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.584678+0000", 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.31e-06 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337650+0000", 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.75297201000000002 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337661+0000", 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81864 5.1 5.996b1c01 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.609437+0000", 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.68743376100002, 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72827973999999995, 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81864 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609437+0000", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609437+0000", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609438+0000", 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0200000000000001e-07 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609462+0000", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4587999999999999e-05 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609471+0000", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.738e-06 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609476+0000", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5909999999999998e-06 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609528+0000", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2247999999999998e-05 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.609533+0000", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2179999999999998e-06 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337706+0000", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72817257000000002 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337717+0000", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1085e-05 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81886 5.1 5.3b6a191 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.656881+0000", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.63999033900001, 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.68087648099999998, 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.329 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81886 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656881+0000", 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656881+0000", 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656883+0000", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.883e-06 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656930+0000", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7635e-05 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656958+0000", 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8246e-05 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.656981+0000", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2353000000000001e-05 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.657082+0000", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010170699999999999 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.657088+0000", 2024-04-15T18:34:07.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2650000000000004e-06 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337746+0000", 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.68065801699999995 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337757+0000", 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1375e-05 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81919 5.1 5.232db611 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.716071+0000", 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.58080029799999, 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.62172704199999995, 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81919 2024-04-15T18:34:07.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716071+0000", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716071+0000", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716072+0000", 2024-04-15T18:34:07.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.082e-06 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716121+0000", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9601999999999997e-05 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716131+0000", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8439999999999993e-06 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716136+0000", 2024-04-15T18:34:07.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1749999999999996e-06 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716194+0000", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7306999999999999e-05 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.716202+0000", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6999999999999997e-06 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337788+0000", 2024-04-15T18:34:07.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.62158538299999999 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337798+0000", 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9490000000000008e-06 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81968 5.1 5.76106e21 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.765508+0000", 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.53136294900003, 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.57232491299999999, 2024-04-15T18:34:07.338 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81968 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765508+0000", 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765508+0000", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765510+0000", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0559999999999999e-06 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765613+0000", 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103131 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765641+0000", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7613000000000001e-05 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765656+0000", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4742e-05 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765761+0000", 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105871 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.765796+0000", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4292999999999998e-05 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337826+0000", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.57202995800000001 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337833+0000", 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2490000000000004e-06 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.342 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81971 5.1 5.b0d4d381 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.770246+0000", 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.52662461800003, 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.56760367300000003, 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81971 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770246+0000", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770246+0000", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770248+0000", 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.703e-06 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770355+0000", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107428 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770385+0000", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9595999999999999e-05 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770402+0000", 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7229000000000001e-05 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770489+0000", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6749000000000006e-05 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.770498+0000", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4420000000000004e-06 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337846+0000", 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.56734769399999996 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337850+0000", 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8319999999999999e-06 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81983 5.1 5.af825931 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.785552+0000", 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.51131906099999, 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55231343899999996, 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81983 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785552+0000", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785552+0000", 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785553+0000", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0529999999999999e-06 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785586+0000", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3334000000000003e-05 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785598+0000", 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1514e-05 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785604+0000", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2940000000000002e-06 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785664+0000", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9678e-05 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.785674+0000", 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0013999999999999e-05 2024-04-15T18:34:07.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337861+0000", 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55218762700000001 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337865+0000", 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9249999999999997e-06 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:81985 5.1 5.44740f31 (undecoded) ondisk+write+known_if_redirected e400)", 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:00.786036+0000", 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 486.510835434, 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.551860393, 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 81985 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.352 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786036+0000", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786036+0000", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786036+0000", 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7199999999999999e-07 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786077+0000", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0794000000000003e-05 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786086+0000", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8559999999999996e-06 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786091+0000", 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0050000000000004e-06 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786134+0000", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2929000000000002e-05 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:00.786139+0000", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9690000000000002e-06 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337889+0000", 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55175003600000005 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:01.337896+0000", 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2320000000000004e-06 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:93127 5.3 5:c04bf541:::benchmark_data_smithi159_116570_object93126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e404)", 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.435187+0000", 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 471.86168400499997, 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71375354800000002, 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:34:07.356 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93127 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435187+0000", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435187+0000", 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435189+0000", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0559999999999999e-06 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435225+0000", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5540999999999997e-05 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435243+0000", 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8892000000000001e-05 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435256+0000", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2221e-05 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435356+0000", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010018699999999999 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435506+0000", 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000150292 2024-04-15T18:34:07.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.435616+0000", 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109986 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.439031+0000", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003415155 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.474951+0000", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.035919896 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148862+0000", 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.67391104300000004 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148907+0000", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4844e-05 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148941+0000", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3435000000000001e-05 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.25447.0:93127 5.3 e404/394 ondisk, result = 0)", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.438365+0000", 2024-04-15T18:34:07.363 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 471.85850595099998, 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71058331799999996, 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.438365+0000", 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.438365+0000", 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.438367+0000", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3530000000000002e-06 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.438377+0000", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6190000000000006e-06 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148788+0000", 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.71041150399999997 2024-04-15T18:34:07.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148794+0000", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6570000000000002e-06 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148851+0000", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6848999999999997e-05 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148858+0000", 2024-04-15T18:34:07.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.674e-06 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.148948+0000", 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0661999999999994e-05 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:93144 5.2 e404/358 5:476f72f4:::benchmark_data_smithi159_116570_object93143:head v 404'5791, mlcod=403'5789)", 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:26:15.474794+0000", 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 471.822076654, 2024-04-15T18:34:07.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92449504699999996, 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.474794+0000", 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.474794+0000", 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.474795+0000", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8000000000000005e-07 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:15.474863+0000", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7416999999999995e-05 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.396595+0000", 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92173269099999999 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.396602+0000", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.477e-06 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.396656+0000", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4595999999999997e-05 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.396673+0000", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6458999999999999e-05 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.399234+0000", 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025615540000000002 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:26:16.399289+0000", 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5072999999999999e-05 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:186314 5.b e536/409 5:db471f5e:::benchmark_data_smithi159_116570_object93154:head v 536'11820, mlcod=536'11817)", 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 114.183081746, 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98939341700000005, 2024-04-15T18:34:07.371 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113790+0000", 2024-04-15T18:34:07.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.066e-06 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113796+0000", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6450000000000002e-06 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098795+0000", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98499919199999997 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098805+0000", 2024-04-15T18:34:07.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9539999999999999e-06 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098871+0000", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5586000000000005e-05 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098893+0000", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2595999999999998e-05 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.103141+0000", 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042477979999999997 2024-04-15T18:34:07.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.103183+0000", 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1579999999999998e-05 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:26892 6.0 e550/541 6:0de6e80a:::benchmark_data_smithi159_139342_object26891:head v 550'1708, mlcod=550'1705)", 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:57.764262+0000", 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 69.532608977999999, 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22802615600000001, 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.375 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.764262+0000", 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.764262+0000", 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.764263+0000", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.032e-06 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.764292+0000", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8659000000000001e-05 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.991207+0000", 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22691581599999999 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.991212+0000", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8760000000000004e-06 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.991268+0000", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5868000000000001e-05 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.991294+0000", 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5605e-05 2024-04-15T18:34:07.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.992244+0000", 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095065000000000004 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.992288+0000", 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3649999999999997e-05 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.379 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:39501 6.a e555/541 6:57bd5a22:::benchmark_data_smithi159_139342_object39500:head v 555'2399, mlcod=555'2396)", 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 46.464231910999999, 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270015279, 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832640+0000", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6700000000000003e-07 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832700+0000", 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0361e-05 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101287+0000", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26858680499999998 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101294+0000", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4719999999999998e-06 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101355+0000", 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0955999999999997e-05 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101379+0000", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3892000000000001e-05 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.102607+0000", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001227749 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.102654+0000", 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7477e-05 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:34:07.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:34:07.384 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:34:07.384 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:07.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:06 smithi159 ceph-mon[45336]: pgmap v1493: 17 pgs: 1 active+clean+scrubbing, 5 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 21486/258429 objects degraded (8.314%) 2024-04-15T18:34:07.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:06 smithi159 ceph-mon[45336]: osdmap e577: 12 total, 11 up, 11 in 2024-04-15T18:34:07.422 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:07.444 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:07.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:07.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:07.587 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:07.616 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:07.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:07.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:07.810 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:07.972 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 89063 89047 62.5271 47.5 0.0265225 0.0159871 2024-04-15T18:34:08.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:07 smithi049 ceph-mon[40087]: 6.1 scrub ok 2024-04-15T18:34:08.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 21486/258429 objects degraded (8.314%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:08.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:07 smithi049 ceph-mon[40087]: osdmap e578: 12 total, 11 up, 11 in 2024-04-15T18:34:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:07 smithi082 ceph-mon[44206]: 6.1 scrub ok 2024-04-15T18:34:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 21486/258429 objects degraded (8.314%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:07 smithi082 ceph-mon[44206]: osdmap e578: 12 total, 11 up, 11 in 2024-04-15T18:34:08.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:07 smithi159 ceph-mon[45336]: 6.1 scrub ok 2024-04-15T18:34:08.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 21486/258429 objects degraded (8.314%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:08.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:07 smithi159 ceph-mon[45336]: osdmap e578: 12 total, 11 up, 11 in 2024-04-15T18:34:08.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:34:08.722 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:08.782 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:08.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:08.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:08.940 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:08.971 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:08.972 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 12 89876 89864 62.3996 51.0625 0.0218737 0.0160169 2024-04-15T18:34:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:08.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:09.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:09.116 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:09.156 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:09.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:09.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:08 smithi049 ceph-mon[40087]: pgmap v1496: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 2 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 23 MiB/s wr, 368 op/s; 21743/262176 objects degraded (8.293%); 577 KiB/s, 7 objects/s recovering 2024-04-15T18:34:09.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:09.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:08 smithi082 ceph-mon[44206]: pgmap v1496: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 2 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 23 MiB/s wr, 368 op/s; 21743/262176 objects degraded (8.293%); 577 KiB/s, 7 objects/s recovering 2024-04-15T18:34:09.285 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:08 smithi159 ceph-mon[45336]: pgmap v1496: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 2 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 23 MiB/s wr, 368 op/s; 21743/262176 objects degraded (8.293%); 577 KiB/s, 7 objects/s recovering 2024-04-15T18:34:09.287 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:09.306 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:09.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:09.311 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:09.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:09.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:09.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:09.424 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:09.444 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:09.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:09.484 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:09.513 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:09.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:09.572 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:09.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:09.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:09.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:09.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:09.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.7692 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 89876 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 61.885 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 33.9287 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 116.312 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 990 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 542.872 2024-04-15T18:34:09.973 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 1861 2024-04-15T18:34:09.974 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 0 2024-04-15T18:34:09.974 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0160585 2024-04-15T18:34:09.974 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.19646 2024-04-15T18:34:09.974 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 15.0148 2024-04-15T18:34:09.974 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00106491 2024-04-15T18:34:09.983 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:34:10.101 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:10.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:10.215 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: 6.2 scrub starts 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: 6.4 scrub starts 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1265224538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: 6.2 scrub starts 2024-04-15T18:34:10.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: 6.4 scrub starts 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1265224538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.263 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.268 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:10.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:10.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: 6.2 scrub starts 2024-04-15T18:34:10.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: 6.4 scrub starts 2024-04-15T18:34:10.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1265224538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:10.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: pgmap v1498: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 493 op/s; 21708/265155 objects degraded (8.187%); 2.1 MiB/s, 32 objects/s recovering 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:10 smithi049 ceph-mon[40087]: 6.2 scrub ok 2024-04-15T18:34:11.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: pgmap v1498: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 493 op/s; 21708/265155 objects degraded (8.187%); 2.1 MiB/s, 32 objects/s recovering 2024-04-15T18:34:11.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:10 smithi082 ceph-mon[44206]: 6.2 scrub ok 2024-04-15T18:34:11.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: pgmap v1498: 17 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 11 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 493 op/s; 21708/265155 objects degraded (8.187%); 2.1 MiB/s, 32 objects/s recovering 2024-04-15T18:34:11.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:11.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:10 smithi159 ceph-mon[45336]: 6.2 scrub ok 2024-04-15T18:34:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:11.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:11.428 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:34:11.444 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:11.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:11.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:11.581 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:11.603 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:11.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:11.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:11.743 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:11.958 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:11 smithi082 ceph-mon[44206]: 6.4 scrub ok 2024-04-15T18:34:12.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:11 smithi049 ceph-mon[40087]: 6.4 scrub ok 2024-04-15T18:34:12.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:11 smithi159 ceph-mon[45336]: 6.4 scrub ok 2024-04-15T18:34:13.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:12 smithi049 ceph-mon[40087]: pgmap v1499: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 526 op/s; 21231/266628 objects degraded (7.963%); 6.7 MiB/s, 106 objects/s recovering 2024-04-15T18:34:13.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 21231/266628 objects degraded (7.963%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:13.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1717147022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:12 smithi082 ceph-mon[44206]: pgmap v1499: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 526 op/s; 21231/266628 objects degraded (7.963%); 6.7 MiB/s, 106 objects/s recovering 2024-04-15T18:34:13.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 21231/266628 objects degraded (7.963%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:13.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1717147022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:12 smithi159 ceph-mon[45336]: pgmap v1499: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 31 MiB/s wr, 526 op/s; 21231/266628 objects degraded (7.963%); 6.7 MiB/s, 106 objects/s recovering 2024-04-15T18:34:13.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 21231/266628 objects degraded (7.963%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:13.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1717147022' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:13.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:34:13.556 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:13.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:13.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:13.702 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:14.026 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:34:14.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1863921568,config=1059521453,health=3043139144,kv=381200665}) 2024-04-15T18:34:14.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2522943856,logm=1032422407}) 2024-04-15T18:34:14.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54172305}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721340623}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923487141}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67155288}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910019489}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066552193}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487532569}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408340601}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603121907}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663758718}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658537291}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158873647}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287902942}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2540796699}) 2024-04-15T18:34:14.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366970885}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184220640}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2271315294}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792257746}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291025418}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323034123}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678854950}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785654052}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926225955}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3680904382}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178149898}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135290105}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847896767}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732029950}) 2024-04-15T18:34:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625083285}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3330908033}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283093111}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668701426}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779861395}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036691526}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632322635}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962739681}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168174409}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660863347}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407647598}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225498983}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3709676346}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979023490}) 2024-04-15T18:34:14.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535558476}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=187271867}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026539367}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686557541}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946964263}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414036723}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045725819}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252957429}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984424195}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935321636}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411303081}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549124670}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054662748}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476335396}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924404717}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=410191971}) 2024-04-15T18:34:14.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287127493}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801873616}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864933299}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316436963}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510707053}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425514705}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322990948}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=47666720,mdsmap=2840952586,mgr=798432949}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mgr=14,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=70} crc {mgr=187916674,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=712996742,mon_config_key=2965594423}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2530393979}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1948904719}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1746849822,monmap=1029477071,osd_metadata=175868875,osd_pg_creating=749074650,osdmap=3576918729}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=42} crc {osdmap=2576795721}) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:14.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:13 smithi049 ceph-mon[40087]: osdmap e579: 12 total, 11 up, 11 in 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1863921568,config=1059521453,health=3043139144,kv=381200665}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2522943856,logm=1032422407}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54172305}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721340623}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923487141}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67155288}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910019489}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066552193}) 2024-04-15T18:34:14.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487532569}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408340601}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603121907}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663758718}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658537291}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158873647}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287902942}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2540796699}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366970885}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184220640}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2271315294}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792257746}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291025418}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323034123}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678854950}) 2024-04-15T18:34:14.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785654052}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926225955}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3680904382}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178149898}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135290105}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847896767}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732029950}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625083285}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3330908033}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283093111}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668701426}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779861395}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036691526}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632322635}) 2024-04-15T18:34:14.282 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962739681}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168174409}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660863347}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407647598}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225498983}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3709676346}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979023490}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535558476}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=187271867}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026539367}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686557541}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946964263}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414036723}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045725819}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252957429}) 2024-04-15T18:34:14.283 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984424195}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935321636}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411303081}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549124670}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054662748}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476335396}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924404717}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=410191971}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287127493}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801873616}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864933299}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316436963}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510707053}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425514705}) 2024-04-15T18:34:14.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322990948}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=47666720,mdsmap=2840952586,mgr=798432949}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mgr=14,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=70} crc {mgr=187916674,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=712996742,mon_config_key=2965594423}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2530393979}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1948904719}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1746849822,monmap=1029477071,osd_metadata=175868875,osd_pg_creating=749074650,osdmap=3576918729}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=42} crc {osdmap=2576795721}) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:14.285 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:13 smithi082 ceph-mon[44206]: osdmap e579: 12 total, 11 up, 11 in 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=1863921568,config=1059521453,health=3043139144,kv=381200665}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2522943856,logm=1032422407}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54172305}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721340623}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923487141}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67155288}) 2024-04-15T18:34:14.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3910019489}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066552193}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2487532569}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408340601}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603121907}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663758718}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658537291}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4158873647}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287902942}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2540796699}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=366970885}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1184220640}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2271315294}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792257746}) 2024-04-15T18:34:14.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3291025418}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323034123}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678854950}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2785654052}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926225955}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3680904382}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2178149898}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135290105}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847896767}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=732029950}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=625083285}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3330908033}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283093111}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=668701426}) 2024-04-15T18:34:14.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779861395}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036691526}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1632322635}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3962739681}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168174409}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660863347}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407647598}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225498983}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3709676346}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1979023490}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535558476}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=187271867}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026539367}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3686557541}) 2024-04-15T18:34:14.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946964263}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414036723}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3045725819}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3252957429}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984424195}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935321636}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2411303081}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=549124670}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054662748}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=476335396}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924404717}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=410191971}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287127493}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801873616}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3864933299}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3316436963}) 2024-04-15T18:34:14.401 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3510707053}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425514705}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=322990948}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=47666720,mdsmap=2840952586,mgr=798432949}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mgr=14,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=70} crc {mgr=187916674,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=712996742,mon_config_key=2965594423}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2530393979}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1948904719}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1746849822,monmap=1029477071,osd_metadata=175868875,osd_pg_creating=749074650,osdmap=3576918729}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=42} crc {osdmap=2576795721}) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:14.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:13 smithi159 ceph-mon[45336]: osdmap e579: 12 total, 11 up, 11 in 2024-04-15T18:34:14.759 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:14.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:14.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:14.903 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:15.116 INFO:teuthology.orchestra.run.smithi082.stdout:6.a 2024-04-15T18:34:15.116 INFO:teuthology.orchestra.run.smithi082.stdout:6.5 2024-04-15T18:34:15.116 INFO:teuthology.orchestra.run.smithi082.stdout:1.0 2024-04-15T18:34:15.116 INFO:teuthology.orchestra.run.smithi082.stdout:6.c 2024-04-15T18:34:15.116 INFO:teuthology.orchestra.run.smithi082.stdout:6.3 2024-04-15T18:34:15.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:14 smithi049 ceph-mon[40087]: pgmap v1500: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 29 MiB/s wr, 674 op/s; 20834/264660 objects degraded (7.872%); 8.2 MiB/s, 129 objects/s recovering 2024-04-15T18:34:15.260 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:15.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:15.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:14 smithi082 ceph-mon[44206]: pgmap v1500: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 29 MiB/s wr, 674 op/s; 20834/264660 objects degraded (7.872%); 8.2 MiB/s, 129 objects/s recovering 2024-04-15T18:34:15.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:15.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:14 smithi159 ceph-mon[45336]: pgmap v1500: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 128 B/s rd, 29 MiB/s wr, 674 op/s; 20834/264660 objects degraded (7.872%); 8.2 MiB/s, 129 objects/s recovering 2024-04-15T18:34:15.403 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:15.487 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:15.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:15.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:15.630 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:15.652 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:15.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:15.759 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:15.793 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:15.837 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:34:15.837 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:34:15.882 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:15.919 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:15.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:16.026 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:16.059 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:16.085 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:16.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:16.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:15 smithi049 ceph-mon[40087]: 6.e scrub starts 2024-04-15T18:34:16.189 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:16.220 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:16.250 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:16.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:16.255 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:15 smithi159 ceph-mon[45336]: 6.e scrub starts 2024-04-15T18:34:16.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:15 smithi082 ceph-mon[44206]: 6.e scrub starts 2024-04-15T18:34:16.355 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:16.389 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:16.407 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:16.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:16.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:16.548 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:16.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:34:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:16 smithi049 ceph-mon[40087]: pgmap v1502: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 19 MiB/s wr, 548 op/s; 20834/263934 objects degraded (7.894%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:34:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:16 smithi049 ceph-mon[40087]: 6.6 scrub starts 2024-04-15T18:34:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:16 smithi049 ceph-mon[40087]: 6.e scrub ok 2024-04-15T18:34:17.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:34:17.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:16 smithi082 ceph-mon[44206]: pgmap v1502: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 19 MiB/s wr, 548 op/s; 20834/263934 objects degraded (7.894%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:34:17.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:16 smithi082 ceph-mon[44206]: 6.6 scrub starts 2024-04-15T18:34:17.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:16 smithi082 ceph-mon[44206]: 6.e scrub ok 2024-04-15T18:34:17.313 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:17.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:17.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:16 smithi159 ceph-mon[45336]: pgmap v1502: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 127 B/s rd, 19 MiB/s wr, 548 op/s; 20834/263934 objects degraded (7.894%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:34:17.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:16 smithi159 ceph-mon[45336]: 6.6 scrub starts 2024-04-15T18:34:17.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:16 smithi159 ceph-mon[45336]: 6.e scrub ok 2024-04-15T18:34:17.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:34:17.450 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:17.478 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:34:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:17.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:34:17.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:17.611 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:17.640 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:17.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:17.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:17.778 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:17.805 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:17.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:17.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:17.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:18.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 20834/263934 objects degraded (7.894%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:18.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 20834/263934 objects degraded (7.894%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:18.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 20834/263934 objects degraded (7.894%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:18.691 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:18.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:18.738 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:18.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:18.812 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:18.857 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:19.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:18 smithi082 ceph-mon[44206]: pgmap v1503: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 624 op/s; 20402/258426 objects degraded (7.895%); 9.0 MiB/s, 144 objects/s recovering 2024-04-15T18:34:19.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:18 smithi082 ceph-mon[44206]: 6.6 scrub ok 2024-04-15T18:34:19.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:18 smithi159 ceph-mon[45336]: pgmap v1503: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 624 op/s; 20402/258426 objects degraded (7.895%); 9.0 MiB/s, 144 objects/s recovering 2024-04-15T18:34:19.402 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:18 smithi159 ceph-mon[45336]: 6.6 scrub ok 2024-04-15T18:34:19.415 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:19.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:19.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:18 smithi049 ceph-mon[40087]: pgmap v1503: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 16 MiB/s wr, 624 op/s; 20402/258426 objects degraded (7.895%); 9.0 MiB/s, 144 objects/s recovering 2024-04-15T18:34:19.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:18 smithi049 ceph-mon[40087]: 6.6 scrub ok 2024-04-15T18:34:19.490 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:19.521 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:19.529 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:19.554 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:19.572 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:19.577 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:19.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:19.614 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:19.652 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:19.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:19.687 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:19.725 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:19.731 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:19.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:19.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:19.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:19.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:19.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:19.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:19.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:19.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:20.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3765424020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:20.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3765424020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:20.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3765424020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:20.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:20.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.089 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:21.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:21.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:21.232 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 9.3 MiB/s wr, 680 op/s; 20051/253710 objects degraded (7.903%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 9.3 MiB/s wr, 680 op/s; 20051/253710 objects degraded (7.903%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: pgmap v1504: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 9.3 MiB/s wr, 680 op/s; 20051/253710 objects degraded (7.903%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:21.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:22.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:22.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.6 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 6.3 --file /var/log/ceph/exp.6.3.6 2024-04-15T18:34:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:23 smithi049 ceph-mon[40087]: pgmap v1505: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 4.9 MiB/s wr, 692 op/s; 19610/250581 objects degraded (7.826%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:23.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 19610/250581 objects degraded (7.826%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:23.198 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:23.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:23.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:23 smithi082 ceph-mon[44206]: pgmap v1505: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 4.9 MiB/s wr, 692 op/s; 19610/250581 objects degraded (7.826%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:23.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 19610/250581 objects degraded (7.826%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:23.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:23.336 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:23 smithi159 ceph-mon[45336]: pgmap v1505: 17 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 102 B/s rd, 4.9 MiB/s wr, 692 op/s; 19610/250581 objects degraded (7.826%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:23.336 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 19610/250581 objects degraded (7.826%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:23.338 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:23.361 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:23.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:23.433 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.6/config 2024-04-15T18:34:23.465 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:23.496 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:23.525 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:23.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:23.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:23.687 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:23.910 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:23.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:34:23.967 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:34:23.967 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:34:23.967 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:34:23.967 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:34:24.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:24.066 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:24.222 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:24.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:24.326 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:24.352 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:24.376 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:24.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:24.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:24.513 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:24.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:34:24.671 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:24.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:24.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:34:24.777 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:24.815 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:24.931 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:34:25.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:25 smithi049 ceph-mon[40087]: pgmap v1506: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 19266/243690 objects degraded (7.906%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:25.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3006496630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:25 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:25 smithi082 ceph-mon[44206]: pgmap v1506: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 19266/243690 objects degraded (7.906%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3006496630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:25 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:25 smithi159 ceph-mon[45336]: pgmap v1506: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 19266/243690 objects degraded (7.906%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:25.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3006496630' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:25 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:25.876 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:25.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:25.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:26.007 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:26.028 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:26.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:26.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:26.178 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:26 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:26.178 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:26 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:26.178 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:26 smithi082 ceph-mon[44206]: osdmap e580: 12 total, 11 up, 11 in 2024-04-15T18:34:26.192 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:26.228 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:26.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:26.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:26.392 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:26.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:26 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:26.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:26 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:26.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:26 smithi159 ceph-mon[45336]: osdmap e580: 12 total, 11 up, 11 in 2024-04-15T18:34:26.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:26 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:26.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:26 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:26.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:26 smithi049 ceph-mon[40087]: osdmap e580: 12 total, 11 up, 11 in 2024-04-15T18:34:26.510 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:26.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:26.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:26.645 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:26.998 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:27.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:27.014 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:27.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:27.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:27.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:27 smithi049 ceph-mon[40087]: pgmap v1508: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 19266/242991 objects degraded (7.929%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:27.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:27 smithi082 ceph-mon[44206]: pgmap v1508: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 19266/242991 objects degraded (7.929%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:27.279 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:27.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:27.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:27.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:27 smithi159 ceph-mon[45336]: pgmap v1508: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 19266/242991 objects degraded (7.929%); 9.8 MiB/s, 156 objects/s recovering 2024-04-15T18:34:27.418 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:27.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:27.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:27.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:27.595 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:28.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:34:28.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:28.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:28.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 19266/242991 objects degraded (7.929%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:28.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3273001760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:28.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:28.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:28.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:28.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 19266/242991 objects degraded (7.929%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:28.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3273001760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:28.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:28.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:28.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:28.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 19266/242991 objects degraded (7.929%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:28.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3273001760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:28.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:28.659 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:28.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:28.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:29.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:29.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:29.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: pgmap v1509: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 18797/237741 objects degraded (7.907%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:34:29.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osdmap e581: 12 total, 11 up, 11 in 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:29.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:29 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:29.287 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:29.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:29.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: pgmap v1509: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 18797/237741 objects degraded (7.907%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:34:29.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.344 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osdmap e581: 12 total, 11 up, 11 in 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:29.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:29 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:29.355 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:29.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:29.383 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:29.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: pgmap v1509: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 18797/237741 objects degraded (7.907%); 10 MiB/s, 160 objects/s recovering 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osdmap e581: 12 total, 11 up, 11 in 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:29 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:29.465 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:29.491 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:29.515 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:29.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:29.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:29.645 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:29.656 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:30.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:30 smithi049 ceph-mon[40087]: osdmap e582: 12 total, 11 up, 11 in 2024-04-15T18:34:30.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:30.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:30.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:30.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:30.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:30 smithi159 ceph-mon[45336]: osdmap e582: 12 total, 11 up, 11 in 2024-04-15T18:34:30.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:30.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:30.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:30.400 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:30.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:30 smithi082 ceph-mon[44206]: osdmap e582: 12 total, 11 up, 11 in 2024-04-15T18:34:30.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:30.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:30.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:30.415 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:30.580 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:34:30.580 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:34:31.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:34:32.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:32 smithi049 ceph-mon[40087]: pgmap v1513: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 17993/229746 objects degraded (7.832%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:34:32.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 17993/229746 objects degraded (7.832%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:32.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:32 smithi082 ceph-mon[44206]: pgmap v1513: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 17993/229746 objects degraded (7.832%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:34:32.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 17993/229746 objects degraded (7.832%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:32 smithi159 ceph-mon[45336]: pgmap v1513: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 17993/229746 objects degraded (7.832%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:34:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 17993/229746 objects degraded (7.832%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:33.359 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:33.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:34:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:33.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:33.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:33.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:33.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:33.654 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:34:33.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:33.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:33.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:33.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:33.787 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:33.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:33.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:33.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:34.940 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:34.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:35.046 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:35.082 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:35.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:35 smithi049 ceph-mon[40087]: pgmap v1514: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 17593/221892 objects degraded (7.929%); 13 MiB/s, 209 objects/s recovering 2024-04-15T18:34:35.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:35 smithi159 ceph-mon[45336]: pgmap v1514: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 17593/221892 objects degraded (7.929%); 13 MiB/s, 209 objects/s recovering 2024-04-15T18:34:35.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:35 smithi082 ceph-mon[44206]: pgmap v1514: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 17593/221892 objects degraded (7.929%); 13 MiB/s, 209 objects/s recovering 2024-04-15T18:34:35.696 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:35.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:35.745 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:35.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:35.765 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:35.801 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:35.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:34:35.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:35.884 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:35.911 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:34:35.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:35.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:34:36.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:36.054 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:36.076 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:36.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:36.111 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:34:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2527218033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.181 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:36.216 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:36.244 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:36.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:36.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:36.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:36.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2527218033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:36.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.394 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:36.437 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:36.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:36.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:36.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2527218033' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:36.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:36.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:36.572 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:36.597 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:36.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:36.703 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:36.745 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:36.749 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:36.775 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:36.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:36.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:36.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:36.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:36.963 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:36.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:37.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:37.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: pgmap v1515: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 17593/221181 objects degraded (7.954%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osdmap e583: 12 total, 11 up, 11 in 2024-04-15T18:34:37.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: pgmap v1515: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 17593/221181 objects degraded (7.954%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:34:37.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osdmap e583: 12 total, 11 up, 11 in 2024-04-15T18:34:37.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: pgmap v1515: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 17593/221181 objects degraded (7.954%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:34:37.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/107080462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osdmap e583: 12 total, 11 up, 11 in 2024-04-15T18:34:37.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:37.658 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:37.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:37.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:37.796 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:37.901 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:37.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:38.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:38.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:38.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:38 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 17593/221181 objects degraded (7.954%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:38.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:38.379 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:38.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:38.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:38 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 17593/221181 objects degraded (7.954%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:38.435 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:34:38.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:38.518 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:38.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:38.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:38 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 17593/221181 objects degraded (7.954%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:38.545 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:38.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:38.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:38.685 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:38.710 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:38.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:38.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:38.845 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:38.868 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:38.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:38.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:39.006 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.034 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:39.141 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:39.172 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.196 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:39.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:39.343 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.371 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:39.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:39 smithi159 ceph-mon[45336]: pgmap v1517: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 17173/215418 objects degraded (7.972%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:34:39.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:39 smithi049 ceph-mon[40087]: pgmap v1517: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 17173/215418 objects degraded (7.972%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:34:39.480 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:34:39.510 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:39 smithi082 ceph-mon[44206]: pgmap v1517: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 17173/215418 objects degraded (7.972%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:34:39.530 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:34:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:34:39.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:39.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:34:39.667 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.705 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:39.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:39.850 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:39.879 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:39.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:39.983 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:40.041 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:40.056 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:40.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:34:40.544 INFO:teuthology.orchestra.run.smithi082.stderr:Exporting 6.3 info 6.3( v 561'3617 (0'0,561'3617] local-lis/les=541/542 n=3617 ec=541/541 lis/c=541/541 les/c/f=542/542/0 sis=541) 2024-04-15T18:34:40.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c002130a:::benchmark_data_smithi159_139342_object39504:head# 2024-04-15T18:34:40.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c002133f:::benchmark_data_smithi159_139342_object19706:head# 2024-04-15T18:34:40.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00285a9:::benchmark_data_smithi159_139342_object38827:head# 2024-04-15T18:34:40.564 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0029ebe:::benchmark_data_smithi159_139342_object12995:head# 2024-04-15T18:34:40.564 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0038130:::benchmark_data_smithi159_139342_object39221:head# 2024-04-15T18:34:40.565 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00792cc:::benchmark_data_smithi159_139342_object34550:head# 2024-04-15T18:34:40.565 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c008c4d5:::benchmark_data_smithi159_139342_object46254:head# 2024-04-15T18:34:40.565 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0098b17:::benchmark_data_smithi159_139342_object24966:head# 2024-04-15T18:34:40.566 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c009b657:::benchmark_data_smithi159_139342_object238:head# 2024-04-15T18:34:40.566 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00a7de3:::benchmark_data_smithi159_139342_object40841:head# 2024-04-15T18:34:40.567 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00bb310:::benchmark_data_smithi159_139342_object2884:head# 2024-04-15T18:34:40.567 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00bf5c9:::benchmark_data_smithi159_139342_object22049:head# 2024-04-15T18:34:40.568 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00cdf97:::benchmark_data_smithi159_139342_object48773:head# 2024-04-15T18:34:40.568 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00dd19e:::benchmark_data_smithi159_139342_object35291:head# 2024-04-15T18:34:40.568 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00e58e2:::benchmark_data_smithi159_139342_object556:head# 2024-04-15T18:34:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c00ea767:::benchmark_data_smithi159_139342_object41515:head# 2024-04-15T18:34:40.569 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c010428f:::benchmark_data_smithi159_139342_object27847:head# 2024-04-15T18:34:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0143a99:::benchmark_data_smithi159_139342_object52461:head# 2024-04-15T18:34:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0146c62:::benchmark_data_smithi159_139342_object16386:head# 2024-04-15T18:34:40.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c014fdd2:::benchmark_data_smithi159_139342_object18051:head# 2024-04-15T18:34:40.571 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0159ba7:::benchmark_data_smithi159_139342_object26964:head# 2024-04-15T18:34:40.572 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0173924:::benchmark_data_smithi159_139342_object12030:head# 2024-04-15T18:34:40.574 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c01bd227:::benchmark_data_smithi159_139342_object54735:head# 2024-04-15T18:34:40.581 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c01c88b2:::benchmark_data_smithi159_139342_object23240:head# 2024-04-15T18:34:40.581 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c01d29b2:::benchmark_data_smithi159_139342_object25027:head# 2024-04-15T18:34:40.582 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c01d69eb:::benchmark_data_smithi159_139342_object56885:head# 2024-04-15T18:34:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0200167:::benchmark_data_smithi159_139342_object32266:head# 2024-04-15T18:34:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c021fe3e:::benchmark_data_smithi159_139342_object34453:head# 2024-04-15T18:34:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0225b23:::benchmark_data_smithi159_139342_object10141:head# 2024-04-15T18:34:40.583 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0257073:::benchmark_data_smithi159_139342_object54794:head# 2024-04-15T18:34:40.584 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0258f5b:::benchmark_data_smithi159_139342_object8094:head# 2024-04-15T18:34:40.586 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c02611d3:::benchmark_data_smithi159_139342_object36330:head# 2024-04-15T18:34:40.586 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c027225a:::benchmark_data_smithi159_139342_object9958:head# 2024-04-15T18:34:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c02a9e2e:::benchmark_data_smithi159_139342_object46664:head# 2024-04-15T18:34:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c02eee17:::benchmark_data_smithi159_139342_object16829:head# 2024-04-15T18:34:40.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c02f4652:::benchmark_data_smithi159_139342_object4369:head# 2024-04-15T18:34:40.592 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c02f69b7:::benchmark_data_smithi159_139342_object35102:head# 2024-04-15T18:34:40.592 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0301139:::benchmark_data_smithi159_139342_object31928:head# 2024-04-15T18:34:40.592 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0301840:::benchmark_data_smithi159_139342_object50480:head# 2024-04-15T18:34:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0308220:::benchmark_data_smithi159_139342_object42109:head# 2024-04-15T18:34:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0332639:::benchmark_data_smithi159_139342_object14290:head# 2024-04-15T18:34:40.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c033891a:::benchmark_data_smithi159_139342_object54203:head# 2024-04-15T18:34:40.596 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0347c88:::benchmark_data_smithi159_139342_object20082:head# 2024-04-15T18:34:40.596 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c034e434:::benchmark_data_smithi159_139342_object10271:head# 2024-04-15T18:34:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c035d917:::benchmark_data_smithi159_139342_object52368:head# 2024-04-15T18:34:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c037fcd7:::benchmark_data_smithi159_139342_object44932:head# 2024-04-15T18:34:40.597 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c03a81e6:::benchmark_data_smithi159_139342_object56720:head# 2024-04-15T18:34:40.598 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c03e5413:::benchmark_data_smithi159_139342_object2151:head# 2024-04-15T18:34:40.598 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c03fd0b6:::benchmark_data_smithi159_139342_object39735:head# 2024-04-15T18:34:40.599 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c042428a:::benchmark_data_smithi159_139342_object30693:head# 2024-04-15T18:34:40.599 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0427ad7:::benchmark_data_smithi159_139342_object36110:head# 2024-04-15T18:34:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c043a567:::benchmark_data_smithi159_139342_object13352:head# 2024-04-15T18:34:40.600 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0444743:::benchmark_data_smithi159_139342_object14073:head# 2024-04-15T18:34:40.601 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0492976:::benchmark_data_smithi159_139342_object21193:head# 2024-04-15T18:34:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c049934b:::benchmark_data_smithi159_139342_object6245:head# 2024-04-15T18:34:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04b262d:::benchmark_data_smithi159_139342_object50792:head# 2024-04-15T18:34:40.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04b3b22:::benchmark_data_smithi159_139342_object42224:head# 2024-04-15T18:34:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04b5563:::benchmark_data_smithi159_139342_object33941:head# 2024-04-15T18:34:40.605 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04bbf82:::benchmark_data_smithi159_139342_object11642:head# 2024-04-15T18:34:40.606 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04c044c:::benchmark_data_smithi159_139342_object45246:head# 2024-04-15T18:34:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04ccf5a:::benchmark_data_smithi159_139342_object137:head# 2024-04-15T18:34:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04d6048:::benchmark_data_smithi159_139342_object42739:head# 2024-04-15T18:34:40.607 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04dc5ac:::benchmark_data_smithi159_139342_object50682:head# 2024-04-15T18:34:40.609 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c04fb026:::benchmark_data_smithi159_139342_object9884:head# 2024-04-15T18:34:40.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0506692:::benchmark_data_smithi159_139342_object36609:head# 2024-04-15T18:34:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0517b72:::benchmark_data_smithi159_139342_object38262:head# 2024-04-15T18:34:40.611 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c054681e:::benchmark_data_smithi159_139342_object11529:head# 2024-04-15T18:34:40.612 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0554649:::benchmark_data_smithi159_139342_object20979:head# 2024-04-15T18:34:40.613 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c056e1c3:::benchmark_data_smithi159_139342_object31367:head# 2024-04-15T18:34:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c058b07a:::benchmark_data_smithi159_139342_object38504:head# 2024-04-15T18:34:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c05929f8:::benchmark_data_smithi159_139342_object1704:head# 2024-04-15T18:34:40.614 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c059df5a:::benchmark_data_smithi159_139342_object27238:head# 2024-04-15T18:34:40.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c059efd8:::benchmark_data_smithi159_139342_object5607:head# 2024-04-15T18:34:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c05d3ca4:::benchmark_data_smithi159_139342_object40119:head# 2024-04-15T18:34:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0617eae:::benchmark_data_smithi159_139342_object43056:head# 2024-04-15T18:34:40.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0623da2:::benchmark_data_smithi159_139342_object14371:head# 2024-04-15T18:34:40.617 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c062772d:::benchmark_data_smithi159_139342_object39401:head# 2024-04-15T18:34:40.617 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c065495b:::benchmark_data_smithi159_139342_object57253:head# 2024-04-15T18:34:40.618 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c065c645:::benchmark_data_smithi159_139342_object4023:head# 2024-04-15T18:34:40.618 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c066d678:::benchmark_data_smithi159_139342_object34397:head# 2024-04-15T18:34:40.619 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0673948:::benchmark_data_smithi159_139342_object30457:head# 2024-04-15T18:34:40.619 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0688d97:::benchmark_data_smithi159_139342_object30424:head# 2024-04-15T18:34:40.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0696de1:::benchmark_data_smithi159_139342_object11112:head# 2024-04-15T18:34:40.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0696e33:::benchmark_data_smithi159_139342_object18121:head# 2024-04-15T18:34:40.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c069da58:::benchmark_data_smithi159_139342_object52710:head# 2024-04-15T18:34:40.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c06a49c5:::benchmark_data_smithi159_139342_object13696:head# 2024-04-15T18:34:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0711721:::benchmark_data_smithi159_139342_object17546:head# 2024-04-15T18:34:40.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0726498:::benchmark_data_smithi159_139342_object15966:head# 2024-04-15T18:34:40.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c073aea1:::benchmark_data_smithi159_139342_object52144:head# 2024-04-15T18:34:40.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c073ebbc:::benchmark_data_smithi159_139342_object27603:head# 2024-04-15T18:34:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0744a76:::benchmark_data_smithi159_139342_object15907:head# 2024-04-15T18:34:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0745b53:::benchmark_data_smithi159_139342_object2945:head# 2024-04-15T18:34:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c075890a:::benchmark_data_smithi159_139342_object52064:head# 2024-04-15T18:34:40.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c075b95b:::benchmark_data_smithi159_139342_object47168:head# 2024-04-15T18:34:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c077a3f0:::benchmark_data_smithi159_139342_object52499:head# 2024-04-15T18:34:40.630 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c078f785:::benchmark_data_smithi159_139342_object54740:head# 2024-04-15T18:34:40.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07af996:::benchmark_data_smithi159_139342_object31020:head# 2024-04-15T18:34:40.637 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07c15b2:::benchmark_data_smithi159_139342_object19527:head# 2024-04-15T18:34:40.638 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07d2fb0:::benchmark_data_smithi159_139342_object15902:head# 2024-04-15T18:34:40.638 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07d8844:::benchmark_data_smithi159_139342_object40076:head# 2024-04-15T18:34:40.644 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07e10eb:::benchmark_data_smithi159_139342_object28812:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c07f1752:::benchmark_data_smithi159_139342_object53473:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c083e7fc:::benchmark_data_smithi159_139342_object7756:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08530f8:::benchmark_data_smithi159_139342_object33305:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0855eec:::benchmark_data_smithi159_139342_object55211:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0857504:::benchmark_data_smithi159_139342_object49337:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c085c0ed:::benchmark_data_smithi159_139342_object17002:head# 2024-04-15T18:34:40.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c085d390:::benchmark_data_smithi159_139342_object27500:head# 2024-04-15T18:34:40.646 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0874a39:::benchmark_data_smithi159_139342_object47034:head# 2024-04-15T18:34:40.646 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08bc6d5:::benchmark_data_smithi159_139342_object2436:head# 2024-04-15T18:34:40.647 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08cd4f0:::benchmark_data_smithi159_139342_object42879:head# 2024-04-15T18:34:40.647 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08dfeba:::benchmark_data_smithi159_139342_object38603:head# 2024-04-15T18:34:40.648 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08eb02a:::benchmark_data_smithi159_139342_object50428:head# 2024-04-15T18:34:40.648 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08f30ea:::benchmark_data_smithi159_139342_object2316:head# 2024-04-15T18:34:40.649 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08f316c:::benchmark_data_smithi159_139342_object20342:head# 2024-04-15T18:34:40.649 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08f4b38:::benchmark_data_smithi159_139342_object38300:head# 2024-04-15T18:34:40.651 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c08f63c4:::benchmark_data_smithi159_139342_object34280:head# 2024-04-15T18:34:40.651 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0906d61:::benchmark_data_smithi159_139342_object53232:head# 2024-04-15T18:34:40.651 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c09341bd:::benchmark_data_smithi159_139342_object2157:head# 2024-04-15T18:34:40.652 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c09380fd:::benchmark_data_smithi159_139342_object38439:head# 2024-04-15T18:34:40.652 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0979c2b:::benchmark_data_smithi159_139342_object22801:head# 2024-04-15T18:34:40.653 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c09b4918:::benchmark_data_smithi159_139342_object13321:head# 2024-04-15T18:34:40.654 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c09c969e:::benchmark_data_smithi159_139342_object43374:head# 2024-04-15T18:34:40.654 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c09c9c7b:::benchmark_data_smithi159_139342_object32656:head# 2024-04-15T18:34:40.658 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a05cb2:::benchmark_data_smithi159_139342_object15039:head# 2024-04-15T18:34:40.659 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a08b32:::benchmark_data_smithi159_139342_object54716:head# 2024-04-15T18:34:40.660 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a0a11d:::benchmark_data_smithi159_139342_object9905:head# 2024-04-15T18:34:40.660 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a24972:::benchmark_data_smithi159_139342_object27213:head# 2024-04-15T18:34:40.661 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a27be2:::benchmark_data_smithi159_139342_object32707:head# 2024-04-15T18:34:40.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a2f6fd:::benchmark_data_smithi159_139342_object48442:head# 2024-04-15T18:34:40.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a37b23:::benchmark_data_smithi159_139342_object22593:head# 2024-04-15T18:34:40.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a50f43:::benchmark_data_smithi159_139342_object15710:head# 2024-04-15T18:34:40.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a7052c:::benchmark_data_smithi159_139342_object5230:head# 2024-04-15T18:34:40.672 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0a97cb9:::benchmark_data_smithi159_139342_object11644:head# 2024-04-15T18:34:40.672 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0adf653:::benchmark_data_smithi159_139342_object831:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0af7272:::benchmark_data_smithi159_139342_object47527:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b05a08:::benchmark_data_smithi159_139342_object2835:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b178b9:::benchmark_data_smithi159_139342_object11316:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b183e3:::benchmark_data_smithi159_139342_object4746:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b428b1:::benchmark_data_smithi159_139342_object51501:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b5e8f6:::benchmark_data_smithi159_139342_object49426:head# 2024-04-15T18:34:40.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0b6406c:::benchmark_data_smithi159_139342_object41499:head# 2024-04-15T18:34:40.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ba4258:::benchmark_data_smithi159_139342_object36232:head# 2024-04-15T18:34:40.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ba5a46:::benchmark_data_smithi159_139342_object6372:head# 2024-04-15T18:34:40.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0bae154:::benchmark_data_smithi159_139342_object35305:head# 2024-04-15T18:34:40.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0bc9a97:::benchmark_data_smithi159_139342_object14919:head# 2024-04-15T18:34:40.675 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0bcae3d:::benchmark_data_smithi159_139342_object50391:head# 2024-04-15T18:34:40.675 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0bcafbd:::benchmark_data_smithi159_139342_object26143:head# 2024-04-15T18:34:40.676 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c0d1c8:::benchmark_data_smithi159_139342_object31130:head# 2024-04-15T18:34:40.676 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c0e3ef:::benchmark_data_smithi159_139342_object15535:head# 2024-04-15T18:34:40.676 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c11552:::benchmark_data_smithi159_139342_object4972:head# 2024-04-15T18:34:40.677 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c14709:::benchmark_data_smithi159_139342_object52854:head# 2024-04-15T18:34:40.677 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c3670e:::benchmark_data_smithi159_139342_object5942:head# 2024-04-15T18:34:40.678 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c6aeb7:::benchmark_data_smithi159_139342_object45735:head# 2024-04-15T18:34:40.678 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c7667a:::benchmark_data_smithi159_139342_object48770:head# 2024-04-15T18:34:40.678 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c84222:::benchmark_data_smithi159_139342_object42921:head# 2024-04-15T18:34:40.679 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c8a594:::benchmark_data_smithi159_139342_object42577:head# 2024-04-15T18:34:40.679 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0c980a5:::benchmark_data_smithi159_139342_object56991:head# 2024-04-15T18:34:40.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0cb8c97:::benchmark_data_smithi159_139342_object7178:head# 2024-04-15T18:34:40.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0cc8f54:::benchmark_data_smithi159_139342_object22101:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0cd62b4:::benchmark_data_smithi159_139342_object39607:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ceb71a:::benchmark_data_smithi159_139342_object51403:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0cfa1a5:::benchmark_data_smithi159_139342_object23214:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0cfcab8:::benchmark_data_smithi159_139342_object35823:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d59503:::benchmark_data_smithi159_139342_object43285:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d68cbd:::benchmark_data_smithi159_139342_object51603:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d6b335:::benchmark_data_smithi159_139342_object13916:head# 2024-04-15T18:34:40.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d6bf09:::benchmark_data_smithi159_139342_object12380:head# 2024-04-15T18:34:40.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d6f95f:::benchmark_data_smithi159_139342_object30931:head# 2024-04-15T18:34:40.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0d94312:::benchmark_data_smithi159_139342_object19667:head# 2024-04-15T18:34:40.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0de97b3:::benchmark_data_smithi159_139342_object50225:head# 2024-04-15T18:34:40.690 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e05a69:::benchmark_data_smithi159_139342_object49212:head# 2024-04-15T18:34:40.690 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e08557:::benchmark_data_smithi159_139342_object37156:head# 2024-04-15T18:34:40.690 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e40452:::benchmark_data_smithi159_139342_object27172:head# 2024-04-15T18:34:40.690 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e452c6:::benchmark_data_smithi159_139342_object55728:head# 2024-04-15T18:34:40.691 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e69664:::benchmark_data_smithi159_139342_object15704:head# 2024-04-15T18:34:40.691 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0e98ff2:::benchmark_data_smithi159_139342_object31240:head# 2024-04-15T18:34:40.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ea9488:::benchmark_data_smithi159_139342_object31013:head# 2024-04-15T18:34:40.693 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ebc682:::benchmark_data_smithi159_139342_object32551:head# 2024-04-15T18:34:40.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ebfe6e:::benchmark_data_smithi159_139342_object22849:head# 2024-04-15T18:34:40.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ec037f:::benchmark_data_smithi159_139342_object53540:head# 2024-04-15T18:34:40.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ecdb83:::benchmark_data_smithi159_139342_object41755:head# 2024-04-15T18:34:40.695 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ee4917:::benchmark_data_smithi159_139342_object31937:head# 2024-04-15T18:34:40.695 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0ef57f3:::benchmark_data_smithi159_139342_object32445:head# 2024-04-15T18:34:40.696 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f0fd40:::benchmark_data_smithi159_139342_object15458:head# 2024-04-15T18:34:40.696 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f1c901:::benchmark_data_smithi159_139342_object8131:head# 2024-04-15T18:34:40.697 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f222d1:::benchmark_data_smithi159_139342_object30144:head# 2024-04-15T18:34:40.697 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f3a49c:::benchmark_data_smithi159_139342_object22023:head# 2024-04-15T18:34:40.698 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f6051c:::benchmark_data_smithi159_139342_object37363:head# 2024-04-15T18:34:40.698 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f6dc6e:::benchmark_data_smithi159_139342_object48884:head# 2024-04-15T18:34:40.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f6f050:::benchmark_data_smithi159_139342_object20265:head# 2024-04-15T18:34:40.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f96e62:::benchmark_data_smithi159_139342_object24884:head# 2024-04-15T18:34:40.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0f9f249:::benchmark_data_smithi159_139342_object57041:head# 2024-04-15T18:34:40.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0fd14ec:::benchmark_data_smithi159_139342_object1631:head# 2024-04-15T18:34:40.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0fe7bfd:::benchmark_data_smithi159_139342_object28553:head# 2024-04-15T18:34:40.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c0fefc07:::benchmark_data_smithi159_139342_object31866:head# 2024-04-15T18:34:40.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10024e6:::benchmark_data_smithi159_139342_object46598:head# 2024-04-15T18:34:40.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c100a9c9:::benchmark_data_smithi159_139342_object22688:head# 2024-04-15T18:34:40.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1010a89:::benchmark_data_smithi159_139342_object28241:head# 2024-04-15T18:34:40.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1039c5f:::benchmark_data_smithi159_139342_object23835:head# 2024-04-15T18:34:40.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1059772:::benchmark_data_smithi159_139342_object57201:head# 2024-04-15T18:34:40.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c105cb78:::benchmark_data_smithi159_139342_object22964:head# 2024-04-15T18:34:40.704 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c105fd6c:::benchmark_data_smithi159_139342_object18202:head# 2024-04-15T18:34:40.704 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1062c70:::benchmark_data_smithi159_139342_object31911:head# 2024-04-15T18:34:40.705 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c107be8c:::benchmark_data_smithi159_139342_object54088:head# 2024-04-15T18:34:40.705 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c108634c:::benchmark_data_smithi159_139342_object19091:head# 2024-04-15T18:34:40.706 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c109c342:::benchmark_data_smithi159_139342_object54772:head# 2024-04-15T18:34:40.706 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10ac302:::benchmark_data_smithi159_139342_object25850:head# 2024-04-15T18:34:40.707 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10be188:::benchmark_data_smithi159_139342_object11090:head# 2024-04-15T18:34:40.707 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10d1f0b:::benchmark_data_smithi159_139342_object24275:head# 2024-04-15T18:34:40.708 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10e07a7:::benchmark_data_smithi159_139342_object56583:head# 2024-04-15T18:34:40.708 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10f2d39:::benchmark_data_smithi159_139342_object56250:head# 2024-04-15T18:34:40.709 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c10f686b:::benchmark_data_smithi159_139342_object16761:head# 2024-04-15T18:34:40.709 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1105b38:::benchmark_data_smithi159_139342_object49285:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1122c11:::benchmark_data_smithi159_139342_object49147:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c112a63a:::benchmark_data_smithi159_139342_object22631:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1141785:::benchmark_data_smithi159_139342_object49663:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c114e1dd:::benchmark_data_smithi159_139342_object42279:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c11564a9:::benchmark_data_smithi159_139342_object39697:head# 2024-04-15T18:34:40.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1197e51:::benchmark_data_smithi159_139342_object32342:head# 2024-04-15T18:34:40.717 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:40.718 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c11b7999:::benchmark_data_smithi159_139342_object4759:head# 2024-04-15T18:34:40.719 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c11d1414:::benchmark_data_smithi159_139342_object4749:head# 2024-04-15T18:34:40.719 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1215c05:::benchmark_data_smithi159_139342_object24839:head# 2024-04-15T18:34:40.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c121e6a4:::benchmark_data_smithi159_139342_object14961:head# 2024-04-15T18:34:40.721 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c124cc08:::benchmark_data_smithi159_139342_object12063:head# 2024-04-15T18:34:40.721 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1250d74:::benchmark_data_smithi159_139342_object22047:head# 2024-04-15T18:34:40.721 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1258f08:::benchmark_data_smithi159_139342_object47093:head# 2024-04-15T18:34:40.723 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1262e54:::benchmark_data_smithi159_139342_object29039:head# 2024-04-15T18:34:40.723 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1288a9e:::benchmark_data_smithi159_139342_object11842:head# 2024-04-15T18:34:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:40.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:40.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c129b1ec:::benchmark_data_smithi159_139342_object49974:head# 2024-04-15T18:34:40.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12cb96f:::benchmark_data_smithi159_139342_object24626:head# 2024-04-15T18:34:40.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12d96e3:::benchmark_data_smithi159_139342_object44431:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12da5ba:::benchmark_data_smithi159_139342_object56507:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12e9893:::benchmark_data_smithi159_139342_object53553:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12ee8a6:::benchmark_data_smithi159_139342_object52147:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c12f6933:::benchmark_data_smithi159_139342_object33742:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1300bf1:::benchmark_data_smithi159_139342_object21558:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c130250b:::benchmark_data_smithi159_139342_object2545:head# 2024-04-15T18:34:40.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c132b787:::benchmark_data_smithi159_139342_object46688:head# 2024-04-15T18:34:40.733 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c133a657:::benchmark_data_smithi159_139342_object10013:head# 2024-04-15T18:34:40.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c135d4fd:::benchmark_data_smithi159_139342_object28707:head# 2024-04-15T18:34:40.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c137da38:::benchmark_data_smithi159_139342_object44545:head# 2024-04-15T18:34:40.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c137defd:::benchmark_data_smithi159_139342_object31414:head# 2024-04-15T18:34:40.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1380756:::benchmark_data_smithi159_139342_object9436:head# 2024-04-15T18:34:40.737 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c138c484:::benchmark_data_smithi159_139342_object489:head# 2024-04-15T18:34:40.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c13a7c2a:::benchmark_data_smithi159_139342_object34756:head# 2024-04-15T18:34:40.745 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c13d26fe:::benchmark_data_smithi159_139342_object45506:head# 2024-04-15T18:34:40.747 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c13daab6:::benchmark_data_smithi159_139342_object37823:head# 2024-04-15T18:34:40.747 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14052fa:::benchmark_data_smithi159_139342_object837:head# 2024-04-15T18:34:40.748 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c140716d:::benchmark_data_smithi159_139342_object18591:head# 2024-04-15T18:34:40.748 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c140f908:::benchmark_data_smithi159_139342_object54814:head# 2024-04-15T18:34:40.749 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1436e69:::benchmark_data_smithi159_139342_object11176:head# 2024-04-15T18:34:40.749 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1440e14:::benchmark_data_smithi159_139342_object31330:head# 2024-04-15T18:34:40.750 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1441e50:::benchmark_data_smithi159_139342_object46931:head# 2024-04-15T18:34:40.750 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1489dcb:::benchmark_data_smithi159_139342_object8690:head# 2024-04-15T18:34:40.750 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c149a079:::benchmark_data_smithi159_139342_object42310:head# 2024-04-15T18:34:40.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c149cfe0:::benchmark_data_smithi159_139342_object38743:head# 2024-04-15T18:34:40.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14b435c:::benchmark_data_smithi159_139342_object37175:head# 2024-04-15T18:34:40.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14b5240:::benchmark_data_smithi159_139342_object7507:head# 2024-04-15T18:34:40.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14bada4:::benchmark_data_smithi159_139342_object54120:head# 2024-04-15T18:34:40.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14cbf00:::benchmark_data_smithi159_139342_object31756:head# 2024-04-15T18:34:40.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14cec93:::benchmark_data_smithi159_139342_object38157:head# 2024-04-15T18:34:40.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14ced4a:::benchmark_data_smithi159_139342_object31791:head# 2024-04-15T18:34:40.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c14d0ec3:::benchmark_data_smithi159_139342_object630:head# 2024-04-15T18:34:40.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1528ae0:::benchmark_data_smithi159_139342_object57254:head# 2024-04-15T18:34:40.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c152e5e8:::benchmark_data_smithi159_139342_object25277:head# 2024-04-15T18:34:40.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c153fe93:::benchmark_data_smithi159_139342_object21336:head# 2024-04-15T18:34:40.759 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15460de:::benchmark_data_smithi159_139342_object39228:head# 2024-04-15T18:34:40.759 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c154c492:::benchmark_data_smithi159_139342_object30391:head# 2024-04-15T18:34:40.760 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15532db:::benchmark_data_smithi159_139342_object19032:head# 2024-04-15T18:34:40.761 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15646ac:::benchmark_data_smithi159_139342_object27760:head# 2024-04-15T18:34:40.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1567d7f:::benchmark_data_smithi159_139342_object54082:head# 2024-04-15T18:34:40.765 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15755db:::benchmark_data_smithi159_139342_object20058:head# 2024-04-15T18:34:40.766 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1576b02:::benchmark_data_smithi159_139342_object16039:head# 2024-04-15T18:34:40.766 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1580e7f:::benchmark_data_smithi159_139342_object56332:head# 2024-04-15T18:34:40.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1591b24:::benchmark_data_smithi159_139342_object19637:head# 2024-04-15T18:34:40.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15950b2:::benchmark_data_smithi159_139342_object7005:head# 2024-04-15T18:34:40.768 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1595fb9:::benchmark_data_smithi159_139342_object13451:head# 2024-04-15T18:34:40.768 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15a4fd0:::benchmark_data_smithi159_139342_object57035:head# 2024-04-15T18:34:40.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15ab195:::benchmark_data_smithi159_139342_object19964:head# 2024-04-15T18:34:40.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15b5257:::benchmark_data_smithi159_139342_object5891:head# 2024-04-15T18:34:40.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15be175:::benchmark_data_smithi159_139342_object26118:head# 2024-04-15T18:34:40.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c15e6d9e:::benchmark_data_smithi159_139342_object13610:head# 2024-04-15T18:34:40.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c160b5c5:::benchmark_data_smithi159_139342_object54619:head# 2024-04-15T18:34:40.773 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1622901:::benchmark_data_smithi159_139342_object44475:head# 2024-04-15T18:34:40.773 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16247a4:::benchmark_data_smithi159_139342_object13247:head# 2024-04-15T18:34:40.774 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c162f2d9:::benchmark_data_smithi159_139342_object18157:head# 2024-04-15T18:34:40.774 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c163ab5c:::benchmark_data_smithi159_139342_object16944:head# 2024-04-15T18:34:40.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c166183c:::benchmark_data_smithi159_139342_object23910:head# 2024-04-15T18:34:40.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c168a638:::benchmark_data_smithi159_139342_object15146:head# 2024-04-15T18:34:40.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c169319a:::benchmark_data_smithi159_139342_object56077:head# 2024-04-15T18:34:40.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c169b458:::benchmark_data_smithi159_139342_object3752:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16a1c91:::benchmark_data_smithi159_139342_object40980:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16b4e39:::benchmark_data_smithi159_139342_object4573:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16b9fb0:::benchmark_data_smithi159_139342_object35805:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16ba253:::benchmark_data_smithi159_139342_object44401:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16d2086:::benchmark_data_smithi159_139342_object54279:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c16e6d7e:::benchmark_data_smithi159_139342_object26910:head# 2024-04-15T18:34:40.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1707d54:::benchmark_data_smithi159_139342_object2603:head# 2024-04-15T18:34:40.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c171744d:::benchmark_data_smithi159_139342_object15061:head# 2024-04-15T18:34:40.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1725ac1:::benchmark_data_smithi159_139342_object7436:head# 2024-04-15T18:34:40.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c174dc4d:::benchmark_data_smithi159_139342_object56298:head# 2024-04-15T18:34:40.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1765781:::benchmark_data_smithi159_139342_object40701:head# 2024-04-15T18:34:40.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17735cb:::benchmark_data_smithi159_139342_object19198:head# 2024-04-15T18:34:40.783 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c178d184:::benchmark_data_smithi159_139342_object53834:head# 2024-04-15T18:34:40.783 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c179e03e:::benchmark_data_smithi159_139342_object11377:head# 2024-04-15T18:34:40.791 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17a300d:::benchmark_data_smithi159_139342_object36864:head# 2024-04-15T18:34:40.791 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17bc0ff:::benchmark_data_smithi159_139342_object334:head# 2024-04-15T18:34:40.791 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17c3338:::benchmark_data_smithi159_139342_object52864:head# 2024-04-15T18:34:40.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17ca83a:::benchmark_data_smithi159_139342_object39763:head# 2024-04-15T18:34:40.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17d94d2:::benchmark_data_smithi159_139342_object38542:head# 2024-04-15T18:34:40.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17e7e64:::benchmark_data_smithi159_139342_object27861:head# 2024-04-15T18:34:40.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17ec8f5:::benchmark_data_smithi159_139342_object14194:head# 2024-04-15T18:34:40.795 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c17f092a:::benchmark_data_smithi159_139342_object45249:head# 2024-04-15T18:34:40.796 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c180023d:::benchmark_data_smithi159_139342_object14595:head# 2024-04-15T18:34:40.796 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1801594:::benchmark_data_smithi159_139342_object47012:head# 2024-04-15T18:34:40.797 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1813486:::benchmark_data_smithi159_139342_object10927:head# 2024-04-15T18:34:40.797 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18252fd:::benchmark_data_smithi159_139342_object8799:head# 2024-04-15T18:34:40.798 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c183a85c:::benchmark_data_smithi159_139342_object13206:head# 2024-04-15T18:34:40.798 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c184b0fb:::benchmark_data_smithi159_139342_object1252:head# 2024-04-15T18:34:40.798 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c184ef7f:::benchmark_data_smithi159_139342_object12248:head# 2024-04-15T18:34:40.801 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1862b02:::benchmark_data_smithi159_139342_object6986:head# 2024-04-15T18:34:40.803 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18982c1:::benchmark_data_smithi159_139342_object35771:head# 2024-04-15T18:34:40.803 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18bb042:::benchmark_data_smithi159_139342_object51765:head# 2024-04-15T18:34:40.808 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18ce85d:::benchmark_data_smithi159_139342_object9351:head# 2024-04-15T18:34:40.812 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18d8b55:::benchmark_data_smithi159_139342_object368:head# 2024-04-15T18:34:40.812 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18dc613:::benchmark_data_smithi159_139342_object42846:head# 2024-04-15T18:34:40.812 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18e4107:::benchmark_data_smithi159_139342_object40184:head# 2024-04-15T18:34:40.812 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18e9926:::benchmark_data_smithi159_139342_object35081:head# 2024-04-15T18:34:40.812 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c18f27ac:::benchmark_data_smithi159_139342_object30593:head# 2024-04-15T18:34:40.813 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1900b12:::benchmark_data_smithi159_139342_object3305:head# 2024-04-15T18:34:40.813 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1917364:::benchmark_data_smithi159_139342_object15542:head# 2024-04-15T18:34:40.815 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c191e98d:::benchmark_data_smithi159_139342_object42532:head# 2024-04-15T18:34:40.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c192551f:::benchmark_data_smithi159_139342_object51481:head# 2024-04-15T18:34:40.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c192c2a8:::benchmark_data_smithi159_139342_object47060:head# 2024-04-15T18:34:40.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c195174c:::benchmark_data_smithi159_139342_object13555:head# 2024-04-15T18:34:40.822 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1958dca:::benchmark_data_smithi159_139342_object55171:head# 2024-04-15T18:34:40.822 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19596fc:::benchmark_data_smithi159_139342_object213:head# 2024-04-15T18:34:40.822 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1970d3a:::benchmark_data_smithi159_139342_object55888:head# 2024-04-15T18:34:40.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1978176:::benchmark_data_smithi159_139342_object7455:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1978230:::benchmark_data_smithi159_139342_object55228:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1985b12:::benchmark_data_smithi159_139342_object13293:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c198fdfb:::benchmark_data_smithi159_139342_object4750:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1992911:::benchmark_data_smithi159_139342_object48066:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19a9be4:::benchmark_data_smithi159_139342_object35635:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19adc71:::benchmark_data_smithi159_139342_object3507:head# 2024-04-15T18:34:40.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19b2d10:::benchmark_data_smithi159_139342_object51174:head# 2024-04-15T18:34:40.828 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19b7e61:::benchmark_data_smithi159_139342_object7207:head# 2024-04-15T18:34:40.829 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19be9cb:::benchmark_data_smithi159_139342_object32692:head# 2024-04-15T18:34:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19e36b1:::benchmark_data_smithi159_139342_object49183:head# 2024-04-15T18:34:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19eed71:::benchmark_data_smithi159_139342_object15516:head# 2024-04-15T18:34:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c19ff268:::benchmark_data_smithi159_139342_object40709:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1a005e7:::benchmark_data_smithi159_139342_object48893:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1a43682:::benchmark_data_smithi159_139342_object24053:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1a517b4:::benchmark_data_smithi159_139342_object8894:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1a67d18:::benchmark_data_smithi159_139342_object48370:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1a99f4c:::benchmark_data_smithi159_139342_object37202:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1aa43fb:::benchmark_data_smithi159_139342_object14313:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1abe1f0:::benchmark_data_smithi159_139342_object49252:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1af3118:::benchmark_data_smithi159_139342_object26782:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b173a9:::benchmark_data_smithi159_139342_object13248:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b1a088:::benchmark_data_smithi159_139342_object28914:head# 2024-04-15T18:34:40.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b2c6da:::benchmark_data_smithi159_139342_object44404:head# 2024-04-15T18:34:40.840 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b2c99d:::benchmark_data_smithi159_139342_object37615:head# 2024-04-15T18:34:40.840 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b356dd:::benchmark_data_smithi159_139342_object42193:head# 2024-04-15T18:34:40.840 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b375fd:::benchmark_data_smithi159_139342_object44964:head# 2024-04-15T18:34:40.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b40209:::benchmark_data_smithi159_139342_object48960:head# 2024-04-15T18:34:40.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b576fe:::benchmark_data_smithi159_139342_object15806:head# 2024-04-15T18:34:40.849 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b81a8a:::benchmark_data_smithi159_139342_object23480:head# 2024-04-15T18:34:40.850 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b8be33:::benchmark_data_smithi159_139342_object29268:head# 2024-04-15T18:34:40.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b90194:::benchmark_data_smithi159_139342_object30582:head# 2024-04-15T18:34:40.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1b95e8b:::benchmark_data_smithi159_139342_object8707:head# 2024-04-15T18:34:40.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1bc4ff0:::benchmark_data_smithi159_139342_object16124:head# 2024-04-15T18:34:40.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1bcd071:::benchmark_data_smithi159_139342_object27426:head# 2024-04-15T18:34:40.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1be573b:::benchmark_data_smithi159_139342_object4484:head# 2024-04-15T18:34:40.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1c82621:::benchmark_data_smithi159_139342_object57286:head# 2024-04-15T18:34:40.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1c87790:::benchmark_data_smithi159_139342_object717:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1c8f0f6:::benchmark_data_smithi159_139342_object16881:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1caa510:::benchmark_data_smithi159_139342_object36272:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1cac427:::benchmark_data_smithi159_139342_object47560:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1ce6ac8:::benchmark_data_smithi159_139342_object9163:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1ce796d:::benchmark_data_smithi159_139342_object660:head# 2024-04-15T18:34:40.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1cf8f49:::benchmark_data_smithi159_139342_object39902:head# 2024-04-15T18:34:40.865 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d1589a:::benchmark_data_smithi159_139342_object13850:head# 2024-04-15T18:34:40.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d375cd:::benchmark_data_smithi159_139342_object51861:head# 2024-04-15T18:34:40.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d4b400:::benchmark_data_smithi159_139342_object13112:head# 2024-04-15T18:34:40.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d76139:::benchmark_data_smithi159_139342_object15271:head# 2024-04-15T18:34:40.870 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d766ad:::benchmark_data_smithi159_139342_object39354:head# 2024-04-15T18:34:40.870 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d7fab9:::benchmark_data_smithi159_139342_object41475:head# 2024-04-15T18:34:40.871 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d84dba:::benchmark_data_smithi159_139342_object17208:head# 2024-04-15T18:34:40.871 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d8ccb4:::benchmark_data_smithi159_139342_object35835:head# 2024-04-15T18:34:40.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1d92aa4:::benchmark_data_smithi159_139342_object50433:head# 2024-04-15T18:34:40.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1dbb7b9:::benchmark_data_smithi159_139342_object14472:head# 2024-04-15T18:34:40.873 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1dc36a9:::benchmark_data_smithi159_139342_object3794:head# 2024-04-15T18:34:40.873 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1dc5b6e:::benchmark_data_smithi159_139342_object26515:head# 2024-04-15T18:34:40.873 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1dc76a4:::benchmark_data_smithi159_139342_object55021:head# 2024-04-15T18:34:40.878 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1dd61a3:::benchmark_data_smithi159_139342_object1827:head# 2024-04-15T18:34:40.878 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1e01600:::benchmark_data_smithi159_139342_object15249:head# 2024-04-15T18:34:40.878 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1e5b814:::benchmark_data_smithi159_139342_object52430:head# 2024-04-15T18:34:40.878 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1e6ce17:::benchmark_data_smithi159_139342_object5098:head# 2024-04-15T18:34:40.885 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1e706c6:::benchmark_data_smithi159_139342_object12669:head# 2024-04-15T18:34:40.885 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1e98f88:::benchmark_data_smithi159_139342_object22916:head# 2024-04-15T18:34:40.886 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1ea6b28:::benchmark_data_smithi159_139342_object35228:head# 2024-04-15T18:34:40.886 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1eaae5c:::benchmark_data_smithi159_139342_object23392:head# 2024-04-15T18:34:40.886 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1eb16f4:::benchmark_data_smithi159_139342_object19632:head# 2024-04-15T18:34:40.886 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1eb39ad:::benchmark_data_smithi159_139342_object54160:head# 2024-04-15T18:34:40.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1eb5b76:::benchmark_data_smithi159_139342_object12070:head# 2024-04-15T18:34:40.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1edc3c9:::benchmark_data_smithi159_139342_object45917:head# 2024-04-15T18:34:40.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f095ee:::benchmark_data_smithi159_139342_object26738:head# 2024-04-15T18:34:40.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f21671:::benchmark_data_smithi159_139342_object35066:head# 2024-04-15T18:34:40.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f21751:::benchmark_data_smithi159_139342_object56780:head# 2024-04-15T18:34:40.895 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f27b83:::benchmark_data_smithi159_139342_object10468:head# 2024-04-15T18:34:40.895 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f28d35:::benchmark_data_smithi159_139342_object10238:head# 2024-04-15T18:34:40.895 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f3d1de:::benchmark_data_smithi159_139342_object38213:head# 2024-04-15T18:34:40.896 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f49ae6:::benchmark_data_smithi159_139342_object3499:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f552e7:::benchmark_data_smithi159_139342_object19703:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f5abe7:::benchmark_data_smithi159_139342_object3814:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f5b360:::benchmark_data_smithi159_139342_object39693:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f6798c:::benchmark_data_smithi159_139342_object54403:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f821ff:::benchmark_data_smithi159_139342_object37409:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f8de26:::benchmark_data_smithi159_139342_object975:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1f9a3cc:::benchmark_data_smithi159_139342_object35965:head# 2024-04-15T18:34:40.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1fc9bbe:::benchmark_data_smithi159_139342_object16771:head# 2024-04-15T18:34:40.901 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1fde58f:::benchmark_data_smithi159_139342_object35625:head# 2024-04-15T18:34:40.901 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1fe5851:::benchmark_data_smithi159_139342_object5081:head# 2024-04-15T18:34:40.902 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1fed14e:::benchmark_data_smithi159_139342_object30551:head# 2024-04-15T18:34:40.903 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c1fedb5c:::benchmark_data_smithi159_139342_object5329:head# 2024-04-15T18:34:40.907 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20046a3:::benchmark_data_smithi159_139342_object8578:head# 2024-04-15T18:34:40.907 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2014c96:::benchmark_data_smithi159_139342_object20214:head# 2024-04-15T18:34:40.908 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2016832:::benchmark_data_smithi159_139342_object40036:head# 2024-04-15T18:34:40.909 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c204c56a:::benchmark_data_smithi159_139342_object11662:head# 2024-04-15T18:34:40.909 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c206103b:::benchmark_data_smithi159_139342_object43140:head# 2024-04-15T18:34:40.910 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:40.922 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20626dd:::benchmark_data_smithi159_139342_object27353:head# 2024-04-15T18:34:40.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c206be03:::benchmark_data_smithi159_139342_object35567:head# 2024-04-15T18:34:40.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2080925:::benchmark_data_smithi159_139342_object24590:head# 2024-04-15T18:34:40.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20bc64e:::benchmark_data_smithi159_139342_object48836:head# 2024-04-15T18:34:40.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20c3410:::benchmark_data_smithi159_139342_object22179:head# 2024-04-15T18:34:40.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20d44fb:::benchmark_data_smithi159_139342_object36010:head# 2024-04-15T18:34:40.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20d4abd:::benchmark_data_smithi159_139342_object2479:head# 2024-04-15T18:34:40.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c20ee788:::benchmark_data_smithi159_139342_object34443:head# 2024-04-15T18:34:40.929 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c210f0b0:::benchmark_data_smithi159_139342_object33618:head# 2024-04-15T18:34:40.929 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2112d2f:::benchmark_data_smithi159_139342_object38901:head# 2024-04-15T18:34:40.930 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c214298b:::benchmark_data_smithi159_139342_object35595:head# 2024-04-15T18:34:40.930 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2153f6e:::benchmark_data_smithi159_139342_object23621:head# 2024-04-15T18:34:40.933 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c215dc9a:::benchmark_data_smithi159_139342_object4640:head# 2024-04-15T18:34:40.936 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21619c4:::benchmark_data_smithi159_139342_object46796:head# 2024-04-15T18:34:40.937 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2162a0b:::benchmark_data_smithi159_139342_object25567:head# 2024-04-15T18:34:40.938 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c216452b:::benchmark_data_smithi159_139342_object20343:head# 2024-04-15T18:34:40.938 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c217a082:::benchmark_data_smithi159_139342_object36435:head# 2024-04-15T18:34:40.939 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21985a2:::benchmark_data_smithi159_139342_object49586:head# 2024-04-15T18:34:40.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21998f8:::benchmark_data_smithi159_139342_object44060:head# 2024-04-15T18:34:40.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21cfc61:::benchmark_data_smithi159_139342_object906:head# 2024-04-15T18:34:40.941 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21dbc34:::benchmark_data_smithi159_139342_object21208:head# 2024-04-15T18:34:40.941 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c21eb8c1:::benchmark_data_smithi159_139342_object37550:head# 2024-04-15T18:34:40.941 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c221ce7b:::benchmark_data_smithi159_139342_object32142:head# 2024-04-15T18:34:40.942 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2227436:::benchmark_data_smithi159_139342_object2859:head# 2024-04-15T18:34:40.944 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c222dbec:::benchmark_data_smithi159_139342_object35784:head# 2024-04-15T18:34:40.944 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2234a84:::benchmark_data_smithi159_139342_object31576:head# 2024-04-15T18:34:40.944 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c223e1b9:::benchmark_data_smithi159_139342_object36189:head# 2024-04-15T18:34:40.945 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c223fffe:::benchmark_data_smithi159_139342_object12034:head# 2024-04-15T18:34:40.945 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2241112:::benchmark_data_smithi159_139342_object4839:head# 2024-04-15T18:34:40.945 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22478fd:::benchmark_data_smithi159_139342_object32726:head# 2024-04-15T18:34:40.946 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2253175:::benchmark_data_smithi159_139342_object46359:head# 2024-04-15T18:34:40.949 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22532b7:::benchmark_data_smithi159_139342_object52171:head# 2024-04-15T18:34:40.950 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:40.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c225c950:::benchmark_data_smithi159_139342_object39310:head# 2024-04-15T18:34:40.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22797fc:::benchmark_data_smithi159_139342_object44639:head# 2024-04-15T18:34:40.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c227dc04:::benchmark_data_smithi159_139342_object10000:head# 2024-04-15T18:34:40.951 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2287af2:::benchmark_data_smithi159_139342_object54096:head# 2024-04-15T18:34:40.951 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22a7ef6:::benchmark_data_smithi159_139342_object21431:head# 2024-04-15T18:34:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:40.955 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22cfc09:::benchmark_data_smithi159_139342_object9448:head# 2024-04-15T18:34:40.955 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22e1d77:::benchmark_data_smithi159_139342_object31658:head# 2024-04-15T18:34:40.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:40.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c22f9445:::benchmark_data_smithi159_139342_object7963:head# 2024-04-15T18:34:40.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2307c45:::benchmark_data_smithi159_139342_object15556:head# 2024-04-15T18:34:40.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c232b291:::benchmark_data_smithi159_139342_object50030:head# 2024-04-15T18:34:40.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c23537b6:::benchmark_data_smithi159_139342_object31266:head# 2024-04-15T18:34:40.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c235aa2e:::benchmark_data_smithi159_139342_object13240:head# 2024-04-15T18:34:40.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c237d6c4:::benchmark_data_smithi159_139342_object24272:head# 2024-04-15T18:34:40.965 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2390605:::benchmark_data_smithi159_139342_object48200:head# 2024-04-15T18:34:40.965 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c23bfdbe:::benchmark_data_smithi159_139342_object23581:head# 2024-04-15T18:34:40.965 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c23c4373:::benchmark_data_smithi159_139342_object31751:head# 2024-04-15T18:34:40.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2420bb4:::benchmark_data_smithi159_139342_object26570:head# 2024-04-15T18:34:40.969 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2442084:::benchmark_data_smithi159_139342_object19130:head# 2024-04-15T18:34:40.969 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c244b1cb:::benchmark_data_smithi159_139342_object40198:head# 2024-04-15T18:34:40.970 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c244e9fe:::benchmark_data_smithi159_139342_object12478:head# 2024-04-15T18:34:40.970 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2452646:::benchmark_data_smithi159_139342_object23437:head# 2024-04-15T18:34:40.970 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c245729c:::benchmark_data_smithi159_139342_object27377:head# 2024-04-15T18:34:40.971 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2458b2b:::benchmark_data_smithi159_139342_object38923:head# 2024-04-15T18:34:40.971 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c245c620:::benchmark_data_smithi159_139342_object41178:head# 2024-04-15T18:34:40.972 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c246220f:::benchmark_data_smithi159_139342_object49543:head# 2024-04-15T18:34:40.972 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24649f8:::benchmark_data_smithi159_139342_object48824:head# 2024-04-15T18:34:40.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24671fa:::benchmark_data_smithi159_139342_object46413:head# 2024-04-15T18:34:40.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c246a1ac:::benchmark_data_smithi159_139342_object52979:head# 2024-04-15T18:34:40.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24848b0:::benchmark_data_smithi159_139342_object49100:head# 2024-04-15T18:34:40.974 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24a5c55:::benchmark_data_smithi159_139342_object28047:head# 2024-04-15T18:34:40.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24add86:::benchmark_data_smithi159_139342_object46805:head# 2024-04-15T18:34:40.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c24bd708:::benchmark_data_smithi159_139342_object55375:head# 2024-04-15T18:34:40.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c250262e:::benchmark_data_smithi159_139342_object46925:head# 2024-04-15T18:34:40.978 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2502b59:::benchmark_data_smithi159_139342_object1071:head# 2024-04-15T18:34:40.978 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c250d3e0:::benchmark_data_smithi159_139342_object8089:head# 2024-04-15T18:34:40.981 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c250f564:::benchmark_data_smithi159_139342_object29651:head# 2024-04-15T18:34:40.981 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c252176b:::benchmark_data_smithi159_139342_object9373:head# 2024-04-15T18:34:40.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2527955:::benchmark_data_smithi159_139342_object35949:head# 2024-04-15T18:34:40.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2543c29:::benchmark_data_smithi159_139342_object5553:head# 2024-04-15T18:34:40.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2544c8c:::benchmark_data_smithi159_139342_object56123:head# 2024-04-15T18:34:40.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c255ba0d:::benchmark_data_smithi159_139342_object16372:head# 2024-04-15T18:34:40.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2578bfc:::benchmark_data_smithi159_139342_object52851:head# 2024-04-15T18:34:40.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2593a64:::benchmark_data_smithi159_139342_object30943:head# 2024-04-15T18:34:40.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c259a5c8:::benchmark_data_smithi159_139342_object13145:head# 2024-04-15T18:34:40.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c25b6cbc:::benchmark_data_smithi159_139342_object24263:head# 2024-04-15T18:34:40.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c25caf92:::benchmark_data_smithi159_139342_object9642:head# 2024-04-15T18:34:40.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c25cb334:::benchmark_data_smithi159_139342_object3598:head# 2024-04-15T18:34:40.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c25da488:::benchmark_data_smithi159_139342_object2745:head# 2024-04-15T18:34:40.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c25f144f:::benchmark_data_smithi159_139342_object42426:head# 2024-04-15T18:34:40.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c262af25:::benchmark_data_smithi159_139342_object41514:head# 2024-04-15T18:34:40.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c262de26:::benchmark_data_smithi159_139342_object27274:head# 2024-04-15T18:34:40.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c264ad14:::benchmark_data_smithi159_139342_object39594:head# 2024-04-15T18:34:40.989 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c264e62f:::benchmark_data_smithi159_139342_object54104:head# 2024-04-15T18:34:40.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c265ee38:::benchmark_data_smithi159_139342_object16789:head# 2024-04-15T18:34:40.993 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26709fd:::benchmark_data_smithi159_139342_object50956:head# 2024-04-15T18:34:40.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c267be44:::benchmark_data_smithi159_139342_object26268:head# 2024-04-15T18:34:40.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c268fa5b:::benchmark_data_smithi159_139342_object46047:head# 2024-04-15T18:34:40.996 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c269af15:::benchmark_data_smithi159_139342_object19516:head# 2024-04-15T18:34:40.996 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26a5cdd:::benchmark_data_smithi159_139342_object52029:head# 2024-04-15T18:34:40.996 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26bd19b:::benchmark_data_smithi159_139342_object42665:head# 2024-04-15T18:34:40.996 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26ce00b:::benchmark_data_smithi159_139342_object18979:head# 2024-04-15T18:34:41.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26d584f:::benchmark_data_smithi159_139342_object44103:head# 2024-04-15T18:34:41.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26dc157:::benchmark_data_smithi159_139342_object47264:head# 2024-04-15T18:34:41.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26dfa57:::benchmark_data_smithi159_139342_object53468:head# 2024-04-15T18:34:41.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26e4e28:::benchmark_data_smithi159_139342_object5102:head# 2024-04-15T18:34:41.002 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c26ee199:::benchmark_data_smithi159_139342_object53276:head# 2024-04-15T18:34:41.002 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2701cad:::benchmark_data_smithi159_139342_object56792:head# 2024-04-15T18:34:41.003 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c271b65c:::benchmark_data_smithi159_139342_object38139:head# 2024-04-15T18:34:41.003 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2730f20:::benchmark_data_smithi159_139342_object40580:head# 2024-04-15T18:34:41.006 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2731045:::benchmark_data_smithi159_139342_object43557:head# 2024-04-15T18:34:41.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c273816d:::benchmark_data_smithi159_139342_object4422:head# 2024-04-15T18:34:41.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c273b0dd:::benchmark_data_smithi159_139342_object51503:head# 2024-04-15T18:34:41.012 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27788d4:::benchmark_data_smithi159_139342_object39331:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c278aa79:::benchmark_data_smithi159_139342_object33312:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c278c9dd:::benchmark_data_smithi159_139342_object3990:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2791fa7:::benchmark_data_smithi159_139342_object14956:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27a89a9:::benchmark_data_smithi159_139342_object33666:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27b81c2:::benchmark_data_smithi159_139342_object31440:head# 2024-04-15T18:34:41.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27c5616:::benchmark_data_smithi159_139342_object21984:head# 2024-04-15T18:34:41.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27dd48f:::benchmark_data_smithi159_139342_object35050:head# 2024-04-15T18:34:41.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c27fb4f8:::benchmark_data_smithi159_139342_object36283:head# 2024-04-15T18:34:41.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2830e66:::benchmark_data_smithi159_139342_object43144:head# 2024-04-15T18:34:41.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c283cfed:::benchmark_data_smithi159_139342_object46955:head# 2024-04-15T18:34:41.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2846464:::benchmark_data_smithi159_139342_object11016:head# 2024-04-15T18:34:41.016 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c284b934:::benchmark_data_smithi159_139342_object26516:head# 2024-04-15T18:34:41.016 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28679e3:::benchmark_data_smithi159_139342_object12218:head# 2024-04-15T18:34:41.017 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2868ba2:::benchmark_data_smithi159_139342_object52549:head# 2024-04-15T18:34:41.018 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c287b72b:::benchmark_data_smithi159_139342_object45569:head# 2024-04-15T18:34:41.018 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28a64de:::benchmark_data_smithi159_139342_object1562:head# 2024-04-15T18:34:41.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28c55af:::benchmark_data_smithi159_139342_object9261:head# 2024-04-15T18:34:41.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28e5376:::benchmark_data_smithi159_139342_object27040:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28eb4ce:::benchmark_data_smithi159_139342_object18640:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28efe31:::benchmark_data_smithi159_139342_object49982:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c28f39f3:::benchmark_data_smithi159_139342_object29792:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29018ce:::benchmark_data_smithi159_139342_object41952:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2920ef0:::benchmark_data_smithi159_139342_object9270:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2924e32:::benchmark_data_smithi159_139342_object13517:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29310b4:::benchmark_data_smithi159_139342_object54408:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c293518c:::benchmark_data_smithi159_139342_object34158:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2938cef:::benchmark_data_smithi159_139342_object2724:head# 2024-04-15T18:34:41.028 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2949234:::benchmark_data_smithi159_139342_object49545:head# 2024-04-15T18:34:41.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c296e39b:::benchmark_data_smithi159_139342_object36758:head# 2024-04-15T18:34:41.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29778c2:::benchmark_data_smithi159_139342_object50223:head# 2024-04-15T18:34:41.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c297833e:::benchmark_data_smithi159_139342_object42723:head# 2024-04-15T18:34:41.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c298294e:::benchmark_data_smithi159_139342_object52667:head# 2024-04-15T18:34:41.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c299dd60:::benchmark_data_smithi159_139342_object51704:head# 2024-04-15T18:34:41.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c299f3a6:::benchmark_data_smithi159_139342_object6089:head# 2024-04-15T18:34:41.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29bde5b:::benchmark_data_smithi159_139342_object21004:head# 2024-04-15T18:34:41.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29c70c9:::benchmark_data_smithi159_139342_object929:head# 2024-04-15T18:34:41.037 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29e484e:::benchmark_data_smithi159_139342_object4802:head# 2024-04-15T18:34:41.038 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29eb799:::benchmark_data_smithi159_139342_object2530:head# 2024-04-15T18:34:41.038 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29f4364:::benchmark_data_smithi159_139342_object35011:head# 2024-04-15T18:34:41.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c29f6bcc:::benchmark_data_smithi159_139342_object40310:head# 2024-04-15T18:34:41.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a09caa:::benchmark_data_smithi159_139342_object18896:head# 2024-04-15T18:34:41.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a14eb8:::benchmark_data_smithi159_139342_object51924:head# 2024-04-15T18:34:41.048 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a2942a:::benchmark_data_smithi159_139342_object37456:head# 2024-04-15T18:34:41.049 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a62a39:::benchmark_data_smithi159_139342_object24964:head# 2024-04-15T18:34:41.049 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a6d37b:::benchmark_data_smithi159_139342_object43683:head# 2024-04-15T18:34:41.050 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a73e5a:::benchmark_data_smithi159_139342_object9947:head# 2024-04-15T18:34:41.050 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2a8d5ef:::benchmark_data_smithi159_139342_object33515:head# 2024-04-15T18:34:41.051 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2aa07f9:::benchmark_data_smithi159_139342_object42762:head# 2024-04-15T18:34:41.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2aa83f2:::benchmark_data_smithi159_139342_object44199:head# 2024-04-15T18:34:41.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ac82d6:::benchmark_data_smithi159_139342_object8752:head# 2024-04-15T18:34:41.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2acd680:::benchmark_data_smithi159_139342_object46276:head# 2024-04-15T18:34:41.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ad2428:::benchmark_data_smithi159_139342_object7513:head# 2024-04-15T18:34:41.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ad4b4b:::benchmark_data_smithi159_139342_object3851:head# 2024-04-15T18:34:41.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2aed96c:::benchmark_data_smithi159_139342_object48376:head# 2024-04-15T18:34:41.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2af1349:::benchmark_data_smithi159_139342_object5499:head# 2024-04-15T18:34:41.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:41.060 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2af9815:::benchmark_data_smithi159_139342_object26530:head# 2024-04-15T18:34:41.061 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b00b39:::benchmark_data_smithi159_139342_object53623:head# 2024-04-15T18:34:41.062 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b2ab79:::benchmark_data_smithi159_139342_object23677:head# 2024-04-15T18:34:41.062 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b426ac:::benchmark_data_smithi159_139342_object14876:head# 2024-04-15T18:34:41.063 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b6537a:::benchmark_data_smithi159_139342_object33982:head# 2024-04-15T18:34:41.063 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b78aea:::benchmark_data_smithi159_139342_object45357:head# 2024-04-15T18:34:41.064 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b85fcb:::benchmark_data_smithi159_139342_object43163:head# 2024-04-15T18:34:41.065 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2b904d2:::benchmark_data_smithi159_139342_object15812:head# 2024-04-15T18:34:41.065 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2bb1ed9:::benchmark_data_smithi159_139342_object483:head# 2024-04-15T18:34:41.069 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2bf1f71:::benchmark_data_smithi159_139342_object49940:head# 2024-04-15T18:34:41.069 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c02322:::benchmark_data_smithi159_139342_object56186:head# 2024-04-15T18:34:41.070 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c09b5d:::benchmark_data_smithi159_139342_object15356:head# 2024-04-15T18:34:41.078 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c09c02:::benchmark_data_smithi159_139342_object4190:head# 2024-04-15T18:34:41.078 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c0edaa:::benchmark_data_smithi159_139342_object9903:head# 2024-04-15T18:34:41.079 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c13fde:::benchmark_data_smithi159_139342_object24345:head# 2024-04-15T18:34:41.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c2d044:::benchmark_data_smithi159_139342_object4569:head# 2024-04-15T18:34:41.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c4c8b1:::benchmark_data_smithi159_139342_object52963:head# 2024-04-15T18:34:41.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c61963:::benchmark_data_smithi159_139342_object36169:head# 2024-04-15T18:34:41.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c62243:::benchmark_data_smithi159_139342_object11032:head# 2024-04-15T18:34:41.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c6ec8d:::benchmark_data_smithi159_139342_object12824:head# 2024-04-15T18:34:41.082 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c8074a:::benchmark_data_smithi159_139342_object46105:head# 2024-04-15T18:34:41.082 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c8bb2f:::benchmark_data_smithi159_139342_object47322:head# 2024-04-15T18:34:41.083 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c8c5c6:::benchmark_data_smithi159_139342_object26277:head# 2024-04-15T18:34:41.083 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c97717:::benchmark_data_smithi159_139342_object15645:head# 2024-04-15T18:34:41.084 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2c9c64a:::benchmark_data_smithi159_139342_object19487:head# 2024-04-15T18:34:41.111 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:41.123 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ca5b2a:::benchmark_data_smithi159_139342_object40576:head# 2024-04-15T18:34:41.128 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2cd2448:::benchmark_data_smithi159_139342_object27458:head# 2024-04-15T18:34:41.131 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2cdf016:::benchmark_data_smithi159_139342_object23117:head# 2024-04-15T18:34:41.132 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2cff720:::benchmark_data_smithi159_139342_object35193:head# 2024-04-15T18:34:41.132 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2d1dc7d:::benchmark_data_smithi159_139342_object29098:head# 2024-04-15T18:34:41.133 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2d1e097:::benchmark_data_smithi159_139342_object40103:head# 2024-04-15T18:34:41.133 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2d343d4:::benchmark_data_smithi159_139342_object36167:head# 2024-04-15T18:34:41.134 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2d7ebf2:::benchmark_data_smithi159_139342_object16183:head# 2024-04-15T18:34:41.134 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2da0018:::benchmark_data_smithi159_139342_object1202:head# 2024-04-15T18:34:41.134 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2db4448:::benchmark_data_smithi159_139342_object28099:head# 2024-04-15T18:34:41.135 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2deb681:::benchmark_data_smithi159_139342_object37603:head# 2024-04-15T18:34:41.135 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2df11db:::benchmark_data_smithi159_139342_object22676:head# 2024-04-15T18:34:41.139 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e075a8:::benchmark_data_smithi159_139342_object55216:head# 2024-04-15T18:34:41.142 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e1064d:::benchmark_data_smithi159_139342_object15140:head# 2024-04-15T18:34:41.142 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:41.143 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e2fc00:::benchmark_data_smithi159_139342_object40682:head# 2024-04-15T18:34:41.143 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e3b65b:::benchmark_data_smithi159_139342_object50578:head# 2024-04-15T18:34:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:41.145 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e3fa9a:::benchmark_data_smithi159_139342_object6398:head# 2024-04-15T18:34:41.145 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e4bcb6:::benchmark_data_smithi159_139342_object3751:head# 2024-04-15T18:34:41.145 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e54633:::benchmark_data_smithi159_139342_object29162:head# 2024-04-15T18:34:41.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:41.149 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e67652:::benchmark_data_smithi159_139342_object40508:head# 2024-04-15T18:34:41.149 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e6f077:::benchmark_data_smithi159_139342_object10215:head# 2024-04-15T18:34:41.150 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e72a2a:::benchmark_data_smithi159_139342_object30285:head# 2024-04-15T18:34:41.150 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e8e425:::benchmark_data_smithi159_139342_object44745:head# 2024-04-15T18:34:41.153 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2e8ea44:::benchmark_data_smithi159_139342_object50770:head# 2024-04-15T18:34:41.160 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ec3b8c:::benchmark_data_smithi159_139342_object14993:head# 2024-04-15T18:34:41.160 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ed3a0e:::benchmark_data_smithi159_139342_object12653:head# 2024-04-15T18:34:41.160 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ed7eb1:::benchmark_data_smithi159_139342_object41155:head# 2024-04-15T18:34:41.160 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2edcdbf:::benchmark_data_smithi159_139342_object13818:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2efe13f:::benchmark_data_smithi159_139342_object24658:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f07289:::benchmark_data_smithi159_139342_object41997:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f11a4c:::benchmark_data_smithi159_139342_object15383:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f2be76:::benchmark_data_smithi159_139342_object36187:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f39adb:::benchmark_data_smithi159_139342_object44808:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f3ad66:::benchmark_data_smithi159_139342_object36322:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f3c99b:::benchmark_data_smithi159_139342_object22185:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f4826d:::benchmark_data_smithi159_139342_object51157:head# 2024-04-15T18:34:41.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f72c3e:::benchmark_data_smithi159_139342_object26879:head# 2024-04-15T18:34:41.162 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2f77032:::benchmark_data_smithi159_139342_object49454:head# 2024-04-15T18:34:41.162 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2fbf04e:::benchmark_data_smithi159_139342_object30782:head# 2024-04-15T18:34:41.162 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2fdaf85:::benchmark_data_smithi159_139342_object53307:head# 2024-04-15T18:34:41.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ff2213:::benchmark_data_smithi159_139342_object21092:head# 2024-04-15T18:34:41.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c2ff8068:::benchmark_data_smithi159_139342_object38554:head# 2024-04-15T18:34:41.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3000ba1:::benchmark_data_smithi159_139342_object44586:head# 2024-04-15T18:34:41.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c300d6df:::benchmark_data_smithi159_139342_object21793:head# 2024-04-15T18:34:41.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c303f4b2:::benchmark_data_smithi159_139342_object55083:head# 2024-04-15T18:34:41.165 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30435b9:::benchmark_data_smithi159_139342_object26512:head# 2024-04-15T18:34:41.165 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3056f85:::benchmark_data_smithi159_139342_object37663:head# 2024-04-15T18:34:41.166 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c306ca04:::benchmark_data_smithi159_139342_object21663:head# 2024-04-15T18:34:41.166 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c307d7f8:::benchmark_data_smithi159_139342_object8008:head# 2024-04-15T18:34:41.167 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30ab2cf:::benchmark_data_smithi159_139342_object33379:head# 2024-04-15T18:34:41.167 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30ae57e:::benchmark_data_smithi159_139342_object47745:head# 2024-04-15T18:34:41.167 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30b22b8:::benchmark_data_smithi159_139342_object48743:head# 2024-04-15T18:34:41.168 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30c3a79:::benchmark_data_smithi159_139342_object56416:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30c6bd6:::benchmark_data_smithi159_139342_object32801:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30d2120:::benchmark_data_smithi159_139342_object29837:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30d3202:::benchmark_data_smithi159_139342_object23170:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30e0dd6:::benchmark_data_smithi159_139342_object20434:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30e1449:::benchmark_data_smithi159_139342_object23955:head# 2024-04-15T18:34:41.172 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c30f2bbe:::benchmark_data_smithi159_139342_object49444:head# 2024-04-15T18:34:41.176 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31286b6:::benchmark_data_smithi159_139342_object22172:head# 2024-04-15T18:34:41.177 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3143f7d:::benchmark_data_smithi159_139342_object42087:head# 2024-04-15T18:34:41.177 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c314bdff:::benchmark_data_smithi159_139342_object29626:head# 2024-04-15T18:34:41.181 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3154d77:::benchmark_data_smithi159_139342_object13990:head# 2024-04-15T18:34:41.181 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c315ca58:::benchmark_data_smithi159_139342_object12991:head# 2024-04-15T18:34:41.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3188011:::benchmark_data_smithi159_139342_object1426:head# 2024-04-15T18:34:41.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c318e558:::benchmark_data_smithi159_139342_object8168:head# 2024-04-15T18:34:41.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c318e5f0:::benchmark_data_smithi159_139342_object43404:head# 2024-04-15T18:34:41.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3190fa0:::benchmark_data_smithi159_139342_object6952:head# 2024-04-15T18:34:41.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31bfa60:::benchmark_data_smithi159_139342_object4102:head# 2024-04-15T18:34:41.184 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31caf8f:::benchmark_data_smithi159_139342_object5016:head# 2024-04-15T18:34:41.184 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31ef486:::benchmark_data_smithi159_139342_object26425:head# 2024-04-15T18:34:41.184 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31f35b0:::benchmark_data_smithi159_139342_object56019:head# 2024-04-15T18:34:41.184 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c31fc9e1:::benchmark_data_smithi159_139342_object53004:head# 2024-04-15T18:34:41.184 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3202b28:::benchmark_data_smithi159_139342_object12783:head# 2024-04-15T18:34:41.185 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32253ce:::benchmark_data_smithi159_139342_object18635:head# 2024-04-15T18:34:41.185 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32267dc:::benchmark_data_smithi159_139342_object37499:head# 2024-04-15T18:34:41.187 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c322dc6e:::benchmark_data_smithi159_139342_object38769:head# 2024-04-15T18:34:41.188 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c322e326:::benchmark_data_smithi159_139342_object33375:head# 2024-04-15T18:34:41.188 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c323b5d6:::benchmark_data_smithi159_139342_object24270:head# 2024-04-15T18:34:41.188 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32488a6:::benchmark_data_smithi159_139342_object32957:head# 2024-04-15T18:34:41.189 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c325be84:::benchmark_data_smithi159_139342_object24038:head# 2024-04-15T18:34:41.189 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c326847f:::benchmark_data_smithi159_139342_object23920:head# 2024-04-15T18:34:41.190 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c327038f:::benchmark_data_smithi159_139342_object29317:head# 2024-04-15T18:34:41.190 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3273f06:::benchmark_data_smithi159_139342_object28865:head# 2024-04-15T18:34:41.191 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32834c1:::benchmark_data_smithi159_139342_object9125:head# 2024-04-15T18:34:41.192 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3295b1b:::benchmark_data_smithi159_139342_object11102:head# 2024-04-15T18:34:41.192 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32bb890:::benchmark_data_smithi159_139342_object50215:head# 2024-04-15T18:34:41.192 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32c2407:::benchmark_data_smithi159_139342_object16959:head# 2024-04-15T18:34:41.192 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32d859a:::benchmark_data_smithi159_139342_object28936:head# 2024-04-15T18:34:41.193 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32d887b:::benchmark_data_smithi159_139342_object9911:head# 2024-04-15T18:34:41.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32e87a7:::benchmark_data_smithi159_139342_object23733:head# 2024-04-15T18:34:41.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32ed5e3:::benchmark_data_smithi159_139342_object12616:head# 2024-04-15T18:34:41.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32f2bd8:::benchmark_data_smithi159_139342_object21168:head# 2024-04-15T18:34:41.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32fcdc3:::benchmark_data_smithi159_139342_object20183:head# 2024-04-15T18:34:41.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32fd787:::benchmark_data_smithi159_139342_object2268:head# 2024-04-15T18:34:41.199 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c32ff6ad:::benchmark_data_smithi159_139342_object45323:head# 2024-04-15T18:34:41.199 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3300a1d:::benchmark_data_smithi159_139342_object47954:head# 2024-04-15T18:34:41.199 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3319122:::benchmark_data_smithi159_139342_object32103:head# 2024-04-15T18:34:41.199 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c331c8f2:::benchmark_data_smithi159_139342_object42386:head# 2024-04-15T18:34:41.199 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c331faac:::benchmark_data_smithi159_139342_object25742:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c332a2db:::benchmark_data_smithi159_139342_object33947:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c337ccea:::benchmark_data_smithi159_139342_object28535:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3394e12:::benchmark_data_smithi159_139342_object47413:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c339abd8:::benchmark_data_smithi159_139342_object48383:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c339fc5d:::benchmark_data_smithi159_139342_object48475:head# 2024-04-15T18:34:41.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33af8fa:::benchmark_data_smithi159_139342_object23629:head# 2024-04-15T18:34:41.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:41 smithi082 ceph-mon[44206]: pgmap v1518: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 16768/210462 objects degraded (7.967%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:41.209 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33b1758:::benchmark_data_smithi159_139342_object1479:head# 2024-04-15T18:34:41.209 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33b379b:::benchmark_data_smithi159_139342_object3625:head# 2024-04-15T18:34:41.209 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33bbd69:::benchmark_data_smithi159_139342_object17667:head# 2024-04-15T18:34:41.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33cdafe:::benchmark_data_smithi159_139342_object44892:head# 2024-04-15T18:34:41.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33d9440:::benchmark_data_smithi159_139342_object46576:head# 2024-04-15T18:34:41.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c33f2790:::benchmark_data_smithi159_139342_object56411:head# 2024-04-15T18:34:41.212 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34066c5:::benchmark_data_smithi159_139342_object1270:head# 2024-04-15T18:34:41.212 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3428489:::benchmark_data_smithi159_139342_object724:head# 2024-04-15T18:34:41.213 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c342927f:::benchmark_data_smithi159_139342_object19238:head# 2024-04-15T18:34:41.213 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c342a15e:::benchmark_data_smithi159_139342_object30662:head# 2024-04-15T18:34:41.213 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3436856:::benchmark_data_smithi159_139342_object36060:head# 2024-04-15T18:34:41.214 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c343d1d3:::benchmark_data_smithi159_139342_object47614:head# 2024-04-15T18:34:41.214 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c344a8be:::benchmark_data_smithi159_139342_object27371:head# 2024-04-15T18:34:41.215 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c346a9c4:::benchmark_data_smithi159_139342_object27029:head# 2024-04-15T18:34:41.215 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c346c5f0:::benchmark_data_smithi159_139342_object38688:head# 2024-04-15T18:34:41.216 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34709ed:::benchmark_data_smithi159_139342_object39067:head# 2024-04-15T18:34:41.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34a049c:::benchmark_data_smithi159_139342_object55512:head# 2024-04-15T18:34:41.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34a07bf:::benchmark_data_smithi159_139342_object13608:head# 2024-04-15T18:34:41.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34a14f6:::benchmark_data_smithi159_139342_object15620:head# 2024-04-15T18:34:41.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c34b08b4:::benchmark_data_smithi159_139342_object40114:head# 2024-04-15T18:34:41.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c350368b:::benchmark_data_smithi159_139342_object32945:head# 2024-04-15T18:34:41.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c352af83:::benchmark_data_smithi159_139342_object55934:head# 2024-04-15T18:34:41.220 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3532bb3:::benchmark_data_smithi159_139342_object3537:head# 2024-04-15T18:34:41.220 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3554d07:::benchmark_data_smithi159_139342_object17950:head# 2024-04-15T18:34:41.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c358a83c:::benchmark_data_smithi159_139342_object36889:head# 2024-04-15T18:34:41.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c35aedc1:::benchmark_data_smithi159_139342_object21305:head# 2024-04-15T18:34:41.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c35c73f0:::benchmark_data_smithi159_139342_object38857:head# 2024-04-15T18:34:41.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c35cb18e:::benchmark_data_smithi159_139342_object47453:head# 2024-04-15T18:34:41.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c35d8302:::benchmark_data_smithi159_139342_object23700:head# 2024-04-15T18:34:41.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c35fc98e:::benchmark_data_smithi159_139342_object6253:head# 2024-04-15T18:34:41.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3634f33:::benchmark_data_smithi159_139342_object3487:head# 2024-04-15T18:34:41.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c363b800:::benchmark_data_smithi159_139342_object49297:head# 2024-04-15T18:34:41.227 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3643526:::benchmark_data_smithi159_139342_object46073:head# 2024-04-15T18:34:41.227 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c364db4e:::benchmark_data_smithi159_139342_object6888:head# 2024-04-15T18:34:41.228 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3669615:::benchmark_data_smithi159_139342_object40047:head# 2024-04-15T18:34:41.228 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3678a86:::benchmark_data_smithi159_139342_object50779:head# 2024-04-15T18:34:41.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c367cf65:::benchmark_data_smithi159_139342_object22244:head# 2024-04-15T18:34:41.231 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c367e128:::benchmark_data_smithi159_139342_object38851:head# 2024-04-15T18:34:41.233 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c36c1264:::benchmark_data_smithi159_139342_object46941:head# 2024-04-15T18:34:41.233 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c36d3b64:::benchmark_data_smithi159_139342_object21549:head# 2024-04-15T18:34:41.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c36e508f:::benchmark_data_smithi159_139342_object28915:head# 2024-04-15T18:34:41.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3703c4f:::benchmark_data_smithi159_139342_object29689:head# 2024-04-15T18:34:41.240 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c370ed8b:::benchmark_data_smithi159_139342_object52291:head# 2024-04-15T18:34:41.241 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37126c8:::benchmark_data_smithi159_139342_object5698:head# 2024-04-15T18:34:41.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:41.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3716a94:::benchmark_data_smithi159_139342_object35518:head# 2024-04-15T18:34:41.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c371bef3:::benchmark_data_smithi159_139342_object52758:head# 2024-04-15T18:34:41.248 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37240f7:::benchmark_data_smithi159_139342_object13:head# 2024-04-15T18:34:41.248 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3727ce2:::benchmark_data_smithi159_139342_object20640:head# 2024-04-15T18:34:41.249 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3728042:::benchmark_data_smithi159_139342_object13786:head# 2024-04-15T18:34:41.249 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c374bcbe:::benchmark_data_smithi159_139342_object53313:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c374cdbd:::benchmark_data_smithi159_139342_object12798:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3771992:::benchmark_data_smithi159_139342_object56513:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3783237:::benchmark_data_smithi159_139342_object31961:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37b1d76:::benchmark_data_smithi159_139342_object22785:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37d565c:::benchmark_data_smithi159_139342_object25202:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37dcff8:::benchmark_data_smithi159_139342_object9052:head# 2024-04-15T18:34:41.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37f3fe6:::benchmark_data_smithi159_139342_object8403:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c37fe111:::benchmark_data_smithi159_139342_object43735:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3804548:::benchmark_data_smithi159_139342_object54646:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c380ed5a:::benchmark_data_smithi159_139342_object19671:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3818506:::benchmark_data_smithi159_139342_object48607:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38308ae:::benchmark_data_smithi159_139342_object43460:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c384e738:::benchmark_data_smithi159_139342_object26307:head# 2024-04-15T18:34:41.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3865a9b:::benchmark_data_smithi159_139342_object4385:head# 2024-04-15T18:34:41.257 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3880cbe:::benchmark_data_smithi159_139342_object41566:head# 2024-04-15T18:34:41.261 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3883a8b:::benchmark_data_smithi159_139342_object42312:head# 2024-04-15T18:34:41.261 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c388a1d9:::benchmark_data_smithi159_139342_object6637:head# 2024-04-15T18:34:41.261 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38aeb00:::benchmark_data_smithi159_139342_object2237:head# 2024-04-15T18:34:41.261 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38b5b74:::benchmark_data_smithi159_139342_object20108:head# 2024-04-15T18:34:41.262 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38cc117:::benchmark_data_smithi159_139342_object40052:head# 2024-04-15T18:34:41.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38dad8c:::benchmark_data_smithi159_139342_object21411:head# 2024-04-15T18:34:41.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38e43b8:::benchmark_data_smithi159_139342_object55652:head# 2024-04-15T18:34:41.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c38ffb5e:::benchmark_data_smithi159_139342_object9449:head# 2024-04-15T18:34:41.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c390da1c:::benchmark_data_smithi159_139342_object24003:head# 2024-04-15T18:34:41.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3910d88:::benchmark_data_smithi159_139342_object25456:head# 2024-04-15T18:34:41.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3915f1c:::benchmark_data_smithi159_139342_object23162:head# 2024-04-15T18:34:41.267 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3928da9:::benchmark_data_smithi159_139342_object45790:head# 2024-04-15T18:34:41.268 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39333df:::benchmark_data_smithi159_139342_object1327:head# 2024-04-15T18:34:41.268 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c394797f:::benchmark_data_smithi159_139342_object53472:head# 2024-04-15T18:34:41.269 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c394e86d:::benchmark_data_smithi159_139342_object39402:head# 2024-04-15T18:34:41.269 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c395b0d8:::benchmark_data_smithi159_139342_object9942:head# 2024-04-15T18:34:41.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3989d51:::benchmark_data_smithi159_139342_object14923:head# 2024-04-15T18:34:41.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3996c2b:::benchmark_data_smithi159_139342_object40998:head# 2024-04-15T18:34:41.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3999bbb:::benchmark_data_smithi159_139342_object52:head# 2024-04-15T18:34:41.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39b62bc:::benchmark_data_smithi159_139342_object2735:head# 2024-04-15T18:34:41.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39d66fa:::benchmark_data_smithi159_139342_object11738:head# 2024-04-15T18:34:41.279 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39e5cf4:::benchmark_data_smithi159_139342_object34673:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39f76a4:::benchmark_data_smithi159_139342_object36194:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39fd3db:::benchmark_data_smithi159_139342_object655:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c39fe099:::benchmark_data_smithi159_139342_object49676:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a05462:::benchmark_data_smithi159_139342_object1895:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a5129b:::benchmark_data_smithi159_139342_object47339:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a6072c:::benchmark_data_smithi159_139342_object11951:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a6d169:::benchmark_data_smithi159_139342_object12284:head# 2024-04-15T18:34:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a704e7:::benchmark_data_smithi159_139342_object14758:head# 2024-04-15T18:34:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a76312:::benchmark_data_smithi159_139342_object41416:head# 2024-04-15T18:34:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3a9f73d:::benchmark_data_smithi159_139342_object2076:head# 2024-04-15T18:34:41.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3abd750:::benchmark_data_smithi159_139342_object10375:head# 2024-04-15T18:34:41.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ac6010:::benchmark_data_smithi159_139342_object42984:head# 2024-04-15T18:34:41.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3af7656:::benchmark_data_smithi159_139342_object19662:head# 2024-04-15T18:34:41.289 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:41.289 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3afd87b:::benchmark_data_smithi159_139342_object47724:head# 2024-04-15T18:34:41.290 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b286f3:::benchmark_data_smithi159_139342_object43672:head# 2024-04-15T18:34:41.290 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b355ac:::benchmark_data_smithi159_139342_object4581:head# 2024-04-15T18:34:41.290 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b4c417:::benchmark_data_smithi159_139342_object49101:head# 2024-04-15T18:34:41.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b59428:::benchmark_data_smithi159_139342_object46026:head# 2024-04-15T18:34:41.294 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b5e65c:::benchmark_data_smithi159_139342_object17194:head# 2024-04-15T18:34:41.294 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3b60655:::benchmark_data_smithi159_139342_object14018:head# 2024-04-15T18:34:41.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ba998a:::benchmark_data_smithi159_139342_object48364:head# 2024-04-15T18:34:41.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3bb9567:::benchmark_data_smithi159_139342_object16215:head# 2024-04-15T18:34:41.296 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3bbb001:::benchmark_data_smithi159_139342_object45641:head# 2024-04-15T18:34:41.296 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3bc774c:::benchmark_data_smithi159_139342_object55147:head# 2024-04-15T18:34:41.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3bc8722:::benchmark_data_smithi159_139342_object32397:head# 2024-04-15T18:34:41.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3be2bd8:::benchmark_data_smithi159_139342_object5818:head# 2024-04-15T18:34:41.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3bf3592:::benchmark_data_smithi159_139342_object262:head# 2024-04-15T18:34:41.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c0288d:::benchmark_data_smithi159_139342_object21135:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c0acd6:::benchmark_data_smithi159_139342_object13594:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c1503a:::benchmark_data_smithi159_139342_object8480:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c45562:::benchmark_data_smithi159_139342_object47083:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c4e457:::benchmark_data_smithi159_139342_object40303:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c4feb8:::benchmark_data_smithi159_139342_object245:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c58ea7:::benchmark_data_smithi159_139342_object27403:head# 2024-04-15T18:34:41.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3c5c65f:::benchmark_data_smithi159_139342_object9035:head# 2024-04-15T18:34:41.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3cb1b3f:::benchmark_data_smithi159_139342_object37452:head# 2024-04-15T18:34:41.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3cb7e0e:::benchmark_data_smithi159_139342_object49443:head# 2024-04-15T18:34:41.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3cc4bb9:::benchmark_data_smithi159_139342_object9777:head# 2024-04-15T18:34:41.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3cd58e8:::benchmark_data_smithi159_139342_object4346:head# 2024-04-15T18:34:41.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3cfa824:::benchmark_data_smithi159_139342_object5115:head# 2024-04-15T18:34:41.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d0bebf:::benchmark_data_smithi159_139342_object34749:head# 2024-04-15T18:34:41.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d20b1b:::benchmark_data_smithi159_139342_object3326:head# 2024-04-15T18:34:41.306 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d20e56:::benchmark_data_smithi159_139342_object3201:head# 2024-04-15T18:34:41.307 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d25540:::benchmark_data_smithi159_139342_object29797:head# 2024-04-15T18:34:41.307 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d302f8:::benchmark_data_smithi159_139342_object12496:head# 2024-04-15T18:34:41.307 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d36634:::benchmark_data_smithi159_139342_object46144:head# 2024-04-15T18:34:41.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d42119:::benchmark_data_smithi159_139342_object4919:head# 2024-04-15T18:34:41.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d4c46a:::benchmark_data_smithi159_139342_object37780:head# 2024-04-15T18:34:41.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d6e23d:::benchmark_data_smithi159_139342_object8467:head# 2024-04-15T18:34:41.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d75892:::benchmark_data_smithi159_139342_object40748:head# 2024-04-15T18:34:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3d77c1f:::benchmark_data_smithi159_139342_object54900:head# 2024-04-15T18:34:41.310 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3da8e76:::benchmark_data_smithi159_139342_object34569:head# 2024-04-15T18:34:41.313 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:41.315 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3db7460:::benchmark_data_smithi159_139342_object27176:head# 2024-04-15T18:34:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:41.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:41.325 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3dc4f4d:::benchmark_data_smithi159_139342_object8590:head# 2024-04-15T18:34:41.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3df476e:::benchmark_data_smithi159_139342_object53353:head# 2024-04-15T18:34:41.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3dfa743:::benchmark_data_smithi159_139342_object41071:head# 2024-04-15T18:34:41.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e0e64b:::benchmark_data_smithi159_139342_object15537:head# 2024-04-15T18:34:41.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e37bf0:::benchmark_data_smithi159_139342_object46511:head# 2024-04-15T18:34:41.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e423b5:::benchmark_data_smithi159_139342_object21419:head# 2024-04-15T18:34:41.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e4f025:::benchmark_data_smithi159_139342_object7493:head# 2024-04-15T18:34:41.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e72050:::benchmark_data_smithi159_139342_object716:head# 2024-04-15T18:34:41.329 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e77b95:::benchmark_data_smithi159_139342_object42335:head# 2024-04-15T18:34:41.330 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e7e717:::benchmark_data_smithi159_139342_object38324:head# 2024-04-15T18:34:41.330 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3e880b5:::benchmark_data_smithi159_139342_object7144:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ea858c:::benchmark_data_smithi159_139342_object37353:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ebaae7:::benchmark_data_smithi159_139342_object25699:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ebf40e:::benchmark_data_smithi159_139342_object55810:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ed7976:::benchmark_data_smithi159_139342_object38701:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f0e1e1:::benchmark_data_smithi159_139342_object27763:head# 2024-04-15T18:34:41.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f273f7:::benchmark_data_smithi159_139342_object39567:head# 2024-04-15T18:34:41.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f2ff94:::benchmark_data_smithi159_139342_object5552:head# 2024-04-15T18:34:41.337 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f3ec59:::benchmark_data_smithi159_139342_object42187:head# 2024-04-15T18:34:41.337 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f713c8:::benchmark_data_smithi159_139342_object27042:head# 2024-04-15T18:34:41.337 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f7c6e7:::benchmark_data_smithi159_139342_object8994:head# 2024-04-15T18:34:41.338 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3f984d5:::benchmark_data_smithi159_139342_object31411:head# 2024-04-15T18:34:41.338 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3fa6343:::benchmark_data_smithi159_139342_object2816:head# 2024-04-15T18:34:41.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3fb6ce7:::benchmark_data_smithi159_139342_object17765:head# 2024-04-15T18:34:41.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3fbee9b:::benchmark_data_smithi159_139342_object15700:head# 2024-04-15T18:34:41.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ff1642:::benchmark_data_smithi159_139342_object13368:head# 2024-04-15T18:34:41.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c3ff3e6c:::benchmark_data_smithi159_139342_object51217:head# 2024-04-15T18:34:41.341 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c400da8f:::benchmark_data_smithi159_139342_object40157:head# 2024-04-15T18:34:41.341 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4016a5b:::benchmark_data_smithi159_139342_object15311:head# 2024-04-15T18:34:41.342 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c402939d:::benchmark_data_smithi159_139342_object36863:head# 2024-04-15T18:34:41.342 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4033b9a:::benchmark_data_smithi159_139342_object18558:head# 2024-04-15T18:34:41.343 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4041b8f:::benchmark_data_smithi159_139342_object46837:head# 2024-04-15T18:34:41.343 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c404c672:::benchmark_data_smithi159_139342_object9869:head# 2024-04-15T18:34:41.344 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4057dfd:::benchmark_data_smithi159_139342_object31799:head# 2024-04-15T18:34:41.344 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4065da0:::benchmark_data_smithi159_139342_object39628:head# 2024-04-15T18:34:41.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c406689f:::benchmark_data_smithi159_139342_object11936:head# 2024-04-15T18:34:41.348 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40692fc:::benchmark_data_smithi159_139342_object33762:head# 2024-04-15T18:34:41.348 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4069cc7:::benchmark_data_smithi159_139342_object39314:head# 2024-04-15T18:34:41.348 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c409c1aa:::benchmark_data_smithi159_139342_object35856:head# 2024-04-15T18:34:41.348 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40a06f8:::benchmark_data_smithi159_139342_object5312:head# 2024-04-15T18:34:41.349 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40ab640:::benchmark_data_smithi159_139342_object47934:head# 2024-04-15T18:34:41.349 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40c25da:::benchmark_data_smithi159_139342_object17304:head# 2024-04-15T18:34:41.349 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40e378d:::benchmark_data_smithi159_139342_object53847:head# 2024-04-15T18:34:41.350 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40e3b1a:::benchmark_data_smithi159_139342_object29022:head# 2024-04-15T18:34:41.354 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c40ef657:::benchmark_data_smithi159_139342_object239:head# 2024-04-15T18:34:41.354 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4101d5b:::benchmark_data_smithi159_139342_object21341:head# 2024-04-15T18:34:41.354 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c410cf4f:::benchmark_data_smithi159_139342_object48247:head# 2024-04-15T18:34:41.354 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c411c883:::benchmark_data_smithi159_139342_object7688:head# 2024-04-15T18:34:41.354 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c412a451:::benchmark_data_smithi159_139342_object36753:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c415269f:::benchmark_data_smithi159_139342_object19048:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c415801d:::benchmark_data_smithi159_139342_object53200:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c415f7e8:::benchmark_data_smithi159_139342_object42839:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4160a5d:::benchmark_data_smithi159_139342_object12157:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4171093:::benchmark_data_smithi159_139342_object37315:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c417a430:::benchmark_data_smithi159_139342_object57249:head# 2024-04-15T18:34:41.360 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4184d92:::benchmark_data_smithi159_139342_object49778:head# 2024-04-15T18:34:41.364 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4193274:::benchmark_data_smithi159_139342_object19136:head# 2024-04-15T18:34:41.364 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c419ead1:::benchmark_data_smithi159_139342_object28979:head# 2024-04-15T18:34:41.365 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41a9d1e:::benchmark_data_smithi159_139342_object13766:head# 2024-04-15T18:34:41.365 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41abc85:::benchmark_data_smithi159_139342_object36200:head# 2024-04-15T18:34:41.365 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41abdfc:::benchmark_data_smithi159_139342_object49031:head# 2024-04-15T18:34:41.365 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41b2891:::benchmark_data_smithi159_139342_object18001:head# 2024-04-15T18:34:41.367 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41f0cea:::benchmark_data_smithi159_139342_object38207:head# 2024-04-15T18:34:41.367 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c41f3421:::benchmark_data_smithi159_139342_object47246:head# 2024-04-15T18:34:41.367 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4204eef:::benchmark_data_smithi159_139342_object26804:head# 2024-04-15T18:34:41.368 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c420b4a7:::benchmark_data_smithi159_139342_object25705:head# 2024-04-15T18:34:41.368 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4217ae8:::benchmark_data_smithi159_139342_object21270:head# 2024-04-15T18:34:41.371 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42350eb:::benchmark_data_smithi159_139342_object28962:head# 2024-04-15T18:34:41.371 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c423ad9f:::benchmark_data_smithi159_139342_object49298:head# 2024-04-15T18:34:41.371 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c423c0dc:::benchmark_data_smithi159_139342_object32452:head# 2024-04-15T18:34:41.372 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4241350:::benchmark_data_smithi159_139342_object45333:head# 2024-04-15T18:34:41.373 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42421cc:::benchmark_data_smithi159_139342_object43180:head# 2024-04-15T18:34:41.377 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42510ad:::benchmark_data_smithi159_139342_object40443:head# 2024-04-15T18:34:41.377 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c426be84:::benchmark_data_smithi159_139342_object39511:head# 2024-04-15T18:34:41.377 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c427bebd:::benchmark_data_smithi159_139342_object38810:head# 2024-04-15T18:34:41.378 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c428b9d4:::benchmark_data_smithi159_139342_object57184:head# 2024-04-15T18:34:41.378 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42aa98f:::benchmark_data_smithi159_139342_object43129:head# 2024-04-15T18:34:41.378 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42b5aea:::benchmark_data_smithi159_139342_object46300:head# 2024-04-15T18:34:41.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42d5c31:::benchmark_data_smithi159_139342_object14633:head# 2024-04-15T18:34:41.384 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c42f6586:::benchmark_data_smithi159_139342_object18662:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c43163f1:::benchmark_data_smithi159_139342_object32382:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4318d73:::benchmark_data_smithi159_139342_object8209:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4326e95:::benchmark_data_smithi159_139342_object27655:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4335af2:::benchmark_data_smithi159_139342_object12351:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4342e60:::benchmark_data_smithi159_139342_object34657:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4346e66:::benchmark_data_smithi159_139342_object44020:head# 2024-04-15T18:34:41.388 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c434d67b:::benchmark_data_smithi159_139342_object55746:head# 2024-04-15T18:34:41.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4355f70:::benchmark_data_smithi159_139342_object36207:head# 2024-04-15T18:34:41.394 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4368118:::benchmark_data_smithi159_139342_object39026:head# 2024-04-15T18:34:41.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c43730a9:::benchmark_data_smithi159_139342_object22638:head# 2024-04-15T18:34:41.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c437496f:::benchmark_data_smithi159_139342_object12361:head# 2024-04-15T18:34:41.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:41 smithi159 ceph-mon[45336]: pgmap v1518: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 16768/210462 objects degraded (7.967%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:41.399 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4388083:::benchmark_data_smithi159_139342_object35597:head# 2024-04-15T18:34:41.399 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c438b0f9:::benchmark_data_smithi159_139342_object21759:head# 2024-04-15T18:34:41.399 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c439539d:::benchmark_data_smithi159_139342_object46005:head# 2024-04-15T18:34:41.401 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c43b59a3:::benchmark_data_smithi159_139342_object10571:head# 2024-04-15T18:34:41.402 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c43bfc9c:::benchmark_data_smithi159_139342_object42493:head# 2024-04-15T18:34:41.402 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c442229d:::benchmark_data_smithi159_139342_object46655:head# 2024-04-15T18:34:41.402 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4426078:::benchmark_data_smithi159_139342_object18773:head# 2024-04-15T18:34:41.403 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44276f9:::benchmark_data_smithi159_139342_object42928:head# 2024-04-15T18:34:41.403 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44288be:::benchmark_data_smithi159_139342_object14406:head# 2024-04-15T18:34:41.404 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c443245a:::benchmark_data_smithi159_139342_object29499:head# 2024-04-15T18:34:41.404 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c443fa81:::benchmark_data_smithi159_139342_object47460:head# 2024-04-15T18:34:41.405 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4450fe6:::benchmark_data_smithi159_139342_object41267:head# 2024-04-15T18:34:41.405 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c446b834:::benchmark_data_smithi159_139342_object29554:head# 2024-04-15T18:34:41.406 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44825a4:::benchmark_data_smithi159_139342_object48965:head# 2024-04-15T18:34:41.406 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4490970:::benchmark_data_smithi159_139342_object6313:head# 2024-04-15T18:34:41.407 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44b43b1:::benchmark_data_smithi159_139342_object21109:head# 2024-04-15T18:34:41.407 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44bba94:::benchmark_data_smithi159_139342_object33352:head# 2024-04-15T18:34:41.408 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44c0dbc:::benchmark_data_smithi159_139342_object11620:head# 2024-04-15T18:34:41.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:41 smithi049 ceph-mon[40087]: pgmap v1518: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 16768/210462 objects degraded (7.967%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:41.415 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44c7162:::benchmark_data_smithi159_139342_object27280:head# 2024-04-15T18:34:41.415 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44d06b8:::benchmark_data_smithi159_139342_object44907:head# 2024-04-15T18:34:41.415 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c44fe3c5:::benchmark_data_smithi159_139342_object29894:head# 2024-04-15T18:34:41.416 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4508d91:::benchmark_data_smithi159_139342_object21955:head# 2024-04-15T18:34:41.416 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c451c1dd:::benchmark_data_smithi159_139342_object33892:head# 2024-04-15T18:34:41.416 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c452bcd5:::benchmark_data_smithi159_139342_object2896:head# 2024-04-15T18:34:41.416 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c45399d8:::benchmark_data_smithi159_139342_object19878:head# 2024-04-15T18:34:41.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:41.424 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4556543:::benchmark_data_smithi159_139342_object56421:head# 2024-04-15T18:34:41.424 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c455d012:::benchmark_data_smithi159_139342_object8843:head# 2024-04-15T18:34:41.424 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4579081:::benchmark_data_smithi159_139342_object34906:head# 2024-04-15T18:34:41.424 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c458ed85:::benchmark_data_smithi159_139342_object32503:head# 2024-04-15T18:34:41.424 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c45a3798:::benchmark_data_smithi159_139342_object35973:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c45b0c6a:::benchmark_data_smithi159_139342_object53274:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c45cb447:::benchmark_data_smithi159_139342_object55453:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c45e7fc9:::benchmark_data_smithi159_139342_object14002:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c460249d:::benchmark_data_smithi159_139342_object20532:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c460e7af:::benchmark_data_smithi159_139342_object12857:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c461348d:::benchmark_data_smithi159_139342_object2934:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4619e57:::benchmark_data_smithi159_139342_object14245:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c461c579:::benchmark_data_smithi159_139342_object33937:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c463333d:::benchmark_data_smithi159_139342_object41914:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4642eed:::benchmark_data_smithi159_139342_object2058:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4677b2c:::benchmark_data_smithi159_139342_object13361:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4688362:::benchmark_data_smithi159_139342_object11601:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46895d4:::benchmark_data_smithi159_139342_object38450:head# 2024-04-15T18:34:41.425 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c468b2db:::benchmark_data_smithi159_139342_object56910:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4692587:::benchmark_data_smithi159_139342_object11865:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46d51c1:::benchmark_data_smithi159_139342_object2801:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46d56c9:::benchmark_data_smithi159_139342_object53408:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46d765b:::benchmark_data_smithi159_139342_object43381:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46f28ec:::benchmark_data_smithi159_139342_object52936:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c46f8d36:::benchmark_data_smithi159_139342_object26445:head# 2024-04-15T18:34:41.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4701d4f:::benchmark_data_smithi159_139342_object54860:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4714d70:::benchmark_data_smithi159_139342_object20974:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c471fd9c:::benchmark_data_smithi159_139342_object28437:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4732944:::benchmark_data_smithi159_139342_object11500:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47329fa:::benchmark_data_smithi159_139342_object18298:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4740242:::benchmark_data_smithi159_139342_object13127:head# 2024-04-15T18:34:41.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47436de:::benchmark_data_smithi159_139342_object5141:head# 2024-04-15T18:34:41.437 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47454e0:::benchmark_data_smithi159_139342_object25335:head# 2024-04-15T18:34:41.437 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c475ff6c:::benchmark_data_smithi159_139342_object46743:head# 2024-04-15T18:34:41.441 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c476f0f6:::benchmark_data_smithi159_139342_object1508:head# 2024-04-15T18:34:41.441 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c478285d:::benchmark_data_smithi159_139342_object3199:head# 2024-04-15T18:34:41.441 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c478bfeb:::benchmark_data_smithi159_139342_object4253:head# 2024-04-15T18:34:41.441 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47920c7:::benchmark_data_smithi159_139342_object29742:head# 2024-04-15T18:34:41.441 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4792eb3:::benchmark_data_smithi159_139342_object21054:head# 2024-04-15T18:34:41.442 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47b258d:::benchmark_data_smithi159_139342_object30665:head# 2024-04-15T18:34:41.442 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47dda14:::benchmark_data_smithi159_139342_object13891:head# 2024-04-15T18:34:41.442 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47ddcf6:::benchmark_data_smithi159_139342_object51786:head# 2024-04-15T18:34:41.442 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47eafc8:::benchmark_data_smithi159_139342_object46721:head# 2024-04-15T18:34:41.442 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47ec696:::benchmark_data_smithi159_139342_object33409:head# 2024-04-15T18:34:41.444 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c47f38f0:::benchmark_data_smithi159_139342_object1915:head# 2024-04-15T18:34:41.444 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c480a717:::benchmark_data_smithi159_139342_object25048:head# 2024-04-15T18:34:41.444 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c480b3c3:::benchmark_data_smithi159_139342_object48300:head# 2024-04-15T18:34:41.444 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4839239:::benchmark_data_smithi159_139342_object27849:head# 2024-04-15T18:34:41.444 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c484f861:::benchmark_data_smithi159_139342_object19769:head# 2024-04-15T18:34:41.445 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4863bdd:::benchmark_data_smithi159_139342_object5117:head# 2024-04-15T18:34:41.445 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4867905:::benchmark_data_smithi159_139342_object40463:head# 2024-04-15T18:34:41.445 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4872f12:::benchmark_data_smithi159_139342_object48533:head# 2024-04-15T18:34:41.446 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4894a02:::benchmark_data_smithi159_139342_object21230:head# 2024-04-15T18:34:41.446 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c489919d:::benchmark_data_smithi159_139342_object5803:head# 2024-04-15T18:34:41.447 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c48ae89e:::benchmark_data_smithi159_139342_object39135:head# 2024-04-15T18:34:41.447 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c48cb081:::benchmark_data_smithi159_139342_object29358:head# 2024-04-15T18:34:41.448 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c48cce4d:::benchmark_data_smithi159_139342_object26528:head# 2024-04-15T18:34:41.448 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c48d5308:::benchmark_data_smithi159_139342_object54929:head# 2024-04-15T18:34:41.449 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4926277:::benchmark_data_smithi159_139342_object16114:head# 2024-04-15T18:34:41.449 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c496f7e9:::benchmark_data_smithi159_139342_object39324:head# 2024-04-15T18:34:41.450 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c498fecd:::benchmark_data_smithi159_139342_object27604:head# 2024-04-15T18:34:41.455 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c499d4cc:::benchmark_data_smithi159_139342_object51548:head# 2024-04-15T18:34:41.455 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49a1007:::benchmark_data_smithi159_139342_object27566:head# 2024-04-15T18:34:41.455 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49be5c8:::benchmark_data_smithi159_139342_object51243:head# 2024-04-15T18:34:41.455 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49cb50d:::benchmark_data_smithi159_139342_object8063:head# 2024-04-15T18:34:41.455 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49cd3c6:::benchmark_data_smithi159_139342_object17709:head# 2024-04-15T18:34:41.456 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49cf3bc:::benchmark_data_smithi159_139342_object44668:head# 2024-04-15T18:34:41.456 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49d130c:::benchmark_data_smithi159_139342_object15173:head# 2024-04-15T18:34:41.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49d61eb:::benchmark_data_smithi159_139342_object5903:head# 2024-04-15T18:34:41.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49dd84c:::benchmark_data_smithi159_139342_object21732:head# 2024-04-15T18:34:41.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49e6fe0:::benchmark_data_smithi159_139342_object23657:head# 2024-04-15T18:34:41.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49e8f15:::benchmark_data_smithi159_139342_object37154:head# 2024-04-15T18:34:41.461 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49f2c40:::benchmark_data_smithi159_139342_object40321:head# 2024-04-15T18:34:41.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c49f5500:::benchmark_data_smithi159_139342_object10627:head# 2024-04-15T18:34:41.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a07c61:::benchmark_data_smithi159_139342_object11789:head# 2024-04-15T18:34:41.463 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a186e8:::benchmark_data_smithi159_139342_object55178:head# 2024-04-15T18:34:41.463 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a233da:::benchmark_data_smithi159_139342_object27879:head# 2024-04-15T18:34:41.464 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a278ed:::benchmark_data_smithi159_139342_object31936:head# 2024-04-15T18:34:41.464 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a3b5cf:::benchmark_data_smithi159_139342_object54858:head# 2024-04-15T18:34:41.464 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a4467f:::benchmark_data_smithi159_139342_object15953:head# 2024-04-15T18:34:41.466 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4a97685:::benchmark_data_smithi159_139342_object26501:head# 2024-04-15T18:34:41.466 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4aa9a5c:::benchmark_data_smithi159_139342_object44619:head# 2024-04-15T18:34:41.466 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4abacac:::benchmark_data_smithi159_139342_object21424:head# 2024-04-15T18:34:41.467 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4abee4f:::benchmark_data_smithi159_139342_object38853:head# 2024-04-15T18:34:41.467 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4af4f17:::benchmark_data_smithi159_139342_object26246:head# 2024-04-15T18:34:41.468 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b0238c:::benchmark_data_smithi159_139342_object9062:head# 2024-04-15T18:34:41.468 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b086bf:::benchmark_data_smithi159_139342_object18096:head# 2024-04-15T18:34:41.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b16be8:::benchmark_data_smithi159_139342_object38481:head# 2024-04-15T18:34:41.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b1c4c0:::benchmark_data_smithi159_139342_object45692:head# 2024-04-15T18:34:41.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b2f236:::benchmark_data_smithi159_139342_object56555:head# 2024-04-15T18:34:41.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b32ed9:::benchmark_data_smithi159_139342_object46358:head# 2024-04-15T18:34:41.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b35056:::benchmark_data_smithi159_139342_object26328:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b3b3bb:::benchmark_data_smithi159_139342_object33847:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b5577b:::benchmark_data_smithi159_139342_object16368:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b5b5a9:::benchmark_data_smithi159_139342_object21234:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4b87b7b:::benchmark_data_smithi159_139342_object18086:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4bcac45:::benchmark_data_smithi159_139342_object27906:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c26e10:::benchmark_data_smithi159_139342_object45866:head# 2024-04-15T18:34:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:41.483 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c47137:::benchmark_data_smithi159_139342_object1689:head# 2024-04-15T18:34:41.483 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c5bb83:::benchmark_data_smithi159_139342_object44809:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c6c66e:::benchmark_data_smithi159_139342_object9465:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c75fb5:::benchmark_data_smithi159_139342_object19898:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4c9ab39:::benchmark_data_smithi159_139342_object34892:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4ca6a38:::benchmark_data_smithi159_139342_object34644:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4cb8a87:::benchmark_data_smithi159_139342_object11244:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4cc02bf:::benchmark_data_smithi159_139342_object24744:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4cca7b1:::benchmark_data_smithi159_139342_object27556:head# 2024-04-15T18:34:41.486 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4ccac09:::benchmark_data_smithi159_139342_object48930:head# 2024-04-15T18:34:41.487 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4cd354b:::benchmark_data_smithi159_139342_object20322:head# 2024-04-15T18:34:41.487 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4cea8a1:::benchmark_data_smithi159_139342_object28624:head# 2024-04-15T18:34:41.487 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4d05232:::benchmark_data_smithi159_139342_object50995:head# 2024-04-15T18:34:41.488 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4d08cf3:::benchmark_data_smithi159_139342_object47874:head# 2024-04-15T18:34:41.488 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4d42b0d:::benchmark_data_smithi159_139342_object36149:head# 2024-04-15T18:34:41.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4d4fceb:::benchmark_data_smithi159_139342_object24541:head# 2024-04-15T18:34:41.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4d5eddc:::benchmark_data_smithi159_139342_object37972:head# 2024-04-15T18:34:41.495 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4da1833:::benchmark_data_smithi159_139342_object19922:head# 2024-04-15T18:34:41.495 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dac947:::benchmark_data_smithi159_139342_object43394:head# 2024-04-15T18:34:41.497 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4db3776:::benchmark_data_smithi159_139342_object29052:head# 2024-04-15T18:34:41.497 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dc4cf2:::benchmark_data_smithi159_139342_object18081:head# 2024-04-15T18:34:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:41.502 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dc8649:::benchmark_data_smithi159_139342_object20932:head# 2024-04-15T18:34:41.503 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dce5ac:::benchmark_data_smithi159_139342_object15239:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4de870c:::benchmark_data_smithi159_139342_object8207:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dfacfc:::benchmark_data_smithi159_139342_object41879:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4dfd46c:::benchmark_data_smithi159_139342_object4566:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e09f4f:::benchmark_data_smithi159_139342_object7894:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e2d664:::benchmark_data_smithi159_139342_object57076:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e33278:::benchmark_data_smithi159_139342_object1691:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e66108:::benchmark_data_smithi159_139342_object42356:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e8e0ba:::benchmark_data_smithi159_139342_object30114:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4e98f69:::benchmark_data_smithi159_139342_object32862:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4eaaf62:::benchmark_data_smithi159_139342_object19394:head# 2024-04-15T18:34:41.526 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4eca703:::benchmark_data_smithi159_139342_object518:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4ed2f5b:::benchmark_data_smithi159_139342_object4455:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4f0cced:::benchmark_data_smithi159_139342_object49081:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4f24822:::benchmark_data_smithi159_139342_object28905:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4f414d3:::benchmark_data_smithi159_139342_object2389:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4f65097:::benchmark_data_smithi159_139342_object2454:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fa8afa:::benchmark_data_smithi159_139342_object17224:head# 2024-04-15T18:34:41.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fa91d1:::benchmark_data_smithi159_139342_object10040:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fc3bc5:::benchmark_data_smithi159_139342_object41849:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fd1477:::benchmark_data_smithi159_139342_object20555:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fe7a69:::benchmark_data_smithi159_139342_object32270:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4fed02d:::benchmark_data_smithi159_139342_object19805:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4ff62ea:::benchmark_data_smithi159_139342_object13411:head# 2024-04-15T18:34:41.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c4ff80ee:::benchmark_data_smithi159_139342_object53418:head# 2024-04-15T18:34:41.529 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c501511f:::benchmark_data_smithi159_139342_object31430:head# 2024-04-15T18:34:41.529 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c502722f:::benchmark_data_smithi159_139342_object35021:head# 2024-04-15T18:34:41.530 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50452aa:::benchmark_data_smithi159_139342_object23934:head# 2024-04-15T18:34:41.530 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c506e422:::benchmark_data_smithi159_139342_object22483:head# 2024-04-15T18:34:41.530 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50a25c5:::benchmark_data_smithi159_139342_object36216:head# 2024-04-15T18:34:41.531 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50c166e:::benchmark_data_smithi159_139342_object26687:head# 2024-04-15T18:34:41.531 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50c47e7:::benchmark_data_smithi159_139342_object30952:head# 2024-04-15T18:34:41.532 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50c546a:::benchmark_data_smithi159_139342_object8214:head# 2024-04-15T18:34:41.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50ca6c2:::benchmark_data_smithi159_139342_object24456:head# 2024-04-15T18:34:41.539 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50d780f:::benchmark_data_smithi159_139342_object23450:head# 2024-04-15T18:34:41.539 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50e3e4e:::benchmark_data_smithi159_139342_object43560:head# 2024-04-15T18:34:41.540 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50ea5af:::benchmark_data_smithi159_139342_object38217:head# 2024-04-15T18:34:41.540 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50f3ed1:::benchmark_data_smithi159_139342_object14236:head# 2024-04-15T18:34:41.542 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50f52b2:::benchmark_data_smithi159_139342_object28659:head# 2024-04-15T18:34:41.543 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c50f86f1:::benchmark_data_smithi159_139342_object39302:head# 2024-04-15T18:34:41.543 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5100c28:::benchmark_data_smithi159_139342_object28834:head# 2024-04-15T18:34:41.544 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5123606:::benchmark_data_smithi159_139342_object22636:head# 2024-04-15T18:34:41.547 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c512a431:::benchmark_data_smithi159_139342_object18384:head# 2024-04-15T18:34:41.547 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5140563:::benchmark_data_smithi159_139342_object26256:head# 2024-04-15T18:34:41.548 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5143a47:::benchmark_data_smithi159_139342_object7575:head# 2024-04-15T18:34:41.548 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5148193:::benchmark_data_smithi159_139342_object3329:head# 2024-04-15T18:34:41.549 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c514ed52:::benchmark_data_smithi159_139342_object19107:head# 2024-04-15T18:34:41.549 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c516f9d0:::benchmark_data_smithi159_139342_object44520:head# 2024-04-15T18:34:41.550 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c518ac7e:::benchmark_data_smithi159_139342_object27686:head# 2024-04-15T18:34:41.552 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c518d0f1:::benchmark_data_smithi159_139342_object7723:head# 2024-04-15T18:34:41.552 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51c5482:::benchmark_data_smithi159_139342_object25546:head# 2024-04-15T18:34:41.553 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51d2967:::benchmark_data_smithi159_139342_object26413:head# 2024-04-15T18:34:41.554 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51d8738:::benchmark_data_smithi159_139342_object5888:head# 2024-04-15T18:34:41.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51e24c4:::benchmark_data_smithi159_139342_object51306:head# 2024-04-15T18:34:41.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51e2831:::benchmark_data_smithi159_139342_object465:head# 2024-04-15T18:34:41.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c51e70dd:::benchmark_data_smithi159_139342_object32040:head# 2024-04-15T18:34:41.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c521d86d:::benchmark_data_smithi159_139342_object30652:head# 2024-04-15T18:34:41.559 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c522c0ed:::benchmark_data_smithi159_139342_object53153:head# 2024-04-15T18:34:41.559 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c525dd83:::benchmark_data_smithi159_139342_object7771:head# 2024-04-15T18:34:41.559 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5268e8a:::benchmark_data_smithi159_139342_object5381:head# 2024-04-15T18:34:41.559 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5279692:::benchmark_data_smithi159_139342_object3296:head# 2024-04-15T18:34:41.559 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c527b357:::benchmark_data_smithi159_139342_object56909:head# 2024-04-15T18:34:41.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c527cc99:::benchmark_data_smithi159_139342_object31659:head# 2024-04-15T18:34:41.564 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c52ac270:::benchmark_data_smithi159_139342_object15820:head# 2024-04-15T18:34:41.569 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c52b061e:::benchmark_data_smithi159_139342_object12294:head# 2024-04-15T18:34:41.569 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c52c4389:::benchmark_data_smithi159_139342_object52814:head# 2024-04-15T18:34:41.569 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c52cd7cf:::benchmark_data_smithi159_139342_object55407:head# 2024-04-15T18:34:41.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c52eb757:::benchmark_data_smithi159_139342_object43719:head# 2024-04-15T18:34:41.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c531efe2:::benchmark_data_smithi159_139342_object44202:head# 2024-04-15T18:34:41.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5325a15:::benchmark_data_smithi159_139342_object18526:head# 2024-04-15T18:34:41.572 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53413cc:::benchmark_data_smithi159_139342_object39768:head# 2024-04-15T18:34:41.572 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53725f6:::benchmark_data_smithi159_139342_object4076:head# 2024-04-15T18:34:41.572 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53730f4:::benchmark_data_smithi159_139342_object17769:head# 2024-04-15T18:34:41.573 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5376123:::benchmark_data_smithi159_139342_object10986:head# 2024-04-15T18:34:41.574 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5388089:::benchmark_data_smithi159_139342_object18325:head# 2024-04-15T18:34:41.574 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c539bfcf:::benchmark_data_smithi159_139342_object1150:head# 2024-04-15T18:34:41.575 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53a51cb:::benchmark_data_smithi159_139342_object41769:head# 2024-04-15T18:34:41.575 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53a72bd:::benchmark_data_smithi159_139342_object19931:head# 2024-04-15T18:34:41.575 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53bdcfe:::benchmark_data_smithi159_139342_object7142:head# 2024-04-15T18:34:41.576 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53ecab9:::benchmark_data_smithi159_139342_object47523:head# 2024-04-15T18:34:41.576 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c53f6c28:::benchmark_data_smithi159_139342_object35256:head# 2024-04-15T18:34:41.577 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54076d9:::benchmark_data_smithi159_139342_object47054:head# 2024-04-15T18:34:41.578 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54378e2:::benchmark_data_smithi159_139342_object27574:head# 2024-04-15T18:34:41.578 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c545ba4e:::benchmark_data_smithi159_139342_object35769:head# 2024-04-15T18:34:41.578 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c546387a:::benchmark_data_smithi159_139342_object41092:head# 2024-04-15T18:34:41.579 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54815a0:::benchmark_data_smithi159_139342_object11913:head# 2024-04-15T18:34:41.579 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54a6039:::benchmark_data_smithi159_139342_object7538:head# 2024-04-15T18:34:41.585 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54bfd0f:::benchmark_data_smithi159_139342_object22408:head# 2024-04-15T18:34:41.585 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54c5ca7:::benchmark_data_smithi159_139342_object44543:head# 2024-04-15T18:34:41.588 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54d4bb0:::benchmark_data_smithi159_139342_object955:head# 2024-04-15T18:34:41.590 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54d9f4d:::benchmark_data_smithi159_139342_object47537:head# 2024-04-15T18:34:41.593 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54f3374:::benchmark_data_smithi159_139342_object54280:head# 2024-04-15T18:34:41.593 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54f3cbc:::benchmark_data_smithi159_139342_object25520:head# 2024-04-15T18:34:41.593 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c54fba45:::benchmark_data_smithi159_139342_object54976:head# 2024-04-15T18:34:41.598 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5501610:::benchmark_data_smithi159_139342_object21939:head# 2024-04-15T18:34:41.602 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c553bb39:::benchmark_data_smithi159_139342_object27826:head# 2024-04-15T18:34:41.602 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5540abb:::benchmark_data_smithi159_139342_object5931:head# 2024-04-15T18:34:41.603 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c554996d:::benchmark_data_smithi159_139342_object6293:head# 2024-04-15T18:34:41.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c555a8c5:::benchmark_data_smithi159_139342_object7934:head# 2024-04-15T18:34:41.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c555b762:::benchmark_data_smithi159_139342_object40173:head# 2024-04-15T18:34:41.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c55946f7:::benchmark_data_smithi159_139342_object47806:head# 2024-04-15T18:34:41.604 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c559ed56:::benchmark_data_smithi159_139342_object39568:head# 2024-04-15T18:34:41.606 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c55b4ab2:::benchmark_data_smithi159_139342_object5139:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c55b9320:::benchmark_data_smithi159_139342_object13402:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c55f27b6:::benchmark_data_smithi159_139342_object20946:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c560e7b6:::benchmark_data_smithi159_139342_object27612:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5642125:::benchmark_data_smithi159_139342_object8473:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c565f679:::benchmark_data_smithi159_139342_object34289:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56608db:::benchmark_data_smithi159_139342_object31606:head# 2024-04-15T18:34:41.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5660c94:::benchmark_data_smithi159_139342_object16598:head# 2024-04-15T18:34:41.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c567b3e1:::benchmark_data_smithi159_139342_object33470:head# 2024-04-15T18:34:41.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c567eda9:::benchmark_data_smithi159_139342_object20841:head# 2024-04-15T18:34:41.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56835c7:::benchmark_data_smithi159_139342_object41341:head# 2024-04-15T18:34:41.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c568420a:::benchmark_data_smithi159_139342_object36888:head# 2024-04-15T18:34:41.619 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56982c9:::benchmark_data_smithi159_139342_object53226:head# 2024-04-15T18:34:41.620 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c569d369:::benchmark_data_smithi159_139342_object1789:head# 2024-04-15T18:34:41.620 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56a25d2:::benchmark_data_smithi159_139342_object51518:head# 2024-04-15T18:34:41.620 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56b00c0:::benchmark_data_smithi159_139342_object44782:head# 2024-04-15T18:34:41.621 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56b600c:::benchmark_data_smithi159_139342_object29714:head# 2024-04-15T18:34:41.621 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56b9858:::benchmark_data_smithi159_139342_object14619:head# 2024-04-15T18:34:41.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56c8051:::benchmark_data_smithi159_139342_object42120:head# 2024-04-15T18:34:41.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56c9d2d:::benchmark_data_smithi159_139342_object53395:head# 2024-04-15T18:34:41.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56d3d4a:::benchmark_data_smithi159_139342_object38049:head# 2024-04-15T18:34:41.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56e2169:::benchmark_data_smithi159_139342_object23571:head# 2024-04-15T18:34:41.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56eaf65:::benchmark_data_smithi159_139342_object36405:head# 2024-04-15T18:34:41.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c56f7b7a:::benchmark_data_smithi159_139342_object1839:head# 2024-04-15T18:34:41.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5709423:::benchmark_data_smithi159_139342_object40100:head# 2024-04-15T18:34:41.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c570a212:::benchmark_data_smithi159_139342_object8803:head# 2024-04-15T18:34:41.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c571ee60:::benchmark_data_smithi159_139342_object29220:head# 2024-04-15T18:34:41.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5738446:::benchmark_data_smithi159_139342_object40894:head# 2024-04-15T18:34:41.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c573f447:::benchmark_data_smithi159_139342_object42144:head# 2024-04-15T18:34:41.628 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5741c7c:::benchmark_data_smithi159_139342_object16520:head# 2024-04-15T18:34:41.628 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5754960:::benchmark_data_smithi159_139342_object10264:head# 2024-04-15T18:34:41.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c57550ad:::benchmark_data_smithi159_139342_object47742:head# 2024-04-15T18:34:41.629 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c575d5d1:::benchmark_data_smithi159_139342_object57247:head# 2024-04-15T18:34:41.630 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c57a827b:::benchmark_data_smithi159_139342_object8249:head# 2024-04-15T18:34:41.630 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c57c0b6a:::benchmark_data_smithi159_139342_object32570:head# 2024-04-15T18:34:41.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c57e4a9a:::benchmark_data_smithi159_139342_object50374:head# 2024-04-15T18:34:41.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c57f9b46:::benchmark_data_smithi159_139342_object54592:head# 2024-04-15T18:34:41.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c580cfd3:::benchmark_data_smithi159_139342_object8570:head# 2024-04-15T18:34:41.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5816aef:::benchmark_data_smithi159_139342_object45349:head# 2024-04-15T18:34:41.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c582efbf:::benchmark_data_smithi159_139342_object32893:head# 2024-04-15T18:34:41.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c583fddd:::benchmark_data_smithi159_139342_object21700:head# 2024-04-15T18:34:41.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c585f6c2:::benchmark_data_smithi159_139342_object44872:head# 2024-04-15T18:34:41.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5874160:::benchmark_data_smithi159_139342_object27497:head# 2024-04-15T18:34:41.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c587b34a:::benchmark_data_smithi159_139342_object10531:head# 2024-04-15T18:34:41.637 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58a89a6:::benchmark_data_smithi159_139342_object13753:head# 2024-04-15T18:34:41.637 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58b746e:::benchmark_data_smithi159_139342_object30957:head# 2024-04-15T18:34:41.637 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58c7af8:::benchmark_data_smithi159_139342_object50218:head# 2024-04-15T18:34:41.637 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58d3825:::benchmark_data_smithi159_139342_object45037:head# 2024-04-15T18:34:41.640 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58d80c9:::benchmark_data_smithi159_139342_object22322:head# 2024-04-15T18:34:41.640 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58dcf1e:::benchmark_data_smithi159_139342_object29177:head# 2024-04-15T18:34:41.640 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58ea049:::benchmark_data_smithi159_139342_object12244:head# 2024-04-15T18:34:41.640 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c58fc3d6:::benchmark_data_smithi159_139342_object7346:head# 2024-04-15T18:34:41.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59224aa:::benchmark_data_smithi159_139342_object51132:head# 2024-04-15T18:34:41.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5925bc7:::benchmark_data_smithi159_139342_object11843:head# 2024-04-15T18:34:41.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59566fe:::benchmark_data_smithi159_139342_object23568:head# 2024-04-15T18:34:41.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59613c6:::benchmark_data_smithi159_139342_object15478:head# 2024-04-15T18:34:41.647 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5988821:::benchmark_data_smithi159_139342_object22309:head# 2024-04-15T18:34:41.647 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59932e9:::benchmark_data_smithi159_139342_object23488:head# 2024-04-15T18:34:41.647 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5993b5b:::benchmark_data_smithi159_139342_object9129:head# 2024-04-15T18:34:41.648 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c599a358:::benchmark_data_smithi159_139342_object54062:head# 2024-04-15T18:34:41.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59a029c:::benchmark_data_smithi159_139342_object39459:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59a84ae:::benchmark_data_smithi159_139342_object32456:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59bcc28:::benchmark_data_smithi159_139342_object29665:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59c2733:::benchmark_data_smithi159_139342_object5163:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59c546e:::benchmark_data_smithi159_139342_object18148:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59cdfdb:::benchmark_data_smithi159_139342_object20675:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59e5b65:::benchmark_data_smithi159_139342_object16468:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59ee26a:::benchmark_data_smithi159_139342_object18446:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c59f93b9:::benchmark_data_smithi159_139342_object36156:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a40ecb:::benchmark_data_smithi159_139342_object49986:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a60f6f:::benchmark_data_smithi159_139342_object45561:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a6e691:::benchmark_data_smithi159_139342_object1151:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a6eb2b:::benchmark_data_smithi159_139342_object40646:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a7c1fa:::benchmark_data_smithi159_139342_object49521:head# 2024-04-15T18:34:41.663 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a89829:::benchmark_data_smithi159_139342_object1410:head# 2024-04-15T18:34:41.664 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a945e1:::benchmark_data_smithi159_139342_object42998:head# 2024-04-15T18:34:41.664 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a94fa8:::benchmark_data_smithi159_139342_object38691:head# 2024-04-15T18:34:41.664 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5a9fda7:::benchmark_data_smithi159_139342_object45100:head# 2024-04-15T18:34:41.664 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5aa894a:::benchmark_data_smithi159_139342_object49204:head# 2024-04-15T18:34:41.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ac4e82:::benchmark_data_smithi159_139342_object51172:head# 2024-04-15T18:34:41.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ad1363:::benchmark_data_smithi159_139342_object23363:head# 2024-04-15T18:34:41.666 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5adf8c5:::benchmark_data_smithi159_139342_object37995:head# 2024-04-15T18:34:41.666 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ae25b9:::benchmark_data_smithi159_139342_object38879:head# 2024-04-15T18:34:41.666 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ae7494:::benchmark_data_smithi159_139342_object45044:head# 2024-04-15T18:34:41.667 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5aeacf5:::benchmark_data_smithi159_139342_object23651:head# 2024-04-15T18:34:41.667 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5af1ec7:::benchmark_data_smithi159_139342_object49784:head# 2024-04-15T18:34:41.671 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b07ade:::benchmark_data_smithi159_139342_object27783:head# 2024-04-15T18:34:41.675 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b25a89:::benchmark_data_smithi159_139342_object41583:head# 2024-04-15T18:34:41.682 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b31e2c:::benchmark_data_smithi159_139342_object47085:head# 2024-04-15T18:34:41.682 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b5cb01:::benchmark_data_smithi159_139342_object16538:head# 2024-04-15T18:34:41.683 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b6892f:::benchmark_data_smithi159_139342_object6267:head# 2024-04-15T18:34:41.683 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b7bec7:::benchmark_data_smithi159_139342_object39679:head# 2024-04-15T18:34:41.683 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b807de:::benchmark_data_smithi159_139342_object654:head# 2024-04-15T18:34:41.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b81cc4:::benchmark_data_smithi159_139342_object2002:head# 2024-04-15T18:34:41.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5b8af1b:::benchmark_data_smithi159_139342_object40924:head# 2024-04-15T18:34:41.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ba1ce3:::benchmark_data_smithi159_139342_object21665:head# 2024-04-15T18:34:41.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ba4bc8:::benchmark_data_smithi159_139342_object8205:head# 2024-04-15T18:34:41.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ba8e74:::benchmark_data_smithi159_139342_object22895:head# 2024-04-15T18:34:41.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5bab319:::benchmark_data_smithi159_139342_object53477:head# 2024-04-15T18:34:41.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5bde023:::benchmark_data_smithi159_139342_object24174:head# 2024-04-15T18:34:41.693 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5bee4fd:::benchmark_data_smithi159_139342_object41410:head# 2024-04-15T18:34:41.693 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5bf2ea0:::benchmark_data_smithi159_139342_object579:head# 2024-04-15T18:34:41.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c00398:::benchmark_data_smithi159_139342_object51497:head# 2024-04-15T18:34:41.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c046f3:::benchmark_data_smithi159_139342_object12109:head# 2024-04-15T18:34:41.695 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c0962a:::benchmark_data_smithi159_139342_object31338:head# 2024-04-15T18:34:41.695 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c1554f:::benchmark_data_smithi159_139342_object50094:head# 2024-04-15T18:34:41.696 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c22be1:::benchmark_data_smithi159_139342_object51521:head# 2024-04-15T18:34:41.696 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c2b258:::benchmark_data_smithi159_139342_object32768:head# 2024-04-15T18:34:41.697 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c2ea5f:::benchmark_data_smithi159_139342_object24347:head# 2024-04-15T18:34:41.697 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c31902:::benchmark_data_smithi159_139342_object6210:head# 2024-04-15T18:34:41.698 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c41423:::benchmark_data_smithi159_139342_object21318:head# 2024-04-15T18:34:41.698 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c7b9a9:::benchmark_data_smithi159_139342_object52502:head# 2024-04-15T18:34:41.698 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5c868ce:::benchmark_data_smithi159_139342_object35488:head# 2024-04-15T18:34:41.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5cea047:::benchmark_data_smithi159_139342_object10549:head# 2024-04-15T18:34:41.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5cf62b2:::benchmark_data_smithi159_139342_object4328:head# 2024-04-15T18:34:41.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5cf88bd:::benchmark_data_smithi159_139342_object46450:head# 2024-04-15T18:34:41.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d1b7a3:::benchmark_data_smithi159_139342_object54767:head# 2024-04-15T18:34:41.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d1c021:::benchmark_data_smithi159_139342_object11516:head# 2024-04-15T18:34:41.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d3b0e0:::benchmark_data_smithi159_139342_object51665:head# 2024-04-15T18:34:41.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d3dec8:::benchmark_data_smithi159_139342_object13712:head# 2024-04-15T18:34:41.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d5a74a:::benchmark_data_smithi159_139342_object42384:head# 2024-04-15T18:34:41.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d5e4bc:::benchmark_data_smithi159_139342_object46228:head# 2024-04-15T18:34:41.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d648c4:::benchmark_data_smithi159_139342_object37721:head# 2024-04-15T18:34:41.704 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d734f0:::benchmark_data_smithi159_139342_object11816:head# 2024-04-15T18:34:41.704 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d7d6cb:::benchmark_data_smithi159_139342_object55085:head# 2024-04-15T18:34:41.704 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d84760:::benchmark_data_smithi159_139342_object26588:head# 2024-04-15T18:34:41.705 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d89969:::benchmark_data_smithi159_139342_object25597:head# 2024-04-15T18:34:41.705 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5d8fde3:::benchmark_data_smithi159_139342_object30908:head# 2024-04-15T18:34:41.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dac53b:::benchmark_data_smithi159_139342_object48848:head# 2024-04-15T18:34:41.716 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5db0c14:::benchmark_data_smithi159_139342_object19059:head# 2024-04-15T18:34:41.716 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dbd6d6:::benchmark_data_smithi159_139342_object38002:head# 2024-04-15T18:34:41.717 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dc84eb:::benchmark_data_smithi159_139342_object8404:head# 2024-04-15T18:34:41.717 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dccac4:::benchmark_data_smithi159_139342_object28628:head# 2024-04-15T18:34:41.718 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dd1ee3:::benchmark_data_smithi159_139342_object24088:head# 2024-04-15T18:34:41.719 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5dd579e:::benchmark_data_smithi159_139342_object24558:head# 2024-04-15T18:34:41.719 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5df7b94:::benchmark_data_smithi159_139342_object44956:head# 2024-04-15T18:34:41.719 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5df9f4c:::benchmark_data_smithi159_139342_object51037:head# 2024-04-15T18:34:41.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e23d10:::benchmark_data_smithi159_139342_object21837:head# 2024-04-15T18:34:41.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e2bbde:::benchmark_data_smithi159_139342_object53234:head# 2024-04-15T18:34:41.723 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e47ada:::benchmark_data_smithi159_139342_object37287:head# 2024-04-15T18:34:41.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e5e35c:::benchmark_data_smithi159_139342_object7922:head# 2024-04-15T18:34:41.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e74bd8:::benchmark_data_smithi159_139342_object9976:head# 2024-04-15T18:34:41.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e8d58f:::benchmark_data_smithi159_139342_object46064:head# 2024-04-15T18:34:41.727 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5e93dcd:::benchmark_data_smithi159_139342_object1277:head# 2024-04-15T18:34:41.727 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5eb3346:::benchmark_data_smithi159_139342_object38620:head# 2024-04-15T18:34:41.727 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5eb6a2f:::benchmark_data_smithi159_139342_object4004:head# 2024-04-15T18:34:41.728 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ec260a:::benchmark_data_smithi159_139342_object44373:head# 2024-04-15T18:34:41.729 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ecc231:::benchmark_data_smithi159_139342_object49359:head# 2024-04-15T18:34:41.729 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5eccaba:::benchmark_data_smithi159_139342_object4491:head# 2024-04-15T18:34:41.730 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ed5f5f:::benchmark_data_smithi159_139342_object38659:head# 2024-04-15T18:34:41.730 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5edfa6c:::benchmark_data_smithi159_139342_object49303:head# 2024-04-15T18:34:41.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f0a8d5:::benchmark_data_smithi159_139342_object11057:head# 2024-04-15T18:34:41.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f0be24:::benchmark_data_smithi159_139342_object10656:head# 2024-04-15T18:34:41.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f249c2:::benchmark_data_smithi159_139342_object33530:head# 2024-04-15T18:34:41.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f3889b:::benchmark_data_smithi159_139342_object3941:head# 2024-04-15T18:34:41.732 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f3ae74:::benchmark_data_smithi159_139342_object48615:head# 2024-04-15T18:34:41.733 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f3cf91:::benchmark_data_smithi159_139342_object49580:head# 2024-04-15T18:34:41.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f4116b:::benchmark_data_smithi159_139342_object18208:head# 2024-04-15T18:34:41.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f56332:::benchmark_data_smithi159_139342_object4087:head# 2024-04-15T18:34:41.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f7898e:::benchmark_data_smithi159_139342_object54745:head# 2024-04-15T18:34:41.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5f8e162:::benchmark_data_smithi159_139342_object21545:head# 2024-04-15T18:34:41.737 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fad283:::benchmark_data_smithi159_139342_object41929:head# 2024-04-15T18:34:41.737 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fb1521:::benchmark_data_smithi159_139342_object14021:head# 2024-04-15T18:34:41.738 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fc4268:::benchmark_data_smithi159_139342_object19297:head# 2024-04-15T18:34:41.738 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fc884e:::benchmark_data_smithi159_139342_object18882:head# 2024-04-15T18:34:41.738 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fcb3db:::benchmark_data_smithi159_139342_object45016:head# 2024-04-15T18:34:41.739 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5fdbd01:::benchmark_data_smithi159_139342_object34735:head# 2024-04-15T18:34:41.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c5ffe5ab:::benchmark_data_smithi159_139342_object26803:head# 2024-04-15T18:34:41.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c602e6ce:::benchmark_data_smithi159_139342_object3009:head# 2024-04-15T18:34:41.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c602e8db:::benchmark_data_smithi159_139342_object17954:head# 2024-04-15T18:34:41.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c60301af:::benchmark_data_smithi159_139342_object43378:head# 2024-04-15T18:34:41.742 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c604cdf5:::benchmark_data_smithi159_139342_object28015:head# 2024-04-15T18:34:41.742 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c605c1bf:::benchmark_data_smithi159_139342_object38955:head# 2024-04-15T18:34:41.745 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c60615c9:::benchmark_data_smithi159_139342_object32735:head# 2024-04-15T18:34:41.751 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6065138:::benchmark_data_smithi159_139342_object35765:head# 2024-04-15T18:34:41.751 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c60eb7b1:::benchmark_data_smithi159_139342_object25825:head# 2024-04-15T18:34:41.751 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c60f1c64:::benchmark_data_smithi159_139342_object31935:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c60fa36e:::benchmark_data_smithi159_139342_object7653:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c611738c:::benchmark_data_smithi159_139342_object25196:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c612664d:::benchmark_data_smithi159_139342_object41580:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c612bcee:::benchmark_data_smithi159_139342_object25340:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6143f59:::benchmark_data_smithi159_139342_object38593:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c615b1c7:::benchmark_data_smithi159_139342_object48447:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61a8d18:::benchmark_data_smithi159_139342_object6008:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61bcfcc:::benchmark_data_smithi159_139342_object22805:head# 2024-04-15T18:34:41.752 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61c0ab0:::benchmark_data_smithi159_139342_object13971:head# 2024-04-15T18:34:41.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61c5806:::benchmark_data_smithi159_139342_object6560:head# 2024-04-15T18:34:41.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61cd590:::benchmark_data_smithi159_139342_object158:head# 2024-04-15T18:34:41.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61d17c5:::benchmark_data_smithi159_139342_object18994:head# 2024-04-15T18:34:41.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61d2432:::benchmark_data_smithi159_139342_object52790:head# 2024-04-15T18:34:41.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61eb2f8:::benchmark_data_smithi159_139342_object23783:head# 2024-04-15T18:34:41.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61f5cab:::benchmark_data_smithi159_139342_object20062:head# 2024-04-15T18:34:41.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c61f63ed:::benchmark_data_smithi159_139342_object36295:head# 2024-04-15T18:34:41.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6202fef:::benchmark_data_smithi159_139342_object49728:head# 2024-04-15T18:34:41.765 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6224e9a:::benchmark_data_smithi159_139342_object18231:head# 2024-04-15T18:34:41.765 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c626d43e:::benchmark_data_smithi159_139342_object43871:head# 2024-04-15T18:34:41.765 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6279ba9:::benchmark_data_smithi159_139342_object25234:head# 2024-04-15T18:34:41.766 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62b1e79:::benchmark_data_smithi159_139342_object6642:head# 2024-04-15T18:34:41.766 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62c252a:::benchmark_data_smithi159_139342_object702:head# 2024-04-15T18:34:41.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62c728f:::benchmark_data_smithi159_139342_object1772:head# 2024-04-15T18:34:41.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62cebea:::benchmark_data_smithi159_139342_object6672:head# 2024-04-15T18:34:41.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62d0de7:::benchmark_data_smithi159_139342_object25571:head# 2024-04-15T18:34:41.767 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62d70f6:::benchmark_data_smithi159_139342_object44454:head# 2024-04-15T18:34:41.768 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62e919d:::benchmark_data_smithi159_139342_object12376:head# 2024-04-15T18:34:41.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c62ee7e3:::benchmark_data_smithi159_139342_object32410:head# 2024-04-15T18:34:41.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c631bfa5:::benchmark_data_smithi159_139342_object41696:head# 2024-04-15T18:34:41.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6322b54:::benchmark_data_smithi159_139342_object36491:head# 2024-04-15T18:34:41.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63280a3:::benchmark_data_smithi159_139342_object47418:head# 2024-04-15T18:34:41.771 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6330acf:::benchmark_data_smithi159_139342_object47115:head# 2024-04-15T18:34:41.772 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6340590:::benchmark_data_smithi159_139342_object44739:head# 2024-04-15T18:34:41.772 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c634399d:::benchmark_data_smithi159_139342_object51475:head# 2024-04-15T18:34:41.773 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63563db:::benchmark_data_smithi159_139342_object16441:head# 2024-04-15T18:34:41.773 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c635c4b7:::benchmark_data_smithi159_139342_object35976:head# 2024-04-15T18:34:41.773 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c638605b:::benchmark_data_smithi159_139342_object41628:head# 2024-04-15T18:34:41.774 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63bf5fc:::benchmark_data_smithi159_139342_object41240:head# 2024-04-15T18:34:41.775 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63c25ff:::benchmark_data_smithi159_139342_object17146:head# 2024-04-15T18:34:41.775 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63d51bf:::benchmark_data_smithi159_139342_object40872:head# 2024-04-15T18:34:41.775 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63f1b5d:::benchmark_data_smithi159_139342_object32165:head# 2024-04-15T18:34:41.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63fac99:::benchmark_data_smithi159_139342_object3882:head# 2024-04-15T18:34:41.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c63fff95:::benchmark_data_smithi159_139342_object249:head# 2024-04-15T18:34:41.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c641f313:::benchmark_data_smithi159_139342_object56638:head# 2024-04-15T18:34:41.777 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6442e59:::benchmark_data_smithi159_139342_object44179:head# 2024-04-15T18:34:41.777 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c646df40:::benchmark_data_smithi159_139342_object25179:head# 2024-04-15T18:34:41.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c647acf3:::benchmark_data_smithi159_139342_object43878:head# 2024-04-15T18:34:41.778 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c647ca5f:::benchmark_data_smithi159_139342_object2625:head# 2024-04-15T18:34:41.779 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c64b47f4:::benchmark_data_smithi159_139342_object53772:head# 2024-04-15T18:34:41.779 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c64d1813:::benchmark_data_smithi159_139342_object34904:head# 2024-04-15T18:34:41.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c64fdde6:::benchmark_data_smithi159_139342_object27886:head# 2024-04-15T18:34:41.780 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c652fda8:::benchmark_data_smithi159_139342_object26116:head# 2024-04-15T18:34:41.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6580524:::benchmark_data_smithi159_139342_object49342:head# 2024-04-15T18:34:41.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c658d747:::benchmark_data_smithi159_139342_object41481:head# 2024-04-15T18:34:41.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6597a6f:::benchmark_data_smithi159_139342_object23998:head# 2024-04-15T18:34:41.784 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65d7d1b:::benchmark_data_smithi159_139342_object6728:head# 2024-04-15T18:34:41.785 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65e6b76:::benchmark_data_smithi159_139342_object31199:head# 2024-04-15T18:34:41.785 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65e9166:::benchmark_data_smithi159_139342_object24150:head# 2024-04-15T18:34:41.788 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65edcb1:::benchmark_data_smithi159_139342_object52591:head# 2024-04-15T18:34:41.788 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65effe1:::benchmark_data_smithi159_139342_object31326:head# 2024-04-15T18:34:41.788 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c65f556d:::benchmark_data_smithi159_139342_object269:head# 2024-04-15T18:34:41.788 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c660af90:::benchmark_data_smithi159_139342_object20619:head# 2024-04-15T18:34:41.789 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6614184:::benchmark_data_smithi159_139342_object11574:head# 2024-04-15T18:34:41.789 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c661de58:::benchmark_data_smithi159_139342_object43229:head# 2024-04-15T18:34:41.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6632fd0:::benchmark_data_smithi159_139342_object46578:head# 2024-04-15T18:34:41.794 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6637d43:::benchmark_data_smithi159_139342_object49467:head# 2024-04-15T18:34:41.795 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c663c122:::benchmark_data_smithi159_139342_object9005:head# 2024-04-15T18:34:41.796 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c665b0d8:::benchmark_data_smithi159_139342_object33293:head# 2024-04-15T18:34:41.796 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6674cb7:::benchmark_data_smithi159_139342_object12593:head# 2024-04-15T18:34:41.796 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66b0126:::benchmark_data_smithi159_139342_object38964:head# 2024-04-15T18:34:41.797 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66cebcd:::benchmark_data_smithi159_139342_object25028:head# 2024-04-15T18:34:41.798 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66e3a8e:::benchmark_data_smithi159_139342_object56084:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66eabfc:::benchmark_data_smithi159_139342_object36342:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66f23f5:::benchmark_data_smithi159_139342_object44998:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c66fb2ac:::benchmark_data_smithi159_139342_object13890:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6701e4f:::benchmark_data_smithi159_139342_object26226:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c671d871:::benchmark_data_smithi159_139342_object11817:head# 2024-04-15T18:34:41.806 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6726d91:::benchmark_data_smithi159_139342_object43352:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6727cf2:::benchmark_data_smithi159_139342_object10542:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6734837:::benchmark_data_smithi159_139342_object48550:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67410fc:::benchmark_data_smithi159_139342_object37415:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6743689:::benchmark_data_smithi159_139342_object21603:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c676f14b:::benchmark_data_smithi159_139342_object34690:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67801f7:::benchmark_data_smithi159_139342_object34348:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67806d4:::benchmark_data_smithi159_139342_object21177:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c679504f:::benchmark_data_smithi159_139342_object20738:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67b8d99:::benchmark_data_smithi159_139342_object53436:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67bca19:::benchmark_data_smithi159_139342_object14337:head# 2024-04-15T18:34:41.807 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67c280c:::benchmark_data_smithi159_139342_object52152:head# 2024-04-15T18:34:41.810 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67cafd6:::benchmark_data_smithi159_139342_object3375:head# 2024-04-15T18:34:41.810 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67cdde5:::benchmark_data_smithi159_139342_object23083:head# 2024-04-15T18:34:41.810 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67d13f8:::benchmark_data_smithi159_139342_object19204:head# 2024-04-15T18:34:41.810 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67d3195:::benchmark_data_smithi159_139342_object25708:head# 2024-04-15T18:34:41.810 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67e0dc9:::benchmark_data_smithi159_139342_object50638:head# 2024-04-15T18:34:41.813 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67e4c1f:::benchmark_data_smithi159_139342_object2692:head# 2024-04-15T18:34:41.815 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67f6f51:::benchmark_data_smithi159_139342_object14933:head# 2024-04-15T18:34:41.816 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c67fb561:::benchmark_data_smithi159_139342_object8015:head# 2024-04-15T18:34:41.816 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6849831:::benchmark_data_smithi159_139342_object14720:head# 2024-04-15T18:34:41.817 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68533f2:::benchmark_data_smithi159_139342_object29254:head# 2024-04-15T18:34:41.817 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68639eb:::benchmark_data_smithi159_139342_object15571:head# 2024-04-15T18:34:41.818 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6868f54:::benchmark_data_smithi159_139342_object30689:head# 2024-04-15T18:34:41.818 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c688124a:::benchmark_data_smithi159_139342_object30417:head# 2024-04-15T18:34:41.819 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c688fae4:::benchmark_data_smithi159_139342_object310:head# 2024-04-15T18:34:41.819 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68bfb04:::benchmark_data_smithi159_139342_object41924:head# 2024-04-15T18:34:41.819 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68de132:::benchmark_data_smithi159_139342_object9534:head# 2024-04-15T18:34:41.820 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68e3875:::benchmark_data_smithi159_139342_object26115:head# 2024-04-15T18:34:41.820 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c68ec22d:::benchmark_data_smithi159_139342_object34910:head# 2024-04-15T18:34:41.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c690fe4d:::benchmark_data_smithi159_139342_object29229:head# 2024-04-15T18:34:41.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c691eb7d:::benchmark_data_smithi159_139342_object52496:head# 2024-04-15T18:34:41.821 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c692ff4c:::benchmark_data_smithi159_139342_object50838:head# 2024-04-15T18:34:41.822 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c696911c:::benchmark_data_smithi159_139342_object28481:head# 2024-04-15T18:34:41.822 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c697978c:::benchmark_data_smithi159_139342_object5442:head# 2024-04-15T18:34:41.823 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6979b5d:::benchmark_data_smithi159_139342_object14120:head# 2024-04-15T18:34:41.823 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c699c73f:::benchmark_data_smithi159_139342_object24379:head# 2024-04-15T18:34:41.824 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c69d1226:::benchmark_data_smithi159_139342_object31650:head# 2024-04-15T18:34:41.824 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c69f8b7b:::benchmark_data_smithi159_139342_object32478:head# 2024-04-15T18:34:41.825 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c69f9d84:::benchmark_data_smithi159_139342_object204:head# 2024-04-15T18:34:41.825 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a068ac:::benchmark_data_smithi159_139342_object12440:head# 2024-04-15T18:34:41.825 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a071f3:::benchmark_data_smithi159_139342_object9049:head# 2024-04-15T18:34:41.826 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a1073d:::benchmark_data_smithi159_139342_object32019:head# 2024-04-15T18:34:41.826 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a11261:::benchmark_data_smithi159_139342_object49831:head# 2024-04-15T18:34:41.828 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a13691:::benchmark_data_smithi159_139342_object25554:head# 2024-04-15T18:34:41.828 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a191db:::benchmark_data_smithi159_139342_object33918:head# 2024-04-15T18:34:41.832 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a24096:::benchmark_data_smithi159_139342_object47000:head# 2024-04-15T18:34:41.832 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a28bcc:::benchmark_data_smithi159_139342_object3111:head# 2024-04-15T18:34:41.832 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a3758e:::benchmark_data_smithi159_139342_object30087:head# 2024-04-15T18:34:41.833 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a61e75:::benchmark_data_smithi159_139342_object55633:head# 2024-04-15T18:34:41.833 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a6a970:::benchmark_data_smithi159_139342_object42765:head# 2024-04-15T18:34:41.833 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6a8b1ce:::benchmark_data_smithi159_139342_object13285:head# 2024-04-15T18:34:41.834 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6aaffde:::benchmark_data_smithi159_139342_object15129:head# 2024-04-15T18:34:41.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ac3928:::benchmark_data_smithi159_139342_object56460:head# 2024-04-15T18:34:41.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ad119e:::benchmark_data_smithi159_139342_object338:head# 2024-04-15T18:34:41.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ae13a2:::benchmark_data_smithi159_139342_object26686:head# 2024-04-15T18:34:41.836 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6af51b9:::benchmark_data_smithi159_139342_object39398:head# 2024-04-15T18:34:41.836 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b0703f:::benchmark_data_smithi159_139342_object42403:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b1da45:::benchmark_data_smithi159_139342_object36547:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b320f6:::benchmark_data_smithi159_139342_object4904:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b38806:::benchmark_data_smithi159_139342_object3238:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b3a14e:::benchmark_data_smithi159_139342_object18478:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b3d5da:::benchmark_data_smithi159_139342_object45241:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b49b61:::benchmark_data_smithi159_139342_object48900:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b6dc17:::benchmark_data_smithi159_139342_object39085:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b6ed4d:::benchmark_data_smithi159_139342_object10440:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6b989c2:::benchmark_data_smithi159_139342_object19192:head# 2024-04-15T18:34:41.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ba2bee:::benchmark_data_smithi159_139342_object48422:head# 2024-04-15T18:34:41.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6bae41c:::benchmark_data_smithi159_139342_object17477:head# 2024-04-15T18:34:41.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6bcab17:::benchmark_data_smithi159_139342_object7:head# 2024-04-15T18:34:41.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6bda468:::benchmark_data_smithi159_139342_object22312:head# 2024-04-15T18:34:41.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c06ab5:::benchmark_data_smithi159_139342_object47816:head# 2024-04-15T18:34:41.843 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c13c49:::benchmark_data_smithi159_139342_object37555:head# 2024-04-15T18:34:41.843 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c264a5:::benchmark_data_smithi159_139342_object12102:head# 2024-04-15T18:34:41.844 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c2f490:::benchmark_data_smithi159_139342_object19578:head# 2024-04-15T18:34:41.844 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c4adfe:::benchmark_data_smithi159_139342_object45451:head# 2024-04-15T18:34:41.845 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c808fe:::benchmark_data_smithi159_139342_object6696:head# 2024-04-15T18:34:41.847 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c863c4:::benchmark_data_smithi159_139342_object36951:head# 2024-04-15T18:34:41.847 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6c898df:::benchmark_data_smithi159_139342_object5639:head# 2024-04-15T18:34:41.849 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ca6d8a:::benchmark_data_smithi159_139342_object38774:head# 2024-04-15T18:34:41.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6cb718f:::benchmark_data_smithi159_139342_object38463:head# 2024-04-15T18:34:41.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6cbf5b2:::benchmark_data_smithi159_139342_object36466:head# 2024-04-15T18:34:41.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6cd31ce:::benchmark_data_smithi159_139342_object42179:head# 2024-04-15T18:34:41.854 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ceaf2b:::benchmark_data_smithi159_139342_object37857:head# 2024-04-15T18:34:41.854 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d0c6de:::benchmark_data_smithi159_139342_object5836:head# 2024-04-15T18:34:41.855 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d1fd97:::benchmark_data_smithi159_139342_object28743:head# 2024-04-15T18:34:41.855 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d433c2:::benchmark_data_smithi159_139342_object13844:head# 2024-04-15T18:34:41.856 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d61b1b:::benchmark_data_smithi159_139342_object48857:head# 2024-04-15T18:34:41.856 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d8381c:::benchmark_data_smithi159_139342_object41504:head# 2024-04-15T18:34:41.857 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d85aea:::benchmark_data_smithi159_139342_object56956:head# 2024-04-15T18:34:41.858 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6d99dd9:::benchmark_data_smithi159_139342_object10989:head# 2024-04-15T18:34:41.860 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6daf3f0:::benchmark_data_smithi159_139342_object4460:head# 2024-04-15T18:34:41.861 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e09df6:::benchmark_data_smithi159_139342_object18975:head# 2024-04-15T18:34:41.862 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e0f46a:::benchmark_data_smithi159_139342_object47553:head# 2024-04-15T18:34:41.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e15a69:::benchmark_data_smithi159_139342_object52248:head# 2024-04-15T18:34:41.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e268a8:::benchmark_data_smithi159_139342_object47452:head# 2024-04-15T18:34:41.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e3112f:::benchmark_data_smithi159_139342_object12672:head# 2024-04-15T18:34:41.865 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e31627:::benchmark_data_smithi159_139342_object4116:head# 2024-04-15T18:34:41.865 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e48013:::benchmark_data_smithi159_139342_object13934:head# 2024-04-15T18:34:41.866 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e48a21:::benchmark_data_smithi159_139342_object44086:head# 2024-04-15T18:34:41.867 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e534b2:::benchmark_data_smithi159_139342_object38305:head# 2024-04-15T18:34:41.867 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e5b0a0:::benchmark_data_smithi159_139342_object578:head# 2024-04-15T18:34:41.868 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e611de:::benchmark_data_smithi159_139342_object19586:head# 2024-04-15T18:34:41.868 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6e89ab7:::benchmark_data_smithi159_139342_object15006:head# 2024-04-15T18:34:41.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ebc167:::benchmark_data_smithi159_139342_object17621:head# 2024-04-15T18:34:41.879 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ed4b11:::benchmark_data_smithi159_139342_object10606:head# 2024-04-15T18:34:41.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6eeb167:::benchmark_data_smithi159_139342_object18199:head# 2024-04-15T18:34:41.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6ef325e:::benchmark_data_smithi159_139342_object45559:head# 2024-04-15T18:34:41.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f1af9d:::benchmark_data_smithi159_139342_object43370:head# 2024-04-15T18:34:41.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f28f34:::benchmark_data_smithi159_139342_object42248:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f316e0:::benchmark_data_smithi159_139342_object12056:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f41ef2:::benchmark_data_smithi159_139342_object43782:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f50810:::benchmark_data_smithi159_139342_object966:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f5f86c:::benchmark_data_smithi159_139342_object28548:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f63ab9:::benchmark_data_smithi159_139342_object19273:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f8778f:::benchmark_data_smithi159_139342_object49085:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c6f9fa76:::benchmark_data_smithi159_139342_object52298:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70094a2:::benchmark_data_smithi159_139342_object25641:head# 2024-04-15T18:34:41.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7009d82:::benchmark_data_smithi159_139342_object28380:head# 2024-04-15T18:34:41.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c700bb5d:::benchmark_data_smithi159_139342_object33553:head# 2024-04-15T18:34:41.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c701707f:::benchmark_data_smithi159_139342_object45219:head# 2024-04-15T18:34:41.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c702aed9:::benchmark_data_smithi159_139342_object28279:head# 2024-04-15T18:34:41.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70333f7:::benchmark_data_smithi159_139342_object16010:head# 2024-04-15T18:34:41.885 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c704cee0:::benchmark_data_smithi159_139342_object7975:head# 2024-04-15T18:34:41.888 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c706f6f3:::benchmark_data_smithi159_139342_object14047:head# 2024-04-15T18:34:41.888 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7085aa1:::benchmark_data_smithi159_139342_object51429:head# 2024-04-15T18:34:41.898 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7090b99:::benchmark_data_smithi159_139342_object13653:head# 2024-04-15T18:34:41.898 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70971c6:::benchmark_data_smithi159_139342_object19333:head# 2024-04-15T18:34:41.898 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7097e1e:::benchmark_data_smithi159_139342_object26461:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70b0185:::benchmark_data_smithi159_139342_object47483:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70d2442:::benchmark_data_smithi159_139342_object36793:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70e0a25:::benchmark_data_smithi159_139342_object4421:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70e28ab:::benchmark_data_smithi159_139342_object46913:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c70f0200:::benchmark_data_smithi159_139342_object19342:head# 2024-04-15T18:34:41.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7119302:::benchmark_data_smithi159_139342_object2133:head# 2024-04-15T18:34:41.903 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71270ca:::benchmark_data_smithi159_139342_object41899:head# 2024-04-15T18:34:41.903 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c712eef7:::benchmark_data_smithi159_139342_object25530:head# 2024-04-15T18:34:41.907 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7138653:::benchmark_data_smithi159_139342_object12528:head# 2024-04-15T18:34:41.908 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7144719:::benchmark_data_smithi159_139342_object36077:head# 2024-04-15T18:34:41.908 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7153f40:::benchmark_data_smithi159_139342_object20439:head# 2024-04-15T18:34:41.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c715b867:::benchmark_data_smithi159_139342_object1709:head# 2024-04-15T18:34:41.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71734ed:::benchmark_data_smithi159_139342_object17863:head# 2024-04-15T18:34:41.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c718eb3a:::benchmark_data_smithi159_139342_object33962:head# 2024-04-15T18:34:41.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c719e3c8:::benchmark_data_smithi159_139342_object37917:head# 2024-04-15T18:34:41.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71d28ef:::benchmark_data_smithi159_139342_object144:head# 2024-04-15T18:34:41.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71d86b2:::benchmark_data_smithi159_139342_object27448:head# 2024-04-15T18:34:41.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71de358:::benchmark_data_smithi159_139342_object11109:head# 2024-04-15T18:34:41.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71e47ab:::benchmark_data_smithi159_139342_object4611:head# 2024-04-15T18:34:41.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c71e59dc:::benchmark_data_smithi159_139342_object27656:head# 2024-04-15T18:34:41.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c721e1a4:::benchmark_data_smithi159_139342_object49838:head# 2024-04-15T18:34:41.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7242099:::benchmark_data_smithi159_139342_object16356:head# 2024-04-15T18:34:41.914 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7244182:::benchmark_data_smithi159_139342_object42982:head# 2024-04-15T18:34:41.915 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72507b7:::benchmark_data_smithi159_139342_object10502:head# 2024-04-15T18:34:41.918 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c726f300:::benchmark_data_smithi159_139342_object52096:head# 2024-04-15T18:34:41.919 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c727636e:::benchmark_data_smithi159_139342_object41805:head# 2024-04-15T18:34:41.919 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c727abde:::benchmark_data_smithi159_139342_object39778:head# 2024-04-15T18:34:41.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7288b4c:::benchmark_data_smithi159_139342_object26342:head# 2024-04-15T18:34:41.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72a0369:::benchmark_data_smithi159_139342_object49386:head# 2024-04-15T18:34:41.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72abb32:::benchmark_data_smithi159_139342_object45368:head# 2024-04-15T18:34:41.921 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72af5f2:::benchmark_data_smithi159_139342_object35929:head# 2024-04-15T18:34:41.921 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72b95a4:::benchmark_data_smithi159_139342_object39415:head# 2024-04-15T18:34:41.922 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72c0d1f:::benchmark_data_smithi159_139342_object54748:head# 2024-04-15T18:34:41.922 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72dd1a7:::benchmark_data_smithi159_139342_object315:head# 2024-04-15T18:34:41.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72e781e:::benchmark_data_smithi159_139342_object53306:head# 2024-04-15T18:34:41.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72f19b3:::benchmark_data_smithi159_139342_object42170:head# 2024-04-15T18:34:41.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72f2de2:::benchmark_data_smithi159_139342_object52401:head# 2024-04-15T18:34:41.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72f829b:::benchmark_data_smithi159_139342_object18747:head# 2024-04-15T18:34:41.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c72fc13f:::benchmark_data_smithi159_139342_object56026:head# 2024-04-15T18:34:41.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7310be9:::benchmark_data_smithi159_139342_object25962:head# 2024-04-15T18:34:41.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7356422:::benchmark_data_smithi159_139342_object911:head# 2024-04-15T18:34:41.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c735c80a:::benchmark_data_smithi159_139342_object30541:head# 2024-04-15T18:34:41.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7361f0e:::benchmark_data_smithi159_139342_object5138:head# 2024-04-15T18:34:41.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7372faa:::benchmark_data_smithi159_139342_object37755:head# 2024-04-15T18:34:41.928 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7378b70:::benchmark_data_smithi159_139342_object47618:head# 2024-04-15T18:34:41.932 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7379190:::benchmark_data_smithi159_139342_object52490:head# 2024-04-15T18:34:41.932 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c737af83:::benchmark_data_smithi159_139342_object7220:head# 2024-04-15T18:34:41.932 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c73a66c6:::benchmark_data_smithi159_139342_object12183:head# 2024-04-15T18:34:41.932 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c73a927e:::benchmark_data_smithi159_139342_object12040:head# 2024-04-15T18:34:41.933 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c73ec18b:::benchmark_data_smithi159_139342_object1393:head# 2024-04-15T18:34:41.934 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7406b92:::benchmark_data_smithi159_139342_object11828:head# 2024-04-15T18:34:41.934 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c741af35:::benchmark_data_smithi159_139342_object28615:head# 2024-04-15T18:34:41.938 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c741b34a:::benchmark_data_smithi159_139342_object17725:head# 2024-04-15T18:34:41.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7445362:::benchmark_data_smithi159_139342_object43515:head# 2024-04-15T18:34:41.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7448317:::benchmark_data_smithi159_139342_object33897:head# 2024-04-15T18:34:41.941 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c74c8f2d:::benchmark_data_smithi159_139342_object1348:head# 2024-04-15T18:34:41.941 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c74f27cc:::benchmark_data_smithi159_139342_object6592:head# 2024-04-15T18:34:41.942 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c74f7f2f:::benchmark_data_smithi159_139342_object16491:head# 2024-04-15T18:34:41.942 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c750273f:::benchmark_data_smithi159_139342_object476:head# 2024-04-15T18:34:41.942 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7553e4a:::benchmark_data_smithi159_139342_object26339:head# 2024-04-15T18:34:41.944 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75d32c8:::benchmark_data_smithi159_139342_object1914:head# 2024-04-15T18:34:41.945 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75d5b60:::benchmark_data_smithi159_139342_object17300:head# 2024-04-15T18:34:41.946 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75dc77d:::benchmark_data_smithi159_139342_object38110:head# 2024-04-15T18:34:41.947 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75f0913:::benchmark_data_smithi159_139342_object33631:head# 2024-04-15T18:34:41.947 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75f0cdc:::benchmark_data_smithi159_139342_object39917:head# 2024-04-15T18:34:41.948 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75f2c16:::benchmark_data_smithi159_139342_object44779:head# 2024-04-15T18:34:41.948 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75f37f0:::benchmark_data_smithi159_139342_object27943:head# 2024-04-15T18:34:41.948 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c75fc6fc:::benchmark_data_smithi159_139342_object7113:head# 2024-04-15T18:34:41.949 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c763ee95:::benchmark_data_smithi159_139342_object39448:head# 2024-04-15T18:34:41.952 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c767edff:::benchmark_data_smithi159_139342_object44981:head# 2024-04-15T18:34:41.959 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76963ab:::benchmark_data_smithi159_139342_object52472:head# 2024-04-15T18:34:41.959 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76af302:::benchmark_data_smithi159_139342_object46835:head# 2024-04-15T18:34:41.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76b16ac:::benchmark_data_smithi159_139342_object39829:head# 2024-04-15T18:34:41.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76b5d71:::benchmark_data_smithi159_139342_object5424:head# 2024-04-15T18:34:41.960 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76b86c8:::benchmark_data_smithi159_139342_object34477:head# 2024-04-15T18:34:41.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76c4767:::benchmark_data_smithi159_139342_object11120:head# 2024-04-15T18:34:41.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76d48a0:::benchmark_data_smithi159_139342_object52275:head# 2024-04-15T18:34:41.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c76ed94e:::benchmark_data_smithi159_139342_object17816:head# 2024-04-15T18:34:41.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c771de2c:::benchmark_data_smithi159_139342_object2628:head# 2024-04-15T18:34:41.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c773223a:::benchmark_data_smithi159_139342_object33528:head# 2024-04-15T18:34:41.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77344da:::benchmark_data_smithi159_139342_object9746:head# 2024-04-15T18:34:41.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7742d91:::benchmark_data_smithi159_139342_object9179:head# 2024-04-15T18:34:41.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77459da:::benchmark_data_smithi159_139342_object38929:head# 2024-04-15T18:34:41.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c776366f:::benchmark_data_smithi159_139342_object45585:head# 2024-04-15T18:34:41.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c778bd82:::benchmark_data_smithi159_139342_object27214:head# 2024-04-15T18:34:41.970 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77ba3de:::benchmark_data_smithi159_139342_object51186:head# 2024-04-15T18:34:41.971 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77ba735:::benchmark_data_smithi159_139342_object45194:head# 2024-04-15T18:34:41.971 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77bea34:::benchmark_data_smithi159_139342_object46741:head# 2024-04-15T18:34:41.971 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77becc6:::benchmark_data_smithi159_139342_object42074:head# 2024-04-15T18:34:41.972 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77edc6f:::benchmark_data_smithi159_139342_object37001:head# 2024-04-15T18:34:41.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77ee614:::benchmark_data_smithi159_139342_object47294:head# 2024-04-15T18:34:41.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c77fb662:::benchmark_data_smithi159_139342_object33072:head# 2024-04-15T18:34:41.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7800004:::benchmark_data_smithi159_139342_object55608:head# 2024-04-15T18:34:41.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7833080:::benchmark_data_smithi159_139342_object46777:head# 2024-04-15T18:34:41.977 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c784aab2:::benchmark_data_smithi159_139342_object42659:head# 2024-04-15T18:34:41.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c784ef11:::benchmark_data_smithi159_139342_object4417:head# 2024-04-15T18:34:41.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7854693:::benchmark_data_smithi159_139342_object3642:head# 2024-04-15T18:34:41.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c787588d:::benchmark_data_smithi159_139342_object6383:head# 2024-04-15T18:34:41.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7879c7a:::benchmark_data_smithi159_139342_object3862:head# 2024-04-15T18:34:41.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7894cd4:::benchmark_data_smithi159_139342_object4973:head# 2024-04-15T18:34:41.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78a27f7:::benchmark_data_smithi159_139342_object49719:head# 2024-04-15T18:34:41.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78a9dc9:::benchmark_data_smithi159_139342_object4493:head# 2024-04-15T18:34:41.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78ac4b2:::benchmark_data_smithi159_139342_object35814:head# 2024-04-15T18:34:41.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78ae233:::benchmark_data_smithi159_139342_object6327:head# 2024-04-15T18:34:41.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78b9f7e:::benchmark_data_smithi159_139342_object49698:head# 2024-04-15T18:34:41.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78e4fdd:::benchmark_data_smithi159_139342_object29550:head# 2024-04-15T18:34:41.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c78f3e4f:::benchmark_data_smithi159_139342_object16493:head# 2024-04-15T18:34:41.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7901337:::benchmark_data_smithi159_139342_object41200:head# 2024-04-15T18:34:41.986 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7901490:::benchmark_data_smithi159_139342_object35827:head# 2024-04-15T18:34:41.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c791935e:::benchmark_data_smithi159_139342_object22771:head# 2024-04-15T18:34:41.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c791db61:::benchmark_data_smithi159_139342_object48807:head# 2024-04-15T18:34:41.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c794fa7d:::benchmark_data_smithi159_139342_object55269:head# 2024-04-15T18:34:41.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c796b1e2:::benchmark_data_smithi159_139342_object14219:head# 2024-04-15T18:34:41.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7974315:::benchmark_data_smithi159_139342_object27701:head# 2024-04-15T18:34:41.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c797b681:::benchmark_data_smithi159_139342_object54877:head# 2024-04-15T18:34:41.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c798318a:::benchmark_data_smithi159_139342_object6052:head# 2024-04-15T18:34:41.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c798cd06:::benchmark_data_smithi159_139342_object13577:head# 2024-04-15T18:34:41.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79919de:::benchmark_data_smithi159_139342_object5086:head# 2024-04-15T18:34:41.995 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79af38f:::benchmark_data_smithi159_139342_object16476:head# 2024-04-15T18:34:41.996 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79bd905:::benchmark_data_smithi159_139342_object31986:head# 2024-04-15T18:34:42.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79c1fd5:::benchmark_data_smithi159_139342_object35105:head# 2024-04-15T18:34:42.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79cca62:::benchmark_data_smithi159_139342_object45460:head# 2024-04-15T18:34:42.008 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79d6fc9:::benchmark_data_smithi159_139342_object20799:head# 2024-04-15T18:34:42.008 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79db2e5:::benchmark_data_smithi159_139342_object49820:head# 2024-04-15T18:34:42.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79e4f46:::benchmark_data_smithi159_139342_object24536:head# 2024-04-15T18:34:42.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79e83f8:::benchmark_data_smithi159_139342_object36444:head# 2024-04-15T18:34:42.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79e8fac:::benchmark_data_smithi159_139342_object18860:head# 2024-04-15T18:34:42.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c79f85a8:::benchmark_data_smithi159_139342_object9732:head# 2024-04-15T18:34:42.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a156e5:::benchmark_data_smithi159_139342_object15870:head# 2024-04-15T18:34:42.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a26be8:::benchmark_data_smithi159_139342_object30976:head# 2024-04-15T18:34:42.011 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a39ba9:::benchmark_data_smithi159_139342_object49465:head# 2024-04-15T18:34:42.011 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a535cb:::benchmark_data_smithi159_139342_object5930:head# 2024-04-15T18:34:42.011 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a65b2a:::benchmark_data_smithi159_139342_object10030:head# 2024-04-15T18:34:42.012 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a75245:::benchmark_data_smithi159_139342_object37337:head# 2024-04-15T18:34:42.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a791c5:::benchmark_data_smithi159_139342_object18275:head# 2024-04-15T18:34:42.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a874a9:::benchmark_data_smithi159_139342_object11562:head# 2024-04-15T18:34:42.013 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7a87f6a:::benchmark_data_smithi159_139342_object21495:head# 2024-04-15T18:34:42.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7aa8fe9:::benchmark_data_smithi159_139342_object42611:head# 2024-04-15T18:34:42.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7ab26a0:::benchmark_data_smithi159_139342_object26862:head# 2024-04-15T18:34:42.018 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7ad1153:::benchmark_data_smithi159_139342_object28432:head# 2024-04-15T18:34:42.018 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b0c594:::benchmark_data_smithi159_139342_object9308:head# 2024-04-15T18:34:42.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b121e0:::benchmark_data_smithi159_139342_object55293:head# 2024-04-15T18:34:42.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b1cfab:::benchmark_data_smithi159_139342_object34116:head# 2024-04-15T18:34:42.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b1d761:::benchmark_data_smithi159_139342_object17291:head# 2024-04-15T18:34:42.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b2bbf5:::benchmark_data_smithi159_139342_object13909:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b3ff2d:::benchmark_data_smithi159_139342_object5468:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b4702b:::benchmark_data_smithi159_139342_object44205:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b47636:::benchmark_data_smithi159_139342_object25154:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b4e288:::benchmark_data_smithi159_139342_object9011:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b59813:::benchmark_data_smithi159_139342_object12255:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b6879a:::benchmark_data_smithi159_139342_object39858:head# 2024-04-15T18:34:42.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b698f0:::benchmark_data_smithi159_139342_object3543:head# 2024-04-15T18:34:42.024 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b7394c:::benchmark_data_smithi159_139342_object55684:head# 2024-04-15T18:34:42.024 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b9ba63:::benchmark_data_smithi159_139342_object17567:head# 2024-04-15T18:34:42.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7b9e12f:::benchmark_data_smithi159_139342_object15993:head# 2024-04-15T18:34:42.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7bbc49b:::benchmark_data_smithi159_139342_object44024:head# 2024-04-15T18:34:42.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7bbcf13:::benchmark_data_smithi159_139342_object50425:head# 2024-04-15T18:34:42.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7bc2114:::benchmark_data_smithi159_139342_object29464:head# 2024-04-15T18:34:42.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7bc425c:::benchmark_data_smithi159_139342_object11234:head# 2024-04-15T18:34:42.026 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7bdfcca:::benchmark_data_smithi159_139342_object27206:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c1834f:::benchmark_data_smithi159_139342_object10094:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c18dab:::benchmark_data_smithi159_139342_object15334:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c2587b:::benchmark_data_smithi159_139342_object13659:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c2684c:::benchmark_data_smithi159_139342_object24677:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c2e273:::benchmark_data_smithi159_139342_object49500:head# 2024-04-15T18:34:42.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c40860:::benchmark_data_smithi159_139342_object52696:head# 2024-04-15T18:34:42.030 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c469ca:::benchmark_data_smithi159_139342_object52876:head# 2024-04-15T18:34:42.030 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c5600a:::benchmark_data_smithi159_139342_object26739:head# 2024-04-15T18:34:42.031 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c74189:::benchmark_data_smithi159_139342_object54724:head# 2024-04-15T18:34:42.031 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c7d8dd:::benchmark_data_smithi159_139342_object55550:head# 2024-04-15T18:34:42.031 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c8d9f0:::benchmark_data_smithi159_139342_object49849:head# 2024-04-15T18:34:42.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7c9ca2f:::benchmark_data_smithi159_139342_object6819:head# 2024-04-15T18:34:42.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7cabcba:::benchmark_data_smithi159_139342_object23265:head# 2024-04-15T18:34:42.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7cd616b:::benchmark_data_smithi159_139342_object52641:head# 2024-04-15T18:34:42.034 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7cd6d80:::benchmark_data_smithi159_139342_object11297:head# 2024-04-15T18:34:42.034 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7cd7da0:::benchmark_data_smithi159_139342_object15676:head# 2024-04-15T18:34:42.034 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d101da:::benchmark_data_smithi159_139342_object5267:head# 2024-04-15T18:34:42.035 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d2dacc:::benchmark_data_smithi159_139342_object56764:head# 2024-04-15T18:34:42.035 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d44994:::benchmark_data_smithi159_139342_object46044:head# 2024-04-15T18:34:42.038 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d44e05:::benchmark_data_smithi159_139342_object48256:head# 2024-04-15T18:34:42.038 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d599a4:::benchmark_data_smithi159_139342_object28876:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d75483:::benchmark_data_smithi159_139342_object13681:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d7c425:::benchmark_data_smithi159_139342_object35138:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7d8a4b3:::benchmark_data_smithi159_139342_object28857:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7da8b1a:::benchmark_data_smithi159_139342_object7753:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7da9fe8:::benchmark_data_smithi159_139342_object3787:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7db47e8:::benchmark_data_smithi159_139342_object39968:head# 2024-04-15T18:34:42.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7db5fb8:::benchmark_data_smithi159_139342_object56115:head# 2024-04-15T18:34:42.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7dbf6f7:::benchmark_data_smithi159_139342_object2257:head# 2024-04-15T18:34:42.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7dd001a:::benchmark_data_smithi159_139342_object43784:head# 2024-04-15T18:34:42.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7dd35f7:::benchmark_data_smithi159_139342_object37765:head# 2024-04-15T18:34:42.042 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7dde8e4:::benchmark_data_smithi159_139342_object1051:head# 2024-04-15T18:34:42.043 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7ddefc4:::benchmark_data_smithi159_139342_object19888:head# 2024-04-15T18:34:42.043 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7deb98f:::benchmark_data_smithi159_139342_object3298:head# 2024-04-15T18:34:42.043 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7def99a:::benchmark_data_smithi159_139342_object47746:head# 2024-04-15T18:34:42.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e12669:::benchmark_data_smithi159_139342_object3546:head# 2024-04-15T18:34:42.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e27a47:::benchmark_data_smithi159_139342_object29538:head# 2024-04-15T18:34:42.045 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e5a0ee:::benchmark_data_smithi159_139342_object33120:head# 2024-04-15T18:34:42.045 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e5b90f:::benchmark_data_smithi159_139342_object53991:head# 2024-04-15T18:34:42.046 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e7a04a:::benchmark_data_smithi159_139342_object55039:head# 2024-04-15T18:34:42.046 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e8492f:::benchmark_data_smithi159_139342_object49434:head# 2024-04-15T18:34:42.046 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e8da70:::benchmark_data_smithi159_139342_object54846:head# 2024-04-15T18:34:42.047 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7e97bde:::benchmark_data_smithi159_139342_object37681:head# 2024-04-15T18:34:42.047 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7ebf3a2:::benchmark_data_smithi159_139342_object9944:head# 2024-04-15T18:34:42.048 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7edd2a7:::benchmark_data_smithi159_139342_object5129:head# 2024-04-15T18:34:42.048 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7ee40ff:::benchmark_data_smithi159_139342_object9629:head# 2024-04-15T18:34:42.052 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f352f4:::benchmark_data_smithi159_139342_object29793:head# 2024-04-15T18:34:42.052 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f43b35:::benchmark_data_smithi159_139342_object9334:head# 2024-04-15T18:34:42.053 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f4a03e:::benchmark_data_smithi159_139342_object2877:head# 2024-04-15T18:34:42.053 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f5a2fb:::benchmark_data_smithi159_139342_object17434:head# 2024-04-15T18:34:42.053 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f68b47:::benchmark_data_smithi159_139342_object49406:head# 2024-04-15T18:34:42.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f70974:::benchmark_data_smithi159_139342_object48990:head# 2024-04-15T18:34:42.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f80058:::benchmark_data_smithi159_139342_object40549:head# 2024-04-15T18:34:42.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f815e6:::benchmark_data_smithi159_139342_object2578:head# 2024-04-15T18:34:42.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f862c2:::benchmark_data_smithi159_139342_object11891:head# 2024-04-15T18:34:42.056 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f96ee8:::benchmark_data_smithi159_139342_object16111:head# 2024-04-15T18:34:42.056 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7f9d982:::benchmark_data_smithi159_139342_object18245:head# 2024-04-15T18:34:42.057 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7fb86da:::benchmark_data_smithi159_139342_object12277:head# 2024-04-15T18:34:42.057 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7fe3ad1:::benchmark_data_smithi159_139342_object16585:head# 2024-04-15T18:34:42.058 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c7fe73d0:::benchmark_data_smithi159_139342_object25072:head# 2024-04-15T18:34:42.058 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80041e3:::benchmark_data_smithi159_139342_object38505:head# 2024-04-15T18:34:42.066 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c800b1ae:::benchmark_data_smithi159_139342_object13254:head# 2024-04-15T18:34:42.069 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c802c427:::benchmark_data_smithi159_139342_object19019:head# 2024-04-15T18:34:42.070 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8061f55:::benchmark_data_smithi159_139342_object3209:head# 2024-04-15T18:34:42.070 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c808474f:::benchmark_data_smithi159_139342_object52326:head# 2024-04-15T18:34:42.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c808a203:::benchmark_data_smithi159_139342_object30103:head# 2024-04-15T18:34:42.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c809bc00:::benchmark_data_smithi159_139342_object43602:head# 2024-04-15T18:34:42.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80ab0c2:::benchmark_data_smithi159_139342_object44248:head# 2024-04-15T18:34:42.072 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80b6b44:::benchmark_data_smithi159_139342_object48790:head# 2024-04-15T18:34:42.072 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80c052b:::benchmark_data_smithi159_139342_object26398:head# 2024-04-15T18:34:42.073 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80c5ab3:::benchmark_data_smithi159_139342_object11854:head# 2024-04-15T18:34:42.073 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c80c7ae1:::benchmark_data_smithi159_139342_object18692:head# 2024-04-15T18:34:42.074 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c810e20f:::benchmark_data_smithi159_139342_object41993:head# 2024-04-15T18:34:42.074 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c810edf3:::benchmark_data_smithi159_139342_object36521:head# 2024-04-15T18:34:42.075 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c811b4bf:::benchmark_data_smithi159_139342_object5047:head# 2024-04-15T18:34:42.075 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8156e91:::benchmark_data_smithi159_139342_object12938:head# 2024-04-15T18:34:42.076 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c81695e8:::benchmark_data_smithi159_139342_object6000:head# 2024-04-15T18:34:42.079 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8174ca7:::benchmark_data_smithi159_139342_object46531:head# 2024-04-15T18:34:42.079 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8191171:::benchmark_data_smithi159_139342_object14390:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8196d81:::benchmark_data_smithi159_139342_object27889:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8198ccf:::benchmark_data_smithi159_139342_object37398:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c81b1784:::benchmark_data_smithi159_139342_object24868:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c81b9aa9:::benchmark_data_smithi159_139342_object52361:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c81d9bef:::benchmark_data_smithi159_139342_object53621:head# 2024-04-15T18:34:42.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c81e94c6:::benchmark_data_smithi159_139342_object56741:head# 2024-04-15T18:34:42.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c821b0bd:::benchmark_data_smithi159_139342_object39465:head# 2024-04-15T18:34:42.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8227d27:::benchmark_data_smithi159_139342_object56753:head# 2024-04-15T18:34:42.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82314cf:::benchmark_data_smithi159_139342_object42278:head# 2024-04-15T18:34:42.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c824033a:::benchmark_data_smithi159_139342_object52011:head# 2024-04-15T18:34:42.081 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8272f61:::benchmark_data_smithi159_139342_object22395:head# 2024-04-15T18:34:42.082 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8286d65:::benchmark_data_smithi159_139342_object44666:head# 2024-04-15T18:34:42.082 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8286e5c:::benchmark_data_smithi159_139342_object31934:head# 2024-04-15T18:34:42.159 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8291987:::benchmark_data_smithi159_139342_object37745:head# 2024-04-15T18:34:42.160 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c829ec24:::benchmark_data_smithi159_139342_object48921:head# 2024-04-15T18:34:42.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82aa56b:::benchmark_data_smithi159_139342_object49813:head# 2024-04-15T18:34:42.161 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82d15a8:::benchmark_data_smithi159_139342_object34869:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82d1b3c:::benchmark_data_smithi159_139342_object21374:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82d22bc:::benchmark_data_smithi159_139342_object36732:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82d6471:::benchmark_data_smithi159_139342_object50355:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c82f3cf5:::benchmark_data_smithi159_139342_object40990:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8306515:::benchmark_data_smithi159_139342_object16794:head# 2024-04-15T18:34:42.164 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c830ecbe:::benchmark_data_smithi159_139342_object45005:head# 2024-04-15T18:34:42.165 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8310eec:::benchmark_data_smithi159_139342_object33926:head# 2024-04-15T18:34:42.165 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8328571:::benchmark_data_smithi159_139342_object7628:head# 2024-04-15T18:34:42.166 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8329261:::benchmark_data_smithi159_139342_object1078:head# 2024-04-15T18:34:42.166 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c832a2b0:::benchmark_data_smithi159_139342_object41348:head# 2024-04-15T18:34:42.166 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c832bf40:::benchmark_data_smithi159_139342_object8075:head# 2024-04-15T18:34:42.167 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8330ee8:::benchmark_data_smithi159_139342_object20909:head# 2024-04-15T18:34:42.168 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c833edb7:::benchmark_data_smithi159_139342_object56453:head# 2024-04-15T18:34:42.168 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c835efa2:::benchmark_data_smithi159_139342_object1762:head# 2024-04-15T18:34:42.169 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83602c4:::benchmark_data_smithi159_139342_object32002:head# 2024-04-15T18:34:42.169 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83673e6:::benchmark_data_smithi159_139342_object44405:head# 2024-04-15T18:34:42.170 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c837e841:::benchmark_data_smithi159_139342_object29593:head# 2024-04-15T18:34:42.170 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8395e8f:::benchmark_data_smithi159_139342_object29451:head# 2024-04-15T18:34:42.178 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83968f3:::benchmark_data_smithi159_139342_object45127:head# 2024-04-15T18:34:42.178 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83ad3b6:::benchmark_data_smithi159_139342_object37503:head# 2024-04-15T18:34:42.178 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83ba34d:::benchmark_data_smithi159_139342_object21118:head# 2024-04-15T18:34:42.178 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83da119:::benchmark_data_smithi159_139342_object31768:head# 2024-04-15T18:34:42.178 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83e299b:::benchmark_data_smithi159_139342_object43678:head# 2024-04-15T18:34:42.180 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83e887a:::benchmark_data_smithi159_139342_object22160:head# 2024-04-15T18:34:42.180 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83f1234:::benchmark_data_smithi159_139342_object36693:head# 2024-04-15T18:34:42.181 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c83fe8e3:::benchmark_data_smithi159_139342_object57141:head# 2024-04-15T18:34:42.181 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8429446:::benchmark_data_smithi159_139342_object1956:head# 2024-04-15T18:34:42.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8439644:::benchmark_data_smithi159_139342_object42455:head# 2024-04-15T18:34:42.182 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c843b424:::benchmark_data_smithi159_139342_object49997:head# 2024-04-15T18:34:42.185 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c843eeb1:::benchmark_data_smithi159_139342_object52640:head# 2024-04-15T18:34:42.185 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8462146:::benchmark_data_smithi159_139342_object23930:head# 2024-04-15T18:34:42.185 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c847c661:::benchmark_data_smithi159_139342_object231:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c84809be:::benchmark_data_smithi159_139342_object40656:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8485963:::benchmark_data_smithi159_139342_object14005:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c84872d6:::benchmark_data_smithi159_139342_object52049:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c849c013:::benchmark_data_smithi159_139342_object12919:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c850927d:::benchmark_data_smithi159_139342_object49876:head# 2024-04-15T18:34:42.186 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c850b70f:::benchmark_data_smithi159_139342_object55672:head# 2024-04-15T18:34:42.187 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85228d8:::benchmark_data_smithi159_139342_object53420:head# 2024-04-15T18:34:42.187 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8537261:::benchmark_data_smithi159_139342_object20432:head# 2024-04-15T18:34:42.187 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c854202e:::benchmark_data_smithi159_139342_object55239:head# 2024-04-15T18:34:42.188 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c854a357:::benchmark_data_smithi159_139342_object13312:head# 2024-04-15T18:34:42.188 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c854d2e1:::benchmark_data_smithi159_139342_object24955:head# 2024-04-15T18:34:42.189 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c856207f:::benchmark_data_smithi159_139342_object47535:head# 2024-04-15T18:34:42.189 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85669d3:::benchmark_data_smithi159_139342_object10912:head# 2024-04-15T18:34:42.190 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c856b5ad:::benchmark_data_smithi159_139342_object20773:head# 2024-04-15T18:34:42.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8583b9a:::benchmark_data_smithi159_139342_object17171:head# 2024-04-15T18:34:42.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c858566a:::benchmark_data_smithi159_139342_object14764:head# 2024-04-15T18:34:42.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8598263:::benchmark_data_smithi159_139342_object21527:head# 2024-04-15T18:34:42.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85a81e5:::benchmark_data_smithi159_139342_object34822:head# 2024-04-15T18:34:42.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85ac9c5:::benchmark_data_smithi159_139342_object51542:head# 2024-04-15T18:34:42.201 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85c145d:::benchmark_data_smithi159_139342_object47210:head# 2024-04-15T18:34:42.201 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85cb1aa:::benchmark_data_smithi159_139342_object54787:head# 2024-04-15T18:34:42.202 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85dffcb:::benchmark_data_smithi159_139342_object26619:head# 2024-04-15T18:34:42.203 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85e0637:::benchmark_data_smithi159_139342_object32509:head# 2024-04-15T18:34:42.205 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c85e6c0c:::benchmark_data_smithi159_139342_object20326:head# 2024-04-15T18:34:42.206 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8616e1c:::benchmark_data_smithi159_139342_object25225:head# 2024-04-15T18:34:42.206 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c864a796:::benchmark_data_smithi159_139342_object12249:head# 2024-04-15T18:34:42.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c865c8ea:::benchmark_data_smithi159_139342_object9646:head# 2024-04-15T18:34:42.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86640b7:::benchmark_data_smithi159_139342_object4626:head# 2024-04-15T18:34:42.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c866c39a:::benchmark_data_smithi159_139342_object19087:head# 2024-04-15T18:34:42.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c866fb16:::benchmark_data_smithi159_139342_object54910:head# 2024-04-15T18:34:42.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8685870:::benchmark_data_smithi159_139342_object34143:head# 2024-04-15T18:34:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8690049:::benchmark_data_smithi159_139342_object43301:head# 2024-04-15T18:34:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86a69b9:::benchmark_data_smithi159_139342_object13059:head# 2024-04-15T18:34:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86c2bca:::benchmark_data_smithi159_139342_object34432:head# 2024-04-15T18:34:42.217 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86db00d:::benchmark_data_smithi159_139342_object31661:head# 2024-04-15T18:34:42.217 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86dd83e:::benchmark_data_smithi159_139342_object54003:head# 2024-04-15T18:34:42.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c86e806a:::benchmark_data_smithi159_139342_object41686:head# 2024-04-15T18:34:42.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8704daa:::benchmark_data_smithi159_139342_object4047:head# 2024-04-15T18:34:42.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8708f13:::benchmark_data_smithi159_139342_object8901:head# 2024-04-15T18:34:42.220 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c871844f:::benchmark_data_smithi159_139342_object43330:head# 2024-04-15T18:34:42.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8728f29:::benchmark_data_smithi159_139342_object42679:head# 2024-04-15T18:34:42.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8745beb:::benchmark_data_smithi159_139342_object17437:head# 2024-04-15T18:34:42.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c875c5ac:::benchmark_data_smithi159_139342_object39199:head# 2024-04-15T18:34:42.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c87602dd:::benchmark_data_smithi159_139342_object56658:head# 2024-04-15T18:34:42.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8773565:::benchmark_data_smithi159_139342_object14198:head# 2024-04-15T18:34:42.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c877d094:::benchmark_data_smithi159_139342_object37665:head# 2024-04-15T18:34:42.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c87815f5:::benchmark_data_smithi159_139342_object20510:head# 2024-04-15T18:34:42.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c878ae90:::benchmark_data_smithi159_139342_object24876:head# 2024-04-15T18:34:42.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c879fa78:::benchmark_data_smithi159_139342_object6278:head# 2024-04-15T18:34:42.225 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c87e3550:::benchmark_data_smithi159_139342_object36862:head# 2024-04-15T18:34:42.225 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c87eeffb:::benchmark_data_smithi159_139342_object31574:head# 2024-04-15T18:34:42.226 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c87f1d91:::benchmark_data_smithi159_139342_object42870:head# 2024-04-15T18:34:42.227 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8837e74:::benchmark_data_smithi159_139342_object19402:head# 2024-04-15T18:34:42.227 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c883808b:::benchmark_data_smithi159_139342_object22567:head# 2024-04-15T18:34:42.228 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8887b6f:::benchmark_data_smithi159_139342_object26904:head# 2024-04-15T18:34:42.228 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c88bbdaf:::benchmark_data_smithi159_139342_object16616:head# 2024-04-15T18:34:42.228 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c88d79ef:::benchmark_data_smithi159_139342_object16108:head# 2024-04-15T18:34:42.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c88ee53d:::benchmark_data_smithi159_139342_object1829:head# 2024-04-15T18:34:42.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8909590:::benchmark_data_smithi159_139342_object44684:head# 2024-04-15T18:34:42.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c891919d:::benchmark_data_smithi159_139342_object6824:head# 2024-04-15T18:34:42.231 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c893009f:::benchmark_data_smithi159_139342_object27024:head# 2024-04-15T18:34:42.231 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8943453:::benchmark_data_smithi159_139342_object15563:head# 2024-04-15T18:34:42.231 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c894e0ef:::benchmark_data_smithi159_139342_object47101:head# 2024-04-15T18:34:42.232 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8960115:::benchmark_data_smithi159_139342_object16775:head# 2024-04-15T18:34:42.232 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c896c41a:::benchmark_data_smithi159_139342_object7350:head# 2024-04-15T18:34:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8972794:::benchmark_data_smithi159_139342_object40953:head# 2024-04-15T18:34:42.233 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c898abee:::benchmark_data_smithi159_139342_object3532:head# 2024-04-15T18:34:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8993d02:::benchmark_data_smithi159_139342_object21594:head# 2024-04-15T18:34:42.234 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c89be36d:::benchmark_data_smithi159_139342_object2951:head# 2024-04-15T18:34:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c89d3582:::benchmark_data_smithi159_139342_object43045:head# 2024-04-15T18:34:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c89fa7ca:::benchmark_data_smithi159_139342_object7353:head# 2024-04-15T18:34:42.235 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a012dd:::benchmark_data_smithi159_139342_object56073:head# 2024-04-15T18:34:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a0dfd2:::benchmark_data_smithi159_139342_object21646:head# 2024-04-15T18:34:42.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a14307:::benchmark_data_smithi159_139342_object24904:head# 2024-04-15T18:34:42.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a26006:::benchmark_data_smithi159_139342_object7856:head# 2024-04-15T18:34:42.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a2b627:::benchmark_data_smithi159_139342_object55482:head# 2024-04-15T18:34:42.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a2c5bd:::benchmark_data_smithi159_139342_object20575:head# 2024-04-15T18:34:42.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a549ac:::benchmark_data_smithi159_139342_object48777:head# 2024-04-15T18:34:42.239 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8a77084:::benchmark_data_smithi159_139342_object10481:head# 2024-04-15T18:34:42.239 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8aa89af:::benchmark_data_smithi159_139342_object22443:head# 2024-04-15T18:34:42.239 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ab2c4f:::benchmark_data_smithi159_139342_object19094:head# 2024-04-15T18:34:42.240 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ab3d4a:::benchmark_data_smithi159_139342_object46583:head# 2024-04-15T18:34:42.240 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8abcb23:::benchmark_data_smithi159_139342_object22493:head# 2024-04-15T18:34:42.241 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ac1a19:::benchmark_data_smithi159_139342_object6818:head# 2024-04-15T18:34:42.244 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ace785:::benchmark_data_smithi159_139342_object27535:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8afcf3e:::benchmark_data_smithi159_139342_object22206:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8afe682:::benchmark_data_smithi159_139342_object7861:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b20a98:::benchmark_data_smithi159_139342_object23204:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b4c15a:::benchmark_data_smithi159_139342_object29804:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b52b27:::benchmark_data_smithi159_139342_object14518:head# 2024-04-15T18:34:42.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b77a93:::benchmark_data_smithi159_139342_object21710:head# 2024-04-15T18:34:42.246 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b77b2f:::benchmark_data_smithi159_139342_object6002:head# 2024-04-15T18:34:42.246 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8b9c192:::benchmark_data_smithi159_139342_object54222:head# 2024-04-15T18:34:42.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ba3755:::benchmark_data_smithi159_139342_object25692:head# 2024-04-15T18:34:42.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8bbe210:::benchmark_data_smithi159_139342_object41261:head# 2024-04-15T18:34:42.248 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8bcd6bb:::benchmark_data_smithi159_139342_object39481:head# 2024-04-15T18:34:42.248 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8bd396f:::benchmark_data_smithi159_139342_object15875:head# 2024-04-15T18:34:42.249 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8bd7787:::benchmark_data_smithi159_139342_object15079:head# 2024-04-15T18:34:42.249 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8be223e:::benchmark_data_smithi159_139342_object10914:head# 2024-04-15T18:34:42.250 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c046ee:::benchmark_data_smithi159_139342_object33992:head# 2024-04-15T18:34:42.250 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c067a7:::benchmark_data_smithi159_139342_object17688:head# 2024-04-15T18:34:42.251 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c092da:::benchmark_data_smithi159_139342_object29781:head# 2024-04-15T18:34:42.251 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c0a7a1:::benchmark_data_smithi159_139342_object5464:head# 2024-04-15T18:34:42.251 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c1388d:::benchmark_data_smithi159_139342_object56540:head# 2024-04-15T18:34:42.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c14b0e:::benchmark_data_smithi159_139342_object31303:head# 2024-04-15T18:34:42.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c17175:::benchmark_data_smithi159_139342_object20926:head# 2024-04-15T18:34:42.253 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c4ad4c:::benchmark_data_smithi159_139342_object35741:head# 2024-04-15T18:34:42.253 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c5ed18:::benchmark_data_smithi159_139342_object17057:head# 2024-04-15T18:34:42.254 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c60cf2:::benchmark_data_smithi159_139342_object33978:head# 2024-04-15T18:34:42.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c62b76:::benchmark_data_smithi159_139342_object8257:head# 2024-04-15T18:34:42.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c808c3:::benchmark_data_smithi159_139342_object22874:head# 2024-04-15T18:34:42.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c87fe2:::benchmark_data_smithi159_139342_object48232:head# 2024-04-15T18:34:42.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8c88186:::benchmark_data_smithi159_139342_object43262:head# 2024-04-15T18:34:42.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8cab0b5:::benchmark_data_smithi159_139342_object38992:head# 2024-04-15T18:34:42.256 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8cb307a:::benchmark_data_smithi159_139342_object38125:head# 2024-04-15T18:34:42.257 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ccfa43:::benchmark_data_smithi159_139342_object22074:head# 2024-04-15T18:34:42.257 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8cd3e7b:::benchmark_data_smithi159_139342_object48248:head# 2024-04-15T18:34:42.258 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ce45a1:::benchmark_data_smithi159_139342_object21181:head# 2024-04-15T18:34:42.258 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d0896b:::benchmark_data_smithi159_139342_object13295:head# 2024-04-15T18:34:42.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d1e671:::benchmark_data_smithi159_139342_object53495:head# 2024-04-15T18:34:42.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d24206:::benchmark_data_smithi159_139342_object21236:head# 2024-04-15T18:34:42.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d28869:::benchmark_data_smithi159_139342_object15063:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d344e9:::benchmark_data_smithi159_139342_object39967:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d3479f:::benchmark_data_smithi159_139342_object46237:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d50ac2:::benchmark_data_smithi159_139342_object29773:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d51949:::benchmark_data_smithi159_139342_object18709:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d58713:::benchmark_data_smithi159_139342_object36234:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d60aea:::benchmark_data_smithi159_139342_object55334:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d816ef:::benchmark_data_smithi159_139342_object415:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d926f1:::benchmark_data_smithi159_139342_object40180:head# 2024-04-15T18:34:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d981ff:::benchmark_data_smithi159_139342_object4304:head# 2024-04-15T18:34:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d9894d:::benchmark_data_smithi159_139342_object17951:head# 2024-04-15T18:34:42.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8d9c6aa:::benchmark_data_smithi159_139342_object41160:head# 2024-04-15T18:34:42.267 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8da2982:::benchmark_data_smithi159_139342_object30492:head# 2024-04-15T18:34:42.270 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8da9e41:::benchmark_data_smithi159_139342_object18870:head# 2024-04-15T18:34:42.270 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8db2869:::benchmark_data_smithi159_139342_object37500:head# 2024-04-15T18:34:42.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8dcdf3f:::benchmark_data_smithi159_139342_object29731:head# 2024-04-15T18:34:42.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8dfe913:::benchmark_data_smithi159_139342_object53875:head# 2024-04-15T18:34:42.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e3436b:::benchmark_data_smithi159_139342_object52878:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e3d17c:::benchmark_data_smithi159_139342_object18751:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e551f4:::benchmark_data_smithi159_139342_object38957:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e68822:::benchmark_data_smithi159_139342_object37070:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e7d128:::benchmark_data_smithi159_139342_object40118:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8e9e078:::benchmark_data_smithi159_139342_object53154:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ea5c99:::benchmark_data_smithi159_139342_object39463:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8eaa6b5:::benchmark_data_smithi159_139342_object48328:head# 2024-04-15T18:34:42.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ec4a91:::benchmark_data_smithi159_139342_object5309:head# 2024-04-15T18:34:42.277 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ecc9de:::benchmark_data_smithi159_139342_object34139:head# 2024-04-15T18:34:42.277 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8ef9124:::benchmark_data_smithi159_139342_object51976:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f28093:::benchmark_data_smithi159_139342_object14901:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f38765:::benchmark_data_smithi159_139342_object24261:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f3ac71:::benchmark_data_smithi159_139342_object51651:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f42440:::benchmark_data_smithi159_139342_object45211:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f5b24a:::benchmark_data_smithi159_139342_object27553:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f67b02:::benchmark_data_smithi159_139342_object5200:head# 2024-04-15T18:34:42.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f803fc:::benchmark_data_smithi159_139342_object31489:head# 2024-04-15T18:34:42.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f848de:::benchmark_data_smithi159_139342_object25654:head# 2024-04-15T18:34:42.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f89388:::benchmark_data_smithi159_139342_object23567:head# 2024-04-15T18:34:42.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8f8f31f:::benchmark_data_smithi159_139342_object24397:head# 2024-04-15T18:34:42.284 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fac0fb:::benchmark_data_smithi159_139342_object13693:head# 2024-04-15T18:34:42.284 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fb1668:::benchmark_data_smithi159_139342_object48491:head# 2024-04-15T18:34:42.285 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fbdf39:::benchmark_data_smithi159_139342_object34359:head# 2024-04-15T18:34:42.291 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fe14be:::benchmark_data_smithi159_139342_object719:head# 2024-04-15T18:34:42.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fea19d:::benchmark_data_smithi159_139342_object51853:head# 2024-04-15T18:34:42.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c8fef31f:::benchmark_data_smithi159_139342_object15236:head# 2024-04-15T18:34:42.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9004a8c:::benchmark_data_smithi159_139342_object40995:head# 2024-04-15T18:34:42.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9068107:::benchmark_data_smithi159_139342_object4138:head# 2024-04-15T18:34:42.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c906a08e:::benchmark_data_smithi159_139342_object7546:head# 2024-04-15T18:34:42.294 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c906b729:::benchmark_data_smithi159_139342_object13220:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c906e88c:::benchmark_data_smithi159_139342_object39766:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9077e93:::benchmark_data_smithi159_139342_object52612:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90aa25e:::benchmark_data_smithi159_139342_object1378:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90b8b04:::benchmark_data_smithi159_139342_object20917:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90caf12:::benchmark_data_smithi159_139342_object14861:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90cf399:::benchmark_data_smithi159_139342_object23468:head# 2024-04-15T18:34:42.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90e4967:::benchmark_data_smithi159_139342_object36236:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c90fa4d4:::benchmark_data_smithi159_139342_object14778:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9101eec:::benchmark_data_smithi159_139342_object6585:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c912355f:::benchmark_data_smithi159_139342_object45796:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9135bd2:::benchmark_data_smithi159_139342_object12449:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c913779c:::benchmark_data_smithi159_139342_object32027:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c914fc4f:::benchmark_data_smithi159_139342_object16593:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c915585f:::benchmark_data_smithi159_139342_object21337:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91570aa:::benchmark_data_smithi159_139342_object11210:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9159ffa:::benchmark_data_smithi159_139342_object32767:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91658b3:::benchmark_data_smithi159_139342_object10862:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c919c26a:::benchmark_data_smithi159_139342_object53479:head# 2024-04-15T18:34:42.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91be162:::benchmark_data_smithi159_139342_object39667:head# 2024-04-15T18:34:42.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91dd2b4:::benchmark_data_smithi159_139342_object1624:head# 2024-04-15T18:34:42.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91e399f:::benchmark_data_smithi159_139342_object38684:head# 2024-04-15T18:34:42.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c91f2304:::benchmark_data_smithi159_139342_object20267:head# 2024-04-15T18:34:42.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92184e0:::benchmark_data_smithi159_139342_object47171:head# 2024-04-15T18:34:42.307 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92365a0:::benchmark_data_smithi159_139342_object37147:head# 2024-04-15T18:34:42.307 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9237ff6:::benchmark_data_smithi159_139342_object54861:head# 2024-04-15T18:34:42.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c923cafa:::benchmark_data_smithi159_139342_object15336:head# 2024-04-15T18:34:42.311 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92819d5:::benchmark_data_smithi159_139342_object26715:head# 2024-04-15T18:34:42.312 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c928c43d:::benchmark_data_smithi159_139342_object31499:head# 2024-04-15T18:34:42.314 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92a6579:::benchmark_data_smithi159_139342_object19171:head# 2024-04-15T18:34:42.314 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92abb2d:::benchmark_data_smithi159_139342_object37636:head# 2024-04-15T18:34:42.314 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92ad0f0:::benchmark_data_smithi159_139342_object37340:head# 2024-04-15T18:34:42.315 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92b845b:::benchmark_data_smithi159_139342_object31092:head# 2024-04-15T18:34:42.315 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92c98b3:::benchmark_data_smithi159_139342_object42729:head# 2024-04-15T18:34:42.317 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92ceacb:::benchmark_data_smithi159_139342_object16084:head# 2024-04-15T18:34:42.318 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92d3777:::benchmark_data_smithi159_139342_object3027:head# 2024-04-15T18:34:42.318 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92efb36:::benchmark_data_smithi159_139342_object7924:head# 2024-04-15T18:34:42.319 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c92f4c39:::benchmark_data_smithi159_139342_object6884:head# 2024-04-15T18:34:42.320 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9312e90:::benchmark_data_smithi159_139342_object39062:head# 2024-04-15T18:34:42.320 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c931edc0:::benchmark_data_smithi159_139342_object19815:head# 2024-04-15T18:34:42.323 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c93273b0:::benchmark_data_smithi159_139342_object23364:head# 2024-04-15T18:34:42.324 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c934bd77:::benchmark_data_smithi159_139342_object17206:head# 2024-04-15T18:34:42.324 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c936626d:::benchmark_data_smithi159_139342_object23175:head# 2024-04-15T18:34:42.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c936ace8:::benchmark_data_smithi159_139342_object32586:head# 2024-04-15T18:34:42.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9383cc2:::benchmark_data_smithi159_139342_object50558:head# 2024-04-15T18:34:42.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c93be78d:::benchmark_data_smithi159_139342_object29584:head# 2024-04-15T18:34:42.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c93f3a4b:::benchmark_data_smithi159_139342_object21803:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c93f67b1:::benchmark_data_smithi159_139342_object43709:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9432050:::benchmark_data_smithi159_139342_object1004:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c944657a:::benchmark_data_smithi159_139342_object54181:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c944d3a8:::benchmark_data_smithi159_139342_object422:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c948a582:::benchmark_data_smithi159_139342_object45674:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c949a482:::benchmark_data_smithi159_139342_object6456:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94a1e69:::benchmark_data_smithi159_139342_object20733:head# 2024-04-15T18:34:42.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94ada26:::benchmark_data_smithi159_139342_object46395:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94adbb9:::benchmark_data_smithi159_139342_object39948:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94b2142:::benchmark_data_smithi159_139342_object26309:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94bb72a:::benchmark_data_smithi159_139342_object39343:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94cd101:::benchmark_data_smithi159_139342_object14987:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94d8786:::benchmark_data_smithi159_139342_object20205:head# 2024-04-15T18:34:42.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94da3d2:::benchmark_data_smithi159_139342_object5038:head# 2024-04-15T18:34:42.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94dda1c:::benchmark_data_smithi159_139342_object25237:head# 2024-04-15T18:34:42.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94e61f2:::benchmark_data_smithi159_139342_object18881:head# 2024-04-15T18:34:42.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94e6243:::benchmark_data_smithi159_139342_object47595:head# 2024-04-15T18:34:42.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94f0195:::benchmark_data_smithi159_139342_object21114:head# 2024-04-15T18:34:42.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c94f62ef:::benchmark_data_smithi159_139342_object46195:head# 2024-04-15T18:34:42.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c953510c:::benchmark_data_smithi159_139342_object36786:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9539eb5:::benchmark_data_smithi159_139342_object50961:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9556eca:::benchmark_data_smithi159_139342_object28482:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9567c36:::benchmark_data_smithi159_139342_object26478:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95690fb:::benchmark_data_smithi159_139342_object38020:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9578429:::benchmark_data_smithi159_139342_object9204:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c958cd36:::benchmark_data_smithi159_139342_object6167:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c959f920:::benchmark_data_smithi159_139342_object46374:head# 2024-04-15T18:34:42.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95a83be:::benchmark_data_smithi159_139342_object40088:head# 2024-04-15T18:34:42.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95b7dbd:::benchmark_data_smithi159_139342_object14812:head# 2024-04-15T18:34:42.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95cdea2:::benchmark_data_smithi159_139342_object37017:head# 2024-04-15T18:34:42.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95d13c5:::benchmark_data_smithi159_139342_object53627:head# 2024-04-15T18:34:42.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95da31a:::benchmark_data_smithi159_139342_object18523:head# 2024-04-15T18:34:42.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95ed828:::benchmark_data_smithi159_139342_object44923:head# 2024-04-15T18:34:42.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c95f5c8b:::benchmark_data_smithi159_139342_object14508:head# 2024-04-15T18:34:42.372 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9600e73:::benchmark_data_smithi159_139342_object23817:head# 2024-04-15T18:34:42.372 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9612d2f:::benchmark_data_smithi159_139342_object19338:head# 2024-04-15T18:34:42.372 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9613fcb:::benchmark_data_smithi159_139342_object45938:head# 2024-04-15T18:34:42.373 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c961fc00:::benchmark_data_smithi159_139342_object17027:head# 2024-04-15T18:34:42.373 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c962f87b:::benchmark_data_smithi159_139342_object11348:head# 2024-04-15T18:34:42.374 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9630d12:::benchmark_data_smithi159_139342_object12451:head# 2024-04-15T18:34:42.374 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9632576:::benchmark_data_smithi159_139342_object15108:head# 2024-04-15T18:34:42.375 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96464ac:::benchmark_data_smithi159_139342_object31945:head# 2024-04-15T18:34:42.375 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96660e8:::benchmark_data_smithi159_139342_object21067:head# 2024-04-15T18:34:42.375 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c966f702:::benchmark_data_smithi159_139342_object30435:head# 2024-04-15T18:34:42.376 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c967e225:::benchmark_data_smithi159_139342_object2310:head# 2024-04-15T18:34:42.376 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c969b259:::benchmark_data_smithi159_139342_object26388:head# 2024-04-15T18:34:42.380 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96a3b91:::benchmark_data_smithi159_139342_object21875:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96beb63:::benchmark_data_smithi159_139342_object2909:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96c6b58:::benchmark_data_smithi159_139342_object35257:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96cd79d:::benchmark_data_smithi159_139342_object17029:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c96cef52:::benchmark_data_smithi159_139342_object39114:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c971365a:::benchmark_data_smithi159_139342_object42624:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9717486:::benchmark_data_smithi159_139342_object31079:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c974e5f9:::benchmark_data_smithi159_139342_object6901:head# 2024-04-15T18:34:42.381 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c975f459:::benchmark_data_smithi159_139342_object16757:head# 2024-04-15T18:34:42.382 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c976f528:::benchmark_data_smithi159_139342_object56797:head# 2024-04-15T18:34:42.382 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c977bf7d:::benchmark_data_smithi159_139342_object1550:head# 2024-04-15T18:34:42.382 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97960b5:::benchmark_data_smithi159_139342_object46150:head# 2024-04-15T18:34:42.382 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c979b715:::benchmark_data_smithi159_139342_object11509:head# 2024-04-15T18:34:42.382 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c979b98e:::benchmark_data_smithi159_139342_object45065:head# 2024-04-15T18:34:42.383 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97a4be4:::benchmark_data_smithi159_139342_object52475:head# 2024-04-15T18:34:42.383 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97b46b4:::benchmark_data_smithi159_139342_object33340:head# 2024-04-15T18:34:42.384 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97cc794:::benchmark_data_smithi159_139342_object4430:head# 2024-04-15T18:34:42.385 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97d3004:::benchmark_data_smithi159_139342_object657:head# 2024-04-15T18:34:42.385 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97daf1b:::benchmark_data_smithi159_139342_object36251:head# 2024-04-15T18:34:42.385 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c97fd1eb:::benchmark_data_smithi159_139342_object11302:head# 2024-04-15T18:34:42.386 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9814b56:::benchmark_data_smithi159_139342_object51722:head# 2024-04-15T18:34:42.386 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9827528:::benchmark_data_smithi159_139342_object16293:head# 2024-04-15T18:34:42.386 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c982e29e:::benchmark_data_smithi159_139342_object56153:head# 2024-04-15T18:34:42.387 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9834518:::benchmark_data_smithi159_139342_object29212:head# 2024-04-15T18:34:42.387 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c98347f1:::benchmark_data_smithi159_139342_object13932:head# 2024-04-15T18:34:42.392 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c986ccb8:::benchmark_data_smithi159_139342_object7672:head# 2024-04-15T18:34:42.392 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9875949:::benchmark_data_smithi159_139342_object14672:head# 2024-04-15T18:34:42.392 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c988b126:::benchmark_data_smithi159_139342_object42682:head# 2024-04-15T18:34:42.392 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c98f2a35:::benchmark_data_smithi159_139342_object39330:head# 2024-04-15T18:34:42.392 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c98f4092:::benchmark_data_smithi159_139342_object43067:head# 2024-04-15T18:34:42.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9960344:::benchmark_data_smithi159_139342_object22204:head# 2024-04-15T18:34:42.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9968c0e:::benchmark_data_smithi159_139342_object9504:head# 2024-04-15T18:34:42.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9973f82:::benchmark_data_smithi159_139342_object13164:head# 2024-04-15T18:34:42.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9974e71:::benchmark_data_smithi159_139342_object28498:head# 2024-04-15T18:34:42.393 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c997d82d:::benchmark_data_smithi159_139342_object12359:head# 2024-04-15T18:34:42.394 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c999ef5b:::benchmark_data_smithi159_139342_object4776:head# 2024-04-15T18:34:42.394 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99ad65b:::benchmark_data_smithi159_139342_object833:head# 2024-04-15T18:34:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99be1de:::benchmark_data_smithi159_139342_object52183:head# 2024-04-15T18:34:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99db7de:::benchmark_data_smithi159_139342_object28119:head# 2024-04-15T18:34:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99def5b:::benchmark_data_smithi159_139342_object32047:head# 2024-04-15T18:34:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99f20c9:::benchmark_data_smithi159_139342_object40359:head# 2024-04-15T18:34:42.395 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99f5584:::benchmark_data_smithi159_139342_object47545:head# 2024-04-15T18:34:42.396 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c99f6ab6:::benchmark_data_smithi159_139342_object48176:head# 2024-04-15T18:34:42.396 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a32466:::benchmark_data_smithi159_139342_object9050:head# 2024-04-15T18:34:42.397 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a473cc:::benchmark_data_smithi159_139342_object49067:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a5471c:::benchmark_data_smithi159_139342_object7867:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a569ae:::benchmark_data_smithi159_139342_object51743:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a712fc:::benchmark_data_smithi159_139342_object18241:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a7565b:::benchmark_data_smithi159_139342_object13834:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a757b5:::benchmark_data_smithi159_139342_object50314:head# 2024-04-15T18:34:42.409 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a818c7:::benchmark_data_smithi159_139342_object3682:head# 2024-04-15T18:34:42.410 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a89058:::benchmark_data_smithi159_139342_object3732:head# 2024-04-15T18:34:42.410 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a973b7:::benchmark_data_smithi159_139342_object19346:head# 2024-04-15T18:34:42.410 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9a9dff6:::benchmark_data_smithi159_139342_object28480:head# 2024-04-15T18:34:42.410 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9aa723d:::benchmark_data_smithi159_139342_object14495:head# 2024-04-15T18:34:42.414 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9aacefa:::benchmark_data_smithi159_139342_object8696:head# 2024-04-15T18:34:42.414 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9aaf28a:::benchmark_data_smithi159_139342_object19217:head# 2024-04-15T18:34:42.414 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ac2338:::benchmark_data_smithi159_139342_object56340:head# 2024-04-15T18:34:42.414 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b046ec:::benchmark_data_smithi159_139342_object21825:head# 2024-04-15T18:34:42.418 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b0e3f9:::benchmark_data_smithi159_139342_object37954:head# 2024-04-15T18:34:42.418 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b1eceb:::benchmark_data_smithi159_139342_object45933:head# 2024-04-15T18:34:42.420 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b47d94:::benchmark_data_smithi159_139342_object11214:head# 2024-04-15T18:34:42.420 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b49bd8:::benchmark_data_smithi159_139342_object33252:head# 2024-04-15T18:34:42.420 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b4c2ea:::benchmark_data_smithi159_139342_object49287:head# 2024-04-15T18:34:42.420 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b5201d:::benchmark_data_smithi159_139342_object11496:head# 2024-04-15T18:34:42.420 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b6df2a:::benchmark_data_smithi159_139342_object33272:head# 2024-04-15T18:34:42.426 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b6f665:::benchmark_data_smithi159_139342_object27629:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b73850:::benchmark_data_smithi159_139342_object33169:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9b94575:::benchmark_data_smithi159_139342_object21952:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ba4fb8:::benchmark_data_smithi159_139342_object9274:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ba7b52:::benchmark_data_smithi159_139342_object4411:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9bb5eef:::benchmark_data_smithi159_139342_object19740:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9beee2a:::benchmark_data_smithi159_139342_object43615:head# 2024-04-15T18:34:42.427 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9bf06ab:::benchmark_data_smithi159_139342_object50341:head# 2024-04-15T18:34:42.428 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9bf6b93:::benchmark_data_smithi159_139342_object37679:head# 2024-04-15T18:34:42.428 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9bfacbb:::benchmark_data_smithi159_139342_object46726:head# 2024-04-15T18:34:42.428 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c06d0e:::benchmark_data_smithi159_139342_object20279:head# 2024-04-15T18:34:42.428 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c0c07c:::benchmark_data_smithi159_139342_object54418:head# 2024-04-15T18:34:42.429 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c120bc:::benchmark_data_smithi159_139342_object16310:head# 2024-04-15T18:34:42.429 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c30b7e:::benchmark_data_smithi159_139342_object9344:head# 2024-04-15T18:34:42.430 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c3a5b7:::benchmark_data_smithi159_139342_object42223:head# 2024-04-15T18:34:42.430 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c3d509:::benchmark_data_smithi159_139342_object54311:head# 2024-04-15T18:34:42.431 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c43252:::benchmark_data_smithi159_139342_object41170:head# 2024-04-15T18:34:42.434 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c48daa:::benchmark_data_smithi159_139342_object8021:head# 2024-04-15T18:34:42.434 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c5fd3c:::benchmark_data_smithi159_139342_object2703:head# 2024-04-15T18:34:42.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c7a851:::benchmark_data_smithi159_139342_object55978:head# 2024-04-15T18:34:42.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c8bef7:::benchmark_data_smithi159_139342_object35790:head# 2024-04-15T18:34:42.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9c8d056:::benchmark_data_smithi159_139342_object39386:head# 2024-04-15T18:34:42.435 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cc10c5:::benchmark_data_smithi159_139342_object22846:head# 2024-04-15T18:34:42.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cd4225:::benchmark_data_smithi159_139342_object34024:head# 2024-04-15T18:34:42.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cda880:::benchmark_data_smithi159_139342_object35559:head# 2024-04-15T18:34:42.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cdf13a:::benchmark_data_smithi159_139342_object29209:head# 2024-04-15T18:34:42.436 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cefae8:::benchmark_data_smithi159_139342_object21011:head# 2024-04-15T18:34:42.440 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9cf161f:::benchmark_data_smithi159_139342_object16264:head# 2024-04-15T18:34:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d0efde:::benchmark_data_smithi159_139342_object20976:head# 2024-04-15T18:34:42.448 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d1b489:::benchmark_data_smithi159_139342_object39883:head# 2024-04-15T18:34:42.449 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d49fe1:::benchmark_data_smithi159_139342_object17457:head# 2024-04-15T18:34:42.450 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d587fc:::benchmark_data_smithi159_139342_object52700:head# 2024-04-15T18:34:42.450 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d71773:::benchmark_data_smithi159_139342_object15341:head# 2024-04-15T18:34:42.450 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9d88934:::benchmark_data_smithi159_139342_object29669:head# 2024-04-15T18:34:42.451 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9dab015:::benchmark_data_smithi159_139342_object29607:head# 2024-04-15T18:34:42.451 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9dba2d7:::benchmark_data_smithi159_139342_object13540:head# 2024-04-15T18:34:42.451 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9dc6a0d:::benchmark_data_smithi159_139342_object12005:head# 2024-04-15T18:34:42.452 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9dd459f:::benchmark_data_smithi159_139342_object36274:head# 2024-04-15T18:34:42.452 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9de50f5:::benchmark_data_smithi159_139342_object23034:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9deb011:::benchmark_data_smithi159_139342_object5578:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9df32b0:::benchmark_data_smithi159_139342_object52970:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e0d46c:::benchmark_data_smithi159_139342_object1256:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e4aaf1:::benchmark_data_smithi159_139342_object26653:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e54e86:::benchmark_data_smithi159_139342_object43228:head# 2024-04-15T18:34:42.457 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e6d68a:::benchmark_data_smithi159_139342_object24961:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e72340:::benchmark_data_smithi159_139342_object46167:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e726f4:::benchmark_data_smithi159_139342_object43145:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9e899ea:::benchmark_data_smithi159_139342_object33339:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ea9314:::benchmark_data_smithi159_139342_object57300:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9eb80e0:::benchmark_data_smithi159_139342_object52396:head# 2024-04-15T18:34:42.458 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ebd2d4:::benchmark_data_smithi159_139342_object33523:head# 2024-04-15T18:34:42.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ecabb9:::benchmark_data_smithi159_139342_object17273:head# 2024-04-15T18:34:42.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ecaec8:::benchmark_data_smithi159_139342_object17866:head# 2024-04-15T18:34:42.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9ee5078:::benchmark_data_smithi159_139342_object19697:head# 2024-04-15T18:34:42.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f12a9b:::benchmark_data_smithi159_139342_object55364:head# 2024-04-15T18:34:42.462 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f14824:::benchmark_data_smithi159_139342_object15851:head# 2024-04-15T18:34:42.463 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f330ca:::benchmark_data_smithi159_139342_object18648:head# 2024-04-15T18:34:42.463 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f44f9f:::benchmark_data_smithi159_139342_object29536:head# 2024-04-15T18:34:42.463 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f50b9e:::benchmark_data_smithi159_139342_object54453:head# 2024-04-15T18:34:42.464 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9f96c77:::benchmark_data_smithi159_139342_object12591:head# 2024-04-15T18:34:42.464 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9fd7353:::benchmark_data_smithi159_139342_object9276:head# 2024-04-15T18:34:42.465 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9fd79d1:::benchmark_data_smithi159_139342_object22982:head# 2024-04-15T18:34:42.465 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:c9fe0e67:::benchmark_data_smithi159_139342_object44583:head# 2024-04-15T18:34:42.466 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca01143b:::benchmark_data_smithi159_139342_object32543:head# 2024-04-15T18:34:42.467 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca018e3f:::benchmark_data_smithi159_139342_object16646:head# 2024-04-15T18:34:42.467 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca01cdfb:::benchmark_data_smithi159_139342_object14354:head# 2024-04-15T18:34:42.468 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca029181:::benchmark_data_smithi159_139342_object32384:head# 2024-04-15T18:34:42.468 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca029bdb:::benchmark_data_smithi159_139342_object38486:head# 2024-04-15T18:34:42.469 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca03c31b:::benchmark_data_smithi159_139342_object21658:head# 2024-04-15T18:34:42.469 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca03f2bd:::benchmark_data_smithi159_139342_object17261:head# 2024-04-15T18:34:42.469 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca05077e:::benchmark_data_smithi159_139342_object53287:head# 2024-04-15T18:34:42.470 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca060c45:::benchmark_data_smithi159_139342_object15317:head# 2024-04-15T18:34:42.470 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca0934c1:::benchmark_data_smithi159_139342_object36791:head# 2024-04-15T18:34:42.471 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca0abec6:::benchmark_data_smithi159_139342_object11558:head# 2024-04-15T18:34:42.472 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca0faef8:::benchmark_data_smithi159_139342_object33307:head# 2024-04-15T18:34:42.472 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca104536:::benchmark_data_smithi159_139342_object20649:head# 2024-04-15T18:34:42.476 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1246c6:::benchmark_data_smithi159_139342_object36879:head# 2024-04-15T18:34:42.476 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca142163:::benchmark_data_smithi159_139342_object29687:head# 2024-04-15T18:34:42.476 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca147537:::benchmark_data_smithi159_139342_object42678:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca178117:::benchmark_data_smithi159_139342_object36568:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca190022:::benchmark_data_smithi159_139342_object8464:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca19bf13:::benchmark_data_smithi159_139342_object17259:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1ae060:::benchmark_data_smithi159_139342_object32451:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1b429e:::benchmark_data_smithi159_139342_object46723:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1b4e4d:::benchmark_data_smithi159_139342_object44551:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1ba149:::benchmark_data_smithi159_139342_object22781:head# 2024-04-15T18:34:42.479 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1c5e88:::benchmark_data_smithi159_139342_object18880:head# 2024-04-15T18:34:42.480 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca1dc0d0:::benchmark_data_smithi159_139342_object11293:head# 2024-04-15T18:34:42.480 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca211128:::benchmark_data_smithi159_139342_object19581:head# 2024-04-15T18:34:42.480 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca223553:::benchmark_data_smithi159_139342_object53235:head# 2024-04-15T18:34:42.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca228c2b:::benchmark_data_smithi159_139342_object46600:head# 2024-04-15T18:34:42.481 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca22ecf8:::benchmark_data_smithi159_139342_object55949:head# 2024-04-15T18:34:42.482 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca231d64:::benchmark_data_smithi159_139342_object29758:head# 2024-04-15T18:34:42.483 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca2357c3:::benchmark_data_smithi159_139342_object29901:head# 2024-04-15T18:34:42.483 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca23bc2a:::benchmark_data_smithi159_139342_object7232:head# 2024-04-15T18:34:42.483 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca243613:::benchmark_data_smithi159_139342_object36648:head# 2024-04-15T18:34:42.484 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca249ff5:::benchmark_data_smithi159_139342_object3252:head# 2024-04-15T18:34:42.484 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca24b8cd:::benchmark_data_smithi159_139342_object53948:head# 2024-04-15T18:34:42.484 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca256b55:::benchmark_data_smithi159_139342_object3916:head# 2024-04-15T18:34:42.485 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca26530c:::benchmark_data_smithi159_139342_object28112:head# 2024-04-15T18:34:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca2691c4:::benchmark_data_smithi159_139342_object5376:head# 2024-04-15T18:34:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca28a963:::benchmark_data_smithi159_139342_object51895:head# 2024-04-15T18:34:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca2a8bc6:::benchmark_data_smithi159_139342_object14483:head# 2024-04-15T18:34:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca2c2991:::benchmark_data_smithi159_139342_object39528:head# 2024-04-15T18:34:42.489 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca2fa1e4:::benchmark_data_smithi159_139342_object49097:head# 2024-04-15T18:34:42.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca308f71:::benchmark_data_smithi159_139342_object15723:head# 2024-04-15T18:34:42.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca30bf92:::benchmark_data_smithi159_139342_object17575:head# 2024-04-15T18:34:42.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3295c0:::benchmark_data_smithi159_139342_object23999:head# 2024-04-15T18:34:42.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca331351:::benchmark_data_smithi159_139342_object6820:head# 2024-04-15T18:34:42.490 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca35ba98:::benchmark_data_smithi159_139342_object24322:head# 2024-04-15T18:34:42.492 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca37edd8:::benchmark_data_smithi159_139342_object40642:head# 2024-04-15T18:34:42.492 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca38d267:::benchmark_data_smithi159_139342_object32895:head# 2024-04-15T18:34:42.494 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca392912:::benchmark_data_smithi159_139342_object16993:head# 2024-04-15T18:34:42.495 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3a5469:::benchmark_data_smithi159_139342_object13800:head# 2024-04-15T18:34:42.496 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3b6a7b:::benchmark_data_smithi159_139342_object40020:head# 2024-04-15T18:34:42.496 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3b89cc:::benchmark_data_smithi159_139342_object57258:head# 2024-04-15T18:34:42.497 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3bcb71:::benchmark_data_smithi159_139342_object664:head# 2024-04-15T18:34:42.497 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3bda2e:::benchmark_data_smithi159_139342_object49375:head# 2024-04-15T18:34:42.497 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3cacf7:::benchmark_data_smithi159_139342_object5034:head# 2024-04-15T18:34:42.498 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca3dd758:::benchmark_data_smithi159_139342_object1898:head# 2024-04-15T18:34:42.499 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca40bd73:::benchmark_data_smithi159_139342_object39993:head# 2024-04-15T18:34:42.499 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca424b45:::benchmark_data_smithi159_139342_object43757:head# 2024-04-15T18:34:42.500 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4267de:::benchmark_data_smithi159_139342_object31780:head# 2024-04-15T18:34:42.500 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4481c8:::benchmark_data_smithi159_139342_object21596:head# 2024-04-15T18:34:42.500 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca44d07a:::benchmark_data_smithi159_139342_object15244:head# 2024-04-15T18:34:42.504 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca44f887:::benchmark_data_smithi159_139342_object19363:head# 2024-04-15T18:34:42.504 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca46798e:::benchmark_data_smithi159_139342_object52251:head# 2024-04-15T18:34:42.504 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca47046e:::benchmark_data_smithi159_139342_object36222:head# 2024-04-15T18:34:42.507 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca485871:::benchmark_data_smithi159_139342_object1369:head# 2024-04-15T18:34:42.507 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4a7813:::benchmark_data_smithi159_139342_object44992:head# 2024-04-15T18:34:42.507 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4baa0d:::benchmark_data_smithi159_139342_object13406:head# 2024-04-15T18:34:42.508 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4bda29:::benchmark_data_smithi159_139342_object27095:head# 2024-04-15T18:34:42.509 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4c5d76:::benchmark_data_smithi159_139342_object21093:head# 2024-04-15T18:34:42.509 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4e6bce:::benchmark_data_smithi159_139342_object16224:head# 2024-04-15T18:34:42.509 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4e8db6:::benchmark_data_smithi159_139342_object30755:head# 2024-04-15T18:34:42.510 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca4f6ca2:::benchmark_data_smithi159_139342_object8067:head# 2024-04-15T18:34:42.510 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca50a125:::benchmark_data_smithi159_139342_object47603:head# 2024-04-15T18:34:42.510 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca50f133:::benchmark_data_smithi159_139342_object33136:head# 2024-04-15T18:34:42.511 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca52d117:::benchmark_data_smithi159_139342_object5745:head# 2024-04-15T18:34:42.511 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5351b8:::benchmark_data_smithi159_139342_object15092:head# 2024-04-15T18:34:42.512 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca53eea5:::benchmark_data_smithi159_139342_object11581:head# 2024-04-15T18:34:42.512 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5429cd:::benchmark_data_smithi159_139342_object54489:head# 2024-04-15T18:34:42.513 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5444ff:::benchmark_data_smithi159_139342_object37712:head# 2024-04-15T18:34:42.513 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5489bf:::benchmark_data_smithi159_139342_object12178:head# 2024-04-15T18:34:42.514 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca549101:::benchmark_data_smithi159_139342_object37127:head# 2024-04-15T18:34:42.514 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca57018d:::benchmark_data_smithi159_139342_object47337:head# 2024-04-15T18:34:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca57da3c:::benchmark_data_smithi159_139342_object56976:head# 2024-04-15T18:34:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5829b5:::benchmark_data_smithi159_139342_object47549:head# 2024-04-15T18:34:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5829de:::benchmark_data_smithi159_139342_object24562:head# 2024-04-15T18:34:42.515 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca585ab7:::benchmark_data_smithi159_139342_object47924:head# 2024-04-15T18:34:42.516 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca59f058:::benchmark_data_smithi159_139342_object34907:head# 2024-04-15T18:34:42.519 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5b22e0:::benchmark_data_smithi159_139342_object21641:head# 2024-04-15T18:34:42.519 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5c0be0:::benchmark_data_smithi159_139342_object17949:head# 2024-04-15T18:34:42.519 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5c0f7b:::benchmark_data_smithi159_139342_object3310:head# 2024-04-15T18:34:42.519 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5cf72e:::benchmark_data_smithi159_139342_object30219:head# 2024-04-15T18:34:42.519 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5d3cc0:::benchmark_data_smithi159_139342_object18482:head# 2024-04-15T18:34:42.520 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5dc39b:::benchmark_data_smithi159_139342_object8872:head# 2024-04-15T18:34:42.520 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5ed00c:::benchmark_data_smithi159_139342_object33271:head# 2024-04-15T18:34:42.520 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca5ee132:::benchmark_data_smithi159_139342_object42183:head# 2024-04-15T18:34:42.521 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca62f916:::benchmark_data_smithi159_139342_object56197:head# 2024-04-15T18:34:42.521 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca638eac:::benchmark_data_smithi159_139342_object24769:head# 2024-04-15T18:34:42.522 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca645551:::benchmark_data_smithi159_139342_object34413:head# 2024-04-15T18:34:42.522 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca655333:::benchmark_data_smithi159_139342_object27111:head# 2024-04-15T18:34:42.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca665e2b:::benchmark_data_smithi159_139342_object55115:head# 2024-04-15T18:34:42.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca66f866:::benchmark_data_smithi159_139342_object26097:head# 2024-04-15T18:34:42.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca67b8e1:::benchmark_data_smithi159_139342_object46657:head# 2024-04-15T18:34:42.527 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca68f50c:::benchmark_data_smithi159_139342_object30215:head# 2024-04-15T18:34:42.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6cb780:::benchmark_data_smithi159_139342_object12566:head# 2024-04-15T18:34:42.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6e3225:::benchmark_data_smithi159_139342_object12521:head# 2024-04-15T18:34:42.528 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6e58eb:::benchmark_data_smithi159_139342_object27773:head# 2024-04-15T18:34:42.529 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6ebd94:::benchmark_data_smithi159_139342_object27068:head# 2024-04-15T18:34:42.529 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6ee4f8:::benchmark_data_smithi159_139342_object13561:head# 2024-04-15T18:34:42.530 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6f6164:::benchmark_data_smithi159_139342_object37364:head# 2024-04-15T18:34:42.530 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca6fc5b7:::benchmark_data_smithi159_139342_object8225:head# 2024-04-15T18:34:42.531 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca71826f:::benchmark_data_smithi159_139342_object6289:head# 2024-04-15T18:34:42.531 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca728db2:::benchmark_data_smithi159_139342_object49714:head# 2024-04-15T18:34:42.531 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7734b8:::benchmark_data_smithi159_139342_object48653:head# 2024-04-15T18:34:42.532 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca783308:::benchmark_data_smithi159_139342_object10292:head# 2024-04-15T18:34:42.532 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca78b476:::benchmark_data_smithi159_139342_object54465:head# 2024-04-15T18:34:42.533 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca790505:::benchmark_data_smithi159_139342_object28443:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7973c1:::benchmark_data_smithi159_139342_object8124:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca79acd8:::benchmark_data_smithi159_139342_object44051:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7a4a59:::benchmark_data_smithi159_139342_object27626:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7a5385:::benchmark_data_smithi159_139342_object50539:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7c9a49:::benchmark_data_smithi159_139342_object56029:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca7fe878:::benchmark_data_smithi159_139342_object51047:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca829dd3:::benchmark_data_smithi159_139342_object2893:head# 2024-04-15T18:34:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8387af:::benchmark_data_smithi159_139342_object34540:head# 2024-04-15T18:34:42.538 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca84b36b:::benchmark_data_smithi159_139342_object38177:head# 2024-04-15T18:34:42.544 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca85e572:::benchmark_data_smithi159_139342_object45887:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca85f4d0:::benchmark_data_smithi159_139342_object18990:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca87355c:::benchmark_data_smithi159_139342_object39011:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8854e8:::benchmark_data_smithi159_139342_object6911:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca890cdd:::benchmark_data_smithi159_139342_object8729:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca89c9bc:::benchmark_data_smithi159_139342_object54937:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8abf2e:::benchmark_data_smithi159_139342_object16422:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8b5ba1:::benchmark_data_smithi159_139342_object20730:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8da232:::benchmark_data_smithi159_139342_object46997:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8dc81e:::benchmark_data_smithi159_139342_object30171:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8ddf37:::benchmark_data_smithi159_139342_object19410:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8df266:::benchmark_data_smithi159_139342_object2907:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca8e6232:::benchmark_data_smithi159_139342_object18869:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9070f7:::benchmark_data_smithi159_139342_object19215:head# 2024-04-15T18:34:42.545 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca91023c:::benchmark_data_smithi159_139342_object13316:head# 2024-04-15T18:34:42.546 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca916a7e:::benchmark_data_smithi159_139342_object26485:head# 2024-04-15T18:34:42.547 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca920be6:::benchmark_data_smithi159_139342_object32211:head# 2024-04-15T18:34:42.547 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca955822:::benchmark_data_smithi159_139342_object28855:head# 2024-04-15T18:34:42.548 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca959fb4:::benchmark_data_smithi159_139342_object34222:head# 2024-04-15T18:34:42.548 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca966723:::benchmark_data_smithi159_139342_object21900:head# 2024-04-15T18:34:42.549 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca98f158:::benchmark_data_smithi159_139342_object29748:head# 2024-04-15T18:34:42.549 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca993ac9:::benchmark_data_smithi159_139342_object30214:head# 2024-04-15T18:34:42.549 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca995bf1:::benchmark_data_smithi159_139342_object46636:head# 2024-04-15T18:34:42.550 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9b9ac8:::benchmark_data_smithi159_139342_object30418:head# 2024-04-15T18:34:42.554 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9c71e3:::benchmark_data_smithi159_139342_object8981:head# 2024-04-15T18:34:42.555 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9d0143:::benchmark_data_smithi159_139342_object19320:head# 2024-04-15T18:34:42.555 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9d1559:::benchmark_data_smithi159_139342_object40969:head# 2024-04-15T18:34:42.555 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9d6030:::benchmark_data_smithi159_139342_object3112:head# 2024-04-15T18:34:42.555 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9df43f:::benchmark_data_smithi159_139342_object46644:head# 2024-04-15T18:34:42.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ca9f4e7c:::benchmark_data_smithi159_139342_object30309:head# 2024-04-15T18:34:42.556 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa09d22:::benchmark_data_smithi159_139342_object25457:head# 2024-04-15T18:34:42.557 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa0caea:::benchmark_data_smithi159_139342_object9816:head# 2024-04-15T18:34:42.557 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa0f56d:::benchmark_data_smithi159_139342_object37168:head# 2024-04-15T18:34:42.557 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa13880:::benchmark_data_smithi159_139342_object33067:head# 2024-04-15T18:34:42.558 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa1b2f9:::benchmark_data_smithi159_139342_object45415:head# 2024-04-15T18:34:42.558 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa25fbc:::benchmark_data_smithi159_139342_object10651:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa2e1e1:::benchmark_data_smithi159_139342_object16139:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa2e24d:::benchmark_data_smithi159_139342_object20471:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa2fc92:::benchmark_data_smithi159_139342_object56291:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa300a9:::benchmark_data_smithi159_139342_object17764:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa34007:::benchmark_data_smithi159_139342_object44435:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa54cba:::benchmark_data_smithi159_139342_object37046:head# 2024-04-15T18:34:42.563 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa76e8c:::benchmark_data_smithi159_139342_object33749:head# 2024-04-15T18:34:42.565 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa86886:::benchmark_data_smithi159_139342_object57010:head# 2024-04-15T18:34:42.566 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caa8e4f6:::benchmark_data_smithi159_139342_object44354:head# 2024-04-15T18:34:42.567 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caaa3d91:::benchmark_data_smithi159_139342_object24298:head# 2024-04-15T18:34:42.567 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caab9576:::benchmark_data_smithi159_139342_object29085:head# 2024-04-15T18:34:42.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caae54b5:::benchmark_data_smithi159_139342_object37779:head# 2024-04-15T18:34:42.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caae5d10:::benchmark_data_smithi159_139342_object1044:head# 2024-04-15T18:34:42.570 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caaeecd7:::benchmark_data_smithi159_139342_object43291:head# 2024-04-15T18:34:42.573 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caaefed9:::benchmark_data_smithi159_139342_object52597:head# 2024-04-15T18:34:42.574 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab18451:::benchmark_data_smithi159_139342_object21:head# 2024-04-15T18:34:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab1e203:::benchmark_data_smithi159_139342_object23714:head# 2024-04-15T18:34:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab23bc7:::benchmark_data_smithi159_139342_object16602:head# 2024-04-15T18:34:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab2d0f9:::benchmark_data_smithi159_139342_object30295:head# 2024-04-15T18:34:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab31afa:::benchmark_data_smithi159_139342_object96:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab37286:::benchmark_data_smithi159_139342_object23346:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab5bab2:::benchmark_data_smithi159_139342_object18876:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab6346f:::benchmark_data_smithi159_139342_object18520:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab8636b:::benchmark_data_smithi159_139342_object56193:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cab90a2a:::benchmark_data_smithi159_139342_object7474:head# 2024-04-15T18:34:42.591 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cabae522:::benchmark_data_smithi159_139342_object54566:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cabbc25a:::benchmark_data_smithi159_139342_object18790:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac1a37f:::benchmark_data_smithi159_139342_object45711:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac2758c:::benchmark_data_smithi159_139342_object56419:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac2db37:::benchmark_data_smithi159_139342_object38707:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac37efb:::benchmark_data_smithi159_139342_object10349:head# 2024-04-15T18:34:42.595 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac385b1:::benchmark_data_smithi159_139342_object8740:head# 2024-04-15T18:34:42.602 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac3b505:::benchmark_data_smithi159_139342_object39845:head# 2024-04-15T18:34:42.602 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac3d56a:::benchmark_data_smithi159_139342_object25258:head# 2024-04-15T18:34:42.603 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac3d8fb:::benchmark_data_smithi159_139342_object34416:head# 2024-04-15T18:34:42.605 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac5d1f7:::benchmark_data_smithi159_139342_object11905:head# 2024-04-15T18:34:42.605 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac5e60f:::benchmark_data_smithi159_139342_object51242:head# 2024-04-15T18:34:42.606 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cac8e829:::benchmark_data_smithi159_139342_object39606:head# 2024-04-15T18:34:42.606 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacb662f:::benchmark_data_smithi159_139342_object22481:head# 2024-04-15T18:34:42.607 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacb75bc:::benchmark_data_smithi159_139342_object9072:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacc2488:::benchmark_data_smithi159_139342_object14156:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacd6ebc:::benchmark_data_smithi159_139342_object42536:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacd7193:::benchmark_data_smithi159_139342_object17179:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cace4700:::benchmark_data_smithi159_139342_object45074:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cacf8ed2:::benchmark_data_smithi159_139342_object12043:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad15776:::benchmark_data_smithi159_139342_object12670:head# 2024-04-15T18:34:42.610 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad22023:::benchmark_data_smithi159_139342_object7253:head# 2024-04-15T18:34:42.613 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad22c52:::benchmark_data_smithi159_139342_object27281:head# 2024-04-15T18:34:42.613 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad29dc3:::benchmark_data_smithi159_139342_object52423:head# 2024-04-15T18:34:42.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad30e85:::benchmark_data_smithi159_139342_object40909:head# 2024-04-15T18:34:42.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad3419e:::benchmark_data_smithi159_139342_object37942:head# 2024-04-15T18:34:42.615 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad41199:::benchmark_data_smithi159_139342_object17689:head# 2024-04-15T18:34:42.616 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad5b8ad:::benchmark_data_smithi159_139342_object4349:head# 2024-04-15T18:34:42.617 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad624d4:::benchmark_data_smithi159_139342_object20942:head# 2024-04-15T18:34:42.617 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad68d0e:::benchmark_data_smithi159_139342_object21525:head# 2024-04-15T18:34:42.617 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad84029:::benchmark_data_smithi159_139342_object25946:head# 2024-04-15T18:34:42.618 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cad909ca:::benchmark_data_smithi159_139342_object53294:head# 2024-04-15T18:34:42.618 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cadb4dfa:::benchmark_data_smithi159_139342_object18818:head# 2024-04-15T18:34:42.623 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae05d02:::benchmark_data_smithi159_139342_object16349:head# 2024-04-15T18:34:42.623 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae0e898:::benchmark_data_smithi159_139342_object32408:head# 2024-04-15T18:34:42.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae19962:::benchmark_data_smithi159_139342_object46765:head# 2024-04-15T18:34:42.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae254ef:::benchmark_data_smithi159_139342_object15705:head# 2024-04-15T18:34:42.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae437d0:::benchmark_data_smithi159_139342_object1331:head# 2024-04-15T18:34:42.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae4bc97:::benchmark_data_smithi159_139342_object30310:head# 2024-04-15T18:34:42.624 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae540dc:::benchmark_data_smithi159_139342_object16703:head# 2024-04-15T18:34:42.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae675a2:::benchmark_data_smithi159_139342_object12446:head# 2024-04-15T18:34:42.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae69971:::benchmark_data_smithi159_139342_object44271:head# 2024-04-15T18:34:42.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae70516:::benchmark_data_smithi159_139342_object39707:head# 2024-04-15T18:34:42.625 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae7394b:::benchmark_data_smithi159_139342_object37011:head# 2024-04-15T18:34:42.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae913ea:::benchmark_data_smithi159_139342_object20727:head# 2024-04-15T18:34:42.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cae9ea0d:::benchmark_data_smithi159_139342_object52577:head# 2024-04-15T18:34:42.626 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caea1e4d:::benchmark_data_smithi159_139342_object23649:head# 2024-04-15T18:34:42.627 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caeab72f:::benchmark_data_smithi159_139342_object56495:head# 2024-04-15T18:34:42.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caec6869:::benchmark_data_smithi159_139342_object11541:head# 2024-04-15T18:34:42.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caed5f62:::benchmark_data_smithi159_139342_object4584:head# 2024-04-15T18:34:42.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caef69bf:::benchmark_data_smithi159_139342_object21953:head# 2024-04-15T18:34:42.631 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caef7d62:::benchmark_data_smithi159_139342_object51492:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf0a338:::benchmark_data_smithi159_139342_object55142:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf0b1ec:::benchmark_data_smithi159_139342_object30800:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf46232:::benchmark_data_smithi159_139342_object24821:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf5ad2e:::benchmark_data_smithi159_139342_object8336:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf5f942:::benchmark_data_smithi159_139342_object23442:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf5fa53:::benchmark_data_smithi159_139342_object16169:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caf9cd37:::benchmark_data_smithi159_139342_object34211:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cafbe3a7:::benchmark_data_smithi159_139342_object44856:head# 2024-04-15T18:34:42.632 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cafdc5f8:::benchmark_data_smithi159_139342_object50734:head# 2024-04-15T18:34:42.633 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caff88d5:::benchmark_data_smithi159_139342_object50112:head# 2024-04-15T18:34:42.633 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:caffecca:::benchmark_data_smithi159_139342_object23056:head# 2024-04-15T18:34:42.634 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb00ce2f:::benchmark_data_smithi159_139342_object36558:head# 2024-04-15T18:34:42.634 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb00e676:::benchmark_data_smithi159_139342_object43162:head# 2024-04-15T18:34:42.635 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb0112eb:::benchmark_data_smithi159_139342_object33133:head# 2024-04-15T18:34:42.635 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb01d086:::benchmark_data_smithi159_139342_object8053:head# 2024-04-15T18:34:42.635 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb025bc1:::benchmark_data_smithi159_139342_object32658:head# 2024-04-15T18:34:42.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb0337e1:::benchmark_data_smithi159_139342_object19002:head# 2024-04-15T18:34:42.636 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb03b4cd:::benchmark_data_smithi159_139342_object6027:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb040619:::benchmark_data_smithi159_139342_object56858:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb0765a5:::benchmark_data_smithi159_139342_object51707:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb077066:::benchmark_data_smithi159_139342_object31918:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb093faf:::benchmark_data_smithi159_139342_object4757:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb0af5a7:::benchmark_data_smithi159_139342_object15742:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb12891b:::benchmark_data_smithi159_139342_object12318:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb161b2d:::benchmark_data_smithi159_139342_object16533:head# 2024-04-15T18:34:42.641 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb163cb3:::benchmark_data_smithi159_139342_object28497:head# 2024-04-15T18:34:42.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb16f552:::benchmark_data_smithi159_139342_object54258:head# 2024-04-15T18:34:42.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb178da2:::benchmark_data_smithi159_139342_object12615:head# 2024-04-15T18:34:42.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb18a502:::benchmark_data_smithi159_139342_object5465:head# 2024-04-15T18:34:42.642 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1bca0a:::benchmark_data_smithi159_139342_object10236:head# 2024-04-15T18:34:42.643 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1c2d49:::benchmark_data_smithi159_139342_object51053:head# 2024-04-15T18:34:42.643 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1cd1e0:::benchmark_data_smithi159_139342_object2011:head# 2024-04-15T18:34:42.643 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1e0e78:::benchmark_data_smithi159_139342_object38670:head# 2024-04-15T18:34:42.644 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1f89f2:::benchmark_data_smithi159_139342_object17111:head# 2024-04-15T18:34:42.645 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb1faa3f:::benchmark_data_smithi159_139342_object45983:head# 2024-04-15T18:34:42.649 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb200007:::benchmark_data_smithi159_139342_object49852:head# 2024-04-15T18:34:42.650 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb205d22:::benchmark_data_smithi159_139342_object16463:head# 2024-04-15T18:34:42.650 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb20a22e:::benchmark_data_smithi159_139342_object35801:head# 2024-04-15T18:34:42.651 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb214d8d:::benchmark_data_smithi159_139342_object49099:head# 2024-04-15T18:34:42.652 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb21cec8:::benchmark_data_smithi159_139342_object41005:head# 2024-04-15T18:34:42.652 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb234d5d:::benchmark_data_smithi159_139342_object12353:head# 2024-04-15T18:34:42.652 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb255bc0:::benchmark_data_smithi159_139342_object16331:head# 2024-04-15T18:34:42.655 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb257fe8:::benchmark_data_smithi159_139342_object10687:head# 2024-04-15T18:34:42.658 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb273d01:::benchmark_data_smithi159_139342_object49705:head# 2024-04-15T18:34:42.659 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb29ef5b:::benchmark_data_smithi159_139342_object47364:head# 2024-04-15T18:34:42.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb2c128d:::benchmark_data_smithi159_139342_object2437:head# 2024-04-15T18:34:42.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb2d401c:::benchmark_data_smithi159_139342_object35729:head# 2024-04-15T18:34:42.662 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb2d9d19:::benchmark_data_smithi159_139342_object9378:head# 2024-04-15T18:34:42.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb2dc79a:::benchmark_data_smithi159_139342_object50904:head# 2024-04-15T18:34:42.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb306871:::benchmark_data_smithi159_139342_object12637:head# 2024-04-15T18:34:42.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb306d45:::benchmark_data_smithi159_139342_object15754:head# 2024-04-15T18:34:42.665 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb31c79d:::benchmark_data_smithi159_139342_object12356:head# 2024-04-15T18:34:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3261a2:::benchmark_data_smithi159_139342_object7721:head# 2024-04-15T18:34:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb358ffa:::benchmark_data_smithi159_139342_object49335:head# 2024-04-15T18:34:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb35f901:::benchmark_data_smithi159_139342_object2263:head# 2024-04-15T18:34:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb372f86:::benchmark_data_smithi159_139342_object53049:head# 2024-04-15T18:34:42.673 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb392e48:::benchmark_data_smithi159_139342_object11230:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3a48b1:::benchmark_data_smithi159_139342_object1756:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3acba2:::benchmark_data_smithi159_139342_object36722:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3c1f6b:::benchmark_data_smithi159_139342_object22342:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3d669e:::benchmark_data_smithi159_139342_object43401:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb3f0db8:::benchmark_data_smithi159_139342_object24915:head# 2024-04-15T18:34:42.674 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb40617e:::benchmark_data_smithi159_139342_object10129:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb418fbc:::benchmark_data_smithi159_139342_object5021:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb41dae4:::benchmark_data_smithi159_139342_object40679:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb42dbae:::benchmark_data_smithi159_139342_object16043:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb44ab99:::benchmark_data_smithi159_139342_object26970:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb45bfc2:::benchmark_data_smithi159_139342_object36833:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb47234d:::benchmark_data_smithi159_139342_object26681:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb475dd8:::benchmark_data_smithi159_139342_object10466:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb48968b:::benchmark_data_smithi159_139342_object16923:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb48c3e9:::benchmark_data_smithi159_139342_object28549:head# 2024-04-15T18:34:42.687 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb49b34c:::benchmark_data_smithi159_139342_object21934:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb4c900a:::benchmark_data_smithi159_139342_object32596:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb4d2f41:::benchmark_data_smithi159_139342_object4698:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb4ddf7c:::benchmark_data_smithi159_139342_object22963:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb50a65b:::benchmark_data_smithi159_139342_object41266:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb515b34:::benchmark_data_smithi159_139342_object10876:head# 2024-04-15T18:34:42.688 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5189a0:::benchmark_data_smithi159_139342_object33779:head# 2024-04-15T18:34:42.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb531bd3:::benchmark_data_smithi159_139342_object51406:head# 2024-04-15T18:34:42.689 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb53b21f:::benchmark_data_smithi159_139342_object24187:head# 2024-04-15T18:34:42.691 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5567cb:::benchmark_data_smithi159_139342_object39872:head# 2024-04-15T18:34:42.691 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb55a27a:::benchmark_data_smithi159_139342_object13887:head# 2024-04-15T18:34:42.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb560fc5:::benchmark_data_smithi159_139342_object17708:head# 2024-04-15T18:34:42.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb573c7c:::benchmark_data_smithi159_139342_object15957:head# 2024-04-15T18:34:42.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb587324:::benchmark_data_smithi159_139342_object1395:head# 2024-04-15T18:34:42.692 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb58773e:::benchmark_data_smithi159_139342_object54828:head# 2024-04-15T18:34:42.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb58f95a:::benchmark_data_smithi159_139342_object14397:head# 2024-04-15T18:34:42.694 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb599034:::benchmark_data_smithi159_139342_object50992:head# 2024-04-15T18:34:42.695 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5a0327:::benchmark_data_smithi159_139342_object15959:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5a6300:::benchmark_data_smithi159_139342_object30997:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5bacc8:::benchmark_data_smithi159_139342_object2957:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5bb5c3:::benchmark_data_smithi159_139342_object49487:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb5d20d7:::benchmark_data_smithi159_139342_object36507:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb617ec6:::benchmark_data_smithi159_139342_object17811:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb61befd:::benchmark_data_smithi159_139342_object26090:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb633551:::benchmark_data_smithi159_139342_object30540:head# 2024-04-15T18:34:42.699 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb663e00:::benchmark_data_smithi159_139342_object5923:head# 2024-04-15T18:34:42.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb680fcd:::benchmark_data_smithi159_139342_object55172:head# 2024-04-15T18:34:42.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb68334e:::benchmark_data_smithi159_139342_object31965:head# 2024-04-15T18:34:42.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb68a7ed:::benchmark_data_smithi159_139342_object3769:head# 2024-04-15T18:34:42.700 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb691b0c:::benchmark_data_smithi159_139342_object54534:head# 2024-04-15T18:34:42.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb694dd4:::benchmark_data_smithi159_139342_object3001:head# 2024-04-15T18:34:42.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb6a39b5:::benchmark_data_smithi159_139342_object35012:head# 2024-04-15T18:34:42.701 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb6a5d8c:::benchmark_data_smithi159_139342_object47352:head# 2024-04-15T18:34:42.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb6bc0ba:::benchmark_data_smithi159_139342_object16675:head# 2024-04-15T18:34:42.702 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb6ee8cb:::benchmark_data_smithi159_139342_object49148:head# 2024-04-15T18:34:42.703 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb6f94fe:::benchmark_data_smithi159_139342_object12727:head# 2024-04-15T18:34:42.709 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7098ce:::benchmark_data_smithi159_139342_object49118:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb70fccb:::benchmark_data_smithi159_139342_object43390:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb728968:::benchmark_data_smithi159_139342_object29952:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb730f79:::benchmark_data_smithi159_139342_object48716:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb757892:::benchmark_data_smithi159_139342_object25248:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb794ad6:::benchmark_data_smithi159_139342_object30867:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7a5f3a:::benchmark_data_smithi159_139342_object9190:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7a8b67:::benchmark_data_smithi159_139342_object40123:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7aff93:::benchmark_data_smithi159_139342_object2938:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7b6047:::benchmark_data_smithi159_139342_object56980:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7b9eae:::benchmark_data_smithi159_139342_object47903:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7c3cd5:::benchmark_data_smithi159_139342_object22857:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7eb7a3:::benchmark_data_smithi159_139342_object22996:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb7ffb47:::benchmark_data_smithi159_139342_object49842:head# 2024-04-15T18:34:42.710 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8165a1:::benchmark_data_smithi159_139342_object32522:head# 2024-04-15T18:34:42.711 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb833d00:::benchmark_data_smithi159_139342_object4527:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb83eecd:::benchmark_data_smithi159_139342_object32618:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb841b62:::benchmark_data_smithi159_139342_object40441:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb85fc6a:::benchmark_data_smithi159_139342_object55484:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb86358f:::benchmark_data_smithi159_139342_object56534:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb86d4aa:::benchmark_data_smithi159_139342_object6447:head# 2024-04-15T18:34:42.712 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8c7ce3:::benchmark_data_smithi159_139342_object34016:head# 2024-04-15T18:34:42.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8c7e26:::benchmark_data_smithi159_139342_object47870:head# 2024-04-15T18:34:42.713 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8d9660:::benchmark_data_smithi159_139342_object29730:head# 2024-04-15T18:34:42.714 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8e0d6a:::benchmark_data_smithi159_139342_object49380:head# 2024-04-15T18:34:42.714 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8e1867:::benchmark_data_smithi159_139342_object21726:head# 2024-04-15T18:34:42.715 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8e5dfe:::benchmark_data_smithi159_139342_object50010:head# 2024-04-15T18:34:42.715 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8e9da0:::benchmark_data_smithi159_139342_object17596:head# 2024-04-15T18:34:42.716 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb8ff95c:::benchmark_data_smithi159_139342_object33980:head# 2024-04-15T18:34:42.716 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9015e9:::benchmark_data_smithi159_139342_object35427:head# 2024-04-15T18:34:42.716 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb904bad:::benchmark_data_smithi159_139342_object56501:head# 2024-04-15T18:34:42.717 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9062ea:::benchmark_data_smithi159_139342_object15138:head# 2024-04-15T18:34:42.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb914eac:::benchmark_data_smithi159_139342_object21731:head# 2024-04-15T18:34:42.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb915641:::benchmark_data_smithi159_139342_object5735:head# 2024-04-15T18:34:42.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb92f6d8:::benchmark_data_smithi159_139342_object9121:head# 2024-04-15T18:34:42.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb945b31:::benchmark_data_smithi159_139342_object22010:head# 2024-04-15T18:34:42.720 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb953546:::benchmark_data_smithi159_139342_object11365:head# 2024-04-15T18:34:42.721 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb953dae:::benchmark_data_smithi159_139342_object24119:head# 2024-04-15T18:34:42.721 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb961cac:::benchmark_data_smithi159_139342_object14293:head# 2024-04-15T18:34:42.722 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb966afd:::benchmark_data_smithi159_139342_object14466:head# 2024-04-15T18:34:42.723 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9684d0:::benchmark_data_smithi159_139342_object48518:head# 2024-04-15T18:34:42.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9918c1:::benchmark_data_smithi159_139342_object2841:head# 2024-04-15T18:34:42.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9b4858:::benchmark_data_smithi159_139342_object31992:head# 2024-04-15T18:34:42.726 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cb9cf83a:::benchmark_data_smithi159_139342_object37715:head# 2024-04-15T18:34:42.727 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba0585e:::benchmark_data_smithi159_139342_object19289:head# 2024-04-15T18:34:42.728 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba0ba3c:::benchmark_data_smithi159_139342_object50092:head# 2024-04-15T18:34:42.728 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba43567:::benchmark_data_smithi159_139342_object42782:head# 2024-04-15T18:34:42.730 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba4add7:::benchmark_data_smithi159_139342_object15488:head# 2024-04-15T18:34:42.730 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba63f70:::benchmark_data_smithi159_139342_object50634:head# 2024-04-15T18:34:42.730 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba64d41:::benchmark_data_smithi159_139342_object23387:head# 2024-04-15T18:34:42.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba89038:::benchmark_data_smithi159_139342_object30618:head# 2024-04-15T18:34:42.731 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cba90298:::benchmark_data_smithi159_139342_object52057:head# 2024-04-15T18:34:42.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbaa5438:::benchmark_data_smithi159_139342_object46585:head# 2024-04-15T18:34:42.735 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbaac7da:::benchmark_data_smithi159_139342_object44177:head# 2024-04-15T18:34:42.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbaadd82:::benchmark_data_smithi159_139342_object56518:head# 2024-04-15T18:34:42.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbab64ec:::benchmark_data_smithi159_139342_object50145:head# 2024-04-15T18:34:42.736 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbacd5ea:::benchmark_data_smithi159_139342_object36784:head# 2024-04-15T18:34:42.737 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbacdd7f:::benchmark_data_smithi159_139342_object26850:head# 2024-04-15T18:34:42.739 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbae1f0d:::benchmark_data_smithi159_139342_object13525:head# 2024-04-15T18:34:42.739 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbae8071:::benchmark_data_smithi159_139342_object9019:head# 2024-04-15T18:34:42.740 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbaf5e3b:::benchmark_data_smithi159_139342_object29838:head# 2024-04-15T18:34:42.740 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb23a5c:::benchmark_data_smithi159_139342_object39418:head# 2024-04-15T18:34:42.740 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb2a7cf:::benchmark_data_smithi159_139342_object26521:head# 2024-04-15T18:34:42.740 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb4ed87:::benchmark_data_smithi159_139342_object19600:head# 2024-04-15T18:34:42.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb5a0df:::benchmark_data_smithi159_139342_object54990:head# 2024-04-15T18:34:42.741 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb670a9:::benchmark_data_smithi159_139342_object45520:head# 2024-04-15T18:34:42.742 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb6da44:::benchmark_data_smithi159_139342_object9659:head# 2024-04-15T18:34:42.742 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb700be:::benchmark_data_smithi159_139342_object30979:head# 2024-04-15T18:34:42.743 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb74742:::benchmark_data_smithi159_139342_object1603:head# 2024-04-15T18:34:42.743 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb7bd9d:::benchmark_data_smithi159_139342_object27451:head# 2024-04-15T18:34:42.743 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb8c627:::benchmark_data_smithi159_139342_object8029:head# 2024-04-15T18:34:42.744 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb8ddcc:::benchmark_data_smithi159_139342_object24929:head# 2024-04-15T18:34:42.744 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbb9a7ae:::benchmark_data_smithi159_139342_object32897:head# 2024-04-15T18:34:42.744 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbbb2441:::benchmark_data_smithi159_139342_object41470:head# 2024-04-15T18:34:42.745 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbbdffe3:::benchmark_data_smithi159_139342_object24049:head# 2024-04-15T18:34:42.745 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbbeb714:::benchmark_data_smithi159_139342_object1089:head# 2024-04-15T18:34:42.746 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbbf3885:::benchmark_data_smithi159_139342_object16290:head# 2024-04-15T18:34:42.746 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc16947:::benchmark_data_smithi159_139342_object29101:head# 2024-04-15T18:34:42.747 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc2d532:::benchmark_data_smithi159_139342_object16188:head# 2024-04-15T18:34:42.747 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc2ef95:::benchmark_data_smithi159_139342_object13111:head# 2024-04-15T18:34:42.747 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc3deb4:::benchmark_data_smithi159_139342_object32300:head# 2024-04-15T18:34:42.748 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc4c257:::benchmark_data_smithi159_139342_object14318:head# 2024-04-15T18:34:42.748 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc533e9:::benchmark_data_smithi159_139342_object35397:head# 2024-04-15T18:34:42.749 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc5a820:::benchmark_data_smithi159_139342_object6151:head# 2024-04-15T18:34:42.749 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc5bbff:::benchmark_data_smithi159_139342_object49885:head# 2024-04-15T18:34:42.749 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc7af31:::benchmark_data_smithi159_139342_object5394:head# 2024-04-15T18:34:42.750 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc93026:::benchmark_data_smithi159_139342_object5451:head# 2024-04-15T18:34:42.750 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc95ba4:::benchmark_data_smithi159_139342_object35768:head# 2024-04-15T18:34:42.751 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbc9aea9:::benchmark_data_smithi159_139342_object56076:head# 2024-04-15T18:34:42.751 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbcaf03c:::benchmark_data_smithi159_139342_object5321:head# 2024-04-15T18:34:42.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbcc128f:::benchmark_data_smithi159_139342_object34167:head# 2024-04-15T18:34:42.753 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbccfb6c:::benchmark_data_smithi159_139342_object24054:head# 2024-04-15T18:34:42.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbcecd43:::benchmark_data_smithi159_139342_object31872:head# 2024-04-15T18:34:42.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd04f72:::benchmark_data_smithi159_139342_object32180:head# 2024-04-15T18:34:42.754 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd23254:::benchmark_data_smithi159_139342_object29235:head# 2024-04-15T18:34:42.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd37467:::benchmark_data_smithi159_139342_object9726:head# 2024-04-15T18:34:42.755 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd4fa3c:::benchmark_data_smithi159_139342_object18094:head# 2024-04-15T18:34:42.756 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd5ba41:::benchmark_data_smithi159_139342_object2827:head# 2024-04-15T18:34:42.757 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd5d9af:::benchmark_data_smithi159_139342_object3479:head# 2024-04-15T18:34:42.757 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd64f5c:::benchmark_data_smithi159_139342_object33178:head# 2024-04-15T18:34:42.758 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbd8563c:::benchmark_data_smithi159_139342_object26928:head# 2024-04-15T18:34:42.758 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbdcb179:::benchmark_data_smithi159_139342_object37379:head# 2024-04-15T18:34:42.760 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbdcce3e:::benchmark_data_smithi159_139342_object21184:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbde0129:::benchmark_data_smithi159_139342_object16795:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbde0a96:::benchmark_data_smithi159_139342_object9446:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbdffe32:::benchmark_data_smithi159_139342_object46785:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe11b60:::benchmark_data_smithi159_139342_object18614:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe1eb56:::benchmark_data_smithi159_139342_object56690:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe2212a:::benchmark_data_smithi159_139342_object40346:head# 2024-04-15T18:34:42.763 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe2a6b5:::benchmark_data_smithi159_139342_object33101:head# 2024-04-15T18:34:42.764 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe3819f:::benchmark_data_smithi159_139342_object21328:head# 2024-04-15T18:34:42.764 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe46078:::benchmark_data_smithi159_139342_object50191:head# 2024-04-15T18:34:42.764 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe4a466:::benchmark_data_smithi159_139342_object52808:head# 2024-04-15T18:34:42.764 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe56459:::benchmark_data_smithi159_139342_object47124:head# 2024-04-15T18:34:42.764 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe62217:::benchmark_data_smithi159_139342_object1484:head# 2024-04-15T18:34:42.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe6f788:::benchmark_data_smithi159_139342_object36635:head# 2024-04-15T18:34:42.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe70e04:::benchmark_data_smithi159_139342_object23599:head# 2024-04-15T18:34:42.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbe9348b:::benchmark_data_smithi159_139342_object6938:head# 2024-04-15T18:34:42.769 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbeb1e6d:::benchmark_data_smithi159_139342_object44412:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbede4af:::benchmark_data_smithi159_139342_object6451:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbee0b9d:::benchmark_data_smithi159_139342_object39032:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf048a8:::benchmark_data_smithi159_139342_object25690:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf08515:::benchmark_data_smithi159_139342_object46746:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf1c162:::benchmark_data_smithi159_139342_object7577:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf1f239:::benchmark_data_smithi159_139342_object8290:head# 2024-04-15T18:34:42.770 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf38988:::benchmark_data_smithi159_139342_object4622:head# 2024-04-15T18:34:42.774 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf3c2a2:::benchmark_data_smithi159_139342_object31361:head# 2024-04-15T18:34:42.775 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf45167:::benchmark_data_smithi159_139342_object26725:head# 2024-04-15T18:34:42.775 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf5c789:::benchmark_data_smithi159_139342_object36895:head# 2024-04-15T18:34:42.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf703d0:::benchmark_data_smithi159_139342_object10666:head# 2024-04-15T18:34:42.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf99cd4:::benchmark_data_smithi159_139342_object34214:head# 2024-04-15T18:34:42.776 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbf9b79b:::benchmark_data_smithi159_139342_object37460:head# 2024-04-15T18:34:42.777 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbfae2eb:::benchmark_data_smithi159_139342_object52836:head# 2024-04-15T18:34:42.777 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbfb1802:::benchmark_data_smithi159_139342_object56170:head# 2024-04-15T18:34:42.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbfdf23a:::benchmark_data_smithi159_139342_object24214:head# 2024-04-15T18:34:42.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbfecccb:::benchmark_data_smithi159_139342_object39836:head# 2024-04-15T18:34:42.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbff042a:::benchmark_data_smithi159_139342_object40582:head# 2024-04-15T18:34:42.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cbffae4a:::benchmark_data_smithi159_139342_object37372:head# 2024-04-15T18:34:42.781 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc002228:::benchmark_data_smithi159_139342_object39961:head# 2024-04-15T18:34:42.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc00f451:::benchmark_data_smithi159_139342_object23489:head# 2024-04-15T18:34:42.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc016e39:::benchmark_data_smithi159_139342_object39480:head# 2024-04-15T18:34:42.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc02b188:::benchmark_data_smithi159_139342_object33046:head# 2024-04-15T18:34:42.782 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc03958c:::benchmark_data_smithi159_139342_object45076:head# 2024-04-15T18:34:42.784 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc039928:::benchmark_data_smithi159_139342_object21795:head# 2024-04-15T18:34:42.784 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc04423a:::benchmark_data_smithi159_139342_object37714:head# 2024-04-15T18:34:42.785 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc0804fb:::benchmark_data_smithi159_139342_object32689:head# 2024-04-15T18:34:42.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc08e577:::benchmark_data_smithi159_139342_object26832:head# 2024-04-15T18:34:42.827 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc0947bd:::benchmark_data_smithi159_139342_object5781:head# 2024-04-15T18:34:42.828 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc09a108:::benchmark_data_smithi159_139342_object24991:head# 2024-04-15T18:34:42.828 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc09c78d:::benchmark_data_smithi159_139342_object3578:head# 2024-04-15T18:34:42.829 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc0f1f16:::benchmark_data_smithi159_139342_object47601:head# 2024-04-15T18:34:42.829 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc12cffc:::benchmark_data_smithi159_139342_object28005:head# 2024-04-15T18:34:42.830 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc15642d:::benchmark_data_smithi159_139342_object3873:head# 2024-04-15T18:34:42.830 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc173ec2:::benchmark_data_smithi159_139342_object12047:head# 2024-04-15T18:34:42.830 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc176838:::benchmark_data_smithi159_139342_object28278:head# 2024-04-15T18:34:42.831 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc176c0d:::benchmark_data_smithi159_139342_object34877:head# 2024-04-15T18:34:42.831 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc187f31:::benchmark_data_smithi159_139342_object15089:head# 2024-04-15T18:34:42.832 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc19e32f:::benchmark_data_smithi159_139342_object49133:head# 2024-04-15T18:34:42.832 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1af9d1:::benchmark_data_smithi159_139342_object40473:head# 2024-04-15T18:34:42.833 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1b32b1:::benchmark_data_smithi159_139342_object43387:head# 2024-04-15T18:34:42.833 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1c4dee:::benchmark_data_smithi159_139342_object49269:head# 2024-04-15T18:34:42.834 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1c8025:::benchmark_data_smithi159_139342_object2777:head# 2024-04-15T18:34:42.834 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1cc3bf:::benchmark_data_smithi159_139342_object19760:head# 2024-04-15T18:34:42.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1e0c0f:::benchmark_data_smithi159_139342_object18212:head# 2024-04-15T18:34:42.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1f5a57:::benchmark_data_smithi159_139342_object20292:head# 2024-04-15T18:34:42.835 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc1fb540:::benchmark_data_smithi159_139342_object8471:head# 2024-04-15T18:34:42.836 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc20193a:::benchmark_data_smithi159_139342_object50230:head# 2024-04-15T18:34:42.836 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc209053:::benchmark_data_smithi159_139342_object45418:head# 2024-04-15T18:34:42.837 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2102aa:::benchmark_data_smithi159_139342_object28093:head# 2024-04-15T18:34:42.837 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc236703:::benchmark_data_smithi159_139342_object7359:head# 2024-04-15T18:34:42.838 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc23a371:::benchmark_data_smithi159_139342_object57265:head# 2024-04-15T18:34:42.838 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2494e9:::benchmark_data_smithi159_139342_object36347:head# 2024-04-15T18:34:42.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2531ba:::benchmark_data_smithi159_139342_object40365:head# 2024-04-15T18:34:42.839 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc278ab3:::benchmark_data_smithi159_139342_object6217:head# 2024-04-15T18:34:42.840 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc280202:::benchmark_data_smithi159_139342_object40369:head# 2024-04-15T18:34:42.840 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc29c2ab:::benchmark_data_smithi159_139342_object35251:head# 2024-04-15T18:34:42.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2a6e13:::benchmark_data_smithi159_139342_object38127:head# 2024-04-15T18:34:42.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2acb12:::benchmark_data_smithi159_139342_object21510:head# 2024-04-15T18:34:42.841 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc2d6046:::benchmark_data_smithi159_139342_object28759:head# 2024-04-15T18:34:42.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc30a879:::benchmark_data_smithi159_139342_object1196:head# 2024-04-15T18:34:42.842 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc31577a:::benchmark_data_smithi159_139342_object28813:head# 2024-04-15T18:34:42.843 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc31a191:::benchmark_data_smithi159_139342_object6095:head# 2024-04-15T18:34:42.843 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc32c10a:::benchmark_data_smithi159_139342_object39380:head# 2024-04-15T18:34:42.844 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc356bd7:::benchmark_data_smithi159_139342_object53373:head# 2024-04-15T18:34:42.844 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc35c103:::benchmark_data_smithi159_139342_object48054:head# 2024-04-15T18:34:42.847 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc35df15:::benchmark_data_smithi159_139342_object28613:head# 2024-04-15T18:34:42.847 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc363d44:::benchmark_data_smithi159_139342_object11248:head# 2024-04-15T18:34:42.848 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc37b4e8:::benchmark_data_smithi159_139342_object29260:head# 2024-04-15T18:34:42.848 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc395cd6:::benchmark_data_smithi159_139342_object2810:head# 2024-04-15T18:34:42.848 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc3c67f2:::benchmark_data_smithi159_139342_object56558:head# 2024-04-15T18:34:42.849 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc3c68ae:::benchmark_data_smithi159_139342_object13405:head# 2024-04-15T18:34:42.849 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc3e53ce:::benchmark_data_smithi159_139342_object55510:head# 2024-04-15T18:34:42.850 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc3f5489:::benchmark_data_smithi159_139342_object1902:head# 2024-04-15T18:34:42.850 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc402fde:::benchmark_data_smithi159_139342_object5182:head# 2024-04-15T18:34:42.851 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4031c3:::benchmark_data_smithi159_139342_object49135:head# 2024-04-15T18:34:42.851 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc408cf4:::benchmark_data_smithi159_139342_object9420:head# 2024-04-15T18:34:42.852 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4114c2:::benchmark_data_smithi159_139342_object35889:head# 2024-04-15T18:34:42.852 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4118d1:::benchmark_data_smithi159_139342_object20461:head# 2024-04-15T18:34:42.852 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc414a22:::benchmark_data_smithi159_139342_object13347:head# 2024-04-15T18:34:42.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc41f90c:::benchmark_data_smithi159_139342_object33878:head# 2024-04-15T18:34:42.853 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc42cc81:::benchmark_data_smithi159_139342_object53228:head# 2024-04-15T18:34:42.854 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc438983:::benchmark_data_smithi159_139342_object41980:head# 2024-04-15T18:34:42.854 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc473e91:::benchmark_data_smithi159_139342_object46079:head# 2024-04-15T18:34:42.855 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc47dd7a:::benchmark_data_smithi159_139342_object50444:head# 2024-04-15T18:34:42.855 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc48c777:::benchmark_data_smithi159_139342_object24789:head# 2024-04-15T18:34:42.856 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4af7c4:::benchmark_data_smithi159_139342_object26006:head# 2024-04-15T18:34:42.858 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4b814f:::benchmark_data_smithi159_139342_object4078:head# 2024-04-15T18:34:42.858 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc4cd124:::benchmark_data_smithi159_139342_object24189:head# 2024-04-15T18:34:42.859 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc51bb23:::benchmark_data_smithi159_139342_object29265:head# 2024-04-15T18:34:42.859 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc532854:::benchmark_data_smithi159_139342_object39916:head# 2024-04-15T18:34:42.860 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5357b2:::benchmark_data_smithi159_139342_object38031:head# 2024-04-15T18:34:42.860 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc53907a:::benchmark_data_smithi159_139342_object26755:head# 2024-04-15T18:34:42.861 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc545f73:::benchmark_data_smithi159_139342_object39910:head# 2024-04-15T18:34:42.861 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc581c10:::benchmark_data_smithi159_139342_object25064:head# 2024-04-15T18:34:42.862 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc58824d:::benchmark_data_smithi159_139342_object12326:head# 2024-04-15T18:34:42.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5a05c8:::benchmark_data_smithi159_139342_object5287:head# 2024-04-15T18:34:42.863 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5b761e:::benchmark_data_smithi159_139342_object1155:head# 2024-04-15T18:34:42.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5ca2ef:::benchmark_data_smithi159_139342_object5890:head# 2024-04-15T18:34:42.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5cd459:::benchmark_data_smithi159_139342_object13643:head# 2024-04-15T18:34:42.864 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5d5c7b:::benchmark_data_smithi159_139342_object36807:head# 2024-04-15T18:34:42.865 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5eec2a:::benchmark_data_smithi159_139342_object34866:head# 2024-04-15T18:34:42.865 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc5f7dc3:::benchmark_data_smithi159_139342_object21850:head# 2024-04-15T18:34:42.866 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc63adaf:::benchmark_data_smithi159_139342_object29248:head# 2024-04-15T18:34:42.866 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc66a4be:::benchmark_data_smithi159_139342_object34659:head# 2024-04-15T18:34:42.866 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc66fc9d:::benchmark_data_smithi159_139342_object50277:head# 2024-04-15T18:34:42.867 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc67e4ce:::benchmark_data_smithi159_139342_object26562:head# 2024-04-15T18:34:42.867 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc68d6f2:::benchmark_data_smithi159_139342_object21251:head# 2024-04-15T18:34:42.868 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6901f3:::benchmark_data_smithi159_139342_object18276:head# 2024-04-15T18:34:42.868 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6c57e1:::benchmark_data_smithi159_139342_object54773:head# 2024-04-15T18:34:42.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6ca681:::benchmark_data_smithi159_139342_object13777:head# 2024-04-15T18:34:42.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6ddb8c:::benchmark_data_smithi159_139342_object28323:head# 2024-04-15T18:34:42.869 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6f5931:::benchmark_data_smithi159_139342_object26989:head# 2024-04-15T18:34:42.870 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc6f9123:::benchmark_data_smithi159_139342_object49791:head# 2024-04-15T18:34:42.870 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc707c8b:::benchmark_data_smithi159_139342_object47419:head# 2024-04-15T18:34:42.870 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc72ef9a:::benchmark_data_smithi159_139342_object53752:head# 2024-04-15T18:34:42.871 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7756bc:::benchmark_data_smithi159_139342_object36105:head# 2024-04-15T18:34:42.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc781200:::benchmark_data_smithi159_139342_object2731:head# 2024-04-15T18:34:42.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7acc2d:::benchmark_data_smithi159_139342_object52561:head# 2024-04-15T18:34:42.872 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7bcacb:::benchmark_data_smithi159_139342_object53759:head# 2024-04-15T18:34:42.873 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7d17f5:::benchmark_data_smithi159_139342_object46824:head# 2024-04-15T18:34:42.873 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7d6350:::benchmark_data_smithi159_139342_object30615:head# 2024-04-15T18:34:42.874 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7e426d:::benchmark_data_smithi159_139342_object33752:head# 2024-04-15T18:34:42.874 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7e942f:::benchmark_data_smithi159_139342_object14972:head# 2024-04-15T18:34:42.875 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc7feda2:::benchmark_data_smithi159_139342_object2187:head# 2024-04-15T18:34:42.875 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8272ef:::benchmark_data_smithi159_139342_object47475:head# 2024-04-15T18:34:42.876 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc84a1f2:::benchmark_data_smithi159_139342_object41289:head# 2024-04-15T18:34:42.876 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc84bc74:::benchmark_data_smithi159_139342_object51041:head# 2024-04-15T18:34:42.877 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8624d6:::benchmark_data_smithi159_139342_object38544:head# 2024-04-15T18:34:42.877 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc87ed1b:::benchmark_data_smithi159_139342_object38716:head# 2024-04-15T18:34:42.877 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc890822:::benchmark_data_smithi159_139342_object13538:head# 2024-04-15T18:34:42.878 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8a4f19:::benchmark_data_smithi159_139342_object4288:head# 2024-04-15T18:34:42.879 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8b4dc1:::benchmark_data_smithi159_139342_object1533:head# 2024-04-15T18:34:42.879 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8d67a7:::benchmark_data_smithi159_139342_object8835:head# 2024-04-15T18:34:42.879 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8f2353:::benchmark_data_smithi159_139342_object9702:head# 2024-04-15T18:34:42.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8f3512:::benchmark_data_smithi159_139342_object33042:head# 2024-04-15T18:34:42.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8f6942:::benchmark_data_smithi159_139342_object38059:head# 2024-04-15T18:34:42.880 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc8faaa3:::benchmark_data_smithi159_139342_object23134:head# 2024-04-15T18:34:42.881 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc902789:::benchmark_data_smithi159_139342_object33728:head# 2024-04-15T18:34:42.881 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc906041:::benchmark_data_smithi159_139342_object45012:head# 2024-04-15T18:34:42.882 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc90ca9a:::benchmark_data_smithi159_139342_object21240:head# 2024-04-15T18:34:42.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9175d1:::benchmark_data_smithi159_139342_object55651:head# 2024-04-15T18:34:42.883 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc91cfe0:::benchmark_data_smithi159_139342_object9916:head# 2024-04-15T18:34:42.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc91e38d:::benchmark_data_smithi159_139342_object33559:head# 2024-04-15T18:34:42.884 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc930cea:::benchmark_data_smithi159_139342_object25466:head# 2024-04-15T18:34:42.885 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9320ee:::benchmark_data_smithi159_139342_object14064:head# 2024-04-15T18:34:42.888 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc95d3b6:::benchmark_data_smithi159_139342_object14003:head# 2024-04-15T18:34:42.891 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc96beb7:::benchmark_data_smithi159_139342_object38089:head# 2024-04-15T18:34:42.891 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc994c96:::benchmark_data_smithi159_139342_object10266:head# 2024-04-15T18:34:42.891 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9a11b7:::benchmark_data_smithi159_139342_object27522:head# 2024-04-15T18:34:42.892 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9b83bb:::benchmark_data_smithi159_139342_object18299:head# 2024-04-15T18:34:42.892 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9bb2b2:::benchmark_data_smithi159_139342_object6072:head# 2024-04-15T18:34:42.893 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9be1b8:::benchmark_data_smithi159_139342_object15985:head# 2024-04-15T18:34:42.893 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9c91c1:::benchmark_data_smithi159_139342_object15824:head# 2024-04-15T18:34:42.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9d7cf7:::benchmark_data_smithi159_139342_object3669:head# 2024-04-15T18:34:42.894 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9e1b8d:::benchmark_data_smithi159_139342_object44042:head# 2024-04-15T18:34:42.895 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9ea4a6:::benchmark_data_smithi159_139342_object23919:head# 2024-04-15T18:34:42.895 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cc9f1880:::benchmark_data_smithi159_139342_object16364:head# 2024-04-15T18:34:42.896 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca03f0f:::benchmark_data_smithi159_139342_object36177:head# 2024-04-15T18:34:42.896 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca1accf:::benchmark_data_smithi159_139342_object47466:head# 2024-04-15T18:34:42.896 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca36c03:::benchmark_data_smithi159_139342_object36701:head# 2024-04-15T18:34:42.897 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca46b61:::benchmark_data_smithi159_139342_object977:head# 2024-04-15T18:34:42.897 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca56360:::benchmark_data_smithi159_139342_object24059:head# 2024-04-15T18:34:42.898 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca56aa0:::benchmark_data_smithi159_139342_object35052:head# 2024-04-15T18:34:42.898 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca64b62:::benchmark_data_smithi159_139342_object41833:head# 2024-04-15T18:34:42.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca69be2:::benchmark_data_smithi159_139342_object24807:head# 2024-04-15T18:34:42.899 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca702fa:::benchmark_data_smithi159_139342_object4168:head# 2024-04-15T18:34:42.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca83bd8:::benchmark_data_smithi159_139342_object42206:head# 2024-04-15T18:34:42.900 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cca86e80:::benchmark_data_smithi159_139342_object53237:head# 2024-04-15T18:34:42.901 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccabf0c8:::benchmark_data_smithi159_139342_object9868:head# 2024-04-15T18:34:42.904 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb07abf:::benchmark_data_smithi159_139342_object54819:head# 2024-04-15T18:34:42.905 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb143dd:::benchmark_data_smithi159_139342_object17710:head# 2024-04-15T18:34:42.905 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb23c46:::benchmark_data_smithi159_139342_object2185:head# 2024-04-15T18:34:42.906 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb39db6:::benchmark_data_smithi159_139342_object31042:head# 2024-04-15T18:34:42.906 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb47245:::benchmark_data_smithi159_139342_object40175:head# 2024-04-15T18:34:42.906 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb53343:::benchmark_data_smithi159_139342_object7813:head# 2024-04-15T18:34:42.907 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb55b05:::benchmark_data_smithi159_139342_object13905:head# 2024-04-15T18:34:42.907 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb5a368:::benchmark_data_smithi159_139342_object35921:head# 2024-04-15T18:34:42.908 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb60804:::benchmark_data_smithi159_139342_object46293:head# 2024-04-15T18:34:42.908 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb7956f:::benchmark_data_smithi159_139342_object46117:head# 2024-04-15T18:34:42.909 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccb7b8af:::benchmark_data_smithi159_139342_object27927:head# 2024-04-15T18:34:42.910 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccbceddf:::benchmark_data_smithi159_139342_object33836:head# 2024-04-15T18:34:42.910 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccbfca3b:::benchmark_data_smithi159_139342_object20275:head# 2024-04-15T18:34:42.910 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc082cf:::benchmark_data_smithi159_139342_object4439:head# 2024-04-15T18:34:42.910 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc08d3c:::benchmark_data_smithi159_139342_object55715:head# 2024-04-15T18:34:42.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc0f687:::benchmark_data_smithi159_139342_object23971:head# 2024-04-15T18:34:42.911 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc43f30:::benchmark_data_smithi159_139342_object21902:head# 2024-04-15T18:34:42.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc47019:::benchmark_data_smithi159_139342_object19100:head# 2024-04-15T18:34:42.912 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc52cb5:::benchmark_data_smithi159_139342_object1761:head# 2024-04-15T18:34:42.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc6fc7e:::benchmark_data_smithi159_139342_object42473:head# 2024-04-15T18:34:42.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccc94393:::benchmark_data_smithi159_139342_object37624:head# 2024-04-15T18:34:42.913 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cccb63ff:::benchmark_data_smithi159_139342_object31814:head# 2024-04-15T18:34:42.914 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cccda18b:::benchmark_data_smithi159_139342_object19347:head# 2024-04-15T18:34:42.914 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cccdc40e:::benchmark_data_smithi159_139342_object52713:head# 2024-04-15T18:34:42.915 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccce1f19:::benchmark_data_smithi159_139342_object55999:head# 2024-04-15T18:34:42.915 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccce35b2:::benchmark_data_smithi159_139342_object43300:head# 2024-04-15T18:34:42.916 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd07903:::benchmark_data_smithi159_139342_object56389:head# 2024-04-15T18:34:42.916 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd0c523:::benchmark_data_smithi159_139342_object41867:head# 2024-04-15T18:34:42.916 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd273e2:::benchmark_data_smithi159_139342_object47966:head# 2024-04-15T18:34:42.917 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd671ad:::benchmark_data_smithi159_139342_object8778:head# 2024-04-15T18:34:42.917 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd861e0:::benchmark_data_smithi159_139342_object44322:head# 2024-04-15T18:34:42.917 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd8f846:::benchmark_data_smithi159_139342_object31645:head# 2024-04-15T18:34:42.918 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccd917e5:::benchmark_data_smithi159_139342_object29899:head# 2024-04-15T18:34:42.919 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccdba03f:::benchmark_data_smithi159_139342_object37980:head# 2024-04-15T18:34:42.919 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccdcc1f3:::benchmark_data_smithi159_139342_object45683:head# 2024-04-15T18:34:42.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce16fd0:::benchmark_data_smithi159_139342_object42209:head# 2024-04-15T18:34:42.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce37433:::benchmark_data_smithi159_139342_object22054:head# 2024-04-15T18:34:42.920 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce56857:::benchmark_data_smithi159_139342_object28884:head# 2024-04-15T18:34:42.921 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce57585:::benchmark_data_smithi159_139342_object1211:head# 2024-04-15T18:34:42.921 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce6ffd7:::benchmark_data_smithi159_139342_object16850:head# 2024-04-15T18:34:42.922 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce7d19a:::benchmark_data_smithi159_139342_object46696:head# 2024-04-15T18:34:42.922 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce80a2c:::benchmark_data_smithi159_139342_object36136:head# 2024-04-15T18:34:42.923 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce8aa73:::benchmark_data_smithi159_139342_object50189:head# 2024-04-15T18:34:42.923 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cce9e243:::benchmark_data_smithi159_139342_object6540:head# 2024-04-15T18:34:42.924 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccea0071:::benchmark_data_smithi159_139342_object7079:head# 2024-04-15T18:34:42.925 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccead6b9:::benchmark_data_smithi159_139342_object42078:head# 2024-04-15T18:34:42.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cceb1846:::benchmark_data_smithi159_139342_object45655:head# 2024-04-15T18:34:42.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cced6a47:::benchmark_data_smithi159_139342_object21412:head# 2024-04-15T18:34:42.926 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccee1c5f:::benchmark_data_smithi159_139342_object6193:head# 2024-04-15T18:34:42.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf1a961:::benchmark_data_smithi159_139342_object51669:head# 2024-04-15T18:34:42.927 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf28f7d:::benchmark_data_smithi159_139342_object10693:head# 2024-04-15T18:34:42.934 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf3c1ad:::benchmark_data_smithi159_139342_object47974:head# 2024-04-15T18:34:42.934 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf4436f:::benchmark_data_smithi159_139342_object7061:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf55d2e:::benchmark_data_smithi159_139342_object9370:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf794fc:::benchmark_data_smithi159_139342_object44251:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccf8a014:::benchmark_data_smithi159_139342_object43627:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccfaff2e:::benchmark_data_smithi159_139342_object39861:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ccfe2f9f:::benchmark_data_smithi159_139342_object46937:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd00d2ee:::benchmark_data_smithi159_139342_object14024:head# 2024-04-15T18:34:42.935 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd014f3c:::benchmark_data_smithi159_139342_object12942:head# 2024-04-15T18:34:42.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd02558a:::benchmark_data_smithi159_139342_object51951:head# 2024-04-15T18:34:42.940 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd028bc6:::benchmark_data_smithi159_139342_object800:head# 2024-04-15T18:34:42.944 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd030f6e:::benchmark_data_smithi159_139342_object56875:head# 2024-04-15T18:34:42.945 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd041ff0:::benchmark_data_smithi159_139342_object50654:head# 2024-04-15T18:34:42.946 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd050da1:::benchmark_data_smithi159_139342_object43733:head# 2024-04-15T18:34:42.946 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd093d8b:::benchmark_data_smithi159_139342_object19355:head# 2024-04-15T18:34:42.946 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd0940df:::benchmark_data_smithi159_139342_object26609:head# 2024-04-15T18:34:42.947 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd09efbb:::benchmark_data_smithi159_139342_object55426:head# 2024-04-15T18:34:42.947 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd0d1261:::benchmark_data_smithi159_139342_object54431:head# 2024-04-15T18:34:42.948 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd0f24b6:::benchmark_data_smithi159_139342_object46244:head# 2024-04-15T18:34:42.949 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd0f7aea:::benchmark_data_smithi159_139342_object52145:head# 2024-04-15T18:34:42.949 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd119b7c:::benchmark_data_smithi159_139342_object51841:head# 2024-04-15T18:34:42.949 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd11d6fd:::benchmark_data_smithi159_139342_object51585:head# 2024-04-15T18:34:42.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd11d831:::benchmark_data_smithi159_139342_object47386:head# 2024-04-15T18:34:42.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd12418c:::benchmark_data_smithi159_139342_object36081:head# 2024-04-15T18:34:42.950 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd12e521:::benchmark_data_smithi159_139342_object50511:head# 2024-04-15T18:34:42.951 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd12e9ca:::benchmark_data_smithi159_139342_object53958:head# 2024-04-15T18:34:42.951 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd13f796:::benchmark_data_smithi159_139342_object25642:head# 2024-04-15T18:34:42.952 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd159a21:::benchmark_data_smithi159_139342_object236:head# 2024-04-15T18:34:42.952 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd15f319:::benchmark_data_smithi159_139342_object15435:head# 2024-04-15T18:34:42.952 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd168592:::benchmark_data_smithi159_139342_object43234:head# 2024-04-15T18:34:42.953 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd17401d:::benchmark_data_smithi159_139342_object5651:head# 2024-04-15T18:34:42.956 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1754e9:::benchmark_data_smithi159_139342_object40789:head# 2024-04-15T18:34:42.956 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd17d6eb:::benchmark_data_smithi159_139342_object24303:head# 2024-04-15T18:34:42.956 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd198ef8:::benchmark_data_smithi159_139342_object24302:head# 2024-04-15T18:34:42.956 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd19e956:::benchmark_data_smithi159_139342_object1031:head# 2024-04-15T18:34:42.957 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1b1d26:::benchmark_data_smithi159_139342_object167:head# 2024-04-15T18:34:42.957 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1b6645:::benchmark_data_smithi159_139342_object28520:head# 2024-04-15T18:34:42.957 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1cdc2e:::benchmark_data_smithi159_139342_object10809:head# 2024-04-15T18:34:42.957 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1d04c5:::benchmark_data_smithi159_139342_object42378:head# 2024-04-15T18:34:42.957 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd1d27ca:::benchmark_data_smithi159_139342_object10821:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd2292b5:::benchmark_data_smithi159_139342_object30785:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd22c901:::benchmark_data_smithi159_139342_object35373:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd24308c:::benchmark_data_smithi159_139342_object27796:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd250af8:::benchmark_data_smithi159_139342_object3773:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd275afe:::benchmark_data_smithi159_139342_object4504:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd28b494:::benchmark_data_smithi159_139342_object10527:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd2a0700:::benchmark_data_smithi159_139342_object23838:head# 2024-04-15T18:34:42.961 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd2a243a:::benchmark_data_smithi159_139342_object26264:head# 2024-04-15T18:34:42.962 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd2a7f23:::benchmark_data_smithi159_139342_object54304:head# 2024-04-15T18:34:42.962 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd2c15ca:::benchmark_data_smithi159_139342_object15632:head# 2024-04-15T18:34:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd32a28c:::benchmark_data_smithi159_139342_object40895:head# 2024-04-15T18:34:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd331a25:::benchmark_data_smithi159_139342_object25805:head# 2024-04-15T18:34:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3457f0:::benchmark_data_smithi159_139342_object10741:head# 2024-04-15T18:34:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd347da0:::benchmark_data_smithi159_139342_object20484:head# 2024-04-15T18:34:42.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd350686:::benchmark_data_smithi159_139342_object46499:head# 2024-04-15T18:34:42.964 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd36cedf:::benchmark_data_smithi159_139342_object45814:head# 2024-04-15T18:34:42.965 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3a629a:::benchmark_data_smithi159_139342_object45884:head# 2024-04-15T18:34:42.965 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3a6ea5:::benchmark_data_smithi159_139342_object12507:head# 2024-04-15T18:34:42.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3af870:::benchmark_data_smithi159_139342_object48681:head# 2024-04-15T18:34:42.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3ea9f8:::benchmark_data_smithi159_139342_object41343:head# 2024-04-15T18:34:42.966 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd3ff58e:::benchmark_data_smithi159_139342_object50060:head# 2024-04-15T18:34:42.967 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd40b5a3:::benchmark_data_smithi159_139342_object15929:head# 2024-04-15T18:34:42.968 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd45189d:::benchmark_data_smithi159_139342_object28907:head# 2024-04-15T18:34:42.968 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd477295:::benchmark_data_smithi159_139342_object44670:head# 2024-04-15T18:34:42.969 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd483574:::benchmark_data_smithi159_139342_object29324:head# 2024-04-15T18:34:42.969 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd48a0b3:::benchmark_data_smithi159_139342_object20286:head# 2024-04-15T18:34:42.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd4bb763:::benchmark_data_smithi159_139342_object2949:head# 2024-04-15T18:34:42.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd4c49c8:::benchmark_data_smithi159_139342_object18285:head# 2024-04-15T18:34:42.973 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd4dff85:::benchmark_data_smithi159_139342_object48622:head# 2024-04-15T18:34:42.974 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd52b300:::benchmark_data_smithi159_139342_object1185:head# 2024-04-15T18:34:42.974 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5341bc:::benchmark_data_smithi159_139342_object23230:head# 2024-04-15T18:34:42.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd546a2b:::benchmark_data_smithi159_139342_object3801:head# 2024-04-15T18:34:42.975 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd55257b:::benchmark_data_smithi159_139342_object30290:head# 2024-04-15T18:34:42.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd557cb7:::benchmark_data_smithi159_139342_object56674:head# 2024-04-15T18:34:42.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd56680e:::benchmark_data_smithi159_139342_object52409:head# 2024-04-15T18:34:42.976 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd581bf1:::benchmark_data_smithi159_139342_object19490:head# 2024-04-15T18:34:42.977 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd588692:::benchmark_data_smithi159_139342_object12013:head# 2024-04-15T18:34:42.977 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd58a373:::benchmark_data_smithi159_139342_object24376:head# 2024-04-15T18:34:42.977 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd59fb0b:::benchmark_data_smithi159_139342_object12470:head# 2024-04-15T18:34:42.978 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5aa755:::benchmark_data_smithi159_139342_object29025:head# 2024-04-15T18:34:42.979 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5ace1a:::benchmark_data_smithi159_139342_object44911:head# 2024-04-15T18:34:42.979 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5d768f:::benchmark_data_smithi159_139342_object5439:head# 2024-04-15T18:34:42.979 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5dafd8:::benchmark_data_smithi159_139342_object35203:head# 2024-04-15T18:34:42.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd5e92b9:::benchmark_data_smithi159_139342_object55405:head# 2024-04-15T18:34:42.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6002f1:::benchmark_data_smithi159_139342_object52668:head# 2024-04-15T18:34:42.980 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd65eaa6:::benchmark_data_smithi159_139342_object20240:head# 2024-04-15T18:34:42.981 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6627de:::benchmark_data_smithi159_139342_object47953:head# 2024-04-15T18:34:42.981 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd668cc5:::benchmark_data_smithi159_139342_object37430:head# 2024-04-15T18:34:42.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6900d2:::benchmark_data_smithi159_139342_object28964:head# 2024-04-15T18:34:42.982 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6b9090:::benchmark_data_smithi159_139342_object37425:head# 2024-04-15T18:34:42.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6be731:::benchmark_data_smithi159_139342_object53625:head# 2024-04-15T18:34:42.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6c856b:::benchmark_data_smithi159_139342_object39298:head# 2024-04-15T18:34:42.983 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd6f21fb:::benchmark_data_smithi159_139342_object32571:head# 2024-04-15T18:34:42.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd72233d:::benchmark_data_smithi159_139342_object15568:head# 2024-04-15T18:34:42.984 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd72c256:::benchmark_data_smithi159_139342_object4355:head# 2024-04-15T18:34:42.985 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd73197a:::benchmark_data_smithi159_139342_object49673:head# 2024-04-15T18:34:42.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd743e0d:::benchmark_data_smithi159_139342_object28427:head# 2024-04-15T18:34:42.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd745ddc:::benchmark_data_smithi159_139342_object16454:head# 2024-04-15T18:34:42.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7539ee:::benchmark_data_smithi159_139342_object10577:head# 2024-04-15T18:34:42.987 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd77d481:::benchmark_data_smithi159_139342_object27484:head# 2024-04-15T18:34:42.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd78303c:::benchmark_data_smithi159_139342_object21313:head# 2024-04-15T18:34:42.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7a2696:::benchmark_data_smithi159_139342_object51206:head# 2024-04-15T18:34:42.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7a32b7:::benchmark_data_smithi159_139342_object45410:head# 2024-04-15T18:34:42.988 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7aa218:::benchmark_data_smithi159_139342_object40459:head# 2024-04-15T18:34:42.989 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7b7666:::benchmark_data_smithi159_139342_object37643:head# 2024-04-15T18:34:42.989 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7c827c:::benchmark_data_smithi159_139342_object40438:head# 2024-04-15T18:34:42.989 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7d36bb:::benchmark_data_smithi159_139342_object41121:head# 2024-04-15T18:34:42.990 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7dcaba:::benchmark_data_smithi159_139342_object26299:head# 2024-04-15T18:34:42.991 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7e5e7d:::benchmark_data_smithi159_139342_object35383:head# 2024-04-15T18:34:42.991 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7f2e0d:::benchmark_data_smithi159_139342_object22908:head# 2024-04-15T18:34:42.991 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd7f6e1c:::benchmark_data_smithi159_139342_object43697:head# 2024-04-15T18:34:42.991 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd807431:::benchmark_data_smithi159_139342_object22231:head# 2024-04-15T18:34:42.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd824406:::benchmark_data_smithi159_139342_object18718:head# 2024-04-15T18:34:42.992 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd83530e:::benchmark_data_smithi159_139342_object50016:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd84c173:::benchmark_data_smithi159_139342_object26703:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd867ee6:::benchmark_data_smithi159_139342_object1939:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd86cf8a:::benchmark_data_smithi159_139342_object32584:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd87e30e:::benchmark_data_smithi159_139342_object21982:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd882a25:::benchmark_data_smithi159_139342_object30030:head# 2024-04-15T18:34:42.997 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd894baa:::benchmark_data_smithi159_139342_object4019:head# 2024-04-15T18:34:43.001 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd895e89:::benchmark_data_smithi159_139342_object36358:head# 2024-04-15T18:34:43.002 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd8c32a9:::benchmark_data_smithi159_139342_object4243:head# 2024-04-15T18:34:43.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd8cb42b:::benchmark_data_smithi159_139342_object21311:head# 2024-04-15T18:34:43.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd8d8c3f:::benchmark_data_smithi159_139342_object27857:head# 2024-04-15T18:34:43.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd90de8a:::benchmark_data_smithi159_139342_object48808:head# 2024-04-15T18:34:43.007 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd914eac:::benchmark_data_smithi159_139342_object56113:head# 2024-04-15T18:34:43.008 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd92f30b:::benchmark_data_smithi159_139342_object32565:head# 2024-04-15T18:34:43.008 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd937300:::benchmark_data_smithi159_139342_object34312:head# 2024-04-15T18:34:43.008 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd93ad87:::benchmark_data_smithi159_139342_object20149:head# 2024-04-15T18:34:43.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd93e72c:::benchmark_data_smithi159_139342_object56404:head# 2024-04-15T18:34:43.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd94cb3f:::benchmark_data_smithi159_139342_object36212:head# 2024-04-15T18:34:43.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd95dd24:::benchmark_data_smithi159_139342_object39871:head# 2024-04-15T18:34:43.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd96403d:::benchmark_data_smithi159_139342_object55968:head# 2024-04-15T18:34:43.009 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd973afc:::benchmark_data_smithi159_139342_object30622:head# 2024-04-15T18:34:43.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd985301:::benchmark_data_smithi159_139342_object17328:head# 2024-04-15T18:34:43.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd990d8a:::benchmark_data_smithi159_139342_object11918:head# 2024-04-15T18:34:43.010 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9a7943:::benchmark_data_smithi159_139342_object32209:head# 2024-04-15T18:34:43.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9d8a9e:::benchmark_data_smithi159_139342_object36159:head# 2024-04-15T18:34:43.014 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9d90cc:::benchmark_data_smithi159_139342_object49802:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9edb70:::benchmark_data_smithi159_139342_object7629:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9f056f:::benchmark_data_smithi159_139342_object44331:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cd9f09d5:::benchmark_data_smithi159_139342_object24740:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda33ba5:::benchmark_data_smithi159_139342_object3203:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda421b0:::benchmark_data_smithi159_139342_object38063:head# 2024-04-15T18:34:43.015 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda63727:::benchmark_data_smithi159_139342_object314:head# 2024-04-15T18:34:43.016 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda6e41e:::benchmark_data_smithi159_139342_object13954:head# 2024-04-15T18:34:43.016 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda92ab5:::benchmark_data_smithi159_139342_object31873:head# 2024-04-15T18:34:43.017 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cda9675f:::benchmark_data_smithi159_139342_object15829:head# 2024-04-15T18:34:43.017 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdac8843:::benchmark_data_smithi159_139342_object9418:head# 2024-04-15T18:34:43.017 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdacfe87:::benchmark_data_smithi159_139342_object12795:head# 2024-04-15T18:34:43.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdad50f7:::benchmark_data_smithi159_139342_object37948:head# 2024-04-15T18:34:43.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdae0fd2:::benchmark_data_smithi159_139342_object34795:head# 2024-04-15T18:34:43.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdaee15c:::benchmark_data_smithi159_139342_object53885:head# 2024-04-15T18:34:43.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb0b6df:::benchmark_data_smithi159_139342_object39537:head# 2024-04-15T18:34:43.019 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb0f1bd:::benchmark_data_smithi159_139342_object2921:head# 2024-04-15T18:34:43.020 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb11b6a:::benchmark_data_smithi159_139342_object5488:head# 2024-04-15T18:34:43.020 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb11f15:::benchmark_data_smithi159_139342_object28509:head# 2024-04-15T18:34:43.021 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb1245a:::benchmark_data_smithi159_139342_object7935:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb1cde7:::benchmark_data_smithi159_139342_object22004:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdb27a5d:::benchmark_data_smithi159_139342_object31623:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdba0b95:::benchmark_data_smithi159_139342_object8806:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdbaf754:::benchmark_data_smithi159_139342_object7697:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdbc7d3d:::benchmark_data_smithi159_139342_object16589:head# 2024-04-15T18:34:43.023 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdbd15bb:::benchmark_data_smithi159_139342_object10998:head# 2024-04-15T18:34:43.024 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdbd8a1a:::benchmark_data_smithi159_139342_object16899:head# 2024-04-15T18:34:43.024 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdbec96e:::benchmark_data_smithi159_139342_object1304:head# 2024-04-15T18:34:43.024 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc216bf:::benchmark_data_smithi159_139342_object28127:head# 2024-04-15T18:34:43.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc29242:::benchmark_data_smithi159_139342_object53088:head# 2024-04-15T18:34:43.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc3db3a:::benchmark_data_smithi159_139342_object49721:head# 2024-04-15T18:34:43.025 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc42519:::benchmark_data_smithi159_139342_object22778:head# 2024-04-15T18:34:43.026 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc43f39:::benchmark_data_smithi159_139342_object19620:head# 2024-04-15T18:34:43.029 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc470b2:::benchmark_data_smithi159_139342_object16560:head# 2024-04-15T18:34:43.031 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc51072:::benchmark_data_smithi159_139342_object42858:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc6a90e:::benchmark_data_smithi159_139342_object11542:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc71427:::benchmark_data_smithi159_139342_object38359:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc8c79a:::benchmark_data_smithi159_139342_object18477:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdc94ba6:::benchmark_data_smithi159_139342_object1125:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdcb41e8:::benchmark_data_smithi159_139342_object42969:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdce4169:::benchmark_data_smithi159_139342_object19446:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdceb9a2:::benchmark_data_smithi159_139342_object15873:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdcedb30:::benchmark_data_smithi159_139342_object39101:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdcf0f88:::benchmark_data_smithi159_139342_object11513:head# 2024-04-15T18:34:43.032 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd207b1:::benchmark_data_smithi159_139342_object49377:head# 2024-04-15T18:34:43.033 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd25121:::benchmark_data_smithi159_139342_object18932:head# 2024-04-15T18:34:43.034 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd4643d:::benchmark_data_smithi159_139342_object10846:head# 2024-04-15T18:34:43.035 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd50cb5:::benchmark_data_smithi159_139342_object48201:head# 2024-04-15T18:34:43.035 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd51003:::benchmark_data_smithi159_139342_object19172:head# 2024-04-15T18:34:43.035 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd5627e:::benchmark_data_smithi159_139342_object14200:head# 2024-04-15T18:34:43.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd99ae2:::benchmark_data_smithi159_139342_object4008:head# 2024-04-15T18:34:43.036 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdd9d42c:::benchmark_data_smithi159_139342_object6236:head# 2024-04-15T18:34:43.037 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdda01a6:::benchmark_data_smithi159_139342_object34786:head# 2024-04-15T18:34:43.037 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cddc9969:::benchmark_data_smithi159_139342_object30933:head# 2024-04-15T18:34:43.039 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cddca648:::benchmark_data_smithi159_139342_object14286:head# 2024-04-15T18:34:43.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cddf3c3c:::benchmark_data_smithi159_139342_object31538:head# 2024-04-15T18:34:43.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cddfa985:::benchmark_data_smithi159_139342_object33912:head# 2024-04-15T18:34:43.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde0c64a:::benchmark_data_smithi159_139342_object1948:head# 2024-04-15T18:34:43.040 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde2c207:::benchmark_data_smithi159_139342_object18524:head# 2024-04-15T18:34:43.041 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde2f3e1:::benchmark_data_smithi159_139342_object47782:head# 2024-04-15T18:34:43.042 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde3b901:::benchmark_data_smithi159_139342_object16855:head# 2024-04-15T18:34:43.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde6b896:::benchmark_data_smithi159_139342_object49331:head# 2024-04-15T18:34:43.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde74fe7:::benchmark_data_smithi159_139342_object3854:head# 2024-04-15T18:34:43.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde860a6:::benchmark_data_smithi159_139342_object12667:head# 2024-04-15T18:34:43.044 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cde9b441:::benchmark_data_smithi159_139342_object51071:head# 2024-04-15T18:34:43.048 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cded046f:::benchmark_data_smithi159_139342_object22477:head# 2024-04-15T18:34:43.048 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cded9755:::benchmark_data_smithi159_139342_object11217:head# 2024-04-15T18:34:43.049 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdeebf41:::benchmark_data_smithi159_139342_object35362:head# 2024-04-15T18:34:43.049 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdef0f09:::benchmark_data_smithi159_139342_object32071:head# 2024-04-15T18:34:43.049 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf3257f:::benchmark_data_smithi159_139342_object23917:head# 2024-04-15T18:34:43.050 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf3ab1c:::benchmark_data_smithi159_139342_object39618:head# 2024-04-15T18:34:43.050 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf5eadc:::benchmark_data_smithi159_139342_object17828:head# 2024-04-15T18:34:43.051 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf65834:::benchmark_data_smithi159_139342_object35851:head# 2024-04-15T18:34:43.051 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf683e4:::benchmark_data_smithi159_139342_object47591:head# 2024-04-15T18:34:43.052 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf6eccd:::benchmark_data_smithi159_139342_object50980:head# 2024-04-15T18:34:43.052 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf6f29c:::benchmark_data_smithi159_139342_object26073:head# 2024-04-15T18:34:43.052 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf73ad6:::benchmark_data_smithi159_139342_object1392:head# 2024-04-15T18:34:43.053 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf77e5d:::benchmark_data_smithi159_139342_object29322:head# 2024-04-15T18:34:43.053 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf7df41:::benchmark_data_smithi159_139342_object50596:head# 2024-04-15T18:34:43.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf7f5a2:::benchmark_data_smithi159_139342_object10696:head# 2024-04-15T18:34:43.054 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf83929:::benchmark_data_smithi159_139342_object13530:head# 2024-04-15T18:34:43.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdf86002:::benchmark_data_smithi159_139342_object5590:head# 2024-04-15T18:34:43.055 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdfb50a6:::benchmark_data_smithi159_139342_object55556:head# 2024-04-15T18:34:43.056 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdfd29d9:::benchmark_data_smithi159_139342_object54385:head# 2024-04-15T18:34:43.056 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdfd335a:::benchmark_data_smithi159_139342_object23302:head# 2024-04-15T18:34:43.060 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdfe571c:::benchmark_data_smithi159_139342_object56865:head# 2024-04-15T18:34:43.060 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cdffd749:::benchmark_data_smithi159_139342_object678:head# 2024-04-15T18:34:43.061 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce000f0a:::benchmark_data_smithi159_139342_object33527:head# 2024-04-15T18:34:43.061 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0016c7:::benchmark_data_smithi159_139342_object10278:head# 2024-04-15T18:34:43.062 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce002b8b:::benchmark_data_smithi159_139342_object45586:head# 2024-04-15T18:34:43.062 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0072d4:::benchmark_data_smithi159_139342_object45669:head# 2024-04-15T18:34:43.062 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce029a11:::benchmark_data_smithi159_139342_object2275:head# 2024-04-15T18:34:43.063 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0321f8:::benchmark_data_smithi159_139342_object36673:head# 2024-04-15T18:34:43.063 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce040ae4:::benchmark_data_smithi159_139342_object37223:head# 2024-04-15T18:34:43.064 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce04bf81:::benchmark_data_smithi159_139342_object7827:head# 2024-04-15T18:34:43.064 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce078093:::benchmark_data_smithi159_139342_object38682:head# 2024-04-15T18:34:43.065 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce09d55d:::benchmark_data_smithi159_139342_object9458:head# 2024-04-15T18:34:43.065 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0a4f95:::benchmark_data_smithi159_139342_object248:head# 2024-04-15T18:34:43.065 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0cca95:::benchmark_data_smithi159_139342_object28227:head# 2024-04-15T18:34:43.066 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0ce361:::benchmark_data_smithi159_139342_object16774:head# 2024-04-15T18:34:43.066 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0dea16:::benchmark_data_smithi159_139342_object50860:head# 2024-04-15T18:34:43.067 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce0f5283:::benchmark_data_smithi159_139342_object41711:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce12b469:::benchmark_data_smithi159_139342_object11812:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce13300d:::benchmark_data_smithi159_139342_object6156:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce13990c:::benchmark_data_smithi159_139342_object1402:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce176ddb:::benchmark_data_smithi159_139342_object5601:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce195639:::benchmark_data_smithi159_139342_object29262:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce1a2327:::benchmark_data_smithi159_139342_object23141:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce1a4d4d:::benchmark_data_smithi159_139342_object26480:head# 2024-04-15T18:34:43.071 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce1b47f5:::benchmark_data_smithi159_139342_object40003:head# 2024-04-15T18:34:43.072 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce1e189b:::benchmark_data_smithi159_139342_object53168:head# 2024-04-15T18:34:43.072 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce1ebaaf:::benchmark_data_smithi159_139342_object24918:head# 2024-04-15T18:34:43.072 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce202c9c:::benchmark_data_smithi159_139342_object15763:head# 2024-04-15T18:34:43.080 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce22aebb:::benchmark_data_smithi159_139342_object40097:head# 2024-04-15T18:34:43.083 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce262a68:::benchmark_data_smithi159_139342_object6961:head# 2024-04-15T18:34:43.083 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce279872:::benchmark_data_smithi159_139342_object33064:head# 2024-04-15T18:34:43.084 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce27d5ad:::benchmark_data_smithi159_139342_object47833:head# 2024-04-15T18:34:43.084 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce28afd3:::benchmark_data_smithi159_139342_object9322:head# 2024-04-15T18:34:43.085 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce293320:::benchmark_data_smithi159_139342_object48802:head# 2024-04-15T18:34:43.085 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce2beb32:::benchmark_data_smithi159_139342_object5947:head# 2024-04-15T18:34:43.086 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce2c1b75:::benchmark_data_smithi159_139342_object48278:head# 2024-04-15T18:34:43.086 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce2e16f1:::benchmark_data_smithi159_139342_object21334:head# 2024-04-15T18:34:43.087 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce2f79c8:::benchmark_data_smithi159_139342_object43053:head# 2024-04-15T18:34:43.087 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce31c56a:::benchmark_data_smithi159_139342_object44867:head# 2024-04-15T18:34:43.087 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce32f5a5:::benchmark_data_smithi159_139342_object6449:head# 2024-04-15T18:34:43.088 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce330932:::benchmark_data_smithi159_139342_object41384:head# 2024-04-15T18:34:43.088 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3463a6:::benchmark_data_smithi159_139342_object37478:head# 2024-04-15T18:34:43.089 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce358765:::benchmark_data_smithi159_139342_object24848:head# 2024-04-15T18:34:43.089 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3692b0:::benchmark_data_smithi159_139342_object1315:head# 2024-04-15T18:34:43.090 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3697e8:::benchmark_data_smithi159_139342_object30946:head# 2024-04-15T18:34:43.090 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce378907:::benchmark_data_smithi159_139342_object26652:head# 2024-04-15T18:34:43.091 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3a6954:::benchmark_data_smithi159_139342_object46125:head# 2024-04-15T18:34:43.091 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3a95eb:::benchmark_data_smithi159_139342_object12620:head# 2024-04-15T18:34:43.092 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3b8be9:::benchmark_data_smithi159_139342_object25513:head# 2024-04-15T18:34:43.092 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3dcfe7:::benchmark_data_smithi159_139342_object31052:head# 2024-04-15T18:34:43.092 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3e31a2:::benchmark_data_smithi159_139342_object50726:head# 2024-04-15T18:34:43.093 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3f995a:::benchmark_data_smithi159_139342_object7532:head# 2024-04-15T18:34:43.093 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce3fe1be:::benchmark_data_smithi159_139342_object55007:head# 2024-04-15T18:34:43.094 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce422433:::benchmark_data_smithi159_139342_object13673:head# 2024-04-15T18:34:43.094 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce42c3b4:::benchmark_data_smithi159_139342_object45046:head# 2024-04-15T18:34:43.095 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce43fa11:::benchmark_data_smithi159_139342_object17387:head# 2024-04-15T18:34:43.095 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce449553:::benchmark_data_smithi159_139342_object11521:head# 2024-04-15T18:34:43.097 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce46fee6:::benchmark_data_smithi159_139342_object41342:head# 2024-04-15T18:34:43.097 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce47ca5e:::benchmark_data_smithi159_139342_object55969:head# 2024-04-15T18:34:43.097 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce48064f:::benchmark_data_smithi159_139342_object42835:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce482d90:::benchmark_data_smithi159_139342_object45400:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce48a29b:::benchmark_data_smithi159_139342_object5983:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4ab47a:::benchmark_data_smithi159_139342_object9509:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4b7426:::benchmark_data_smithi159_139342_object20693:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4bb1b6:::benchmark_data_smithi159_139342_object43186:head# 2024-04-15T18:34:43.098 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4c71d1:::benchmark_data_smithi159_139342_object19966:head# 2024-04-15T18:34:43.099 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4d7381:::benchmark_data_smithi159_139342_object57344:head# 2024-04-15T18:34:43.099 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce4e3336:::benchmark_data_smithi159_139342_object506:head# 2024-04-15T18:34:43.100 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce51d25b:::benchmark_data_smithi159_139342_object8108:head# 2024-04-15T18:34:43.100 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce533cec:::benchmark_data_smithi159_139342_object25156:head# 2024-04-15T18:34:43.101 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce54dd4d:::benchmark_data_smithi159_139342_object45604:head# 2024-04-15T18:34:43.101 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce55cbf2:::benchmark_data_smithi159_139342_object5293:head# 2024-04-15T18:34:43.101 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce561240:::benchmark_data_smithi159_139342_object2445:head# 2024-04-15T18:34:43.102 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce56ed74:::benchmark_data_smithi159_139342_object11160:head# 2024-04-15T18:34:43.102 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce57fb96:::benchmark_data_smithi159_139342_object12119:head# 2024-04-15T18:34:43.102 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce583ace:::benchmark_data_smithi159_139342_object53629:head# 2024-04-15T18:34:43.103 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce59548a:::benchmark_data_smithi159_139342_object9304:head# 2024-04-15T18:34:43.103 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce59ae78:::benchmark_data_smithi159_139342_object7714:head# 2024-04-15T18:34:43.104 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5b4782:::benchmark_data_smithi159_139342_object11703:head# 2024-04-15T18:34:43.104 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5bdfc3:::benchmark_data_smithi159_139342_object12694:head# 2024-04-15T18:34:43.105 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5cd693:::benchmark_data_smithi159_139342_object6745:head# 2024-04-15T18:34:43.105 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5cd803:::benchmark_data_smithi159_139342_object49587:head# 2024-04-15T18:34:43.105 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5df863:::benchmark_data_smithi159_139342_object43000:head# 2024-04-15T18:34:43.109 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5dff1e:::benchmark_data_smithi159_139342_object47092:head# 2024-04-15T18:34:43.109 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5e8b62:::benchmark_data_smithi159_139342_object50633:head# 2024-04-15T18:34:43.109 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce5e9be1:::benchmark_data_smithi159_139342_object7611:head# 2024-04-15T18:34:43.109 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce606283:::benchmark_data_smithi159_139342_object25700:head# 2024-04-15T18:34:43.110 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce60f159:::benchmark_data_smithi159_139342_object40950:head# 2024-04-15T18:34:43.110 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce613b93:::benchmark_data_smithi159_139342_object37504:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce656833:::benchmark_data_smithi159_139342_object51876:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce662ca4:::benchmark_data_smithi159_139342_object25257:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6743b0:::benchmark_data_smithi159_139342_object17692:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce67c54c:::benchmark_data_smithi159_139342_object29416:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce68250e:::benchmark_data_smithi159_139342_object51322:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce694e46:::benchmark_data_smithi159_139342_object24678:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6a8b7c:::benchmark_data_smithi159_139342_object8183:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6a92b0:::benchmark_data_smithi159_139342_object22084:head# 2024-04-15T18:34:43.194 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6aa550:::benchmark_data_smithi159_139342_object33710:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6b4efd:::benchmark_data_smithi159_139342_object48194:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6b6879:::benchmark_data_smithi159_139342_object41196:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6bf868:::benchmark_data_smithi159_139342_object28546:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6cd51f:::benchmark_data_smithi159_139342_object23753:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6cd607:::benchmark_data_smithi159_139342_object1517:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6cfbe6:::benchmark_data_smithi159_139342_object20465:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6d3b7e:::benchmark_data_smithi159_139342_object56027:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6f599b:::benchmark_data_smithi159_139342_object15614:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce6fb0a2:::benchmark_data_smithi159_139342_object48583:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce719125:::benchmark_data_smithi159_139342_object29035:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce743637:::benchmark_data_smithi159_139342_object41614:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7834eb:::benchmark_data_smithi159_139342_object50200:head# 2024-04-15T18:34:43.195 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7994cf:::benchmark_data_smithi159_139342_object34468:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7bea64:::benchmark_data_smithi159_139342_object548:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7db884:::benchmark_data_smithi159_139342_object49224:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7e6d8b:::benchmark_data_smithi159_139342_object41126:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce7f66c4:::benchmark_data_smithi159_139342_object2523:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce80802d:::benchmark_data_smithi159_139342_object50081:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8165ae:::benchmark_data_smithi159_139342_object41497:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce83bb34:::benchmark_data_smithi159_139342_object29556:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce83d51c:::benchmark_data_smithi159_139342_object8556:head# 2024-04-15T18:34:43.196 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce84d3dd:::benchmark_data_smithi159_139342_object32225:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce854e83:::benchmark_data_smithi159_139342_object2836:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8961be:::benchmark_data_smithi159_139342_object49405:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce89fa55:::benchmark_data_smithi159_139342_object17699:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8b7295:::benchmark_data_smithi159_139342_object47737:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8d1897:::benchmark_data_smithi159_139342_object47962:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8de812:::benchmark_data_smithi159_139342_object4121:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8e9867:::benchmark_data_smithi159_139342_object3425:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce8fae68:::benchmark_data_smithi159_139342_object20673:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce90ed2f:::benchmark_data_smithi159_139342_object30347:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce927ef8:::benchmark_data_smithi159_139342_object17591:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce95e693:::benchmark_data_smithi159_139342_object39713:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce984f7d:::benchmark_data_smithi159_139342_object52596:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce99ecd1:::benchmark_data_smithi159_139342_object38814:head# 2024-04-15T18:34:43.197 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce9aa07b:::benchmark_data_smithi159_139342_object44699:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce9c9834:::benchmark_data_smithi159_139342_object9747:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ce9f00d3:::benchmark_data_smithi159_139342_object56871:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea23b77:::benchmark_data_smithi159_139342_object39913:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea428df:::benchmark_data_smithi159_139342_object33462:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea5d44b:::benchmark_data_smithi159_139342_object36092:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea7036b:::benchmark_data_smithi159_139342_object22128:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea770f0:::benchmark_data_smithi159_139342_object36048:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea78789:::benchmark_data_smithi159_139342_object20154:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea7a8aa:::benchmark_data_smithi159_139342_object11393:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea80952:::benchmark_data_smithi159_139342_object5688:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea81fd7:::benchmark_data_smithi159_139342_object43293:head# 2024-04-15T18:34:43.198 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cea9645c:::benchmark_data_smithi159_139342_object49082:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceaa4bfd:::benchmark_data_smithi159_139342_object52605:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceab2a1c:::benchmark_data_smithi159_139342_object4261:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceac345a:::benchmark_data_smithi159_139342_object33671:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceacbda7:::benchmark_data_smithi159_139342_object27602:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceae7cb2:::benchmark_data_smithi159_139342_object11487:head# 2024-04-15T18:34:43.200 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb06ed7:::benchmark_data_smithi159_139342_object52893:head# 2024-04-15T18:34:43.201 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb2636a:::benchmark_data_smithi159_139342_object39153:head# 2024-04-15T18:34:43.201 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb2bcde:::benchmark_data_smithi159_139342_object51946:head# 2024-04-15T18:34:43.201 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb2dfd6:::benchmark_data_smithi159_139342_object40648:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb5a38b:::benchmark_data_smithi159_139342_object36019:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb89cdc:::benchmark_data_smithi159_139342_object38740:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceb8a3a4:::benchmark_data_smithi159_139342_object24324:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceba59f9:::benchmark_data_smithi159_139342_object26144:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cebaa8a8:::benchmark_data_smithi159_139342_object3563:head# 2024-04-15T18:34:43.204 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cebb0a83:::benchmark_data_smithi159_139342_object33194:head# 2024-04-15T18:34:43.209 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cebebcaa:::benchmark_data_smithi159_139342_object23637:head# 2024-04-15T18:34:43.209 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec042f8:::benchmark_data_smithi159_139342_object15385:head# 2024-04-15T18:34:43.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec12be6:::benchmark_data_smithi159_139342_object21407:head# 2024-04-15T18:34:43.210 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec1f004:::benchmark_data_smithi159_139342_object28204:head# 2024-04-15T18:34:43.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec21e68:::benchmark_data_smithi159_139342_object51966:head# 2024-04-15T18:34:43.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec36210:::benchmark_data_smithi159_139342_object9865:head# 2024-04-15T18:34:43.211 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec52588:::benchmark_data_smithi159_139342_object11968:head# 2024-04-15T18:34:43.212 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec63d26:::benchmark_data_smithi159_139342_object5979:head# 2024-04-15T18:34:43.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec68e9e:::benchmark_data_smithi159_139342_object10526:head# 2024-04-15T18:34:43.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec6bdda:::benchmark_data_smithi159_139342_object13650:head# 2024-04-15T18:34:43.218 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec8320c:::benchmark_data_smithi159_139342_object21480:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cec9a400:::benchmark_data_smithi159_139342_object36519:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceca2957:::benchmark_data_smithi159_139342_object20905:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cecb6bc2:::benchmark_data_smithi159_139342_object4039:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cecbc588:::benchmark_data_smithi159_139342_object1316:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cecbf3ee:::benchmark_data_smithi159_139342_object8342:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cecca810:::benchmark_data_smithi159_139342_object1838:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceccb323:::benchmark_data_smithi159_139342_object16118:head# 2024-04-15T18:34:43.219 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ced0e010:::benchmark_data_smithi159_139342_object6440:head# 2024-04-15T18:34:43.220 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ced6a365:::benchmark_data_smithi159_139342_object30342:head# 2024-04-15T18:34:43.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ced7b1de:::benchmark_data_smithi159_139342_object9967:head# 2024-04-15T18:34:43.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceda098e:::benchmark_data_smithi159_139342_object26953:head# 2024-04-15T18:34:43.221 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceda7ba9:::benchmark_data_smithi159_139342_object52083:head# 2024-04-15T18:34:43.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cedb391a:::benchmark_data_smithi159_139342_object28691:head# 2024-04-15T18:34:43.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cedec929:::benchmark_data_smithi159_139342_object27001:head# 2024-04-15T18:34:43.222 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee2fa85:::benchmark_data_smithi159_139342_object18265:head# 2024-04-15T18:34:43.223 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee35830:::benchmark_data_smithi159_139342_object22727:head# 2024-04-15T18:34:43.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee3d574:::benchmark_data_smithi159_139342_object28821:head# 2024-04-15T18:34:43.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee43b33:::benchmark_data_smithi159_139342_object33486:head# 2024-04-15T18:34:43.224 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee51853:::benchmark_data_smithi159_139342_object3924:head# 2024-04-15T18:34:43.225 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee562b9:::benchmark_data_smithi159_139342_object29107:head# 2024-04-15T18:34:43.225 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee9af9b:::benchmark_data_smithi159_139342_object4021:head# 2024-04-15T18:34:43.226 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cee9eed3:::benchmark_data_smithi159_139342_object53915:head# 2024-04-15T18:34:43.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceea306e:::benchmark_data_smithi159_139342_object32144:head# 2024-04-15T18:34:43.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceea50e7:::benchmark_data_smithi159_139342_object10726:head# 2024-04-15T18:34:43.229 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceec4115:::benchmark_data_smithi159_139342_object34684:head# 2024-04-15T18:34:43.230 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceedd387:::benchmark_data_smithi159_139342_object30016:head# 2024-04-15T18:34:43.234 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceef52c6:::benchmark_data_smithi159_139342_object48649:head# 2024-04-15T18:34:43.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef0e72b:::benchmark_data_smithi159_139342_object22926:head# 2024-04-15T18:34:43.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef1aa72:::benchmark_data_smithi159_139342_object12573:head# 2024-04-15T18:34:43.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef1c564:::benchmark_data_smithi159_139342_object22430:head# 2024-04-15T18:34:43.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef335c0:::benchmark_data_smithi159_139342_object55422:head# 2024-04-15T18:34:43.236 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef3cc80:::benchmark_data_smithi159_139342_object37840:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef4a6d6:::benchmark_data_smithi159_139342_object46574:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef55b7f:::benchmark_data_smithi159_139342_object13815:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cef62184:::benchmark_data_smithi159_139342_object19016:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefa2741:::benchmark_data_smithi159_139342_object23948:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefbc9d4:::benchmark_data_smithi159_139342_object11528:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefc5564:::benchmark_data_smithi159_139342_object55230:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefcd041:::benchmark_data_smithi159_139342_object7639:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefe7bc9:::benchmark_data_smithi159_139342_object32179:head# 2024-04-15T18:34:43.237 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cefebc0d:::benchmark_data_smithi159_139342_object36314:head# 2024-04-15T18:34:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:ceff968a:::benchmark_data_smithi159_139342_object17939:head# 2024-04-15T18:34:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf022aea:::benchmark_data_smithi159_139342_object35967:head# 2024-04-15T18:34:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf030d65:::benchmark_data_smithi159_139342_object29108:head# 2024-04-15T18:34:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf03362c:::benchmark_data_smithi159_139342_object17356:head# 2024-04-15T18:34:43.238 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf03c4c6:::benchmark_data_smithi159_139342_object45139:head# 2024-04-15T18:34:43.239 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf05d309:::benchmark_data_smithi159_139342_object30478:head# 2024-04-15T18:34:43.239 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf07158f:::benchmark_data_smithi159_139342_object3961:head# 2024-04-15T18:34:43.240 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf0dab6d:::benchmark_data_smithi159_139342_object36058:head# 2024-04-15T18:34:43.240 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf0fdc42:::benchmark_data_smithi159_139342_object13655:head# 2024-04-15T18:34:43.241 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf111190:::benchmark_data_smithi159_139342_object42099:head# 2024-04-15T18:34:43.241 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf126bcc:::benchmark_data_smithi159_139342_object42563:head# 2024-04-15T18:34:43.241 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1308ae:::benchmark_data_smithi159_139342_object53451:head# 2024-04-15T18:34:43.242 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1346e6:::benchmark_data_smithi159_139342_object19055:head# 2024-04-15T18:34:43.242 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf159f84:::benchmark_data_smithi159_139342_object22188:head# 2024-04-15T18:34:43.243 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf19577b:::benchmark_data_smithi159_139342_object12640:head# 2024-04-15T18:34:43.243 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1a2980:::benchmark_data_smithi159_139342_object6648:head# 2024-04-15T18:34:43.243 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1a9cd6:::benchmark_data_smithi159_139342_object20541:head# 2024-04-15T18:34:43.244 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1d105b:::benchmark_data_smithi159_139342_object42865:head# 2024-04-15T18:34:43.244 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf1e15e2:::benchmark_data_smithi159_139342_object6246:head# 2024-04-15T18:34:43.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf201058:::benchmark_data_smithi159_139342_object32928:head# 2024-04-15T18:34:43.245 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf209fc8:::benchmark_data_smithi159_139342_object51038:head# 2024-04-15T18:34:43.246 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf212b9e:::benchmark_data_smithi159_139342_object49843:head# 2024-04-15T18:34:43.246 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf215f91:::benchmark_data_smithi159_139342_object33159:head# 2024-04-15T18:34:43.246 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf222e8c:::benchmark_data_smithi159_139342_object31422:head# 2024-04-15T18:34:43.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf235aca:::benchmark_data_smithi159_139342_object28334:head# 2024-04-15T18:34:43.247 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf23e753:::benchmark_data_smithi159_139342_object24831:head# 2024-04-15T18:34:43.248 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf24b744:::benchmark_data_smithi159_139342_object12427:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf25d3d5:::benchmark_data_smithi159_139342_object15981:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf25fe2b:::benchmark_data_smithi159_139342_object1779:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf289455:::benchmark_data_smithi159_139342_object42821:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2967ed:::benchmark_data_smithi159_139342_object51054:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf298081:::benchmark_data_smithi159_139342_object55454:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf298794:::benchmark_data_smithi159_139342_object18530:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2b9b12:::benchmark_data_smithi159_139342_object40917:head# 2024-04-15T18:34:43.252 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2bef03:::benchmark_data_smithi159_139342_object3654:head# 2024-04-15T18:34:43.253 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2d06d1:::benchmark_data_smithi159_139342_object15550:head# 2024-04-15T18:34:43.254 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2d4e3f:::benchmark_data_smithi159_139342_object2387:head# 2024-04-15T18:34:43.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf2f1853:::benchmark_data_smithi159_139342_object45700:head# 2024-04-15T18:34:43.255 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf30ec16:::benchmark_data_smithi159_139342_object38382:head# 2024-04-15T18:34:43.258 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf32209d:::benchmark_data_smithi159_139342_object35916:head# 2024-04-15T18:34:43.258 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf35bd3a:::benchmark_data_smithi159_139342_object11438:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf362521:::benchmark_data_smithi159_139342_object6956:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3725d7:::benchmark_data_smithi159_139342_object44054:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf38e99a:::benchmark_data_smithi159_139342_object41366:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf38ebab:::benchmark_data_smithi159_139342_object53108:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf39833b:::benchmark_data_smithi159_139342_object43931:head# 2024-04-15T18:34:43.259 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf39a0c7:::benchmark_data_smithi159_139342_object20226:head# 2024-04-15T18:34:43.260 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3a0a1a:::benchmark_data_smithi159_139342_object7851:head# 2024-04-15T18:34:43.260 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3a59bc:::benchmark_data_smithi159_139342_object42646:head# 2024-04-15T18:34:43.260 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3bf692:::benchmark_data_smithi159_139342_object14088:head# 2024-04-15T18:34:43.260 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3c5793:::benchmark_data_smithi159_139342_object18149:head# 2024-04-15T18:34:43.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3d2810:::benchmark_data_smithi159_139342_object1389:head# 2024-04-15T18:34:43.263 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3d5ff4:::benchmark_data_smithi159_139342_object31609:head# 2024-04-15T18:34:43.264 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3db14e:::benchmark_data_smithi159_139342_object44937:head# 2024-04-15T18:34:43.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3eae47:::benchmark_data_smithi159_139342_object45511:head# 2024-04-15T18:34:43.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf3f181e:::benchmark_data_smithi159_139342_object39482:head# 2024-04-15T18:34:43.266 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf42333c:::benchmark_data_smithi159_139342_object29548:head# 2024-04-15T18:34:43.268 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf42b09b:::benchmark_data_smithi159_139342_object43882:head# 2024-04-15T18:34:43.268 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf45c740:::benchmark_data_smithi159_139342_object18498:head# 2024-04-15T18:34:43.269 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf46cb2e:::benchmark_data_smithi159_139342_object48875:head# 2024-04-15T18:34:43.269 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf46fa99:::benchmark_data_smithi159_139342_object49773:head# 2024-04-15T18:34:43.270 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf49698e:::benchmark_data_smithi159_139342_object1406:head# 2024-04-15T18:34:43.270 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf4a7984:::benchmark_data_smithi159_139342_object5842:head# 2024-04-15T18:34:43.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf4d0fe6:::benchmark_data_smithi159_139342_object45532:head# 2024-04-15T18:34:43.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf4f18da:::benchmark_data_smithi159_139342_object14976:head# 2024-04-15T18:34:43.271 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf4fa0d8:::benchmark_data_smithi159_139342_object19482:head# 2024-04-15T18:34:43.272 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf501e97:::benchmark_data_smithi159_139342_object19783:head# 2024-04-15T18:34:43.272 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf507fee:::benchmark_data_smithi159_139342_object29920:head# 2024-04-15T18:34:43.272 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf527b05:::benchmark_data_smithi159_139342_object50454:head# 2024-04-15T18:34:43.273 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf55d274:::benchmark_data_smithi159_139342_object34352:head# 2024-04-15T18:34:43.273 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf56181f:::benchmark_data_smithi159_139342_object8553:head# 2024-04-15T18:34:43.274 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf57cdec:::benchmark_data_smithi159_139342_object46842:head# 2024-04-15T18:34:43.274 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf592315:::benchmark_data_smithi159_139342_object30093:head# 2024-04-15T18:34:43.275 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5b7c40:::benchmark_data_smithi159_139342_object33440:head# 2024-04-15T18:34:43.275 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5baa7c:::benchmark_data_smithi159_139342_object9032:head# 2024-04-15T18:34:43.275 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5c5c2f:::benchmark_data_smithi159_139342_object8251:head# 2024-04-15T18:34:43.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5c61dd:::benchmark_data_smithi159_139342_object839:head# 2024-04-15T18:34:43.276 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5dbb60:::benchmark_data_smithi159_139342_object2855:head# 2024-04-15T18:34:43.277 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5e03ef:::benchmark_data_smithi159_139342_object32787:head# 2024-04-15T18:34:43.277 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf5f63da:::benchmark_data_smithi159_139342_object29643:head# 2024-04-15T18:34:43.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf605188:::benchmark_data_smithi159_139342_object1497:head# 2024-04-15T18:34:43.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6218cd:::benchmark_data_smithi159_139342_object11289:head# 2024-04-15T18:34:43.278 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf631181:::benchmark_data_smithi159_139342_object54060:head# 2024-04-15T18:34:43.279 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf637e33:::benchmark_data_smithi159_139342_object23676:head# 2024-04-15T18:34:43.279 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf639e85:::benchmark_data_smithi159_139342_object44768:head# 2024-04-15T18:34:43.280 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6420f4:::benchmark_data_smithi159_139342_object50265:head# 2024-04-15T18:34:43.280 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf646c1b:::benchmark_data_smithi159_139342_object23500:head# 2024-04-15T18:34:43.281 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf64e970:::benchmark_data_smithi159_139342_object34577:head# 2024-04-15T18:34:43.281 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf65262a:::benchmark_data_smithi159_139342_object39595:head# 2024-04-15T18:34:43.281 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf675909:::benchmark_data_smithi159_139342_object57045:head# 2024-04-15T18:34:43.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf678487:::benchmark_data_smithi159_139342_object24732:head# 2024-04-15T18:34:43.282 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf67b7ba:::benchmark_data_smithi159_139342_object53047:head# 2024-04-15T18:34:43.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6a3c32:::benchmark_data_smithi159_139342_object54161:head# 2024-04-15T18:34:43.283 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6a9350:::benchmark_data_smithi159_139342_object20871:head# 2024-04-15T18:34:43.284 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6c16d9:::benchmark_data_smithi159_139342_object2495:head# 2024-04-15T18:34:43.284 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6c7496:::benchmark_data_smithi159_139342_object11198:head# 2024-04-15T18:34:43.285 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6de300:::benchmark_data_smithi159_139342_object48046:head# 2024-04-15T18:34:43.285 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6e02e8:::benchmark_data_smithi159_139342_object6951:head# 2024-04-15T18:34:43.285 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6e46cc:::benchmark_data_smithi159_139342_object34996:head# 2024-04-15T18:34:43.286 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6e7f21:::benchmark_data_smithi159_139342_object48973:head# 2024-04-15T18:34:43.286 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6ede3c:::benchmark_data_smithi159_139342_object30299:head# 2024-04-15T18:34:43.286 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf6f67f8:::benchmark_data_smithi159_139342_object2844:head# 2024-04-15T18:34:43.287 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf72056b:::benchmark_data_smithi159_139342_object15680:head# 2024-04-15T18:34:43.287 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf727aff:::benchmark_data_smithi159_139342_object42520:head# 2024-04-15T18:34:43.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf72f18b:::benchmark_data_smithi159_139342_object55117:head# 2024-04-15T18:34:43.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf752163:::benchmark_data_smithi159_139342_object16489:head# 2024-04-15T18:34:43.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf75278d:::benchmark_data_smithi159_139342_object36196:head# 2024-04-15T18:34:43.288 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf758d49:::benchmark_data_smithi159_139342_object34821:head# 2024-04-15T18:34:43.289 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf78375e:::benchmark_data_smithi159_139342_object54471:head# 2024-04-15T18:34:43.289 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf78556a:::benchmark_data_smithi159_139342_object31679:head# 2024-04-15T18:34:43.290 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7a06fb:::benchmark_data_smithi159_139342_object20603:head# 2024-04-15T18:34:43.291 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7a7c51:::benchmark_data_smithi159_139342_object35328:head# 2024-04-15T18:34:43.291 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7a884a:::benchmark_data_smithi159_139342_object8989:head# 2024-04-15T18:34:43.291 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7bdb44:::benchmark_data_smithi159_139342_object32123:head# 2024-04-15T18:34:43.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7eb2dd:::benchmark_data_smithi159_139342_object33620:head# 2024-04-15T18:34:43.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf7eda83:::benchmark_data_smithi159_139342_object45836:head# 2024-04-15T18:34:43.292 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf800817:::benchmark_data_smithi159_139342_object4714:head# 2024-04-15T18:34:43.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8021b8:::benchmark_data_smithi159_139342_object41765:head# 2024-04-15T18:34:43.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8139ce:::benchmark_data_smithi159_139342_object40246:head# 2024-04-15T18:34:43.293 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8190de:::benchmark_data_smithi159_139342_object14564:head# 2024-04-15T18:34:43.294 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf81a444:::benchmark_data_smithi159_139342_object51823:head# 2024-04-15T18:34:43.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf82a8c6:::benchmark_data_smithi159_139342_object806:head# 2024-04-15T18:34:43.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf82d9c1:::benchmark_data_smithi159_139342_object26322:head# 2024-04-15T18:34:43.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf835092:::benchmark_data_smithi159_139342_object847:head# 2024-04-15T18:34:43.295 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf83f085:::benchmark_data_smithi159_139342_object172:head# 2024-04-15T18:34:43.296 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf84f9be:::benchmark_data_smithi159_139342_object1025:head# 2024-04-15T18:34:43.296 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf867fa6:::benchmark_data_smithi159_139342_object42406:head# 2024-04-15T18:34:43.297 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8a1ee0:::benchmark_data_smithi159_139342_object11369:head# 2024-04-15T18:34:43.297 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8a2f7d:::benchmark_data_smithi159_139342_object20255:head# 2024-04-15T18:34:43.297 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8aa3e9:::benchmark_data_smithi159_139342_object14814:head# 2024-04-15T18:34:43.298 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8e41f9:::benchmark_data_smithi159_139342_object23663:head# 2024-04-15T18:34:43.298 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8e6e53:::benchmark_data_smithi159_139342_object19224:head# 2024-04-15T18:34:43.299 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8e93e5:::benchmark_data_smithi159_139342_object45920:head# 2024-04-15T18:34:43.299 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8ebd87:::benchmark_data_smithi159_139342_object37183:head# 2024-04-15T18:34:43.299 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf8ff9e1:::benchmark_data_smithi159_139342_object6055:head# 2024-04-15T18:34:43.300 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf90352a:::benchmark_data_smithi159_139342_object25844:head# 2024-04-15T18:34:43.300 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf917fed:::benchmark_data_smithi159_139342_object50718:head# 2024-04-15T18:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf921581:::benchmark_data_smithi159_139342_object23914:head# 2024-04-15T18:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf926e1b:::benchmark_data_smithi159_139342_object44812:head# 2024-04-15T18:34:43.301 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf94a98a:::benchmark_data_smithi159_139342_object1790:head# 2024-04-15T18:34:43.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf954253:::benchmark_data_smithi159_139342_object42637:head# 2024-04-15T18:34:43.302 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf965d5c:::benchmark_data_smithi159_139342_object7128:head# 2024-04-15T18:34:43.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf96a7a9:::benchmark_data_smithi159_139342_object30272:head# 2024-04-15T18:34:43.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9706d5:::benchmark_data_smithi159_139342_object44334:head# 2024-04-15T18:34:43.303 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf97cb39:::benchmark_data_smithi159_139342_object27119:head# 2024-04-15T18:34:43.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf99b510:::benchmark_data_smithi159_139342_object54084:head# 2024-04-15T18:34:43.304 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9aec18:::benchmark_data_smithi159_139342_object54016:head# 2024-04-15T18:34:43.305 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9b36be:::benchmark_data_smithi159_139342_object8599:head# 2024-04-15T18:34:43.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9b92be:::benchmark_data_smithi159_139342_object7264:head# 2024-04-15T18:34:43.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9d2a1f:::benchmark_data_smithi159_139342_object14874:head# 2024-04-15T18:34:43.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cf9fa6bd:::benchmark_data_smithi159_139342_object15509:head# 2024-04-15T18:34:43.308 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfa09ba6:::benchmark_data_smithi159_139342_object39157:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfa5974d:::benchmark_data_smithi159_139342_object33534:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfa66004:::benchmark_data_smithi159_139342_object41695:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfa71040:::benchmark_data_smithi159_139342_object44472:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfa74fa7:::benchmark_data_smithi159_139342_object50288:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfaa3fe5:::benchmark_data_smithi159_139342_object6164:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfaaadba:::benchmark_data_smithi159_139342_object2348:head# 2024-04-15T18:34:43.309 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfaacefa:::benchmark_data_smithi159_139342_object53247:head# 2024-04-15T18:34:43.310 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfac5787:::benchmark_data_smithi159_139342_object15047:head# 2024-04-15T18:34:43.310 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfac6f4b:::benchmark_data_smithi159_139342_object9662:head# 2024-04-15T18:34:43.310 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfaf7bd3:::benchmark_data_smithi159_139342_object22427:head# 2024-04-15T18:34:43.311 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb00ef8:::benchmark_data_smithi159_139342_object37211:head# 2024-04-15T18:34:43.311 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb0afef:::benchmark_data_smithi159_139342_object29471:head# 2024-04-15T18:34:43.312 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb0ba91:::benchmark_data_smithi159_139342_object24739:head# 2024-04-15T18:34:43.312 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb0ec9b:::benchmark_data_smithi159_139342_object27207:head# 2024-04-15T18:34:43.312 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb0ffe5:::benchmark_data_smithi159_139342_object38057:head# 2024-04-15T18:34:43.313 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb1c11e:::benchmark_data_smithi159_139342_object24859:head# 2024-04-15T18:34:43.313 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb317c0:::benchmark_data_smithi159_139342_object7258:head# 2024-04-15T18:34:43.314 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb3af28:::benchmark_data_smithi159_139342_object19518:head# 2024-04-15T18:34:43.314 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb4266c:::benchmark_data_smithi159_139342_object46712:head# 2024-04-15T18:34:43.315 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb5b87a:::benchmark_data_smithi159_139342_object49738:head# 2024-04-15T18:34:43.316 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb90acd:::benchmark_data_smithi159_139342_object511:head# 2024-04-15T18:34:43.316 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb94e76:::benchmark_data_smithi159_139342_object7078:head# 2024-04-15T18:34:43.316 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfb9ef78:::benchmark_data_smithi159_139342_object19754:head# 2024-04-15T18:34:43.317 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfbd396f:::benchmark_data_smithi159_139342_object30336:head# 2024-04-15T18:34:43.317 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfbdd221:::benchmark_data_smithi159_139342_object24177:head# 2024-04-15T18:34:43.317 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfbe6c11:::benchmark_data_smithi159_139342_object24474:head# 2024-04-15T18:34:43.318 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfc07b7c:::benchmark_data_smithi159_139342_object53833:head# 2024-04-15T18:34:43.318 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfc2b9b1:::benchmark_data_smithi159_139342_object6840:head# 2024-04-15T18:34:43.322 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfc443cf:::benchmark_data_smithi159_139342_object57060:head# 2024-04-15T18:34:43.325 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfc832a9:::benchmark_data_smithi159_139342_object13363:head# 2024-04-15T18:34:43.325 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfca8520:::benchmark_data_smithi159_139342_object13333:head# 2024-04-15T18:34:43.325 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfcb818b:::benchmark_data_smithi159_139342_object51957:head# 2024-04-15T18:34:43.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd03a12:::benchmark_data_smithi159_139342_object29762:head# 2024-04-15T18:34:43.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd0e2f0:::benchmark_data_smithi159_139342_object43734:head# 2024-04-15T18:34:43.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd1895b:::benchmark_data_smithi159_139342_object4736:head# 2024-04-15T18:34:43.326 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd2be2c:::benchmark_data_smithi159_139342_object54170:head# 2024-04-15T18:34:43.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd468b9:::benchmark_data_smithi159_139342_object30821:head# 2024-04-15T18:34:43.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd5f593:::benchmark_data_smithi159_139342_object44609:head# 2024-04-15T18:34:43.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd7451f:::benchmark_data_smithi159_139342_object32362:head# 2024-04-15T18:34:43.327 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd79abb:::benchmark_data_smithi159_139342_object28504:head# 2024-04-15T18:34:43.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfd80860:::benchmark_data_smithi159_139342_object18098:head# 2024-04-15T18:34:43.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfded8c7:::benchmark_data_smithi159_139342_object25210:head# 2024-04-15T18:34:43.328 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfdee9e8:::benchmark_data_smithi159_139342_object34630:head# 2024-04-15T18:34:43.329 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe382db:::benchmark_data_smithi159_139342_object17854:head# 2024-04-15T18:34:43.329 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe4134c:::benchmark_data_smithi159_139342_object43334:head# 2024-04-15T18:34:43.330 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe45815:::benchmark_data_smithi159_139342_object4258:head# 2024-04-15T18:34:43.330 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe564d4:::benchmark_data_smithi159_139342_object22373:head# 2024-04-15T18:34:43.331 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe5b56e:::benchmark_data_smithi159_139342_object6534:head# 2024-04-15T18:34:43.331 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe657ed:::benchmark_data_smithi159_139342_object291:head# 2024-04-15T18:34:43.332 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe6d72a:::benchmark_data_smithi159_139342_object22488:head# 2024-04-15T18:34:43.332 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe6e1a1:::benchmark_data_smithi159_139342_object40847:head# 2024-04-15T18:34:43.332 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe7462f:::benchmark_data_smithi159_139342_object27570:head# 2024-04-15T18:34:43.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe7b65b:::benchmark_data_smithi159_139342_object47241:head# 2024-04-15T18:34:43.333 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe7da31:::benchmark_data_smithi159_139342_object20778:head# 2024-04-15T18:34:43.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfe96e74:::benchmark_data_smithi159_139342_object11571:head# 2024-04-15T18:34:43.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfea1a6b:::benchmark_data_smithi159_139342_object54713:head# 2024-04-15T18:34:43.334 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfea8fb6:::benchmark_data_smithi159_139342_object52422:head# 2024-04-15T18:34:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfeae918:::benchmark_data_smithi159_139342_object49026:head# 2024-04-15T18:34:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfeb2956:::benchmark_data_smithi159_139342_object30140:head# 2024-04-15T18:34:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfecb13f:::benchmark_data_smithi159_139342_object53203:head# 2024-04-15T18:34:43.336 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfece703:::benchmark_data_smithi159_139342_object18479:head# 2024-04-15T18:34:43.336 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfed7478:::benchmark_data_smithi159_139342_object30977:head# 2024-04-15T18:34:43.336 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfee12f0:::benchmark_data_smithi159_139342_object2397:head# 2024-04-15T18:34:43.337 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfee4b47:::benchmark_data_smithi159_139342_object17517:head# 2024-04-15T18:34:43.338 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff2c08c:::benchmark_data_smithi159_139342_object52770:head# 2024-04-15T18:34:43.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff4b57d:::benchmark_data_smithi159_139342_object26799:head# 2024-04-15T18:34:43.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff4e3d1:::benchmark_data_smithi159_139342_object37453:head# 2024-04-15T18:34:43.339 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff517ba:::benchmark_data_smithi159_139342_object4209:head# 2024-04-15T18:34:43.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff6d612:::benchmark_data_smithi159_139342_object48469:head# 2024-04-15T18:34:43.340 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff8711e:::benchmark_data_smithi159_139342_object31272:head# 2024-04-15T18:34:43.341 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff8ee22:::benchmark_data_smithi159_139342_object13071:head# 2024-04-15T18:34:43.344 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff9067c:::benchmark_data_smithi159_139342_object48928:head# 2024-04-15T18:34:43.344 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff95e4a:::benchmark_data_smithi159_139342_object33168:head# 2024-04-15T18:34:43.344 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cff97226:::benchmark_data_smithi159_139342_object34909:head# 2024-04-15T18:34:43.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cffb2c32:::benchmark_data_smithi159_139342_object52498:head# 2024-04-15T18:34:43.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cffc60fa:::benchmark_data_smithi159_139342_object21120:head# 2024-04-15T18:34:43.345 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cffc7723:::benchmark_data_smithi159_139342_object5008:head# 2024-04-15T18:34:43.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cffd2df4:::benchmark_data_smithi159_139342_object54107:head# 2024-04-15T18:34:43.346 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cffde47e:::benchmark_data_smithi159_139342_object37273:head# 2024-04-15T18:34:43.347 INFO:teuthology.orchestra.run.smithi082.stderr:Read #6:cfff17da:::benchmark_data_smithi159_139342_object37511:head# 2024-04-15T18:34:43.349 INFO:teuthology.orchestra.run.smithi082.stderr:Export successful 2024-04-15T18:34:43.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:43 smithi159 ceph-mon[45336]: pgmap v1519: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s; 16342/206958 objects degraded (7.896%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:43.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:43 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 16342/206958 objects degraded (7.896%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:43.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:43 smithi049 ceph-mon[40087]: pgmap v1519: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s; 16342/206958 objects degraded (7.896%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:43.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:43 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 16342/206958 objects degraded (7.896%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:43.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:43 smithi082 ceph-mon[44206]: pgmap v1519: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s; 16342/206958 objects degraded (7.896%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:43.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:43 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 16342/206958 objects degraded (7.896%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:43.852 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:43.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:43.960 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:43.994 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:44.021 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:44.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:44.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:44.166 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:44.189 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:34:44.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:44.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:44.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3269219040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.399 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:34:44.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:44.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3269219040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.471 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:34:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:44.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:34:44.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3269219040' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:34:44.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:34:44.614 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:45.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:45 smithi049 ceph-mon[40087]: pgmap v1520: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 15965/199860 objects degraded (7.988%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:45.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:45 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:45.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:45.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:45 smithi049 ceph-mon[40087]: osdmap e584: 12 total, 11 up, 11 in 2024-04-15T18:34:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:34:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:45.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:34:45.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:45 smithi159 ceph-mon[45336]: pgmap v1520: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 15965/199860 objects degraded (7.988%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:45.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:45 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:45.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:45.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:45 smithi159 ceph-mon[45336]: osdmap e584: 12 total, 11 up, 11 in 2024-04-15T18:34:45.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:45.518 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:45.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:45 smithi082 ceph-mon[44206]: pgmap v1520: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 15965/199860 objects degraded (7.988%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:34:45.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:45 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:45.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:34:45.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:45 smithi082 ceph-mon[44206]: osdmap e584: 12 total, 11 up, 11 in 2024-04-15T18:34:45.540 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:45.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:45.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:34:45.699 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:45.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:45.726 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:34:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:45.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:34:45.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:45.778 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:45.815 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:45.832 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:45.855 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:45.871 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:45.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:45.898 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:45.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:45.923 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:45.950 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:45.980 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:46.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:46.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:46.049 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:46.050 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:46.050 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:46.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:46.077 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:46.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:46.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:46.219 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:46.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4171941933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:46.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4171941933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:46.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4171941933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:46.860 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:47.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: pgmap v1522: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 15965/199236 objects degraded (8.013%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.162 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.162 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: pgmap v1522: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 15965/199236 objects degraded (8.013%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:34:47.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.501 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: pgmap v1522: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 15965/199236 objects degraded (8.013%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:34:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:47.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:48.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:48 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 15965/199236 objects degraded (8.013%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:48.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:48 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 15965/199236 objects degraded (8.013%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:48.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:48 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 15965/199236 objects degraded (8.013%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:48.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:34:49.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:49 smithi049 ceph-mon[40087]: pgmap v1523: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 15485/194055 objects degraded (7.980%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:49.521 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:49.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:49 smithi082 ceph-mon[44206]: pgmap v1523: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 15485/194055 objects degraded (7.980%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:49.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:49 smithi159 ceph-mon[45336]: pgmap v1523: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 15485/194055 objects degraded (7.980%); 10 MiB/s, 168 objects/s recovering 2024-04-15T18:34:50.064 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:50.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:50.168 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:34:50.172 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:34:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:50.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:50.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:50.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: pgmap v1524: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 15073/189531 objects degraded (7.953%); 11 MiB/s, 169 objects/s recovering 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: osdmap e585: 12 total, 11 up, 11 in 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:51.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: pgmap v1524: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 15073/189531 objects degraded (7.953%); 11 MiB/s, 169 objects/s recovering 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: osdmap e585: 12 total, 11 up, 11 in 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:51.611 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: pgmap v1524: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 15073/189531 objects degraded (7.953%); 11 MiB/s, 169 objects/s recovering 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/71054724' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: osdmap e585: 12 total, 11 up, 11 in 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:52.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:52 smithi049 ceph-mon[40087]: osdmap e586: 12 total, 11 up, 11 in 2024-04-15T18:34:52.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:52 smithi082 ceph-mon[44206]: osdmap e586: 12 total, 11 up, 11 in 2024-04-15T18:34:52.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:52 smithi159 ceph-mon[45336]: osdmap e586: 12 total, 11 up, 11 in 2024-04-15T18:34:52.778 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:52.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:52.853 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:52.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:52.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:52.974 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:53.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:53.050 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:53.087 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:53.129 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:53.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:53.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:53.223 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:34:53.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:53 smithi049 ceph-mon[40087]: pgmap v1527: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14545/185940 objects degraded (7.822%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:34:53.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 14545/185940 objects degraded (7.822%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:53.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/459365205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:53.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:53 smithi082 ceph-mon[44206]: pgmap v1527: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14545/185940 objects degraded (7.822%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:34:53.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 14545/185940 objects degraded (7.822%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:53.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/459365205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:53.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:53 smithi159 ceph-mon[45336]: pgmap v1527: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 14545/185940 objects degraded (7.822%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:34:53.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 14545/185940 objects degraded (7.822%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:53.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/459365205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:53.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:34:53.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:34:54.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:34:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:54.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:34:54.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:34:54.306 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.505 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:54.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:34:55.257 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:34:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:55.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:34:55.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:55.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:55 smithi049 ceph-mon[40087]: pgmap v1528: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 14157/178935 objects degraded (7.912%); 14 MiB/s, 225 objects/s recovering 2024-04-15T18:34:55.414 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:55.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:55 smithi082 ceph-mon[44206]: pgmap v1528: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 14157/178935 objects degraded (7.912%); 14 MiB/s, 225 objects/s recovering 2024-04-15T18:34:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:55 smithi159 ceph-mon[45336]: pgmap v1528: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 14157/178935 objects degraded (7.912%); 14 MiB/s, 225 objects/s recovering 2024-04-15T18:34:55.929 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:55.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:55.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.6 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 6.3 2024-04-15T18:34:55.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:34:56.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:56.215 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:56.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:56.280 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.6/config 2024-04-15T18:34:56.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:56.375 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:56.597 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:34:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:57 smithi049 ceph-mon[40087]: pgmap v1529: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 657 op/s; 14157/178269 objects degraded (7.941%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:57.223 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:34:57.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:57 smithi082 ceph-mon[44206]: pgmap v1529: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 657 op/s; 14157/178269 objects degraded (7.941%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:57.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:57.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:57 smithi159 ceph-mon[45336]: pgmap v1529: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 657 op/s; 14157/178269 objects degraded (7.941%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:34:57.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:34:58.346 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:34:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:58.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:34:58.453 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: osdmap e587: 12 total, 11 up, 11 in 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 14157/178269 objects degraded (7.941%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:58.464 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:58 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: osdmap e587: 12 total, 11 up, 11 in 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 14157/178269 objects degraded (7.941%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:58.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:58 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4290263851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: osdmap e587: 12 total, 11 up, 11 in 2024-04-15T18:34:58.478 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 14157/178269 objects degraded (7.941%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:34:58.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:34:58.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:34:58.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:34:58.479 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:58 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:34:58.481 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:58.500 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:34:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:58.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:34:58.577 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:34:58.577 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:34:58.578 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:34:58.578 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:34:58.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:34:58.633 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:58.657 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:34:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:58.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:34:58.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:58.815 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:58.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:58.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:59.006 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:59.071 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:59.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:59.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:34:59.235 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:59.261 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:34:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:59.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:34:59.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:34:59.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:34:59.350 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:34:59.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:34:59.398 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:34:59.418 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:59.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:34:59.455 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:34:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:34:59.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:34:59.495 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:34:59.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:34:59.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:34:59.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:34:59.603 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:34:59.626 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:34:59.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:34:59.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:34:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:34:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:34:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:34:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:34:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:34:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:34:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/655578535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:59.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:59.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:34:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/655578535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:34:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:34:59.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:34:59.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:34:59.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:34:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/655578535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:00.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:00.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: pgmap v1532: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 13129/168303 objects degraded (7.801%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: pgmap v1532: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 13129/168303 objects degraded (7.801%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: pgmap v1532: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 13129/168303 objects degraded (7.801%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:00.980 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:01.012 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:01.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:01.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:01.143 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:01.166 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:01.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:01.270 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:01.297 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:01.328 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:01.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:01.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:01.471 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:01.498 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:01.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:01.538 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:01.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:01.666 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:01.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:01.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:02.252 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:02.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:02.356 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:02.386 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:02.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:35:02.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12736/164898 objects degraded (7.724%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:35:02.693 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12736/164898 objects degraded (7.724%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:35:02.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:35:02.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:35:02.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:35:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:35:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:35:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:35:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:35:02.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12736/164898 objects degraded (7.724%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:35:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:03.507 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:35:03.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:03.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:03.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: pgmap v1534: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 12277/157758 objects degraded (7.782%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:04.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:04 smithi049 ceph-mon[40087]: osdmap e588: 12 total, 11 up, 11 in 2024-04-15T18:35:04.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:04.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:04.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: pgmap v1534: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 12277/157758 objects degraded (7.782%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:35:04.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:04.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:04.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:04 smithi082 ceph-mon[44206]: osdmap e588: 12 total, 11 up, 11 in 2024-04-15T18:35:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: pgmap v1534: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 12277/157758 objects degraded (7.782%); 12 MiB/s, 187 objects/s recovering 2024-04-15T18:35:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2309193254' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:04 smithi159 ceph-mon[45336]: osdmap e588: 12 total, 11 up, 11 in 2024-04-15T18:35:05.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:05.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:05.743 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:05.781 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:05.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:05.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:05.875 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:05.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:05.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:05.943 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:05.977 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:06.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:06.051 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:06.051 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:06.052 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: pgmap v1536: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 12277/157068 objects degraded (7.816%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: osdmap e589: 12 total, 11 up, 11 in 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/293586534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:06.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: pgmap v1536: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 12277/157068 objects degraded (7.816%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: osdmap e589: 12 total, 11 up, 11 in 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/293586534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: pgmap v1536: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 12277/157068 objects degraded (7.816%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:06.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:06.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:06.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: osdmap e589: 12 total, 11 up, 11 in 2024-04-15T18:35:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/293586534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:06.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:35:07.472 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12277/157068 objects degraded (7.816%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12277/157068 objects degraded (7.816%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:07.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12277/157068 objects degraded (7.816%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:08.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:08 smithi049 ceph-mon[40087]: pgmap v1538: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 11758/151929 objects degraded (7.739%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:35:08.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:35:08.755 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:35:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:08.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:08.830 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:08 smithi082 ceph-mon[44206]: pgmap v1538: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 11758/151929 objects degraded (7.739%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:35:08.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:08.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:08 smithi159 ceph-mon[45336]: pgmap v1538: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 11758/151929 objects degraded (7.739%); 11 MiB/s, 171 objects/s recovering 2024-04-15T18:35:08.897 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:09.516 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:09.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:09.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:09.807 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:09 smithi049 ceph-mon[40087]: osdmap e590: 12 total, 11 up, 11 in 2024-04-15T18:35:09.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:09.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:09.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:09 smithi159 ceph-mon[45336]: osdmap e590: 12 total, 11 up, 11 in 2024-04-15T18:35:10.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:10.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:10.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:09 smithi082 ceph-mon[44206]: osdmap e590: 12 total, 11 up, 11 in 2024-04-15T18:35:10.563 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:10.572 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:35:10.630 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:35:10.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:10.722 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:10.841 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:35:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:10 smithi159 ceph-mon[45336]: pgmap v1540: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 11286/147261 objects degraded (7.664%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:35:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:10 smithi159 ceph-mon[45336]: osdmap e591: 12 total, 11 up, 11 in 2024-04-15T18:35:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:10.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:10 smithi049 ceph-mon[40087]: pgmap v1540: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 11286/147261 objects degraded (7.664%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:35:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:10 smithi049 ceph-mon[40087]: osdmap e591: 12 total, 11 up, 11 in 2024-04-15T18:35:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:11.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:11.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:11.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:10 smithi082 ceph-mon[44206]: pgmap v1540: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 11286/147261 objects degraded (7.664%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:35:11.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:10 smithi082 ceph-mon[44206]: osdmap e591: 12 total, 11 up, 11 in 2024-04-15T18:35:11.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:11.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:11 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:11.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:11.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:11 smithi049 ceph-mon[40087]: osdmap e592: 12 total, 11 up, 11 in 2024-04-15T18:35:11.768 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:11 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:11.768 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:11.768 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:11 smithi082 ceph-mon[44206]: osdmap e592: 12 total, 11 up, 11 in 2024-04-15T18:35:11.790 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:11.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:11.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:11.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:11 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:11.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2679484086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:11.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:11 smithi159 ceph-mon[45336]: osdmap e592: 12 total, 11 up, 11 in 2024-04-15T18:35:11.929 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:12.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:12.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:12.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:12.213 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:12.245 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:12.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:12.353 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:12.397 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:12.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:12.503 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:12.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:12.574 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:12.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:12.653 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:12.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:12.732 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:12.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:12.800 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:12.837 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:12.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:12 smithi049 ceph-mon[40087]: pgmap v1543: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 10786/143964 objects degraded (7.492%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:12.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 10786/143964 objects degraded (7.492%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:12.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1023189248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:12.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:12 smithi159 ceph-mon[45336]: pgmap v1543: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 10786/143964 objects degraded (7.492%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:12.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 10786/143964 objects degraded (7.492%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:12.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1023189248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:12 smithi082 ceph-mon[44206]: pgmap v1543: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 10786/143964 objects degraded (7.492%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 10786/143964 objects degraded (7.492%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1023189248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:13.675 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:13.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.853 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:13.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.153 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: pgmap v1544: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 836 op/s; 10332/136872 objects degraded (7.549%); 15 MiB/s, 237 objects/s recovering 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:14.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:14.800 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:14.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:14.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: pgmap v1544: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 836 op/s; 10332/136872 objects degraded (7.549%); 15 MiB/s, 237 objects/s recovering 2024-04-15T18:35:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:14.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:14.938 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:14.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:14.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: pgmap v1544: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 836 op/s; 10332/136872 objects degraded (7.549%); 15 MiB/s, 237 objects/s recovering 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:15.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:15.065 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:15.121 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:15.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:35:15.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:15 smithi049 ceph-mon[40087]: osdmap e593: 12 total, 11 up, 11 in 2024-04-15T18:35:15.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:15 smithi159 ceph-mon[45336]: osdmap e593: 12 total, 11 up, 11 in 2024-04-15T18:35:16.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:15 smithi082 ceph-mon[44206]: osdmap e593: 12 total, 11 up, 11 in 2024-04-15T18:35:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:16 smithi159 ceph-mon[45336]: pgmap v1546: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 10332/136221 objects degraded (7.585%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:16.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:16.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:16 smithi049 ceph-mon[40087]: pgmap v1546: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 10332/136221 objects degraded (7.585%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:16.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:16.998 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:35:17.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:16 smithi082 ceph-mon[44206]: pgmap v1546: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 613 op/s; 10332/136221 objects degraded (7.585%); 9.9 MiB/s, 158 objects/s recovering 2024-04-15T18:35:17.022 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:17.057 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:17.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:17.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:17.217 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:17.274 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:17.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:17.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:17.407 INFO:teuthology.orchestra.run.smithi082.stdout: marking collection for removal 2024-04-15T18:35:17.413 INFO:teuthology.orchestra.run.smithi082.stdout:setting '_remove' omap key 2024-04-15T18:35:17.419 INFO:teuthology.orchestra.run.smithi082.stdout:finish_remove_pgs 6.3_head removing 6.3 2024-04-15T18:35:17.420 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:17.510 INFO:teuthology.orchestra.run.smithi082.stdout:Remove successful 2024-04-15T18:35:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:17 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:17.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:17 smithi049 ceph-mon[40087]: osdmap e594: 12 total, 11 up, 11 in 2024-04-15T18:35:17.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 10332/136221 objects degraded (7.585%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:17.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:17 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:17.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:17.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:17 smithi159 ceph-mon[45336]: osdmap e594: 12 total, 11 up, 11 in 2024-04-15T18:35:17.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 10332/136221 objects degraded (7.585%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:17.991 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:17 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:17.991 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3590129276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:17.991 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:17 smithi082 ceph-mon[44206]: osdmap e594: 12 total, 11 up, 11 in 2024-04-15T18:35:17.992 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 10332/136221 objects degraded (7.585%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:18.159 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:35:18.159 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:35:18.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:18 smithi159 ceph-mon[45336]: pgmap v1548: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 9845/131103 objects degraded (7.509%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:35:18.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:18 smithi049 ceph-mon[40087]: pgmap v1548: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 9845/131103 objects degraded (7.509%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:35:18.975 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:18.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:18.992 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:18 smithi082 ceph-mon[44206]: pgmap v1548: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 9845/131103 objects degraded (7.509%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:35:19.036 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:35:19.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:19.128 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:19.406 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:19.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.6 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.3.6 2024-04-15T18:35:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:19.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:19.472 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:35:19.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:19.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:19.641 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:19.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:19.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:19.750 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.6/config 2024-04-15T18:35:19.784 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:19.804 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:19.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:19.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:19.957 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:19.985 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:19.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:20.093 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:20.137 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:20.405 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:20.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:20.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:20.541 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:20.564 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:20.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:20.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:20.713 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:20 smithi082 ceph-mon[44206]: pgmap v1549: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 9401/126630 objects degraded (7.424%); 11 MiB/s, 173 objects/s recovering 2024-04-15T18:35:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:20 smithi082 ceph-mon[44206]: osdmap e595: 12 total, 11 up, 11 in 2024-04-15T18:35:20.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:20.854 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:20.854 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:20.854 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:20 smithi049 ceph-mon[40087]: pgmap v1549: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 9401/126630 objects degraded (7.424%); 11 MiB/s, 173 objects/s recovering 2024-04-15T18:35:20.854 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:20 smithi049 ceph-mon[40087]: osdmap e595: 12 total, 11 up, 11 in 2024-04-15T18:35:20.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:20.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:20.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:20.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:20 smithi159 ceph-mon[45336]: pgmap v1549: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 9401/126630 objects degraded (7.424%); 11 MiB/s, 173 objects/s recovering 2024-04-15T18:35:20.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:20 smithi159 ceph-mon[45336]: osdmap e595: 12 total, 11 up, 11 in 2024-04-15T18:35:20.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:21.000 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:21.064 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:21.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:21.172 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:21.209 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:21.422 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:21.488 INFO:teuthology.orchestra.run.smithi082.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:35:21.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:35:21.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:21.547 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:21.567 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:21.589 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:21.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:21.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:21.645 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:21.681 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:21.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:21.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:21.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:21.751 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:21.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:21.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:21.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1146664658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:21.834 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:21.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:21.885 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:21.886 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1146664658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:21.889 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:21.910 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:21.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:21.922 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:21.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:21.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1146664658' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:22.010 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:22.010 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:22.013 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:22.061 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:22.118 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:22.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:22.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:22.262 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:22.295 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:22.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:22.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:22.443 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:22.469 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:22.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:22.577 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:22.626 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: pgmap v1551: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 8778/123345 objects degraded (7.117%); 14 MiB/s, 227 objects/s recovering 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.676 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.677 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8778/123345 objects degraded (7.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:22.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: pgmap v1551: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 8778/123345 objects degraded (7.117%); 14 MiB/s, 227 objects/s recovering 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:22.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8778/123345 objects degraded (7.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:23.038 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:23.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:23.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: pgmap v1551: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 8778/123345 objects degraded (7.117%); 14 MiB/s, 227 objects/s recovering 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8778/123345 objects degraded (7.117%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:23.169 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:23.188 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:23.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:23.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:23.321 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:23.522 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:23.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:23.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:23.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:23.669 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:23.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:23.941 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:23.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:23.986 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:23.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:24.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:get_pg_num_history pg_num_history pg_num_history(e561 pg_nums {1={19=1},5={358=16},6={541=16}} deleted_pools 539,5) 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Importing pgid 6.3 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c002130a:::benchmark_data_smithi159_139342_object39504:head# 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c002133f:::benchmark_data_smithi159_139342_object19706:head# 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00285a9:::benchmark_data_smithi159_139342_object38827:head# 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0029ebe:::benchmark_data_smithi159_139342_object12995:head# 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.207 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0038130:::benchmark_data_smithi159_139342_object39221:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00792cc:::benchmark_data_smithi159_139342_object34550:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c008c4d5:::benchmark_data_smithi159_139342_object46254:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0098b17:::benchmark_data_smithi159_139342_object24966:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c009b657:::benchmark_data_smithi159_139342_object238:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00a7de3:::benchmark_data_smithi159_139342_object40841:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00bb310:::benchmark_data_smithi159_139342_object2884:head# 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.208 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00bf5c9:::benchmark_data_smithi159_139342_object22049:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00cdf97:::benchmark_data_smithi159_139342_object48773:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00dd19e:::benchmark_data_smithi159_139342_object35291:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00e58e2:::benchmark_data_smithi159_139342_object556:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c00ea767:::benchmark_data_smithi159_139342_object41515:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c010428f:::benchmark_data_smithi159_139342_object27847:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0143a99:::benchmark_data_smithi159_139342_object52461:head# 2024-04-15T18:35:24.209 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0146c62:::benchmark_data_smithi159_139342_object16386:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c014fdd2:::benchmark_data_smithi159_139342_object18051:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0159ba7:::benchmark_data_smithi159_139342_object26964:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0173924:::benchmark_data_smithi159_139342_object12030:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c01bd227:::benchmark_data_smithi159_139342_object54735:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c01c88b2:::benchmark_data_smithi159_139342_object23240:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c01d29b2:::benchmark_data_smithi159_139342_object25027:head# 2024-04-15T18:35:24.210 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c01d69eb:::benchmark_data_smithi159_139342_object56885:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0200167:::benchmark_data_smithi159_139342_object32266:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c021fe3e:::benchmark_data_smithi159_139342_object34453:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0225b23:::benchmark_data_smithi159_139342_object10141:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0257073:::benchmark_data_smithi159_139342_object54794:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0258f5b:::benchmark_data_smithi159_139342_object8094:head# 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c02611d3:::benchmark_data_smithi159_139342_object36330:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c027225a:::benchmark_data_smithi159_139342_object9958:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c02a9e2e:::benchmark_data_smithi159_139342_object46664:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c02eee17:::benchmark_data_smithi159_139342_object16829:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c02f4652:::benchmark_data_smithi159_139342_object4369:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c02f69b7:::benchmark_data_smithi159_139342_object35102:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0301139:::benchmark_data_smithi159_139342_object31928:head# 2024-04-15T18:35:24.212 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0301840:::benchmark_data_smithi159_139342_object50480:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0308220:::benchmark_data_smithi159_139342_object42109:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0332639:::benchmark_data_smithi159_139342_object14290:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c033891a:::benchmark_data_smithi159_139342_object54203:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0347c88:::benchmark_data_smithi159_139342_object20082:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c034e434:::benchmark_data_smithi159_139342_object10271:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c035d917:::benchmark_data_smithi159_139342_object52368:head# 2024-04-15T18:35:24.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c037fcd7:::benchmark_data_smithi159_139342_object44932:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c03a81e6:::benchmark_data_smithi159_139342_object56720:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c03e5413:::benchmark_data_smithi159_139342_object2151:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c03fd0b6:::benchmark_data_smithi159_139342_object39735:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c042428a:::benchmark_data_smithi159_139342_object30693:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0427ad7:::benchmark_data_smithi159_139342_object36110:head# 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c043a567:::benchmark_data_smithi159_139342_object13352:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0444743:::benchmark_data_smithi159_139342_object14073:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0492976:::benchmark_data_smithi159_139342_object21193:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c049934b:::benchmark_data_smithi159_139342_object6245:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04b262d:::benchmark_data_smithi159_139342_object50792:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04b3b22:::benchmark_data_smithi159_139342_object42224:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04b5563:::benchmark_data_smithi159_139342_object33941:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04bbf82:::benchmark_data_smithi159_139342_object11642:head# 2024-04-15T18:35:24.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04c044c:::benchmark_data_smithi159_139342_object45246:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04ccf5a:::benchmark_data_smithi159_139342_object137:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04d6048:::benchmark_data_smithi159_139342_object42739:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04dc5ac:::benchmark_data_smithi159_139342_object50682:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c04fb026:::benchmark_data_smithi159_139342_object9884:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0506692:::benchmark_data_smithi159_139342_object36609:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0517b72:::benchmark_data_smithi159_139342_object38262:head# 2024-04-15T18:35:24.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c054681e:::benchmark_data_smithi159_139342_object11529:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0554649:::benchmark_data_smithi159_139342_object20979:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c056e1c3:::benchmark_data_smithi159_139342_object31367:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c058b07a:::benchmark_data_smithi159_139342_object38504:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c05929f8:::benchmark_data_smithi159_139342_object1704:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c059df5a:::benchmark_data_smithi159_139342_object27238:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c059efd8:::benchmark_data_smithi159_139342_object5607:head# 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c05d3ca4:::benchmark_data_smithi159_139342_object40119:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0617eae:::benchmark_data_smithi159_139342_object43056:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0623da2:::benchmark_data_smithi159_139342_object14371:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c062772d:::benchmark_data_smithi159_139342_object39401:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c065495b:::benchmark_data_smithi159_139342_object57253:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c065c645:::benchmark_data_smithi159_139342_object4023:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c066d678:::benchmark_data_smithi159_139342_object34397:head# 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.218 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0673948:::benchmark_data_smithi159_139342_object30457:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0688d97:::benchmark_data_smithi159_139342_object30424:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0696de1:::benchmark_data_smithi159_139342_object11112:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0696e33:::benchmark_data_smithi159_139342_object18121:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c069da58:::benchmark_data_smithi159_139342_object52710:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c06a49c5:::benchmark_data_smithi159_139342_object13696:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0711721:::benchmark_data_smithi159_139342_object17546:head# 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0726498:::benchmark_data_smithi159_139342_object15966:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c073aea1:::benchmark_data_smithi159_139342_object52144:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c073ebbc:::benchmark_data_smithi159_139342_object27603:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0744a76:::benchmark_data_smithi159_139342_object15907:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0745b53:::benchmark_data_smithi159_139342_object2945:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c075890a:::benchmark_data_smithi159_139342_object52064:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c075b95b:::benchmark_data_smithi159_139342_object47168:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c077a3f0:::benchmark_data_smithi159_139342_object52499:head# 2024-04-15T18:35:24.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c078f785:::benchmark_data_smithi159_139342_object54740:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07af996:::benchmark_data_smithi159_139342_object31020:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07c15b2:::benchmark_data_smithi159_139342_object19527:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07d2fb0:::benchmark_data_smithi159_139342_object15902:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07d8844:::benchmark_data_smithi159_139342_object40076:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07e10eb:::benchmark_data_smithi159_139342_object28812:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c07f1752:::benchmark_data_smithi159_139342_object53473:head# 2024-04-15T18:35:24.221 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c083e7fc:::benchmark_data_smithi159_139342_object7756:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08530f8:::benchmark_data_smithi159_139342_object33305:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0855eec:::benchmark_data_smithi159_139342_object55211:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0857504:::benchmark_data_smithi159_139342_object49337:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c085c0ed:::benchmark_data_smithi159_139342_object17002:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c085d390:::benchmark_data_smithi159_139342_object27500:head# 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0874a39:::benchmark_data_smithi159_139342_object47034:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08bc6d5:::benchmark_data_smithi159_139342_object2436:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08cd4f0:::benchmark_data_smithi159_139342_object42879:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08dfeba:::benchmark_data_smithi159_139342_object38603:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08eb02a:::benchmark_data_smithi159_139342_object50428:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08f30ea:::benchmark_data_smithi159_139342_object2316:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08f316c:::benchmark_data_smithi159_139342_object20342:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08f4b38:::benchmark_data_smithi159_139342_object38300:head# 2024-04-15T18:35:24.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c08f63c4:::benchmark_data_smithi159_139342_object34280:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0906d61:::benchmark_data_smithi159_139342_object53232:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c09341bd:::benchmark_data_smithi159_139342_object2157:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c09380fd:::benchmark_data_smithi159_139342_object38439:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0979c2b:::benchmark_data_smithi159_139342_object22801:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c09b4918:::benchmark_data_smithi159_139342_object13321:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c09c969e:::benchmark_data_smithi159_139342_object43374:head# 2024-04-15T18:35:24.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c09c9c7b:::benchmark_data_smithi159_139342_object32656:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a05cb2:::benchmark_data_smithi159_139342_object15039:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a08b32:::benchmark_data_smithi159_139342_object54716:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a0a11d:::benchmark_data_smithi159_139342_object9905:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a24972:::benchmark_data_smithi159_139342_object27213:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a27be2:::benchmark_data_smithi159_139342_object32707:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a2f6fd:::benchmark_data_smithi159_139342_object48442:head# 2024-04-15T18:35:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a37b23:::benchmark_data_smithi159_139342_object22593:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a50f43:::benchmark_data_smithi159_139342_object15710:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a7052c:::benchmark_data_smithi159_139342_object5230:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0a97cb9:::benchmark_data_smithi159_139342_object11644:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0adf653:::benchmark_data_smithi159_139342_object831:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0af7272:::benchmark_data_smithi159_139342_object47527:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b05a08:::benchmark_data_smithi159_139342_object2835:head# 2024-04-15T18:35:24.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b178b9:::benchmark_data_smithi159_139342_object11316:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b183e3:::benchmark_data_smithi159_139342_object4746:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b428b1:::benchmark_data_smithi159_139342_object51501:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b5e8f6:::benchmark_data_smithi159_139342_object49426:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0b6406c:::benchmark_data_smithi159_139342_object41499:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ba4258:::benchmark_data_smithi159_139342_object36232:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ba5a46:::benchmark_data_smithi159_139342_object6372:head# 2024-04-15T18:35:24.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0bae154:::benchmark_data_smithi159_139342_object35305:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0bc9a97:::benchmark_data_smithi159_139342_object14919:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0bcae3d:::benchmark_data_smithi159_139342_object50391:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0bcafbd:::benchmark_data_smithi159_139342_object26143:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c0d1c8:::benchmark_data_smithi159_139342_object31130:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c0e3ef:::benchmark_data_smithi159_139342_object15535:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c11552:::benchmark_data_smithi159_139342_object4972:head# 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.228 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c14709:::benchmark_data_smithi159_139342_object52854:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c3670e:::benchmark_data_smithi159_139342_object5942:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c6aeb7:::benchmark_data_smithi159_139342_object45735:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c7667a:::benchmark_data_smithi159_139342_object48770:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c84222:::benchmark_data_smithi159_139342_object42921:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c8a594:::benchmark_data_smithi159_139342_object42577:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0c980a5:::benchmark_data_smithi159_139342_object56991:head# 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.229 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0cb8c97:::benchmark_data_smithi159_139342_object7178:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0cc8f54:::benchmark_data_smithi159_139342_object22101:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0cd62b4:::benchmark_data_smithi159_139342_object39607:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ceb71a:::benchmark_data_smithi159_139342_object51403:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0cfa1a5:::benchmark_data_smithi159_139342_object23214:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0cfcab8:::benchmark_data_smithi159_139342_object35823:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d59503:::benchmark_data_smithi159_139342_object43285:head# 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.230 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d68cbd:::benchmark_data_smithi159_139342_object51603:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d6b335:::benchmark_data_smithi159_139342_object13916:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d6bf09:::benchmark_data_smithi159_139342_object12380:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d6f95f:::benchmark_data_smithi159_139342_object30931:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0d94312:::benchmark_data_smithi159_139342_object19667:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0de97b3:::benchmark_data_smithi159_139342_object50225:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e05a69:::benchmark_data_smithi159_139342_object49212:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e08557:::benchmark_data_smithi159_139342_object37156:head# 2024-04-15T18:35:24.231 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e40452:::benchmark_data_smithi159_139342_object27172:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e452c6:::benchmark_data_smithi159_139342_object55728:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e69664:::benchmark_data_smithi159_139342_object15704:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0e98ff2:::benchmark_data_smithi159_139342_object31240:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ea9488:::benchmark_data_smithi159_139342_object31013:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ebc682:::benchmark_data_smithi159_139342_object32551:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ebfe6e:::benchmark_data_smithi159_139342_object22849:head# 2024-04-15T18:35:24.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ec037f:::benchmark_data_smithi159_139342_object53540:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ecdb83:::benchmark_data_smithi159_139342_object41755:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ee4917:::benchmark_data_smithi159_139342_object31937:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0ef57f3:::benchmark_data_smithi159_139342_object32445:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f0fd40:::benchmark_data_smithi159_139342_object15458:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f1c901:::benchmark_data_smithi159_139342_object8131:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f222d1:::benchmark_data_smithi159_139342_object30144:head# 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.233 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f3a49c:::benchmark_data_smithi159_139342_object22023:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f6051c:::benchmark_data_smithi159_139342_object37363:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f6dc6e:::benchmark_data_smithi159_139342_object48884:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f6f050:::benchmark_data_smithi159_139342_object20265:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f96e62:::benchmark_data_smithi159_139342_object24884:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0f9f249:::benchmark_data_smithi159_139342_object57041:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0fd14ec:::benchmark_data_smithi159_139342_object1631:head# 2024-04-15T18:35:24.234 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0fe7bfd:::benchmark_data_smithi159_139342_object28553:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c0fefc07:::benchmark_data_smithi159_139342_object31866:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10024e6:::benchmark_data_smithi159_139342_object46598:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c100a9c9:::benchmark_data_smithi159_139342_object22688:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1010a89:::benchmark_data_smithi159_139342_object28241:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1039c5f:::benchmark_data_smithi159_139342_object23835:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1059772:::benchmark_data_smithi159_139342_object57201:head# 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.235 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c105cb78:::benchmark_data_smithi159_139342_object22964:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c105fd6c:::benchmark_data_smithi159_139342_object18202:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1062c70:::benchmark_data_smithi159_139342_object31911:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c107be8c:::benchmark_data_smithi159_139342_object54088:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c108634c:::benchmark_data_smithi159_139342_object19091:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c109c342:::benchmark_data_smithi159_139342_object54772:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10ac302:::benchmark_data_smithi159_139342_object25850:head# 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.236 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10be188:::benchmark_data_smithi159_139342_object11090:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10d1f0b:::benchmark_data_smithi159_139342_object24275:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10e07a7:::benchmark_data_smithi159_139342_object56583:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10f2d39:::benchmark_data_smithi159_139342_object56250:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c10f686b:::benchmark_data_smithi159_139342_object16761:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1105b38:::benchmark_data_smithi159_139342_object49285:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1122c11:::benchmark_data_smithi159_139342_object49147:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c112a63a:::benchmark_data_smithi159_139342_object22631:head# 2024-04-15T18:35:24.237 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1141785:::benchmark_data_smithi159_139342_object49663:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c114e1dd:::benchmark_data_smithi159_139342_object42279:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c11564a9:::benchmark_data_smithi159_139342_object39697:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1197e51:::benchmark_data_smithi159_139342_object32342:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c11b7999:::benchmark_data_smithi159_139342_object4759:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c11d1414:::benchmark_data_smithi159_139342_object4749:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1215c05:::benchmark_data_smithi159_139342_object24839:head# 2024-04-15T18:35:24.238 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c121e6a4:::benchmark_data_smithi159_139342_object14961:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c124cc08:::benchmark_data_smithi159_139342_object12063:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1250d74:::benchmark_data_smithi159_139342_object22047:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1258f08:::benchmark_data_smithi159_139342_object47093:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1262e54:::benchmark_data_smithi159_139342_object29039:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1288a9e:::benchmark_data_smithi159_139342_object11842:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c129b1ec:::benchmark_data_smithi159_139342_object49974:head# 2024-04-15T18:35:24.239 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12cb96f:::benchmark_data_smithi159_139342_object24626:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12d96e3:::benchmark_data_smithi159_139342_object44431:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12da5ba:::benchmark_data_smithi159_139342_object56507:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12e9893:::benchmark_data_smithi159_139342_object53553:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12ee8a6:::benchmark_data_smithi159_139342_object52147:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c12f6933:::benchmark_data_smithi159_139342_object33742:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1300bf1:::benchmark_data_smithi159_139342_object21558:head# 2024-04-15T18:35:24.240 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c130250b:::benchmark_data_smithi159_139342_object2545:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c132b787:::benchmark_data_smithi159_139342_object46688:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c133a657:::benchmark_data_smithi159_139342_object10013:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c135d4fd:::benchmark_data_smithi159_139342_object28707:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c137da38:::benchmark_data_smithi159_139342_object44545:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c137defd:::benchmark_data_smithi159_139342_object31414:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1380756:::benchmark_data_smithi159_139342_object9436:head# 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.241 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c138c484:::benchmark_data_smithi159_139342_object489:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c13a7c2a:::benchmark_data_smithi159_139342_object34756:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c13d26fe:::benchmark_data_smithi159_139342_object45506:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c13daab6:::benchmark_data_smithi159_139342_object37823:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14052fa:::benchmark_data_smithi159_139342_object837:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c140716d:::benchmark_data_smithi159_139342_object18591:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c140f908:::benchmark_data_smithi159_139342_object54814:head# 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.242 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1436e69:::benchmark_data_smithi159_139342_object11176:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1440e14:::benchmark_data_smithi159_139342_object31330:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1441e50:::benchmark_data_smithi159_139342_object46931:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1489dcb:::benchmark_data_smithi159_139342_object8690:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c149a079:::benchmark_data_smithi159_139342_object42310:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c149cfe0:::benchmark_data_smithi159_139342_object38743:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14b435c:::benchmark_data_smithi159_139342_object37175:head# 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.243 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14b5240:::benchmark_data_smithi159_139342_object7507:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14bada4:::benchmark_data_smithi159_139342_object54120:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14cbf00:::benchmark_data_smithi159_139342_object31756:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14cec93:::benchmark_data_smithi159_139342_object38157:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14ced4a:::benchmark_data_smithi159_139342_object31791:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c14d0ec3:::benchmark_data_smithi159_139342_object630:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1528ae0:::benchmark_data_smithi159_139342_object57254:head# 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c152e5e8:::benchmark_data_smithi159_139342_object25277:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c153fe93:::benchmark_data_smithi159_139342_object21336:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15460de:::benchmark_data_smithi159_139342_object39228:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c154c492:::benchmark_data_smithi159_139342_object30391:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15532db:::benchmark_data_smithi159_139342_object19032:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15646ac:::benchmark_data_smithi159_139342_object27760:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1567d7f:::benchmark_data_smithi159_139342_object54082:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15755db:::benchmark_data_smithi159_139342_object20058:head# 2024-04-15T18:35:24.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1576b02:::benchmark_data_smithi159_139342_object16039:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1580e7f:::benchmark_data_smithi159_139342_object56332:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1591b24:::benchmark_data_smithi159_139342_object19637:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15950b2:::benchmark_data_smithi159_139342_object7005:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1595fb9:::benchmark_data_smithi159_139342_object13451:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15a4fd0:::benchmark_data_smithi159_139342_object57035:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15ab195:::benchmark_data_smithi159_139342_object19964:head# 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.246 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15b5257:::benchmark_data_smithi159_139342_object5891:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15be175:::benchmark_data_smithi159_139342_object26118:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c15e6d9e:::benchmark_data_smithi159_139342_object13610:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c160b5c5:::benchmark_data_smithi159_139342_object54619:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1622901:::benchmark_data_smithi159_139342_object44475:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16247a4:::benchmark_data_smithi159_139342_object13247:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c162f2d9:::benchmark_data_smithi159_139342_object18157:head# 2024-04-15T18:35:24.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c163ab5c:::benchmark_data_smithi159_139342_object16944:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c166183c:::benchmark_data_smithi159_139342_object23910:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c168a638:::benchmark_data_smithi159_139342_object15146:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c169319a:::benchmark_data_smithi159_139342_object56077:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c169b458:::benchmark_data_smithi159_139342_object3752:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16a1c91:::benchmark_data_smithi159_139342_object40980:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16b4e39:::benchmark_data_smithi159_139342_object4573:head# 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16b9fb0:::benchmark_data_smithi159_139342_object35805:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16ba253:::benchmark_data_smithi159_139342_object44401:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16d2086:::benchmark_data_smithi159_139342_object54279:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c16e6d7e:::benchmark_data_smithi159_139342_object26910:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1707d54:::benchmark_data_smithi159_139342_object2603:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c171744d:::benchmark_data_smithi159_139342_object15061:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1725ac1:::benchmark_data_smithi159_139342_object7436:head# 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c174dc4d:::benchmark_data_smithi159_139342_object56298:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1765781:::benchmark_data_smithi159_139342_object40701:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17735cb:::benchmark_data_smithi159_139342_object19198:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c178d184:::benchmark_data_smithi159_139342_object53834:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c179e03e:::benchmark_data_smithi159_139342_object11377:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17a300d:::benchmark_data_smithi159_139342_object36864:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17bc0ff:::benchmark_data_smithi159_139342_object334:head# 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17c3338:::benchmark_data_smithi159_139342_object52864:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17ca83a:::benchmark_data_smithi159_139342_object39763:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17d94d2:::benchmark_data_smithi159_139342_object38542:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17e7e64:::benchmark_data_smithi159_139342_object27861:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17ec8f5:::benchmark_data_smithi159_139342_object14194:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c17f092a:::benchmark_data_smithi159_139342_object45249:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c180023d:::benchmark_data_smithi159_139342_object14595:head# 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.251 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1801594:::benchmark_data_smithi159_139342_object47012:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1813486:::benchmark_data_smithi159_139342_object10927:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18252fd:::benchmark_data_smithi159_139342_object8799:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c183a85c:::benchmark_data_smithi159_139342_object13206:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c184b0fb:::benchmark_data_smithi159_139342_object1252:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c184ef7f:::benchmark_data_smithi159_139342_object12248:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1862b02:::benchmark_data_smithi159_139342_object6986:head# 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.252 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18982c1:::benchmark_data_smithi159_139342_object35771:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18bb042:::benchmark_data_smithi159_139342_object51765:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18ce85d:::benchmark_data_smithi159_139342_object9351:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18d8b55:::benchmark_data_smithi159_139342_object368:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18dc613:::benchmark_data_smithi159_139342_object42846:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18e4107:::benchmark_data_smithi159_139342_object40184:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18e9926:::benchmark_data_smithi159_139342_object35081:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c18f27ac:::benchmark_data_smithi159_139342_object30593:head# 2024-04-15T18:35:24.253 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1900b12:::benchmark_data_smithi159_139342_object3305:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1917364:::benchmark_data_smithi159_139342_object15542:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c191e98d:::benchmark_data_smithi159_139342_object42532:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c192551f:::benchmark_data_smithi159_139342_object51481:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c192c2a8:::benchmark_data_smithi159_139342_object47060:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c195174c:::benchmark_data_smithi159_139342_object13555:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1958dca:::benchmark_data_smithi159_139342_object55171:head# 2024-04-15T18:35:24.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19596fc:::benchmark_data_smithi159_139342_object213:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1970d3a:::benchmark_data_smithi159_139342_object55888:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1978176:::benchmark_data_smithi159_139342_object7455:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1978230:::benchmark_data_smithi159_139342_object55228:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1985b12:::benchmark_data_smithi159_139342_object13293:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c198fdfb:::benchmark_data_smithi159_139342_object4750:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1992911:::benchmark_data_smithi159_139342_object48066:head# 2024-04-15T18:35:24.255 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19a9be4:::benchmark_data_smithi159_139342_object35635:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19adc71:::benchmark_data_smithi159_139342_object3507:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19b2d10:::benchmark_data_smithi159_139342_object51174:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19b7e61:::benchmark_data_smithi159_139342_object7207:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19be9cb:::benchmark_data_smithi159_139342_object32692:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19e36b1:::benchmark_data_smithi159_139342_object49183:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19eed71:::benchmark_data_smithi159_139342_object15516:head# 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c19ff268:::benchmark_data_smithi159_139342_object40709:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1a005e7:::benchmark_data_smithi159_139342_object48893:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1a43682:::benchmark_data_smithi159_139342_object24053:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1a517b4:::benchmark_data_smithi159_139342_object8894:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1a67d18:::benchmark_data_smithi159_139342_object48370:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1a99f4c:::benchmark_data_smithi159_139342_object37202:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1aa43fb:::benchmark_data_smithi159_139342_object14313:head# 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1abe1f0:::benchmark_data_smithi159_139342_object49252:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1af3118:::benchmark_data_smithi159_139342_object26782:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b173a9:::benchmark_data_smithi159_139342_object13248:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b1a088:::benchmark_data_smithi159_139342_object28914:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b2c6da:::benchmark_data_smithi159_139342_object44404:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b2c99d:::benchmark_data_smithi159_139342_object37615:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b356dd:::benchmark_data_smithi159_139342_object42193:head# 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b375fd:::benchmark_data_smithi159_139342_object44964:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b40209:::benchmark_data_smithi159_139342_object48960:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b576fe:::benchmark_data_smithi159_139342_object15806:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b81a8a:::benchmark_data_smithi159_139342_object23480:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b8be33:::benchmark_data_smithi159_139342_object29268:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b90194:::benchmark_data_smithi159_139342_object30582:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1b95e8b:::benchmark_data_smithi159_139342_object8707:head# 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.259 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1bc4ff0:::benchmark_data_smithi159_139342_object16124:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1bcd071:::benchmark_data_smithi159_139342_object27426:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1be573b:::benchmark_data_smithi159_139342_object4484:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1c82621:::benchmark_data_smithi159_139342_object57286:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1c87790:::benchmark_data_smithi159_139342_object717:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1c8f0f6:::benchmark_data_smithi159_139342_object16881:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1caa510:::benchmark_data_smithi159_139342_object36272:head# 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.260 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1cac427:::benchmark_data_smithi159_139342_object47560:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1ce6ac8:::benchmark_data_smithi159_139342_object9163:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1ce796d:::benchmark_data_smithi159_139342_object660:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1cf8f49:::benchmark_data_smithi159_139342_object39902:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d1589a:::benchmark_data_smithi159_139342_object13850:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d375cd:::benchmark_data_smithi159_139342_object51861:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d4b400:::benchmark_data_smithi159_139342_object13112:head# 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.261 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d76139:::benchmark_data_smithi159_139342_object15271:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d766ad:::benchmark_data_smithi159_139342_object39354:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d7fab9:::benchmark_data_smithi159_139342_object41475:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d84dba:::benchmark_data_smithi159_139342_object17208:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d8ccb4:::benchmark_data_smithi159_139342_object35835:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1d92aa4:::benchmark_data_smithi159_139342_object50433:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1dbb7b9:::benchmark_data_smithi159_139342_object14472:head# 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.262 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1dc36a9:::benchmark_data_smithi159_139342_object3794:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1dc5b6e:::benchmark_data_smithi159_139342_object26515:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1dc76a4:::benchmark_data_smithi159_139342_object55021:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1dd61a3:::benchmark_data_smithi159_139342_object1827:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1e01600:::benchmark_data_smithi159_139342_object15249:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1e5b814:::benchmark_data_smithi159_139342_object52430:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1e6ce17:::benchmark_data_smithi159_139342_object5098:head# 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1e706c6:::benchmark_data_smithi159_139342_object12669:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1e98f88:::benchmark_data_smithi159_139342_object22916:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1ea6b28:::benchmark_data_smithi159_139342_object35228:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1eaae5c:::benchmark_data_smithi159_139342_object23392:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1eb16f4:::benchmark_data_smithi159_139342_object19632:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1eb39ad:::benchmark_data_smithi159_139342_object54160:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1eb5b76:::benchmark_data_smithi159_139342_object12070:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1edc3c9:::benchmark_data_smithi159_139342_object45917:head# 2024-04-15T18:35:24.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f095ee:::benchmark_data_smithi159_139342_object26738:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f21671:::benchmark_data_smithi159_139342_object35066:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f21751:::benchmark_data_smithi159_139342_object56780:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f27b83:::benchmark_data_smithi159_139342_object10468:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f28d35:::benchmark_data_smithi159_139342_object10238:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f3d1de:::benchmark_data_smithi159_139342_object38213:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f49ae6:::benchmark_data_smithi159_139342_object3499:head# 2024-04-15T18:35:24.265 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f552e7:::benchmark_data_smithi159_139342_object19703:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f5abe7:::benchmark_data_smithi159_139342_object3814:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f5b360:::benchmark_data_smithi159_139342_object39693:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f6798c:::benchmark_data_smithi159_139342_object54403:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f821ff:::benchmark_data_smithi159_139342_object37409:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f8de26:::benchmark_data_smithi159_139342_object975:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1f9a3cc:::benchmark_data_smithi159_139342_object35965:head# 2024-04-15T18:35:24.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1fc9bbe:::benchmark_data_smithi159_139342_object16771:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1fde58f:::benchmark_data_smithi159_139342_object35625:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1fe5851:::benchmark_data_smithi159_139342_object5081:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1fed14e:::benchmark_data_smithi159_139342_object30551:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c1fedb5c:::benchmark_data_smithi159_139342_object5329:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20046a3:::benchmark_data_smithi159_139342_object8578:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2014c96:::benchmark_data_smithi159_139342_object20214:head# 2024-04-15T18:35:24.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2016832:::benchmark_data_smithi159_139342_object40036:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c204c56a:::benchmark_data_smithi159_139342_object11662:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c206103b:::benchmark_data_smithi159_139342_object43140:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20626dd:::benchmark_data_smithi159_139342_object27353:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c206be03:::benchmark_data_smithi159_139342_object35567:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2080925:::benchmark_data_smithi159_139342_object24590:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20bc64e:::benchmark_data_smithi159_139342_object48836:head# 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.268 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20c3410:::benchmark_data_smithi159_139342_object22179:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20d44fb:::benchmark_data_smithi159_139342_object36010:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20d4abd:::benchmark_data_smithi159_139342_object2479:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c20ee788:::benchmark_data_smithi159_139342_object34443:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c210f0b0:::benchmark_data_smithi159_139342_object33618:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2112d2f:::benchmark_data_smithi159_139342_object38901:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c214298b:::benchmark_data_smithi159_139342_object35595:head# 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.269 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2153f6e:::benchmark_data_smithi159_139342_object23621:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c215dc9a:::benchmark_data_smithi159_139342_object4640:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21619c4:::benchmark_data_smithi159_139342_object46796:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2162a0b:::benchmark_data_smithi159_139342_object25567:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c216452b:::benchmark_data_smithi159_139342_object20343:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c217a082:::benchmark_data_smithi159_139342_object36435:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21985a2:::benchmark_data_smithi159_139342_object49586:head# 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.270 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21998f8:::benchmark_data_smithi159_139342_object44060:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21cfc61:::benchmark_data_smithi159_139342_object906:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21dbc34:::benchmark_data_smithi159_139342_object21208:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c21eb8c1:::benchmark_data_smithi159_139342_object37550:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c221ce7b:::benchmark_data_smithi159_139342_object32142:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2227436:::benchmark_data_smithi159_139342_object2859:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c222dbec:::benchmark_data_smithi159_139342_object35784:head# 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.271 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2234a84:::benchmark_data_smithi159_139342_object31576:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c223e1b9:::benchmark_data_smithi159_139342_object36189:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c223fffe:::benchmark_data_smithi159_139342_object12034:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2241112:::benchmark_data_smithi159_139342_object4839:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22478fd:::benchmark_data_smithi159_139342_object32726:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2253175:::benchmark_data_smithi159_139342_object46359:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22532b7:::benchmark_data_smithi159_139342_object52171:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c225c950:::benchmark_data_smithi159_139342_object39310:head# 2024-04-15T18:35:24.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22797fc:::benchmark_data_smithi159_139342_object44639:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c227dc04:::benchmark_data_smithi159_139342_object10000:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2287af2:::benchmark_data_smithi159_139342_object54096:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22a7ef6:::benchmark_data_smithi159_139342_object21431:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22cfc09:::benchmark_data_smithi159_139342_object9448:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22e1d77:::benchmark_data_smithi159_139342_object31658:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c22f9445:::benchmark_data_smithi159_139342_object7963:head# 2024-04-15T18:35:24.273 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2307c45:::benchmark_data_smithi159_139342_object15556:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c232b291:::benchmark_data_smithi159_139342_object50030:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c23537b6:::benchmark_data_smithi159_139342_object31266:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c235aa2e:::benchmark_data_smithi159_139342_object13240:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c237d6c4:::benchmark_data_smithi159_139342_object24272:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2390605:::benchmark_data_smithi159_139342_object48200:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c23bfdbe:::benchmark_data_smithi159_139342_object23581:head# 2024-04-15T18:35:24.274 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c23c4373:::benchmark_data_smithi159_139342_object31751:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2420bb4:::benchmark_data_smithi159_139342_object26570:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2442084:::benchmark_data_smithi159_139342_object19130:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c244b1cb:::benchmark_data_smithi159_139342_object40198:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c244e9fe:::benchmark_data_smithi159_139342_object12478:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2452646:::benchmark_data_smithi159_139342_object23437:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c245729c:::benchmark_data_smithi159_139342_object27377:head# 2024-04-15T18:35:24.275 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2458b2b:::benchmark_data_smithi159_139342_object38923:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c245c620:::benchmark_data_smithi159_139342_object41178:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c246220f:::benchmark_data_smithi159_139342_object49543:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24649f8:::benchmark_data_smithi159_139342_object48824:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24671fa:::benchmark_data_smithi159_139342_object46413:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c246a1ac:::benchmark_data_smithi159_139342_object52979:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24848b0:::benchmark_data_smithi159_139342_object49100:head# 2024-04-15T18:35:24.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24a5c55:::benchmark_data_smithi159_139342_object28047:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24add86:::benchmark_data_smithi159_139342_object46805:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c24bd708:::benchmark_data_smithi159_139342_object55375:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c250262e:::benchmark_data_smithi159_139342_object46925:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2502b59:::benchmark_data_smithi159_139342_object1071:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c250d3e0:::benchmark_data_smithi159_139342_object8089:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c250f564:::benchmark_data_smithi159_139342_object29651:head# 2024-04-15T18:35:24.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c252176b:::benchmark_data_smithi159_139342_object9373:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2527955:::benchmark_data_smithi159_139342_object35949:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2543c29:::benchmark_data_smithi159_139342_object5553:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2544c8c:::benchmark_data_smithi159_139342_object56123:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c255ba0d:::benchmark_data_smithi159_139342_object16372:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2578bfc:::benchmark_data_smithi159_139342_object52851:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2593a64:::benchmark_data_smithi159_139342_object30943:head# 2024-04-15T18:35:24.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c259a5c8:::benchmark_data_smithi159_139342_object13145:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c25b6cbc:::benchmark_data_smithi159_139342_object24263:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c25caf92:::benchmark_data_smithi159_139342_object9642:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c25cb334:::benchmark_data_smithi159_139342_object3598:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c25da488:::benchmark_data_smithi159_139342_object2745:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c25f144f:::benchmark_data_smithi159_139342_object42426:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c262af25:::benchmark_data_smithi159_139342_object41514:head# 2024-04-15T18:35:24.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c262de26:::benchmark_data_smithi159_139342_object27274:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c264ad14:::benchmark_data_smithi159_139342_object39594:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c264e62f:::benchmark_data_smithi159_139342_object54104:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c265ee38:::benchmark_data_smithi159_139342_object16789:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26709fd:::benchmark_data_smithi159_139342_object50956:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c267be44:::benchmark_data_smithi159_139342_object26268:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c268fa5b:::benchmark_data_smithi159_139342_object46047:head# 2024-04-15T18:35:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c269af15:::benchmark_data_smithi159_139342_object19516:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26a5cdd:::benchmark_data_smithi159_139342_object52029:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26bd19b:::benchmark_data_smithi159_139342_object42665:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26ce00b:::benchmark_data_smithi159_139342_object18979:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26d584f:::benchmark_data_smithi159_139342_object44103:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26dc157:::benchmark_data_smithi159_139342_object47264:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26dfa57:::benchmark_data_smithi159_139342_object53468:head# 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26e4e28:::benchmark_data_smithi159_139342_object5102:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c26ee199:::benchmark_data_smithi159_139342_object53276:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2701cad:::benchmark_data_smithi159_139342_object56792:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c271b65c:::benchmark_data_smithi159_139342_object38139:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2730f20:::benchmark_data_smithi159_139342_object40580:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2731045:::benchmark_data_smithi159_139342_object43557:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c273816d:::benchmark_data_smithi159_139342_object4422:head# 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c273b0dd:::benchmark_data_smithi159_139342_object51503:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27788d4:::benchmark_data_smithi159_139342_object39331:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c278aa79:::benchmark_data_smithi159_139342_object33312:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c278c9dd:::benchmark_data_smithi159_139342_object3990:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2791fa7:::benchmark_data_smithi159_139342_object14956:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27a89a9:::benchmark_data_smithi159_139342_object33666:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27b81c2:::benchmark_data_smithi159_139342_object31440:head# 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27c5616:::benchmark_data_smithi159_139342_object21984:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27dd48f:::benchmark_data_smithi159_139342_object35050:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c27fb4f8:::benchmark_data_smithi159_139342_object36283:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2830e66:::benchmark_data_smithi159_139342_object43144:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c283cfed:::benchmark_data_smithi159_139342_object46955:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2846464:::benchmark_data_smithi159_139342_object11016:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c284b934:::benchmark_data_smithi159_139342_object26516:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28679e3:::benchmark_data_smithi159_139342_object12218:head# 2024-04-15T18:35:24.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2868ba2:::benchmark_data_smithi159_139342_object52549:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c287b72b:::benchmark_data_smithi159_139342_object45569:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28a64de:::benchmark_data_smithi159_139342_object1562:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28c55af:::benchmark_data_smithi159_139342_object9261:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28e5376:::benchmark_data_smithi159_139342_object27040:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28eb4ce:::benchmark_data_smithi159_139342_object18640:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28efe31:::benchmark_data_smithi159_139342_object49982:head# 2024-04-15T18:35:24.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c28f39f3:::benchmark_data_smithi159_139342_object29792:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29018ce:::benchmark_data_smithi159_139342_object41952:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2920ef0:::benchmark_data_smithi159_139342_object9270:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2924e32:::benchmark_data_smithi159_139342_object13517:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29310b4:::benchmark_data_smithi159_139342_object54408:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c293518c:::benchmark_data_smithi159_139342_object34158:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2938cef:::benchmark_data_smithi159_139342_object2724:head# 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.286 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2949234:::benchmark_data_smithi159_139342_object49545:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c296e39b:::benchmark_data_smithi159_139342_object36758:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29778c2:::benchmark_data_smithi159_139342_object50223:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c297833e:::benchmark_data_smithi159_139342_object42723:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c298294e:::benchmark_data_smithi159_139342_object52667:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c299dd60:::benchmark_data_smithi159_139342_object51704:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c299f3a6:::benchmark_data_smithi159_139342_object6089:head# 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.287 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29bde5b:::benchmark_data_smithi159_139342_object21004:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29c70c9:::benchmark_data_smithi159_139342_object929:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29e484e:::benchmark_data_smithi159_139342_object4802:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29eb799:::benchmark_data_smithi159_139342_object2530:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29f4364:::benchmark_data_smithi159_139342_object35011:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c29f6bcc:::benchmark_data_smithi159_139342_object40310:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a09caa:::benchmark_data_smithi159_139342_object18896:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a14eb8:::benchmark_data_smithi159_139342_object51924:head# 2024-04-15T18:35:24.288 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a2942a:::benchmark_data_smithi159_139342_object37456:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a62a39:::benchmark_data_smithi159_139342_object24964:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a6d37b:::benchmark_data_smithi159_139342_object43683:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a73e5a:::benchmark_data_smithi159_139342_object9947:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2a8d5ef:::benchmark_data_smithi159_139342_object33515:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2aa07f9:::benchmark_data_smithi159_139342_object42762:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2aa83f2:::benchmark_data_smithi159_139342_object44199:head# 2024-04-15T18:35:24.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ac82d6:::benchmark_data_smithi159_139342_object8752:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2acd680:::benchmark_data_smithi159_139342_object46276:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ad2428:::benchmark_data_smithi159_139342_object7513:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ad4b4b:::benchmark_data_smithi159_139342_object3851:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2aed96c:::benchmark_data_smithi159_139342_object48376:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2af1349:::benchmark_data_smithi159_139342_object5499:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2af9815:::benchmark_data_smithi159_139342_object26530:head# 2024-04-15T18:35:24.290 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b00b39:::benchmark_data_smithi159_139342_object53623:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b2ab79:::benchmark_data_smithi159_139342_object23677:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b426ac:::benchmark_data_smithi159_139342_object14876:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b6537a:::benchmark_data_smithi159_139342_object33982:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b78aea:::benchmark_data_smithi159_139342_object45357:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b85fcb:::benchmark_data_smithi159_139342_object43163:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2b904d2:::benchmark_data_smithi159_139342_object15812:head# 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.291 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2bb1ed9:::benchmark_data_smithi159_139342_object483:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2bf1f71:::benchmark_data_smithi159_139342_object49940:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c02322:::benchmark_data_smithi159_139342_object56186:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c09b5d:::benchmark_data_smithi159_139342_object15356:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c09c02:::benchmark_data_smithi159_139342_object4190:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c0edaa:::benchmark_data_smithi159_139342_object9903:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c13fde:::benchmark_data_smithi159_139342_object24345:head# 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.292 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c2d044:::benchmark_data_smithi159_139342_object4569:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c4c8b1:::benchmark_data_smithi159_139342_object52963:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c61963:::benchmark_data_smithi159_139342_object36169:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c62243:::benchmark_data_smithi159_139342_object11032:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c6ec8d:::benchmark_data_smithi159_139342_object12824:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c8074a:::benchmark_data_smithi159_139342_object46105:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c8bb2f:::benchmark_data_smithi159_139342_object47322:head# 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c8c5c6:::benchmark_data_smithi159_139342_object26277:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c97717:::benchmark_data_smithi159_139342_object15645:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2c9c64a:::benchmark_data_smithi159_139342_object19487:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ca5b2a:::benchmark_data_smithi159_139342_object40576:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2cd2448:::benchmark_data_smithi159_139342_object27458:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2cdf016:::benchmark_data_smithi159_139342_object23117:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2cff720:::benchmark_data_smithi159_139342_object35193:head# 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2d1dc7d:::benchmark_data_smithi159_139342_object29098:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2d1e097:::benchmark_data_smithi159_139342_object40103:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2d343d4:::benchmark_data_smithi159_139342_object36167:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2d7ebf2:::benchmark_data_smithi159_139342_object16183:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2da0018:::benchmark_data_smithi159_139342_object1202:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2db4448:::benchmark_data_smithi159_139342_object28099:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2deb681:::benchmark_data_smithi159_139342_object37603:head# 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.295 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2df11db:::benchmark_data_smithi159_139342_object22676:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e075a8:::benchmark_data_smithi159_139342_object55216:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e1064d:::benchmark_data_smithi159_139342_object15140:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e2fc00:::benchmark_data_smithi159_139342_object40682:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e3b65b:::benchmark_data_smithi159_139342_object50578:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e3fa9a:::benchmark_data_smithi159_139342_object6398:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e4bcb6:::benchmark_data_smithi159_139342_object3751:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e54633:::benchmark_data_smithi159_139342_object29162:head# 2024-04-15T18:35:24.296 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e67652:::benchmark_data_smithi159_139342_object40508:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e6f077:::benchmark_data_smithi159_139342_object10215:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e72a2a:::benchmark_data_smithi159_139342_object30285:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e8e425:::benchmark_data_smithi159_139342_object44745:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2e8ea44:::benchmark_data_smithi159_139342_object50770:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ec3b8c:::benchmark_data_smithi159_139342_object14993:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ed3a0e:::benchmark_data_smithi159_139342_object12653:head# 2024-04-15T18:35:24.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ed7eb1:::benchmark_data_smithi159_139342_object41155:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2edcdbf:::benchmark_data_smithi159_139342_object13818:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2efe13f:::benchmark_data_smithi159_139342_object24658:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f07289:::benchmark_data_smithi159_139342_object41997:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f11a4c:::benchmark_data_smithi159_139342_object15383:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f2be76:::benchmark_data_smithi159_139342_object36187:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f39adb:::benchmark_data_smithi159_139342_object44808:head# 2024-04-15T18:35:24.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f3ad66:::benchmark_data_smithi159_139342_object36322:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f3c99b:::benchmark_data_smithi159_139342_object22185:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f4826d:::benchmark_data_smithi159_139342_object51157:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f72c3e:::benchmark_data_smithi159_139342_object26879:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2f77032:::benchmark_data_smithi159_139342_object49454:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2fbf04e:::benchmark_data_smithi159_139342_object30782:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2fdaf85:::benchmark_data_smithi159_139342_object53307:head# 2024-04-15T18:35:24.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ff2213:::benchmark_data_smithi159_139342_object21092:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c2ff8068:::benchmark_data_smithi159_139342_object38554:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3000ba1:::benchmark_data_smithi159_139342_object44586:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c300d6df:::benchmark_data_smithi159_139342_object21793:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c303f4b2:::benchmark_data_smithi159_139342_object55083:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30435b9:::benchmark_data_smithi159_139342_object26512:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3056f85:::benchmark_data_smithi159_139342_object37663:head# 2024-04-15T18:35:24.300 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c306ca04:::benchmark_data_smithi159_139342_object21663:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c307d7f8:::benchmark_data_smithi159_139342_object8008:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30ab2cf:::benchmark_data_smithi159_139342_object33379:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30ae57e:::benchmark_data_smithi159_139342_object47745:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30b22b8:::benchmark_data_smithi159_139342_object48743:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30c3a79:::benchmark_data_smithi159_139342_object56416:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30c6bd6:::benchmark_data_smithi159_139342_object32801:head# 2024-04-15T18:35:24.301 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30d2120:::benchmark_data_smithi159_139342_object29837:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30d3202:::benchmark_data_smithi159_139342_object23170:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30e0dd6:::benchmark_data_smithi159_139342_object20434:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30e1449:::benchmark_data_smithi159_139342_object23955:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c30f2bbe:::benchmark_data_smithi159_139342_object49444:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31286b6:::benchmark_data_smithi159_139342_object22172:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3143f7d:::benchmark_data_smithi159_139342_object42087:head# 2024-04-15T18:35:24.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c314bdff:::benchmark_data_smithi159_139342_object29626:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3154d77:::benchmark_data_smithi159_139342_object13990:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c315ca58:::benchmark_data_smithi159_139342_object12991:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3188011:::benchmark_data_smithi159_139342_object1426:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c318e558:::benchmark_data_smithi159_139342_object8168:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c318e5f0:::benchmark_data_smithi159_139342_object43404:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3190fa0:::benchmark_data_smithi159_139342_object6952:head# 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31bfa60:::benchmark_data_smithi159_139342_object4102:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31caf8f:::benchmark_data_smithi159_139342_object5016:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31ef486:::benchmark_data_smithi159_139342_object26425:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31f35b0:::benchmark_data_smithi159_139342_object56019:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c31fc9e1:::benchmark_data_smithi159_139342_object53004:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3202b28:::benchmark_data_smithi159_139342_object12783:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32253ce:::benchmark_data_smithi159_139342_object18635:head# 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32267dc:::benchmark_data_smithi159_139342_object37499:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c322dc6e:::benchmark_data_smithi159_139342_object38769:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c322e326:::benchmark_data_smithi159_139342_object33375:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c323b5d6:::benchmark_data_smithi159_139342_object24270:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32488a6:::benchmark_data_smithi159_139342_object32957:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c325be84:::benchmark_data_smithi159_139342_object24038:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c326847f:::benchmark_data_smithi159_139342_object23920:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c327038f:::benchmark_data_smithi159_139342_object29317:head# 2024-04-15T18:35:24.305 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3273f06:::benchmark_data_smithi159_139342_object28865:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32834c1:::benchmark_data_smithi159_139342_object9125:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3295b1b:::benchmark_data_smithi159_139342_object11102:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32bb890:::benchmark_data_smithi159_139342_object50215:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32c2407:::benchmark_data_smithi159_139342_object16959:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32d859a:::benchmark_data_smithi159_139342_object28936:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32d887b:::benchmark_data_smithi159_139342_object9911:head# 2024-04-15T18:35:24.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32e87a7:::benchmark_data_smithi159_139342_object23733:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32ed5e3:::benchmark_data_smithi159_139342_object12616:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32f2bd8:::benchmark_data_smithi159_139342_object21168:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32fcdc3:::benchmark_data_smithi159_139342_object20183:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32fd787:::benchmark_data_smithi159_139342_object2268:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c32ff6ad:::benchmark_data_smithi159_139342_object45323:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3300a1d:::benchmark_data_smithi159_139342_object47954:head# 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3319122:::benchmark_data_smithi159_139342_object32103:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c331c8f2:::benchmark_data_smithi159_139342_object42386:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c331faac:::benchmark_data_smithi159_139342_object25742:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c332a2db:::benchmark_data_smithi159_139342_object33947:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c337ccea:::benchmark_data_smithi159_139342_object28535:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3394e12:::benchmark_data_smithi159_139342_object47413:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c339abd8:::benchmark_data_smithi159_139342_object48383:head# 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c339fc5d:::benchmark_data_smithi159_139342_object48475:head# 2024-04-15T18:35:24.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33af8fa:::benchmark_data_smithi159_139342_object23629:head# 2024-04-15T18:35:24.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33b1758:::benchmark_data_smithi159_139342_object1479:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33b379b:::benchmark_data_smithi159_139342_object3625:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33bbd69:::benchmark_data_smithi159_139342_object17667:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33cdafe:::benchmark_data_smithi159_139342_object44892:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33d9440:::benchmark_data_smithi159_139342_object46576:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c33f2790:::benchmark_data_smithi159_139342_object56411:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34066c5:::benchmark_data_smithi159_139342_object1270:head# 2024-04-15T18:35:24.322 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3428489:::benchmark_data_smithi159_139342_object724:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c342927f:::benchmark_data_smithi159_139342_object19238:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c342a15e:::benchmark_data_smithi159_139342_object30662:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3436856:::benchmark_data_smithi159_139342_object36060:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c343d1d3:::benchmark_data_smithi159_139342_object47614:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c344a8be:::benchmark_data_smithi159_139342_object27371:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c346a9c4:::benchmark_data_smithi159_139342_object27029:head# 2024-04-15T18:35:24.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c346c5f0:::benchmark_data_smithi159_139342_object38688:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34709ed:::benchmark_data_smithi159_139342_object39067:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34a049c:::benchmark_data_smithi159_139342_object55512:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34a07bf:::benchmark_data_smithi159_139342_object13608:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34a14f6:::benchmark_data_smithi159_139342_object15620:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c34b08b4:::benchmark_data_smithi159_139342_object40114:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c350368b:::benchmark_data_smithi159_139342_object32945:head# 2024-04-15T18:35:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c352af83:::benchmark_data_smithi159_139342_object55934:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3532bb3:::benchmark_data_smithi159_139342_object3537:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3554d07:::benchmark_data_smithi159_139342_object17950:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c358a83c:::benchmark_data_smithi159_139342_object36889:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c35aedc1:::benchmark_data_smithi159_139342_object21305:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c35c73f0:::benchmark_data_smithi159_139342_object38857:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c35cb18e:::benchmark_data_smithi159_139342_object47453:head# 2024-04-15T18:35:24.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c35d8302:::benchmark_data_smithi159_139342_object23700:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c35fc98e:::benchmark_data_smithi159_139342_object6253:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3634f33:::benchmark_data_smithi159_139342_object3487:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c363b800:::benchmark_data_smithi159_139342_object49297:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3643526:::benchmark_data_smithi159_139342_object46073:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c364db4e:::benchmark_data_smithi159_139342_object6888:head# 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3669615:::benchmark_data_smithi159_139342_object40047:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3678a86:::benchmark_data_smithi159_139342_object50779:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c367cf65:::benchmark_data_smithi159_139342_object22244:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c367e128:::benchmark_data_smithi159_139342_object38851:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c36c1264:::benchmark_data_smithi159_139342_object46941:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c36d3b64:::benchmark_data_smithi159_139342_object21549:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c36e508f:::benchmark_data_smithi159_139342_object28915:head# 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3703c4f:::benchmark_data_smithi159_139342_object29689:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c370ed8b:::benchmark_data_smithi159_139342_object52291:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37126c8:::benchmark_data_smithi159_139342_object5698:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3716a94:::benchmark_data_smithi159_139342_object35518:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c371bef3:::benchmark_data_smithi159_139342_object52758:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37240f7:::benchmark_data_smithi159_139342_object13:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3727ce2:::benchmark_data_smithi159_139342_object20640:head# 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3728042:::benchmark_data_smithi159_139342_object13786:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c374bcbe:::benchmark_data_smithi159_139342_object53313:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c374cdbd:::benchmark_data_smithi159_139342_object12798:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3771992:::benchmark_data_smithi159_139342_object56513:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3783237:::benchmark_data_smithi159_139342_object31961:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37b1d76:::benchmark_data_smithi159_139342_object22785:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37d565c:::benchmark_data_smithi159_139342_object25202:head# 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37dcff8:::benchmark_data_smithi159_139342_object9052:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37f3fe6:::benchmark_data_smithi159_139342_object8403:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c37fe111:::benchmark_data_smithi159_139342_object43735:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3804548:::benchmark_data_smithi159_139342_object54646:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c380ed5a:::benchmark_data_smithi159_139342_object19671:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3818506:::benchmark_data_smithi159_139342_object48607:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38308ae:::benchmark_data_smithi159_139342_object43460:head# 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.330 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c384e738:::benchmark_data_smithi159_139342_object26307:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3865a9b:::benchmark_data_smithi159_139342_object4385:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3880cbe:::benchmark_data_smithi159_139342_object41566:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3883a8b:::benchmark_data_smithi159_139342_object42312:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c388a1d9:::benchmark_data_smithi159_139342_object6637:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38aeb00:::benchmark_data_smithi159_139342_object2237:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38b5b74:::benchmark_data_smithi159_139342_object20108:head# 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.331 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38cc117:::benchmark_data_smithi159_139342_object40052:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38dad8c:::benchmark_data_smithi159_139342_object21411:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38e43b8:::benchmark_data_smithi159_139342_object55652:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c38ffb5e:::benchmark_data_smithi159_139342_object9449:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c390da1c:::benchmark_data_smithi159_139342_object24003:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3910d88:::benchmark_data_smithi159_139342_object25456:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3915f1c:::benchmark_data_smithi159_139342_object23162:head# 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3928da9:::benchmark_data_smithi159_139342_object45790:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39333df:::benchmark_data_smithi159_139342_object1327:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c394797f:::benchmark_data_smithi159_139342_object53472:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c394e86d:::benchmark_data_smithi159_139342_object39402:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c395b0d8:::benchmark_data_smithi159_139342_object9942:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3989d51:::benchmark_data_smithi159_139342_object14923:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3996c2b:::benchmark_data_smithi159_139342_object40998:head# 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3999bbb:::benchmark_data_smithi159_139342_object52:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39b62bc:::benchmark_data_smithi159_139342_object2735:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39d66fa:::benchmark_data_smithi159_139342_object11738:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39e5cf4:::benchmark_data_smithi159_139342_object34673:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39f76a4:::benchmark_data_smithi159_139342_object36194:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39fd3db:::benchmark_data_smithi159_139342_object655:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c39fe099:::benchmark_data_smithi159_139342_object49676:head# 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.334 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a05462:::benchmark_data_smithi159_139342_object1895:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a5129b:::benchmark_data_smithi159_139342_object47339:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a6072c:::benchmark_data_smithi159_139342_object11951:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a6d169:::benchmark_data_smithi159_139342_object12284:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a704e7:::benchmark_data_smithi159_139342_object14758:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a76312:::benchmark_data_smithi159_139342_object41416:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3a9f73d:::benchmark_data_smithi159_139342_object2076:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3abd750:::benchmark_data_smithi159_139342_object10375:head# 2024-04-15T18:35:24.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ac6010:::benchmark_data_smithi159_139342_object42984:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3af7656:::benchmark_data_smithi159_139342_object19662:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3afd87b:::benchmark_data_smithi159_139342_object47724:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b286f3:::benchmark_data_smithi159_139342_object43672:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b355ac:::benchmark_data_smithi159_139342_object4581:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b4c417:::benchmark_data_smithi159_139342_object49101:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b59428:::benchmark_data_smithi159_139342_object46026:head# 2024-04-15T18:35:24.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b5e65c:::benchmark_data_smithi159_139342_object17194:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3b60655:::benchmark_data_smithi159_139342_object14018:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ba998a:::benchmark_data_smithi159_139342_object48364:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3bb9567:::benchmark_data_smithi159_139342_object16215:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3bbb001:::benchmark_data_smithi159_139342_object45641:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3bc774c:::benchmark_data_smithi159_139342_object55147:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3bc8722:::benchmark_data_smithi159_139342_object32397:head# 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.337 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3be2bd8:::benchmark_data_smithi159_139342_object5818:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3bf3592:::benchmark_data_smithi159_139342_object262:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c0288d:::benchmark_data_smithi159_139342_object21135:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c0acd6:::benchmark_data_smithi159_139342_object13594:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c1503a:::benchmark_data_smithi159_139342_object8480:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c45562:::benchmark_data_smithi159_139342_object47083:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c4e457:::benchmark_data_smithi159_139342_object40303:head# 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.338 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c4feb8:::benchmark_data_smithi159_139342_object245:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c58ea7:::benchmark_data_smithi159_139342_object27403:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3c5c65f:::benchmark_data_smithi159_139342_object9035:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3cb1b3f:::benchmark_data_smithi159_139342_object37452:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3cb7e0e:::benchmark_data_smithi159_139342_object49443:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3cc4bb9:::benchmark_data_smithi159_139342_object9777:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3cd58e8:::benchmark_data_smithi159_139342_object4346:head# 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.339 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3cfa824:::benchmark_data_smithi159_139342_object5115:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d0bebf:::benchmark_data_smithi159_139342_object34749:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d20b1b:::benchmark_data_smithi159_139342_object3326:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d20e56:::benchmark_data_smithi159_139342_object3201:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d25540:::benchmark_data_smithi159_139342_object29797:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d302f8:::benchmark_data_smithi159_139342_object12496:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d36634:::benchmark_data_smithi159_139342_object46144:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d42119:::benchmark_data_smithi159_139342_object4919:head# 2024-04-15T18:35:24.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d4c46a:::benchmark_data_smithi159_139342_object37780:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d6e23d:::benchmark_data_smithi159_139342_object8467:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d75892:::benchmark_data_smithi159_139342_object40748:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3d77c1f:::benchmark_data_smithi159_139342_object54900:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3da8e76:::benchmark_data_smithi159_139342_object34569:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3db7460:::benchmark_data_smithi159_139342_object27176:head# 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3dc4f4d:::benchmark_data_smithi159_139342_object8590:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3df476e:::benchmark_data_smithi159_139342_object53353:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3dfa743:::benchmark_data_smithi159_139342_object41071:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e0e64b:::benchmark_data_smithi159_139342_object15537:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e37bf0:::benchmark_data_smithi159_139342_object46511:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e423b5:::benchmark_data_smithi159_139342_object21419:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e4f025:::benchmark_data_smithi159_139342_object7493:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e72050:::benchmark_data_smithi159_139342_object716:head# 2024-04-15T18:35:24.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e77b95:::benchmark_data_smithi159_139342_object42335:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e7e717:::benchmark_data_smithi159_139342_object38324:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3e880b5:::benchmark_data_smithi159_139342_object7144:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ea858c:::benchmark_data_smithi159_139342_object37353:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ebaae7:::benchmark_data_smithi159_139342_object25699:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ebf40e:::benchmark_data_smithi159_139342_object55810:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ed7976:::benchmark_data_smithi159_139342_object38701:head# 2024-04-15T18:35:24.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f0e1e1:::benchmark_data_smithi159_139342_object27763:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f273f7:::benchmark_data_smithi159_139342_object39567:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f2ff94:::benchmark_data_smithi159_139342_object5552:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f3ec59:::benchmark_data_smithi159_139342_object42187:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f713c8:::benchmark_data_smithi159_139342_object27042:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f7c6e7:::benchmark_data_smithi159_139342_object8994:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3f984d5:::benchmark_data_smithi159_139342_object31411:head# 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3fa6343:::benchmark_data_smithi159_139342_object2816:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3fb6ce7:::benchmark_data_smithi159_139342_object17765:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3fbee9b:::benchmark_data_smithi159_139342_object15700:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ff1642:::benchmark_data_smithi159_139342_object13368:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c3ff3e6c:::benchmark_data_smithi159_139342_object51217:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c400da8f:::benchmark_data_smithi159_139342_object40157:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4016a5b:::benchmark_data_smithi159_139342_object15311:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c402939d:::benchmark_data_smithi159_139342_object36863:head# 2024-04-15T18:35:24.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4033b9a:::benchmark_data_smithi159_139342_object18558:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4041b8f:::benchmark_data_smithi159_139342_object46837:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c404c672:::benchmark_data_smithi159_139342_object9869:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4057dfd:::benchmark_data_smithi159_139342_object31799:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4065da0:::benchmark_data_smithi159_139342_object39628:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c406689f:::benchmark_data_smithi159_139342_object11936:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40692fc:::benchmark_data_smithi159_139342_object33762:head# 2024-04-15T18:35:24.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4069cc7:::benchmark_data_smithi159_139342_object39314:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c409c1aa:::benchmark_data_smithi159_139342_object35856:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40a06f8:::benchmark_data_smithi159_139342_object5312:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40ab640:::benchmark_data_smithi159_139342_object47934:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40c25da:::benchmark_data_smithi159_139342_object17304:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40e378d:::benchmark_data_smithi159_139342_object53847:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40e3b1a:::benchmark_data_smithi159_139342_object29022:head# 2024-04-15T18:35:24.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c40ef657:::benchmark_data_smithi159_139342_object239:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4101d5b:::benchmark_data_smithi159_139342_object21341:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c410cf4f:::benchmark_data_smithi159_139342_object48247:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c411c883:::benchmark_data_smithi159_139342_object7688:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c412a451:::benchmark_data_smithi159_139342_object36753:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c415269f:::benchmark_data_smithi159_139342_object19048:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c415801d:::benchmark_data_smithi159_139342_object53200:head# 2024-04-15T18:35:24.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c415f7e8:::benchmark_data_smithi159_139342_object42839:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4160a5d:::benchmark_data_smithi159_139342_object12157:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4171093:::benchmark_data_smithi159_139342_object37315:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c417a430:::benchmark_data_smithi159_139342_object57249:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4184d92:::benchmark_data_smithi159_139342_object49778:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4193274:::benchmark_data_smithi159_139342_object19136:head# 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.349 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c419ead1:::benchmark_data_smithi159_139342_object28979:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41a9d1e:::benchmark_data_smithi159_139342_object13766:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41abc85:::benchmark_data_smithi159_139342_object36200:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41abdfc:::benchmark_data_smithi159_139342_object49031:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41b2891:::benchmark_data_smithi159_139342_object18001:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41f0cea:::benchmark_data_smithi159_139342_object38207:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c41f3421:::benchmark_data_smithi159_139342_object47246:head# 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4204eef:::benchmark_data_smithi159_139342_object26804:head# 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c420b4a7:::benchmark_data_smithi159_139342_object25705:head# 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4217ae8:::benchmark_data_smithi159_139342_object21270:head# 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42350eb:::benchmark_data_smithi159_139342_object28962:head# 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c423ad9f:::benchmark_data_smithi159_139342_object49298:head# 2024-04-15T18:35:24.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c423c0dc:::benchmark_data_smithi159_139342_object32452:head# 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4241350:::benchmark_data_smithi159_139342_object45333:head# 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42421cc:::benchmark_data_smithi159_139342_object43180:head# 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42510ad:::benchmark_data_smithi159_139342_object40443:head# 2024-04-15T18:35:24.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c426be84:::benchmark_data_smithi159_139342_object39511:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c427bebd:::benchmark_data_smithi159_139342_object38810:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c428b9d4:::benchmark_data_smithi159_139342_object57184:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42aa98f:::benchmark_data_smithi159_139342_object43129:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42b5aea:::benchmark_data_smithi159_139342_object46300:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42d5c31:::benchmark_data_smithi159_139342_object14633:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c42f6586:::benchmark_data_smithi159_139342_object18662:head# 2024-04-15T18:35:24.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c43163f1:::benchmark_data_smithi159_139342_object32382:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4318d73:::benchmark_data_smithi159_139342_object8209:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4326e95:::benchmark_data_smithi159_139342_object27655:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4335af2:::benchmark_data_smithi159_139342_object12351:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4342e60:::benchmark_data_smithi159_139342_object34657:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4346e66:::benchmark_data_smithi159_139342_object44020:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c434d67b:::benchmark_data_smithi159_139342_object55746:head# 2024-04-15T18:35:24.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4355f70:::benchmark_data_smithi159_139342_object36207:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4368118:::benchmark_data_smithi159_139342_object39026:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c43730a9:::benchmark_data_smithi159_139342_object22638:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c437496f:::benchmark_data_smithi159_139342_object12361:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4388083:::benchmark_data_smithi159_139342_object35597:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c438b0f9:::benchmark_data_smithi159_139342_object21759:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c439539d:::benchmark_data_smithi159_139342_object46005:head# 2024-04-15T18:35:24.356 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c43b59a3:::benchmark_data_smithi159_139342_object10571:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c43bfc9c:::benchmark_data_smithi159_139342_object42493:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c442229d:::benchmark_data_smithi159_139342_object46655:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4426078:::benchmark_data_smithi159_139342_object18773:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44276f9:::benchmark_data_smithi159_139342_object42928:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44288be:::benchmark_data_smithi159_139342_object14406:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c443245a:::benchmark_data_smithi159_139342_object29499:head# 2024-04-15T18:35:24.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c443fa81:::benchmark_data_smithi159_139342_object47460:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4450fe6:::benchmark_data_smithi159_139342_object41267:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c446b834:::benchmark_data_smithi159_139342_object29554:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44825a4:::benchmark_data_smithi159_139342_object48965:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4490970:::benchmark_data_smithi159_139342_object6313:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44b43b1:::benchmark_data_smithi159_139342_object21109:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44bba94:::benchmark_data_smithi159_139342_object33352:head# 2024-04-15T18:35:24.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44c0dbc:::benchmark_data_smithi159_139342_object11620:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44c7162:::benchmark_data_smithi159_139342_object27280:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44d06b8:::benchmark_data_smithi159_139342_object44907:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c44fe3c5:::benchmark_data_smithi159_139342_object29894:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4508d91:::benchmark_data_smithi159_139342_object21955:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c451c1dd:::benchmark_data_smithi159_139342_object33892:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c452bcd5:::benchmark_data_smithi159_139342_object2896:head# 2024-04-15T18:35:24.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c45399d8:::benchmark_data_smithi159_139342_object19878:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4556543:::benchmark_data_smithi159_139342_object56421:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c455d012:::benchmark_data_smithi159_139342_object8843:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4579081:::benchmark_data_smithi159_139342_object34906:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c458ed85:::benchmark_data_smithi159_139342_object32503:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c45a3798:::benchmark_data_smithi159_139342_object35973:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c45b0c6a:::benchmark_data_smithi159_139342_object53274:head# 2024-04-15T18:35:24.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c45cb447:::benchmark_data_smithi159_139342_object55453:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c45e7fc9:::benchmark_data_smithi159_139342_object14002:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c460249d:::benchmark_data_smithi159_139342_object20532:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c460e7af:::benchmark_data_smithi159_139342_object12857:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c461348d:::benchmark_data_smithi159_139342_object2934:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4619e57:::benchmark_data_smithi159_139342_object14245:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c461c579:::benchmark_data_smithi159_139342_object33937:head# 2024-04-15T18:35:24.361 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c463333d:::benchmark_data_smithi159_139342_object41914:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4642eed:::benchmark_data_smithi159_139342_object2058:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4677b2c:::benchmark_data_smithi159_139342_object13361:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4688362:::benchmark_data_smithi159_139342_object11601:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46895d4:::benchmark_data_smithi159_139342_object38450:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c468b2db:::benchmark_data_smithi159_139342_object56910:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4692587:::benchmark_data_smithi159_139342_object11865:head# 2024-04-15T18:35:24.362 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46d51c1:::benchmark_data_smithi159_139342_object2801:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46d56c9:::benchmark_data_smithi159_139342_object53408:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46d765b:::benchmark_data_smithi159_139342_object43381:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46f28ec:::benchmark_data_smithi159_139342_object52936:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c46f8d36:::benchmark_data_smithi159_139342_object26445:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4701d4f:::benchmark_data_smithi159_139342_object54860:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4714d70:::benchmark_data_smithi159_139342_object20974:head# 2024-04-15T18:35:24.363 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c471fd9c:::benchmark_data_smithi159_139342_object28437:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4732944:::benchmark_data_smithi159_139342_object11500:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47329fa:::benchmark_data_smithi159_139342_object18298:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4740242:::benchmark_data_smithi159_139342_object13127:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47436de:::benchmark_data_smithi159_139342_object5141:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47454e0:::benchmark_data_smithi159_139342_object25335:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c475ff6c:::benchmark_data_smithi159_139342_object46743:head# 2024-04-15T18:35:24.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c476f0f6:::benchmark_data_smithi159_139342_object1508:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c478285d:::benchmark_data_smithi159_139342_object3199:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c478bfeb:::benchmark_data_smithi159_139342_object4253:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47920c7:::benchmark_data_smithi159_139342_object29742:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4792eb3:::benchmark_data_smithi159_139342_object21054:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47b258d:::benchmark_data_smithi159_139342_object30665:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47dda14:::benchmark_data_smithi159_139342_object13891:head# 2024-04-15T18:35:24.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47ddcf6:::benchmark_data_smithi159_139342_object51786:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47eafc8:::benchmark_data_smithi159_139342_object46721:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47ec696:::benchmark_data_smithi159_139342_object33409:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c47f38f0:::benchmark_data_smithi159_139342_object1915:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c480a717:::benchmark_data_smithi159_139342_object25048:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c480b3c3:::benchmark_data_smithi159_139342_object48300:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4839239:::benchmark_data_smithi159_139342_object27849:head# 2024-04-15T18:35:24.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c484f861:::benchmark_data_smithi159_139342_object19769:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4863bdd:::benchmark_data_smithi159_139342_object5117:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4867905:::benchmark_data_smithi159_139342_object40463:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4872f12:::benchmark_data_smithi159_139342_object48533:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4894a02:::benchmark_data_smithi159_139342_object21230:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c489919d:::benchmark_data_smithi159_139342_object5803:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c48ae89e:::benchmark_data_smithi159_139342_object39135:head# 2024-04-15T18:35:24.367 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c48cb081:::benchmark_data_smithi159_139342_object29358:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c48cce4d:::benchmark_data_smithi159_139342_object26528:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c48d5308:::benchmark_data_smithi159_139342_object54929:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4926277:::benchmark_data_smithi159_139342_object16114:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c496f7e9:::benchmark_data_smithi159_139342_object39324:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c498fecd:::benchmark_data_smithi159_139342_object27604:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c499d4cc:::benchmark_data_smithi159_139342_object51548:head# 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.368 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49a1007:::benchmark_data_smithi159_139342_object27566:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49be5c8:::benchmark_data_smithi159_139342_object51243:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49cb50d:::benchmark_data_smithi159_139342_object8063:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49cd3c6:::benchmark_data_smithi159_139342_object17709:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49cf3bc:::benchmark_data_smithi159_139342_object44668:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49d130c:::benchmark_data_smithi159_139342_object15173:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49d61eb:::benchmark_data_smithi159_139342_object5903:head# 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.369 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49dd84c:::benchmark_data_smithi159_139342_object21732:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49e6fe0:::benchmark_data_smithi159_139342_object23657:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49e8f15:::benchmark_data_smithi159_139342_object37154:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49f2c40:::benchmark_data_smithi159_139342_object40321:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c49f5500:::benchmark_data_smithi159_139342_object10627:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a07c61:::benchmark_data_smithi159_139342_object11789:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a186e8:::benchmark_data_smithi159_139342_object55178:head# 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a233da:::benchmark_data_smithi159_139342_object27879:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a278ed:::benchmark_data_smithi159_139342_object31936:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a3b5cf:::benchmark_data_smithi159_139342_object54858:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a4467f:::benchmark_data_smithi159_139342_object15953:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4a97685:::benchmark_data_smithi159_139342_object26501:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4aa9a5c:::benchmark_data_smithi159_139342_object44619:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4abacac:::benchmark_data_smithi159_139342_object21424:head# 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4abee4f:::benchmark_data_smithi159_139342_object38853:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4af4f17:::benchmark_data_smithi159_139342_object26246:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b0238c:::benchmark_data_smithi159_139342_object9062:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b086bf:::benchmark_data_smithi159_139342_object18096:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b16be8:::benchmark_data_smithi159_139342_object38481:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b1c4c0:::benchmark_data_smithi159_139342_object45692:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b2f236:::benchmark_data_smithi159_139342_object56555:head# 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b32ed9:::benchmark_data_smithi159_139342_object46358:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b35056:::benchmark_data_smithi159_139342_object26328:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b3b3bb:::benchmark_data_smithi159_139342_object33847:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b5577b:::benchmark_data_smithi159_139342_object16368:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b5b5a9:::benchmark_data_smithi159_139342_object21234:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4b87b7b:::benchmark_data_smithi159_139342_object18086:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4bcac45:::benchmark_data_smithi159_139342_object27906:head# 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.373 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c26e10:::benchmark_data_smithi159_139342_object45866:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c47137:::benchmark_data_smithi159_139342_object1689:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c5bb83:::benchmark_data_smithi159_139342_object44809:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c6c66e:::benchmark_data_smithi159_139342_object9465:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c75fb5:::benchmark_data_smithi159_139342_object19898:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4c9ab39:::benchmark_data_smithi159_139342_object34892:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4ca6a38:::benchmark_data_smithi159_139342_object34644:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4cb8a87:::benchmark_data_smithi159_139342_object11244:head# 2024-04-15T18:35:24.374 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4cc02bf:::benchmark_data_smithi159_139342_object24744:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4cca7b1:::benchmark_data_smithi159_139342_object27556:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4ccac09:::benchmark_data_smithi159_139342_object48930:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4cd354b:::benchmark_data_smithi159_139342_object20322:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4cea8a1:::benchmark_data_smithi159_139342_object28624:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4d05232:::benchmark_data_smithi159_139342_object50995:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4d08cf3:::benchmark_data_smithi159_139342_object47874:head# 2024-04-15T18:35:24.375 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4d42b0d:::benchmark_data_smithi159_139342_object36149:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4d4fceb:::benchmark_data_smithi159_139342_object24541:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4d5eddc:::benchmark_data_smithi159_139342_object37972:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4da1833:::benchmark_data_smithi159_139342_object19922:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dac947:::benchmark_data_smithi159_139342_object43394:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4db3776:::benchmark_data_smithi159_139342_object29052:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dc4cf2:::benchmark_data_smithi159_139342_object18081:head# 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.376 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dc8649:::benchmark_data_smithi159_139342_object20932:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dce5ac:::benchmark_data_smithi159_139342_object15239:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4de870c:::benchmark_data_smithi159_139342_object8207:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dfacfc:::benchmark_data_smithi159_139342_object41879:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4dfd46c:::benchmark_data_smithi159_139342_object4566:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e09f4f:::benchmark_data_smithi159_139342_object7894:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e2d664:::benchmark_data_smithi159_139342_object57076:head# 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.377 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e33278:::benchmark_data_smithi159_139342_object1691:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e66108:::benchmark_data_smithi159_139342_object42356:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e8e0ba:::benchmark_data_smithi159_139342_object30114:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4e98f69:::benchmark_data_smithi159_139342_object32862:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4eaaf62:::benchmark_data_smithi159_139342_object19394:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4eca703:::benchmark_data_smithi159_139342_object518:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4ed2f5b:::benchmark_data_smithi159_139342_object4455:head# 2024-04-15T18:35:24.378 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4f0cced:::benchmark_data_smithi159_139342_object49081:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4f24822:::benchmark_data_smithi159_139342_object28905:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4f414d3:::benchmark_data_smithi159_139342_object2389:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4f65097:::benchmark_data_smithi159_139342_object2454:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fa8afa:::benchmark_data_smithi159_139342_object17224:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fa91d1:::benchmark_data_smithi159_139342_object10040:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fc3bc5:::benchmark_data_smithi159_139342_object41849:head# 2024-04-15T18:35:24.379 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fd1477:::benchmark_data_smithi159_139342_object20555:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fe7a69:::benchmark_data_smithi159_139342_object32270:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4fed02d:::benchmark_data_smithi159_139342_object19805:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4ff62ea:::benchmark_data_smithi159_139342_object13411:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c4ff80ee:::benchmark_data_smithi159_139342_object53418:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c501511f:::benchmark_data_smithi159_139342_object31430:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c502722f:::benchmark_data_smithi159_139342_object35021:head# 2024-04-15T18:35:24.380 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50452aa:::benchmark_data_smithi159_139342_object23934:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c506e422:::benchmark_data_smithi159_139342_object22483:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50a25c5:::benchmark_data_smithi159_139342_object36216:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50c166e:::benchmark_data_smithi159_139342_object26687:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50c47e7:::benchmark_data_smithi159_139342_object30952:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50c546a:::benchmark_data_smithi159_139342_object8214:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50ca6c2:::benchmark_data_smithi159_139342_object24456:head# 2024-04-15T18:35:24.381 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50d780f:::benchmark_data_smithi159_139342_object23450:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50e3e4e:::benchmark_data_smithi159_139342_object43560:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50ea5af:::benchmark_data_smithi159_139342_object38217:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50f3ed1:::benchmark_data_smithi159_139342_object14236:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50f52b2:::benchmark_data_smithi159_139342_object28659:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c50f86f1:::benchmark_data_smithi159_139342_object39302:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5100c28:::benchmark_data_smithi159_139342_object28834:head# 2024-04-15T18:35:24.382 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5123606:::benchmark_data_smithi159_139342_object22636:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c512a431:::benchmark_data_smithi159_139342_object18384:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5140563:::benchmark_data_smithi159_139342_object26256:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5143a47:::benchmark_data_smithi159_139342_object7575:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5148193:::benchmark_data_smithi159_139342_object3329:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c514ed52:::benchmark_data_smithi159_139342_object19107:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c516f9d0:::benchmark_data_smithi159_139342_object44520:head# 2024-04-15T18:35:24.383 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c518ac7e:::benchmark_data_smithi159_139342_object27686:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c518d0f1:::benchmark_data_smithi159_139342_object7723:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51c5482:::benchmark_data_smithi159_139342_object25546:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51d2967:::benchmark_data_smithi159_139342_object26413:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51d8738:::benchmark_data_smithi159_139342_object5888:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51e24c4:::benchmark_data_smithi159_139342_object51306:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51e2831:::benchmark_data_smithi159_139342_object465:head# 2024-04-15T18:35:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c51e70dd:::benchmark_data_smithi159_139342_object32040:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c521d86d:::benchmark_data_smithi159_139342_object30652:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c522c0ed:::benchmark_data_smithi159_139342_object53153:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c525dd83:::benchmark_data_smithi159_139342_object7771:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5268e8a:::benchmark_data_smithi159_139342_object5381:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5279692:::benchmark_data_smithi159_139342_object3296:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c527b357:::benchmark_data_smithi159_139342_object56909:head# 2024-04-15T18:35:24.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c527cc99:::benchmark_data_smithi159_139342_object31659:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c52ac270:::benchmark_data_smithi159_139342_object15820:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c52b061e:::benchmark_data_smithi159_139342_object12294:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c52c4389:::benchmark_data_smithi159_139342_object52814:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c52cd7cf:::benchmark_data_smithi159_139342_object55407:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c52eb757:::benchmark_data_smithi159_139342_object43719:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c531efe2:::benchmark_data_smithi159_139342_object44202:head# 2024-04-15T18:35:24.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5325a15:::benchmark_data_smithi159_139342_object18526:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53413cc:::benchmark_data_smithi159_139342_object39768:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53725f6:::benchmark_data_smithi159_139342_object4076:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53730f4:::benchmark_data_smithi159_139342_object17769:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5376123:::benchmark_data_smithi159_139342_object10986:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5388089:::benchmark_data_smithi159_139342_object18325:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c539bfcf:::benchmark_data_smithi159_139342_object1150:head# 2024-04-15T18:35:24.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53a51cb:::benchmark_data_smithi159_139342_object41769:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53a72bd:::benchmark_data_smithi159_139342_object19931:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53bdcfe:::benchmark_data_smithi159_139342_object7142:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53ecab9:::benchmark_data_smithi159_139342_object47523:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c53f6c28:::benchmark_data_smithi159_139342_object35256:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54076d9:::benchmark_data_smithi159_139342_object47054:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54378e2:::benchmark_data_smithi159_139342_object27574:head# 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c545ba4e:::benchmark_data_smithi159_139342_object35769:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c546387a:::benchmark_data_smithi159_139342_object41092:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54815a0:::benchmark_data_smithi159_139342_object11913:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54a6039:::benchmark_data_smithi159_139342_object7538:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54bfd0f:::benchmark_data_smithi159_139342_object22408:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54c5ca7:::benchmark_data_smithi159_139342_object44543:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54d4bb0:::benchmark_data_smithi159_139342_object955:head# 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.389 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54d9f4d:::benchmark_data_smithi159_139342_object47537:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54f3374:::benchmark_data_smithi159_139342_object54280:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54f3cbc:::benchmark_data_smithi159_139342_object25520:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c54fba45:::benchmark_data_smithi159_139342_object54976:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5501610:::benchmark_data_smithi159_139342_object21939:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c553bb39:::benchmark_data_smithi159_139342_object27826:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5540abb:::benchmark_data_smithi159_139342_object5931:head# 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c554996d:::benchmark_data_smithi159_139342_object6293:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c555a8c5:::benchmark_data_smithi159_139342_object7934:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c555b762:::benchmark_data_smithi159_139342_object40173:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c55946f7:::benchmark_data_smithi159_139342_object47806:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c559ed56:::benchmark_data_smithi159_139342_object39568:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c55b4ab2:::benchmark_data_smithi159_139342_object5139:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c55b9320:::benchmark_data_smithi159_139342_object13402:head# 2024-04-15T18:35:24.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c55f27b6:::benchmark_data_smithi159_139342_object20946:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c560e7b6:::benchmark_data_smithi159_139342_object27612:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5642125:::benchmark_data_smithi159_139342_object8473:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c565f679:::benchmark_data_smithi159_139342_object34289:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56608db:::benchmark_data_smithi159_139342_object31606:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5660c94:::benchmark_data_smithi159_139342_object16598:head# 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c567b3e1:::benchmark_data_smithi159_139342_object33470:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c567eda9:::benchmark_data_smithi159_139342_object20841:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56835c7:::benchmark_data_smithi159_139342_object41341:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c568420a:::benchmark_data_smithi159_139342_object36888:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56982c9:::benchmark_data_smithi159_139342_object53226:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c569d369:::benchmark_data_smithi159_139342_object1789:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56a25d2:::benchmark_data_smithi159_139342_object51518:head# 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.393 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56b00c0:::benchmark_data_smithi159_139342_object44782:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56b600c:::benchmark_data_smithi159_139342_object29714:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56b9858:::benchmark_data_smithi159_139342_object14619:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56c8051:::benchmark_data_smithi159_139342_object42120:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56c9d2d:::benchmark_data_smithi159_139342_object53395:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56d3d4a:::benchmark_data_smithi159_139342_object38049:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56e2169:::benchmark_data_smithi159_139342_object23571:head# 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.394 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56eaf65:::benchmark_data_smithi159_139342_object36405:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c56f7b7a:::benchmark_data_smithi159_139342_object1839:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5709423:::benchmark_data_smithi159_139342_object40100:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c570a212:::benchmark_data_smithi159_139342_object8803:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c571ee60:::benchmark_data_smithi159_139342_object29220:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5738446:::benchmark_data_smithi159_139342_object40894:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c573f447:::benchmark_data_smithi159_139342_object42144:head# 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.395 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5741c7c:::benchmark_data_smithi159_139342_object16520:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5754960:::benchmark_data_smithi159_139342_object10264:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c57550ad:::benchmark_data_smithi159_139342_object47742:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c575d5d1:::benchmark_data_smithi159_139342_object57247:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c57a827b:::benchmark_data_smithi159_139342_object8249:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c57c0b6a:::benchmark_data_smithi159_139342_object32570:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c57e4a9a:::benchmark_data_smithi159_139342_object50374:head# 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.396 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c57f9b46:::benchmark_data_smithi159_139342_object54592:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c580cfd3:::benchmark_data_smithi159_139342_object8570:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5816aef:::benchmark_data_smithi159_139342_object45349:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c582efbf:::benchmark_data_smithi159_139342_object32893:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c583fddd:::benchmark_data_smithi159_139342_object21700:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c585f6c2:::benchmark_data_smithi159_139342_object44872:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5874160:::benchmark_data_smithi159_139342_object27497:head# 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.397 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c587b34a:::benchmark_data_smithi159_139342_object10531:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58a89a6:::benchmark_data_smithi159_139342_object13753:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58b746e:::benchmark_data_smithi159_139342_object30957:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58c7af8:::benchmark_data_smithi159_139342_object50218:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58d3825:::benchmark_data_smithi159_139342_object45037:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58d80c9:::benchmark_data_smithi159_139342_object22322:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58dcf1e:::benchmark_data_smithi159_139342_object29177:head# 2024-04-15T18:35:24.398 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58ea049:::benchmark_data_smithi159_139342_object12244:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c58fc3d6:::benchmark_data_smithi159_139342_object7346:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59224aa:::benchmark_data_smithi159_139342_object51132:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5925bc7:::benchmark_data_smithi159_139342_object11843:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59566fe:::benchmark_data_smithi159_139342_object23568:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59613c6:::benchmark_data_smithi159_139342_object15478:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5988821:::benchmark_data_smithi159_139342_object22309:head# 2024-04-15T18:35:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59932e9:::benchmark_data_smithi159_139342_object23488:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5993b5b:::benchmark_data_smithi159_139342_object9129:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c599a358:::benchmark_data_smithi159_139342_object54062:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59a029c:::benchmark_data_smithi159_139342_object39459:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59a84ae:::benchmark_data_smithi159_139342_object32456:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59bcc28:::benchmark_data_smithi159_139342_object29665:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59c2733:::benchmark_data_smithi159_139342_object5163:head# 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59c546e:::benchmark_data_smithi159_139342_object18148:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59cdfdb:::benchmark_data_smithi159_139342_object20675:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59e5b65:::benchmark_data_smithi159_139342_object16468:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59ee26a:::benchmark_data_smithi159_139342_object18446:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c59f93b9:::benchmark_data_smithi159_139342_object36156:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a40ecb:::benchmark_data_smithi159_139342_object49986:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a60f6f:::benchmark_data_smithi159_139342_object45561:head# 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.401 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a6e691:::benchmark_data_smithi159_139342_object1151:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a6eb2b:::benchmark_data_smithi159_139342_object40646:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a7c1fa:::benchmark_data_smithi159_139342_object49521:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a89829:::benchmark_data_smithi159_139342_object1410:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a945e1:::benchmark_data_smithi159_139342_object42998:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a94fa8:::benchmark_data_smithi159_139342_object38691:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5a9fda7:::benchmark_data_smithi159_139342_object45100:head# 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.402 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5aa894a:::benchmark_data_smithi159_139342_object49204:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ac4e82:::benchmark_data_smithi159_139342_object51172:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ad1363:::benchmark_data_smithi159_139342_object23363:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5adf8c5:::benchmark_data_smithi159_139342_object37995:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ae25b9:::benchmark_data_smithi159_139342_object38879:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ae7494:::benchmark_data_smithi159_139342_object45044:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5aeacf5:::benchmark_data_smithi159_139342_object23651:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5af1ec7:::benchmark_data_smithi159_139342_object49784:head# 2024-04-15T18:35:24.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b07ade:::benchmark_data_smithi159_139342_object27783:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b25a89:::benchmark_data_smithi159_139342_object41583:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b31e2c:::benchmark_data_smithi159_139342_object47085:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b5cb01:::benchmark_data_smithi159_139342_object16538:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b6892f:::benchmark_data_smithi159_139342_object6267:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b7bec7:::benchmark_data_smithi159_139342_object39679:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b807de:::benchmark_data_smithi159_139342_object654:head# 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b81cc4:::benchmark_data_smithi159_139342_object2002:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5b8af1b:::benchmark_data_smithi159_139342_object40924:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ba1ce3:::benchmark_data_smithi159_139342_object21665:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ba4bc8:::benchmark_data_smithi159_139342_object8205:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ba8e74:::benchmark_data_smithi159_139342_object22895:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5bab319:::benchmark_data_smithi159_139342_object53477:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5bde023:::benchmark_data_smithi159_139342_object24174:head# 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5bee4fd:::benchmark_data_smithi159_139342_object41410:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5bf2ea0:::benchmark_data_smithi159_139342_object579:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c00398:::benchmark_data_smithi159_139342_object51497:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c046f3:::benchmark_data_smithi159_139342_object12109:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c0962a:::benchmark_data_smithi159_139342_object31338:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c1554f:::benchmark_data_smithi159_139342_object50094:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c22be1:::benchmark_data_smithi159_139342_object51521:head# 2024-04-15T18:35:24.406 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c2b258:::benchmark_data_smithi159_139342_object32768:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c2ea5f:::benchmark_data_smithi159_139342_object24347:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c31902:::benchmark_data_smithi159_139342_object6210:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c41423:::benchmark_data_smithi159_139342_object21318:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c7b9a9:::benchmark_data_smithi159_139342_object52502:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5c868ce:::benchmark_data_smithi159_139342_object35488:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5cea047:::benchmark_data_smithi159_139342_object10549:head# 2024-04-15T18:35:24.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5cf62b2:::benchmark_data_smithi159_139342_object4328:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5cf88bd:::benchmark_data_smithi159_139342_object46450:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d1b7a3:::benchmark_data_smithi159_139342_object54767:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d1c021:::benchmark_data_smithi159_139342_object11516:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d3b0e0:::benchmark_data_smithi159_139342_object51665:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d3dec8:::benchmark_data_smithi159_139342_object13712:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d5a74a:::benchmark_data_smithi159_139342_object42384:head# 2024-04-15T18:35:24.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d5e4bc:::benchmark_data_smithi159_139342_object46228:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d648c4:::benchmark_data_smithi159_139342_object37721:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d734f0:::benchmark_data_smithi159_139342_object11816:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d7d6cb:::benchmark_data_smithi159_139342_object55085:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d84760:::benchmark_data_smithi159_139342_object26588:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d89969:::benchmark_data_smithi159_139342_object25597:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5d8fde3:::benchmark_data_smithi159_139342_object30908:head# 2024-04-15T18:35:24.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dac53b:::benchmark_data_smithi159_139342_object48848:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5db0c14:::benchmark_data_smithi159_139342_object19059:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dbd6d6:::benchmark_data_smithi159_139342_object38002:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dc84eb:::benchmark_data_smithi159_139342_object8404:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dccac4:::benchmark_data_smithi159_139342_object28628:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dd1ee3:::benchmark_data_smithi159_139342_object24088:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5dd579e:::benchmark_data_smithi159_139342_object24558:head# 2024-04-15T18:35:24.410 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5df7b94:::benchmark_data_smithi159_139342_object44956:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5df9f4c:::benchmark_data_smithi159_139342_object51037:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e23d10:::benchmark_data_smithi159_139342_object21837:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e2bbde:::benchmark_data_smithi159_139342_object53234:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e47ada:::benchmark_data_smithi159_139342_object37287:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e5e35c:::benchmark_data_smithi159_139342_object7922:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e74bd8:::benchmark_data_smithi159_139342_object9976:head# 2024-04-15T18:35:24.411 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e8d58f:::benchmark_data_smithi159_139342_object46064:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5e93dcd:::benchmark_data_smithi159_139342_object1277:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5eb3346:::benchmark_data_smithi159_139342_object38620:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5eb6a2f:::benchmark_data_smithi159_139342_object4004:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ec260a:::benchmark_data_smithi159_139342_object44373:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ecc231:::benchmark_data_smithi159_139342_object49359:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5eccaba:::benchmark_data_smithi159_139342_object4491:head# 2024-04-15T18:35:24.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ed5f5f:::benchmark_data_smithi159_139342_object38659:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5edfa6c:::benchmark_data_smithi159_139342_object49303:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f0a8d5:::benchmark_data_smithi159_139342_object11057:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f0be24:::benchmark_data_smithi159_139342_object10656:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f249c2:::benchmark_data_smithi159_139342_object33530:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f3889b:::benchmark_data_smithi159_139342_object3941:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f3ae74:::benchmark_data_smithi159_139342_object48615:head# 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f3cf91:::benchmark_data_smithi159_139342_object49580:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f4116b:::benchmark_data_smithi159_139342_object18208:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f56332:::benchmark_data_smithi159_139342_object4087:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f7898e:::benchmark_data_smithi159_139342_object54745:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5f8e162:::benchmark_data_smithi159_139342_object21545:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fad283:::benchmark_data_smithi159_139342_object41929:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fb1521:::benchmark_data_smithi159_139342_object14021:head# 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fc4268:::benchmark_data_smithi159_139342_object19297:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fc884e:::benchmark_data_smithi159_139342_object18882:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fcb3db:::benchmark_data_smithi159_139342_object45016:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5fdbd01:::benchmark_data_smithi159_139342_object34735:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c5ffe5ab:::benchmark_data_smithi159_139342_object26803:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c602e6ce:::benchmark_data_smithi159_139342_object3009:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c602e8db:::benchmark_data_smithi159_139342_object17954:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c60301af:::benchmark_data_smithi159_139342_object43378:head# 2024-04-15T18:35:24.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c604cdf5:::benchmark_data_smithi159_139342_object28015:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c605c1bf:::benchmark_data_smithi159_139342_object38955:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c60615c9:::benchmark_data_smithi159_139342_object32735:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6065138:::benchmark_data_smithi159_139342_object35765:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c60eb7b1:::benchmark_data_smithi159_139342_object25825:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c60f1c64:::benchmark_data_smithi159_139342_object31935:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c60fa36e:::benchmark_data_smithi159_139342_object7653:head# 2024-04-15T18:35:24.416 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c611738c:::benchmark_data_smithi159_139342_object25196:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c612664d:::benchmark_data_smithi159_139342_object41580:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c612bcee:::benchmark_data_smithi159_139342_object25340:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6143f59:::benchmark_data_smithi159_139342_object38593:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c615b1c7:::benchmark_data_smithi159_139342_object48447:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61a8d18:::benchmark_data_smithi159_139342_object6008:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61bcfcc:::benchmark_data_smithi159_139342_object22805:head# 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.417 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61c0ab0:::benchmark_data_smithi159_139342_object13971:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61c5806:::benchmark_data_smithi159_139342_object6560:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61cd590:::benchmark_data_smithi159_139342_object158:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61d17c5:::benchmark_data_smithi159_139342_object18994:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61d2432:::benchmark_data_smithi159_139342_object52790:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61eb2f8:::benchmark_data_smithi159_139342_object23783:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61f5cab:::benchmark_data_smithi159_139342_object20062:head# 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c61f63ed:::benchmark_data_smithi159_139342_object36295:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6202fef:::benchmark_data_smithi159_139342_object49728:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6224e9a:::benchmark_data_smithi159_139342_object18231:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c626d43e:::benchmark_data_smithi159_139342_object43871:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6279ba9:::benchmark_data_smithi159_139342_object25234:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62b1e79:::benchmark_data_smithi159_139342_object6642:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62c252a:::benchmark_data_smithi159_139342_object702:head# 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62c728f:::benchmark_data_smithi159_139342_object1772:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62cebea:::benchmark_data_smithi159_139342_object6672:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62d0de7:::benchmark_data_smithi159_139342_object25571:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62d70f6:::benchmark_data_smithi159_139342_object44454:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62e919d:::benchmark_data_smithi159_139342_object12376:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c62ee7e3:::benchmark_data_smithi159_139342_object32410:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c631bfa5:::benchmark_data_smithi159_139342_object41696:head# 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.420 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6322b54:::benchmark_data_smithi159_139342_object36491:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63280a3:::benchmark_data_smithi159_139342_object47418:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6330acf:::benchmark_data_smithi159_139342_object47115:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6340590:::benchmark_data_smithi159_139342_object44739:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c634399d:::benchmark_data_smithi159_139342_object51475:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63563db:::benchmark_data_smithi159_139342_object16441:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c635c4b7:::benchmark_data_smithi159_139342_object35976:head# 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.421 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c638605b:::benchmark_data_smithi159_139342_object41628:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63bf5fc:::benchmark_data_smithi159_139342_object41240:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63c25ff:::benchmark_data_smithi159_139342_object17146:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63d51bf:::benchmark_data_smithi159_139342_object40872:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63f1b5d:::benchmark_data_smithi159_139342_object32165:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63fac99:::benchmark_data_smithi159_139342_object3882:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c63fff95:::benchmark_data_smithi159_139342_object249:head# 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.422 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c641f313:::benchmark_data_smithi159_139342_object56638:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6442e59:::benchmark_data_smithi159_139342_object44179:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c646df40:::benchmark_data_smithi159_139342_object25179:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c647acf3:::benchmark_data_smithi159_139342_object43878:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c647ca5f:::benchmark_data_smithi159_139342_object2625:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c64b47f4:::benchmark_data_smithi159_139342_object53772:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c64d1813:::benchmark_data_smithi159_139342_object34904:head# 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c64fdde6:::benchmark_data_smithi159_139342_object27886:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c652fda8:::benchmark_data_smithi159_139342_object26116:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6580524:::benchmark_data_smithi159_139342_object49342:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c658d747:::benchmark_data_smithi159_139342_object41481:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6597a6f:::benchmark_data_smithi159_139342_object23998:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65d7d1b:::benchmark_data_smithi159_139342_object6728:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65e6b76:::benchmark_data_smithi159_139342_object31199:head# 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65e9166:::benchmark_data_smithi159_139342_object24150:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65edcb1:::benchmark_data_smithi159_139342_object52591:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65effe1:::benchmark_data_smithi159_139342_object31326:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c65f556d:::benchmark_data_smithi159_139342_object269:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c660af90:::benchmark_data_smithi159_139342_object20619:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6614184:::benchmark_data_smithi159_139342_object11574:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c661de58:::benchmark_data_smithi159_139342_object43229:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6632fd0:::benchmark_data_smithi159_139342_object46578:head# 2024-04-15T18:35:24.425 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6637d43:::benchmark_data_smithi159_139342_object49467:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c663c122:::benchmark_data_smithi159_139342_object9005:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c665b0d8:::benchmark_data_smithi159_139342_object33293:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6674cb7:::benchmark_data_smithi159_139342_object12593:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66b0126:::benchmark_data_smithi159_139342_object38964:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66cebcd:::benchmark_data_smithi159_139342_object25028:head# 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66e3a8e:::benchmark_data_smithi159_139342_object56084:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66eabfc:::benchmark_data_smithi159_139342_object36342:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66f23f5:::benchmark_data_smithi159_139342_object44998:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c66fb2ac:::benchmark_data_smithi159_139342_object13890:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6701e4f:::benchmark_data_smithi159_139342_object26226:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c671d871:::benchmark_data_smithi159_139342_object11817:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6726d91:::benchmark_data_smithi159_139342_object43352:head# 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.427 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6727cf2:::benchmark_data_smithi159_139342_object10542:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6734837:::benchmark_data_smithi159_139342_object48550:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67410fc:::benchmark_data_smithi159_139342_object37415:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6743689:::benchmark_data_smithi159_139342_object21603:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c676f14b:::benchmark_data_smithi159_139342_object34690:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67801f7:::benchmark_data_smithi159_139342_object34348:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67806d4:::benchmark_data_smithi159_139342_object21177:head# 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.428 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c679504f:::benchmark_data_smithi159_139342_object20738:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67b8d99:::benchmark_data_smithi159_139342_object53436:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67bca19:::benchmark_data_smithi159_139342_object14337:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67c280c:::benchmark_data_smithi159_139342_object52152:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67cafd6:::benchmark_data_smithi159_139342_object3375:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67cdde5:::benchmark_data_smithi159_139342_object23083:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67d13f8:::benchmark_data_smithi159_139342_object19204:head# 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.429 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67d3195:::benchmark_data_smithi159_139342_object25708:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67e0dc9:::benchmark_data_smithi159_139342_object50638:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67e4c1f:::benchmark_data_smithi159_139342_object2692:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67f6f51:::benchmark_data_smithi159_139342_object14933:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c67fb561:::benchmark_data_smithi159_139342_object8015:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6849831:::benchmark_data_smithi159_139342_object14720:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68533f2:::benchmark_data_smithi159_139342_object29254:head# 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.430 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68639eb:::benchmark_data_smithi159_139342_object15571:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6868f54:::benchmark_data_smithi159_139342_object30689:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c688124a:::benchmark_data_smithi159_139342_object30417:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c688fae4:::benchmark_data_smithi159_139342_object310:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68bfb04:::benchmark_data_smithi159_139342_object41924:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68de132:::benchmark_data_smithi159_139342_object9534:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68e3875:::benchmark_data_smithi159_139342_object26115:head# 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.431 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c68ec22d:::benchmark_data_smithi159_139342_object34910:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c690fe4d:::benchmark_data_smithi159_139342_object29229:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c691eb7d:::benchmark_data_smithi159_139342_object52496:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c692ff4c:::benchmark_data_smithi159_139342_object50838:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c696911c:::benchmark_data_smithi159_139342_object28481:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c697978c:::benchmark_data_smithi159_139342_object5442:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6979b5d:::benchmark_data_smithi159_139342_object14120:head# 2024-04-15T18:35:24.432 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c699c73f:::benchmark_data_smithi159_139342_object24379:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c69d1226:::benchmark_data_smithi159_139342_object31650:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c69f8b7b:::benchmark_data_smithi159_139342_object32478:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c69f9d84:::benchmark_data_smithi159_139342_object204:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a068ac:::benchmark_data_smithi159_139342_object12440:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a071f3:::benchmark_data_smithi159_139342_object9049:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a1073d:::benchmark_data_smithi159_139342_object32019:head# 2024-04-15T18:35:24.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a11261:::benchmark_data_smithi159_139342_object49831:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a13691:::benchmark_data_smithi159_139342_object25554:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a191db:::benchmark_data_smithi159_139342_object33918:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a24096:::benchmark_data_smithi159_139342_object47000:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a28bcc:::benchmark_data_smithi159_139342_object3111:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a3758e:::benchmark_data_smithi159_139342_object30087:head# 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a61e75:::benchmark_data_smithi159_139342_object55633:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a6a970:::benchmark_data_smithi159_139342_object42765:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6a8b1ce:::benchmark_data_smithi159_139342_object13285:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6aaffde:::benchmark_data_smithi159_139342_object15129:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ac3928:::benchmark_data_smithi159_139342_object56460:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ad119e:::benchmark_data_smithi159_139342_object338:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ae13a2:::benchmark_data_smithi159_139342_object26686:head# 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6af51b9:::benchmark_data_smithi159_139342_object39398:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b0703f:::benchmark_data_smithi159_139342_object42403:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b1da45:::benchmark_data_smithi159_139342_object36547:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b320f6:::benchmark_data_smithi159_139342_object4904:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b38806:::benchmark_data_smithi159_139342_object3238:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b3a14e:::benchmark_data_smithi159_139342_object18478:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b3d5da:::benchmark_data_smithi159_139342_object45241:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b49b61:::benchmark_data_smithi159_139342_object48900:head# 2024-04-15T18:35:24.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b6dc17:::benchmark_data_smithi159_139342_object39085:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b6ed4d:::benchmark_data_smithi159_139342_object10440:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6b989c2:::benchmark_data_smithi159_139342_object19192:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ba2bee:::benchmark_data_smithi159_139342_object48422:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6bae41c:::benchmark_data_smithi159_139342_object17477:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6bcab17:::benchmark_data_smithi159_139342_object7:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6bda468:::benchmark_data_smithi159_139342_object22312:head# 2024-04-15T18:35:24.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c06ab5:::benchmark_data_smithi159_139342_object47816:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c13c49:::benchmark_data_smithi159_139342_object37555:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c264a5:::benchmark_data_smithi159_139342_object12102:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c2f490:::benchmark_data_smithi159_139342_object19578:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c4adfe:::benchmark_data_smithi159_139342_object45451:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c808fe:::benchmark_data_smithi159_139342_object6696:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c863c4:::benchmark_data_smithi159_139342_object36951:head# 2024-04-15T18:35:24.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6c898df:::benchmark_data_smithi159_139342_object5639:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ca6d8a:::benchmark_data_smithi159_139342_object38774:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6cb718f:::benchmark_data_smithi159_139342_object38463:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6cbf5b2:::benchmark_data_smithi159_139342_object36466:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6cd31ce:::benchmark_data_smithi159_139342_object42179:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ceaf2b:::benchmark_data_smithi159_139342_object37857:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d0c6de:::benchmark_data_smithi159_139342_object5836:head# 2024-04-15T18:35:24.439 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d1fd97:::benchmark_data_smithi159_139342_object28743:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d433c2:::benchmark_data_smithi159_139342_object13844:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d61b1b:::benchmark_data_smithi159_139342_object48857:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d8381c:::benchmark_data_smithi159_139342_object41504:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d85aea:::benchmark_data_smithi159_139342_object56956:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6d99dd9:::benchmark_data_smithi159_139342_object10989:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6daf3f0:::benchmark_data_smithi159_139342_object4460:head# 2024-04-15T18:35:24.440 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e09df6:::benchmark_data_smithi159_139342_object18975:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e0f46a:::benchmark_data_smithi159_139342_object47553:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e15a69:::benchmark_data_smithi159_139342_object52248:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e268a8:::benchmark_data_smithi159_139342_object47452:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e3112f:::benchmark_data_smithi159_139342_object12672:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e31627:::benchmark_data_smithi159_139342_object4116:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e48013:::benchmark_data_smithi159_139342_object13934:head# 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.441 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e48a21:::benchmark_data_smithi159_139342_object44086:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e534b2:::benchmark_data_smithi159_139342_object38305:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e5b0a0:::benchmark_data_smithi159_139342_object578:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e611de:::benchmark_data_smithi159_139342_object19586:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6e89ab7:::benchmark_data_smithi159_139342_object15006:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ebc167:::benchmark_data_smithi159_139342_object17621:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ed4b11:::benchmark_data_smithi159_139342_object10606:head# 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.442 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6eeb167:::benchmark_data_smithi159_139342_object18199:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6ef325e:::benchmark_data_smithi159_139342_object45559:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f1af9d:::benchmark_data_smithi159_139342_object43370:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f28f34:::benchmark_data_smithi159_139342_object42248:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f316e0:::benchmark_data_smithi159_139342_object12056:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f41ef2:::benchmark_data_smithi159_139342_object43782:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f50810:::benchmark_data_smithi159_139342_object966:head# 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f5f86c:::benchmark_data_smithi159_139342_object28548:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f63ab9:::benchmark_data_smithi159_139342_object19273:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f8778f:::benchmark_data_smithi159_139342_object49085:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c6f9fa76:::benchmark_data_smithi159_139342_object52298:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70094a2:::benchmark_data_smithi159_139342_object25641:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7009d82:::benchmark_data_smithi159_139342_object28380:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c700bb5d:::benchmark_data_smithi159_139342_object33553:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c701707f:::benchmark_data_smithi159_139342_object45219:head# 2024-04-15T18:35:24.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c702aed9:::benchmark_data_smithi159_139342_object28279:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70333f7:::benchmark_data_smithi159_139342_object16010:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c704cee0:::benchmark_data_smithi159_139342_object7975:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c706f6f3:::benchmark_data_smithi159_139342_object14047:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7085aa1:::benchmark_data_smithi159_139342_object51429:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7090b99:::benchmark_data_smithi159_139342_object13653:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70971c6:::benchmark_data_smithi159_139342_object19333:head# 2024-04-15T18:35:24.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7097e1e:::benchmark_data_smithi159_139342_object26461:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70b0185:::benchmark_data_smithi159_139342_object47483:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70d2442:::benchmark_data_smithi159_139342_object36793:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70e0a25:::benchmark_data_smithi159_139342_object4421:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70e28ab:::benchmark_data_smithi159_139342_object46913:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c70f0200:::benchmark_data_smithi159_139342_object19342:head# 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.446 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7119302:::benchmark_data_smithi159_139342_object2133:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71270ca:::benchmark_data_smithi159_139342_object41899:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c712eef7:::benchmark_data_smithi159_139342_object25530:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7138653:::benchmark_data_smithi159_139342_object12528:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7144719:::benchmark_data_smithi159_139342_object36077:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7153f40:::benchmark_data_smithi159_139342_object20439:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c715b867:::benchmark_data_smithi159_139342_object1709:head# 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.447 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71734ed:::benchmark_data_smithi159_139342_object17863:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c718eb3a:::benchmark_data_smithi159_139342_object33962:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c719e3c8:::benchmark_data_smithi159_139342_object37917:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71d28ef:::benchmark_data_smithi159_139342_object144:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71d86b2:::benchmark_data_smithi159_139342_object27448:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71de358:::benchmark_data_smithi159_139342_object11109:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71e47ab:::benchmark_data_smithi159_139342_object4611:head# 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.448 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c71e59dc:::benchmark_data_smithi159_139342_object27656:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c721e1a4:::benchmark_data_smithi159_139342_object49838:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7242099:::benchmark_data_smithi159_139342_object16356:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7244182:::benchmark_data_smithi159_139342_object42982:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72507b7:::benchmark_data_smithi159_139342_object10502:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c726f300:::benchmark_data_smithi159_139342_object52096:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c727636e:::benchmark_data_smithi159_139342_object41805:head# 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.449 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c727abde:::benchmark_data_smithi159_139342_object39778:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7288b4c:::benchmark_data_smithi159_139342_object26342:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72a0369:::benchmark_data_smithi159_139342_object49386:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72abb32:::benchmark_data_smithi159_139342_object45368:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72af5f2:::benchmark_data_smithi159_139342_object35929:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72b95a4:::benchmark_data_smithi159_139342_object39415:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72c0d1f:::benchmark_data_smithi159_139342_object54748:head# 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.450 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72dd1a7:::benchmark_data_smithi159_139342_object315:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72e781e:::benchmark_data_smithi159_139342_object53306:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72f19b3:::benchmark_data_smithi159_139342_object42170:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72f2de2:::benchmark_data_smithi159_139342_object52401:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72f829b:::benchmark_data_smithi159_139342_object18747:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c72fc13f:::benchmark_data_smithi159_139342_object56026:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7310be9:::benchmark_data_smithi159_139342_object25962:head# 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7356422:::benchmark_data_smithi159_139342_object911:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c735c80a:::benchmark_data_smithi159_139342_object30541:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7361f0e:::benchmark_data_smithi159_139342_object5138:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7372faa:::benchmark_data_smithi159_139342_object37755:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7378b70:::benchmark_data_smithi159_139342_object47618:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7379190:::benchmark_data_smithi159_139342_object52490:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c737af83:::benchmark_data_smithi159_139342_object7220:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c73a66c6:::benchmark_data_smithi159_139342_object12183:head# 2024-04-15T18:35:24.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c73a927e:::benchmark_data_smithi159_139342_object12040:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c73ec18b:::benchmark_data_smithi159_139342_object1393:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7406b92:::benchmark_data_smithi159_139342_object11828:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c741af35:::benchmark_data_smithi159_139342_object28615:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c741b34a:::benchmark_data_smithi159_139342_object17725:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7445362:::benchmark_data_smithi159_139342_object43515:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7448317:::benchmark_data_smithi159_139342_object33897:head# 2024-04-15T18:35:24.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c74c8f2d:::benchmark_data_smithi159_139342_object1348:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c74f27cc:::benchmark_data_smithi159_139342_object6592:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c74f7f2f:::benchmark_data_smithi159_139342_object16491:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c750273f:::benchmark_data_smithi159_139342_object476:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7553e4a:::benchmark_data_smithi159_139342_object26339:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75d32c8:::benchmark_data_smithi159_139342_object1914:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75d5b60:::benchmark_data_smithi159_139342_object17300:head# 2024-04-15T18:35:24.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75dc77d:::benchmark_data_smithi159_139342_object38110:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75f0913:::benchmark_data_smithi159_139342_object33631:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75f0cdc:::benchmark_data_smithi159_139342_object39917:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75f2c16:::benchmark_data_smithi159_139342_object44779:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75f37f0:::benchmark_data_smithi159_139342_object27943:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c75fc6fc:::benchmark_data_smithi159_139342_object7113:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c763ee95:::benchmark_data_smithi159_139342_object39448:head# 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c767edff:::benchmark_data_smithi159_139342_object44981:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76963ab:::benchmark_data_smithi159_139342_object52472:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76af302:::benchmark_data_smithi159_139342_object46835:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76b16ac:::benchmark_data_smithi159_139342_object39829:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76b5d71:::benchmark_data_smithi159_139342_object5424:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76b86c8:::benchmark_data_smithi159_139342_object34477:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76c4767:::benchmark_data_smithi159_139342_object11120:head# 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76d48a0:::benchmark_data_smithi159_139342_object52275:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c76ed94e:::benchmark_data_smithi159_139342_object17816:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c771de2c:::benchmark_data_smithi159_139342_object2628:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c773223a:::benchmark_data_smithi159_139342_object33528:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77344da:::benchmark_data_smithi159_139342_object9746:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7742d91:::benchmark_data_smithi159_139342_object9179:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77459da:::benchmark_data_smithi159_139342_object38929:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c776366f:::benchmark_data_smithi159_139342_object45585:head# 2024-04-15T18:35:24.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c778bd82:::benchmark_data_smithi159_139342_object27214:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77ba3de:::benchmark_data_smithi159_139342_object51186:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77ba735:::benchmark_data_smithi159_139342_object45194:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77bea34:::benchmark_data_smithi159_139342_object46741:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77becc6:::benchmark_data_smithi159_139342_object42074:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77edc6f:::benchmark_data_smithi159_139342_object37001:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77ee614:::benchmark_data_smithi159_139342_object47294:head# 2024-04-15T18:35:24.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.459 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c77fb662:::benchmark_data_smithi159_139342_object33072:head# 2024-04-15T18:35:24.459 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.459 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7800004:::benchmark_data_smithi159_139342_object55608:head# 2024-04-15T18:35:24.459 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.491 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7833080:::benchmark_data_smithi159_139342_object46777:head# 2024-04-15T18:35:24.491 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.491 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c784aab2:::benchmark_data_smithi159_139342_object42659:head# 2024-04-15T18:35:24.491 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.491 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c784ef11:::benchmark_data_smithi159_139342_object4417:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7854693:::benchmark_data_smithi159_139342_object3642:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c787588d:::benchmark_data_smithi159_139342_object6383:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7879c7a:::benchmark_data_smithi159_139342_object3862:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7894cd4:::benchmark_data_smithi159_139342_object4973:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78a27f7:::benchmark_data_smithi159_139342_object49719:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78a9dc9:::benchmark_data_smithi159_139342_object4493:head# 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78ac4b2:::benchmark_data_smithi159_139342_object35814:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78ae233:::benchmark_data_smithi159_139342_object6327:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78b9f7e:::benchmark_data_smithi159_139342_object49698:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78e4fdd:::benchmark_data_smithi159_139342_object29550:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c78f3e4f:::benchmark_data_smithi159_139342_object16493:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7901337:::benchmark_data_smithi159_139342_object41200:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7901490:::benchmark_data_smithi159_139342_object35827:head# 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c791935e:::benchmark_data_smithi159_139342_object22771:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c791db61:::benchmark_data_smithi159_139342_object48807:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c794fa7d:::benchmark_data_smithi159_139342_object55269:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c796b1e2:::benchmark_data_smithi159_139342_object14219:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7974315:::benchmark_data_smithi159_139342_object27701:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c797b681:::benchmark_data_smithi159_139342_object54877:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c798318a:::benchmark_data_smithi159_139342_object6052:head# 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.494 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c798cd06:::benchmark_data_smithi159_139342_object13577:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79919de:::benchmark_data_smithi159_139342_object5086:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79af38f:::benchmark_data_smithi159_139342_object16476:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79bd905:::benchmark_data_smithi159_139342_object31986:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79c1fd5:::benchmark_data_smithi159_139342_object35105:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79cca62:::benchmark_data_smithi159_139342_object45460:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79d6fc9:::benchmark_data_smithi159_139342_object20799:head# 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.495 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79db2e5:::benchmark_data_smithi159_139342_object49820:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79e4f46:::benchmark_data_smithi159_139342_object24536:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79e83f8:::benchmark_data_smithi159_139342_object36444:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79e8fac:::benchmark_data_smithi159_139342_object18860:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c79f85a8:::benchmark_data_smithi159_139342_object9732:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a156e5:::benchmark_data_smithi159_139342_object15870:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a26be8:::benchmark_data_smithi159_139342_object30976:head# 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a39ba9:::benchmark_data_smithi159_139342_object49465:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a535cb:::benchmark_data_smithi159_139342_object5930:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a65b2a:::benchmark_data_smithi159_139342_object10030:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a75245:::benchmark_data_smithi159_139342_object37337:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a791c5:::benchmark_data_smithi159_139342_object18275:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a874a9:::benchmark_data_smithi159_139342_object11562:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7a87f6a:::benchmark_data_smithi159_139342_object21495:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7aa8fe9:::benchmark_data_smithi159_139342_object42611:head# 2024-04-15T18:35:24.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7ab26a0:::benchmark_data_smithi159_139342_object26862:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7ad1153:::benchmark_data_smithi159_139342_object28432:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b0c594:::benchmark_data_smithi159_139342_object9308:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b121e0:::benchmark_data_smithi159_139342_object55293:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b1cfab:::benchmark_data_smithi159_139342_object34116:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b1d761:::benchmark_data_smithi159_139342_object17291:head# 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.498 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b2bbf5:::benchmark_data_smithi159_139342_object13909:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b3ff2d:::benchmark_data_smithi159_139342_object5468:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b4702b:::benchmark_data_smithi159_139342_object44205:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b47636:::benchmark_data_smithi159_139342_object25154:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b4e288:::benchmark_data_smithi159_139342_object9011:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b59813:::benchmark_data_smithi159_139342_object12255:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b6879a:::benchmark_data_smithi159_139342_object39858:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b698f0:::benchmark_data_smithi159_139342_object3543:head# 2024-04-15T18:35:24.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b7394c:::benchmark_data_smithi159_139342_object55684:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b9ba63:::benchmark_data_smithi159_139342_object17567:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7b9e12f:::benchmark_data_smithi159_139342_object15993:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7bbc49b:::benchmark_data_smithi159_139342_object44024:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7bbcf13:::benchmark_data_smithi159_139342_object50425:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7bc2114:::benchmark_data_smithi159_139342_object29464:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7bc425c:::benchmark_data_smithi159_139342_object11234:head# 2024-04-15T18:35:24.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7bdfcca:::benchmark_data_smithi159_139342_object27206:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c1834f:::benchmark_data_smithi159_139342_object10094:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c18dab:::benchmark_data_smithi159_139342_object15334:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c2587b:::benchmark_data_smithi159_139342_object13659:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c2684c:::benchmark_data_smithi159_139342_object24677:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c2e273:::benchmark_data_smithi159_139342_object49500:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c40860:::benchmark_data_smithi159_139342_object52696:head# 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c469ca:::benchmark_data_smithi159_139342_object52876:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c5600a:::benchmark_data_smithi159_139342_object26739:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c74189:::benchmark_data_smithi159_139342_object54724:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c7d8dd:::benchmark_data_smithi159_139342_object55550:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c8d9f0:::benchmark_data_smithi159_139342_object49849:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7c9ca2f:::benchmark_data_smithi159_139342_object6819:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7cabcba:::benchmark_data_smithi159_139342_object23265:head# 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.502 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7cd616b:::benchmark_data_smithi159_139342_object52641:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7cd6d80:::benchmark_data_smithi159_139342_object11297:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7cd7da0:::benchmark_data_smithi159_139342_object15676:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d101da:::benchmark_data_smithi159_139342_object5267:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d2dacc:::benchmark_data_smithi159_139342_object56764:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d44994:::benchmark_data_smithi159_139342_object46044:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d44e05:::benchmark_data_smithi159_139342_object48256:head# 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d599a4:::benchmark_data_smithi159_139342_object28876:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d75483:::benchmark_data_smithi159_139342_object13681:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d7c425:::benchmark_data_smithi159_139342_object35138:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7d8a4b3:::benchmark_data_smithi159_139342_object28857:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7da8b1a:::benchmark_data_smithi159_139342_object7753:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7da9fe8:::benchmark_data_smithi159_139342_object3787:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7db47e8:::benchmark_data_smithi159_139342_object39968:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7db5fb8:::benchmark_data_smithi159_139342_object56115:head# 2024-04-15T18:35:24.504 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7dbf6f7:::benchmark_data_smithi159_139342_object2257:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7dd001a:::benchmark_data_smithi159_139342_object43784:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7dd35f7:::benchmark_data_smithi159_139342_object37765:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7dde8e4:::benchmark_data_smithi159_139342_object1051:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7ddefc4:::benchmark_data_smithi159_139342_object19888:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7deb98f:::benchmark_data_smithi159_139342_object3298:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7def99a:::benchmark_data_smithi159_139342_object47746:head# 2024-04-15T18:35:24.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e12669:::benchmark_data_smithi159_139342_object3546:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e27a47:::benchmark_data_smithi159_139342_object29538:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e5a0ee:::benchmark_data_smithi159_139342_object33120:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e5b90f:::benchmark_data_smithi159_139342_object53991:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e7a04a:::benchmark_data_smithi159_139342_object55039:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e8492f:::benchmark_data_smithi159_139342_object49434:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e8da70:::benchmark_data_smithi159_139342_object54846:head# 2024-04-15T18:35:24.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7e97bde:::benchmark_data_smithi159_139342_object37681:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7ebf3a2:::benchmark_data_smithi159_139342_object9944:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7edd2a7:::benchmark_data_smithi159_139342_object5129:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7ee40ff:::benchmark_data_smithi159_139342_object9629:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f352f4:::benchmark_data_smithi159_139342_object29793:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f43b35:::benchmark_data_smithi159_139342_object9334:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f4a03e:::benchmark_data_smithi159_139342_object2877:head# 2024-04-15T18:35:24.507 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f5a2fb:::benchmark_data_smithi159_139342_object17434:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f68b47:::benchmark_data_smithi159_139342_object49406:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f70974:::benchmark_data_smithi159_139342_object48990:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f80058:::benchmark_data_smithi159_139342_object40549:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f815e6:::benchmark_data_smithi159_139342_object2578:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f862c2:::benchmark_data_smithi159_139342_object11891:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f96ee8:::benchmark_data_smithi159_139342_object16111:head# 2024-04-15T18:35:24.508 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7f9d982:::benchmark_data_smithi159_139342_object18245:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7fb86da:::benchmark_data_smithi159_139342_object12277:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7fe3ad1:::benchmark_data_smithi159_139342_object16585:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c7fe73d0:::benchmark_data_smithi159_139342_object25072:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80041e3:::benchmark_data_smithi159_139342_object38505:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c800b1ae:::benchmark_data_smithi159_139342_object13254:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c802c427:::benchmark_data_smithi159_139342_object19019:head# 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.509 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8061f55:::benchmark_data_smithi159_139342_object3209:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c808474f:::benchmark_data_smithi159_139342_object52326:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c808a203:::benchmark_data_smithi159_139342_object30103:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c809bc00:::benchmark_data_smithi159_139342_object43602:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80ab0c2:::benchmark_data_smithi159_139342_object44248:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80b6b44:::benchmark_data_smithi159_139342_object48790:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80c052b:::benchmark_data_smithi159_139342_object26398:head# 2024-04-15T18:35:24.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80c5ab3:::benchmark_data_smithi159_139342_object11854:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c80c7ae1:::benchmark_data_smithi159_139342_object18692:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c810e20f:::benchmark_data_smithi159_139342_object41993:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c810edf3:::benchmark_data_smithi159_139342_object36521:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c811b4bf:::benchmark_data_smithi159_139342_object5047:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8156e91:::benchmark_data_smithi159_139342_object12938:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c81695e8:::benchmark_data_smithi159_139342_object6000:head# 2024-04-15T18:35:24.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8174ca7:::benchmark_data_smithi159_139342_object46531:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8191171:::benchmark_data_smithi159_139342_object14390:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8196d81:::benchmark_data_smithi159_139342_object27889:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8198ccf:::benchmark_data_smithi159_139342_object37398:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c81b1784:::benchmark_data_smithi159_139342_object24868:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c81b9aa9:::benchmark_data_smithi159_139342_object52361:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c81d9bef:::benchmark_data_smithi159_139342_object53621:head# 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.512 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c81e94c6:::benchmark_data_smithi159_139342_object56741:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c821b0bd:::benchmark_data_smithi159_139342_object39465:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8227d27:::benchmark_data_smithi159_139342_object56753:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82314cf:::benchmark_data_smithi159_139342_object42278:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c824033a:::benchmark_data_smithi159_139342_object52011:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8272f61:::benchmark_data_smithi159_139342_object22395:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8286d65:::benchmark_data_smithi159_139342_object44666:head# 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.513 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8286e5c:::benchmark_data_smithi159_139342_object31934:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8291987:::benchmark_data_smithi159_139342_object37745:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c829ec24:::benchmark_data_smithi159_139342_object48921:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82aa56b:::benchmark_data_smithi159_139342_object49813:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82d15a8:::benchmark_data_smithi159_139342_object34869:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82d1b3c:::benchmark_data_smithi159_139342_object21374:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82d22bc:::benchmark_data_smithi159_139342_object36732:head# 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.514 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82d6471:::benchmark_data_smithi159_139342_object50355:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c82f3cf5:::benchmark_data_smithi159_139342_object40990:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8306515:::benchmark_data_smithi159_139342_object16794:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c830ecbe:::benchmark_data_smithi159_139342_object45005:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8310eec:::benchmark_data_smithi159_139342_object33926:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8328571:::benchmark_data_smithi159_139342_object7628:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8329261:::benchmark_data_smithi159_139342_object1078:head# 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.515 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c832a2b0:::benchmark_data_smithi159_139342_object41348:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c832bf40:::benchmark_data_smithi159_139342_object8075:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8330ee8:::benchmark_data_smithi159_139342_object20909:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c833edb7:::benchmark_data_smithi159_139342_object56453:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c835efa2:::benchmark_data_smithi159_139342_object1762:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83602c4:::benchmark_data_smithi159_139342_object32002:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83673e6:::benchmark_data_smithi159_139342_object44405:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c837e841:::benchmark_data_smithi159_139342_object29593:head# 2024-04-15T18:35:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8395e8f:::benchmark_data_smithi159_139342_object29451:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83968f3:::benchmark_data_smithi159_139342_object45127:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83ad3b6:::benchmark_data_smithi159_139342_object37503:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83ba34d:::benchmark_data_smithi159_139342_object21118:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83da119:::benchmark_data_smithi159_139342_object31768:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83e299b:::benchmark_data_smithi159_139342_object43678:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83e887a:::benchmark_data_smithi159_139342_object22160:head# 2024-04-15T18:35:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83f1234:::benchmark_data_smithi159_139342_object36693:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c83fe8e3:::benchmark_data_smithi159_139342_object57141:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8429446:::benchmark_data_smithi159_139342_object1956:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8439644:::benchmark_data_smithi159_139342_object42455:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c843b424:::benchmark_data_smithi159_139342_object49997:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c843eeb1:::benchmark_data_smithi159_139342_object52640:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8462146:::benchmark_data_smithi159_139342_object23930:head# 2024-04-15T18:35:24.518 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c847c661:::benchmark_data_smithi159_139342_object231:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c84809be:::benchmark_data_smithi159_139342_object40656:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8485963:::benchmark_data_smithi159_139342_object14005:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c84872d6:::benchmark_data_smithi159_139342_object52049:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c849c013:::benchmark_data_smithi159_139342_object12919:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c850927d:::benchmark_data_smithi159_139342_object49876:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c850b70f:::benchmark_data_smithi159_139342_object55672:head# 2024-04-15T18:35:24.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85228d8:::benchmark_data_smithi159_139342_object53420:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8537261:::benchmark_data_smithi159_139342_object20432:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c854202e:::benchmark_data_smithi159_139342_object55239:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c854a357:::benchmark_data_smithi159_139342_object13312:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c854d2e1:::benchmark_data_smithi159_139342_object24955:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c856207f:::benchmark_data_smithi159_139342_object47535:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85669d3:::benchmark_data_smithi159_139342_object10912:head# 2024-04-15T18:35:24.520 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c856b5ad:::benchmark_data_smithi159_139342_object20773:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8583b9a:::benchmark_data_smithi159_139342_object17171:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c858566a:::benchmark_data_smithi159_139342_object14764:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8598263:::benchmark_data_smithi159_139342_object21527:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85a81e5:::benchmark_data_smithi159_139342_object34822:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85ac9c5:::benchmark_data_smithi159_139342_object51542:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85c145d:::benchmark_data_smithi159_139342_object47210:head# 2024-04-15T18:35:24.521 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85cb1aa:::benchmark_data_smithi159_139342_object54787:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85dffcb:::benchmark_data_smithi159_139342_object26619:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85e0637:::benchmark_data_smithi159_139342_object32509:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c85e6c0c:::benchmark_data_smithi159_139342_object20326:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8616e1c:::benchmark_data_smithi159_139342_object25225:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c864a796:::benchmark_data_smithi159_139342_object12249:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c865c8ea:::benchmark_data_smithi159_139342_object9646:head# 2024-04-15T18:35:24.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86640b7:::benchmark_data_smithi159_139342_object4626:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c866c39a:::benchmark_data_smithi159_139342_object19087:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c866fb16:::benchmark_data_smithi159_139342_object54910:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8685870:::benchmark_data_smithi159_139342_object34143:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8690049:::benchmark_data_smithi159_139342_object43301:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86a69b9:::benchmark_data_smithi159_139342_object13059:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86c2bca:::benchmark_data_smithi159_139342_object34432:head# 2024-04-15T18:35:24.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86db00d:::benchmark_data_smithi159_139342_object31661:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86dd83e:::benchmark_data_smithi159_139342_object54003:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c86e806a:::benchmark_data_smithi159_139342_object41686:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8704daa:::benchmark_data_smithi159_139342_object4047:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8708f13:::benchmark_data_smithi159_139342_object8901:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c871844f:::benchmark_data_smithi159_139342_object43330:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8728f29:::benchmark_data_smithi159_139342_object42679:head# 2024-04-15T18:35:24.524 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8745beb:::benchmark_data_smithi159_139342_object17437:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c875c5ac:::benchmark_data_smithi159_139342_object39199:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c87602dd:::benchmark_data_smithi159_139342_object56658:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8773565:::benchmark_data_smithi159_139342_object14198:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c877d094:::benchmark_data_smithi159_139342_object37665:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c87815f5:::benchmark_data_smithi159_139342_object20510:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c878ae90:::benchmark_data_smithi159_139342_object24876:head# 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c879fa78:::benchmark_data_smithi159_139342_object6278:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c87e3550:::benchmark_data_smithi159_139342_object36862:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c87eeffb:::benchmark_data_smithi159_139342_object31574:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c87f1d91:::benchmark_data_smithi159_139342_object42870:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8837e74:::benchmark_data_smithi159_139342_object19402:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c883808b:::benchmark_data_smithi159_139342_object22567:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8887b6f:::benchmark_data_smithi159_139342_object26904:head# 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c88bbdaf:::benchmark_data_smithi159_139342_object16616:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c88d79ef:::benchmark_data_smithi159_139342_object16108:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c88ee53d:::benchmark_data_smithi159_139342_object1829:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8909590:::benchmark_data_smithi159_139342_object44684:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c891919d:::benchmark_data_smithi159_139342_object6824:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c893009f:::benchmark_data_smithi159_139342_object27024:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8943453:::benchmark_data_smithi159_139342_object15563:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c894e0ef:::benchmark_data_smithi159_139342_object47101:head# 2024-04-15T18:35:24.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8960115:::benchmark_data_smithi159_139342_object16775:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c896c41a:::benchmark_data_smithi159_139342_object7350:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8972794:::benchmark_data_smithi159_139342_object40953:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c898abee:::benchmark_data_smithi159_139342_object3532:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8993d02:::benchmark_data_smithi159_139342_object21594:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c89be36d:::benchmark_data_smithi159_139342_object2951:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c89d3582:::benchmark_data_smithi159_139342_object43045:head# 2024-04-15T18:35:24.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c89fa7ca:::benchmark_data_smithi159_139342_object7353:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a012dd:::benchmark_data_smithi159_139342_object56073:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a0dfd2:::benchmark_data_smithi159_139342_object21646:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a14307:::benchmark_data_smithi159_139342_object24904:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a26006:::benchmark_data_smithi159_139342_object7856:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a2b627:::benchmark_data_smithi159_139342_object55482:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a2c5bd:::benchmark_data_smithi159_139342_object20575:head# 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a549ac:::benchmark_data_smithi159_139342_object48777:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8a77084:::benchmark_data_smithi159_139342_object10481:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8aa89af:::benchmark_data_smithi159_139342_object22443:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ab2c4f:::benchmark_data_smithi159_139342_object19094:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ab3d4a:::benchmark_data_smithi159_139342_object46583:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8abcb23:::benchmark_data_smithi159_139342_object22493:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ac1a19:::benchmark_data_smithi159_139342_object6818:head# 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ace785:::benchmark_data_smithi159_139342_object27535:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8afcf3e:::benchmark_data_smithi159_139342_object22206:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8afe682:::benchmark_data_smithi159_139342_object7861:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b20a98:::benchmark_data_smithi159_139342_object23204:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b4c15a:::benchmark_data_smithi159_139342_object29804:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b52b27:::benchmark_data_smithi159_139342_object14518:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b77a93:::benchmark_data_smithi159_139342_object21710:head# 2024-04-15T18:35:24.531 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b77b2f:::benchmark_data_smithi159_139342_object6002:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8b9c192:::benchmark_data_smithi159_139342_object54222:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ba3755:::benchmark_data_smithi159_139342_object25692:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8bbe210:::benchmark_data_smithi159_139342_object41261:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8bcd6bb:::benchmark_data_smithi159_139342_object39481:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8bd396f:::benchmark_data_smithi159_139342_object15875:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8bd7787:::benchmark_data_smithi159_139342_object15079:head# 2024-04-15T18:35:24.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8be223e:::benchmark_data_smithi159_139342_object10914:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c046ee:::benchmark_data_smithi159_139342_object33992:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c067a7:::benchmark_data_smithi159_139342_object17688:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c092da:::benchmark_data_smithi159_139342_object29781:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c0a7a1:::benchmark_data_smithi159_139342_object5464:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c1388d:::benchmark_data_smithi159_139342_object56540:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c14b0e:::benchmark_data_smithi159_139342_object31303:head# 2024-04-15T18:35:24.533 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c17175:::benchmark_data_smithi159_139342_object20926:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c4ad4c:::benchmark_data_smithi159_139342_object35741:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c5ed18:::benchmark_data_smithi159_139342_object17057:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c60cf2:::benchmark_data_smithi159_139342_object33978:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c62b76:::benchmark_data_smithi159_139342_object8257:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c808c3:::benchmark_data_smithi159_139342_object22874:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c87fe2:::benchmark_data_smithi159_139342_object48232:head# 2024-04-15T18:35:24.534 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8c88186:::benchmark_data_smithi159_139342_object43262:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8cab0b5:::benchmark_data_smithi159_139342_object38992:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8cb307a:::benchmark_data_smithi159_139342_object38125:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ccfa43:::benchmark_data_smithi159_139342_object22074:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8cd3e7b:::benchmark_data_smithi159_139342_object48248:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ce45a1:::benchmark_data_smithi159_139342_object21181:head# 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.535 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d0896b:::benchmark_data_smithi159_139342_object13295:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d1e671:::benchmark_data_smithi159_139342_object53495:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d24206:::benchmark_data_smithi159_139342_object21236:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d28869:::benchmark_data_smithi159_139342_object15063:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d344e9:::benchmark_data_smithi159_139342_object39967:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d3479f:::benchmark_data_smithi159_139342_object46237:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d50ac2:::benchmark_data_smithi159_139342_object29773:head# 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d51949:::benchmark_data_smithi159_139342_object18709:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d58713:::benchmark_data_smithi159_139342_object36234:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d60aea:::benchmark_data_smithi159_139342_object55334:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d816ef:::benchmark_data_smithi159_139342_object415:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d926f1:::benchmark_data_smithi159_139342_object40180:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d981ff:::benchmark_data_smithi159_139342_object4304:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d9894d:::benchmark_data_smithi159_139342_object17951:head# 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8d9c6aa:::benchmark_data_smithi159_139342_object41160:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8da2982:::benchmark_data_smithi159_139342_object30492:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8da9e41:::benchmark_data_smithi159_139342_object18870:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8db2869:::benchmark_data_smithi159_139342_object37500:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8dcdf3f:::benchmark_data_smithi159_139342_object29731:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8dfe913:::benchmark_data_smithi159_139342_object53875:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e3436b:::benchmark_data_smithi159_139342_object52878:head# 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e3d17c:::benchmark_data_smithi159_139342_object18751:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e551f4:::benchmark_data_smithi159_139342_object38957:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e68822:::benchmark_data_smithi159_139342_object37070:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e7d128:::benchmark_data_smithi159_139342_object40118:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8e9e078:::benchmark_data_smithi159_139342_object53154:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ea5c99:::benchmark_data_smithi159_139342_object39463:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8eaa6b5:::benchmark_data_smithi159_139342_object48328:head# 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ec4a91:::benchmark_data_smithi159_139342_object5309:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ecc9de:::benchmark_data_smithi159_139342_object34139:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8ef9124:::benchmark_data_smithi159_139342_object51976:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f28093:::benchmark_data_smithi159_139342_object14901:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f38765:::benchmark_data_smithi159_139342_object24261:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f3ac71:::benchmark_data_smithi159_139342_object51651:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f42440:::benchmark_data_smithi159_139342_object45211:head# 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f5b24a:::benchmark_data_smithi159_139342_object27553:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f67b02:::benchmark_data_smithi159_139342_object5200:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f803fc:::benchmark_data_smithi159_139342_object31489:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f848de:::benchmark_data_smithi159_139342_object25654:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f89388:::benchmark_data_smithi159_139342_object23567:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8f8f31f:::benchmark_data_smithi159_139342_object24397:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fac0fb:::benchmark_data_smithi159_139342_object13693:head# 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fb1668:::benchmark_data_smithi159_139342_object48491:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fbdf39:::benchmark_data_smithi159_139342_object34359:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fe14be:::benchmark_data_smithi159_139342_object719:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fea19d:::benchmark_data_smithi159_139342_object51853:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c8fef31f:::benchmark_data_smithi159_139342_object15236:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9004a8c:::benchmark_data_smithi159_139342_object40995:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9068107:::benchmark_data_smithi159_139342_object4138:head# 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c906a08e:::benchmark_data_smithi159_139342_object7546:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c906b729:::benchmark_data_smithi159_139342_object13220:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c906e88c:::benchmark_data_smithi159_139342_object39766:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9077e93:::benchmark_data_smithi159_139342_object52612:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90aa25e:::benchmark_data_smithi159_139342_object1378:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90b8b04:::benchmark_data_smithi159_139342_object20917:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90caf12:::benchmark_data_smithi159_139342_object14861:head# 2024-04-15T18:35:24.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90cf399:::benchmark_data_smithi159_139342_object23468:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90e4967:::benchmark_data_smithi159_139342_object36236:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c90fa4d4:::benchmark_data_smithi159_139342_object14778:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9101eec:::benchmark_data_smithi159_139342_object6585:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c912355f:::benchmark_data_smithi159_139342_object45796:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9135bd2:::benchmark_data_smithi159_139342_object12449:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c913779c:::benchmark_data_smithi159_139342_object32027:head# 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c914fc4f:::benchmark_data_smithi159_139342_object16593:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c915585f:::benchmark_data_smithi159_139342_object21337:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91570aa:::benchmark_data_smithi159_139342_object11210:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9159ffa:::benchmark_data_smithi159_139342_object32767:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91658b3:::benchmark_data_smithi159_139342_object10862:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c919c26a:::benchmark_data_smithi159_139342_object53479:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91be162:::benchmark_data_smithi159_139342_object39667:head# 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.545 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91dd2b4:::benchmark_data_smithi159_139342_object1624:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91e399f:::benchmark_data_smithi159_139342_object38684:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c91f2304:::benchmark_data_smithi159_139342_object20267:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92184e0:::benchmark_data_smithi159_139342_object47171:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92365a0:::benchmark_data_smithi159_139342_object37147:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9237ff6:::benchmark_data_smithi159_139342_object54861:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c923cafa:::benchmark_data_smithi159_139342_object15336:head# 2024-04-15T18:35:24.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92819d5:::benchmark_data_smithi159_139342_object26715:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c928c43d:::benchmark_data_smithi159_139342_object31499:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92a6579:::benchmark_data_smithi159_139342_object19171:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92abb2d:::benchmark_data_smithi159_139342_object37636:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92ad0f0:::benchmark_data_smithi159_139342_object37340:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92b845b:::benchmark_data_smithi159_139342_object31092:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92c98b3:::benchmark_data_smithi159_139342_object42729:head# 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92ceacb:::benchmark_data_smithi159_139342_object16084:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92d3777:::benchmark_data_smithi159_139342_object3027:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92efb36:::benchmark_data_smithi159_139342_object7924:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c92f4c39:::benchmark_data_smithi159_139342_object6884:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9312e90:::benchmark_data_smithi159_139342_object39062:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c931edc0:::benchmark_data_smithi159_139342_object19815:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c93273b0:::benchmark_data_smithi159_139342_object23364:head# 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c934bd77:::benchmark_data_smithi159_139342_object17206:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c936626d:::benchmark_data_smithi159_139342_object23175:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c936ace8:::benchmark_data_smithi159_139342_object32586:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9383cc2:::benchmark_data_smithi159_139342_object50558:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c93be78d:::benchmark_data_smithi159_139342_object29584:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c93f3a4b:::benchmark_data_smithi159_139342_object21803:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c93f67b1:::benchmark_data_smithi159_139342_object43709:head# 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9432050:::benchmark_data_smithi159_139342_object1004:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c944657a:::benchmark_data_smithi159_139342_object54181:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c944d3a8:::benchmark_data_smithi159_139342_object422:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c948a582:::benchmark_data_smithi159_139342_object45674:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c949a482:::benchmark_data_smithi159_139342_object6456:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94a1e69:::benchmark_data_smithi159_139342_object20733:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94ada26:::benchmark_data_smithi159_139342_object46395:head# 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.550 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94adbb9:::benchmark_data_smithi159_139342_object39948:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94b2142:::benchmark_data_smithi159_139342_object26309:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94bb72a:::benchmark_data_smithi159_139342_object39343:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94cd101:::benchmark_data_smithi159_139342_object14987:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94d8786:::benchmark_data_smithi159_139342_object20205:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94da3d2:::benchmark_data_smithi159_139342_object5038:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94dda1c:::benchmark_data_smithi159_139342_object25237:head# 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.551 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94e61f2:::benchmark_data_smithi159_139342_object18881:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94e6243:::benchmark_data_smithi159_139342_object47595:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94f0195:::benchmark_data_smithi159_139342_object21114:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c94f62ef:::benchmark_data_smithi159_139342_object46195:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c953510c:::benchmark_data_smithi159_139342_object36786:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9539eb5:::benchmark_data_smithi159_139342_object50961:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9556eca:::benchmark_data_smithi159_139342_object28482:head# 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.552 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9567c36:::benchmark_data_smithi159_139342_object26478:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95690fb:::benchmark_data_smithi159_139342_object38020:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9578429:::benchmark_data_smithi159_139342_object9204:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c958cd36:::benchmark_data_smithi159_139342_object6167:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c959f920:::benchmark_data_smithi159_139342_object46374:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95a83be:::benchmark_data_smithi159_139342_object40088:head# 2024-04-15T18:35:24.553 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95b7dbd:::benchmark_data_smithi159_139342_object14812:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95cdea2:::benchmark_data_smithi159_139342_object37017:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95d13c5:::benchmark_data_smithi159_139342_object53627:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95da31a:::benchmark_data_smithi159_139342_object18523:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95ed828:::benchmark_data_smithi159_139342_object44923:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c95f5c8b:::benchmark_data_smithi159_139342_object14508:head# 2024-04-15T18:35:24.558 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9600e73:::benchmark_data_smithi159_139342_object23817:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9612d2f:::benchmark_data_smithi159_139342_object19338:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9613fcb:::benchmark_data_smithi159_139342_object45938:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c961fc00:::benchmark_data_smithi159_139342_object17027:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c962f87b:::benchmark_data_smithi159_139342_object11348:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9630d12:::benchmark_data_smithi159_139342_object12451:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9632576:::benchmark_data_smithi159_139342_object15108:head# 2024-04-15T18:35:24.559 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96464ac:::benchmark_data_smithi159_139342_object31945:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96660e8:::benchmark_data_smithi159_139342_object21067:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c966f702:::benchmark_data_smithi159_139342_object30435:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c967e225:::benchmark_data_smithi159_139342_object2310:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c969b259:::benchmark_data_smithi159_139342_object26388:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96a3b91:::benchmark_data_smithi159_139342_object21875:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96beb63:::benchmark_data_smithi159_139342_object2909:head# 2024-04-15T18:35:24.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96c6b58:::benchmark_data_smithi159_139342_object35257:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96cd79d:::benchmark_data_smithi159_139342_object17029:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c96cef52:::benchmark_data_smithi159_139342_object39114:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c971365a:::benchmark_data_smithi159_139342_object42624:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9717486:::benchmark_data_smithi159_139342_object31079:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c974e5f9:::benchmark_data_smithi159_139342_object6901:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c975f459:::benchmark_data_smithi159_139342_object16757:head# 2024-04-15T18:35:24.561 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c976f528:::benchmark_data_smithi159_139342_object56797:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c977bf7d:::benchmark_data_smithi159_139342_object1550:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97960b5:::benchmark_data_smithi159_139342_object46150:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c979b715:::benchmark_data_smithi159_139342_object11509:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c979b98e:::benchmark_data_smithi159_139342_object45065:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97a4be4:::benchmark_data_smithi159_139342_object52475:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97b46b4:::benchmark_data_smithi159_139342_object33340:head# 2024-04-15T18:35:24.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97cc794:::benchmark_data_smithi159_139342_object4430:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97d3004:::benchmark_data_smithi159_139342_object657:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97daf1b:::benchmark_data_smithi159_139342_object36251:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c97fd1eb:::benchmark_data_smithi159_139342_object11302:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9814b56:::benchmark_data_smithi159_139342_object51722:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9827528:::benchmark_data_smithi159_139342_object16293:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c982e29e:::benchmark_data_smithi159_139342_object56153:head# 2024-04-15T18:35:24.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9834518:::benchmark_data_smithi159_139342_object29212:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c98347f1:::benchmark_data_smithi159_139342_object13932:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c986ccb8:::benchmark_data_smithi159_139342_object7672:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9875949:::benchmark_data_smithi159_139342_object14672:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c988b126:::benchmark_data_smithi159_139342_object42682:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c98f2a35:::benchmark_data_smithi159_139342_object39330:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c98f4092:::benchmark_data_smithi159_139342_object43067:head# 2024-04-15T18:35:24.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9960344:::benchmark_data_smithi159_139342_object22204:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9968c0e:::benchmark_data_smithi159_139342_object9504:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9973f82:::benchmark_data_smithi159_139342_object13164:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9974e71:::benchmark_data_smithi159_139342_object28498:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c997d82d:::benchmark_data_smithi159_139342_object12359:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c999ef5b:::benchmark_data_smithi159_139342_object4776:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99ad65b:::benchmark_data_smithi159_139342_object833:head# 2024-04-15T18:35:24.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99be1de:::benchmark_data_smithi159_139342_object52183:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99db7de:::benchmark_data_smithi159_139342_object28119:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99def5b:::benchmark_data_smithi159_139342_object32047:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99f20c9:::benchmark_data_smithi159_139342_object40359:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99f5584:::benchmark_data_smithi159_139342_object47545:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c99f6ab6:::benchmark_data_smithi159_139342_object48176:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a32466:::benchmark_data_smithi159_139342_object9050:head# 2024-04-15T18:35:24.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a473cc:::benchmark_data_smithi159_139342_object49067:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a5471c:::benchmark_data_smithi159_139342_object7867:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a569ae:::benchmark_data_smithi159_139342_object51743:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a712fc:::benchmark_data_smithi159_139342_object18241:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a7565b:::benchmark_data_smithi159_139342_object13834:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a757b5:::benchmark_data_smithi159_139342_object50314:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a818c7:::benchmark_data_smithi159_139342_object3682:head# 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a89058:::benchmark_data_smithi159_139342_object3732:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a973b7:::benchmark_data_smithi159_139342_object19346:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9a9dff6:::benchmark_data_smithi159_139342_object28480:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9aa723d:::benchmark_data_smithi159_139342_object14495:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9aacefa:::benchmark_data_smithi159_139342_object8696:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9aaf28a:::benchmark_data_smithi159_139342_object19217:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ac2338:::benchmark_data_smithi159_139342_object56340:head# 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b046ec:::benchmark_data_smithi159_139342_object21825:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b0e3f9:::benchmark_data_smithi159_139342_object37954:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b1eceb:::benchmark_data_smithi159_139342_object45933:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b47d94:::benchmark_data_smithi159_139342_object11214:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b49bd8:::benchmark_data_smithi159_139342_object33252:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b4c2ea:::benchmark_data_smithi159_139342_object49287:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b5201d:::benchmark_data_smithi159_139342_object11496:head# 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.569 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b6df2a:::benchmark_data_smithi159_139342_object33272:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b6f665:::benchmark_data_smithi159_139342_object27629:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b73850:::benchmark_data_smithi159_139342_object33169:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9b94575:::benchmark_data_smithi159_139342_object21952:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ba4fb8:::benchmark_data_smithi159_139342_object9274:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ba7b52:::benchmark_data_smithi159_139342_object4411:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9bb5eef:::benchmark_data_smithi159_139342_object19740:head# 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.570 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9beee2a:::benchmark_data_smithi159_139342_object43615:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9bf06ab:::benchmark_data_smithi159_139342_object50341:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9bf6b93:::benchmark_data_smithi159_139342_object37679:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9bfacbb:::benchmark_data_smithi159_139342_object46726:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c06d0e:::benchmark_data_smithi159_139342_object20279:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c0c07c:::benchmark_data_smithi159_139342_object54418:head# 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.571 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c120bc:::benchmark_data_smithi159_139342_object16310:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c30b7e:::benchmark_data_smithi159_139342_object9344:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c3a5b7:::benchmark_data_smithi159_139342_object42223:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c3d509:::benchmark_data_smithi159_139342_object54311:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c43252:::benchmark_data_smithi159_139342_object41170:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c48daa:::benchmark_data_smithi159_139342_object8021:head# 2024-04-15T18:35:24.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.573 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c5fd3c:::benchmark_data_smithi159_139342_object2703:head# 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c7a851:::benchmark_data_smithi159_139342_object55978:head# 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c8bef7:::benchmark_data_smithi159_139342_object35790:head# 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9c8d056:::benchmark_data_smithi159_139342_object39386:head# 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cc10c5:::benchmark_data_smithi159_139342_object22846:head# 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cd4225:::benchmark_data_smithi159_139342_object34024:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cda880:::benchmark_data_smithi159_139342_object35559:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cdf13a:::benchmark_data_smithi159_139342_object29209:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cefae8:::benchmark_data_smithi159_139342_object21011:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9cf161f:::benchmark_data_smithi159_139342_object16264:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d0efde:::benchmark_data_smithi159_139342_object20976:head# 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d1b489:::benchmark_data_smithi159_139342_object39883:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d49fe1:::benchmark_data_smithi159_139342_object17457:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d587fc:::benchmark_data_smithi159_139342_object52700:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d71773:::benchmark_data_smithi159_139342_object15341:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9d88934:::benchmark_data_smithi159_139342_object29669:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9dab015:::benchmark_data_smithi159_139342_object29607:head# 2024-04-15T18:35:24.576 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.577 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9dba2d7:::benchmark_data_smithi159_139342_object13540:head# 2024-04-15T18:35:24.577 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9dc6a0d:::benchmark_data_smithi159_139342_object12005:head# 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9dd459f:::benchmark_data_smithi159_139342_object36274:head# 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9de50f5:::benchmark_data_smithi159_139342_object23034:head# 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.578 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9deb011:::benchmark_data_smithi159_139342_object5578:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9df32b0:::benchmark_data_smithi159_139342_object52970:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e0d46c:::benchmark_data_smithi159_139342_object1256:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e4aaf1:::benchmark_data_smithi159_139342_object26653:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e54e86:::benchmark_data_smithi159_139342_object43228:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e6d68a:::benchmark_data_smithi159_139342_object24961:head# 2024-04-15T18:35:24.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e72340:::benchmark_data_smithi159_139342_object46167:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e726f4:::benchmark_data_smithi159_139342_object43145:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9e899ea:::benchmark_data_smithi159_139342_object33339:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ea9314:::benchmark_data_smithi159_139342_object57300:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9eb80e0:::benchmark_data_smithi159_139342_object52396:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ebd2d4:::benchmark_data_smithi159_139342_object33523:head# 2024-04-15T18:35:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.582 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ecabb9:::benchmark_data_smithi159_139342_object17273:head# 2024-04-15T18:35:24.582 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.582 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ecaec8:::benchmark_data_smithi159_139342_object17866:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9ee5078:::benchmark_data_smithi159_139342_object19697:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f12a9b:::benchmark_data_smithi159_139342_object55364:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f14824:::benchmark_data_smithi159_139342_object15851:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f330ca:::benchmark_data_smithi159_139342_object18648:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f44f9f:::benchmark_data_smithi159_139342_object29536:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f50b9e:::benchmark_data_smithi159_139342_object54453:head# 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9f96c77:::benchmark_data_smithi159_139342_object12591:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9fd7353:::benchmark_data_smithi159_139342_object9276:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9fd79d1:::benchmark_data_smithi159_139342_object22982:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:c9fe0e67:::benchmark_data_smithi159_139342_object44583:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca01143b:::benchmark_data_smithi159_139342_object32543:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca018e3f:::benchmark_data_smithi159_139342_object16646:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca01cdfb:::benchmark_data_smithi159_139342_object14354:head# 2024-04-15T18:35:24.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca029181:::benchmark_data_smithi159_139342_object32384:head# 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca029bdb:::benchmark_data_smithi159_139342_object38486:head# 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca03c31b:::benchmark_data_smithi159_139342_object21658:head# 2024-04-15T18:35:24.585 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.586 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca03f2bd:::benchmark_data_smithi159_139342_object17261:head# 2024-04-15T18:35:24.586 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.586 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca05077e:::benchmark_data_smithi159_139342_object53287:head# 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca060c45:::benchmark_data_smithi159_139342_object15317:head# 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca0934c1:::benchmark_data_smithi159_139342_object36791:head# 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.587 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca0abec6:::benchmark_data_smithi159_139342_object11558:head# 2024-04-15T18:35:24.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.588 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca0faef8:::benchmark_data_smithi159_139342_object33307:head# 2024-04-15T18:35:24.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.588 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca104536:::benchmark_data_smithi159_139342_object20649:head# 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1246c6:::benchmark_data_smithi159_139342_object36879:head# 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca142163:::benchmark_data_smithi159_139342_object29687:head# 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca147537:::benchmark_data_smithi159_139342_object42678:head# 2024-04-15T18:35:24.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca178117:::benchmark_data_smithi159_139342_object36568:head# 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca190022:::benchmark_data_smithi159_139342_object8464:head# 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca19bf13:::benchmark_data_smithi159_139342_object17259:head# 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1ae060:::benchmark_data_smithi159_139342_object32451:head# 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1b429e:::benchmark_data_smithi159_139342_object46723:head# 2024-04-15T18:35:24.591 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1b4e4d:::benchmark_data_smithi159_139342_object44551:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1ba149:::benchmark_data_smithi159_139342_object22781:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1c5e88:::benchmark_data_smithi159_139342_object18880:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca1dc0d0:::benchmark_data_smithi159_139342_object11293:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca211128:::benchmark_data_smithi159_139342_object19581:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca223553:::benchmark_data_smithi159_139342_object53235:head# 2024-04-15T18:35:24.592 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.593 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca228c2b:::benchmark_data_smithi159_139342_object46600:head# 2024-04-15T18:35:24.593 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.593 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca22ecf8:::benchmark_data_smithi159_139342_object55949:head# 2024-04-15T18:35:24.593 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.593 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca231d64:::benchmark_data_smithi159_139342_object29758:head# 2024-04-15T18:35:24.594 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.594 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca2357c3:::benchmark_data_smithi159_139342_object29901:head# 2024-04-15T18:35:24.594 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.597 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca23bc2a:::benchmark_data_smithi159_139342_object7232:head# 2024-04-15T18:35:24.597 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.598 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca243613:::benchmark_data_smithi159_139342_object36648:head# 2024-04-15T18:35:24.598 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.598 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca249ff5:::benchmark_data_smithi159_139342_object3252:head# 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca24b8cd:::benchmark_data_smithi159_139342_object53948:head# 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca256b55:::benchmark_data_smithi159_139342_object3916:head# 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca26530c:::benchmark_data_smithi159_139342_object28112:head# 2024-04-15T18:35:24.599 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca2691c4:::benchmark_data_smithi159_139342_object5376:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca28a963:::benchmark_data_smithi159_139342_object51895:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca2a8bc6:::benchmark_data_smithi159_139342_object14483:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca2c2991:::benchmark_data_smithi159_139342_object39528:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca2fa1e4:::benchmark_data_smithi159_139342_object49097:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca308f71:::benchmark_data_smithi159_139342_object15723:head# 2024-04-15T18:35:24.601 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.602 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca30bf92:::benchmark_data_smithi159_139342_object17575:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3295c0:::benchmark_data_smithi159_139342_object23999:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca331351:::benchmark_data_smithi159_139342_object6820:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca35ba98:::benchmark_data_smithi159_139342_object24322:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca37edd8:::benchmark_data_smithi159_139342_object40642:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca38d267:::benchmark_data_smithi159_139342_object32895:head# 2024-04-15T18:35:24.603 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.604 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca392912:::benchmark_data_smithi159_139342_object16993:head# 2024-04-15T18:35:24.604 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.604 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3a5469:::benchmark_data_smithi159_139342_object13800:head# 2024-04-15T18:35:24.604 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3b6a7b:::benchmark_data_smithi159_139342_object40020:head# 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3b89cc:::benchmark_data_smithi159_139342_object57258:head# 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3bcb71:::benchmark_data_smithi159_139342_object664:head# 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3bda2e:::benchmark_data_smithi159_139342_object49375:head# 2024-04-15T18:35:24.605 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3cacf7:::benchmark_data_smithi159_139342_object5034:head# 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca3dd758:::benchmark_data_smithi159_139342_object1898:head# 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca40bd73:::benchmark_data_smithi159_139342_object39993:head# 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca424b45:::benchmark_data_smithi159_139342_object43757:head# 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.606 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4267de:::benchmark_data_smithi159_139342_object31780:head# 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4481c8:::benchmark_data_smithi159_139342_object21596:head# 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca44d07a:::benchmark_data_smithi159_139342_object15244:head# 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.607 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca44f887:::benchmark_data_smithi159_139342_object19363:head# 2024-04-15T18:35:24.608 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca46798e:::benchmark_data_smithi159_139342_object52251:head# 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca47046e:::benchmark_data_smithi159_139342_object36222:head# 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca485871:::benchmark_data_smithi159_139342_object1369:head# 2024-04-15T18:35:24.614 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.618 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4a7813:::benchmark_data_smithi159_139342_object44992:head# 2024-04-15T18:35:24.619 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4baa0d:::benchmark_data_smithi159_139342_object13406:head# 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4bda29:::benchmark_data_smithi159_139342_object27095:head# 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4c5d76:::benchmark_data_smithi159_139342_object21093:head# 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4e6bce:::benchmark_data_smithi159_139342_object16224:head# 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4e8db6:::benchmark_data_smithi159_139342_object30755:head# 2024-04-15T18:35:24.625 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.626 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca4f6ca2:::benchmark_data_smithi159_139342_object8067:head# 2024-04-15T18:35:24.626 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.626 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca50a125:::benchmark_data_smithi159_139342_object47603:head# 2024-04-15T18:35:24.626 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca50f133:::benchmark_data_smithi159_139342_object33136:head# 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca52d117:::benchmark_data_smithi159_139342_object5745:head# 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5351b8:::benchmark_data_smithi159_139342_object15092:head# 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca53eea5:::benchmark_data_smithi159_139342_object11581:head# 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5429cd:::benchmark_data_smithi159_139342_object54489:head# 2024-04-15T18:35:24.627 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5444ff:::benchmark_data_smithi159_139342_object37712:head# 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5489bf:::benchmark_data_smithi159_139342_object12178:head# 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca549101:::benchmark_data_smithi159_139342_object37127:head# 2024-04-15T18:35:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.630 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca57018d:::benchmark_data_smithi159_139342_object47337:head# 2024-04-15T18:35:24.630 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.631 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca57da3c:::benchmark_data_smithi159_139342_object56976:head# 2024-04-15T18:35:24.631 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.631 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5829b5:::benchmark_data_smithi159_139342_object47549:head# 2024-04-15T18:35:24.631 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.631 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5829de:::benchmark_data_smithi159_139342_object24562:head# 2024-04-15T18:35:24.632 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.632 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca585ab7:::benchmark_data_smithi159_139342_object47924:head# 2024-04-15T18:35:24.632 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca59f058:::benchmark_data_smithi159_139342_object34907:head# 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5b22e0:::benchmark_data_smithi159_139342_object21641:head# 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5c0be0:::benchmark_data_smithi159_139342_object17949:head# 2024-04-15T18:35:24.633 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5c0f7b:::benchmark_data_smithi159_139342_object3310:head# 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5cf72e:::benchmark_data_smithi159_139342_object30219:head# 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5d3cc0:::benchmark_data_smithi159_139342_object18482:head# 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5dc39b:::benchmark_data_smithi159_139342_object8872:head# 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.634 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5ed00c:::benchmark_data_smithi159_139342_object33271:head# 2024-04-15T18:35:24.635 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca5ee132:::benchmark_data_smithi159_139342_object42183:head# 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca62f916:::benchmark_data_smithi159_139342_object56197:head# 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca638eac:::benchmark_data_smithi159_139342_object24769:head# 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca645551:::benchmark_data_smithi159_139342_object34413:head# 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca655333:::benchmark_data_smithi159_139342_object27111:head# 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.637 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca665e2b:::benchmark_data_smithi159_139342_object55115:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca66f866:::benchmark_data_smithi159_139342_object26097:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca67b8e1:::benchmark_data_smithi159_139342_object46657:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca68f50c:::benchmark_data_smithi159_139342_object30215:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6cb780:::benchmark_data_smithi159_139342_object12566:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6e3225:::benchmark_data_smithi159_139342_object12521:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6e58eb:::benchmark_data_smithi159_139342_object27773:head# 2024-04-15T18:35:24.638 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6ebd94:::benchmark_data_smithi159_139342_object27068:head# 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6ee4f8:::benchmark_data_smithi159_139342_object13561:head# 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6f6164:::benchmark_data_smithi159_139342_object37364:head# 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca6fc5b7:::benchmark_data_smithi159_139342_object8225:head# 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca71826f:::benchmark_data_smithi159_139342_object6289:head# 2024-04-15T18:35:24.640 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca728db2:::benchmark_data_smithi159_139342_object49714:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7734b8:::benchmark_data_smithi159_139342_object48653:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca783308:::benchmark_data_smithi159_139342_object10292:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca78b476:::benchmark_data_smithi159_139342_object54465:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca790505:::benchmark_data_smithi159_139342_object28443:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7973c1:::benchmark_data_smithi159_139342_object8124:head# 2024-04-15T18:35:24.641 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.642 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca79acd8:::benchmark_data_smithi159_139342_object44051:head# 2024-04-15T18:35:24.642 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.642 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7a4a59:::benchmark_data_smithi159_139342_object27626:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7a5385:::benchmark_data_smithi159_139342_object50539:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7c9a49:::benchmark_data_smithi159_139342_object56029:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca7fe878:::benchmark_data_smithi159_139342_object51047:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca829dd3:::benchmark_data_smithi159_139342_object2893:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8387af:::benchmark_data_smithi159_139342_object34540:head# 2024-04-15T18:35:24.643 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.644 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca84b36b:::benchmark_data_smithi159_139342_object38177:head# 2024-04-15T18:35:24.644 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.647 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca85e572:::benchmark_data_smithi159_139342_object45887:head# 2024-04-15T18:35:24.647 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.647 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca85f4d0:::benchmark_data_smithi159_139342_object18990:head# 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca87355c:::benchmark_data_smithi159_139342_object39011:head# 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8854e8:::benchmark_data_smithi159_139342_object6911:head# 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca890cdd:::benchmark_data_smithi159_139342_object8729:head# 2024-04-15T18:35:24.648 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca89c9bc:::benchmark_data_smithi159_139342_object54937:head# 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8abf2e:::benchmark_data_smithi159_139342_object16422:head# 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8b5ba1:::benchmark_data_smithi159_139342_object20730:head# 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8da232:::benchmark_data_smithi159_139342_object46997:head# 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8dc81e:::benchmark_data_smithi159_139342_object30171:head# 2024-04-15T18:35:24.649 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.650 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8ddf37:::benchmark_data_smithi159_139342_object19410:head# 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8df266:::benchmark_data_smithi159_139342_object2907:head# 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca8e6232:::benchmark_data_smithi159_139342_object18869:head# 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9070f7:::benchmark_data_smithi159_139342_object19215:head# 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca91023c:::benchmark_data_smithi159_139342_object13316:head# 2024-04-15T18:35:24.651 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.653 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca916a7e:::benchmark_data_smithi159_139342_object26485:head# 2024-04-15T18:35:24.653 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.653 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca920be6:::benchmark_data_smithi159_139342_object32211:head# 2024-04-15T18:35:24.653 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca955822:::benchmark_data_smithi159_139342_object28855:head# 2024-04-15T18:35:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca959fb4:::benchmark_data_smithi159_139342_object34222:head# 2024-04-15T18:35:24.654 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.655 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca966723:::benchmark_data_smithi159_139342_object21900:head# 2024-04-15T18:35:24.655 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.655 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca98f158:::benchmark_data_smithi159_139342_object29748:head# 2024-04-15T18:35:24.655 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca993ac9:::benchmark_data_smithi159_139342_object30214:head# 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca995bf1:::benchmark_data_smithi159_139342_object46636:head# 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9b9ac8:::benchmark_data_smithi159_139342_object30418:head# 2024-04-15T18:35:24.656 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9c71e3:::benchmark_data_smithi159_139342_object8981:head# 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9d0143:::benchmark_data_smithi159_139342_object19320:head# 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9d1559:::benchmark_data_smithi159_139342_object40969:head# 2024-04-15T18:35:24.657 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.659 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9d6030:::benchmark_data_smithi159_139342_object3112:head# 2024-04-15T18:35:24.659 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9df43f:::benchmark_data_smithi159_139342_object46644:head# 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ca9f4e7c:::benchmark_data_smithi159_139342_object30309:head# 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa09d22:::benchmark_data_smithi159_139342_object25457:head# 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa0caea:::benchmark_data_smithi159_139342_object9816:head# 2024-04-15T18:35:24.660 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.661 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa0f56d:::benchmark_data_smithi159_139342_object37168:head# 2024-04-15T18:35:24.661 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.661 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa13880:::benchmark_data_smithi159_139342_object33067:head# 2024-04-15T18:35:24.661 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa1b2f9:::benchmark_data_smithi159_139342_object45415:head# 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa25fbc:::benchmark_data_smithi159_139342_object10651:head# 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa2e1e1:::benchmark_data_smithi159_139342_object16139:head# 2024-04-15T18:35:24.662 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa2e24d:::benchmark_data_smithi159_139342_object20471:head# 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa2fc92:::benchmark_data_smithi159_139342_object56291:head# 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa300a9:::benchmark_data_smithi159_139342_object17764:head# 2024-04-15T18:35:24.663 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa34007:::benchmark_data_smithi159_139342_object44435:head# 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa54cba:::benchmark_data_smithi159_139342_object37046:head# 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa76e8c:::benchmark_data_smithi159_139342_object33749:head# 2024-04-15T18:35:24.671 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa86886:::benchmark_data_smithi159_139342_object57010:head# 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caa8e4f6:::benchmark_data_smithi159_139342_object44354:head# 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caaa3d91:::benchmark_data_smithi159_139342_object24298:head# 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caab9576:::benchmark_data_smithi159_139342_object29085:head# 2024-04-15T18:35:24.672 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.673 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caae54b5:::benchmark_data_smithi159_139342_object37779:head# 2024-04-15T18:35:24.673 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.673 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caae5d10:::benchmark_data_smithi159_139342_object1044:head# 2024-04-15T18:35:24.673 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caaeecd7:::benchmark_data_smithi159_139342_object43291:head# 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caaefed9:::benchmark_data_smithi159_139342_object52597:head# 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab18451:::benchmark_data_smithi159_139342_object21:head# 2024-04-15T18:35:24.674 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab1e203:::benchmark_data_smithi159_139342_object23714:head# 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab23bc7:::benchmark_data_smithi159_139342_object16602:head# 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab2d0f9:::benchmark_data_smithi159_139342_object30295:head# 2024-04-15T18:35:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.677 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab31afa:::benchmark_data_smithi159_139342_object96:head# 2024-04-15T18:35:24.677 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.677 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab37286:::benchmark_data_smithi159_139342_object23346:head# 2024-04-15T18:35:24.677 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.678 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab5bab2:::benchmark_data_smithi159_139342_object18876:head# 2024-04-15T18:35:24.678 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.680 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab6346f:::benchmark_data_smithi159_139342_object18520:head# 2024-04-15T18:35:24.680 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.680 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab8636b:::benchmark_data_smithi159_139342_object56193:head# 2024-04-15T18:35:24.680 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cab90a2a:::benchmark_data_smithi159_139342_object7474:head# 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cabae522:::benchmark_data_smithi159_139342_object54566:head# 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cabbc25a:::benchmark_data_smithi159_139342_object18790:head# 2024-04-15T18:35:24.681 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.682 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac1a37f:::benchmark_data_smithi159_139342_object45711:head# 2024-04-15T18:35:24.682 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.682 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac2758c:::benchmark_data_smithi159_139342_object56419:head# 2024-04-15T18:35:24.682 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac2db37:::benchmark_data_smithi159_139342_object38707:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac37efb:::benchmark_data_smithi159_139342_object10349:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac385b1:::benchmark_data_smithi159_139342_object8740:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac3b505:::benchmark_data_smithi159_139342_object39845:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac3d56a:::benchmark_data_smithi159_139342_object25258:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac3d8fb:::benchmark_data_smithi159_139342_object34416:head# 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.690 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac5d1f7:::benchmark_data_smithi159_139342_object11905:head# 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac5e60f:::benchmark_data_smithi159_139342_object51242:head# 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cac8e829:::benchmark_data_smithi159_139342_object39606:head# 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacb662f:::benchmark_data_smithi159_139342_object22481:head# 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacb75bc:::benchmark_data_smithi159_139342_object9072:head# 2024-04-15T18:35:24.691 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.694 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacc2488:::benchmark_data_smithi159_139342_object14156:head# 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacd6ebc:::benchmark_data_smithi159_139342_object42536:head# 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacd7193:::benchmark_data_smithi159_139342_object17179:head# 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cace4700:::benchmark_data_smithi159_139342_object45074:head# 2024-04-15T18:35:24.695 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.704 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cacf8ed2:::benchmark_data_smithi159_139342_object12043:head# 2024-04-15T18:35:24.704 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad15776:::benchmark_data_smithi159_139342_object12670:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad22023:::benchmark_data_smithi159_139342_object7253:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad22c52:::benchmark_data_smithi159_139342_object27281:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad29dc3:::benchmark_data_smithi159_139342_object52423:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad30e85:::benchmark_data_smithi159_139342_object40909:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad3419e:::benchmark_data_smithi159_139342_object37942:head# 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.710 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad41199:::benchmark_data_smithi159_139342_object17689:head# 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad5b8ad:::benchmark_data_smithi159_139342_object4349:head# 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad624d4:::benchmark_data_smithi159_139342_object20942:head# 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad68d0e:::benchmark_data_smithi159_139342_object21525:head# 2024-04-15T18:35:24.711 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.720 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad84029:::benchmark_data_smithi159_139342_object25946:head# 2024-04-15T18:35:24.720 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cad909ca:::benchmark_data_smithi159_139342_object53294:head# 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cadb4dfa:::benchmark_data_smithi159_139342_object18818:head# 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae05d02:::benchmark_data_smithi159_139342_object16349:head# 2024-04-15T18:35:24.721 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae0e898:::benchmark_data_smithi159_139342_object32408:head# 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae19962:::benchmark_data_smithi159_139342_object46765:head# 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae254ef:::benchmark_data_smithi159_139342_object15705:head# 2024-04-15T18:35:24.736 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae437d0:::benchmark_data_smithi159_139342_object1331:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae4bc97:::benchmark_data_smithi159_139342_object30310:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae540dc:::benchmark_data_smithi159_139342_object16703:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae675a2:::benchmark_data_smithi159_139342_object12446:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae69971:::benchmark_data_smithi159_139342_object44271:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae70516:::benchmark_data_smithi159_139342_object39707:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae7394b:::benchmark_data_smithi159_139342_object37011:head# 2024-04-15T18:35:24.737 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae913ea:::benchmark_data_smithi159_139342_object20727:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cae9ea0d:::benchmark_data_smithi159_139342_object52577:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caea1e4d:::benchmark_data_smithi159_139342_object23649:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caeab72f:::benchmark_data_smithi159_139342_object56495:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caec6869:::benchmark_data_smithi159_139342_object11541:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caed5f62:::benchmark_data_smithi159_139342_object4584:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caef69bf:::benchmark_data_smithi159_139342_object21953:head# 2024-04-15T18:35:24.738 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caef7d62:::benchmark_data_smithi159_139342_object51492:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf0a338:::benchmark_data_smithi159_139342_object55142:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf0b1ec:::benchmark_data_smithi159_139342_object30800:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf46232:::benchmark_data_smithi159_139342_object24821:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf5ad2e:::benchmark_data_smithi159_139342_object8336:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf5f942:::benchmark_data_smithi159_139342_object23442:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf5fa53:::benchmark_data_smithi159_139342_object16169:head# 2024-04-15T18:35:24.739 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caf9cd37:::benchmark_data_smithi159_139342_object34211:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cafbe3a7:::benchmark_data_smithi159_139342_object44856:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cafdc5f8:::benchmark_data_smithi159_139342_object50734:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caff88d5:::benchmark_data_smithi159_139342_object50112:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:caffecca:::benchmark_data_smithi159_139342_object23056:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb00ce2f:::benchmark_data_smithi159_139342_object36558:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb00e676:::benchmark_data_smithi159_139342_object43162:head# 2024-04-15T18:35:24.740 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb0112eb:::benchmark_data_smithi159_139342_object33133:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb01d086:::benchmark_data_smithi159_139342_object8053:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb025bc1:::benchmark_data_smithi159_139342_object32658:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb0337e1:::benchmark_data_smithi159_139342_object19002:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb03b4cd:::benchmark_data_smithi159_139342_object6027:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb040619:::benchmark_data_smithi159_139342_object56858:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb0765a5:::benchmark_data_smithi159_139342_object51707:head# 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.741 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb077066:::benchmark_data_smithi159_139342_object31918:head# 2024-04-15T18:35:24.742 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb093faf:::benchmark_data_smithi159_139342_object4757:head# 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb0af5a7:::benchmark_data_smithi159_139342_object15742:head# 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb12891b:::benchmark_data_smithi159_139342_object12318:head# 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb161b2d:::benchmark_data_smithi159_139342_object16533:head# 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb163cb3:::benchmark_data_smithi159_139342_object28497:head# 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.743 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb16f552:::benchmark_data_smithi159_139342_object54258:head# 2024-04-15T18:35:24.744 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.744 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb178da2:::benchmark_data_smithi159_139342_object12615:head# 2024-04-15T18:35:24.744 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb18a502:::benchmark_data_smithi159_139342_object5465:head# 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1bca0a:::benchmark_data_smithi159_139342_object10236:head# 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1c2d49:::benchmark_data_smithi159_139342_object51053:head# 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1cd1e0:::benchmark_data_smithi159_139342_object2011:head# 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1e0e78:::benchmark_data_smithi159_139342_object38670:head# 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.745 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1f89f2:::benchmark_data_smithi159_139342_object17111:head# 2024-04-15T18:35:24.746 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.750 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb1faa3f:::benchmark_data_smithi159_139342_object45983:head# 2024-04-15T18:35:24.750 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.757 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb200007:::benchmark_data_smithi159_139342_object49852:head# 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb205d22:::benchmark_data_smithi159_139342_object16463:head# 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb20a22e:::benchmark_data_smithi159_139342_object35801:head# 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb214d8d:::benchmark_data_smithi159_139342_object49099:head# 2024-04-15T18:35:24.758 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb21cec8:::benchmark_data_smithi159_139342_object41005:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb234d5d:::benchmark_data_smithi159_139342_object12353:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb255bc0:::benchmark_data_smithi159_139342_object16331:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb257fe8:::benchmark_data_smithi159_139342_object10687:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb273d01:::benchmark_data_smithi159_139342_object49705:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb29ef5b:::benchmark_data_smithi159_139342_object47364:head# 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.762 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb2c128d:::benchmark_data_smithi159_139342_object2437:head# 2024-04-15T18:35:24.763 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.768 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb2d401c:::benchmark_data_smithi159_139342_object35729:head# 2024-04-15T18:35:24.768 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.772 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb2d9d19:::benchmark_data_smithi159_139342_object9378:head# 2024-04-15T18:35:24.772 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb2dc79a:::benchmark_data_smithi159_139342_object50904:head# 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb306871:::benchmark_data_smithi159_139342_object12637:head# 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb306d45:::benchmark_data_smithi159_139342_object15754:head# 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb31c79d:::benchmark_data_smithi159_139342_object12356:head# 2024-04-15T18:35:24.777 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3261a2:::benchmark_data_smithi159_139342_object7721:head# 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb358ffa:::benchmark_data_smithi159_139342_object49335:head# 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb35f901:::benchmark_data_smithi159_139342_object2263:head# 2024-04-15T18:35:24.778 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.780 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb372f86:::benchmark_data_smithi159_139342_object53049:head# 2024-04-15T18:35:24.780 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.780 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb392e48:::benchmark_data_smithi159_139342_object11230:head# 2024-04-15T18:35:24.780 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.781 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3a48b1:::benchmark_data_smithi159_139342_object1756:head# 2024-04-15T18:35:24.781 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.781 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3acba2:::benchmark_data_smithi159_139342_object36722:head# 2024-04-15T18:35:24.781 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.788 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3c1f6b:::benchmark_data_smithi159_139342_object22342:head# 2024-04-15T18:35:24.788 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.793 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3d669e:::benchmark_data_smithi159_139342_object43401:head# 2024-04-15T18:35:24.793 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.794 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb3f0db8:::benchmark_data_smithi159_139342_object24915:head# 2024-04-15T18:35:24.794 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.795 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb40617e:::benchmark_data_smithi159_139342_object10129:head# 2024-04-15T18:35:24.795 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.795 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb418fbc:::benchmark_data_smithi159_139342_object5021:head# 2024-04-15T18:35:24.795 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.795 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb41dae4:::benchmark_data_smithi159_139342_object40679:head# 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb42dbae:::benchmark_data_smithi159_139342_object16043:head# 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb44ab99:::benchmark_data_smithi159_139342_object26970:head# 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.796 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb45bfc2:::benchmark_data_smithi159_139342_object36833:head# 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb47234d:::benchmark_data_smithi159_139342_object26681:head# 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb475dd8:::benchmark_data_smithi159_139342_object10466:head# 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb48968b:::benchmark_data_smithi159_139342_object16923:head# 2024-04-15T18:35:24.797 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.808 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb48c3e9:::benchmark_data_smithi159_139342_object28549:head# 2024-04-15T18:35:24.808 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.808 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb49b34c:::benchmark_data_smithi159_139342_object21934:head# 2024-04-15T18:35:24.808 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.809 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb4c900a:::benchmark_data_smithi159_139342_object32596:head# 2024-04-15T18:35:24.809 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb4d2f41:::benchmark_data_smithi159_139342_object4698:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb4ddf7c:::benchmark_data_smithi159_139342_object22963:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb50a65b:::benchmark_data_smithi159_139342_object41266:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb515b34:::benchmark_data_smithi159_139342_object10876:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5189a0:::benchmark_data_smithi159_139342_object33779:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb531bd3:::benchmark_data_smithi159_139342_object51406:head# 2024-04-15T18:35:24.812 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.813 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb53b21f:::benchmark_data_smithi159_139342_object24187:head# 2024-04-15T18:35:24.813 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.822 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5567cb:::benchmark_data_smithi159_139342_object39872:head# 2024-04-15T18:35:24.822 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb55a27a:::benchmark_data_smithi159_139342_object13887:head# 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb560fc5:::benchmark_data_smithi159_139342_object17708:head# 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb573c7c:::benchmark_data_smithi159_139342_object15957:head# 2024-04-15T18:35:24.836 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.837 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb587324:::benchmark_data_smithi159_139342_object1395:head# 2024-04-15T18:35:24.837 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.837 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb58773e:::benchmark_data_smithi159_139342_object54828:head# 2024-04-15T18:35:24.837 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.838 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb58f95a:::benchmark_data_smithi159_139342_object14397:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb599034:::benchmark_data_smithi159_139342_object50992:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5a0327:::benchmark_data_smithi159_139342_object15959:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5a6300:::benchmark_data_smithi159_139342_object30997:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5bacc8:::benchmark_data_smithi159_139342_object2957:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5bb5c3:::benchmark_data_smithi159_139342_object49487:head# 2024-04-15T18:35:24.839 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.840 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb5d20d7:::benchmark_data_smithi159_139342_object36507:head# 2024-04-15T18:35:24.840 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.844 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb617ec6:::benchmark_data_smithi159_139342_object17811:head# 2024-04-15T18:35:24.844 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb61befd:::benchmark_data_smithi159_139342_object26090:head# 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb633551:::benchmark_data_smithi159_139342_object30540:head# 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb663e00:::benchmark_data_smithi159_139342_object5923:head# 2024-04-15T18:35:24.845 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb680fcd:::benchmark_data_smithi159_139342_object55172:head# 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb68334e:::benchmark_data_smithi159_139342_object31965:head# 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb68a7ed:::benchmark_data_smithi159_139342_object3769:head# 2024-04-15T18:35:24.846 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.852 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb691b0c:::benchmark_data_smithi159_139342_object54534:head# 2024-04-15T18:35:24.852 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.853 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb694dd4:::benchmark_data_smithi159_139342_object3001:head# 2024-04-15T18:35:24.853 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.853 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb6a39b5:::benchmark_data_smithi159_139342_object35012:head# 2024-04-15T18:35:24.853 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.862 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb6a5d8c:::benchmark_data_smithi159_139342_object47352:head# 2024-04-15T18:35:24.862 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.863 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb6bc0ba:::benchmark_data_smithi159_139342_object16675:head# 2024-04-15T18:35:24.863 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.869 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb6ee8cb:::benchmark_data_smithi159_139342_object49148:head# 2024-04-15T18:35:24.869 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.871 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb6f94fe:::benchmark_data_smithi159_139342_object12727:head# 2024-04-15T18:35:24.871 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.874 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7098ce:::benchmark_data_smithi159_139342_object49118:head# 2024-04-15T18:35:24.874 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.874 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb70fccb:::benchmark_data_smithi159_139342_object43390:head# 2024-04-15T18:35:24.874 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb728968:::benchmark_data_smithi159_139342_object29952:head# 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb730f79:::benchmark_data_smithi159_139342_object48716:head# 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb757892:::benchmark_data_smithi159_139342_object25248:head# 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb794ad6:::benchmark_data_smithi159_139342_object30867:head# 2024-04-15T18:35:24.875 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7a5f3a:::benchmark_data_smithi159_139342_object9190:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7a8b67:::benchmark_data_smithi159_139342_object40123:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7aff93:::benchmark_data_smithi159_139342_object2938:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7b6047:::benchmark_data_smithi159_139342_object56980:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7b9eae:::benchmark_data_smithi159_139342_object47903:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7c3cd5:::benchmark_data_smithi159_139342_object22857:head# 2024-04-15T18:35:24.877 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7eb7a3:::benchmark_data_smithi159_139342_object22996:head# 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb7ffb47:::benchmark_data_smithi159_139342_object49842:head# 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8165a1:::benchmark_data_smithi159_139342_object32522:head# 2024-04-15T18:35:24.878 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.879 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb833d00:::benchmark_data_smithi159_139342_object4527:head# 2024-04-15T18:35:24.879 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.879 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb83eecd:::benchmark_data_smithi159_139342_object32618:head# 2024-04-15T18:35:24.879 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.881 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb841b62:::benchmark_data_smithi159_139342_object40441:head# 2024-04-15T18:35:24.881 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.883 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb85fc6a:::benchmark_data_smithi159_139342_object55484:head# 2024-04-15T18:35:24.883 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.883 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb86358f:::benchmark_data_smithi159_139342_object56534:head# 2024-04-15T18:35:24.883 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.886 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb86d4aa:::benchmark_data_smithi159_139342_object6447:head# 2024-04-15T18:35:24.886 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.887 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8c7ce3:::benchmark_data_smithi159_139342_object34016:head# 2024-04-15T18:35:24.888 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8c7e26:::benchmark_data_smithi159_139342_object47870:head# 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8d9660:::benchmark_data_smithi159_139342_object29730:head# 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8e0d6a:::benchmark_data_smithi159_139342_object49380:head# 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8e1867:::benchmark_data_smithi159_139342_object21726:head# 2024-04-15T18:35:24.889 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.895 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8e5dfe:::benchmark_data_smithi159_139342_object50010:head# 2024-04-15T18:35:24.895 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.898 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8e9da0:::benchmark_data_smithi159_139342_object17596:head# 2024-04-15T18:35:24.898 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb8ff95c:::benchmark_data_smithi159_139342_object33980:head# 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9015e9:::benchmark_data_smithi159_139342_object35427:head# 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb904bad:::benchmark_data_smithi159_139342_object56501:head# 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.899 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9062ea:::benchmark_data_smithi159_139342_object15138:head# 2024-04-15T18:35:24.900 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.912 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb914eac:::benchmark_data_smithi159_139342_object21731:head# 2024-04-15T18:35:24.912 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.921 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb915641:::benchmark_data_smithi159_139342_object5735:head# 2024-04-15T18:35:24.921 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.923 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb92f6d8:::benchmark_data_smithi159_139342_object9121:head# 2024-04-15T18:35:24.923 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.923 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb945b31:::benchmark_data_smithi159_139342_object22010:head# 2024-04-15T18:35:24.924 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.924 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb953546:::benchmark_data_smithi159_139342_object11365:head# 2024-04-15T18:35:24.924 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb953dae:::benchmark_data_smithi159_139342_object24119:head# 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb961cac:::benchmark_data_smithi159_139342_object14293:head# 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb966afd:::benchmark_data_smithi159_139342_object14466:head# 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9684d0:::benchmark_data_smithi159_139342_object48518:head# 2024-04-15T18:35:24.925 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9918c1:::benchmark_data_smithi159_139342_object2841:head# 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9b4858:::benchmark_data_smithi159_139342_object31992:head# 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cb9cf83a:::benchmark_data_smithi159_139342_object37715:head# 2024-04-15T18:35:24.926 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.927 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba0585e:::benchmark_data_smithi159_139342_object19289:head# 2024-04-15T18:35:24.927 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.933 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba0ba3c:::benchmark_data_smithi159_139342_object50092:head# 2024-04-15T18:35:24.933 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba43567:::benchmark_data_smithi159_139342_object42782:head# 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba4add7:::benchmark_data_smithi159_139342_object15488:head# 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba63f70:::benchmark_data_smithi159_139342_object50634:head# 2024-04-15T18:35:24.934 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.935 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba64d41:::benchmark_data_smithi159_139342_object23387:head# 2024-04-15T18:35:24.935 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.935 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba89038:::benchmark_data_smithi159_139342_object30618:head# 2024-04-15T18:35:24.935 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cba90298:::benchmark_data_smithi159_139342_object52057:head# 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbaa5438:::benchmark_data_smithi159_139342_object46585:head# 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbaac7da:::benchmark_data_smithi159_139342_object44177:head# 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbaadd82:::benchmark_data_smithi159_139342_object56518:head# 2024-04-15T18:35:24.936 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbab64ec:::benchmark_data_smithi159_139342_object50145:head# 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbacd5ea:::benchmark_data_smithi159_139342_object36784:head# 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbacdd7f:::benchmark_data_smithi159_139342_object26850:head# 2024-04-15T18:35:24.938 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbae1f0d:::benchmark_data_smithi159_139342_object13525:head# 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbae8071:::benchmark_data_smithi159_139342_object9019:head# 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbaf5e3b:::benchmark_data_smithi159_139342_object29838:head# 2024-04-15T18:35:24.939 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.940 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb23a5c:::benchmark_data_smithi159_139342_object39418:head# 2024-04-15T18:35:24.940 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.940 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb2a7cf:::benchmark_data_smithi159_139342_object26521:head# 2024-04-15T18:35:24.941 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.941 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb4ed87:::benchmark_data_smithi159_139342_object19600:head# 2024-04-15T18:35:24.941 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb5a0df:::benchmark_data_smithi159_139342_object54990:head# 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb670a9:::benchmark_data_smithi159_139342_object45520:head# 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb6da44:::benchmark_data_smithi159_139342_object9659:head# 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb700be:::benchmark_data_smithi159_139342_object30979:head# 2024-04-15T18:35:24.942 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.946 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb74742:::benchmark_data_smithi159_139342_object1603:head# 2024-04-15T18:35:24.946 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.947 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb7bd9d:::benchmark_data_smithi159_139342_object27451:head# 2024-04-15T18:35:24.947 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.947 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb8c627:::benchmark_data_smithi159_139342_object8029:head# 2024-04-15T18:35:24.947 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.947 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:24.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:24 smithi082 ceph-mon[44206]: pgmap v1552: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 8344/115944 objects degraded (7.197%); 16 MiB/s, 248 objects/s recovering 2024-04-15T18:35:24.950 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb8ddcc:::benchmark_data_smithi159_139342_object24929:head# 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbb9a7ae:::benchmark_data_smithi159_139342_object32897:head# 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbbb2441:::benchmark_data_smithi159_139342_object41470:head# 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbbdffe3:::benchmark_data_smithi159_139342_object24049:head# 2024-04-15T18:35:24.951 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:24.954 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbbeb714:::benchmark_data_smithi159_139342_object1089:head# 2024-04-15T18:35:24.954 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.959 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbbf3885:::benchmark_data_smithi159_139342_object16290:head# 2024-04-15T18:35:24.959 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.962 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc16947:::benchmark_data_smithi159_139342_object29101:head# 2024-04-15T18:35:24.962 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.966 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc2d532:::benchmark_data_smithi159_139342_object16188:head# 2024-04-15T18:35:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc2ef95:::benchmark_data_smithi159_139342_object13111:head# 2024-04-15T18:35:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc3deb4:::benchmark_data_smithi159_139342_object32300:head# 2024-04-15T18:35:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:24 smithi049 ceph-mon[40087]: pgmap v1552: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 8344/115944 objects degraded (7.197%); 16 MiB/s, 248 objects/s recovering 2024-04-15T18:35:24.973 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc4c257:::benchmark_data_smithi159_139342_object14318:head# 2024-04-15T18:35:24.973 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.974 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc533e9:::benchmark_data_smithi159_139342_object35397:head# 2024-04-15T18:35:24.974 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc5a820:::benchmark_data_smithi159_139342_object6151:head# 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc5bbff:::benchmark_data_smithi159_139342_object49885:head# 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc7af31:::benchmark_data_smithi159_139342_object5394:head# 2024-04-15T18:35:24.975 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc93026:::benchmark_data_smithi159_139342_object5451:head# 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc95ba4:::benchmark_data_smithi159_139342_object35768:head# 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbc9aea9:::benchmark_data_smithi159_139342_object56076:head# 2024-04-15T18:35:24.976 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.977 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbcaf03c:::benchmark_data_smithi159_139342_object5321:head# 2024-04-15T18:35:24.977 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.977 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbcc128f:::benchmark_data_smithi159_139342_object34167:head# 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbccfb6c:::benchmark_data_smithi159_139342_object24054:head# 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbcecd43:::benchmark_data_smithi159_139342_object31872:head# 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd04f72:::benchmark_data_smithi159_139342_object32180:head# 2024-04-15T18:35:24.978 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd23254:::benchmark_data_smithi159_139342_object29235:head# 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd37467:::benchmark_data_smithi159_139342_object9726:head# 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd4fa3c:::benchmark_data_smithi159_139342_object18094:head# 2024-04-15T18:35:24.979 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd5ba41:::benchmark_data_smithi159_139342_object2827:head# 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd5d9af:::benchmark_data_smithi159_139342_object3479:head# 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd64f5c:::benchmark_data_smithi159_139342_object33178:head# 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbd8563c:::benchmark_data_smithi159_139342_object26928:head# 2024-04-15T18:35:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.990 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbdcb179:::benchmark_data_smithi159_139342_object37379:head# 2024-04-15T18:35:24.990 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbdcce3e:::benchmark_data_smithi159_139342_object21184:head# 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbde0129:::benchmark_data_smithi159_139342_object16795:head# 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbde0a96:::benchmark_data_smithi159_139342_object9446:head# 2024-04-15T18:35:24.991 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbdffe32:::benchmark_data_smithi159_139342_object46785:head# 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe11b60:::benchmark_data_smithi159_139342_object18614:head# 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe1eb56:::benchmark_data_smithi159_139342_object56690:head# 2024-04-15T18:35:24.992 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe2212a:::benchmark_data_smithi159_139342_object40346:head# 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe2a6b5:::benchmark_data_smithi159_139342_object33101:head# 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe3819f:::benchmark_data_smithi159_139342_object21328:head# 2024-04-15T18:35:24.996 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe46078:::benchmark_data_smithi159_139342_object50191:head# 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe4a466:::benchmark_data_smithi159_139342_object52808:head# 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe56459:::benchmark_data_smithi159_139342_object47124:head# 2024-04-15T18:35:24.997 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe62217:::benchmark_data_smithi159_139342_object1484:head# 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe6f788:::benchmark_data_smithi159_139342_object36635:head# 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe70e04:::benchmark_data_smithi159_139342_object23599:head# 2024-04-15T18:35:24.998 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:24.999 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbe9348b:::benchmark_data_smithi159_139342_object6938:head# 2024-04-15T18:35:24.999 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.002 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbeb1e6d:::benchmark_data_smithi159_139342_object44412:head# 2024-04-15T18:35:25.002 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.010 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbede4af:::benchmark_data_smithi159_139342_object6451:head# 2024-04-15T18:35:25.010 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbee0b9d:::benchmark_data_smithi159_139342_object39032:head# 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf048a8:::benchmark_data_smithi159_139342_object25690:head# 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf08515:::benchmark_data_smithi159_139342_object46746:head# 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf1c162:::benchmark_data_smithi159_139342_object7577:head# 2024-04-15T18:35:25.019 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf1f239:::benchmark_data_smithi159_139342_object8290:head# 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf38988:::benchmark_data_smithi159_139342_object4622:head# 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf3c2a2:::benchmark_data_smithi159_139342_object31361:head# 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf45167:::benchmark_data_smithi159_139342_object26725:head# 2024-04-15T18:35:25.021 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf5c789:::benchmark_data_smithi159_139342_object36895:head# 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf703d0:::benchmark_data_smithi159_139342_object10666:head# 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf99cd4:::benchmark_data_smithi159_139342_object34214:head# 2024-04-15T18:35:25.022 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbf9b79b:::benchmark_data_smithi159_139342_object37460:head# 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbfae2eb:::benchmark_data_smithi159_139342_object52836:head# 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbfb1802:::benchmark_data_smithi159_139342_object56170:head# 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbfdf23a:::benchmark_data_smithi159_139342_object24214:head# 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbfecccb:::benchmark_data_smithi159_139342_object39836:head# 2024-04-15T18:35:25.023 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.024 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbff042a:::benchmark_data_smithi159_139342_object40582:head# 2024-04-15T18:35:25.024 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.024 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cbffae4a:::benchmark_data_smithi159_139342_object37372:head# 2024-04-15T18:35:25.024 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.025 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc002228:::benchmark_data_smithi159_139342_object39961:head# 2024-04-15T18:35:25.025 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.026 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc00f451:::benchmark_data_smithi159_139342_object23489:head# 2024-04-15T18:35:25.026 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc016e39:::benchmark_data_smithi159_139342_object39480:head# 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc02b188:::benchmark_data_smithi159_139342_object33046:head# 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc03958c:::benchmark_data_smithi159_139342_object45076:head# 2024-04-15T18:35:25.033 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.034 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc039928:::benchmark_data_smithi159_139342_object21795:head# 2024-04-15T18:35:25.034 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.038 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc04423a:::benchmark_data_smithi159_139342_object37714:head# 2024-04-15T18:35:25.038 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.038 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc0804fb:::benchmark_data_smithi159_139342_object32689:head# 2024-04-15T18:35:25.039 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.039 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc08e577:::benchmark_data_smithi159_139342_object26832:head# 2024-04-15T18:35:25.039 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc0947bd:::benchmark_data_smithi159_139342_object5781:head# 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc09a108:::benchmark_data_smithi159_139342_object24991:head# 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc09c78d:::benchmark_data_smithi159_139342_object3578:head# 2024-04-15T18:35:25.044 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.048 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc0f1f16:::benchmark_data_smithi159_139342_object47601:head# 2024-04-15T18:35:25.048 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.048 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc12cffc:::benchmark_data_smithi159_139342_object28005:head# 2024-04-15T18:35:25.048 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.049 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc15642d:::benchmark_data_smithi159_139342_object3873:head# 2024-04-15T18:35:25.049 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.049 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc173ec2:::benchmark_data_smithi159_139342_object12047:head# 2024-04-15T18:35:25.049 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.050 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc176838:::benchmark_data_smithi159_139342_object28278:head# 2024-04-15T18:35:25.050 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.050 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc176c0d:::benchmark_data_smithi159_139342_object34877:head# 2024-04-15T18:35:25.050 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc187f31:::benchmark_data_smithi159_139342_object15089:head# 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc19e32f:::benchmark_data_smithi159_139342_object49133:head# 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1af9d1:::benchmark_data_smithi159_139342_object40473:head# 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1b32b1:::benchmark_data_smithi159_139342_object43387:head# 2024-04-15T18:35:25.051 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.052 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1c4dee:::benchmark_data_smithi159_139342_object49269:head# 2024-04-15T18:35:25.052 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.052 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1c8025:::benchmark_data_smithi159_139342_object2777:head# 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1cc3bf:::benchmark_data_smithi159_139342_object19760:head# 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1e0c0f:::benchmark_data_smithi159_139342_object18212:head# 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1f5a57:::benchmark_data_smithi159_139342_object20292:head# 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc1fb540:::benchmark_data_smithi159_139342_object8471:head# 2024-04-15T18:35:25.053 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.053 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc20193a:::benchmark_data_smithi159_139342_object50230:head# 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc209053:::benchmark_data_smithi159_139342_object45418:head# 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2102aa:::benchmark_data_smithi159_139342_object28093:head# 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc236703:::benchmark_data_smithi159_139342_object7359:head# 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.055 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc23a371:::benchmark_data_smithi159_139342_object57265:head# 2024-04-15T18:35:25.056 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.056 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2494e9:::benchmark_data_smithi159_139342_object36347:head# 2024-04-15T18:35:25.056 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.056 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2531ba:::benchmark_data_smithi159_139342_object40365:head# 2024-04-15T18:35:25.056 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.057 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc278ab3:::benchmark_data_smithi159_139342_object6217:head# 2024-04-15T18:35:25.057 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.057 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc280202:::benchmark_data_smithi159_139342_object40369:head# 2024-04-15T18:35:25.057 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.058 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc29c2ab:::benchmark_data_smithi159_139342_object35251:head# 2024-04-15T18:35:25.058 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.061 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2a6e13:::benchmark_data_smithi159_139342_object38127:head# 2024-04-15T18:35:25.061 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2acb12:::benchmark_data_smithi159_139342_object21510:head# 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc2d6046:::benchmark_data_smithi159_139342_object28759:head# 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc30a879:::benchmark_data_smithi159_139342_object1196:head# 2024-04-15T18:35:25.062 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc31577a:::benchmark_data_smithi159_139342_object28813:head# 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc31a191:::benchmark_data_smithi159_139342_object6095:head# 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc32c10a:::benchmark_data_smithi159_139342_object39380:head# 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.063 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc356bd7:::benchmark_data_smithi159_139342_object53373:head# 2024-04-15T18:35:25.064 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc35c103:::benchmark_data_smithi159_139342_object48054:head# 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc35df15:::benchmark_data_smithi159_139342_object28613:head# 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc363d44:::benchmark_data_smithi159_139342_object11248:head# 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc37b4e8:::benchmark_data_smithi159_139342_object29260:head# 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc395cd6:::benchmark_data_smithi159_139342_object2810:head# 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.067 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc3c67f2:::benchmark_data_smithi159_139342_object56558:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc3c68ae:::benchmark_data_smithi159_139342_object13405:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc3e53ce:::benchmark_data_smithi159_139342_object55510:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc3f5489:::benchmark_data_smithi159_139342_object1902:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc402fde:::benchmark_data_smithi159_139342_object5182:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4031c3:::benchmark_data_smithi159_139342_object49135:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc408cf4:::benchmark_data_smithi159_139342_object9420:head# 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.068 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4114c2:::benchmark_data_smithi159_139342_object35889:head# 2024-04-15T18:35:25.069 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.069 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4118d1:::benchmark_data_smithi159_139342_object20461:head# 2024-04-15T18:35:25.069 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.070 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc414a22:::benchmark_data_smithi159_139342_object13347:head# 2024-04-15T18:35:25.070 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.070 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc41f90c:::benchmark_data_smithi159_139342_object33878:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc42cc81:::benchmark_data_smithi159_139342_object53228:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc438983:::benchmark_data_smithi159_139342_object41980:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc473e91:::benchmark_data_smithi159_139342_object46079:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc47dd7a:::benchmark_data_smithi159_139342_object50444:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc48c777:::benchmark_data_smithi159_139342_object24789:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4af7c4:::benchmark_data_smithi159_139342_object26006:head# 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.071 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4b814f:::benchmark_data_smithi159_139342_object4078:head# 2024-04-15T18:35:25.072 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.074 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc4cd124:::benchmark_data_smithi159_139342_object24189:head# 2024-04-15T18:35:25.074 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.074 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc51bb23:::benchmark_data_smithi159_139342_object29265:head# 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc532854:::benchmark_data_smithi159_139342_object39916:head# 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5357b2:::benchmark_data_smithi159_139342_object38031:head# 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc53907a:::benchmark_data_smithi159_139342_object26755:head# 2024-04-15T18:35:25.075 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.078 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc545f73:::benchmark_data_smithi159_139342_object39910:head# 2024-04-15T18:35:25.078 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.080 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc581c10:::benchmark_data_smithi159_139342_object25064:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc58824d:::benchmark_data_smithi159_139342_object12326:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5a05c8:::benchmark_data_smithi159_139342_object5287:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5b761e:::benchmark_data_smithi159_139342_object1155:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5ca2ef:::benchmark_data_smithi159_139342_object5890:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5cd459:::benchmark_data_smithi159_139342_object13643:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5d5c7b:::benchmark_data_smithi159_139342_object36807:head# 2024-04-15T18:35:25.081 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.082 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5eec2a:::benchmark_data_smithi159_139342_object34866:head# 2024-04-15T18:35:25.082 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.082 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc5f7dc3:::benchmark_data_smithi159_139342_object21850:head# 2024-04-15T18:35:25.082 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc63adaf:::benchmark_data_smithi159_139342_object29248:head# 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc66a4be:::benchmark_data_smithi159_139342_object34659:head# 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc66fc9d:::benchmark_data_smithi159_139342_object50277:head# 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc67e4ce:::benchmark_data_smithi159_139342_object26562:head# 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc68d6f2:::benchmark_data_smithi159_139342_object21251:head# 2024-04-15T18:35:25.084 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6901f3:::benchmark_data_smithi159_139342_object18276:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6c57e1:::benchmark_data_smithi159_139342_object54773:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6ca681:::benchmark_data_smithi159_139342_object13777:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6ddb8c:::benchmark_data_smithi159_139342_object28323:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6f5931:::benchmark_data_smithi159_139342_object26989:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc6f9123:::benchmark_data_smithi159_139342_object49791:head# 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc707c8b:::benchmark_data_smithi159_139342_object47419:head# 2024-04-15T18:35:25.086 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:24 smithi159 ceph-mon[45336]: pgmap v1552: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 8344/115944 objects degraded (7.197%); 16 MiB/s, 248 objects/s recovering 2024-04-15T18:35:25.087 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc72ef9a:::benchmark_data_smithi159_139342_object53752:head# 2024-04-15T18:35:25.087 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.087 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7756bc:::benchmark_data_smithi159_139342_object36105:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc781200:::benchmark_data_smithi159_139342_object2731:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7acc2d:::benchmark_data_smithi159_139342_object52561:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7bcacb:::benchmark_data_smithi159_139342_object53759:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7d17f5:::benchmark_data_smithi159_139342_object46824:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7d6350:::benchmark_data_smithi159_139342_object30615:head# 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.088 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7e426d:::benchmark_data_smithi159_139342_object33752:head# 2024-04-15T18:35:25.089 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.089 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.094 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7e942f:::benchmark_data_smithi159_139342_object14972:head# 2024-04-15T18:35:25.094 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.096 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc7feda2:::benchmark_data_smithi159_139342_object2187:head# 2024-04-15T18:35:25.096 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.100 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8272ef:::benchmark_data_smithi159_139342_object47475:head# 2024-04-15T18:35:25.100 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.100 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc84a1f2:::benchmark_data_smithi159_139342_object41289:head# 2024-04-15T18:35:25.100 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.100 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc84bc74:::benchmark_data_smithi159_139342_object51041:head# 2024-04-15T18:35:25.101 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.101 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8624d6:::benchmark_data_smithi159_139342_object38544:head# 2024-04-15T18:35:25.101 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.101 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc87ed1b:::benchmark_data_smithi159_139342_object38716:head# 2024-04-15T18:35:25.101 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc890822:::benchmark_data_smithi159_139342_object13538:head# 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8a4f19:::benchmark_data_smithi159_139342_object4288:head# 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8b4dc1:::benchmark_data_smithi159_139342_object1533:head# 2024-04-15T18:35:25.105 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8d67a7:::benchmark_data_smithi159_139342_object8835:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8f2353:::benchmark_data_smithi159_139342_object9702:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8f3512:::benchmark_data_smithi159_139342_object33042:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8f6942:::benchmark_data_smithi159_139342_object38059:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc8faaa3:::benchmark_data_smithi159_139342_object23134:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc902789:::benchmark_data_smithi159_139342_object33728:head# 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.106 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc906041:::benchmark_data_smithi159_139342_object45012:head# 2024-04-15T18:35:25.107 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc90ca9a:::benchmark_data_smithi159_139342_object21240:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9175d1:::benchmark_data_smithi159_139342_object55651:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc91cfe0:::benchmark_data_smithi159_139342_object9916:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc91e38d:::benchmark_data_smithi159_139342_object33559:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc930cea:::benchmark_data_smithi159_139342_object25466:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9320ee:::benchmark_data_smithi159_139342_object14064:head# 2024-04-15T18:35:25.108 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.109 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc95d3b6:::benchmark_data_smithi159_139342_object14003:head# 2024-04-15T18:35:25.109 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc96beb7:::benchmark_data_smithi159_139342_object38089:head# 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc994c96:::benchmark_data_smithi159_139342_object10266:head# 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9a11b7:::benchmark_data_smithi159_139342_object27522:head# 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9b83bb:::benchmark_data_smithi159_139342_object18299:head# 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9bb2b2:::benchmark_data_smithi159_139342_object6072:head# 2024-04-15T18:35:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9be1b8:::benchmark_data_smithi159_139342_object15985:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9c91c1:::benchmark_data_smithi159_139342_object15824:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9d7cf7:::benchmark_data_smithi159_139342_object3669:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9e1b8d:::benchmark_data_smithi159_139342_object44042:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9ea4a6:::benchmark_data_smithi159_139342_object23919:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cc9f1880:::benchmark_data_smithi159_139342_object16364:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca03f0f:::benchmark_data_smithi159_139342_object36177:head# 2024-04-15T18:35:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.114 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca1accf:::benchmark_data_smithi159_139342_object47466:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca36c03:::benchmark_data_smithi159_139342_object36701:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca46b61:::benchmark_data_smithi159_139342_object977:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca56360:::benchmark_data_smithi159_139342_object24059:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca56aa0:::benchmark_data_smithi159_139342_object35052:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca64b62:::benchmark_data_smithi159_139342_object41833:head# 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca69be2:::benchmark_data_smithi159_139342_object24807:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca702fa:::benchmark_data_smithi159_139342_object4168:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca83bd8:::benchmark_data_smithi159_139342_object42206:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cca86e80:::benchmark_data_smithi159_139342_object53237:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccabf0c8:::benchmark_data_smithi159_139342_object9868:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb07abf:::benchmark_data_smithi159_139342_object54819:head# 2024-04-15T18:35:25.116 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.117 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb143dd:::benchmark_data_smithi159_139342_object17710:head# 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb23c46:::benchmark_data_smithi159_139342_object2185:head# 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb39db6:::benchmark_data_smithi159_139342_object31042:head# 2024-04-15T18:35:25.118 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:25.123 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb47245:::benchmark_data_smithi159_139342_object40175:head# 2024-04-15T18:35:25.123 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.124 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb53343:::benchmark_data_smithi159_139342_object7813:head# 2024-04-15T18:35:25.124 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.125 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb55b05:::benchmark_data_smithi159_139342_object13905:head# 2024-04-15T18:35:25.125 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.125 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb5a368:::benchmark_data_smithi159_139342_object35921:head# 2024-04-15T18:35:25.125 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.125 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb60804:::benchmark_data_smithi159_139342_object46293:head# 2024-04-15T18:35:25.126 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.126 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb7956f:::benchmark_data_smithi159_139342_object46117:head# 2024-04-15T18:35:25.126 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccb7b8af:::benchmark_data_smithi159_139342_object27927:head# 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccbceddf:::benchmark_data_smithi159_139342_object33836:head# 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccbfca3b:::benchmark_data_smithi159_139342_object20275:head# 2024-04-15T18:35:25.132 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc082cf:::benchmark_data_smithi159_139342_object4439:head# 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc08d3c:::benchmark_data_smithi159_139342_object55715:head# 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc0f687:::benchmark_data_smithi159_139342_object23971:head# 2024-04-15T18:35:25.133 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc43f30:::benchmark_data_smithi159_139342_object21902:head# 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc47019:::benchmark_data_smithi159_139342_object19100:head# 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc52cb5:::benchmark_data_smithi159_139342_object1761:head# 2024-04-15T18:35:25.134 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.143 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc6fc7e:::benchmark_data_smithi159_139342_object42473:head# 2024-04-15T18:35:25.143 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.143 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccc94393:::benchmark_data_smithi159_139342_object37624:head# 2024-04-15T18:35:25.143 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.143 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cccb63ff:::benchmark_data_smithi159_139342_object31814:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cccda18b:::benchmark_data_smithi159_139342_object19347:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cccdc40e:::benchmark_data_smithi159_139342_object52713:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccce1f19:::benchmark_data_smithi159_139342_object55999:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccce35b2:::benchmark_data_smithi159_139342_object43300:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd07903:::benchmark_data_smithi159_139342_object56389:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd0c523:::benchmark_data_smithi159_139342_object41867:head# 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.144 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd273e2:::benchmark_data_smithi159_139342_object47966:head# 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd671ad:::benchmark_data_smithi159_139342_object8778:head# 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd861e0:::benchmark_data_smithi159_139342_object44322:head# 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd8f846:::benchmark_data_smithi159_139342_object31645:head# 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccd917e5:::benchmark_data_smithi159_139342_object29899:head# 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.145 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccdba03f:::benchmark_data_smithi159_139342_object37980:head# 2024-04-15T18:35:25.146 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.149 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccdcc1f3:::benchmark_data_smithi159_139342_object45683:head# 2024-04-15T18:35:25.149 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.154 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce16fd0:::benchmark_data_smithi159_139342_object42209:head# 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce37433:::benchmark_data_smithi159_139342_object22054:head# 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce56857:::benchmark_data_smithi159_139342_object28884:head# 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce57585:::benchmark_data_smithi159_139342_object1211:head# 2024-04-15T18:35:25.155 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.157 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce6ffd7:::benchmark_data_smithi159_139342_object16850:head# 2024-04-15T18:35:25.157 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce7d19a:::benchmark_data_smithi159_139342_object46696:head# 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce80a2c:::benchmark_data_smithi159_139342_object36136:head# 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce8aa73:::benchmark_data_smithi159_139342_object50189:head# 2024-04-15T18:35:25.158 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cce9e243:::benchmark_data_smithi159_139342_object6540:head# 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccea0071:::benchmark_data_smithi159_139342_object7079:head# 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccead6b9:::benchmark_data_smithi159_139342_object42078:head# 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.159 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cceb1846:::benchmark_data_smithi159_139342_object45655:head# 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cced6a47:::benchmark_data_smithi159_139342_object21412:head# 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccee1c5f:::benchmark_data_smithi159_139342_object6193:head# 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.160 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf1a961:::benchmark_data_smithi159_139342_object51669:head# 2024-04-15T18:35:25.161 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.161 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf28f7d:::benchmark_data_smithi159_139342_object10693:head# 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf3c1ad:::benchmark_data_smithi159_139342_object47974:head# 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf4436f:::benchmark_data_smithi159_139342_object7061:head# 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf55d2e:::benchmark_data_smithi159_139342_object9370:head# 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf794fc:::benchmark_data_smithi159_139342_object44251:head# 2024-04-15T18:35:25.162 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.170 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccf8a014:::benchmark_data_smithi159_139342_object43627:head# 2024-04-15T18:35:25.170 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.170 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccfaff2e:::benchmark_data_smithi159_139342_object39861:head# 2024-04-15T18:35:25.170 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.171 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ccfe2f9f:::benchmark_data_smithi159_139342_object46937:head# 2024-04-15T18:35:25.171 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.171 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd00d2ee:::benchmark_data_smithi159_139342_object14024:head# 2024-04-15T18:35:25.171 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd014f3c:::benchmark_data_smithi159_139342_object12942:head# 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd02558a:::benchmark_data_smithi159_139342_object51951:head# 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd028bc6:::benchmark_data_smithi159_139342_object800:head# 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd030f6e:::benchmark_data_smithi159_139342_object56875:head# 2024-04-15T18:35:25.172 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd041ff0:::benchmark_data_smithi159_139342_object50654:head# 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd050da1:::benchmark_data_smithi159_139342_object43733:head# 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd093d8b:::benchmark_data_smithi159_139342_object19355:head# 2024-04-15T18:35:25.173 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.180 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd0940df:::benchmark_data_smithi159_139342_object26609:head# 2024-04-15T18:35:25.180 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.180 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd09efbb:::benchmark_data_smithi159_139342_object55426:head# 2024-04-15T18:35:25.180 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.181 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd0d1261:::benchmark_data_smithi159_139342_object54431:head# 2024-04-15T18:35:25.181 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.181 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd0f24b6:::benchmark_data_smithi159_139342_object46244:head# 2024-04-15T18:35:25.181 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd0f7aea:::benchmark_data_smithi159_139342_object52145:head# 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd119b7c:::benchmark_data_smithi159_139342_object51841:head# 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd11d6fd:::benchmark_data_smithi159_139342_object51585:head# 2024-04-15T18:35:25.182 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd11d831:::benchmark_data_smithi159_139342_object47386:head# 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd12418c:::benchmark_data_smithi159_139342_object36081:head# 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd12e521:::benchmark_data_smithi159_139342_object50511:head# 2024-04-15T18:35:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.184 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd12e9ca:::benchmark_data_smithi159_139342_object53958:head# 2024-04-15T18:35:25.184 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.184 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd13f796:::benchmark_data_smithi159_139342_object25642:head# 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd159a21:::benchmark_data_smithi159_139342_object236:head# 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd15f319:::benchmark_data_smithi159_139342_object15435:head# 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd168592:::benchmark_data_smithi159_139342_object43234:head# 2024-04-15T18:35:25.185 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.186 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd17401d:::benchmark_data_smithi159_139342_object5651:head# 2024-04-15T18:35:25.186 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.186 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1754e9:::benchmark_data_smithi159_139342_object40789:head# 2024-04-15T18:35:25.186 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd17d6eb:::benchmark_data_smithi159_139342_object24303:head# 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd198ef8:::benchmark_data_smithi159_139342_object24302:head# 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd19e956:::benchmark_data_smithi159_139342_object1031:head# 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1b1d26:::benchmark_data_smithi159_139342_object167:head# 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1b6645:::benchmark_data_smithi159_139342_object28520:head# 2024-04-15T18:35:25.187 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1cdc2e:::benchmark_data_smithi159_139342_object10809:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1d04c5:::benchmark_data_smithi159_139342_object42378:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd1d27ca:::benchmark_data_smithi159_139342_object10821:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd2292b5:::benchmark_data_smithi159_139342_object30785:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd22c901:::benchmark_data_smithi159_139342_object35373:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd24308c:::benchmark_data_smithi159_139342_object27796:head# 2024-04-15T18:35:25.189 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd250af8:::benchmark_data_smithi159_139342_object3773:head# 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd275afe:::benchmark_data_smithi159_139342_object4504:head# 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd28b494:::benchmark_data_smithi159_139342_object10527:head# 2024-04-15T18:35:25.190 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd2a0700:::benchmark_data_smithi159_139342_object23838:head# 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd2a243a:::benchmark_data_smithi159_139342_object26264:head# 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd2a7f23:::benchmark_data_smithi159_139342_object54304:head# 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd2c15ca:::benchmark_data_smithi159_139342_object15632:head# 2024-04-15T18:35:25.191 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.195 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd32a28c:::benchmark_data_smithi159_139342_object40895:head# 2024-04-15T18:35:25.195 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.195 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd331a25:::benchmark_data_smithi159_139342_object25805:head# 2024-04-15T18:35:25.195 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.196 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3457f0:::benchmark_data_smithi159_139342_object10741:head# 2024-04-15T18:35:25.196 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.196 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd347da0:::benchmark_data_smithi159_139342_object20484:head# 2024-04-15T18:35:25.196 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.197 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd350686:::benchmark_data_smithi159_139342_object46499:head# 2024-04-15T18:35:25.197 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.197 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd36cedf:::benchmark_data_smithi159_139342_object45814:head# 2024-04-15T18:35:25.197 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.198 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3a629a:::benchmark_data_smithi159_139342_object45884:head# 2024-04-15T18:35:25.198 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.198 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3a6ea5:::benchmark_data_smithi159_139342_object12507:head# 2024-04-15T18:35:25.198 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.199 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3af870:::benchmark_data_smithi159_139342_object48681:head# 2024-04-15T18:35:25.199 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.199 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3ea9f8:::benchmark_data_smithi159_139342_object41343:head# 2024-04-15T18:35:25.199 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd3ff58e:::benchmark_data_smithi159_139342_object50060:head# 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd40b5a3:::benchmark_data_smithi159_139342_object15929:head# 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd45189d:::benchmark_data_smithi159_139342_object28907:head# 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd477295:::benchmark_data_smithi159_139342_object44670:head# 2024-04-15T18:35:25.200 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.201 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd483574:::benchmark_data_smithi159_139342_object29324:head# 2024-04-15T18:35:25.201 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.201 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd48a0b3:::benchmark_data_smithi159_139342_object20286:head# 2024-04-15T18:35:25.201 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.206 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd4bb763:::benchmark_data_smithi159_139342_object2949:head# 2024-04-15T18:35:25.206 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd4c49c8:::benchmark_data_smithi159_139342_object18285:head# 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd4dff85:::benchmark_data_smithi159_139342_object48622:head# 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd52b300:::benchmark_data_smithi159_139342_object1185:head# 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5341bc:::benchmark_data_smithi159_139342_object23230:head# 2024-04-15T18:35:25.211 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.212 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd546a2b:::benchmark_data_smithi159_139342_object3801:head# 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd55257b:::benchmark_data_smithi159_139342_object30290:head# 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd557cb7:::benchmark_data_smithi159_139342_object56674:head# 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd56680e:::benchmark_data_smithi159_139342_object52409:head# 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd581bf1:::benchmark_data_smithi159_139342_object19490:head# 2024-04-15T18:35:25.213 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd588692:::benchmark_data_smithi159_139342_object12013:head# 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd58a373:::benchmark_data_smithi159_139342_object24376:head# 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd59fb0b:::benchmark_data_smithi159_139342_object12470:head# 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.214 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5aa755:::benchmark_data_smithi159_139342_object29025:head# 2024-04-15T18:35:25.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5ace1a:::benchmark_data_smithi159_139342_object44911:head# 2024-04-15T18:35:25.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.215 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5d768f:::benchmark_data_smithi159_139342_object5439:head# 2024-04-15T18:35:25.215 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5dafd8:::benchmark_data_smithi159_139342_object35203:head# 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd5e92b9:::benchmark_data_smithi159_139342_object55405:head# 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6002f1:::benchmark_data_smithi159_139342_object52668:head# 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd65eaa6:::benchmark_data_smithi159_139342_object20240:head# 2024-04-15T18:35:25.216 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6627de:::benchmark_data_smithi159_139342_object47953:head# 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd668cc5:::benchmark_data_smithi159_139342_object37430:head# 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6900d2:::benchmark_data_smithi159_139342_object28964:head# 2024-04-15T18:35:25.217 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6b9090:::benchmark_data_smithi159_139342_object37425:head# 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6be731:::benchmark_data_smithi159_139342_object53625:head# 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6c856b:::benchmark_data_smithi159_139342_object39298:head# 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd6f21fb:::benchmark_data_smithi159_139342_object32571:head# 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd72233d:::benchmark_data_smithi159_139342_object15568:head# 2024-04-15T18:35:25.219 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd72c256:::benchmark_data_smithi159_139342_object4355:head# 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd73197a:::benchmark_data_smithi159_139342_object49673:head# 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd743e0d:::benchmark_data_smithi159_139342_object28427:head# 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd745ddc:::benchmark_data_smithi159_139342_object16454:head# 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7539ee:::benchmark_data_smithi159_139342_object10577:head# 2024-04-15T18:35:25.220 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd77d481:::benchmark_data_smithi159_139342_object27484:head# 2024-04-15T18:35:25.222 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.222 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd78303c:::benchmark_data_smithi159_139342_object21313:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7a2696:::benchmark_data_smithi159_139342_object51206:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7a32b7:::benchmark_data_smithi159_139342_object45410:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7aa218:::benchmark_data_smithi159_139342_object40459:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7b7666:::benchmark_data_smithi159_139342_object37643:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7c827c:::benchmark_data_smithi159_139342_object40438:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7d36bb:::benchmark_data_smithi159_139342_object41121:head# 2024-04-15T18:35:25.223 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7dcaba:::benchmark_data_smithi159_139342_object26299:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7e5e7d:::benchmark_data_smithi159_139342_object35383:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7f2e0d:::benchmark_data_smithi159_139342_object22908:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd7f6e1c:::benchmark_data_smithi159_139342_object43697:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd807431:::benchmark_data_smithi159_139342_object22231:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd824406:::benchmark_data_smithi159_139342_object18718:head# 2024-04-15T18:35:25.224 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:25.225 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd83530e:::benchmark_data_smithi159_139342_object50016:head# 2024-04-15T18:35:25.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd84c173:::benchmark_data_smithi159_139342_object26703:head# 2024-04-15T18:35:25.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.226 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd867ee6:::benchmark_data_smithi159_139342_object1939:head# 2024-04-15T18:35:25.226 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.227 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd86cf8a:::benchmark_data_smithi159_139342_object32584:head# 2024-04-15T18:35:25.227 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.232 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd87e30e:::benchmark_data_smithi159_139342_object21982:head# 2024-04-15T18:35:25.232 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.244 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd882a25:::benchmark_data_smithi159_139342_object30030:head# 2024-04-15T18:35:25.244 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.245 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd894baa:::benchmark_data_smithi159_139342_object4019:head# 2024-04-15T18:35:25.245 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.247 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd895e89:::benchmark_data_smithi159_139342_object36358:head# 2024-04-15T18:35:25.247 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd8c32a9:::benchmark_data_smithi159_139342_object4243:head# 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd8cb42b:::benchmark_data_smithi159_139342_object21311:head# 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd8d8c3f:::benchmark_data_smithi159_139342_object27857:head# 2024-04-15T18:35:25.248 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd90de8a:::benchmark_data_smithi159_139342_object48808:head# 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd914eac:::benchmark_data_smithi159_139342_object56113:head# 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd92f30b:::benchmark_data_smithi159_139342_object32565:head# 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd937300:::benchmark_data_smithi159_139342_object34312:head# 2024-04-15T18:35:25.249 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd93ad87:::benchmark_data_smithi159_139342_object20149:head# 2024-04-15T18:35:25.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.250 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd93e72c:::benchmark_data_smithi159_139342_object56404:head# 2024-04-15T18:35:25.250 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.254 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd94cb3f:::benchmark_data_smithi159_139342_object36212:head# 2024-04-15T18:35:25.254 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd95dd24:::benchmark_data_smithi159_139342_object39871:head# 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd96403d:::benchmark_data_smithi159_139342_object55968:head# 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd973afc:::benchmark_data_smithi159_139342_object30622:head# 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd985301:::benchmark_data_smithi159_139342_object17328:head# 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.256 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd990d8a:::benchmark_data_smithi159_139342_object11918:head# 2024-04-15T18:35:25.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.257 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9a7943:::benchmark_data_smithi159_139342_object32209:head# 2024-04-15T18:35:25.257 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9d8a9e:::benchmark_data_smithi159_139342_object36159:head# 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9d90cc:::benchmark_data_smithi159_139342_object49802:head# 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9edb70:::benchmark_data_smithi159_139342_object7629:head# 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9f056f:::benchmark_data_smithi159_139342_object44331:head# 2024-04-15T18:35:25.258 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.262 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.263 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cd9f09d5:::benchmark_data_smithi159_139342_object24740:head# 2024-04-15T18:35:25.263 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda33ba5:::benchmark_data_smithi159_139342_object3203:head# 2024-04-15T18:35:25.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.264 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda421b0:::benchmark_data_smithi159_139342_object38063:head# 2024-04-15T18:35:25.264 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.265 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda63727:::benchmark_data_smithi159_139342_object314:head# 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda6e41e:::benchmark_data_smithi159_139342_object13954:head# 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda92ab5:::benchmark_data_smithi159_139342_object31873:head# 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cda9675f:::benchmark_data_smithi159_139342_object15829:head# 2024-04-15T18:35:25.266 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdac8843:::benchmark_data_smithi159_139342_object9418:head# 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdacfe87:::benchmark_data_smithi159_139342_object12795:head# 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdad50f7:::benchmark_data_smithi159_139342_object37948:head# 2024-04-15T18:35:25.267 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdae0fd2:::benchmark_data_smithi159_139342_object34795:head# 2024-04-15T18:35:25.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.272 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdaee15c:::benchmark_data_smithi159_139342_object53885:head# 2024-04-15T18:35:25.272 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb0b6df:::benchmark_data_smithi159_139342_object39537:head# 2024-04-15T18:35:25.276 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.276 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb0f1bd:::benchmark_data_smithi159_139342_object2921:head# 2024-04-15T18:35:25.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.277 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb11b6a:::benchmark_data_smithi159_139342_object5488:head# 2024-04-15T18:35:25.277 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb11f15:::benchmark_data_smithi159_139342_object28509:head# 2024-04-15T18:35:25.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb1245a:::benchmark_data_smithi159_139342_object7935:head# 2024-04-15T18:35:25.278 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.278 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb1cde7:::benchmark_data_smithi159_139342_object22004:head# 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdb27a5d:::benchmark_data_smithi159_139342_object31623:head# 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdba0b95:::benchmark_data_smithi159_139342_object8806:head# 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdbaf754:::benchmark_data_smithi159_139342_object7697:head# 2024-04-15T18:35:25.279 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.280 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdbc7d3d:::benchmark_data_smithi159_139342_object16589:head# 2024-04-15T18:35:25.280 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdbd15bb:::benchmark_data_smithi159_139342_object10998:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdbd8a1a:::benchmark_data_smithi159_139342_object16899:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdbec96e:::benchmark_data_smithi159_139342_object1304:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc216bf:::benchmark_data_smithi159_139342_object28127:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc29242:::benchmark_data_smithi159_139342_object53088:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc3db3a:::benchmark_data_smithi159_139342_object49721:head# 2024-04-15T18:35:25.281 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc42519:::benchmark_data_smithi159_139342_object22778:head# 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc43f39:::benchmark_data_smithi159_139342_object19620:head# 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc470b2:::benchmark_data_smithi159_139342_object16560:head# 2024-04-15T18:35:25.282 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc51072:::benchmark_data_smithi159_139342_object42858:head# 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc6a90e:::benchmark_data_smithi159_139342_object11542:head# 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc71427:::benchmark_data_smithi159_139342_object38359:head# 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc8c79a:::benchmark_data_smithi159_139342_object18477:head# 2024-04-15T18:35:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdc94ba6:::benchmark_data_smithi159_139342_object1125:head# 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdcb41e8:::benchmark_data_smithi159_139342_object42969:head# 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdce4169:::benchmark_data_smithi159_139342_object19446:head# 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.284 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdceb9a2:::benchmark_data_smithi159_139342_object15873:head# 2024-04-15T18:35:25.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.285 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdcedb30:::benchmark_data_smithi159_139342_object39101:head# 2024-04-15T18:35:25.285 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdcf0f88:::benchmark_data_smithi159_139342_object11513:head# 2024-04-15T18:35:25.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.289 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd207b1:::benchmark_data_smithi159_139342_object49377:head# 2024-04-15T18:35:25.289 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.292 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:25.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd25121:::benchmark_data_smithi159_139342_object18932:head# 2024-04-15T18:35:25.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.293 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd4643d:::benchmark_data_smithi159_139342_object10846:head# 2024-04-15T18:35:25.293 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd50cb5:::benchmark_data_smithi159_139342_object48201:head# 2024-04-15T18:35:25.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.294 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd51003:::benchmark_data_smithi159_139342_object19172:head# 2024-04-15T18:35:25.294 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd5627e:::benchmark_data_smithi159_139342_object14200:head# 2024-04-15T18:35:25.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd99ae2:::benchmark_data_smithi159_139342_object4008:head# 2024-04-15T18:35:25.297 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.297 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdd9d42c:::benchmark_data_smithi159_139342_object6236:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdda01a6:::benchmark_data_smithi159_139342_object34786:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cddc9969:::benchmark_data_smithi159_139342_object30933:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cddca648:::benchmark_data_smithi159_139342_object14286:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cddf3c3c:::benchmark_data_smithi159_139342_object31538:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cddfa985:::benchmark_data_smithi159_139342_object33912:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde0c64a:::benchmark_data_smithi159_139342_object1948:head# 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.298 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde2c207:::benchmark_data_smithi159_139342_object18524:head# 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde2f3e1:::benchmark_data_smithi159_139342_object47782:head# 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde3b901:::benchmark_data_smithi159_139342_object16855:head# 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde6b896:::benchmark_data_smithi159_139342_object49331:head# 2024-04-15T18:35:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde74fe7:::benchmark_data_smithi159_139342_object3854:head# 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde860a6:::benchmark_data_smithi159_139342_object12667:head# 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cde9b441:::benchmark_data_smithi159_139342_object51071:head# 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cded046f:::benchmark_data_smithi159_139342_object22477:head# 2024-04-15T18:35:25.302 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cded9755:::benchmark_data_smithi159_139342_object11217:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdeebf41:::benchmark_data_smithi159_139342_object35362:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdef0f09:::benchmark_data_smithi159_139342_object32071:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf3257f:::benchmark_data_smithi159_139342_object23917:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf3ab1c:::benchmark_data_smithi159_139342_object39618:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf5eadc:::benchmark_data_smithi159_139342_object17828:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf65834:::benchmark_data_smithi159_139342_object35851:head# 2024-04-15T18:35:25.303 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf683e4:::benchmark_data_smithi159_139342_object47591:head# 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf6eccd:::benchmark_data_smithi159_139342_object50980:head# 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf6f29c:::benchmark_data_smithi159_139342_object26073:head# 2024-04-15T18:35:25.304 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf73ad6:::benchmark_data_smithi159_139342_object1392:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf77e5d:::benchmark_data_smithi159_139342_object29322:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf7df41:::benchmark_data_smithi159_139342_object50596:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf7f5a2:::benchmark_data_smithi159_139342_object10696:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf83929:::benchmark_data_smithi159_139342_object13530:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdf86002:::benchmark_data_smithi159_139342_object5590:head# 2024-04-15T18:35:25.306 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdfb50a6:::benchmark_data_smithi159_139342_object55556:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdfd29d9:::benchmark_data_smithi159_139342_object54385:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdfd335a:::benchmark_data_smithi159_139342_object23302:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdfe571c:::benchmark_data_smithi159_139342_object56865:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cdffd749:::benchmark_data_smithi159_139342_object678:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce000f0a:::benchmark_data_smithi159_139342_object33527:head# 2024-04-15T18:35:25.307 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0016c7:::benchmark_data_smithi159_139342_object10278:head# 2024-04-15T18:35:25.308 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.308 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce002b8b:::benchmark_data_smithi159_139342_object45586:head# 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0072d4:::benchmark_data_smithi159_139342_object45669:head# 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce029a11:::benchmark_data_smithi159_139342_object2275:head# 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0321f8:::benchmark_data_smithi159_139342_object36673:head# 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce040ae4:::benchmark_data_smithi159_139342_object37223:head# 2024-04-15T18:35:25.309 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce04bf81:::benchmark_data_smithi159_139342_object7827:head# 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce078093:::benchmark_data_smithi159_139342_object38682:head# 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce09d55d:::benchmark_data_smithi159_139342_object9458:head# 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0a4f95:::benchmark_data_smithi159_139342_object248:head# 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.312 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0cca95:::benchmark_data_smithi159_139342_object28227:head# 2024-04-15T18:35:25.313 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.313 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0ce361:::benchmark_data_smithi159_139342_object16774:head# 2024-04-15T18:35:25.313 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.313 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0dea16:::benchmark_data_smithi159_139342_object50860:head# 2024-04-15T18:35:25.313 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.314 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce0f5283:::benchmark_data_smithi159_139342_object41711:head# 2024-04-15T18:35:25.314 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.314 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce12b469:::benchmark_data_smithi159_139342_object11812:head# 2024-04-15T18:35:25.314 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.319 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce13300d:::benchmark_data_smithi159_139342_object6156:head# 2024-04-15T18:35:25.319 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.319 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce13990c:::benchmark_data_smithi159_139342_object1402:head# 2024-04-15T18:35:25.319 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.320 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce176ddb:::benchmark_data_smithi159_139342_object5601:head# 2024-04-15T18:35:25.320 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.320 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce195639:::benchmark_data_smithi159_139342_object29262:head# 2024-04-15T18:35:25.320 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce1a2327:::benchmark_data_smithi159_139342_object23141:head# 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce1a4d4d:::benchmark_data_smithi159_139342_object26480:head# 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce1b47f5:::benchmark_data_smithi159_139342_object40003:head# 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce1e189b:::benchmark_data_smithi159_139342_object53168:head# 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce1ebaaf:::benchmark_data_smithi159_139342_object24918:head# 2024-04-15T18:35:25.321 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce202c9c:::benchmark_data_smithi159_139342_object15763:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce22aebb:::benchmark_data_smithi159_139342_object40097:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce262a68:::benchmark_data_smithi159_139342_object6961:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce279872:::benchmark_data_smithi159_139342_object33064:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce27d5ad:::benchmark_data_smithi159_139342_object47833:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce28afd3:::benchmark_data_smithi159_139342_object9322:head# 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.323 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce293320:::benchmark_data_smithi159_139342_object48802:head# 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce2beb32:::benchmark_data_smithi159_139342_object5947:head# 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce2c1b75:::benchmark_data_smithi159_139342_object48278:head# 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.324 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce2e16f1:::benchmark_data_smithi159_139342_object21334:head# 2024-04-15T18:35:25.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce2f79c8:::benchmark_data_smithi159_139342_object43053:head# 2024-04-15T18:35:25.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.325 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce31c56a:::benchmark_data_smithi159_139342_object44867:head# 2024-04-15T18:35:25.325 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.326 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce32f5a5:::benchmark_data_smithi159_139342_object6449:head# 2024-04-15T18:35:25.326 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce330932:::benchmark_data_smithi159_139342_object41384:head# 2024-04-15T18:35:25.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.327 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3463a6:::benchmark_data_smithi159_139342_object37478:head# 2024-04-15T18:35:25.327 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce358765:::benchmark_data_smithi159_139342_object24848:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3692b0:::benchmark_data_smithi159_139342_object1315:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3697e8:::benchmark_data_smithi159_139342_object30946:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce378907:::benchmark_data_smithi159_139342_object26652:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3a6954:::benchmark_data_smithi159_139342_object46125:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3a95eb:::benchmark_data_smithi159_139342_object12620:head# 2024-04-15T18:35:25.328 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.329 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3b8be9:::benchmark_data_smithi159_139342_object25513:head# 2024-04-15T18:35:25.329 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3dcfe7:::benchmark_data_smithi159_139342_object31052:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3e31a2:::benchmark_data_smithi159_139342_object50726:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3f995a:::benchmark_data_smithi159_139342_object7532:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce3fe1be:::benchmark_data_smithi159_139342_object55007:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce422433:::benchmark_data_smithi159_139342_object13673:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce42c3b4:::benchmark_data_smithi159_139342_object45046:head# 2024-04-15T18:35:25.335 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce43fa11:::benchmark_data_smithi159_139342_object17387:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce449553:::benchmark_data_smithi159_139342_object11521:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce46fee6:::benchmark_data_smithi159_139342_object41342:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce47ca5e:::benchmark_data_smithi159_139342_object55969:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce48064f:::benchmark_data_smithi159_139342_object42835:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce482d90:::benchmark_data_smithi159_139342_object45400:head# 2024-04-15T18:35:25.336 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce48a29b:::benchmark_data_smithi159_139342_object5983:head# 2024-04-15T18:35:25.340 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.340 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4ab47a:::benchmark_data_smithi159_139342_object9509:head# 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4b7426:::benchmark_data_smithi159_139342_object20693:head# 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4bb1b6:::benchmark_data_smithi159_139342_object43186:head# 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.341 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4c71d1:::benchmark_data_smithi159_139342_object19966:head# 2024-04-15T18:35:25.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4d7381:::benchmark_data_smithi159_139342_object57344:head# 2024-04-15T18:35:25.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.342 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce4e3336:::benchmark_data_smithi159_139342_object506:head# 2024-04-15T18:35:25.342 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce51d25b:::benchmark_data_smithi159_139342_object8108:head# 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce533cec:::benchmark_data_smithi159_139342_object25156:head# 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce54dd4d:::benchmark_data_smithi159_139342_object45604:head# 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce55cbf2:::benchmark_data_smithi159_139342_object5293:head# 2024-04-15T18:35:25.343 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce561240:::benchmark_data_smithi159_139342_object2445:head# 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce56ed74:::benchmark_data_smithi159_139342_object11160:head# 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce57fb96:::benchmark_data_smithi159_139342_object12119:head# 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce583ace:::benchmark_data_smithi159_139342_object53629:head# 2024-04-15T18:35:25.344 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce59548a:::benchmark_data_smithi159_139342_object9304:head# 2024-04-15T18:35:25.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.345 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce59ae78:::benchmark_data_smithi159_139342_object7714:head# 2024-04-15T18:35:25.345 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5b4782:::benchmark_data_smithi159_139342_object11703:head# 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5bdfc3:::benchmark_data_smithi159_139342_object12694:head# 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5cd693:::benchmark_data_smithi159_139342_object6745:head# 2024-04-15T18:35:25.346 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5cd803:::benchmark_data_smithi159_139342_object49587:head# 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5df863:::benchmark_data_smithi159_139342_object43000:head# 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5dff1e:::benchmark_data_smithi159_139342_object47092:head# 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5e8b62:::benchmark_data_smithi159_139342_object50633:head# 2024-04-15T18:35:25.347 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce5e9be1:::benchmark_data_smithi159_139342_object7611:head# 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce606283:::benchmark_data_smithi159_139342_object25700:head# 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce60f159:::benchmark_data_smithi159_139342_object40950:head# 2024-04-15T18:35:25.348 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce613b93:::benchmark_data_smithi159_139342_object37504:head# 2024-04-15T18:35:25.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce656833:::benchmark_data_smithi159_139342_object51876:head# 2024-04-15T18:35:25.350 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.350 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce662ca4:::benchmark_data_smithi159_139342_object25257:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6743b0:::benchmark_data_smithi159_139342_object17692:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce67c54c:::benchmark_data_smithi159_139342_object29416:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce68250e:::benchmark_data_smithi159_139342_object51322:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce694e46:::benchmark_data_smithi159_139342_object24678:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6a8b7c:::benchmark_data_smithi159_139342_object8183:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6a92b0:::benchmark_data_smithi159_139342_object22084:head# 2024-04-15T18:35:25.351 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.352 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6aa550:::benchmark_data_smithi159_139342_object33710:head# 2024-04-15T18:35:25.352 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6b4efd:::benchmark_data_smithi159_139342_object48194:head# 2024-04-15T18:35:25.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.353 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6b6879:::benchmark_data_smithi159_139342_object41196:head# 2024-04-15T18:35:25.353 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6bf868:::benchmark_data_smithi159_139342_object28546:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6cd51f:::benchmark_data_smithi159_139342_object23753:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6cd607:::benchmark_data_smithi159_139342_object1517:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6cfbe6:::benchmark_data_smithi159_139342_object20465:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6d3b7e:::benchmark_data_smithi159_139342_object56027:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6f599b:::benchmark_data_smithi159_139342_object15614:head# 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.354 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce6fb0a2:::benchmark_data_smithi159_139342_object48583:head# 2024-04-15T18:35:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce719125:::benchmark_data_smithi159_139342_object29035:head# 2024-04-15T18:35:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce743637:::benchmark_data_smithi159_139342_object41614:head# 2024-04-15T18:35:25.355 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7834eb:::benchmark_data_smithi159_139342_object50200:head# 2024-04-15T18:35:25.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7994cf:::benchmark_data_smithi159_139342_object34468:head# 2024-04-15T18:35:25.357 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.357 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7bea64:::benchmark_data_smithi159_139342_object548:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7db884:::benchmark_data_smithi159_139342_object49224:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7e6d8b:::benchmark_data_smithi159_139342_object41126:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce7f66c4:::benchmark_data_smithi159_139342_object2523:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce80802d:::benchmark_data_smithi159_139342_object50081:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8165ae:::benchmark_data_smithi159_139342_object41497:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce83bb34:::benchmark_data_smithi159_139342_object29556:head# 2024-04-15T18:35:25.358 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce83d51c:::benchmark_data_smithi159_139342_object8556:head# 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce84d3dd:::benchmark_data_smithi159_139342_object32225:head# 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce854e83:::benchmark_data_smithi159_139342_object2836:head# 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8961be:::benchmark_data_smithi159_139342_object49405:head# 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce89fa55:::benchmark_data_smithi159_139342_object17699:head# 2024-04-15T18:35:25.359 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.360 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8b7295:::benchmark_data_smithi159_139342_object47737:head# 2024-04-15T18:35:25.360 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8d1897:::benchmark_data_smithi159_139342_object47962:head# 2024-04-15T18:35:25.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.364 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8de812:::benchmark_data_smithi159_139342_object4121:head# 2024-04-15T18:35:25.364 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8e9867:::benchmark_data_smithi159_139342_object3425:head# 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce8fae68:::benchmark_data_smithi159_139342_object20673:head# 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce90ed2f:::benchmark_data_smithi159_139342_object30347:head# 2024-04-15T18:35:25.365 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce927ef8:::benchmark_data_smithi159_139342_object17591:head# 2024-04-15T18:35:25.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.366 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce95e693:::benchmark_data_smithi159_139342_object39713:head# 2024-04-15T18:35:25.366 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce984f7d:::benchmark_data_smithi159_139342_object52596:head# 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce99ecd1:::benchmark_data_smithi159_139342_object38814:head# 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce9aa07b:::benchmark_data_smithi159_139342_object44699:head# 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce9c9834:::benchmark_data_smithi159_139342_object9747:head# 2024-04-15T18:35:25.370 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ce9f00d3:::benchmark_data_smithi159_139342_object56871:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea23b77:::benchmark_data_smithi159_139342_object39913:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea428df:::benchmark_data_smithi159_139342_object33462:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea5d44b:::benchmark_data_smithi159_139342_object36092:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea7036b:::benchmark_data_smithi159_139342_object22128:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea770f0:::benchmark_data_smithi159_139342_object36048:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea78789:::benchmark_data_smithi159_139342_object20154:head# 2024-04-15T18:35:25.371 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.372 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea7a8aa:::benchmark_data_smithi159_139342_object11393:head# 2024-04-15T18:35:25.372 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea80952:::benchmark_data_smithi159_139342_object5688:head# 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea81fd7:::benchmark_data_smithi159_139342_object43293:head# 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cea9645c:::benchmark_data_smithi159_139342_object49082:head# 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceaa4bfd:::benchmark_data_smithi159_139342_object52605:head# 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceab2a1c:::benchmark_data_smithi159_139342_object4261:head# 2024-04-15T18:35:25.385 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceac345a:::benchmark_data_smithi159_139342_object33671:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceacbda7:::benchmark_data_smithi159_139342_object27602:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceae7cb2:::benchmark_data_smithi159_139342_object11487:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb06ed7:::benchmark_data_smithi159_139342_object52893:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb2636a:::benchmark_data_smithi159_139342_object39153:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb2bcde:::benchmark_data_smithi159_139342_object51946:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb2dfd6:::benchmark_data_smithi159_139342_object40648:head# 2024-04-15T18:35:25.386 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb5a38b:::benchmark_data_smithi159_139342_object36019:head# 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb89cdc:::benchmark_data_smithi159_139342_object38740:head# 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceb8a3a4:::benchmark_data_smithi159_139342_object24324:head# 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceba59f9:::benchmark_data_smithi159_139342_object26144:head# 2024-04-15T18:35:25.387 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cebaa8a8:::benchmark_data_smithi159_139342_object3563:head# 2024-04-15T18:35:25.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.388 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cebb0a83:::benchmark_data_smithi159_139342_object33194:head# 2024-04-15T18:35:25.388 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cebebcaa:::benchmark_data_smithi159_139342_object23637:head# 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec042f8:::benchmark_data_smithi159_139342_object15385:head# 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec12be6:::benchmark_data_smithi159_139342_object21407:head# 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec1f004:::benchmark_data_smithi159_139342_object28204:head# 2024-04-15T18:35:25.390 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec21e68:::benchmark_data_smithi159_139342_object51966:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec36210:::benchmark_data_smithi159_139342_object9865:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec52588:::benchmark_data_smithi159_139342_object11968:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec63d26:::benchmark_data_smithi159_139342_object5979:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec68e9e:::benchmark_data_smithi159_139342_object10526:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec6bdda:::benchmark_data_smithi159_139342_object13650:head# 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.391 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec8320c:::benchmark_data_smithi159_139342_object21480:head# 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cec9a400:::benchmark_data_smithi159_139342_object36519:head# 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceca2957:::benchmark_data_smithi159_139342_object20905:head# 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cecb6bc2:::benchmark_data_smithi159_139342_object4039:head# 2024-04-15T18:35:25.392 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cecbc588:::benchmark_data_smithi159_139342_object1316:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cecbf3ee:::benchmark_data_smithi159_139342_object8342:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cecca810:::benchmark_data_smithi159_139342_object1838:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceccb323:::benchmark_data_smithi159_139342_object16118:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ced0e010:::benchmark_data_smithi159_139342_object6440:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ced6a365:::benchmark_data_smithi159_139342_object30342:head# 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.403 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ced7b1de:::benchmark_data_smithi159_139342_object9967:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceda098e:::benchmark_data_smithi159_139342_object26953:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceda7ba9:::benchmark_data_smithi159_139342_object52083:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cedb391a:::benchmark_data_smithi159_139342_object28691:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cedec929:::benchmark_data_smithi159_139342_object27001:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee2fa85:::benchmark_data_smithi159_139342_object18265:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee35830:::benchmark_data_smithi159_139342_object22727:head# 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.404 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee3d574:::benchmark_data_smithi159_139342_object28821:head# 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee43b33:::benchmark_data_smithi159_139342_object33486:head# 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee51853:::benchmark_data_smithi159_139342_object3924:head# 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee562b9:::benchmark_data_smithi159_139342_object29107:head# 2024-04-15T18:35:25.405 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee9af9b:::benchmark_data_smithi159_139342_object4021:head# 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cee9eed3:::benchmark_data_smithi159_139342_object53915:head# 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceea306e:::benchmark_data_smithi159_139342_object32144:head# 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.407 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceea50e7:::benchmark_data_smithi159_139342_object10726:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceec4115:::benchmark_data_smithi159_139342_object34684:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceedd387:::benchmark_data_smithi159_139342_object30016:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceef52c6:::benchmark_data_smithi159_139342_object48649:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef0e72b:::benchmark_data_smithi159_139342_object22926:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef1aa72:::benchmark_data_smithi159_139342_object12573:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef1c564:::benchmark_data_smithi159_139342_object22430:head# 2024-04-15T18:35:25.408 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef335c0:::benchmark_data_smithi159_139342_object55422:head# 2024-04-15T18:35:25.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.409 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef3cc80:::benchmark_data_smithi159_139342_object37840:head# 2024-04-15T18:35:25.409 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.411 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef4a6d6:::benchmark_data_smithi159_139342_object46574:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef55b7f:::benchmark_data_smithi159_139342_object13815:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cef62184:::benchmark_data_smithi159_139342_object19016:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefa2741:::benchmark_data_smithi159_139342_object23948:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefbc9d4:::benchmark_data_smithi159_139342_object11528:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefc5564:::benchmark_data_smithi159_139342_object55230:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefcd041:::benchmark_data_smithi159_139342_object7639:head# 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.412 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefe7bc9:::benchmark_data_smithi159_139342_object32179:head# 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cefebc0d:::benchmark_data_smithi159_139342_object36314:head# 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:ceff968a:::benchmark_data_smithi159_139342_object17939:head# 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf022aea:::benchmark_data_smithi159_139342_object35967:head# 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf030d65:::benchmark_data_smithi159_139342_object29108:head# 2024-04-15T18:35:25.413 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf03362c:::benchmark_data_smithi159_139342_object17356:head# 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf03c4c6:::benchmark_data_smithi159_139342_object45139:head# 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf05d309:::benchmark_data_smithi159_139342_object30478:head# 2024-04-15T18:35:25.414 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.415 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf07158f:::benchmark_data_smithi159_139342_object3961:head# 2024-04-15T18:35:25.415 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf0dab6d:::benchmark_data_smithi159_139342_object36058:head# 2024-04-15T18:35:25.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.418 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf0fdc42:::benchmark_data_smithi159_139342_object13655:head# 2024-04-15T18:35:25.418 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf111190:::benchmark_data_smithi159_139342_object42099:head# 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf126bcc:::benchmark_data_smithi159_139342_object42563:head# 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1308ae:::benchmark_data_smithi159_139342_object53451:head# 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1346e6:::benchmark_data_smithi159_139342_object19055:head# 2024-04-15T18:35:25.419 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf159f84:::benchmark_data_smithi159_139342_object22188:head# 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf19577b:::benchmark_data_smithi159_139342_object12640:head# 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1a2980:::benchmark_data_smithi159_139342_object6648:head# 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1a9cd6:::benchmark_data_smithi159_139342_object20541:head# 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1d105b:::benchmark_data_smithi159_139342_object42865:head# 2024-04-15T18:35:25.423 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf1e15e2:::benchmark_data_smithi159_139342_object6246:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf201058:::benchmark_data_smithi159_139342_object32928:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf209fc8:::benchmark_data_smithi159_139342_object51038:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf212b9e:::benchmark_data_smithi159_139342_object49843:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf215f91:::benchmark_data_smithi159_139342_object33159:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf222e8c:::benchmark_data_smithi159_139342_object31422:head# 2024-04-15T18:35:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.426 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf235aca:::benchmark_data_smithi159_139342_object28334:head# 2024-04-15T18:35:25.426 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf23e753:::benchmark_data_smithi159_139342_object24831:head# 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf24b744:::benchmark_data_smithi159_139342_object12427:head# 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf25d3d5:::benchmark_data_smithi159_139342_object15981:head# 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf25fe2b:::benchmark_data_smithi159_139342_object1779:head# 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.433 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf289455:::benchmark_data_smithi159_139342_object42821:head# 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2967ed:::benchmark_data_smithi159_139342_object51054:head# 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf298081:::benchmark_data_smithi159_139342_object55454:head# 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf298794:::benchmark_data_smithi159_139342_object18530:head# 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2b9b12:::benchmark_data_smithi159_139342_object40917:head# 2024-04-15T18:35:25.434 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2bef03:::benchmark_data_smithi159_139342_object3654:head# 2024-04-15T18:35:25.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2d06d1:::benchmark_data_smithi159_139342_object15550:head# 2024-04-15T18:35:25.435 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.435 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2d4e3f:::benchmark_data_smithi159_139342_object2387:head# 2024-04-15T18:35:25.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf2f1853:::benchmark_data_smithi159_139342_object45700:head# 2024-04-15T18:35:25.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.436 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf30ec16:::benchmark_data_smithi159_139342_object38382:head# 2024-04-15T18:35:25.436 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf32209d:::benchmark_data_smithi159_139342_object35916:head# 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf35bd3a:::benchmark_data_smithi159_139342_object11438:head# 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf362521:::benchmark_data_smithi159_139342_object6956:head# 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3725d7:::benchmark_data_smithi159_139342_object44054:head# 2024-04-15T18:35:25.437 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.438 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf38e99a:::benchmark_data_smithi159_139342_object41366:head# 2024-04-15T18:35:25.438 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.442 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.443 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf38ebab:::benchmark_data_smithi159_139342_object53108:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf39833b:::benchmark_data_smithi159_139342_object43931:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf39a0c7:::benchmark_data_smithi159_139342_object20226:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3a0a1a:::benchmark_data_smithi159_139342_object7851:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3a59bc:::benchmark_data_smithi159_139342_object42646:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3bf692:::benchmark_data_smithi159_139342_object14088:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3c5793:::benchmark_data_smithi159_139342_object18149:head# 2024-04-15T18:35:25.444 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3d2810:::benchmark_data_smithi159_139342_object1389:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3d5ff4:::benchmark_data_smithi159_139342_object31609:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3db14e:::benchmark_data_smithi159_139342_object44937:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3eae47:::benchmark_data_smithi159_139342_object45511:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf3f181e:::benchmark_data_smithi159_139342_object39482:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf42333c:::benchmark_data_smithi159_139342_object29548:head# 2024-04-15T18:35:25.445 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.451 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf42b09b:::benchmark_data_smithi159_139342_object43882:head# 2024-04-15T18:35:25.451 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf45c740:::benchmark_data_smithi159_139342_object18498:head# 2024-04-15T18:35:25.452 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.452 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf46cb2e:::benchmark_data_smithi159_139342_object48875:head# 2024-04-15T18:35:25.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.453 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf46fa99:::benchmark_data_smithi159_139342_object49773:head# 2024-04-15T18:35:25.453 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf49698e:::benchmark_data_smithi159_139342_object1406:head# 2024-04-15T18:35:25.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.454 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf4a7984:::benchmark_data_smithi159_139342_object5842:head# 2024-04-15T18:35:25.454 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf4d0fe6:::benchmark_data_smithi159_139342_object45532:head# 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf4f18da:::benchmark_data_smithi159_139342_object14976:head# 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf4fa0d8:::benchmark_data_smithi159_139342_object19482:head# 2024-04-15T18:35:25.455 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.456 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf501e97:::benchmark_data_smithi159_139342_object19783:head# 2024-04-15T18:35:25.456 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf507fee:::benchmark_data_smithi159_139342_object29920:head# 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf527b05:::benchmark_data_smithi159_139342_object50454:head# 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf55d274:::benchmark_data_smithi159_139342_object34352:head# 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.457 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf56181f:::benchmark_data_smithi159_139342_object8553:head# 2024-04-15T18:35:25.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf57cdec:::benchmark_data_smithi159_139342_object46842:head# 2024-04-15T18:35:25.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.458 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf592315:::benchmark_data_smithi159_139342_object30093:head# 2024-04-15T18:35:25.458 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5b7c40:::benchmark_data_smithi159_139342_object33440:head# 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5baa7c:::benchmark_data_smithi159_139342_object9032:head# 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5c5c2f:::benchmark_data_smithi159_139342_object8251:head# 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5c61dd:::benchmark_data_smithi159_139342_object839:head# 2024-04-15T18:35:25.460 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5dbb60:::benchmark_data_smithi159_139342_object2855:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5e03ef:::benchmark_data_smithi159_139342_object32787:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf5f63da:::benchmark_data_smithi159_139342_object29643:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf605188:::benchmark_data_smithi159_139342_object1497:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6218cd:::benchmark_data_smithi159_139342_object11289:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf631181:::benchmark_data_smithi159_139342_object54060:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf637e33:::benchmark_data_smithi159_139342_object23676:head# 2024-04-15T18:35:25.461 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf639e85:::benchmark_data_smithi159_139342_object44768:head# 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6420f4:::benchmark_data_smithi159_139342_object50265:head# 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf646c1b:::benchmark_data_smithi159_139342_object23500:head# 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf64e970:::benchmark_data_smithi159_139342_object34577:head# 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.463 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf65262a:::benchmark_data_smithi159_139342_object39595:head# 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf675909:::benchmark_data_smithi159_139342_object57045:head# 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf678487:::benchmark_data_smithi159_139342_object24732:head# 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf67b7ba:::benchmark_data_smithi159_139342_object53047:head# 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6a3c32:::benchmark_data_smithi159_139342_object54161:head# 2024-04-15T18:35:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6a9350:::benchmark_data_smithi159_139342_object20871:head# 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6c16d9:::benchmark_data_smithi159_139342_object2495:head# 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6c7496:::benchmark_data_smithi159_139342_object11198:head# 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6de300:::benchmark_data_smithi159_139342_object48046:head# 2024-04-15T18:35:25.469 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6e02e8:::benchmark_data_smithi159_139342_object6951:head# 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6e46cc:::benchmark_data_smithi159_139342_object34996:head# 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6e7f21:::benchmark_data_smithi159_139342_object48973:head# 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6ede3c:::benchmark_data_smithi159_139342_object30299:head# 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:25.474 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf6f67f8:::benchmark_data_smithi159_139342_object2844:head# 2024-04-15T18:35:25.474 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.474 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf72056b:::benchmark_data_smithi159_139342_object15680:head# 2024-04-15T18:35:25.474 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.484 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf727aff:::benchmark_data_smithi159_139342_object42520:head# 2024-04-15T18:35:25.484 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.488 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf72f18b:::benchmark_data_smithi159_139342_object55117:head# 2024-04-15T18:35:25.488 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.490 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf752163:::benchmark_data_smithi159_139342_object16489:head# 2024-04-15T18:35:25.491 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.491 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf75278d:::benchmark_data_smithi159_139342_object36196:head# 2024-04-15T18:35:25.491 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.491 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf758d49:::benchmark_data_smithi159_139342_object34821:head# 2024-04-15T18:35:25.491 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.492 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf78375e:::benchmark_data_smithi159_139342_object54471:head# 2024-04-15T18:35:25.492 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.493 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf78556a:::benchmark_data_smithi159_139342_object31679:head# 2024-04-15T18:35:25.493 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.496 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7a06fb:::benchmark_data_smithi159_139342_object20603:head# 2024-04-15T18:35:25.496 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7a7c51:::benchmark_data_smithi159_139342_object35328:head# 2024-04-15T18:35:25.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.497 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7a884a:::benchmark_data_smithi159_139342_object8989:head# 2024-04-15T18:35:25.497 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7bdb44:::benchmark_data_smithi159_139342_object32123:head# 2024-04-15T18:35:25.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.499 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7eb2dd:::benchmark_data_smithi159_139342_object33620:head# 2024-04-15T18:35:25.499 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf7eda83:::benchmark_data_smithi159_139342_object45836:head# 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf800817:::benchmark_data_smithi159_139342_object4714:head# 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8021b8:::benchmark_data_smithi159_139342_object41765:head# 2024-04-15T18:35:25.500 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.501 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8139ce:::benchmark_data_smithi159_139342_object40246:head# 2024-04-15T18:35:25.501 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8190de:::benchmark_data_smithi159_139342_object14564:head# 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf81a444:::benchmark_data_smithi159_139342_object51823:head# 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf82a8c6:::benchmark_data_smithi159_139342_object806:head# 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.505 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf82d9c1:::benchmark_data_smithi159_139342_object26322:head# 2024-04-15T18:35:25.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf835092:::benchmark_data_smithi159_139342_object847:head# 2024-04-15T18:35:25.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.506 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf83f085:::benchmark_data_smithi159_139342_object172:head# 2024-04-15T18:35:25.506 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.510 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf84f9be:::benchmark_data_smithi159_139342_object1025:head# 2024-04-15T18:35:25.510 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf867fa6:::benchmark_data_smithi159_139342_object42406:head# 2024-04-15T18:35:25.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.511 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8a1ee0:::benchmark_data_smithi159_139342_object11369:head# 2024-04-15T18:35:25.511 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.519 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8a2f7d:::benchmark_data_smithi159_139342_object20255:head# 2024-04-15T18:35:25.519 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.522 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8aa3e9:::benchmark_data_smithi159_139342_object14814:head# 2024-04-15T18:35:25.522 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.523 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8e41f9:::benchmark_data_smithi159_139342_object23663:head# 2024-04-15T18:35:25.523 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8e6e53:::benchmark_data_smithi159_139342_object19224:head# 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8e93e5:::benchmark_data_smithi159_139342_object45920:head# 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8ebd87:::benchmark_data_smithi159_139342_object37183:head# 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.526 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf8ff9e1:::benchmark_data_smithi159_139342_object6055:head# 2024-04-15T18:35:25.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.527 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf90352a:::benchmark_data_smithi159_139342_object25844:head# 2024-04-15T18:35:25.527 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf917fed:::benchmark_data_smithi159_139342_object50718:head# 2024-04-15T18:35:25.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf921581:::benchmark_data_smithi159_139342_object23914:head# 2024-04-15T18:35:25.528 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.528 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf926e1b:::benchmark_data_smithi159_139342_object44812:head# 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf94a98a:::benchmark_data_smithi159_139342_object1790:head# 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf954253:::benchmark_data_smithi159_139342_object42637:head# 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf965d5c:::benchmark_data_smithi159_139342_object7128:head# 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf96a7a9:::benchmark_data_smithi159_139342_object30272:head# 2024-04-15T18:35:25.529 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9706d5:::benchmark_data_smithi159_139342_object44334:head# 2024-04-15T18:35:25.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.530 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf97cb39:::benchmark_data_smithi159_139342_object27119:head# 2024-04-15T18:35:25.530 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.531 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf99b510:::benchmark_data_smithi159_139342_object54084:head# 2024-04-15T18:35:25.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.532 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9aec18:::benchmark_data_smithi159_139342_object54016:head# 2024-04-15T18:35:25.532 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9b36be:::benchmark_data_smithi159_139342_object8599:head# 2024-04-15T18:35:25.536 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.536 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9b92be:::benchmark_data_smithi159_139342_object7264:head# 2024-04-15T18:35:25.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.537 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9d2a1f:::benchmark_data_smithi159_139342_object14874:head# 2024-04-15T18:35:25.537 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cf9fa6bd:::benchmark_data_smithi159_139342_object15509:head# 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfa09ba6:::benchmark_data_smithi159_139342_object39157:head# 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfa5974d:::benchmark_data_smithi159_139342_object33534:head# 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfa66004:::benchmark_data_smithi159_139342_object41695:head# 2024-04-15T18:35:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfa71040:::benchmark_data_smithi159_139342_object44472:head# 2024-04-15T18:35:25.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfa74fa7:::benchmark_data_smithi159_139342_object50288:head# 2024-04-15T18:35:25.539 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.539 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfaa3fe5:::benchmark_data_smithi159_139342_object6164:head# 2024-04-15T18:35:25.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.540 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfaaadba:::benchmark_data_smithi159_139342_object2348:head# 2024-04-15T18:35:25.540 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.542 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfaacefa:::benchmark_data_smithi159_139342_object53247:head# 2024-04-15T18:35:25.542 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfac5787:::benchmark_data_smithi159_139342_object15047:head# 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfac6f4b:::benchmark_data_smithi159_139342_object9662:head# 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfaf7bd3:::benchmark_data_smithi159_139342_object22427:head# 2024-04-15T18:35:25.543 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb00ef8:::benchmark_data_smithi159_139342_object37211:head# 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb0afef:::benchmark_data_smithi159_139342_object29471:head# 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb0ba91:::benchmark_data_smithi159_139342_object24739:head# 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.544 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb0ec9b:::benchmark_data_smithi159_139342_object27207:head# 2024-04-15T18:35:25.545 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb0ffe5:::benchmark_data_smithi159_139342_object38057:head# 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb1c11e:::benchmark_data_smithi159_139342_object24859:head# 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb317c0:::benchmark_data_smithi159_139342_object7258:head# 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb3af28:::benchmark_data_smithi159_139342_object19518:head# 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb4266c:::benchmark_data_smithi159_139342_object46712:head# 2024-04-15T18:35:25.546 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb5b87a:::benchmark_data_smithi159_139342_object49738:head# 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb90acd:::benchmark_data_smithi159_139342_object511:head# 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb94e76:::benchmark_data_smithi159_139342_object7078:head# 2024-04-15T18:35:25.547 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfb9ef78:::benchmark_data_smithi159_139342_object19754:head# 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfbd396f:::benchmark_data_smithi159_139342_object30336:head# 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfbdd221:::benchmark_data_smithi159_139342_object24177:head# 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.548 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfbe6c11:::benchmark_data_smithi159_139342_object24474:head# 2024-04-15T18:35:25.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfc07b7c:::benchmark_data_smithi159_139342_object53833:head# 2024-04-15T18:35:25.549 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.549 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfc2b9b1:::benchmark_data_smithi159_139342_object6840:head# 2024-04-15T18:35:25.550 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfc443cf:::benchmark_data_smithi159_139342_object57060:head# 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfc832a9:::benchmark_data_smithi159_139342_object13363:head# 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfca8520:::benchmark_data_smithi159_139342_object13333:head# 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfcb818b:::benchmark_data_smithi159_139342_object51957:head# 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd03a12:::benchmark_data_smithi159_139342_object29762:head# 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.555 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd0e2f0:::benchmark_data_smithi159_139342_object43734:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd1895b:::benchmark_data_smithi159_139342_object4736:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd2be2c:::benchmark_data_smithi159_139342_object54170:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd468b9:::benchmark_data_smithi159_139342_object30821:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd5f593:::benchmark_data_smithi159_139342_object44609:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd7451f:::benchmark_data_smithi159_139342_object32362:head# 2024-04-15T18:35:25.556 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.560 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd79abb:::benchmark_data_smithi159_139342_object28504:head# 2024-04-15T18:35:25.560 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfd80860:::benchmark_data_smithi159_139342_object18098:head# 2024-04-15T18:35:25.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.562 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfded8c7:::benchmark_data_smithi159_139342_object25210:head# 2024-04-15T18:35:25.562 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfdee9e8:::benchmark_data_smithi159_139342_object34630:head# 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe382db:::benchmark_data_smithi159_139342_object17854:head# 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe4134c:::benchmark_data_smithi159_139342_object43334:head# 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe45815:::benchmark_data_smithi159_139342_object4258:head# 2024-04-15T18:35:25.563 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe564d4:::benchmark_data_smithi159_139342_object22373:head# 2024-04-15T18:35:25.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.564 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe5b56e:::benchmark_data_smithi159_139342_object6534:head# 2024-04-15T18:35:25.564 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe657ed:::benchmark_data_smithi159_139342_object291:head# 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe6d72a:::benchmark_data_smithi159_139342_object22488:head# 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe6e1a1:::benchmark_data_smithi159_139342_object40847:head# 2024-04-15T18:35:25.565 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:35:25.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe7462f:::benchmark_data_smithi159_139342_object27570:head# 2024-04-15T18:35:25.566 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.566 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe7b65b:::benchmark_data_smithi159_139342_object47241:head# 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe7da31:::benchmark_data_smithi159_139342_object20778:head# 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfe96e74:::benchmark_data_smithi159_139342_object11571:head# 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfea1a6b:::benchmark_data_smithi159_139342_object54713:head# 2024-04-15T18:35:25.567 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.568 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfea8fb6:::benchmark_data_smithi159_139342_object52422:head# 2024-04-15T18:35:25.568 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.572 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfeae918:::benchmark_data_smithi159_139342_object49026:head# 2024-04-15T18:35:25.572 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:25.574 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfeb2956:::benchmark_data_smithi159_139342_object30140:head# 2024-04-15T18:35:25.574 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.575 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfecb13f:::benchmark_data_smithi159_139342_object53203:head# 2024-04-15T18:35:25.575 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfece703:::benchmark_data_smithi159_139342_object18479:head# 2024-04-15T18:35:25.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.579 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfed7478:::benchmark_data_smithi159_139342_object30977:head# 2024-04-15T18:35:25.579 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfee12f0:::benchmark_data_smithi159_139342_object2397:head# 2024-04-15T18:35:25.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.583 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfee4b47:::benchmark_data_smithi159_139342_object17517:head# 2024-04-15T18:35:25.583 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff2c08c:::benchmark_data_smithi159_139342_object52770:head# 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff4b57d:::benchmark_data_smithi159_139342_object26799:head# 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff4e3d1:::benchmark_data_smithi159_139342_object37453:head# 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff517ba:::benchmark_data_smithi159_139342_object4209:head# 2024-04-15T18:35:25.584 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.586 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff6d612:::benchmark_data_smithi159_139342_object48469:head# 2024-04-15T18:35:25.586 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.587 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff8711e:::benchmark_data_smithi159_139342_object31272:head# 2024-04-15T18:35:25.587 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.587 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff8ee22:::benchmark_data_smithi159_139342_object13071:head# 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff9067c:::benchmark_data_smithi159_139342_object48928:head# 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff95e4a:::benchmark_data_smithi159_139342_object33168:head# 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cff97226:::benchmark_data_smithi159_139342_object34909:head# 2024-04-15T18:35:25.588 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cffb2c32:::benchmark_data_smithi159_139342_object52498:head# 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cffc60fa:::benchmark_data_smithi159_139342_object21120:head# 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cffc7723:::benchmark_data_smithi159_139342_object5008:head# 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cffd2df4:::benchmark_data_smithi159_139342_object54107:head# 2024-04-15T18:35:25.589 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cffde47e:::benchmark_data_smithi159_139342_object37273:head# 2024-04-15T18:35:25.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.590 INFO:teuthology.orchestra.run.smithi082.stdout:Write #6:cfff17da:::benchmark_data_smithi159_139342_object37511:head# 2024-04-15T18:35:25.590 INFO:teuthology.orchestra.run.smithi082.stdout:snapset 0=[]:{} 2024-04-15T18:35:25.598 INFO:teuthology.orchestra.run.smithi082.stdout:write_pg epoch 561 info 6.3( v 561'3617 (0'0,561'3617] local-lis/les=541/542 n=3617 ec=541/541 lis/c=541/541 les/c/f=542/542/0 sis=541) 2024-04-15T18:35:25.602 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.625 INFO:teuthology.orchestra.run.smithi082.stdout:Import successful 2024-04-15T18:35:25.625 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:25.705 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:35:25.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:25.767 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.806 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:25.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:25 smithi159 ceph-mon[45336]: 6.6 scrub starts 2024-04-15T18:35:25.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:35:25.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:25.963 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:25.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:25 smithi049 ceph-mon[40087]: 6.6 scrub starts 2024-04-15T18:35:25.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:35:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:25.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:26.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:25 smithi082 ceph-mon[44206]: 6.6 scrub starts 2024-04-15T18:35:26.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:35:26.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:26.131 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:75414 5.f 5:f2f9aa70:::benchmark_data_smithi159_116570_object75413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:50.844845+0000", 2024-04-15T18:35:26.165 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.31846508399997, 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53091057600000002, 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 75414 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844845+0000", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844845+0000", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844846+0000", 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.353e-06 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844902+0000", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5736999999999998e-05 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844912+0000", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.045e-05 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844918+0000", 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5879999999999997e-06 2024-04-15T18:35:26.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.844954+0000", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6279999999999998e-05 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.845057+0000", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010341 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.845174+0000", 2024-04-15T18:35:26.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116697 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.847241+0000", 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020670139999999998 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.848250+0000", 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010085820000000001 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375561+0000", 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52731106800000005 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375633+0000", 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2107000000000006e-05 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375755+0000", 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012229 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:75414 5.f e392/377 ondisk, result = 0)", 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:25:50.847559+0000", 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.31575052100004, 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52820214399999998, 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.847559+0000", 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.847559+0000", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.847560+0000", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3799999999999996e-07 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:50.847566+0000", 2024-04-15T18:35:26.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6849999999999998e-06 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375476+0000", 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52790998600000005 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375483+0000", 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1570000000000003e-06 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375542+0000", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9487000000000002e-05 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375553+0000", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.096e-05 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:25:51.375761+0000", 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020793100000000001 2024-04-15T18:35:26.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93121 5.a 5:590d7c05:::benchmark_data_smithi159_116570_object93120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.433915+0000", 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72939441599999, 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43449275599999998, 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.176 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93121 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433915+0000", 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433915+0000", 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.433917+0000", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4959999999999999e-06 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434055+0000", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013867799999999999 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434068+0000", 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2653999999999999e-05 2024-04-15T18:35:26.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434078+0000", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8330000000000006e-06 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436519+0000", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002440869 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436530+0000", 2024-04-15T18:35:26.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1039000000000001e-05 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.857579+0000", 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.42104956500000001 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.858992+0000", 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001413128 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.859115+0000", 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122285 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.866295+0000", 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0071798829999999998 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.866306+0000", 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1080999999999999e-05 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.866358+0000", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2509000000000002e-05 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.866391+0000", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2521999999999998e-05 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.866440+0000", 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9129999999999999e-05 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868408+0000", 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019680840000000002 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93122 5.a 5:5fa52b01:::benchmark_data_smithi159_116570_object93121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434093+0000", 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72921709399998, 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43642836099999999, 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.183 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93122 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434093+0000", 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434093+0000", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434093+0000", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6899999999999998e-07 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434145+0000", 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2479999999999999e-05 2024-04-15T18:35:26.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434154+0000", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5790000000000004e-06 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434160+0000", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5040000000000002e-06 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436576+0000", 2024-04-15T18:35:26.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002416634 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436586+0000", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9499999999999996e-06 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.863663+0000", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.42707650699999999 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.863789+0000", 2024-04-15T18:35:26.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000126153 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.863912+0000", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000122714 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868495+0000", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045836949999999996 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868544+0000", 2024-04-15T18:35:26.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9120999999999999e-05 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868580+0000", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5187e-05 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868627+0000", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7701000000000003e-05 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.868667+0000", 2024-04-15T18:35:26.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0034999999999999e-05 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.870521+0000", 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001853632 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93124 5.a 5:5860cf80:::benchmark_data_smithi159_116570_object93123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434168+0000", 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72914175100004, 2024-04-15T18:35:26.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46561628500000002, 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93124 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434168+0000", 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434168+0000", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434168+0000", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7599999999999998e-07 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434219+0000", 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1246999999999999e-05 2024-04-15T18:35:26.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434227+0000", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5469999999999999e-06 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434232+0000", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9479999999999999e-06 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436619+0000", 2024-04-15T18:35:26.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002387129 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436624+0000", 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6410000000000001e-06 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.870580+0000", 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43395657500000001 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.870737+0000", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015709599999999999 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.870915+0000", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000177271 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879266+0000", 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0083512090000000001 2024-04-15T18:35:26.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879304+0000", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7666000000000002e-05 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879343+0000", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9520000000000001e-05 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.897812+0000", 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.018468931000000001 2024-04-15T18:35:26.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.897847+0000", 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4662000000000001e-05 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.899784+0000", 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001937467 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93126 5.a 5:526c9c94:::benchmark_data_smithi159_116570_object93125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434239+0000", 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72907107599997, 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.466710034, 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93126 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434239+0000", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434239+0000", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434239+0000", 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.34e-07 2024-04-15T18:35:26.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434288+0000", 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9617999999999998e-05 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434297+0000", 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0809999999999994e-06 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434301+0000", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8899999999999998e-06 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436655+0000", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002353693 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436663+0000", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7619999999999995e-06 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.873854+0000", 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43719123199999999 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.873964+0000", 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.874075+0000", 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110487 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879355+0000", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0052801499999999999 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.885076+0000", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0057215850000000004 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887069+0000", 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019929150000000001 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.899817+0000", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012747741 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.899855+0000", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8238000000000002e-05 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.900949+0000", 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010933080000000001 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93129 5.a 5:5bc56943:::benchmark_data_smithi159_116570_object93128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434308+0000", 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.729001928, 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46775032900000002, 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93129 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434308+0000", 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434308+0000", 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434308+0000", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6899999999999998e-07 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434360+0000", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2055000000000002e-05 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434368+0000", 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9000000000000006e-06 2024-04-15T18:35:26.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434373+0000", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7759999999999997e-06 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436700+0000", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023273790000000001 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436710+0000", 2024-04-15T18:35:26.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5519999999999993e-06 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.876361+0000", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43965133499999998 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.876514+0000", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015322600000000001 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.876685+0000", 2024-04-15T18:35:26.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017054099999999999 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.885018+0000", 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0083331640000000005 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887122+0000", 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021043720000000002 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887159+0000", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7014000000000002e-05 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.900982+0000", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.013822615999999999 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.901021+0000", 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9048999999999998e-05 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.902058+0000", 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001036981 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93139 5.a 5:5eaad1da:::benchmark_data_smithi159_116570_object93138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434380+0000", 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72892934499998, 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46878445099999999, 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.212 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93139 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434380+0000", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434380+0000", 2024-04-15T18:35:26.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434381+0000", 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4799999999999999e-07 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434435+0000", 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3998e-05 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434443+0000", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0830000000000004e-06 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434448+0000", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9359999999999998e-06 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436755+0000", 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023068960000000001 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436762+0000", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7659999999999998e-06 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879378+0000", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44261540700000002 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879478+0000", 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100383 2024-04-15T18:35:26.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.879579+0000", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100518 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887039+0000", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0074600480000000004 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887190+0000", 2024-04-15T18:35:26.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015166299999999999 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887221+0000", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0516000000000001e-05 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.902103+0000", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014882375 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.902136+0000", 2024-04-15T18:35:26.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3225999999999997e-05 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.903165+0000", 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001028288 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93143 5.a 5:55cec24a:::benchmark_data_smithi159_116570_object93142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434465+0000", 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72884476000002, 2024-04-15T18:35:26.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46997232500000002, 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93143 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434465+0000", 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434465+0000", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434465+0000", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9200000000000002e-07 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434515+0000", 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0096999999999998e-05 2024-04-15T18:35:26.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434525+0000", 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5699999999999999e-06 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434530+0000", 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9640000000000002e-06 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436827+0000", 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022973749999999999 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436834+0000", 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6939999999999998e-06 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.882167+0000", 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44533329100000002 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.882314+0000", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014718700000000001 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.882496+0000", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000181812 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887079+0000", 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045828190000000001 2024-04-15T18:35:26.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887249+0000", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017027899999999999 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887284+0000", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4298999999999998e-05 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.903241+0000", 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015957091999999999 2024-04-15T18:35:26.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.903289+0000", 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8130999999999997e-05 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.904437+0000", 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011483229999999999 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:93152 5.a 5:53f56814:::benchmark_data_smithi159_116570_object93151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:17.434536+0000", 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 548.72877337199998, 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47101279400000001, 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93152 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434536+0000", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434536+0000", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434537+0000", 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5100000000000001e-07 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434583+0000", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6242000000000002e-05 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434591+0000", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7109999999999995e-06 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.434595+0000", 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6809999999999997e-06 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436873+0000", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022777299999999999 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.436883+0000", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0325e-05 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887309+0000", 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45042525500000002 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887414+0000", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000105338 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.887535+0000", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012113200000000001 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.897724+0000", 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010189452999999999 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.897735+0000", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0502999999999999e-05 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.897777+0000", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2494999999999999e-05 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.904493+0000", 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0067157049999999998 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.904531+0000", 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7908999999999999e-05 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:17.905549+0000", 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010179640000000001 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.233 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.a 419/405 [PushReplyOp(5:5f885ffc:::benchmark_data_smithi159_116570_object20517:head)])", 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:45.105683+0000", 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 521.05762690999995, 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.56594298200000004, 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.105683+0000", 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.105683+0000", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.105684+0000", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0589999999999999e-06 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.105690+0000", 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0399999999999998e-06 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.668501+0000", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.56281090600000006 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.668510+0000", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7159999999999998e-06 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.668577+0000", 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6595999999999997e-05 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:45.671626+0000", 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030486649999999999 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.a 420/405 [PushReplyOp(5:5b48313f:::benchmark_data_smithi159_116570_object22923:head)])", 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:47.554390+0000", 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 518.60891995700001, 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.54279947200000001, 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.237 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.554390+0000", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.554390+0000", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.554391+0000", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8400000000000003e-07 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.554396+0000", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2329999999999997e-06 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.094475+0000", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.540079596 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.094484+0000", 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0259999999999996e-06 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.094580+0000", 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5172000000000003e-05 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.097189+0000", 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026095609999999998 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.240 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.a 420/405 [PushReplyOp(5:523944d0:::benchmark_data_smithi159_116570_object22930:head)])", 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:47.563889+0000", 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 518.59942013099999, 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.54464624699999997, 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.563889+0000", 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.563889+0000", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.563890+0000", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:47.563895+0000", 2024-04-15T18:35:26.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6399999999999996e-06 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.106015+0000", 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.54212012899999995 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.106023+0000", 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3580000000000003e-06 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.106085+0000", 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1928999999999996e-05 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:48.108536+0000", 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024503099999999998 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.a 422/405 [PushReplyOp(5:55c04187:::benchmark_data_smithi159_116570_object28415:head)])", 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:52.773683+0000", 2024-04-15T18:35:26.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 513.38962639700003, 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.59944710899999998, 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.773683+0000", 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.773683+0000", 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.773684+0000", 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.054e-06 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:52.773690+0000", 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.857e-06 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.369403+0000", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.59571264800000001 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.369408+0000", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1760000000000001e-06 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.369438+0000", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9947999999999999e-05 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:53.373130+0000", 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003692426 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(5.a 423/405 [PushReplyOp(5:5c2fef5b:::benchmark_data_smithi159_116570_object31419:head)])", 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:26:56.119819+0000", 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 510.04349106000001, 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.573196649, 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:26.248 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.119819+0000", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.119819+0000", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.119820+0000", 2024-04-15T18:35:26.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.046e-06 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.119825+0000", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5090000000000001e-06 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.690431+0000", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57060601 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.690438+0000", 2024-04-15T18:35:26.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2250000000000003e-06 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.690487+0000", 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9118e-05 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:26:56.693015+0000", 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002527741 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:113046 5.a 5:53f8d519:::benchmark_data_smithi159_116570_object19886:head [delete] snapc 0=[] ondisk+write+known_if_redirected e436)", 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:33.440122+0000", 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 472.72318751, 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50539747800000001, 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113046 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.252 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440122+0000", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440122+0000", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440123+0000", 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3790000000000001e-06 2024-04-15T18:35:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440130+0000", 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.19e-06 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440143+0000", 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2835e-05 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440149+0000", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6830000000000002e-06 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440196+0000", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6464000000000001e-05 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440319+0000", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012317599999999999 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,8,10", 2024-04-15T18:35:26.255 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.440398+0000", 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9079000000000006e-05 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444164+0000", 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037661249999999999 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444200+0000", 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5475999999999997e-05 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.448453+0000", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042536270000000003 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945226+0000", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49677261 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945267+0000", 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1545000000000003e-05 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945520+0000", 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000252289 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:113046 5.a e436/405 ondisk, result = 0)", 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:33.444184+0000", 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 472.71912589900001, 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50134110600000004, 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.258 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444184+0000", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444184+0000", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444184+0000", 2024-04-15T18:35:26.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3000000000000001e-07 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.444187+0000", 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7879999999999998e-06 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.944644+0000", 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50045741200000005 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.944648+0000", 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8630000000000001e-06 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945215+0000", 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000567247 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945221+0000", 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0220000000000001e-06 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:33.945525+0000", 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030334399999999999 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:127441 5.f e481/393 5:f29729c2:::benchmark_data_smithi159_116570_object34281:head v 481'8118, mlcod=481'8115)", 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:15.186122+0000", 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 370.97718801600001, 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50947705799999998, 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.262 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186122+0000", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186122+0000", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186122+0000", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186129+0000", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3690000000000003e-06 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.692395+0000", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50626629599999995 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.692402+0000", 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.764e-06 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.692442+0000", 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0657000000000001e-05 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.692484+0000", 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1318e-05 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695570+0000", 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003086334 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695599+0000", 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8551000000000002e-05 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:57366 6.e 6:7868e193:::benchmark_data_smithi159_139342_object57365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 109.597219626, 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.461560207, 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57366 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566092+0000", 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5719999999999999e-06 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566149+0000", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7744999999999997e-05 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566162+0000", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2685e-05 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566170+0000", 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0239999999999997e-06 2024-04-15T18:35:26.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566220+0000", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9595000000000002e-05 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566225+0000", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567183+0000", 2024-04-15T18:35:26.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095810800000000005 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567310+0000", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012681300000000001 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,2", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567438+0000", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000127935 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.569385+0000", 2024-04-15T18:35:26.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001947465 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.570421+0000", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010361540000000001 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027576+0000", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45715493899999998 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027618+0000", 2024-04-15T18:35:26.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1644999999999999e-05 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027650+0000", 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2159000000000002e-05 2024-04-15T18:35:26.273 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:57386 6.2 6:46635af6:::benchmark_data_smithi159_139342_object57385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:36.666699+0000", 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 109.496611083, 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.363897203, 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:26.274 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57386 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666699+0000", 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:26.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666699+0000", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666700+0000", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4449999999999999e-06 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666752+0000", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2178000000000001e-05 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:26.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666763+0000", 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1168e-05 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666769+0000", 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.913e-06 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666824+0000", 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4389999999999999e-05 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.666968+0000", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014486300000000001 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.667129+0000", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016078300000000001 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.668911+0000", 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017821460000000001 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.669562+0000", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000651083 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.030512+0000", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.36094901099999999 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.030549+0000", 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7604000000000001e-05 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:26.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.030596+0000", 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6619000000000002e-05 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:26.280 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:35:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:26.281 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:26.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:26.382 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:26.418 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:26.453 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:26.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:26.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:26.582 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:26.602 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:26.709 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: 6.6 scrub ok 2024-04-15T18:35:26.709 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: 6.e scrub starts 2024-04-15T18:35:26.709 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: pgmap v1553: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 8344/115314 objects degraded (7.236%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:35:26.709 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:26.709 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:35:26.710 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:26 smithi159 ceph-mon[45336]: osdmap e596: 12 total, 11 up, 11 in 2024-04-15T18:35:26.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: 6.6 scrub ok 2024-04-15T18:35:26.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: 6.e scrub starts 2024-04-15T18:35:26.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: pgmap v1553: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 8344/115314 objects degraded (7.236%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:35:26.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:26.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:35:26.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:26 smithi049 ceph-mon[40087]: osdmap e596: 12 total, 11 up, 11 in 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: 6.6 scrub ok 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: 6.e scrub starts 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: pgmap v1553: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 8344/115314 objects degraded (7.236%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1461454474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:26 smithi082 ceph-mon[44206]: osdmap e596: 12 total, 11 up, 11 in 2024-04-15T18:35:27.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8344/115314 objects degraded (7.236%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:28.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8344/115314 objects degraded (7.236%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:28.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8344/115314 objects degraded (7.236%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:28.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:35:28.737 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:28.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:28.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:28.867 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:28.903 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:28.941 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:28.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:28 smithi049 ceph-mon[40087]: pgmap v1555: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 7758/111789 objects degraded (6.940%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:35:28.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1846152053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:28.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:29.021 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:29.024 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:29.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:28 smithi082 ceph-mon[44206]: pgmap v1555: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 7758/111789 objects degraded (6.940%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:35:29.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1846152053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:29.057 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:29.087 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:29.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:29.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:29.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:29.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:29.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:29.117 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:29.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:28 smithi159 ceph-mon[45336]: pgmap v1555: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 7758/111789 objects degraded (6.940%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:35:29.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1846152053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:29.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:30.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:30.974 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:30.980 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.980 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.980 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.980 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:30.980 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:30 smithi082 ceph-mon[44206]: pgmap v1556: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 7236/106107 objects degraded (6.820%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:35:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:30.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:31.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:31.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:30 smithi049 ceph-mon[40087]: pgmap v1556: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 7236/106107 objects degraded (6.820%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:35:31.118 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:30 smithi159 ceph-mon[45336]: pgmap v1556: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 7236/106107 objects degraded (6.820%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:35:31.180 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:31.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:31.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:31.329 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:31.357 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:31.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:31.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:31.493 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:31.515 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:31.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:31.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:31.656 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:32.765 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:35:32.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:35:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:32 smithi082 ceph-mon[44206]: pgmap v1557: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 6592/102858 objects degraded (6.409%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:35:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/275846626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 6592/102858 objects degraded (6.409%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:33.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:32 smithi159 ceph-mon[45336]: pgmap v1557: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 6592/102858 objects degraded (6.409%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:35:33.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/275846626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 6592/102858 objects degraded (6.409%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:33.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:32 smithi049 ceph-mon[40087]: pgmap v1557: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 6592/102858 objects degraded (6.409%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:35:33.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/275846626' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:33.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 6592/102858 objects degraded (6.409%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:33.762 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:33.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:33 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:33.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:33 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:33.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:33 smithi049 ceph-mon[40087]: osdmap e597: 12 total, 11 up, 11 in 2024-04-15T18:35:34.012 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:33 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:34.012 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:33 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:34.012 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:33 smithi082 ceph-mon[44206]: osdmap e597: 12 total, 11 up, 11 in 2024-04-15T18:35:34.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:33 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:34.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:33 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:34.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:33 smithi159 ceph-mon[45336]: osdmap e597: 12 total, 11 up, 11 in 2024-04-15T18:35:34.399 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:34.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:34.445 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:35:34.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/exp.6.3.6 2024-04-15T18:35:34.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:34.554 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:34.613 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-15T18:35:34.613 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:35:34.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:35:34.635 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:34.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:34.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:34.773 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:34.803 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:34.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:34.906 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:35:34.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:34.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:34 smithi049 ceph-mon[40087]: pgmap v1559: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 6055/95634 objects degraded (6.331%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:35:34.944 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:35.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:34 smithi082 ceph-mon[44206]: pgmap v1559: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 6055/95634 objects degraded (6.331%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:35:35.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:34 smithi159 ceph-mon[45336]: pgmap v1559: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 6055/95634 objects degraded (6.331%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:35:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:35.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:35.404 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:35.447 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:35.478 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:35.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:35.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:35.616 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:35.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:35:35.643 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:35.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:35.746 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:35.779 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:35.811 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:35.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:35.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:35.969 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:35.991 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:36.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:36.080 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:36.088 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:36.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:36.192 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:36.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:36.258 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:36.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:36.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:36.310 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:36.348 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:36.364 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:36.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:36.413 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:36.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:36.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:36.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:36.491 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:36.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:36.492 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:36.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:36.579 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:36.605 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:36.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:36.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:36.763 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: pgmap v1560: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6055/95001 objects degraded (6.374%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/700486775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: pgmap v1560: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6055/95001 objects degraded (6.374%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/700486775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.159 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:35:37.159 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:35:37.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.159 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:151338 6.5 6:a1b059b8:::benchmark_data_smithi159_139342_object61459:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.157323+0000", 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.010464218000000001, 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010493227000000001, 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 151338 2024-04-15T18:35:37.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157323+0000", 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157323+0000", 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157324+0000", 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.195e-06 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157330+0000", 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7599999999999999e-06 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157342+0000", 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2354999999999999e-05 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157350+0000", 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0299999999999994e-06 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.162725+0000", 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.005374701 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.162883+0000", 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015846600000000001 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,9", 2024-04-15T18:35:37.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163040+0000", 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015697399999999999 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167044+0000", 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040040379999999997 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:151343 6.8 6:15d16d62:::benchmark_data_smithi159_139342_object61464:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.163353+0000", 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.0044335540000000001, 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004551434, 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:37.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 151343 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163353+0000", 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163353+0000", 2024-04-15T18:35:37.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163355+0000", 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.403e-06 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163358+0000", 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5669999999999999e-06 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163368+0000", 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6120000000000005e-06 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163374+0000", 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2589999999999997e-06 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:37.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163434+0000", 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0077000000000001e-05 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163565+0000", 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013124100000000001 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163669+0000", 2024-04-15T18:35:37.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103707 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:151345 6.8 6.db55018 (undecoded) ondisk+write+known_if_redirected e597)", 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.166431+0000", 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.001355773, 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015582009999999999, 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.171 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 151345 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166431+0000", 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166431+0000", 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166432+0000", 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1489999999999999e-06 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166437+0000", 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6020000000000002e-06 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166448+0000", 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.133e-05 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.166461+0000", 2024-04-15T18:35:37.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3094e-05 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26224.0:151343 6.8 e597/541)", 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.167971+0000", 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4294967295.9998159, 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3893000000000006e-05, 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.175 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167971+0000", 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167971+0000", 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167972+0000", 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8700000000000004e-07 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167976+0000", 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1099999999999996e-06 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.167998+0000", 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2473e-05 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.168004+0000", 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6690000000000003e-06 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:151334 6.b 6:d618f96a:::benchmark_data_smithi159_139342_object61455:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.152281+0000", 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.015505787, 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015820799, 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 151334 2024-04-15T18:35:37.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152281+0000", 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152281+0000", 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152282+0000", 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1999999999999998e-07 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152287+0000", 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8799999999999999e-06 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152296+0000", 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0939999999999997e-06 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152302+0000", 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.455e-06 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:37.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152360+0000", 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7386999999999998e-05 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152506+0000", 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014642799999999999 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,3", 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.152621+0000", 2024-04-15T18:35:37.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115252 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163048+0000", 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010426517 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163193+0000", 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014562900000000001 2024-04-15T18:35:37.184 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:151336 6.3 6:ce3e0558:::benchmark_data_smithi159_139342_object61457:head [delete] snapc 0=[] ondisk+write+known_if_redirected e597)", 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:37.155485+0000", 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.012301565, 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.012701829, 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:37.185 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 151336 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155485+0000", 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155485+0000", 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155486+0000", 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.19e-06 2024-04-15T18:35:37.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155491+0000", 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9350000000000002e-06 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155506+0000", 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4253999999999999e-05 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.155513+0000", 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3309999999999998e-06 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157339+0000", 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001825777 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157483+0000", 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144091 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 8,11", 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.157628+0000", 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000145525 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163058+0000", 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0054293090000000002 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:37.163220+0000", 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016274200000000001 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.191 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:35:37.192 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 6 2024-04-15T18:35:37.192 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:35:37.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: pgmap v1560: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 696 op/s; 6055/95001 objects degraded (6.374%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:35:37.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/700486775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:37.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.858 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.859 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.859 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.859 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.859 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.859 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 6055/95001 objects degraded (6.374%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:37.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:37.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 6055/95001 objects degraded (6.374%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:38.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:38.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:38.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:38.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:38.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:38.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 6055/95001 objects degraded (6.374%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:38.325 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:35:38.884 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:38 smithi082 ceph-mon[44206]: pgmap v1561: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 5393/91062 objects degraded (5.922%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:39.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:38 smithi159 ceph-mon[45336]: pgmap v1561: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 5393/91062 objects degraded (5.922%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:39.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:38 smithi049 ceph-mon[40087]: pgmap v1561: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 5393/91062 objects degraded (5.922%); 15 MiB/s, 236 objects/s recovering 2024-04-15T18:35:39.644 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:39.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:39.753 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:39.782 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:39.805 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:39.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:39.913 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:39.948 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:39.972 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:39.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:40.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:40.115 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:40.132 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:40.159 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:40.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:40.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:40.302 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:40.331 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:35:40.331 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:35:40.331 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:35:40.331 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:35:40.331 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:35:40.332 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:40.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:40.441 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:40.483 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:40.587 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:40.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:40.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:40.743 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:41.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:40 smithi082 ceph-mon[44206]: pgmap v1562: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 4737/85278 objects degraded (5.555%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:35:41.082 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:35:41.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:40 smithi159 ceph-mon[45336]: pgmap v1562: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 4737/85278 objects degraded (5.555%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:35:41.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:40 smithi049 ceph-mon[40087]: pgmap v1562: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 4737/85278 objects degraded (5.555%); 16 MiB/s, 249 objects/s recovering 2024-04-15T18:35:41.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:41.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:41.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:41.353 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:41.383 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:41.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:41.482 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:35:41.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:41.526 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:41.559 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:41.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:41.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:41.707 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:41.756 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:41.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:41.831 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:35:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1827193785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:41.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:41 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:41.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:41 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:41.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:41 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:41.862 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:41.919 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1827193785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:41 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:41 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:42.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:41 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:42.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:35:42.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1827193785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:42.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:41 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:35:42.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:41 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:42.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:41 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:42.302 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:35:42.319 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:35:42.886 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:42.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:42.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:43.034 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:43.061 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:43.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: pgmap v1563: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 4062/81684 objects degraded (4.973%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:43.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:43.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:43.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: osdmap e598: 12 total, 11 up, 11 in 2024-04-15T18:35:43.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: from='client.16698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:43.065 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 4062/81684 objects degraded (4.973%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:43.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:43.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:43.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: pgmap v1563: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 4062/81684 objects degraded (4.973%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:43.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:43.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:43.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: osdmap e598: 12 total, 11 up, 11 in 2024-04-15T18:35:43.174 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: from='client.16698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:43.174 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 4062/81684 objects degraded (4.973%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: pgmap v1563: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 705 op/s; 4062/81684 objects degraded (4.973%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: osdmap e598: 12 total, 11 up, 11 in 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: from='client.16698 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:43.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 4062/81684 objects degraded (4.973%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:43.195 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:43.498 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119557 5.9 5:9526d67b:::benchmark_data_smithi159_116570_object26397:head [delete] snapc 0=[] ondisk+write+known_if_redirected e461)", 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 436.879382706, 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49196453200000001, 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.499 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119557 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625864+0000", 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625866+0000", 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6190000000000001e-06 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625873+0000", 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3540000000000002e-06 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625891+0000", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7382000000000001e-05 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625899+0000", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9570000000000002e-06 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.625962+0000", 2024-04-15T18:35:43.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3751999999999995e-05 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.626087+0000", 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000124306 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.626200+0000", 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113007 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.629266+0000", 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030667070000000001 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:26.663677+0000", 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.034410145000000003 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117667+0000", 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45399008600000001 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117757+0000", 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0679000000000004e-05 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:27.117829+0000", 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1538000000000005e-05 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.505 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34054 6.5 6:a99ded54:::benchmark_data_smithi159_139342_object34053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.581990+0000", 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.923256759, 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 15.014487111999999, 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34054 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.506 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.581990+0000", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.581990+0000", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.581991+0000", 2024-04-15T18:35:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5000000000000002e-07 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582015+0000", 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3869000000000001e-05 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582026+0000", 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0779e-05 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582032+0000", 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0390000000000001e-06 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582062+0000", 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0281e-05 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582138+0000", 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5766999999999995e-05 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.509 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582222+0000", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4452999999999999e-05 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.582980+0000", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075765800000000003 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.583132+0000", 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015233099999999999 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596380+0000", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 15.013247934000001 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596426+0000", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6208000000000002e-05 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596477+0000", 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1042999999999999e-05 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.511 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34056 6.5 6:a212fbee:::benchmark_data_smithi159_139342_object34055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.634703+0000", 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.87054425400001, 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.961891753, 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.512 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34056 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634703+0000", 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634703+0000", 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634704+0000", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0800000000000004e-07 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634774+0000", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0389000000000007e-05 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634785+0000", 2024-04-15T18:35:43.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0604999999999999e-05 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634790+0000", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8239999999999996e-06 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634830+0000", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9295999999999998e-05 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.634924+0000", 2024-04-15T18:35:43.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4574999999999995e-05 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.635012+0000", 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7740999999999999e-05 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.635798+0000", 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00078633499999999999 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.636082+0000", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00028401199999999999 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596523+0000", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.960440988 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596562+0000", 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9195e-05 2024-04-15T18:35:43.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.596595+0000", 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2085000000000002e-05 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34067 6.5 6:a3f0f63b:::benchmark_data_smithi159_139342_object34066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.657534+0000", 2024-04-15T18:35:43.518 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.84771277300001, 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.940645426, 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34067 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657534+0000", 2024-04-15T18:35:43.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657534+0000", 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657535+0000", 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0809999999999999e-06 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657596+0000", 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0313999999999999e-05 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657604+0000", 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7150000000000004e-06 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657610+0000", 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2259999999999996e-06 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657651+0000", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1662000000000002e-05 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657744+0000", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3026e-05 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.657834+0000", 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9627000000000001e-05 2024-04-15T18:35:43.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.658940+0000", 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001106306 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.659102+0000", 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016156300000000001 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.598096+0000", 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.938994326 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.598135+0000", 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8909000000000003e-05 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.598180+0000", 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4671e-05 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.524 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34086 6.5 6:a824a12c:::benchmark_data_smithi159_139342_object34085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.737152+0000", 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.76809503800001, 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.862932645000001, 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.525 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34086 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737152+0000", 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737152+0000", 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.526 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737153+0000", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0499999999999999e-06 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737211+0000", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8334000000000001e-05 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737222+0000", 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0667e-05 2024-04-15T18:35:43.527 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737228+0000", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.553e-06 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737267+0000", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8884000000000002e-05 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737354+0000", 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7909000000000001e-05 2024-04-15T18:35:43.528 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.737444+0000", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9965999999999999e-05 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.739256+0000", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0018117910000000001 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.739679+0000", 2024-04-15T18:35:43.529 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042327899999999998 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600004+0000", 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.860324699 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600056+0000", 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1783999999999998e-05 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.530 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600085+0000", 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8728999999999999e-05 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34095 6.5 6:aac1b905:::benchmark_data_smithi159_139342_object34094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.772168+0000", 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.73307890300001, 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.828042538, 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.531 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34095 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772168+0000", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.532 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772168+0000", 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772169+0000", 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6700000000000002e-07 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772212+0000", 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3143999999999999e-05 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.533 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772223+0000", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0417e-05 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772228+0000", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.22e-06 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772285+0000", 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7426000000000002e-05 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.534 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772384+0000", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8680999999999999e-05 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.772482+0000", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8018999999999998e-05 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.773579+0000", 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001096631 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.535 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.773856+0000", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00027763799999999997 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600133+0000", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.826276379999999 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600170+0000", 2024-04-15T18:35:43.536 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7366000000000001e-05 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600211+0000", 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0748999999999999e-05 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34096 6.5 6:a6e479ed:::benchmark_data_smithi159_139342_object34095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.820962+0000", 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.68428486900001, 2024-04-15T18:35:43.537 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.779334172, 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34096 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.820962+0000", 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.538 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.820962+0000", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.820963+0000", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5700000000000001e-07 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821012+0000", 2024-04-15T18:35:43.539 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8458999999999998e-05 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821022+0000", 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0913e-05 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821028+0000", 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7089999999999999e-06 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.540 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821068+0000", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9576999999999997e-05 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821152+0000", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4091999999999994e-05 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821240+0000", 2024-04-15T18:35:43.541 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7917999999999994e-05 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.821996+0000", 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075638500000000004 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.822328+0000", 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033185199999999999 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.542 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600240+0000", 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.777912384 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600273+0000", 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2947999999999997e-05 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.600296+0000", 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3078000000000002e-05 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.543 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34098 6.5 6:adccc0ea:::benchmark_data_smithi159_139342_object34097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.876228+0000", 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.62901894800001, 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.726299821, 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.544 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34098 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876228+0000", 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876228+0000", 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.545 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876229+0000", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1999999999999998e-07 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876284+0000", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4570000000000001e-05 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876295+0000", 2024-04-15T18:35:43.546 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1788e-05 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876301+0000", 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8409999999999996e-06 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876340+0000", 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8277e-05 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.547 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876422+0000", 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2149000000000002e-05 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.876525+0000", 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010376499999999999 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.877341+0000", 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00081520099999999999 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.548 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.877662+0000", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000321293 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.602411+0000", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.724748841 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.602482+0000", 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1522999999999999e-05 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.549 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.602528+0000", 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5652999999999998e-05 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34123 6.5 6:ac51d62f:::benchmark_data_smithi159_139342_object34122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.939519+0000", 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.56572832800001, 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.663704541, 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.550 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34123 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939519+0000", 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939519+0000", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939520+0000", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0860000000000001e-06 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939574+0000", 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4129999999999998e-05 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939585+0000", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1015e-05 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939591+0000", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8189999999999997e-06 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939633+0000", 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2168e-05 2024-04-15T18:35:43.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939728+0000", 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4569999999999997e-05 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.939827+0000", 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9242999999999998e-05 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.940880+0000", 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010535919999999999 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.941183+0000", 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030216299999999999 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603124+0000", 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.661941498999999 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603181+0000", 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6777000000000002e-05 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603223+0000", 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2478999999999998e-05 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.556 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34135 6.5 6:a4b673af:::benchmark_data_smithi159_139342_object34134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:02.997411+0000", 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.50783617799999, 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.606433251, 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34135 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997411+0000", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997411+0000", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997412+0000", 2024-04-15T18:35:43.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.063e-06 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997436+0000", 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4148e-05 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997447+0000", 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0888e-05 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997462+0000", 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4986e-05 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997506+0000", 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4527000000000002e-05 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997608+0000", 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010124 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.997697+0000", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9553000000000001e-05 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.998726+0000", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010288630000000001 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:02.999134+0000", 2024-04-15T18:35:43.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00040801299999999998 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603741+0000", 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.604606513 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603794+0000", 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3658e-05 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.603844+0000", 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9799000000000002e-05 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34167 6.5 6:a0e5258a:::benchmark_data_smithi159_139342_object34166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.216245+0000", 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.28900254800001, 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.388352202, 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.563 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34167 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216245+0000", 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216245+0000", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216246+0000", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3060000000000001e-06 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216290+0000", 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3819999999999997e-05 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216303+0000", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3548e-05 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216310+0000", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4110000000000001e-06 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216366+0000", 2024-04-15T18:35:43.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6131000000000002e-05 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216479+0000", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011292400000000001 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.216572+0000", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3529999999999994e-05 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.217770+0000", 2024-04-15T18:35:43.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011982869999999999 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.217990+0000", 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00021903600000000001 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.604533+0000", 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.386543156 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.604569+0000", 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6059000000000002e-05 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.604597+0000", 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7994e-05 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34170 6.5 6:af9ed0a4:::benchmark_data_smithi159_139342_object34169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.569 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.224204+0000", 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.28104288599999, 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.38132341, 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34170 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224204+0000", 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224204+0000", 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224205+0000", 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.23e-07 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224260+0000", 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5198000000000002e-05 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224269+0000", 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5390000000000007e-06 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224275+0000", 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8589999999999999e-06 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224324+0000", 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8715999999999999e-05 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224405+0000", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0791000000000006e-05 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.224507+0000", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102258 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.225488+0000", 2024-04-15T18:35:43.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098056500000000008 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.225826+0000", 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033769200000000001 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.605448+0000", 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.379622331 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.605494+0000", 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5770999999999999e-05 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.605528+0000", 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3967000000000002e-05 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34230 6.5 6:a12f0a8e:::benchmark_data_smithi159_139342_object34229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.578 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.378764+0000", 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.12648288400001, 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.227748543000001, 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34230 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378764+0000", 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378764+0000", 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378766+0000", 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8500000000000001e-06 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378814+0000", 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7692999999999999e-05 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378841+0000", 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7713e-05 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378857+0000", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5216e-05 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.378994+0000", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000137491 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.379236+0000", 2024-04-15T18:35:43.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000242258 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.379376+0000", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013951100000000001 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.380546+0000", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001170557 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.380858+0000", 2024-04-15T18:35:43.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00031121600000000001 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606390+0000", 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.225532380000001 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606463+0000", 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2928000000000004e-05 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606513+0000", 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.973e-05 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34231 6.5 6:a2e8723b:::benchmark_data_smithi159_139342_object34230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.442017+0000", 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.063230492, 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.164743154, 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.585 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34231 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442017+0000", 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.586 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442017+0000", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442017+0000", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699999999999997e-07 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442070+0000", 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2541999999999999e-05 2024-04-15T18:35:43.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442080+0000", 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0101999999999999e-05 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442086+0000", 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4970000000000001e-06 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442133+0000", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6971999999999997e-05 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442260+0000", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012717300000000001 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.442409+0000", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000148787 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.443564+0000", 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011557080000000001 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.443611+0000", 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6943000000000001e-05 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606643+0000", 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.163031809 2024-04-15T18:35:43.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606698+0000", 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5025000000000003e-05 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.606760+0000", 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1679000000000004e-05 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34252 6.5 6:a56ffaf3:::benchmark_data_smithi159_139342_object34251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.482351+0000", 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 160.02289576199999, 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.125747388000001, 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34252 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.592 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482351+0000", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482351+0000", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482352+0000", 2024-04-15T18:35:43.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2900000000000002e-07 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482420+0000", 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8112000000000001e-05 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482431+0000", 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0276e-05 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482436+0000", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5350000000000004e-06 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482488+0000", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2154999999999998e-05 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482570+0000", 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1914000000000002e-05 2024-04-15T18:35:43.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.482657+0000", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7311999999999993e-05 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.483863+0000", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012058209999999999 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.484225+0000", 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00036224199999999998 2024-04-15T18:35:43.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608034+0000", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 14.123808126 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608071+0000", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7063000000000001e-05 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608099+0000", 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8002999999999999e-05 2024-04-15T18:35:43.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34257 6.5 6:a387a011:::benchmark_data_smithi159_139342_object34256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:03.646775+0000", 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 159.85847158499999, 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 13.961535507000001, 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.598 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34257 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646775+0000", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646775+0000", 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646777+0000", 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.204e-06 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646818+0000", 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0914000000000002e-05 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646829+0000", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1331999999999999e-05 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646834+0000", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5509999999999999e-06 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.646921+0000", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6067000000000001e-05 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.647015+0000", 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4347999999999999e-05 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.647110+0000", 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5402000000000006e-05 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.648276+0000", 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001165695 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:03.648639+0000", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00036334699999999998 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608155+0000", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 13.959515242 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608229+0000", 2024-04-15T18:35:43.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4802000000000007e-05 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608311+0000", 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1602999999999998e-05 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:34275 6.5 6:a82b7281:::benchmark_data_smithi159_139342_object34274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e552)", 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:04.308566+0000", 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 159.19668129199999, 2024-04-15T18:35:43.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 13.299918641, 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 34275 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308566+0000", 2024-04-15T18:35:43.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308566+0000", 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308568+0000", 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2570000000000002e-06 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308697+0000", 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129121 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308727+0000", 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0171999999999998e-05 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308749+0000", 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1926999999999999e-05 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308883+0000", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133645 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.308996+0000", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000113188 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 6,9", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.309099+0000", 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010247999999999999 2024-04-15T18:35:43.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.310652+0000", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015532149999999999 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:04.311067+0000", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00041543999999999999 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608360+0000", 2024-04-15T18:35:43.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 13.297292793 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608434+0000", 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3615999999999994e-05 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:17.608484+0000", 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0787e-05 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:87343 6.5 6.3e478735 (undecoded) ondisk+write+known_if_redirected e574)", 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:34:03.405447+0000", 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 100.099799733, 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.99944986099999999, 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87343 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.611 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405447+0000", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405447+0000", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405448+0000", 2024-04-15T18:35:43.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2600000000000002e-07 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405536+0000", 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7979000000000006e-05 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405547+0000", 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0711e-05 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405554+0000", 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2389999999999996e-06 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405600+0000", 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5769000000000002e-05 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.405608+0000", 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0099999999999995e-06 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:04.404861+0000", 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.99925290099999997 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:04.404897+0000", 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6526000000000002e-05 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:87364 6.5 6.7625d675 (undecoded) ondisk+write+known_if_redirected e574)", 2024-04-15T18:35:43.615 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:34:03.682520+0000", 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 99.822727154999995, 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.72242963900000001, 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 87364 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682520+0000", 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682520+0000", 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682526+0000", 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4520000000000002e-06 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682580+0000", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4032e-05 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682593+0000", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2238e-05 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682600+0000", 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3119999999999996e-06 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682651+0000", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0605000000000001e-05 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:03.682658+0000", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8699999999999992e-06 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:04.404938+0000", 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.72227946499999995 2024-04-15T18:35:43.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:04.404950+0000", 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1664999999999999e-05 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:89876 6.8 6:11dcbba0:::benchmark_data_smithi159_139342_object89875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:34:08.973137+0000", 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 94.532110041999999, 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50968392600000001, 2024-04-15T18:35:43.620 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 89876 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973137+0000", 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973137+0000", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973138+0000", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0690000000000001e-06 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973194+0000", 2024-04-15T18:35:43.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5812e-05 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973207+0000", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3543999999999999e-05 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973216+0000", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1999999999999994e-06 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973264+0000", 2024-04-15T18:35:43.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8741e-05 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973373+0000", 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108212 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,11", 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.973479+0000", 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106232 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:35:43.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.979197+0000", 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0057184640000000004 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:08.982627+0000", 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003429629 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:09.482681+0000", 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.50005420700000003 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:09.482759+0000", 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7451000000000001e-05 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:34:09.482821+0000", 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2365000000000003e-05 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:35:43.626 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:35:44.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:44.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:44.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:43 smithi159 ceph-mon[45336]: osdmap e599: 12 total, 11 up, 11 in 2024-04-15T18:35:44.156 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:44.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:44.163 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:44.163 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:44.163 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:43 smithi049 ceph-mon[40087]: osdmap e599: 12 total, 11 up, 11 in 2024-04-15T18:35:44.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:44.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:44.219 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:43 smithi082 ceph-mon[44206]: osdmap e599: 12 total, 11 up, 11 in 2024-04-15T18:35:44.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:35:44.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:44.299 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:44.323 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:44.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:44.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:44.462 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:44.493 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:44.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:44.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:44.641 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:35:44.647 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:44.675 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:44.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:44.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:44.848 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:44.881 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1564,"stamp":"2024-04-15T18:35:41.810726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1784807992,"num_objects":27228,"num_object_clones":0,"num_object_copies":81684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27228,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":152709,"num_write_kb":5755003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17819,"num_bytes_recovered":1168796864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152826,"ondisk_log_size":152826,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9372280,"kb_used_data":5101472,"kb_used_omap":176,"kb_used_meta":4270479,"kb_avail":1021824392,"statfs":{"total":1055945392128,"available":1046348177408,"internally_reserved":0,"allocated":5223907328,"data_stored":5220641196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":180226,"internal_metadata":4372971518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,6,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-404422656,"num_objects":-6171,"num_object_clones":0,"num_object_copies":-18513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6171,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":165806080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.048240"},"pg_stats":[{"pgid":"6.f","version":"597'9565","reported_seq":9687,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.368470+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:35:37.368470+0000","last_peered":"2024-04-15T18:35:37.368470+0000","last_clean":"2024-04-15T18:35:37.368470+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:35:37.368470+0000","last_undegraded":"2024-04-15T18:35:37.368470+0000","last_fullsized":"2024-04-15T18:35:37.368470+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9565,"log_dups_size":0,"ondisk_log_size":9565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9565,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"597'9367","reported_seq":9561,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.782342+0000","last_change":"2024-04-15T18:35:25.838944+0000","last_active":"2024-04-15T18:35:37.782342+0000","last_peered":"2024-04-15T18:35:37.782342+0000","last_clean":"2024-04-15T18:35:37.782342+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:35:37.782342+0000","last_undegraded":"2024-04-15T18:35:37.782342+0000","last_fullsized":"2024-04-15T18:35:37.782342+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'5863","last_scrub_stamp":"2024-04-15T18:34:16.326617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:16.326617+0000","objects_scrubbed":799,"log_size":9367,"log_dups_size":0,"ondisk_log_size":9367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9367,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"597'9630","reported_seq":9798,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.808751+0000","last_change":"2024-04-15T18:33:37.267620+0000","last_active":"2024-04-15T18:35:37.808751+0000","last_peered":"2024-04-15T18:35:37.808751+0000","last_clean":"2024-04-15T18:35:37.808751+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:35:37.808751+0000","last_undegraded":"2024-04-15T18:35:37.808751+0000","last_fullsized":"2024-04-15T18:35:37.808751+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'3631","last_scrub_stamp":"2024-04-15T18:33:37.267494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:37.267494+0000","objects_scrubbed":3631,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195352,"num_objects":1774,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9630,"num_write_kb":364865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"597'9675","reported_seq":17352,"reported_epoch":597,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:37.789906+0000","last_change":"2024-04-15T18:34:05.653272+0000","last_active":"2024-04-15T18:35:37.789906+0000","last_peered":"2024-04-15T18:35:37.789906+0000","last_clean":"2024-04-15T18:33:36.501626+0000","last_became_active":"2024-04-15T18:34:05.621869+0000","last_became_peered":"2024-04-15T18:34:05.621869+0000","last_unstale":"2024-04-15T18:35:37.789906+0000","last_undegraded":"2024-04-15T18:34:05.439986+0000","last_fullsized":"2024-04-15T18:34:05.405814+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9675,"log_dups_size":0,"ondisk_log_size":9675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116981760,"num_objects":1785,"num_object_clones":0,"num_object_copies":5355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9675,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":246284288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"597'9783","reported_seq":9906,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:41.274363+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:35:41.274363+0000","last_peered":"2024-04-15T18:35:41.274363+0000","last_clean":"2024-04-15T18:35:41.274363+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:35:41.274363+0000","last_undegraded":"2024-04-15T18:35:41.274363+0000","last_fullsized":"2024-04-15T18:35:41.274363+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99024896,"num_objects":1511,"num_object_clones":0,"num_object_copies":4533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9783,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"597'9452","reported_seq":17994,"reported_epoch":597,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:37.813364+0000","last_change":"2024-04-15T18:34:05.605023+0000","last_active":"2024-04-15T18:35:37.813364+0000","last_peered":"2024-04-15T18:35:37.813364+0000","last_clean":"2024-04-15T18:33:36.551954+0000","last_became_active":"2024-04-15T18:34:05.572185+0000","last_became_peered":"2024-04-15T18:34:05.572185+0000","last_unstale":"2024-04-15T18:35:37.813364+0000","last_undegraded":"2024-04-15T18:34:05.446261+0000","last_fullsized":"2024-04-15T18:34:05.405630+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":274,"log_size":9452,"log_dups_size":0,"ondisk_log_size":9452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9293,"num_write_kb":352704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4200,"num_bytes_recovered":274989056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,9],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":586}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"597'9263","reported_seq":9427,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.366128+0000","last_change":"2024-04-15T18:33:34.960140+0000","last_active":"2024-04-15T18:35:37.366128+0000","last_peered":"2024-04-15T18:35:37.366128+0000","last_clean":"2024-04-15T18:35:37.366128+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:35:37.366128+0000","last_undegraded":"2024-04-15T18:35:37.366128+0000","last_fullsized":"2024-04-15T18:35:37.366128+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'3367","last_scrub_stamp":"2024-04-15T18:33:34.960015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:34.960015+0000","objects_scrubbed":3347,"log_size":9263,"log_dups_size":0,"ondisk_log_size":9263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118685696,"num_objects":1811,"num_object_clones":0,"num_object_copies":5433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"597'9634","reported_seq":9756,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:41.295494+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:35:41.295494+0000","last_peered":"2024-04-15T18:35:41.295494+0000","last_clean":"2024-04-15T18:35:41.295494+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:35:41.295494+0000","last_undegraded":"2024-04-15T18:35:41.295494+0000","last_fullsized":"2024-04-15T18:35:41.295494+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9634,"log_dups_size":0,"ondisk_log_size":9634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":4626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"597'9539","reported_seq":9755,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.786617+0000","last_change":"2024-04-15T18:35:25.111330+0000","last_active":"2024-04-15T18:35:37.786617+0000","last_peered":"2024-04-15T18:35:37.786617+0000","last_clean":"2024-04-15T18:35:37.786617+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:35:37.786617+0000","last_undegraded":"2024-04-15T18:35:37.786617+0000","last_fullsized":"2024-04-15T18:35:37.786617+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'8959","last_scrub_stamp":"2024-04-15T18:35:25.111058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:25.111058+0000","objects_scrubbed":2343,"log_size":9539,"log_dups_size":0,"ondisk_log_size":9539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:48.179384+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114622464,"num_objects":1749,"num_object_clones":0,"num_object_copies":5247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":361216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"597'9647","reported_seq":19331,"reported_epoch":597,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:41.292849+0000","last_change":"2024-04-15T18:34:05.758545+0000","last_active":"2024-04-15T18:35:41.292849+0000","last_peered":"2024-04-15T18:35:41.292849+0000","last_clean":"2024-04-15T18:33:36.504527+0000","last_became_active":"2024-04-15T18:34:05.717737+0000","last_became_peered":"2024-04-15T18:34:05.717737+0000","last_unstale":"2024-04-15T18:35:41.292849+0000","last_undegraded":"2024-04-15T18:34:05.437288+0000","last_fullsized":"2024-04-15T18:34:05.404259+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9647,"log_dups_size":0,"ondisk_log_size":9647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107413504,"num_objects":1639,"num_object_clones":0,"num_object_copies":4917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4778,"num_bytes_recovered":313131008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[{"shards":"7,9","objects":936}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"597'9296","reported_seq":9483,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.388457+0000","last_change":"2024-04-15T18:34:10.988860+0000","last_active":"2024-04-15T18:35:37.388457+0000","last_peered":"2024-04-15T18:35:37.388457+0000","last_clean":"2024-04-15T18:35:37.388457+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:35:37.388457+0000","last_undegraded":"2024-04-15T18:35:37.388457+0000","last_fullsized":"2024-04-15T18:35:37.388457+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'5581","last_scrub_stamp":"2024-04-15T18:34:10.988590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:10.988590+0000","objects_scrubbed":5508,"log_size":9296,"log_dups_size":0,"ondisk_log_size":9296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:35:38.259389+0000","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9296,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"597'9760","reported_seq":19998,"reported_epoch":597,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:41.295918+0000","last_change":"2024-04-15T18:34:05.652298+0000","last_active":"2024-04-15T18:35:41.295918+0000","last_peered":"2024-04-15T18:35:41.295918+0000","last_clean":"2024-04-15T18:33:36.554918+0000","last_became_active":"2024-04-15T18:34:05.626705+0000","last_became_peered":"2024-04-15T18:34:05.626705+0000","last_unstale":"2024-04-15T18:35:41.295918+0000","last_undegraded":"2024-04-15T18:34:05.443559+0000","last_fullsized":"2024-04-15T18:34:05.404607+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":1536,"num_object_clones":0,"num_object_copies":4608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9760,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5054,"num_bytes_recovered":331218944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11],"avail_no_missing":["7","11"],"object_location_counts":[{"shards":"7,11","objects":1100}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"597'9425","reported_seq":9611,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.807206+0000","last_change":"2024-04-15T18:34:10.134837+0000","last_active":"2024-04-15T18:35:37.807206+0000","last_peered":"2024-04-15T18:35:37.807206+0000","last_clean":"2024-04-15T18:35:37.807206+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:35:37.807206+0000","last_undegraded":"2024-04-15T18:35:37.807206+0000","last_fullsized":"2024-04-15T18:35:37.807206+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'5620","last_scrub_stamp":"2024-04-15T18:34:10.134713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:10.134713+0000","objects_scrubbed":5606,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117112832,"num_objects":1787,"num_object_clones":0,"num_object_copies":5361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9425,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"597'9742","reported_seq":9927,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:40.342043+0000","last_change":"2024-04-15T18:34:06.446602+0000","last_active":"2024-04-15T18:35:40.342043+0000","last_peered":"2024-04-15T18:35:40.342043+0000","last_clean":"2024-04-15T18:35:40.342043+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:35:40.342043+0000","last_undegraded":"2024-04-15T18:35:40.342043+0000","last_fullsized":"2024-04-15T18:35:40.342043+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'5552","last_scrub_stamp":"2024-04-15T18:34:06.446495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:06.446495+0000","objects_scrubbed":5528,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"597'9462","reported_seq":9585,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.378556+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:35:37.378556+0000","last_peered":"2024-04-15T18:35:37.378556+0000","last_clean":"2024-04-15T18:35:37.378556+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:35:37.378556+0000","last_undegraded":"2024-04-15T18:35:37.378556+0000","last_fullsized":"2024-04-15T18:35:37.378556+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9462,"log_dups_size":0,"ondisk_log_size":9462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115736576,"num_objects":1766,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"597'9447","reported_seq":9570,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:37.776177+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:35:37.776177+0000","last_peered":"2024-04-15T18:35:37.776177+0000","last_clean":"2024-04-15T18:35:37.776177+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:35:37.776177+0000","last_undegraded":"2024-04-15T18:35:37.776177+0000","last_fullsized":"2024-04-15T18:35:37.776177+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114884608,"num_objects":1753,"num_object_clones":0,"num_object_copies":5259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1487,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-04-15T18:35:32.772149+0000","last_change":"2024-04-15T18:34:07.451425+0000","last_active":"2024-04-15T18:35:32.772149+0000","last_peered":"2024-04-15T18:35:32.772149+0000","last_clean":"2024-04-15T18:35:32.772149+0000","last_became_active":"2024-04-15T18:34:07.451006+0000","last_became_peered":"2024-04-15T18:34:07.451006+0000","last_unstale":"2024-04-15T18:35:32.772149+0000","last_undegraded":"2024-04-15T18:35:32.772149+0000","last_fullsized":"2024-04-15T18:35:32.772149+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:33:18.376351+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:33:18.376351+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1784217624,"num_objects":27226,"num_object_clones":0,"num_object_copies":81678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27226,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152528,"num_write_kb":5752065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17807,"num_bytes_recovered":1165623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6079262720,"data_stored":6079250504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152687,"ondisk_log_size":152687,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926142,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691204,"kb_used_data":212988,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93053948,"statfs":{"total":95995035648,"available":95287242752,"internally_reserved":0,"allocated":218099712,"data_stored":217801300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":489683396},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187779,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696384,"kb_used_data":340344,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93048768,"statfs":{"total":95995035648,"available":95281938432,"internally_reserved":0,"allocated":348512256,"data_stored":348213344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934276,"kb_used_data":515708,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92810876,"statfs":{"total":95995035648,"available":95038337024,"internally_reserved":0,"allocated":528084992,"data_stored":527786600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":428538304},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578364,"kb_used_data":312564,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93166788,"statfs":{"total":95995035648,"available":95402790912,"internally_reserved":0,"allocated":320065536,"data_stored":319770696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168913,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934916,"kb_used_data":516412,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92810236,"statfs":{"total":95995035648,"available":95037681664,"internally_reserved":0,"allocated":528805888,"data_stored":528507496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575219,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899648,"kb_used_data":453368,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92845504,"statfs":{"total":95995035648,"available":95073796096,"internally_reserved":0,"allocated":464248832,"data_stored":463949920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738741,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789884,"kb_used_data":317236,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92955268,"statfs":{"total":95995035648,"available":95186194432,"internally_reserved":0,"allocated":324849664,"data_stored":324554824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193991,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730300,"kb_used_data":505332,"kb_used_omap":114,"kb_used_meta":224845,"kb_avail":93014852,"statfs":{"total":95995035648,"available":95247208448,"internally_reserved":0,"allocated":517459968,"data_stored":517165128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":230241521},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609121,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1384636,"kb_used_data":904436,"kb_used_omap":44,"kb_used_meta":480147,"kb_avail":92360516,"statfs":{"total":95995035648,"available":94577168384,"internally_reserved":0,"allocated":926142464,"data_stored":925847624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":491671082},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986527,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881468,"kb_used_data":453236,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92863684,"statfs":{"total":95995035648,"available":95092412416,"internally_reserved":0,"allocated":464113664,"data_stored":463818824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":851200,"kb_used_data":569848,"kb_used_omap":4,"kb_used_meta":281339,"kb_avail":92893952,"statfs":{"total":95995035648,"available":95123406848,"internally_reserved":0,"allocated":583524352,"data_stored":583225440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":288091841},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322699264,"data_stored":322699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":457773056,"data_stored":457768984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":515309568,"data_stored":515309568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":924647424,"data_stored":924647424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":459341824,"data_stored":459341824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":574558208,"data_stored":574554136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":513474560,"data_stored":513474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":318701568,"data_stored":318701568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":515244032,"data_stored":515244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":339611648,"data_stored":339607576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":211091456,"data_stored":211091456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:35:44.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:35:44.893 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:44.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:44.924 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:44.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:45.061 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:45.088 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:45.092 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:44 smithi159 ceph-mon[45336]: pgmap v1565: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3284/74334 objects degraded (4.418%); 17 MiB/s, 277 objects/s recovering 2024-04-15T18:35:45.092 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:44 smithi159 ceph-mon[45336]: 6.e scrub starts 2024-04-15T18:35:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:45.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:45.194 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:45.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:44 smithi049 ceph-mon[40087]: pgmap v1565: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3284/74334 objects degraded (4.418%); 17 MiB/s, 277 objects/s recovering 2024-04-15T18:35:45.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:44 smithi049 ceph-mon[40087]: 6.e scrub starts 2024-04-15T18:35:45.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:44 smithi082 ceph-mon[44206]: pgmap v1565: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 709 op/s; 3284/74334 objects degraded (4.418%); 17 MiB/s, 277 objects/s recovering 2024-04-15T18:35:45.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:44 smithi082 ceph-mon[44206]: 6.e scrub starts 2024-04-15T18:35:45.282 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:46.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:45 smithi159 ceph-mon[45336]: 6.e scrub ok 2024-04-15T18:35:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:45 smithi049 ceph-mon[40087]: 6.e scrub ok 2024-04-15T18:35:46.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:45 smithi082 ceph-mon[44206]: 6.e scrub ok 2024-04-15T18:35:46.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:35:46.379 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:46.546 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:46.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:46.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:46 smithi049 ceph-mon[40087]: pgmap v1567: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 3284/73758 objects degraded (4.452%); 16 MiB/s, 263 objects/s recovering 2024-04-15T18:35:47.004 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:46 smithi159 ceph-mon[45336]: pgmap v1567: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 3284/73758 objects degraded (4.452%); 16 MiB/s, 263 objects/s recovering 2024-04-15T18:35:47.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:46 smithi082 ceph-mon[44206]: pgmap v1567: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 3284/73758 objects degraded (4.452%); 16 MiB/s, 263 objects/s recovering 2024-04-15T18:35:48.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:47 smithi159 ceph-mon[45336]: osdmap e600: 12 total, 11 up, 11 in 2024-04-15T18:35:48.184 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3284/73758 objects degraded (4.452%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:48.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:47 smithi049 ceph-mon[40087]: osdmap e600: 12 total, 11 up, 11 in 2024-04-15T18:35:48.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3284/73758 objects degraded (4.452%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:48.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:47 smithi082 ceph-mon[44206]: osdmap e600: 12 total, 11 up, 11 in 2024-04-15T18:35:48.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3284/73758 objects degraded (4.452%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:48.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:48.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:48.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:48.373 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:48.410 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:48.448 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:48.448 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:48.448 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:48.448 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:48.449 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:48.671 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:48.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:48.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:48.823 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:48.862 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:48.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:48.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: 6.4 scrub starts 2024-04-15T18:35:48.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: 6.4 scrub ok 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: pgmap v1569: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 2489/70080 objects degraded (3.552%); 16 MiB/s, 261 objects/s recovering 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osdmap e601: 12 total, 11 up, 11 in 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/894737209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:48.967 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:49.032 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:35:49.032 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: 6.4 scrub starts 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: 6.4 scrub ok 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: pgmap v1569: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 2489/70080 objects degraded (3.552%); 16 MiB/s, 261 objects/s recovering 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osdmap e601: 12 total, 11 up, 11 in 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/894737209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.063 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.064 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.165 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:49.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: 6.4 scrub starts 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: 6.4 scrub ok 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: pgmap v1569: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 644 op/s; 2489/70080 objects degraded (3.552%); 16 MiB/s, 261 objects/s recovering 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osdmap e601: 12 total, 11 up, 11 in 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/894737209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.320 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:49.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:49.424 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:35:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:49.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:49.457 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:49.667 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:49.816 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:35:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:49.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:35:49.850 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:35:49.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:49.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:49.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.000 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:50.045 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:50.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:50.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:50.219 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:50.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:50.285 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:35:50.321 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:50.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:50.331 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:35:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:35:50.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:50.468 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:51.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:50 smithi159 ceph-mon[45336]: pgmap v1571: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 516 op/s; 1718/65037 objects degraded (2.642%); 16 MiB/s, 260 objects/s recovering 2024-04-15T18:35:51.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:50 smithi159 ceph-mon[45336]: from='client.26492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:51.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:50 smithi049 ceph-mon[40087]: pgmap v1571: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 516 op/s; 1718/65037 objects degraded (2.642%); 16 MiB/s, 260 objects/s recovering 2024-04-15T18:35:51.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:50 smithi049 ceph-mon[40087]: from='client.26492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:51.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:50 smithi082 ceph-mon[44206]: pgmap v1571: 17 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 516 op/s; 1718/65037 objects degraded (2.642%); 16 MiB/s, 260 objects/s recovering 2024-04-15T18:35:51.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:50 smithi082 ceph-mon[44206]: from='client.26492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:51.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:35:51.741 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2024-04-15T18:35:49.436596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1421148728,"num_objects":21679,"num_object_clones":0,"num_object_copies":65037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21679,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":158258,"num_write_kb":5755003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20163,"num_bytes_recovered":1322413248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158375,"ondisk_log_size":158375,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8412856,"kb_used_data":4132232,"kb_used_omap":176,"kb_used_meta":4280207,"kb_avail":1022783816,"statfs":{"total":1055945392128,"available":1047330627584,"internally_reserved":0,"allocated":4231405568,"data_stored":4228510549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":180226,"internal_metadata":4382932990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,0,4,1,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-203096064,"num_objects":-3099,"num_object_clones":0,"num_object_copies":-9297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1566,"num_bytes_recovered":102629376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005466"},"pg_stats":[{"pgid":"6.f","version":"600'9979","reported_seq":10108,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.270193+0000","last_change":"2024-04-15T18:32:30.639972+0000","last_active":"2024-04-15T18:35:47.270193+0000","last_peered":"2024-04-15T18:35:47.270193+0000","last_clean":"2024-04-15T18:35:47.270193+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:35:47.270193+0000","last_undegraded":"2024-04-15T18:35:47.270193+0000","last_fullsized":"2024-04-15T18:35:47.270193+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86179840,"num_objects":1315,"num_object_clones":0,"num_object_copies":3945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"600'9773","reported_seq":9995,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.405996+0000","last_change":"2024-04-15T18:35:45.010064+0000","last_active":"2024-04-15T18:35:47.405996+0000","last_peered":"2024-04-15T18:35:47.405996+0000","last_clean":"2024-04-15T18:35:47.405996+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:35:47.405996+0000","last_undegraded":"2024-04-15T18:35:47.405996+0000","last_fullsized":"2024-04-15T18:35:47.405996+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'9681","last_scrub_stamp":"2024-04-15T18:35:45.009884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:45.009884+0000","objects_scrubbed":1421,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:10.298794+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86179840,"num_objects":1315,"num_object_clones":0,"num_object_copies":3945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"600'10046","reported_seq":10220,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.403320+0000","last_change":"2024-04-15T18:33:37.267620+0000","last_active":"2024-04-15T18:35:47.403320+0000","last_peered":"2024-04-15T18:35:47.403320+0000","last_clean":"2024-04-15T18:35:47.403320+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:35:47.403320+0000","last_undegraded":"2024-04-15T18:35:47.403320+0000","last_fullsized":"2024-04-15T18:35:47.403320+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'3631","last_scrub_stamp":"2024-04-15T18:33:37.267494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:37.267494+0000","objects_scrubbed":3631,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932376,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10046,"num_write_kb":364865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"600'10068","reported_seq":19191,"reported_epoch":600,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:47.831876+0000","last_change":"2024-04-15T18:34:05.653272+0000","last_active":"2024-04-15T18:35:47.831876+0000","last_peered":"2024-04-15T18:35:47.831876+0000","last_clean":"2024-04-15T18:33:36.501626+0000","last_became_active":"2024-04-15T18:34:05.621869+0000","last_became_peered":"2024-04-15T18:34:05.621869+0000","last_unstale":"2024-04-15T18:35:47.831876+0000","last_undegraded":"2024-04-15T18:34:05.439986+0000","last_fullsized":"2024-04-15T18:34:05.405814+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10068,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4495,"num_bytes_recovered":293470208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":332}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"599'9966","reported_seq":10093,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-15T18:35:46.291168+0000","last_change":"2024-04-15T18:32:30.475224+0000","last_active":"2024-04-15T18:35:46.291168+0000","last_peered":"2024-04-15T18:35:46.291168+0000","last_clean":"2024-04-15T18:35:46.291168+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:35:46.291168+0000","last_undegraded":"2024-04-15T18:35:46.291168+0000","last_fullsized":"2024-04-15T18:35:46.291168+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9966,"log_dups_size":0,"ondisk_log_size":9966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87031808,"num_objects":1328,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"600'9857","reported_seq":20063,"reported_epoch":600,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:47.631384+0000","last_change":"2024-04-15T18:34:05.605023+0000","last_active":"2024-04-15T18:35:47.631384+0000","last_peered":"2024-04-15T18:35:47.631384+0000","last_clean":"2024-04-15T18:33:36.551954+0000","last_became_active":"2024-04-15T18:34:05.572185+0000","last_became_peered":"2024-04-15T18:34:05.572185+0000","last_unstale":"2024-04-15T18:35:47.631384+0000","last_undegraded":"2024-04-15T18:34:05.446261+0000","last_fullsized":"2024-04-15T18:34:05.405630+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":274,"log_size":9857,"log_dups_size":0,"ondisk_log_size":9857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86769664,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":352704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5029,"num_bytes_recovered":329318400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,9],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1010}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"600'9716","reported_seq":9885,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.410485+0000","last_change":"2024-04-15T18:33:34.960140+0000","last_active":"2024-04-15T18:35:47.410485+0000","last_peered":"2024-04-15T18:35:47.410485+0000","last_clean":"2024-04-15T18:35:47.410485+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:35:47.410485+0000","last_undegraded":"2024-04-15T18:35:47.410485+0000","last_fullsized":"2024-04-15T18:35:47.410485+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'3367","last_scrub_stamp":"2024-04-15T18:33:34.960015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:34.960015+0000","objects_scrubbed":3347,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88997888,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"599'9859","reported_seq":9986,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-15T18:35:46.299561+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:35:46.299561+0000","last_peered":"2024-04-15T18:35:46.299561+0000","last_clean":"2024-04-15T18:35:46.299561+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:35:46.299561+0000","last_undegraded":"2024-04-15T18:35:46.299561+0000","last_fullsized":"2024-04-15T18:35:46.299561+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86310912,"num_objects":1317,"num_object_clones":0,"num_object_copies":3951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9859,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"600'9927","reported_seq":10149,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.604973+0000","last_change":"2024-04-15T18:35:25.111330+0000","last_active":"2024-04-15T18:35:47.604973+0000","last_peered":"2024-04-15T18:35:47.604973+0000","last_clean":"2024-04-15T18:35:47.604973+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:35:47.604973+0000","last_undegraded":"2024-04-15T18:35:47.604973+0000","last_fullsized":"2024-04-15T18:35:47.604973+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'8959","last_scrub_stamp":"2024-04-15T18:35:25.111058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:25.111058+0000","objects_scrubbed":2343,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:48.179384+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":4083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":361216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"599'9857","reported_seq":20261,"reported_epoch":599,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:35:46.308582+0000","last_change":"2024-04-15T18:34:05.758545+0000","last_active":"2024-04-15T18:35:46.308582+0000","last_peered":"2024-04-15T18:35:46.308582+0000","last_clean":"2024-04-15T18:33:36.504527+0000","last_became_active":"2024-04-15T18:34:05.717737+0000","last_became_peered":"2024-04-15T18:34:05.717737+0000","last_unstale":"2024-04-15T18:35:46.308582+0000","last_undegraded":"2024-04-15T18:34:05.437288+0000","last_fullsized":"2024-04-15T18:34:05.404259+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9857,"log_dups_size":0,"ondisk_log_size":9857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93650944,"num_objects":1429,"num_object_clones":0,"num_object_copies":4287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":344,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9857,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5137,"num_bytes_recovered":336658432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9],"avail_no_missing":["7","9"],"object_location_counts":[{"shards":"7,9","objects":1085}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"600'9728","reported_seq":9942,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.391146+0000","last_change":"2024-04-15T18:35:47.370846+0000","last_active":"2024-04-15T18:35:47.391146+0000","last_peered":"2024-04-15T18:35:47.391146+0000","last_clean":"2024-04-15T18:35:47.391146+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:35:47.391146+0000","last_undegraded":"2024-04-15T18:35:47.391146+0000","last_fullsized":"2024-04-15T18:35:47.391146+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'9726","last_scrub_stamp":"2024-04-15T18:35:47.370689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:47.370689+0000","objects_scrubbed":1326,"log_size":9728,"log_dups_size":0,"ondisk_log_size":9728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:04.173269+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9728,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"599'9965","reported_seq":21080,"reported_epoch":599,"state":"active+undersized+remapped","last_fresh":"2024-04-15T18:35:46.283611+0000","last_change":"2024-04-15T18:35:46.078703+0000","last_active":"2024-04-15T18:35:46.283611+0000","last_peered":"2024-04-15T18:35:46.283611+0000","last_clean":"2024-04-15T18:33:36.554918+0000","last_became_active":"2024-04-15T18:34:05.626705+0000","last_became_peered":"2024-04-15T18:34:05.626705+0000","last_unstale":"2024-04-15T18:35:46.283611+0000","last_undegraded":"2024-04-15T18:35:46.283611+0000","last_fullsized":"2024-04-15T18:34:05.404607+0000","mapping_epoch":575,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87228416,"num_objects":1331,"num_object_clones":0,"num_object_copies":3993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5490,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11],"avail_no_missing":["7","8","11"],"object_location_counts":[{"shards":"7,11","objects":1331}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"600'9830","reported_seq":10022,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.479384+0000","last_change":"2024-04-15T18:34:10.134837+0000","last_active":"2024-04-15T18:35:47.479384+0000","last_peered":"2024-04-15T18:35:47.479384+0000","last_clean":"2024-04-15T18:35:47.479384+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:35:47.479384+0000","last_undegraded":"2024-04-15T18:35:47.479384+0000","last_fullsized":"2024-04-15T18:35:47.479384+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'5620","last_scrub_stamp":"2024-04-15T18:34:10.134713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:10.134713+0000","objects_scrubbed":5606,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"599'9934","reported_seq":10123,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-15T18:35:45.377814+0000","last_change":"2024-04-15T18:34:06.446602+0000","last_active":"2024-04-15T18:35:45.377814+0000","last_peered":"2024-04-15T18:35:45.377814+0000","last_clean":"2024-04-15T18:35:45.377814+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:35:45.377814+0000","last_undegraded":"2024-04-15T18:35:45.377814+0000","last_fullsized":"2024-04-15T18:35:45.377814+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'5552","last_scrub_stamp":"2024-04-15T18:34:06.446495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:06.446495+0000","objects_scrubbed":5528,"log_size":9934,"log_dups_size":0,"ondisk_log_size":9934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"600'9876","reported_seq":10005,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.396440+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:35:47.396440+0000","last_peered":"2024-04-15T18:35:47.396440+0000","last_clean":"2024-04-15T18:35:47.396440+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:35:47.396440+0000","last_undegraded":"2024-04-15T18:35:47.396440+0000","last_fullsized":"2024-04-15T18:35:47.396440+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604672,"num_objects":1352,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9876,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"600'9855","reported_seq":9984,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-04-15T18:35:47.264713+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:35:47.264713+0000","last_peered":"2024-04-15T18:35:47.264713+0000","last_clean":"2024-04-15T18:35:47.264713+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:35:47.264713+0000","last_undegraded":"2024-04-15T18:35:47.264713+0000","last_fullsized":"2024-04-15T18:35:47.264713+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88145920,"num_objects":1345,"num_object_clones":0,"num_object_copies":4035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1491,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-04-15T18:35:43.456483+0000","last_change":"2024-04-15T18:34:07.451425+0000","last_active":"2024-04-15T18:35:43.456483+0000","last_peered":"2024-04-15T18:35:43.456483+0000","last_clean":"2024-04-15T18:35:43.456483+0000","last_became_active":"2024-04-15T18:34:07.451006+0000","last_became_peered":"2024-04-15T18:34:07.451006+0000","last_unstale":"2024-04-15T18:35:43.456483+0000","last_undegraded":"2024-04-15T18:35:43.456483+0000","last_fullsized":"2024-04-15T18:35:43.456483+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:33:18.376351+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:33:18.376351+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1420558360,"num_objects":21677,"num_object_clones":0,"num_object_copies":65031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158077,"num_write_kb":5752065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20151,"num_bytes_recovered":1319239680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5122895872,"data_stored":5122883656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158236,"ondisk_log_size":158236,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926143,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663012,"kb_used_data":184668,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93082140,"statfs":{"total":95995035648,"available":95316111360,"internally_reserved":0,"allocated":189100032,"data_stored":188837775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":489683396},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187781,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619884,"kb_used_data":259684,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93125268,"statfs":{"total":95995035648,"available":95360274432,"internally_reserved":0,"allocated":265916416,"data_stored":265649227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839908,"kb_used_data":425564,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92905244,"statfs":{"total":95995035648,"available":95134969856,"internally_reserved":0,"allocated":435777536,"data_stored":435515299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874573,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603680,"kb_used_data":311508,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93141472,"statfs":{"total":95995035648,"available":95376867328,"internally_reserved":0,"allocated":318984192,"data_stored":318725507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168914,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855592,"kb_used_data":449564,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92889560,"statfs":{"total":95995035648,"available":95118909440,"internally_reserved":0,"allocated":460353536,"data_stored":460091299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575221,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":795184,"kb_used_data":340580,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92949968,"statfs":{"total":95995035648,"available":95180767232,"internally_reserved":0,"allocated":348753920,"data_stored":348486731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738742,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753948,"kb_used_data":277140,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":92991204,"statfs":{"total":95995035648,"available":95222992896,"internally_reserved":0,"allocated":283791360,"data_stored":283532675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193993,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":632172,"kb_used_data":415520,"kb_used_omap":114,"kb_used_meta":216525,"kb_avail":93112980,"statfs":{"total":95995035648,"available":95347691520,"internally_reserved":0,"allocated":425492480,"data_stored":425229363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":221721841},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609123,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165100,"kb_used_data":689056,"kb_used_omap":44,"kb_used_meta":475987,"kb_avail":92580052,"statfs":{"total":95995035648,"available":94801973248,"internally_reserved":0,"allocated":705593344,"data_stored":705330227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":487411242},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986529,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778988,"kb_used_data":346592,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92966164,"statfs":{"total":95995035648,"available":95197351936,"internally_reserved":0,"allocated":354910208,"data_stored":354647091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705388,"kb_used_data":432356,"kb_used_omap":4,"kb_used_meta":273019,"kb_avail":93039764,"statfs":{"total":95995035648,"available":95272718336,"internally_reserved":0,"allocated":442732544,"data_stored":442465355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":279572161},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280821760,"data_stored":280821760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":347082752,"data_stored":347078680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":425525248,"data_stored":425525248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":704905216,"data_stored":704905216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":354091008,"data_stored":354091008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":441520128,"data_stored":441516056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":445775872,"data_stored":445775872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":318767104,"data_stored":318767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":433455104,"data_stored":433455104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":260837376,"data_stored":260833304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":183304192,"data_stored":183304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:35:51.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:35:51.936 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:51.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:51.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:51 smithi049 ceph-mon[40087]: osdmap e602: 12 total, 11 up, 11 in 2024-04-15T18:35:52.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:52.097 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:51 smithi159 ceph-mon[45336]: osdmap e602: 12 total, 11 up, 11 in 2024-04-15T18:35:52.104 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:52.116 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:35:52.116 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:35:52.133 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:52.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:52.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:52.270 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:52.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:51 smithi082 ceph-mon[44206]: osdmap e602: 12 total, 11 up, 11 in 2024-04-15T18:35:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:35:52.378 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:52.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:52.482 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:52.515 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:52.787 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:52.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:52.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:35:52.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:35:52.936 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:52.958 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:35:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:52.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:35:53.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:53.104 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:53.136 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:53.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:52 smithi082 ceph-mon[44206]: pgmap v1573: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 1439/62436 objects degraded (2.305%); 19 MiB/s, 307 objects/s recovering 2024-04-15T18:35:53.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:52 smithi082 ceph-mon[44206]: osdmap e603: 12 total, 11 up, 11 in 2024-04-15T18:35:53.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1439/62436 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:53.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2630627707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:53.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:53.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:52 smithi159 ceph-mon[45336]: pgmap v1573: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 1439/62436 objects degraded (2.305%); 19 MiB/s, 307 objects/s recovering 2024-04-15T18:35:53.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:52 smithi159 ceph-mon[45336]: osdmap e603: 12 total, 11 up, 11 in 2024-04-15T18:35:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1439/62436 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2630627707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:52 smithi049 ceph-mon[40087]: pgmap v1573: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 1439/62436 objects degraded (2.305%); 19 MiB/s, 307 objects/s recovering 2024-04-15T18:35:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:52 smithi049 ceph-mon[40087]: osdmap e603: 12 total, 11 up, 11 in 2024-04-15T18:35:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1439/62436 objects degraded (2.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2630627707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:35:53.225 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:35:53.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:53.280 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:53.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:53.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:53.416 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:53.434 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:35:53.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:53.472 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:53.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:53.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:53.604 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:54.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:53 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:54.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:53 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:54.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:53 smithi159 ceph-mon[45336]: osdmap e604: 12 total, 11 up, 11 in 2024-04-15T18:35:54.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:53 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:54.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:53 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:54.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:53 smithi049 ceph-mon[40087]: osdmap e604: 12 total, 11 up, 11 in 2024-04-15T18:35:54.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:53 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:35:54.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:53 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:35:54.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:53 smithi082 ceph-mon[44206]: osdmap e604: 12 total, 11 up, 11 in 2024-04-15T18:35:54.354 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:54.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:54.459 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:54.493 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:54.540 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:54.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:54.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:54.712 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:54.852 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:54.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:54.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:54.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:54 smithi049 ceph-mon[40087]: pgmap v1575: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 607/56565 objects degraded (1.073%); 18 MiB/s, 313 objects/s recovering 2024-04-15T18:35:54.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:54 smithi049 ceph-mon[40087]: 6.b scrub starts 2024-04-15T18:35:54.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:54 smithi049 ceph-mon[40087]: osdmap e605: 12 total, 11 up, 11 in 2024-04-15T18:35:54.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:54 smithi049 ceph-mon[40087]: 6.b scrub ok 2024-04-15T18:35:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:55.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:54 smithi159 ceph-mon[45336]: pgmap v1575: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 607/56565 objects degraded (1.073%); 18 MiB/s, 313 objects/s recovering 2024-04-15T18:35:55.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:54 smithi159 ceph-mon[45336]: 6.b scrub starts 2024-04-15T18:35:55.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:54 smithi159 ceph-mon[45336]: osdmap e605: 12 total, 11 up, 11 in 2024-04-15T18:35:55.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:54 smithi159 ceph-mon[45336]: 6.b scrub ok 2024-04-15T18:35:55.147 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:35:55.147 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:35:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:55.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:55.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:35:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:54 smithi082 ceph-mon[44206]: pgmap v1575: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 607/56565 objects degraded (1.073%); 18 MiB/s, 313 objects/s recovering 2024-04-15T18:35:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:54 smithi082 ceph-mon[44206]: 6.b scrub starts 2024-04-15T18:35:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:54 smithi082 ceph-mon[44206]: osdmap e605: 12 total, 11 up, 11 in 2024-04-15T18:35:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:54 smithi082 ceph-mon[44206]: 6.b scrub ok 2024-04-15T18:35:55.312 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:55.414 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:35:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:55.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:35:55.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:55.553 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:55.577 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:55.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:55.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:55.741 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:55.767 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:55.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:55.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:35:55.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:55.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:35:55.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:35:55.934 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:55.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:35:56.022 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:35:56.060 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:35:56.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:35:56.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:35:56.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:55 smithi159 ceph-mon[45336]: 6.2 scrub starts 2024-04-15T18:35:56.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/789097795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:56.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:35:56.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:55 smithi049 ceph-mon[40087]: 6.2 scrub starts 2024-04-15T18:35:56.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/789097795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:56.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:35:56.268 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:35:56.269 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:35:56.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:55 smithi082 ceph-mon[44206]: 6.2 scrub starts 2024-04-15T18:35:56.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/789097795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:35:56.688 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:56.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:56.701 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:35:56.795 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:35:56.830 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:56.873 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:35:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:56.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:35:56.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:57.021 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:57.122 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: 6.f scrub starts 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: pgmap v1578: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 607/56070 objects degraded (1.083%); 13 MiB/s, 242 objects/s recovering 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: 6.f scrub ok 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: 6.2 scrub ok 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:56 smithi159 ceph-mon[45336]: 6.8 scrub starts 2024-04-15T18:35:57.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: 6.f scrub starts 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: pgmap v1578: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 607/56070 objects degraded (1.083%); 13 MiB/s, 242 objects/s recovering 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: 6.f scrub ok 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: 6.2 scrub ok 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.151 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:56 smithi049 ceph-mon[40087]: 6.8 scrub starts 2024-04-15T18:35:57.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: 6.f scrub starts 2024-04-15T18:35:57.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: pgmap v1578: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 607/56070 objects degraded (1.083%); 13 MiB/s, 242 objects/s recovering 2024-04-15T18:35:57.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: 6.f scrub ok 2024-04-15T18:35:57.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: 6.2 scrub ok 2024-04-15T18:35:57.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:57.280 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:56 smithi082 ceph-mon[44206]: 6.8 scrub starts 2024-04-15T18:35:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:57.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:57.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:35:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:57.849 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:35:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:57.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:35:57.952 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:57.980 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:58.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:35:58.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:35:58.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:35:58.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:58.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: 6.8 scrub ok 2024-04-15T18:35:58.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:35:58.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:35:58.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: 6.a scrub starts 2024-04-15T18:35:58.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 607/56070 objects degraded (1.083%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:58.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: osdmap e606: 12 total, 11 up, 11 in 2024-04-15T18:35:58.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:57 smithi049 ceph-mon[40087]: 6.a scrub ok 2024-04-15T18:35:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: 6.8 scrub ok 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: 6.a scrub starts 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 607/56070 objects degraded (1.083%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: osdmap e606: 12 total, 11 up, 11 in 2024-04-15T18:35:58.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:57 smithi159 ceph-mon[45336]: 6.a scrub ok 2024-04-15T18:35:58.154 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:35:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:58.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:35:58.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: 6.8 scrub ok 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: 6.a scrub starts 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 607/56070 objects degraded (1.083%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: osdmap e606: 12 total, 11 up, 11 in 2024-04-15T18:35:58.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:57 smithi082 ceph-mon[44206]: 6.a scrub ok 2024-04-15T18:35:58.294 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:35:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:58.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:35:58.421 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:58.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:58.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:35:58.791 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.831 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:35:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:58.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:35:58.899 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:35:58.904 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:35:58.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:35:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:164800 6.9 6:96435217:::benchmark_data_smithi159_139342_object74921:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:58.928929+0000", 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.063749262000000001, 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.063777207000000002, 2024-04-15T18:35:58.983 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164800 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928929+0000", 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:35:58.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928929+0000", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928931+0000", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5939999999999999e-06 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928937+0000", 2024-04-15T18:35:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2099999999999998e-06 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928947+0000", 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.01e-05 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.928953+0000", 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1999999999999999e-06 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.929018+0000", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5182e-05 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.929144+0000", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012557700000000001 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.929281+0000", 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013694099999999999 2024-04-15T18:35:58.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.934386+0000", 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051051769999999998 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.976174+0000", 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041787739999999997 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.988 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:164831 6.9 6:93794c21:::benchmark_data_smithi159_139342_object74952:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:58.971099+0000", 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.021579912999999999, 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021675059, 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164831 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971099+0000", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971099+0000", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971100+0000", 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4389999999999999e-06 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971107+0000", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5209999999999997e-06 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971121+0000", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3651e-05 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971129+0000", 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2430000000000005e-06 2024-04-15T18:35:58.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971189+0000", 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9815999999999998e-05 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971291+0000", 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102093 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:35:58.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.971383+0000", 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1710000000000001e-05 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:58.976254+0000", 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048711190000000001 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 2 2024-04-15T18:35:58.993 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:35:59.015 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:58 smithi159 ceph-mon[45336]: pgmap v1579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 542/52830 objects degraded (1.026%); 7.7 MiB/s, 149 objects/s recovering 2024-04-15T18:35:59.015 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:58 smithi159 ceph-mon[45336]: 6.7 scrub starts 2024-04-15T18:35:59.015 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:58 smithi159 ceph-mon[45336]: osdmap e607: 12 total, 11 up, 11 in 2024-04-15T18:35:59.015 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:58 smithi159 ceph-mon[45336]: 6.7 scrub ok 2024-04-15T18:35:59.185 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:35:59.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:58 smithi049 ceph-mon[40087]: pgmap v1579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 542/52830 objects degraded (1.026%); 7.7 MiB/s, 149 objects/s recovering 2024-04-15T18:35:59.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:58 smithi049 ceph-mon[40087]: 6.7 scrub starts 2024-04-15T18:35:59.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:58 smithi049 ceph-mon[40087]: osdmap e607: 12 total, 11 up, 11 in 2024-04-15T18:35:59.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:58 smithi049 ceph-mon[40087]: 6.7 scrub ok 2024-04-15T18:35:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:59.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:35:59.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:58 smithi082 ceph-mon[44206]: pgmap v1579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 533 op/s; 542/52830 objects degraded (1.026%); 7.7 MiB/s, 149 objects/s recovering 2024-04-15T18:35:59.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:58 smithi082 ceph-mon[44206]: 6.7 scrub starts 2024-04-15T18:35:59.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:58 smithi082 ceph-mon[44206]: osdmap e607: 12 total, 11 up, 11 in 2024-04-15T18:35:59.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:58 smithi082 ceph-mon[44206]: 6.7 scrub ok 2024-04-15T18:35:59.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:35:59.344 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:59.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:35:59.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:35:59.720 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:35:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:59.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:35:59.763 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2024-04-15T18:35:58.457951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1036780088,"num_objects":15814,"num_object_clones":0,"num_object_copies":47442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15814,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":164123,"num_write_kb":5755003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21881,"num_bytes_recovered":1399155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":7424640,"kb_used_data":3128600,"kb_used_omap":176,"kb_used_meta":4295503,"kb_avail":1023772032,"statfs":{"total":1055945392128,"available":1048342560768,"internally_reserved":0,"allocated":3203686400,"data_stored":3200522896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":180226,"internal_metadata":4398596094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199294976,"num_objects":-3041,"num_object_clones":0,"num_object_copies":-9123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":14090240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.020541"},"pg_stats":[{"pgid":"6.f","version":"605'10320","reported_seq":10476,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.396131+0000","last_change":"2024-04-15T18:35:55.478313+0000","last_active":"2024-04-15T18:35:57.396131+0000","last_peered":"2024-04-15T18:35:57.396131+0000","last_clean":"2024-04-15T18:35:57.396131+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:35:57.396131+0000","last_undegraded":"2024-04-15T18:35:57.396131+0000","last_fullsized":"2024-04-15T18:35:57.396131+0000","mapping_epoch":541,"log_start":"547'300","ondisk_log_start":"547'300","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10228","last_scrub_stamp":"2024-04-15T18:35:55.478006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.478006+0000","objects_scrubbed":1072,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:01.919682+0000","scrub_duration":252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10320,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"606'10159","reported_seq":10394,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.720408+0000","last_change":"2024-04-15T18:35:45.010064+0000","last_active":"2024-04-15T18:35:57.720408+0000","last_peered":"2024-04-15T18:35:57.720408+0000","last_clean":"2024-04-15T18:35:57.720408+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:35:57.720408+0000","last_undegraded":"2024-04-15T18:35:57.720408+0000","last_fullsized":"2024-04-15T18:35:57.720408+0000","mapping_epoch":541,"log_start":"546'100","ondisk_log_start":"546'100","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'9681","last_scrub_stamp":"2024-04-15T18:35:45.009884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:45.009884+0000","objects_scrubbed":1421,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:10.298794+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60882944,"num_objects":929,"num_object_clones":0,"num_object_copies":2787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"606'10439","reported_seq":10625,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.664841+0000","last_change":"2024-04-15T18:33:37.267620+0000","last_active":"2024-04-15T18:35:57.664841+0000","last_peered":"2024-04-15T18:35:57.664841+0000","last_clean":"2024-04-15T18:35:57.664841+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:35:57.664841+0000","last_undegraded":"2024-04-15T18:35:57.664841+0000","last_fullsized":"2024-04-15T18:35:57.664841+0000","mapping_epoch":541,"log_start":"547'400","ondisk_log_start":"547'400","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'3631","last_scrub_stamp":"2024-04-15T18:33:37.267494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:37.267494+0000","objects_scrubbed":3631,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176728,"num_objects":965,"num_object_clones":0,"num_object_copies":2895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10439,"num_write_kb":364865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"605'10424","reported_seq":21680,"reported_epoch":605,"state":"peering","last_fresh":"2024-04-15T18:35:57.416908+0000","last_change":"2024-04-15T18:35:57.464171+0000","last_active":"2024-04-15T18:35:57.464171+0000","last_peered":"2024-04-15T18:35:57.416908+0000","last_clean":"2024-04-15T18:33:36.501626+0000","last_became_active":"2024-04-15T18:34:05.621869+0000","last_became_peered":"2024-04-15T18:34:05.621869+0000","last_unstale":"2024-04-15T18:35:57.416908+0000","last_undegraded":"2024-04-15T18:35:57.416908+0000","last_fullsized":"2024-04-15T18:34:05.405814+0000","mapping_epoch":606,"log_start":"547'400","ondisk_log_start":"547'400","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67895296,"num_objects":1036,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":337248256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"605'10271","reported_seq":10429,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:56.268496+0000","last_change":"2024-04-15T18:35:54.492058+0000","last_active":"2024-04-15T18:35:56.268496+0000","last_peered":"2024-04-15T18:35:56.268496+0000","last_clean":"2024-04-15T18:35:56.268496+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:35:56.268496+0000","last_undegraded":"2024-04-15T18:35:56.268496+0000","last_fullsized":"2024-04-15T18:35:56.268496+0000","mapping_epoch":541,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10193","last_scrub_stamp":"2024-04-15T18:35:54.491971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:54.491971+0000","objects_scrubbed":1101,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:59.573582+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67043328,"num_objects":1023,"num_object_clones":0,"num_object_copies":3069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10271,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"606'10237","reported_seq":21105,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.801929+0000","last_change":"2024-04-15T18:35:57.590647+0000","last_active":"2024-04-15T18:35:57.801929+0000","last_peered":"2024-04-15T18:35:57.801929+0000","last_clean":"2024-04-15T18:35:57.801929+0000","last_became_active":"2024-04-15T18:35:51.865798+0000","last_became_peered":"2024-04-15T18:35:51.865798+0000","last_unstale":"2024-04-15T18:35:57.801929+0000","last_undegraded":"2024-04-15T18:35:57.801929+0000","last_fullsized":"2024-04-15T18:35:57.801929+0000","mapping_epoch":602,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'10225","last_scrub_stamp":"2024-04-15T18:35:57.590483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:57.590483+0000","objects_scrubbed":966,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:03.223994+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61865984,"num_objects":944,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":352704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5343,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,9],"acting":[5,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"605'10093","reported_seq":10273,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.432895+0000","last_change":"2024-04-15T18:33:34.960140+0000","last_active":"2024-04-15T18:35:57.432895+0000","last_peered":"2024-04-15T18:35:57.432895+0000","last_clean":"2024-04-15T18:35:57.432895+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:35:57.432895+0000","last_undegraded":"2024-04-15T18:35:57.432895+0000","last_fullsized":"2024-04-15T18:35:57.432895+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'3367","last_scrub_stamp":"2024-04-15T18:33:34.960015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:33:34.960015+0000","objects_scrubbed":3347,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10093,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"605'10195","reported_seq":10334,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:56.285284+0000","last_change":"2024-04-15T18:32:30.736047+0000","last_active":"2024-04-15T18:35:56.285284+0000","last_peered":"2024-04-15T18:35:56.285284+0000","last_clean":"2024-04-15T18:35:56.285284+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:35:56.285284+0000","last_undegraded":"2024-04-15T18:35:56.285284+0000","last_fullsized":"2024-04-15T18:35:56.285284+0000","mapping_epoch":541,"log_start":"546'100","ondisk_log_start":"546'100","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"606'10302","reported_seq":10536,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.822344+0000","last_change":"2024-04-15T18:35:25.111330+0000","last_active":"2024-04-15T18:35:57.822344+0000","last_peered":"2024-04-15T18:35:57.822344+0000","last_clean":"2024-04-15T18:35:57.822344+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:35:57.822344+0000","last_undegraded":"2024-04-15T18:35:57.822344+0000","last_fullsized":"2024-04-15T18:35:57.822344+0000","mapping_epoch":541,"log_start":"547'300","ondisk_log_start":"547'300","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'8959","last_scrub_stamp":"2024-04-15T18:35:25.111058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:25.111058+0000","objects_scrubbed":2343,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:48.179384+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64618496,"num_objects":986,"num_object_clones":0,"num_object_copies":2958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10302,"num_write_kb":361216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"605'10212","reported_seq":21325,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:56.292248+0000","last_change":"2024-04-15T18:35:54.474764+0000","last_active":"2024-04-15T18:35:56.292248+0000","last_peered":"2024-04-15T18:35:56.292248+0000","last_clean":"2024-04-15T18:35:56.292248+0000","last_became_active":"2024-04-15T18:35:54.445912+0000","last_became_peered":"2024-04-15T18:35:54.445912+0000","last_unstale":"2024-04-15T18:35:56.292248+0000","last_undegraded":"2024-04-15T18:35:56.292248+0000","last_fullsized":"2024-04-15T18:35:56.292248+0000","mapping_epoch":604,"log_start":"547'200","ondisk_log_start":"547'200","created":541,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5481,"num_bytes_recovered":359202816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"605'10084","reported_seq":10307,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.416956+0000","last_change":"2024-04-15T18:35:47.370846+0000","last_active":"2024-04-15T18:35:57.416956+0000","last_peered":"2024-04-15T18:35:57.416956+0000","last_clean":"2024-04-15T18:35:57.416956+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:35:57.416956+0000","last_undegraded":"2024-04-15T18:35:57.416956+0000","last_fullsized":"2024-04-15T18:35:57.416956+0000","mapping_epoch":541,"log_start":"0'0","ondisk_log_start":"0'0","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'9726","last_scrub_stamp":"2024-04-15T18:35:47.370689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:47.370689+0000","objects_scrubbed":1326,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:04.173269+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10084,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"605'10329","reported_seq":21460,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:56.284037+0000","last_change":"2024-04-15T18:35:47.868268+0000","last_active":"2024-04-15T18:35:56.284037+0000","last_peered":"2024-04-15T18:35:56.284037+0000","last_clean":"2024-04-15T18:35:56.284037+0000","last_became_active":"2024-04-15T18:35:47.841503+0000","last_became_peered":"2024-04-15T18:35:47.841503+0000","last_unstale":"2024-04-15T18:35:56.284037+0000","last_undegraded":"2024-04-15T18:35:56.284037+0000","last_fullsized":"2024-04-15T18:35:56.284037+0000","mapping_epoch":600,"log_start":"547'300","ondisk_log_start":"547'300","created":541,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10329,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5490,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"606'10224","reported_seq":10446,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.755113+0000","last_change":"2024-04-15T18:35:55.811254+0000","last_active":"2024-04-15T18:35:57.755113+0000","last_peered":"2024-04-15T18:35:57.755113+0000","last_clean":"2024-04-15T18:35:57.755113+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:35:57.755113+0000","last_undegraded":"2024-04-15T18:35:57.755113+0000","last_fullsized":"2024-04-15T18:35:57.755113+0000","mapping_epoch":541,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10120","last_scrub_stamp":"2024-04-15T18:35:55.810909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.810909+0000","objects_scrubbed":1092,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:14.214223+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10224,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"605'10282","reported_seq":10483,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:55.379806+0000","last_change":"2024-04-15T18:34:06.446602+0000","last_active":"2024-04-15T18:35:55.379806+0000","last_peered":"2024-04-15T18:35:55.379806+0000","last_clean":"2024-04-15T18:35:55.379806+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:35:55.379806+0000","last_undegraded":"2024-04-15T18:35:55.379806+0000","last_fullsized":"2024-04-15T18:35:55.379806+0000","mapping_epoch":541,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'5552","last_scrub_stamp":"2024-04-15T18:34:06.446495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:34:06.446495+0000","objects_scrubbed":5528,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10282,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"605'10258","reported_seq":10397,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.412129+0000","last_change":"2024-04-15T18:32:30.982510+0000","last_active":"2024-04-15T18:35:57.412129+0000","last_peered":"2024-04-15T18:35:57.412129+0000","last_clean":"2024-04-15T18:35:57.412129+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:35:57.412129+0000","last_undegraded":"2024-04-15T18:35:57.412129+0000","last_fullsized":"2024-04-15T18:35:57.412129+0000","mapping_epoch":541,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"606'10272","reported_seq":10412,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-04-15T18:35:57.724986+0000","last_change":"2024-04-15T18:32:30.478715+0000","last_active":"2024-04-15T18:35:57.724986+0000","last_peered":"2024-04-15T18:35:57.724986+0000","last_clean":"2024-04-15T18:35:57.724986+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:35:57.724986+0000","last_undegraded":"2024-04-15T18:35:57.724986+0000","last_fullsized":"2024-04-15T18:35:57.724986+0000","mapping_epoch":541,"log_start":"546'200","ondisk_log_start":"546'200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1503,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-04-15T18:35:54.444024+0000","last_change":"2024-04-15T18:34:07.451425+0000","last_active":"2024-04-15T18:35:54.444024+0000","last_peered":"2024-04-15T18:35:54.444024+0000","last_clean":"2024-04-15T18:35:54.444024+0000","last_became_active":"2024-04-15T18:34:07.451006+0000","last_became_peered":"2024-04-15T18:34:07.451006+0000","last_unstale":"2024-04-15T18:35:54.444024+0000","last_undegraded":"2024-04-15T18:35:54.444024+0000","last_fullsized":"2024-04-15T18:35:54.444024+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:33:18.376351+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:30:46.224467+0000","last_clean_scrub_stamp":"2024-04-15T18:33:18.376351+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1036189720,"num_objects":15812,"num_object_clones":0,"num_object_copies":47436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15812,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163942,"num_write_kb":5752065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21869,"num_bytes_recovered":1395982336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4063703040,"data_stored":4063690824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926145,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":620900,"kb_used_data":138520,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93124252,"statfs":{"total":95995035648,"available":95359234048,"internally_reserved":0,"allocated":141844480,"data_stored":141559905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187783,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":554344,"kb_used_data":189984,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93190808,"statfs":{"total":95995035648,"available":95427387392,"internally_reserved":0,"allocated":194543616,"data_stored":194254030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514829,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736496,"kb_used_data":330468,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93008656,"statfs":{"total":95995035648,"available":95240863744,"internally_reserved":0,"allocated":338399232,"data_stored":338110166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":415758784},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874575,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":584356,"kb_used_data":282460,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93160796,"statfs":{"total":95995035648,"available":95396655104,"internally_reserved":0,"allocated":289239040,"data_stored":288953526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168916,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":757228,"kb_used_data":342884,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92987924,"statfs":{"total":95995035648,"available":95219634176,"internally_reserved":0,"allocated":351113216,"data_stored":350824150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575223,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692520,"kb_used_data":250400,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93052632,"statfs":{"total":95995035648,"available":95285895168,"internally_reserved":0,"allocated":256409600,"data_stored":256120014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738744,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672808,"kb_used_data":208476,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93072344,"statfs":{"total":95995035648,"available":95306080256,"internally_reserved":0,"allocated":213479424,"data_stored":213193910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193995,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543716,"kb_used_data":318492,"kb_used_omap":114,"kb_used_meta":224845,"kb_avail":93201436,"statfs":{"total":95995035648,"available":95438270464,"internally_reserved":0,"allocated":326135808,"data_stored":325850294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":230241521},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609125,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992676,"kb_used_data":502748,"kb_used_omap":44,"kb_used_meta":489875,"kb_avail":92752476,"statfs":{"total":95995035648,"available":94978535424,"internally_reserved":0,"allocated":514813952,"data_stored":514528438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":501632554},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672244,"kb_used_data":248168,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93072908,"statfs":{"total":95995035648,"available":95306657792,"internally_reserved":0,"allocated":254124032,"data_stored":253834049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703426,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597352,"kb_used_data":316000,"kb_used_omap":4,"kb_used_meta":281339,"kb_avail":93147800,"statfs":{"total":95995035648,"available":95383347200,"internally_reserved":0,"allocated":323584000,"data_stored":323294414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":288091841},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211812352,"data_stored":211812352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":253300736,"data_stored":253296664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321257472,"data_stored":321257472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":513867776,"data_stored":513867776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":251002880,"data_stored":251002880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":316215296,"data_stored":316211224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":333840384,"data_stored":333840384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":283836416,"data_stored":283836416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":323551232,"data_stored":323551232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":190124032,"data_stored":190119960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":138084352,"data_stored":138084352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:35:59.764 INFO:tasks.cephadm.ceph_manager.ceph:PG 6.c is not active+clean 2024-04-15T18:35:59.765 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '6.c', 'version': "605'10424", 'reported_seq': 21680, 'reported_epoch': 605, 'state': 'peering', 'last_fresh': '2024-04-15T18:35:57.416908+0000', 'last_change': '2024-04-15T18:35:57.464171+0000', 'last_active': '2024-04-15T18:35:57.464171+0000', 'last_peered': '2024-04-15T18:35:57.416908+0000', 'last_clean': '2024-04-15T18:33:36.501626+0000', 'last_became_active': '2024-04-15T18:34:05.621869+0000', 'last_became_peered': '2024-04-15T18:34:05.621869+0000', 'last_unstale': '2024-04-15T18:35:57.416908+0000', 'last_undegraded': '2024-04-15T18:35:57.416908+0000', 'last_fullsized': '2024-04-15T18:34:05.405814+0000', 'mapping_epoch': 606, 'log_start': "547'400", 'ondisk_log_start': "547'400", 'created': 541, 'last_epoch_clean': 542, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:32:29.457742+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:32:29.457742+0000', 'last_clean_scrub_stamp': '2024-04-15T18:32:29.457742+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 67895296, 'num_objects': 1036, 'num_object_clones': 0, 'num_object_copies': 3108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10424, 'num_write_kb': 366720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5555, 'num_bytes_recovered': 337248256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3, 8], 'acting': [5, 3, 8], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3, 8], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-15T18:35:59.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:35:59.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:35:59.868 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:35:59.894 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:35:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:35:59.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:35:59.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:35:59 smithi049 ceph-mon[40087]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:35:59.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:00.036 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:00.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:36:00.077 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:00.114 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:00.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:00.134 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:00.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:35:59 smithi159 ceph-mon[45336]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:00.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:00.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:35:59 smithi082 ceph-mon[44206]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:00.446 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:00.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:00.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:00.604 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:00.822 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:00.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:00.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:00.954 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:00.974 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:00.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:01.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:01.111 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:00 smithi049 ceph-mon[40087]: pgmap v1582: 17 pgs: 1 peering, 16 active+clean; 989 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2.2 MiB/s, 101 objects/s recovering 2024-04-15T18:36:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:00 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 542/52830 objects degraded (1.026%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:36:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:00 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:36:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:00 smithi049 ceph-mon[40087]: 6.1 deep-scrub starts 2024-04-15T18:36:01.221 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:01.222 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:00 smithi159 ceph-mon[45336]: pgmap v1582: 17 pgs: 1 peering, 16 active+clean; 989 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2.2 MiB/s, 101 objects/s recovering 2024-04-15T18:36:01.222 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:00 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 542/52830 objects degraded (1.026%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:36:01.222 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:00 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:36:01.223 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:00 smithi159 ceph-mon[45336]: 6.1 deep-scrub starts 2024-04-15T18:36:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:01.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:00 smithi082 ceph-mon[44206]: pgmap v1582: 17 pgs: 1 peering, 16 active+clean; 989 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2.2 MiB/s, 101 objects/s recovering 2024-04-15T18:36:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:00 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 542/52830 objects degraded (1.026%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:36:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:00 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:36:01.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:00 smithi082 ceph-mon[44206]: 6.1 deep-scrub starts 2024-04-15T18:36:01.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:01.366 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:01.947 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:36:01.947 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:36:01.947 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:36:01.947 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:36:01.947 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:36:02.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:01 smithi049 ceph-mon[40087]: 6.9 scrub starts 2024-04-15T18:36:02.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:01 smithi049 ceph-mon[40087]: 6.5 scrub starts 2024-04-15T18:36:02.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:01 smithi049 ceph-mon[40087]: 6.5 scrub ok 2024-04-15T18:36:02.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:01 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:36:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:01 smithi082 ceph-mon[44206]: 6.9 scrub starts 2024-04-15T18:36:02.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:01 smithi082 ceph-mon[44206]: 6.5 scrub starts 2024-04-15T18:36:02.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:01 smithi082 ceph-mon[44206]: 6.5 scrub ok 2024-04-15T18:36:02.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:01 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:36:02.318 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:02.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:02.386 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:02.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:01 smithi159 ceph-mon[45336]: 6.9 scrub starts 2024-04-15T18:36:02.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:01 smithi159 ceph-mon[45336]: 6.5 scrub starts 2024-04-15T18:36:02.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:01 smithi159 ceph-mon[45336]: 6.5 scrub ok 2024-04-15T18:36:02.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:01 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:36:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:02.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:02.524 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:02.979 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:02.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:03.087 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:03.141 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:03.194 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:03.194 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:02 smithi159 ceph-mon[45336]: 6.0 scrub starts 2024-04-15T18:36:03.194 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:02 smithi159 ceph-mon[45336]: 6.0 scrub ok 2024-04-15T18:36:03.194 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:02 smithi159 ceph-mon[45336]: pgmap v1583: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 931 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 1.9 MiB/s, 86 objects/s recovering 2024-04-15T18:36:03.194 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:02 smithi159 ceph-mon[45336]: 6.9 scrub ok 2024-04-15T18:36:03.195 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:02 smithi159 ceph-mon[45336]: 6.c deep-scrub starts 2024-04-15T18:36:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:03.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:02 smithi049 ceph-mon[40087]: 6.0 scrub starts 2024-04-15T18:36:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:02 smithi049 ceph-mon[40087]: 6.0 scrub ok 2024-04-15T18:36:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:02 smithi049 ceph-mon[40087]: pgmap v1583: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 931 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 1.9 MiB/s, 86 objects/s recovering 2024-04-15T18:36:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:02 smithi049 ceph-mon[40087]: 6.9 scrub ok 2024-04-15T18:36:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:02 smithi049 ceph-mon[40087]: 6.c deep-scrub starts 2024-04-15T18:36:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:02 smithi082 ceph-mon[44206]: 6.0 scrub starts 2024-04-15T18:36:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:02 smithi082 ceph-mon[44206]: 6.0 scrub ok 2024-04-15T18:36:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:02 smithi082 ceph-mon[44206]: pgmap v1583: 17 pgs: 1 active+clean+scrubbing+deep, 1 peering, 15 active+clean; 931 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 1.9 MiB/s, 86 objects/s recovering 2024-04-15T18:36:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:02 smithi082 ceph-mon[44206]: 6.9 scrub ok 2024-04-15T18:36:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:02 smithi082 ceph-mon[44206]: 6.c deep-scrub starts 2024-04-15T18:36:03.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:03.338 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:03.364 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:03.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:03.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:03.511 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:03.576 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:36:04.011 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:04.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:36:04.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:36:04.059 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:36:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:36:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:36:04.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:36:04.283 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:36:04.283 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:36:04.283 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:36:05.134 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:36:05.135 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1912,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":607,"num_osds":12,"num_up_osds":11,"osd_up_since":1713206016,"num_in_osds":11,"osd_in_since":1713206043,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+clean+scrubbing+deep","count":2}],"num_pgs":17,"num_pools":2,"num_objects":12962,"data_bytes":849871416,"bytes_used":7077044224,"bytes_avail":1048868347904,"bytes_total":1055945392128,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":1760886,"recovering_keys_per_sec":0,"num_objects_recovered":607,"num_bytes_recovered":14090240,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":715},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":44,"modified":"2024-04-15T18:35:51.437075+0000","services":{}},"progress_events":{}} 2024-04-15T18:36:05.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:05 smithi049 ceph-mon[40087]: pgmap v1584: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1.7 MiB/s, 75 objects/s recovering 2024-04-15T18:36:05.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:05 smithi049 ceph-mon[40087]: 6.d scrub starts 2024-04-15T18:36:05.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:05 smithi049 ceph-mon[40087]: 6.d scrub ok 2024-04-15T18:36:05.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:05 smithi049 ceph-mon[40087]: 6.3 scrub starts 2024-04-15T18:36:05.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3325749456' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:36:05.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:05 smithi159 ceph-mon[45336]: pgmap v1584: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1.7 MiB/s, 75 objects/s recovering 2024-04-15T18:36:05.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:05 smithi159 ceph-mon[45336]: 6.d scrub starts 2024-04-15T18:36:05.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:05 smithi159 ceph-mon[45336]: 6.d scrub ok 2024-04-15T18:36:05.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:05 smithi159 ceph-mon[45336]: 6.3 scrub starts 2024-04-15T18:36:05.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3325749456' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:36:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:05 smithi082 ceph-mon[44206]: pgmap v1584: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1.7 MiB/s, 75 objects/s recovering 2024-04-15T18:36:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:05 smithi082 ceph-mon[44206]: 6.d scrub starts 2024-04-15T18:36:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:05 smithi082 ceph-mon[44206]: 6.d scrub ok 2024-04-15T18:36:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:05 smithi082 ceph-mon[44206]: 6.3 scrub starts 2024-04-15T18:36:05.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3325749456' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.978 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 404/377 [PushReplyOp(5:dd0ee7d7:::benchmark_data_smithi159_116570_object27502:head)])", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 590.51676359199996, 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39222512599999998, 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:05.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469290+0000", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000004e-07 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.469295+0000", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6469999999999997e-06 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:05.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860402+0000", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39110686099999997 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860410+0000", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.860475+0000", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5375000000000003e-05 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:05.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:15.861515+0000", 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010396470000000001 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.b 405/377 [PushReplyOp(5:d997bfb1:::benchmark_data_smithi159_116570_object30349:head)])", 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 589.22896633100004, 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43079569699999998, 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:36:05.982 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757087+0000", 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:05.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757088+0000", 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7000000000000005e-07 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:16.757092+0000", 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6079999999999998e-06 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186476+0000", 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42938425299999999 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186493+0000", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6646e-05 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.186666+0000", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017301299999999999 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.187883+0000", 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001216607 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118576 5.e 5:782c6d1f:::benchmark_data_smithi159_116570_object25416:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 467.27723859100001, 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.461995184, 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118576 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.986 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708815+0000", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708816+0000", 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.75e-06 2024-04-15T18:36:05.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708824+0000", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1359999999999999e-06 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708842+0000", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8142e-05 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708852+0000", 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0181e-05 2024-04-15T18:36:05.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.708919+0000", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7094000000000005e-05 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709064+0000", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014527 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.709192+0000", 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012819900000000001 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712786+0000", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00359377 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.712964+0000", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000177651 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170740+0000", 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45777645 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170776+0000", 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5828000000000003e-05 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170810+0000", 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3713000000000001e-05 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.991 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:118576 5.e e459/358 ondisk, result = 0)", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 467.27297518500001, 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45773745599999999, 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:05.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713078+0000", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713079+0000", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000001e-07 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.713085+0000", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:05.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170680+0000", 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45759575699999999 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170687+0000", 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3509999999999998e-06 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170726+0000", 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9140000000000001e-05 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170736+0000", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0000000000000001e-05 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.170816+0000", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9723999999999998e-05 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:118591 5.e 5:79283452:::benchmark_data_smithi159_116570_object25431:head [delete] snapc 0=[] ondisk+write+known_if_redirected e459)", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:36:05.995 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 467.15911771600003, 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.451595888, 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 118591 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826936+0000", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826937+0000", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.375e-06 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826944+0000", 2024-04-15T18:36:05.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2350000000000002e-06 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826959+0000", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5013e-05 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.826969+0000", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0023e-05 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827033+0000", 2024-04-15T18:36:05.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3962999999999997e-05 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827123+0000", 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9382999999999994e-05 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.827201+0000", 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8164000000000006e-05 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:05.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830060+0000", 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028591860000000001 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:18.830383+0000", 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032335000000000003 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278451+0000", 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44806768699999999 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278489+0000", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7829e-05 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:19.278531+0000", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2679999999999998e-05 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:119911 5.e 5:78ed0cf0:::benchmark_data_smithi159_116570_object26751:head [delete] snapc 0=[] ondisk+write+known_if_redirected e463)", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 457.35746048700003, 2024-04-15T18:36:06.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42595297199999999, 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 119911 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628593+0000", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628594+0000", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.556e-06 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628601+0000", 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628614+0000", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2962999999999999e-05 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628622+0000", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1240000000000005e-06 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628689+0000", 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6351000000000002e-05 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628833+0000", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014472500000000001 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.628965+0000", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013210300000000001 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632755+0000", 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037895329999999999 2024-04-15T18:36:06.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632768+0000", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2626000000000001e-05 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054455+0000", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42168709399999998 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054508+0000", 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3328999999999997e-05 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054546+0000", 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7734999999999997e-05 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:119911 5.e e463/358 ondisk, result = 0)", 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 457.35403359899999, 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42253153399999999, 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.007 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632020+0000", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632021+0000", 2024-04-15T18:36:06.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.632027+0000", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9689999999999999e-06 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054377+0000", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42235067500000001 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054386+0000", 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4190000000000002e-06 2024-04-15T18:36:06.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054439+0000", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3771999999999999e-05 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054448+0000", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4149999999999999e-06 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:29.054551+0000", 2024-04-15T18:36:06.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103403 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186297 5.e 5:7bc39edb:::benchmark_data_smithi159_116570_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 232.90050783000001, 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.38087093300000002, 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:06.011 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186297 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085545+0000", 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085546+0000", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3399999999999997e-07 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085552+0000", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4530000000000001e-06 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085564+0000", 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2542e-05 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.085572+0000", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7610000000000007e-06 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089739+0000", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004166477 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089871+0000", 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000132669 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.089950+0000", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8208999999999996e-05 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.094319+0000", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043699189999999999 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.094340+0000", 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0687999999999999e-05 2024-04-15T18:36:06.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466324+0000", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37198407 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466368+0000", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3843e-05 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466416+0000", 2024-04-15T18:36:06.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8368000000000002e-05 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186306 5.e 5:78abf885:::benchmark_data_smithi159_116570_object93146:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 232.887909097, 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67465055600000001, 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:06.017 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186306 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098144+0000", 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098146+0000", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098152+0000", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0310000000000004e-06 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098163+0000", 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1181e-05 2024-04-15T18:36:06.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098171+0000", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0239999999999997e-06 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098226+0000", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5457000000000001e-05 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098370+0000", 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014345300000000001 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.098497+0000", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012726299999999999 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103398+0000", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049014469999999997 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.108448+0000", 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050492990000000001 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772670+0000", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.66422262099999996 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772755+0000", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5030000000000004e-05 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772795+0000", 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9406999999999997e-05 2024-04-15T18:36:06.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:186297 5.e e536/358 ondisk, result = 0)", 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 232.87805974899999, 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35842624299999998, 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107994+0000", 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.107995+0000", 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.68e-06 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.108002+0000", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6390000000000003e-06 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466258+0000", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35825611800000001 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466265+0000", 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5169999999999996e-06 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466312+0000", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7539e-05 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466320+0000", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4680000000000003e-06 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466420+0000", 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100282 2024-04-15T18:36:06.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186306 5.e 5:78abf885:::benchmark_data_smithi159_116570_object93146:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e536)", 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 232.87288185, 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.65959372100000002, 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:06.027 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186306 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113171+0000", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113172+0000", 2024-04-15T18:36:06.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7899999999999999e-07 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113175+0000", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3239999999999999e-06 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113183+0000", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5290000000000003e-06 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113189+0000", 2024-04-15T18:36:06.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1299999999999998e-06 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113434+0000", 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024558700000000002 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for ondisk", 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113477+0000", 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3093000000000002e-05 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.772765+0000", 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.65928767899999996 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:186314 5.b 5:db471f5e:::benchmark_data_smithi159_116570_object93154:head [delete] snapc 0=[] ondisk+write+known_if_redirected e536)", 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 232.87281840899999, 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.99055303699999997, 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.031 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 186314 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113235+0000", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5600000000000001e-07 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113238+0000", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7130000000000001e-06 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113248+0000", 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0137e-05 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113252+0000", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000001e-06 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113371+0000", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118981 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113479+0000", 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010749800000000001 2024-04-15T18:36:06.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.113615+0000", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013696199999999999 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.117350+0000", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003734818 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.120268+0000", 2024-04-15T18:36:06.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029180479999999999 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103702+0000", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.983433848 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103742+0000", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9328e-05 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:14.103788+0000", 2024-04-15T18:36:06.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6397000000000003e-05 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:89863 6.9 6:91fd2ab3:::benchmark_data_smithi159_139342_object89862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.040996932, 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37690326899999999, 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 89863 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945058+0000", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3689999999999999e-06 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945076+0000", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8085999999999999e-05 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945086+0000", 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7739999999999999e-06 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945091+0000", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1290000000000004e-06 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945132+0000", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1081000000000002e-05 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945234+0000", 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102428 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945337+0000", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010299899999999999 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.950097+0000", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047599950000000004 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.950294+0000", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000196813 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321843+0000", 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.371548516 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321905+0000", 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2470000000000003e-05 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321960+0000", 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4608999999999998e-05 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:155611 6.a e598/575 6:5c79a8ec:::benchmark_data_smithi159_139342_object65732:head v 598'9678, mlcod=567'4930)", 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.867256002000001, 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.232901525, 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:36:06.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118798+0000", 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4799999999999997e-07 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118803+0000", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0830000000000003e-06 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348129+0000", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229325801 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348135+0000", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4189999999999999e-06 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348195+0000", 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9808000000000001e-05 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348223+0000", 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7732000000000001e-05 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.351657+0000", 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034345170000000002 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.351699+0000", 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1417e-05 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:158103 6.9 6:94e72de9:::benchmark_data_smithi159_139342_object68224:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:46.885705+0000", 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.100348284999999, 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20105256299999999, 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.047 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158103 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885705+0000", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885705+0000", 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885706+0000", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1939999999999999e-06 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885712+0000", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.356e-06 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885724+0000", 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221999999999999e-05 2024-04-15T18:36:06.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885733+0000", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5070000000000006e-06 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885796+0000", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2261000000000006e-05 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885902+0000", 2024-04-15T18:36:06.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010671900000000001 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.885999+0000", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6800999999999997e-05 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.889982+0000", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039833739999999996 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:46.890160+0000", 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017796199999999999 2024-04-15T18:36:06.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.086668+0000", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19650795900000001 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.086705+0000", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6936e-05 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.086758+0000", 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2271999999999997e-05 2024-04-15T18:36:06.052 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:158190 6.9 6:9d4dbcde:::benchmark_data_smithi159_139342_object68311:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.853376989000001, 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24825063999999999, 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158190 2024-04-15T18:36:06.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132677+0000", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132683+0000", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5289999999999999e-06 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132694+0000", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1341e-05 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132700+0000", 2024-04-15T18:36:06.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5350000000000004e-06 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132742+0000", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2536000000000001e-05 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132839+0000", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6197000000000001e-05 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132915+0000", 2024-04-15T18:36:06.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6719999999999997e-05 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.137560+0000", 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004644385 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.137882+0000", 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032201499999999999 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380850+0000", 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24296841699999999 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380890+0000", 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0098999999999997e-05 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380927+0000", 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6789000000000003e-05 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.058 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.059 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:158202 6.9 6:937e2fa2:::benchmark_data_smithi159_139342_object68323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.816148101, 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.231209677, 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.061 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158202 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169906+0000", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.187e-06 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169911+0000", 2024-04-15T18:36:06.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0329999999999999e-06 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169923+0000", 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1868000000000001e-05 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169929+0000", 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7610000000000004e-06 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169984+0000", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4746000000000001e-05 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.170084+0000", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010006200000000001 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.170160+0000", 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6305999999999998e-05 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.173978+0000", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003817745 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.174149+0000", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017113999999999999 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401048+0000", 2024-04-15T18:36:06.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22689936099999999 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401084+0000", 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5321000000000003e-05 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401115+0000", 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1146999999999998e-05 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:167300 6.9 6:91185da9:::benchmark_data_smithi159_139342_object77421:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:03.227746+0000", 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.7583069130000002, 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14690705300000001, 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 167300 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.068 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227746+0000", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227746+0000", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227748+0000", 2024-04-15T18:36:06.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.536e-06 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227754+0000", 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6779999999999997e-06 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227764+0000", 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0631e-05 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.227773+0000", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6240000000000008e-06 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.238885+0000", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011112479999999999 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.241486+0000", 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026002159999999998 2024-04-15T18:36:06.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.242278+0000", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00079205200000000001 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.248947+0000", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006669426 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.250035+0000", 2024-04-15T18:36:06.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001088107 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.374557+0000", 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.12452152800000001 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.374612+0000", 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4849999999999998e-05 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.374653+0000", 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1925000000000003e-05 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:167315 6.a e607/602 6:5548ec4a:::benchmark_data_smithi159_139342_object77436:head v 607'10428, mlcod=607'10425)", 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:03.254335+0000", 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.7317184540000001, 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18491997299999999, 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.074 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.254335+0000", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.254335+0000", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.254336+0000", 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9899999999999999e-07 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.254342+0000", 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2500000000000003e-06 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.431234+0000", 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17689180700000001 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.431241+0000", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.075e-06 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.431287+0000", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6582999999999998e-05 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.431301+0000", 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3910000000000001e-05 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.439222+0000", 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079204400000000008 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:03.439255+0000", 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3108999999999998e-05 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:168318 6.9 6:9c1fb062:::benchmark_data_smithi159_139342_object78439:head [delete] snapc 0=[] ondisk+write+known_if_redirected e607)", 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:05.424869+0000", 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.56118405199999999, 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.159871918, 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 168318 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:06.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424869+0000", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424869+0000", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424871+0000", 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5009999999999999e-06 2024-04-15T18:36:06.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424877+0000", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424890+0000", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3081999999999999e-05 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.424898+0000", 2024-04-15T18:36:06.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6380000000000004e-06 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.426765+0000", 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018667550000000001 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.426893+0000", 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012806699999999999 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.427021+0000", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012782200000000001 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.434503+0000", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074820779999999996 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.438664+0000", 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041614879999999996 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.584628+0000", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14596437100000001 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.584685+0000", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6199000000000002e-05 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:05.584741+0000", 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6508000000000002e-05 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.084 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:06.085 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:06.085 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:36:06.571 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:36:06.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:36:06.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:06 smithi049 ceph-mon[40087]: 6.3 scrub ok 2024-04-15T18:36:06.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:06 smithi049 ceph-mon[40087]: 1.0 deep-scrub starts 2024-04-15T18:36:06.645 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:06 smithi049 ceph-mon[40087]: 1.0 deep-scrub ok 2024-04-15T18:36:06.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:06 smithi159 ceph-mon[45336]: 6.3 scrub ok 2024-04-15T18:36:06.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:06 smithi159 ceph-mon[45336]: 1.0 deep-scrub starts 2024-04-15T18:36:06.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:06 smithi159 ceph-mon[45336]: 1.0 deep-scrub ok 2024-04-15T18:36:06.722 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:36:06.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:06 smithi082 ceph-mon[44206]: 6.3 scrub ok 2024-04-15T18:36:06.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:06 smithi082 ceph-mon[44206]: 1.0 deep-scrub starts 2024-04-15T18:36:06.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:06 smithi082 ceph-mon[44206]: 1.0 deep-scrub ok 2024-04-15T18:36:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:06.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:07.062 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:07.089 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:07.123 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:07.152 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:07.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:07.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:07.332 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:07.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:07 smithi082 ceph-mon[44206]: pgmap v1585: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 1.2 MiB/s, 67 objects/s recovering 2024-04-15T18:36:07.597 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:07.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:07 smithi049 ceph-mon[40087]: pgmap v1585: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 1.2 MiB/s, 67 objects/s recovering 2024-04-15T18:36:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:07.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:07.617 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:07.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:07.684 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:07.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:07.723 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:07.732 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:07.754 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:07.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:07.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:07.797 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:07.835 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:07.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:07.868 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:07 smithi159 ceph-mon[45336]: pgmap v1585: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 811 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 1.2 MiB/s, 67 objects/s recovering 2024-04-15T18:36:07.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:07.907 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:07.940 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:07.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:08.005 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:08.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:08.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:08.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:08.005 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:08.006 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:08.200 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:08.373 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: 6.1 deep-scrub ok 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2032355306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/282029118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: 6.1 deep-scrub ok 2024-04-15T18:36:08.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2032355306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/282029118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: 6.1 deep-scrub ok 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2032355306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/282029118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:36:08.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:36:09.353 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:09.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:09.459 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:09.488 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 761 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 962 KiB/s, 54 objects/s recovering 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: 6.c deep-scrub ok 2024-04-15T18:36:09.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:09.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:09.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:09 smithi159 ceph-mon[45336]: osdmap e608: 12 total, 11 up, 11 in 2024-04-15T18:36:09.689 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:09.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:09.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 761 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 962 KiB/s, 54 objects/s recovering 2024-04-15T18:36:09.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: 6.c deep-scrub ok 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:09.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:09 smithi049 ceph-mon[40087]: osdmap e608: 12 total, 11 up, 11 in 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: pgmap v1586: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 761 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 962 KiB/s, 54 objects/s recovering 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: 6.c deep-scrub ok 2024-04-15T18:36:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:09.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:09 smithi082 ceph-mon[44206]: osdmap e608: 12 total, 11 up, 11 in 2024-04-15T18:36:09.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:09.848 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:09.876 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:09.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:09.975 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:09.975 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:09.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:10.030 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:10.142 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:10.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:10.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:10.277 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:10.685 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:36:10.729 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:10.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:10.835 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:10.867 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:10.892 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:10.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:10.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:36:10.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:11.034 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:11.058 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:11.064 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:11.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:11.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:11.259 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:11.303 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:11.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:11.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:11.439 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:11.466 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:11.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:11.485 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:11.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:11.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:11 smithi159 ceph-mon[45336]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 658 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 529 op/s 2024-04-15T18:36:11.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:11.578 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:11.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:11 smithi049 ceph-mon[40087]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 658 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 529 op/s 2024-04-15T18:36:11.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:11.637 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:11.637 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:11.642 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:36:11.657 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:36:11.676 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:11.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:11 smithi082 ceph-mon[44206]: pgmap v1588: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 658 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 529 op/s 2024-04-15T18:36:11.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:11.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:11.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:11.819 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:11.949 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:11.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:12.054 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:12.382 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:12 smithi159 ceph-mon[45336]: osdmap e609: 12 total, 11 up, 11 in 2024-04-15T18:36:12.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:12 smithi159 ceph-mon[45336]: pgmap v1590: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 596 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 428 op/s 2024-04-15T18:36:12.383 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:12 smithi159 ceph-mon[45336]: from='client.16731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:12.508 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:12 smithi049 ceph-mon[40087]: osdmap e609: 12 total, 11 up, 11 in 2024-04-15T18:36:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:12 smithi049 ceph-mon[40087]: pgmap v1590: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 596 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 428 op/s 2024-04-15T18:36:12.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:12 smithi049 ceph-mon[40087]: from='client.16731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:12.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:12 smithi082 ceph-mon[44206]: osdmap e609: 12 total, 11 up, 11 in 2024-04-15T18:36:12.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:12 smithi082 ceph-mon[44206]: pgmap v1590: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 596 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 428 op/s 2024-04-15T18:36:12.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:12 smithi082 ceph-mon[44206]: from='client.16731 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:12.861 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:12.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:12.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:12.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:36:12.999 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:13.450 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:13.451 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:13.453 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:13.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:13.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:13.616 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:13.657 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:13.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:13.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:13.835 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:13.859 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:13.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:13.969 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:14.004 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:14.106 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:14.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:14.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:36:14.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:14.249 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:14.532 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:14 smithi049 ceph-mon[40087]: pgmap v1591: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-04-15T18:36:14.569 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2024-04-15T18:36:11.441964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625148472,"num_objects":9533,"num_object_clones":0,"num_object_copies":28599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9533,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":170404,"num_write_kb":5755003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21881,"num_bytes_recovered":1399155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6262956,"kb_used_data":1921168,"kb_used_omap":176,"kb_used_meta":4341327,"kb_avail":1024933716,"statfs":{"total":1055945392128,"available":1049532125184,"internally_reserved":0,"allocated":1967276032,"data_stored":1963995195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":180226,"internal_metadata":4445519870},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,8,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-224722944,"num_objects":-3429,"num_object_clones":0,"num_object_copies":-10287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001488"},"pg_stats":[{"pgid":"6.f","version":"607'10635","reported_seq":10795,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.292579+0000","last_change":"2024-04-15T18:35:55.478313+0000","last_active":"2024-04-15T18:36:07.292579+0000","last_peered":"2024-04-15T18:36:07.292579+0000","last_clean":"2024-04-15T18:36:07.292579+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:36:07.292579+0000","last_undegraded":"2024-04-15T18:36:07.292579+0000","last_fullsized":"2024-04-15T18:36:07.292579+0000","mapping_epoch":541,"log_start":"547'600","ondisk_log_start":"547'600","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10228","last_scrub_stamp":"2024-04-15T18:35:55.478006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.478006+0000","objects_scrubbed":1072,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:01.919682+0000","scrub_duration":252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188224,"num_objects":659,"num_object_clones":0,"num_object_copies":1977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10635,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"607'10465","reported_seq":10702,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.729558+0000","last_change":"2024-04-15T18:35:45.010064+0000","last_active":"2024-04-15T18:36:07.729558+0000","last_peered":"2024-04-15T18:36:07.729558+0000","last_clean":"2024-04-15T18:36:07.729558+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:36:07.729558+0000","last_undegraded":"2024-04-15T18:36:07.729558+0000","last_fullsized":"2024-04-15T18:36:07.729558+0000","mapping_epoch":541,"log_start":"547'400","ondisk_log_start":"547'400","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'9681","last_scrub_stamp":"2024-04-15T18:35:45.009884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:45.009884+0000","objects_scrubbed":1421,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:10.298794+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10465,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"607'10749","reported_seq":10951,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.796531+0000","last_change":"2024-04-15T18:36:04.452909+0000","last_active":"2024-04-15T18:36:07.796531+0000","last_peered":"2024-04-15T18:36:07.796531+0000","last_clean":"2024-04-15T18:36:07.796531+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:36:07.796531+0000","last_undegraded":"2024-04-15T18:36:07.796531+0000","last_fullsized":"2024-04-15T18:36:07.796531+0000","mapping_epoch":541,"log_start":"547'700","ondisk_log_start":"547'700","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10636","last_scrub_stamp":"2024-04-15T18:36:04.452727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:04.452727+0000","objects_scrubbed":771,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:06.836658+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860568,"num_objects":655,"num_object_clones":0,"num_object_copies":1965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10749,"num_write_kb":364865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"607'11524","reported_seq":22826,"reported_epoch":607,"state":"active+clean+scrubbing+deep","last_fresh":"2024-04-15T18:36:07.836401+0000","last_change":"2024-04-15T18:36:02.289117+0000","last_active":"2024-04-15T18:36:07.836401+0000","last_peered":"2024-04-15T18:36:07.836401+0000","last_clean":"2024-04-15T18:36:07.836401+0000","last_became_active":"2024-04-15T18:35:58.477290+0000","last_became_peered":"2024-04-15T18:35:58.477290+0000","last_unstale":"2024-04-15T18:36:07.836401+0000","last_undegraded":"2024-04-15T18:36:07.836401+0000","last_fullsized":"2024-04-15T18:36:07.836401+0000","mapping_epoch":606,"log_start":"549'1501","ondisk_log_start":"549'1501","created":541,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:32:29.457742+0000","objects_scrubbed":754,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45219840,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":337248256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"608'10807","reported_seq":10971,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:11.303984+0000","last_change":"2024-04-15T18:35:54.492058+0000","last_active":"2024-04-15T18:36:11.303984+0000","last_peered":"2024-04-15T18:36:11.303984+0000","last_clean":"2024-04-15T18:36:11.303984+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:36:11.303984+0000","last_undegraded":"2024-04-15T18:36:11.303984+0000","last_fullsized":"2024-04-15T18:36:11.303984+0000","mapping_epoch":541,"log_start":"547'800","ondisk_log_start":"547'800","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10193","last_scrub_stamp":"2024-04-15T18:35:54.491971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:54.491971+0000","objects_scrubbed":1101,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:59.573582+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31916032,"num_objects":487,"num_object_clones":0,"num_object_copies":1461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"607'10562","reported_seq":21432,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.782101+0000","last_change":"2024-04-15T18:35:57.590647+0000","last_active":"2024-04-15T18:36:07.782101+0000","last_peered":"2024-04-15T18:36:07.782101+0000","last_clean":"2024-04-15T18:36:07.782101+0000","last_became_active":"2024-04-15T18:35:51.865798+0000","last_became_peered":"2024-04-15T18:35:51.865798+0000","last_unstale":"2024-04-15T18:36:07.782101+0000","last_undegraded":"2024-04-15T18:36:07.782101+0000","last_fullsized":"2024-04-15T18:36:07.782101+0000","mapping_epoch":602,"log_start":"547'500","ondisk_log_start":"547'500","created":541,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'10225","last_scrub_stamp":"2024-04-15T18:35:57.590483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:57.590483+0000","objects_scrubbed":966,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:03.223994+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":352704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5343,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,9],"acting":[5,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"607'10419","reported_seq":10621,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.369836+0000","last_change":"2024-04-15T18:36:01.917202+0000","last_active":"2024-04-15T18:36:07.369836+0000","last_peered":"2024-04-15T18:36:07.369836+0000","last_clean":"2024-04-15T18:36:07.369836+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:36:07.369836+0000","last_undegraded":"2024-04-15T18:36:07.369836+0000","last_fullsized":"2024-04-15T18:36:07.369836+0000","mapping_epoch":541,"log_start":"547'400","ondisk_log_start":"547'400","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10251","last_scrub_stamp":"2024-04-15T18:36:01.916949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.916949+0000","objects_scrubbed":848,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:03.231241+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42926080,"num_objects":655,"num_object_clones":0,"num_object_copies":1965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"608'10738","reported_seq":10901,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:11.278448+0000","last_change":"2024-04-15T18:35:56.721189+0000","last_active":"2024-04-15T18:36:11.278448+0000","last_peered":"2024-04-15T18:36:11.278448+0000","last_clean":"2024-04-15T18:36:11.278448+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:36:11.278448+0000","last_undegraded":"2024-04-15T18:36:11.278448+0000","last_fullsized":"2024-04-15T18:36:11.278448+0000","mapping_epoch":541,"log_start":"547'700","ondisk_log_start":"547'700","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10217","last_scrub_stamp":"2024-04-15T18:35:56.720948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:56.720948+0000","objects_scrubbed":968,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:01.442649+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10738,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"607'10642","reported_seq":10878,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.824959+0000","last_change":"2024-04-15T18:35:25.111330+0000","last_active":"2024-04-15T18:36:07.824959+0000","last_peered":"2024-04-15T18:36:07.824959+0000","last_clean":"2024-04-15T18:36:07.824959+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:36:07.824959+0000","last_undegraded":"2024-04-15T18:36:07.824959+0000","last_fullsized":"2024-04-15T18:36:07.824959+0000","mapping_epoch":541,"log_start":"547'600","ondisk_log_start":"547'600","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'8959","last_scrub_stamp":"2024-04-15T18:35:25.111058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:25.111058+0000","objects_scrubbed":2343,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:48.179384+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42336256,"num_objects":646,"num_object_clones":0,"num_object_copies":1938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10642,"num_write_kb":361216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"608'10781","reported_seq":21916,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:11.304027+0000","last_change":"2024-04-15T18:36:01.585943+0000","last_active":"2024-04-15T18:36:11.304027+0000","last_peered":"2024-04-15T18:36:11.304027+0000","last_clean":"2024-04-15T18:36:11.304027+0000","last_became_active":"2024-04-15T18:35:54.445912+0000","last_became_peered":"2024-04-15T18:35:54.445912+0000","last_unstale":"2024-04-15T18:36:11.304027+0000","last_undegraded":"2024-04-15T18:36:11.304027+0000","last_fullsized":"2024-04-15T18:36:11.304027+0000","mapping_epoch":604,"log_start":"547'700","ondisk_log_start":"547'700","created":541,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10410","last_scrub_stamp":"2024-04-15T18:36:01.585745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.585745+0000","objects_scrubbed":878,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:13.405968+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33095680,"num_objects":505,"num_object_clones":0,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5481,"num_bytes_recovered":359202816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"607'10389","reported_seq":10616,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.426981+0000","last_change":"2024-04-15T18:35:47.370846+0000","last_active":"2024-04-15T18:36:07.426981+0000","last_peered":"2024-04-15T18:36:07.426981+0000","last_clean":"2024-04-15T18:36:07.426981+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:36:07.426981+0000","last_undegraded":"2024-04-15T18:36:07.426981+0000","last_fullsized":"2024-04-15T18:36:07.426981+0000","mapping_epoch":541,"log_start":"547'300","ondisk_log_start":"547'300","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'9726","last_scrub_stamp":"2024-04-15T18:35:47.370689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:47.370689+0000","objects_scrubbed":1326,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:04.173269+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"608'10860","reported_seq":22013,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:11.296799+0000","last_change":"2024-04-15T18:36:04.752350+0000","last_active":"2024-04-15T18:36:11.296799+0000","last_peered":"2024-04-15T18:36:11.296799+0000","last_clean":"2024-04-15T18:36:11.296799+0000","last_became_active":"2024-04-15T18:35:47.841503+0000","last_became_peered":"2024-04-15T18:35:47.841503+0000","last_unstale":"2024-04-15T18:36:11.296799+0000","last_undegraded":"2024-04-15T18:36:11.296799+0000","last_fullsized":"2024-04-15T18:36:11.296799+0000","mapping_epoch":600,"log_start":"547'800","ondisk_log_start":"547'800","created":541,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10609","last_scrub_stamp":"2024-04-15T18:36:04.751692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:04.751692+0000","objects_scrubbed":691,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:08.590026+0000","scrub_duration":310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5490,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"607'10562","reported_seq":10786,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.744889+0000","last_change":"2024-04-15T18:35:55.811254+0000","last_active":"2024-04-15T18:36:07.744889+0000","last_peered":"2024-04-15T18:36:07.744889+0000","last_clean":"2024-04-15T18:36:07.744889+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:36:07.744889+0000","last_undegraded":"2024-04-15T18:36:07.744889+0000","last_fullsized":"2024-04-15T18:36:07.744889+0000","mapping_epoch":541,"log_start":"547'500","ondisk_log_start":"547'500","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10120","last_scrub_stamp":"2024-04-15T18:35:55.810909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.810909+0000","objects_scrubbed":1092,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:14.214223+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42598400,"num_objects":650,"num_object_clones":0,"num_object_copies":1950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10562,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"608'11629","reported_seq":11880,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:10.525779+0000","last_change":"2024-04-15T18:36:06.701881+0000","last_active":"2024-04-15T18:36:10.525779+0000","last_peered":"2024-04-15T18:36:10.525779+0000","last_clean":"2024-04-15T18:36:10.525779+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:36:10.525779+0000","last_undegraded":"2024-04-15T18:36:10.525779+0000","last_fullsized":"2024-04-15T18:36:10.525779+0000","mapping_epoch":541,"log_start":"550'1601","ondisk_log_start":"550'1601","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'11484","last_scrub_stamp":"2024-04-15T18:36:06.701721+0000","last_deep_scrub":"607'11484","last_deep_scrub_stamp":"2024-04-15T18:36:06.701721+0000","last_clean_scrub_stamp":"2024-04-15T18:36:06.701721+0000","objects_scrubbed":818,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:34.054642+0000","scrub_duration":6572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421056,"num_objects":571,"num_object_clones":0,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"607'10594","reported_seq":10752,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.392523+0000","last_change":"2024-04-15T18:36:01.392695+0000","last_active":"2024-04-15T18:36:07.392523+0000","last_peered":"2024-04-15T18:36:07.392523+0000","last_clean":"2024-04-15T18:36:07.392523+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:36:07.392523+0000","last_undegraded":"2024-04-15T18:36:07.392523+0000","last_fullsized":"2024-04-15T18:36:07.392523+0000","mapping_epoch":541,"log_start":"547'500","ondisk_log_start":"547'500","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10402","last_scrub_stamp":"2024-04-15T18:36:01.392490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.392490+0000","objects_scrubbed":829,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:18.367555+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41549824,"num_objects":634,"num_object_clones":0,"num_object_copies":1902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"607'10598","reported_seq":10758,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-04-15T18:36:07.783337+0000","last_change":"2024-04-15T18:35:58.646760+0000","last_active":"2024-04-15T18:36:07.783337+0000","last_peered":"2024-04-15T18:36:07.783337+0000","last_clean":"2024-04-15T18:36:07.783337+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:36:07.783337+0000","last_undegraded":"2024-04-15T18:36:07.783337+0000","last_fullsized":"2024-04-15T18:36:07.783337+0000","mapping_epoch":541,"log_start":"547'500","ondisk_log_start":"547'500","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10289","last_scrub_stamp":"2024-04-15T18:35:58.646432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:58.646432+0000","objects_scrubbed":919,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:08.206941+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10598,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1518,"reported_epoch":608,"state":"active+clean","last_fresh":"2024-04-15T18:36:08.374160+0000","last_change":"2024-04-15T18:36:05.553043+0000","last_active":"2024-04-15T18:36:08.374160+0000","last_peered":"2024-04-15T18:36:08.374160+0000","last_clean":"2024-04-15T18:36:08.374160+0000","last_became_active":"2024-04-15T18:34:07.451006+0000","last_became_peered":"2024-04-15T18:34:07.451006+0000","last_unstale":"2024-04-15T18:36:08.374160+0000","last_undegraded":"2024-04-15T18:36:08.374160+0000","last_fullsized":"2024-04-15T18:36:08.374160+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:36:05.552905+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:29.521306+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":624558104,"num_objects":9531,"num_object_clones":0,"num_object_copies":28593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170223,"num_write_kb":5752065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21869,"num_bytes_recovered":1395982336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2830381056,"data_stored":2830368840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926148,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556628,"kb_used_data":70088,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93188524,"statfs":{"total":95995035648,"available":95425048576,"internally_reserved":0,"allocated":71770112,"data_stored":71467469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187785,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484416,"kb_used_data":128376,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93260736,"statfs":{"total":95995035648,"available":95498993664,"internally_reserved":0,"allocated":131457024,"data_stored":131158429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514831,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672068,"kb_used_data":221564,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93073084,"statfs":{"total":95995035648,"available":95306838016,"internally_reserved":0,"allocated":226881536,"data_stored":226583461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":461240768},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874578,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440520,"kb_used_data":138624,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93304632,"statfs":{"total":95995035648,"available":95543943168,"internally_reserved":0,"allocated":141950976,"data_stored":141651905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168919,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571536,"kb_used_data":165512,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93173616,"statfs":{"total":95995035648,"available":95409782784,"internally_reserved":0,"allocated":169484288,"data_stored":169181665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575225,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622272,"kb_used_data":171832,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93122880,"statfs":{"total":95995035648,"available":95357829120,"internally_reserved":0,"allocated":175955968,"data_stored":175657373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571016,"kb_used_data":102528,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93174136,"statfs":{"total":95995035648,"available":95410315264,"internally_reserved":0,"allocated":104988672,"data_stored":104689601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193997,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428668,"kb_used_data":211892,"kb_used_omap":114,"kb_used_meta":216525,"kb_avail":93316484,"statfs":{"total":95995035648,"available":95556079616,"internally_reserved":0,"allocated":216977408,"data_stored":216682885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":221721841},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609127,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820028,"kb_used_data":334260,"kb_used_omap":44,"kb_used_meta":485715,"kb_avail":92925124,"statfs":{"total":95995035648,"available":95155326976,"internally_reserved":0,"allocated":342282240,"data_stored":341987717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":497372714},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610556,"kb_used_data":164276,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93134596,"statfs":{"total":95995035648,"available":95369826304,"internally_reserved":0,"allocated":168218624,"data_stored":167924101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703428,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":485248,"kb_used_data":212216,"kb_used_omap":4,"kb_used_meta":273019,"kb_avail":93259904,"statfs":{"total":95995035648,"available":95498141696,"internally_reserved":0,"allocated":217309184,"data_stored":217010589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":279572161},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103153664,"data_stored":103153664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172232704,"data_stored":172228632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":213975040,"data_stored":213975040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340393984,"data_stored":340393984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":165085184,"data_stored":165085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":210374656,"data_stored":210370584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":152436736,"data_stored":152436736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":139001856,"data_stored":139001856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":216399872,"data_stored":216399872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":125308928,"data_stored":125304856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":65208320,"data_stored":65208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:36:14.600 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:14 smithi082 ceph-mon[44206]: pgmap v1591: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-04-15T18:36:14.809 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:14.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:14.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:14 smithi159 ceph-mon[45336]: pgmap v1591: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-04-15T18:36:14.918 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:14.954 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:14.991 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:14.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:15.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:15.158 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:15.163 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:15.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:15.280 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:15.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:15.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:15.412 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:15.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:15.456 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:15.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:15.485 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:15.534 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:15.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:15.580 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:15.614 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:15.633 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:15.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:15.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:15.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:15.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:15.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4001159722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:15.775 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:15.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4001159722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:15.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:15.833 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4001159722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:15.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:15.924 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:16.013 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:16.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:16.119 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:16.169 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:16.197 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:16.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:16.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:16.386 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93112 5.4 5:26f9a09c:::benchmark_data_smithi159_116570_object93111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 599.03719822599999, 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44799008099999998, 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.464 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93112 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431723+0000", 2024-04-15T18:36:16.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431724+0000", 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.703e-06 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431750+0000", 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5607000000000001e-05 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431765+0000", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4961e-05 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431776+0000", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1491000000000001e-05 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432594+0000", 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081732599999999995 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432605+0000", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1847e-05 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.869983+0000", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43737783200000002 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.870128+0000", 2024-04-15T18:36:16.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014457399999999999 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.870245+0000", 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000117124 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877470+0000", 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072249510000000003 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877516+0000", 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6137999999999998e-05 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877527+0000", 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0603e-05 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.877566+0000", 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9755999999999997e-05 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879713+0000", 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021461679999999999 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93117 5.4 5:2c3e2625:::benchmark_data_smithi159_116570_object93116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 599.03712143799999, 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45760223, 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.471 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93117 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431799+0000", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431800+0000", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7899999999999999e-07 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431817+0000", 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.768e-05 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431826+0000", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6899999999999998e-06 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431832+0000", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9429999999999997e-06 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432666+0000", 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083379299999999999 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432675+0000", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0019999999999995e-06 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.874919+0000", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44224381400000001 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.875032+0000", 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113428 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.875146+0000", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879881+0000", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047348360000000001 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.885930+0000", 2024-04-15T18:36:16.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0060485570000000004 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.885992+0000", 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1898999999999998e-05 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.886063+0000", 2024-04-15T18:36:16.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1125999999999996e-05 2024-04-15T18:36:16.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889402+0000", 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033388329999999998 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93154 5.4 5:2e09de61:::benchmark_data_smithi159_116570_object93153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e406)", 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 599.03708041899995, 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45964630499999998, 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.479 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93154 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431840+0000", 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431841+0000", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3799999999999998e-07 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431859+0000", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8561000000000001e-05 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431867+0000", 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2030000000000003e-06 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431871+0000", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9189999999999998e-06 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432711+0000", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083922700000000001 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432716+0000", 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1889999999999998e-06 2024-04-15T18:36:16.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.879919+0000", 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44720337100000002 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.880100+0000", 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018037500000000001 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:36:16.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.880257+0000", 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015770700000000001 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889476+0000", 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092190499999999995 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889494+0000", 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.806e-05 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889604+0000", 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109131 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889667+0000", 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3540999999999994e-05 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.891487+0000", 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018195329999999999 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:93157 5.4 5:26fe9593:::benchmark_data_smithi159_116570_object93156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e406)", 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 599.03704265600004, 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46360710100000002, 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.486 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93157 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431878+0000", 2024-04-15T18:36:16.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431879+0000", 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.89e-07 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431898+0000", 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9908999999999999e-05 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431906+0000", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4880000000000001e-06 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.431911+0000", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1789999999999999e-06 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432738+0000", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00082660600000000002 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:36:16.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.432742+0000", 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6190000000000002e-06 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883136+0000", 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45039331599999999 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883297+0000", 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000161521 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.883486+0000", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018915499999999999 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.889557+0000", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006070565 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.891498+0000", 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001940893 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.894242+0000", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002744607 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.894283+0000", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0658999999999998e-05 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:17.895485+0000", 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012021950000000001 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.492 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.4 419/405 [PushReplyOp(5:2fa64db3:::benchmark_data_smithi159_116570_object21356:head)])", 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:44.627824+0000", 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 571.84109646499996, 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.88145360500000003, 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:44.627824+0000", 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:44.627824+0000", 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:44.627825+0000", 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.063e-06 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:44.627831+0000", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9739999999999999e-06 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:45.506379+0000", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.87854758200000005 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:45.506406+0000", 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7559000000000002e-05 2024-04-15T18:36:16.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:45.506478+0000", 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1081000000000006e-05 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:45.509278+0000", 2024-04-15T18:36:16.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002800346 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.4 420/405 [PushReplyOp(5:2ba1ebb2:::benchmark_data_smithi159_116570_object25910:head)])", 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:49.421728+0000", 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 567.04719254099996, 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.70426207799999996, 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:36:16.498 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:49.421728+0000", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:49.421728+0000", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:49.421729+0000", 2024-04-15T18:36:16.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8000000000000005e-07 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:49.421734+0000", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8400000000000002e-06 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:50.123399+0000", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.70166549 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:50.123408+0000", 2024-04-15T18:36:16.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0839999999999998e-06 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:50.123467+0000", 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8952000000000001e-05 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:50.125990+0000", 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025229319999999999 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:108213 5.0 5:05263343:::benchmark_data_smithi159_116570_object15053:head [delete] snapc 0=[] ondisk+write+known_if_redirected e422)", 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:54.365338+0000", 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 562.10358301099996, 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44983003700000002, 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 108213 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365338+0000", 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365338+0000", 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365339+0000", 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.629e-06 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365347+0000", 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7309999999999993e-06 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365366+0000", 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9218000000000001e-05 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365376+0000", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4190000000000008e-06 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365437+0000", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0857999999999999e-05 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365560+0000", 2024-04-15T18:36:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000123316 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.365663+0000", 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010346 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.369044+0000", 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033805329999999998 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.369176+0000", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013171999999999999 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815103+0000", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44592758399999999 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815136+0000", 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2707999999999999e-05 2024-04-15T18:36:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815168+0000", 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1860999999999999e-05 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:108213 5.0 e422/358 ondisk, result = 0)", 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:54.368791+0000", 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 562.10012928699996, 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44638086199999999, 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.368791+0000", 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.368791+0000", 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.368792+0000", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000001e-07 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.368798+0000", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9710000000000001e-06 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815045+0000", 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.446246319 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815053+0000", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5159999999999994e-06 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815094+0000", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1118999999999997e-05 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815100+0000", 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.383e-06 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.815172+0000", 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2689999999999997e-05 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:108218 5.0 5:0847a2fc:::benchmark_data_smithi159_116570_object15058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e423)", 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:54.412530+0000", 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 562.056391265, 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43101314400000001, 2024-04-15T18:36:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 108218 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412530+0000", 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412530+0000", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412531+0000", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0839999999999999e-06 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412536+0000", 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0810000000000001e-06 2024-04-15T18:36:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412547+0000", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1171e-05 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412553+0000", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0870000000000003e-06 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412608+0000", 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5287000000000001e-05 2024-04-15T18:36:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.412753+0000", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014438999999999999 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.413107+0000", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035451500000000002 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416293+0000", 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031862850000000001 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416396+0000", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010241700000000001 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843468+0000", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42707181399999999 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843511+0000", 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3763000000000001e-05 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843543+0000", 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1250000000000001e-05 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:108218 5.0 e423/358 ondisk, result = 0)", 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:26:54.416214+0000", 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 562.05270674400003, 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42733364899999998, 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416214+0000", 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416214+0000", 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416215+0000", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1500000000000005e-07 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.416222+0000", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0169999999999999e-06 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843380+0000", 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42715891499999997 2024-04-15T18:36:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843389+0000", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3189999999999995e-06 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843453+0000", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3912999999999996e-05 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843462+0000", 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0059999999999998e-06 2024-04-15T18:36:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:26:54.843548+0000", 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5964000000000006e-05 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(5.4 428/405 [PushReplyOp(5:2a95ea57:::benchmark_data_smithi159_116570_object45454:head)])", 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:08.805877+0000", 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 547.663043292, 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.70674705100000002, 2024-04-15T18:36:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.805877+0000", 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.805877+0000", 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.805878+0000", 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200000000000002e-07 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:08.805886+0000", 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0669999999999992e-06 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.510103+0000", 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.70421699199999999 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.510112+0000", 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3820000000000004e-06 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.510170+0000", 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8091000000000003e-05 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:09.512625+0000", 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002454707 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:113218 5.0 5:07ee6a04:::benchmark_data_smithi159_116570_object20058:head [delete] snapc 0=[] ondisk+write+known_if_redirected e436)", 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:34.893063+0000", 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 521.57585800200002, 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.448705142, 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 113218 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893063+0000", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893063+0000", 2024-04-15T18:36:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893064+0000", 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0419999999999999e-06 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893069+0000", 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.429e-06 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893080+0000", 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1089e-05 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893088+0000", 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1340000000000004e-06 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893146+0000", 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7580000000000001e-05 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893264+0000", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011833299999999999 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.893376+0000", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000111162 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.896903+0000", 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003527138 2024-04-15T18:36:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:34.897064+0000", 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016102600000000001 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:35.341687+0000", 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44462352500000002 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:35.341734+0000", 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6718999999999998e-05 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:35.341768+0000", 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3964999999999998e-05 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:115759 5.8 e449/358 5:110e281b:::benchmark_data_smithi159_116570_object22599:head v 449'7301, mlcod=449'7298)", 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:27:56.581255+0000", 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 499.88766601600003, 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.52002717600000004, 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:56.581255+0000", 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:56.581255+0000", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:56.581256+0000", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0359999999999999e-06 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:56.581264+0000", 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.097825+0000", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.51656034399999995 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.097831+0000", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6680000000000004e-06 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.097883+0000", 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1897000000000001e-05 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.097907+0000", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4290999999999998e-05 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.101232+0000", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033245089999999998 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:27:57.101282+0000", 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-04-15T18:36:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:119741 5.0 5:042709b5:::benchmark_data_smithi159_116570_object26581:head [delete] snapc 0=[] ondisk+write+known_if_redirected e462)", 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:27.575262+0000", 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 468.89365923000003, 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43833338799999999, 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 119741 2024-04-15T18:36:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575262+0000", 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575262+0000", 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575263+0000", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4699999999999999e-06 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575269+0000", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3609999999999997e-06 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575283+0000", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3181e-05 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575292+0000", 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2490000000000007e-06 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575348+0000", 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6224000000000002e-05 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575466+0000", 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011750400000000001 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.575545+0000", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9845999999999994e-05 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.578777+0000", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003231215 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.578793+0000", 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6620999999999999e-05 2024-04-15T18:36:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013510+0000", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43471707799999998 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013547+0000", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6967e-05 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013595+0000", 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7672e-05 2024-04-15T18:36:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:119741 5.0 e462/358 ondisk, result = 0)", 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:27.577376+0000", 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 468.89154507799998, 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43622424199999998, 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.577376+0000", 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.577376+0000", 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.577377+0000", 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2929999999999999e-06 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:27.577389+0000", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1945e-05 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013454+0000", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43606537099999998 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013462+0000", 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3150000000000003e-06 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013501+0000", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9289000000000003e-05 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013506+0000", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4949999999999999e-06 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:28.013600+0000", 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3534000000000003e-05 2024-04-15T18:36:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.25447.0:121314 5.0 e466/358 ondisk, result = 0)", 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:28:37.744772+0000", 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 458.72414886600001, 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48195966800000001, 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:37.744772+0000", 2024-04-15T18:36:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:37.744772+0000", 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:37.744773+0000", 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.066e-06 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:37.744781+0000", 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2150000000000001e-06 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:38.226554+0000", 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48177330000000002 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:38.226560+0000", 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7459999999999997e-06 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:38.226603+0000", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2447000000000002e-05 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:38.226612+0000", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2280000000000004e-06 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:28:38.226732+0000", 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119666 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.25447.0:130115 5.0 5:003838ad:::benchmark_data_smithi159_116570_object36955:head [delete] snapc 0=[] ondisk+write+known_if_redirected e487)", 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 405.07713932399997, 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.632512768, 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.25447", 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 130115 2024-04-15T18:36:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391781+0000", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391783+0000", 2024-04-15T18:36:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7489999999999999e-06 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391788+0000", 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1619999999999999e-06 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391803+0000", 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4533e-05 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391812+0000", 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2709999999999998e-06 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391866+0000", 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.391977+0000", 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000110729 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.392083+0000", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106679 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.396079+0000", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039953519999999998 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:31.396202+0000", 2024-04-15T18:36:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012359 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:32.024173+0000", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62797061899999995 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:32.024240+0000", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6928999999999996e-05 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:29:32.024294+0000", 2024-04-15T18:36:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4545e-05 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:141747 6.f e595/541 6:fa5b3ac3:::benchmark_data_smithi159_139342_object51868:head v 595'8933, mlcod=595'8930)", 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 53.400673423999997, 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.256791241, 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068248+0000", 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1030000000000001e-06 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068258+0000", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.679e-06 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321692+0000", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25343376699999998 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321699+0000", 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7170000000000002e-06 2024-04-15T18:36:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321769+0000", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0734000000000004e-05 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321790+0000", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0996999999999999e-05 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.325000+0000", 2024-04-15T18:36:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032095650000000002 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.325039+0000", 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8679e-05 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:175777 6.f e609/541 6:f7f4a2f1:::benchmark_data_smithi159_139342_object85898:head v 609'11036, mlcod=609'11033)", 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:16.175829+0000", 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.293091555, 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.097783384000000001, 2024-04-15T18:36:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.175829+0000", 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.175829+0000", 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.175830+0000", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.257e-06 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.175842+0000", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1372000000000001e-05 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.269352+0000", 2024-04-15T18:36:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.093510105999999996 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.269364+0000", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1941e-05 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.269422+0000", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7827e-05 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.269446+0000", 2024-04-15T18:36:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3799999999999999e-05 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.273573+0000", 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041276259999999997 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.273613+0000", 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9455000000000001e-05 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:175926 6.f e609/541 6:fe18b49b:::benchmark_data_smithi159_139342_object86047:head v 609'11042, mlcod=609'11039)", 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:16.338903+0000", 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.13001784799999999, 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.068052177000000005, 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.338903+0000", 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.338903+0000", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.338904+0000", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-06 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.338912+0000", 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9459999999999998e-06 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.402460+0000", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.063547751999999999 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.402469+0000", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5219999999999996e-06 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.402544+0000", 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5133e-05 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.402568+0000", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3506000000000002e-05 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.406904+0000", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004336393 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:16.406955+0000", 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0858e-05 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:36:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:36:16.570 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: pgmap v1592: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.720 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: pgmap v1592: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: pgmap v1592: 17 pgs: 17 active+clean; 439 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 643 op/s 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.822 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.823 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.823 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:16.823 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:17.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:36:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:17.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:17.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:17.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:17.550 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:17.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:36:17.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:17.775 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:17.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:17.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:36:17.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:36:17.828 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:17.880 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:17.917 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:17.944 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:17.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:18.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:18.083 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:18.104 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:18.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:18.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:18.310 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:18.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:18.379 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:18.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:18.472 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:18.545 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:18.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:18.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:18.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:18 smithi049 ceph-mon[40087]: pgmap v1593: 17 pgs: 17 active+clean; 364 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:36:18.732 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:18.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:18.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:18.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:18 smithi159 ceph-mon[45336]: pgmap v1593: 17 pgs: 17 active+clean; 364 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:36:18.913 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:18 smithi082 ceph-mon[44206]: pgmap v1593: 17 pgs: 17 active+clean; 364 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 517 op/s 2024-04-15T18:36:18.913 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:18.942 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:18.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:19.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:19.098 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:19.555 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:36:19.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:36:19.858 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2565776289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:19.859 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:19 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:19.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2565776289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:19.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:19 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:20.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2565776289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:20.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:19 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:20.173 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:36:20.173 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:36:20.174 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:36:20.174 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:36:20.772 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:20.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:20.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:20 smithi159 ceph-mon[45336]: pgmap v1594: 17 pgs: 17 active+clean; 197 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-04-15T18:36:20.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:20 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:20.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:20.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:20 smithi159 ceph-mon[45336]: osdmap e610: 12 total, 11 up, 11 in 2024-04-15T18:36:20.880 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:20.901 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:36:20.915 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:20.952 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:20.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:20.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:20 smithi049 ceph-mon[40087]: pgmap v1594: 17 pgs: 17 active+clean; 197 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-04-15T18:36:20.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:20 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:20.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:20.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:20 smithi049 ceph-mon[40087]: osdmap e610: 12 total, 11 up, 11 in 2024-04-15T18:36:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:20 smithi082 ceph-mon[44206]: pgmap v1594: 17 pgs: 17 active+clean; 197 MiB data, 4.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-04-15T18:36:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:20 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:20 smithi082 ceph-mon[44206]: osdmap e610: 12 total, 11 up, 11 in 2024-04-15T18:36:21.058 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:21.098 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:21.134 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:21.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:21.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:21.285 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:21.332 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:21.658 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:36:21.660 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:36:21.898 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:22.159 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:22.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:22.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:22.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:22.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:21 smithi049 ceph-mon[40087]: osdmap e611: 12 total, 11 up, 11 in 2024-04-15T18:36:22.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:22.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:22.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 89876 objects 2024-04-15T18:36:22.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :132.279 2024-04-15T18:36:22.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:22.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:22.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:22.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:21 smithi082 ceph-mon[44206]: osdmap e611: 12 total, 11 up, 11 in 2024-04-15T18:36:22.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:22.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:22.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:22.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:22.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:21 smithi159 ceph-mon[45336]: osdmap e611: 12 total, 11 up, 11 in 2024-04-15T18:36:22.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:22.303 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:22.365 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:22.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:22.469 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:22.496 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:22.596 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:22.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:22.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:22.751 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:22.773 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:22.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:22.819 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-04-15T18:36:21.443822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":128516664,"num_objects":1955,"num_object_clones":0,"num_object_copies":5865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1955,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":177982,"num_write_kb":5755003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21881,"num_bytes_recovered":1399155904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161088,"ondisk_log_size":161088,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":4918720,"kb_used_data":488344,"kb_used_omap":176,"kb_used_meta":4430095,"kb_avail":1026277952,"statfs":{"total":1055945392128,"available":1050908622848,"internally_reserved":0,"allocated":500064256,"data_stored":496683913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":180226,"internal_metadata":4536418302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,2,10,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-332136448,"num_objects":-5068,"num_object_clones":0,"num_object_copies":-15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5068,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001499"},"pg_stats":[{"pgid":"6.f","version":"609'11098","reported_seq":11260,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.451337+0000","last_change":"2024-04-15T18:35:55.478313+0000","last_active":"2024-04-15T18:36:17.451337+0000","last_peered":"2024-04-15T18:36:17.451337+0000","last_clean":"2024-04-15T18:36:17.451337+0000","last_became_active":"2024-04-15T18:32:30.639649+0000","last_became_peered":"2024-04-15T18:32:30.639649+0000","last_unstale":"2024-04-15T18:36:17.451337+0000","last_undegraded":"2024-04-15T18:36:17.451337+0000","last_fullsized":"2024-04-15T18:36:17.451337+0000","mapping_epoch":541,"log_start":"547'1000","ondisk_log_start":"547'1000","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10228","last_scrub_stamp":"2024-04-15T18:35:55.478006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.478006+0000","objects_scrubbed":1072,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:01.919682+0000","scrub_duration":252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12845056,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"609'10913","reported_seq":11154,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.819950+0000","last_change":"2024-04-15T18:35:45.010064+0000","last_active":"2024-04-15T18:36:17.819950+0000","last_peered":"2024-04-15T18:36:17.819950+0000","last_clean":"2024-04-15T18:36:17.819950+0000","last_became_active":"2024-04-15T18:32:30.477286+0000","last_became_peered":"2024-04-15T18:32:30.477286+0000","last_unstale":"2024-04-15T18:36:17.819950+0000","last_undegraded":"2024-04-15T18:36:17.819950+0000","last_fullsized":"2024-04-15T18:36:17.819950+0000","mapping_epoch":541,"log_start":"547'900","ondisk_log_start":"547'900","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"599'9681","last_scrub_stamp":"2024-04-15T18:35:45.009884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:45.009884+0000","objects_scrubbed":1421,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:10.298794+0000","scrub_duration":426,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11468800,"num_objects":175,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10913,"num_write_kb":354816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"609'11227","reported_seq":11433,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.825551+0000","last_change":"2024-04-15T18:36:04.452909+0000","last_active":"2024-04-15T18:36:17.825551+0000","last_peered":"2024-04-15T18:36:17.825551+0000","last_clean":"2024-04-15T18:36:17.825551+0000","last_became_active":"2024-04-15T18:32:30.640066+0000","last_became_peered":"2024-04-15T18:32:30.640066+0000","last_unstale":"2024-04-15T18:36:17.825551+0000","last_undegraded":"2024-04-15T18:36:17.825551+0000","last_fullsized":"2024-04-15T18:36:17.825551+0000","mapping_epoch":541,"log_start":"548'1200","ondisk_log_start":"548'1200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10636","last_scrub_stamp":"2024-04-15T18:36:04.452727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:04.452727+0000","objects_scrubbed":771,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:06.836658+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534360,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11227,"num_write_kb":364865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"609'12054","reported_seq":23366,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.840549+0000","last_change":"2024-04-15T18:36:08.033820+0000","last_active":"2024-04-15T18:36:17.840549+0000","last_peered":"2024-04-15T18:36:17.840549+0000","last_clean":"2024-04-15T18:36:17.840549+0000","last_became_active":"2024-04-15T18:35:58.477290+0000","last_became_peered":"2024-04-15T18:35:58.477290+0000","last_unstale":"2024-04-15T18:36:17.840549+0000","last_undegraded":"2024-04-15T18:36:17.840549+0000","last_fullsized":"2024-04-15T18:36:17.840549+0000","mapping_epoch":606,"log_start":"552'2001","ondisk_log_start":"552'2001","created":541,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'11555","last_scrub_stamp":"2024-04-15T18:36:08.033644+0000","last_deep_scrub":"607'11555","last_deep_scrub_stamp":"2024-04-15T18:36:08.033644+0000","last_clean_scrub_stamp":"2024-04-15T18:36:08.033644+0000","objects_scrubbed":773,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:09.413555+0000","scrub_duration":5744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":337248256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"611'11293","reported_seq":11463,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-04-15T18:36:21.319644+0000","last_change":"2024-04-15T18:35:54.492058+0000","last_active":"2024-04-15T18:36:21.319644+0000","last_peered":"2024-04-15T18:36:21.319644+0000","last_clean":"2024-04-15T18:36:21.319644+0000","last_became_active":"2024-04-15T18:32:30.474877+0000","last_became_peered":"2024-04-15T18:32:30.474877+0000","last_unstale":"2024-04-15T18:36:21.319644+0000","last_undegraded":"2024-04-15T18:36:21.319644+0000","last_fullsized":"2024-04-15T18:36:21.319644+0000","mapping_epoch":541,"log_start":"548'1200","ondisk_log_start":"548'1200","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10193","last_scrub_stamp":"2024-04-15T18:35:54.491971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:54.491971+0000","objects_scrubbed":1101,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:59.573582+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11293,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"609'11008","reported_seq":21883,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.773195+0000","last_change":"2024-04-15T18:35:57.590647+0000","last_active":"2024-04-15T18:36:17.773195+0000","last_peered":"2024-04-15T18:36:17.773195+0000","last_clean":"2024-04-15T18:36:17.773195+0000","last_became_active":"2024-04-15T18:35:51.865798+0000","last_became_peered":"2024-04-15T18:35:51.865798+0000","last_unstale":"2024-04-15T18:36:17.773195+0000","last_undegraded":"2024-04-15T18:36:17.773195+0000","last_fullsized":"2024-04-15T18:36:17.773195+0000","mapping_epoch":602,"log_start":"547'1000","ondisk_log_start":"547'1000","created":541,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'10225","last_scrub_stamp":"2024-04-15T18:35:57.590483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:57.590483+0000","objects_scrubbed":966,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:03.223994+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":352704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5343,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,9],"acting":[5,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"609'10891","reported_seq":11096,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.397625+0000","last_change":"2024-04-15T18:36:01.917202+0000","last_active":"2024-04-15T18:36:17.397625+0000","last_peered":"2024-04-15T18:36:17.397625+0000","last_clean":"2024-04-15T18:36:17.397625+0000","last_became_active":"2024-04-15T18:32:30.981832+0000","last_became_peered":"2024-04-15T18:32:30.981832+0000","last_unstale":"2024-04-15T18:36:17.397625+0000","last_undegraded":"2024-04-15T18:36:17.397625+0000","last_fullsized":"2024-04-15T18:36:17.397625+0000","mapping_epoch":541,"log_start":"547'800","ondisk_log_start":"547'800","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10251","last_scrub_stamp":"2024-04-15T18:36:01.916949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.916949+0000","objects_scrubbed":848,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:03.231241+0000","scrub_duration":1366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10891,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,8],"acting":[9,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.8","version":"611'11172","reported_seq":11341,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-04-15T18:36:21.308438+0000","last_change":"2024-04-15T18:35:56.721189+0000","last_active":"2024-04-15T18:36:21.308438+0000","last_peered":"2024-04-15T18:36:21.308438+0000","last_clean":"2024-04-15T18:36:21.308438+0000","last_became_active":"2024-04-15T18:32:30.735680+0000","last_became_peered":"2024-04-15T18:32:30.735680+0000","last_unstale":"2024-04-15T18:36:21.308438+0000","last_undegraded":"2024-04-15T18:36:21.308438+0000","last_fullsized":"2024-04-15T18:36:21.308438+0000","mapping_epoch":541,"log_start":"548'1100","ondisk_log_start":"548'1100","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10217","last_scrub_stamp":"2024-04-15T18:35:56.720948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:56.720948+0000","objects_scrubbed":968,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:01.442649+0000","scrub_duration":359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":357632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"609'11134","reported_seq":11374,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.802653+0000","last_change":"2024-04-15T18:35:25.111330+0000","last_active":"2024-04-15T18:36:17.802653+0000","last_peered":"2024-04-15T18:36:17.802653+0000","last_clean":"2024-04-15T18:36:17.802653+0000","last_became_active":"2024-04-15T18:32:30.479725+0000","last_became_peered":"2024-04-15T18:32:30.479725+0000","last_unstale":"2024-04-15T18:36:17.802653+0000","last_undegraded":"2024-04-15T18:36:17.802653+0000","last_fullsized":"2024-04-15T18:36:17.802653+0000","mapping_epoch":541,"log_start":"548'1100","ondisk_log_start":"548'1100","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"595'8959","last_scrub_stamp":"2024-04-15T18:35:25.111058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:25.111058+0000","objects_scrubbed":2343,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:36:48.179384+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":361216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"611'11284","reported_seq":22425,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-04-15T18:36:21.293806+0000","last_change":"2024-04-15T18:36:01.585943+0000","last_active":"2024-04-15T18:36:21.293806+0000","last_peered":"2024-04-15T18:36:21.293806+0000","last_clean":"2024-04-15T18:36:21.293806+0000","last_became_active":"2024-04-15T18:35:54.445912+0000","last_became_peered":"2024-04-15T18:35:54.445912+0000","last_unstale":"2024-04-15T18:36:21.293806+0000","last_undegraded":"2024-04-15T18:36:21.293806+0000","last_fullsized":"2024-04-15T18:36:21.293806+0000","mapping_epoch":604,"log_start":"548'1200","ondisk_log_start":"548'1200","created":541,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10410","last_scrub_stamp":"2024-04-15T18:36:01.585745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.585745+0000","objects_scrubbed":878,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:13.405968+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11284,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5481,"num_bytes_recovered":359202816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"609'10897","reported_seq":11128,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.413934+0000","last_change":"2024-04-15T18:35:47.370846+0000","last_active":"2024-04-15T18:36:17.413934+0000","last_peered":"2024-04-15T18:36:17.413934+0000","last_clean":"2024-04-15T18:36:17.413934+0000","last_became_active":"2024-04-15T18:32:30.477443+0000","last_became_peered":"2024-04-15T18:32:30.477443+0000","last_unstale":"2024-04-15T18:36:17.413934+0000","last_undegraded":"2024-04-15T18:36:17.413934+0000","last_fullsized":"2024-04-15T18:36:17.413934+0000","mapping_epoch":541,"log_start":"547'800","ondisk_log_start":"547'800","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'9726","last_scrub_stamp":"2024-04-15T18:35:47.370689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:47.370689+0000","objects_scrubbed":1326,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:04.173269+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":353600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"611'11294","reported_seq":22452,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-04-15T18:36:21.267524+0000","last_change":"2024-04-15T18:36:04.752350+0000","last_active":"2024-04-15T18:36:21.267524+0000","last_peered":"2024-04-15T18:36:21.267524+0000","last_clean":"2024-04-15T18:36:21.267524+0000","last_became_active":"2024-04-15T18:35:47.841503+0000","last_became_peered":"2024-04-15T18:35:47.841503+0000","last_unstale":"2024-04-15T18:36:21.267524+0000","last_undegraded":"2024-04-15T18:36:21.267524+0000","last_fullsized":"2024-04-15T18:36:21.267524+0000","mapping_epoch":600,"log_start":"548'1200","ondisk_log_start":"548'1200","created":541,"last_epoch_clean":601,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10609","last_scrub_stamp":"2024-04-15T18:36:04.751692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:04.751692+0000","objects_scrubbed":691,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:08.590026+0000","scrub_duration":310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5490,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"609'11041","reported_seq":11269,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.840543+0000","last_change":"2024-04-15T18:35:55.811254+0000","last_active":"2024-04-15T18:36:17.840543+0000","last_peered":"2024-04-15T18:36:17.840543+0000","last_clean":"2024-04-15T18:36:17.840543+0000","last_became_active":"2024-04-15T18:32:30.477599+0000","last_became_peered":"2024-04-15T18:32:30.477599+0000","last_unstale":"2024-04-15T18:36:17.840543+0000","last_undegraded":"2024-04-15T18:36:17.840543+0000","last_fullsized":"2024-04-15T18:36:17.840543+0000","mapping_epoch":541,"log_start":"547'1000","ondisk_log_start":"547'1000","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'10120","last_scrub_stamp":"2024-04-15T18:35:55.810909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:55.810909+0000","objects_scrubbed":1092,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:14.214223+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"610'12157","reported_seq":12412,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-04-15T18:36:20.528737+0000","last_change":"2024-04-15T18:36:06.701881+0000","last_active":"2024-04-15T18:36:20.528737+0000","last_peered":"2024-04-15T18:36:20.528737+0000","last_clean":"2024-04-15T18:36:20.528737+0000","last_became_active":"2024-04-15T18:32:30.639506+0000","last_became_peered":"2024-04-15T18:32:30.639506+0000","last_unstale":"2024-04-15T18:36:20.528737+0000","last_undegraded":"2024-04-15T18:36:20.528737+0000","last_fullsized":"2024-04-15T18:36:20.528737+0000","mapping_epoch":541,"log_start":"552'2101","ondisk_log_start":"552'2101","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'11484","last_scrub_stamp":"2024-04-15T18:36:06.701721+0000","last_deep_scrub":"607'11484","last_deep_scrub_stamp":"2024-04-15T18:36:06.701721+0000","last_clean_scrub_stamp":"2024-04-15T18:36:06.701721+0000","objects_scrubbed":818,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:34.054642+0000","scrub_duration":6572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":364224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"609'11051","reported_seq":11214,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.363368+0000","last_change":"2024-04-15T18:36:01.392695+0000","last_active":"2024-04-15T18:36:17.363368+0000","last_peered":"2024-04-15T18:36:17.363368+0000","last_clean":"2024-04-15T18:36:17.363368+0000","last_became_active":"2024-04-15T18:32:30.982115+0000","last_became_peered":"2024-04-15T18:32:30.982115+0000","last_unstale":"2024-04-15T18:36:17.363368+0000","last_undegraded":"2024-04-15T18:36:17.363368+0000","last_fullsized":"2024-04-15T18:36:17.363368+0000","mapping_epoch":541,"log_start":"547'1000","ondisk_log_start":"547'1000","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10402","last_scrub_stamp":"2024-04-15T18:36:01.392490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:36:01.392490+0000","objects_scrubbed":829,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:18.367555+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"609'11037","reported_seq":11201,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-04-15T18:36:17.735223+0000","last_change":"2024-04-15T18:35:58.646760+0000","last_active":"2024-04-15T18:36:17.735223+0000","last_peered":"2024-04-15T18:36:17.735223+0000","last_clean":"2024-04-15T18:36:17.735223+0000","last_became_active":"2024-04-15T18:32:30.478365+0000","last_became_peered":"2024-04-15T18:32:30.478365+0000","last_unstale":"2024-04-15T18:36:17.735223+0000","last_undegraded":"2024-04-15T18:36:17.735223+0000","last_fullsized":"2024-04-15T18:36:17.735223+0000","mapping_epoch":541,"log_start":"547'1000","ondisk_log_start":"547'1000","created":541,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10289","last_scrub_stamp":"2024-04-15T18:35:58.646432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:32:29.457742+0000","last_clean_scrub_stamp":"2024-04-15T18:35:58.646432+0000","objects_scrubbed":919,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:08.206941+0000","scrub_duration":290,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1524,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-04-15T18:36:20.582408+0000","last_change":"2024-04-15T18:36:05.553043+0000","last_active":"2024-04-15T18:36:20.582408+0000","last_peered":"2024-04-15T18:36:20.582408+0000","last_clean":"2024-04-15T18:36:20.582408+0000","last_became_active":"2024-04-15T18:34:07.451006+0000","last_became_peered":"2024-04-15T18:34:07.451006+0000","last_unstale":"2024-04-15T18:36:20.582408+0000","last_undegraded":"2024-04-15T18:36:20.582408+0000","last_fullsized":"2024-04-15T18:36:20.582408+0000","mapping_epoch":577,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":578,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:36:05.552905+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:37:29.521306+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,9],"acting":[7,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":127926296,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177801,"num_write_kb":5752065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21869,"num_bytes_recovered":1395982336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1353527296,"data_stored":1353515080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160949,"ondisk_log_size":160949,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":502316,"kb_used_data":11616,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93242836,"statfs":{"total":95995035648,"available":95480664064,"internally_reserved":0,"allocated":11894784,"data_stored":11583095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":502462916},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187787,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416152,"kb_used_data":37904,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93329000,"statfs":{"total":95995035648,"available":95568896000,"internally_reserved":0,"allocated":38813696,"data_stored":38506055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514833,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520796,"kb_used_data":78676,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93224356,"statfs":{"total":95995035648,"available":95461740544,"internally_reserved":0,"allocated":80564224,"data_stored":80257103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":452721088},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874580,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325988,"kb_used_data":15768,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93419164,"statfs":{"total":95995035648,"available":95661223936,"internally_reserved":0,"allocated":16146432,"data_stored":15838315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168921,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436332,"kb_used_data":17824,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93308820,"statfs":{"total":95995035648,"available":95548231680,"internally_reserved":0,"allocated":18251776,"data_stored":17940107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575227,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":504344,"kb_used_data":49744,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93240808,"statfs":{"total":95995035648,"available":95478587392,"internally_reserved":0,"allocated":50937856,"data_stored":50630215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738749,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484196,"kb_used_data":11544,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93260956,"statfs":{"total":95995035648,"available":95499218944,"internally_reserved":0,"allocated":11821056,"data_stored":11512939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483193999,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289876,"kb_used_data":64908,"kb_used_omap":114,"kb_used_meta":224845,"kb_avail":93455276,"statfs":{"total":95995035648,"available":95698202624,"internally_reserved":0,"allocated":66465792,"data_stored":66162223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":230241521},"hb_peers":[0,1,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609129,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592340,"kb_used_data":94092,"kb_used_omap":44,"kb_used_meta":498195,"kb_avail":93152812,"statfs":{"total":95995035648,"available":95388479488,"internally_reserved":0,"allocated":96350208,"data_stored":96046639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":510152234},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986535,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500308,"kb_used_data":45708,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93244844,"statfs":{"total":95995035648,"available":95482720256,"internally_reserved":0,"allocated":46804992,"data_stored":46501423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703430,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":346072,"kb_used_data":60560,"kb_used_omap":4,"kb_used_meta":285499,"kb_avail":93399080,"statfs":{"total":95995035648,"available":95640657920,"internally_reserved":0,"allocated":62013440,"data_stored":61705799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":292351681},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9306112,"data_stored":9306112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46927872,"data_stored":46923800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61865984,"data_stored":61865984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":93454336,"data_stored":93454336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43253760,"data_stored":43253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54398976,"data_stored":54394904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":926810112,"data_stored":926810112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":524288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10223616,"data_stored":10223616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":63569920,"data_stored":63569920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":33034240,"data_stored":33030168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10158080,"data_stored":10158080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:36:22.820 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:36:22.820 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-15T18:36:22.878 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:22.907 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:22.927 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:22.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: pgmap v1597: 17 pgs: 17 active+clean; 123 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 633 op/s 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: osdmap e612: 12 total, 11 up, 11 in 2024-04-15T18:36:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:22 smithi082 ceph-mon[44206]: osdmap e613: 12 total, 11 up, 11 in 2024-04-15T18:36:23.031 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:23.060 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:23.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:23.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:23.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:23.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: pgmap v1597: 17 pgs: 17 active+clean; 123 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 633 op/s 2024-04-15T18:36:23.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:23.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:23.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:23.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: osdmap e612: 12 total, 11 up, 11 in 2024-04-15T18:36:23.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:22 smithi159 ceph-mon[45336]: osdmap e613: 12 total, 11 up, 11 in 2024-04-15T18:36:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:23.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:23.155 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:36:23.155 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:36:23.155 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:36:23.155 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:36:23.155 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:36:23.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: pgmap v1597: 17 pgs: 17 active+clean; 123 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 633 op/s 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: from='client.16740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: osdmap e612: 12 total, 11 up, 11 in 2024-04-15T18:36:23.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:22 smithi049 ceph-mon[40087]: osdmap e613: 12 total, 11 up, 11 in 2024-04-15T18:36:23.237 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:23.266 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:23.286 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:23.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:23.390 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:23.419 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:23.438 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:23.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:23.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:23.571 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:23.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:36:23.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:36:23.769 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:23.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:23.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:23.903 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:23.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:23.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:24.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:24.028 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:36:24.054 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:24.054 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:24.054 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:23 smithi159 ceph-mon[45336]: osdmap e614: 12 total, 11 up, 11 in 2024-04-15T18:36:24.056 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:24.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:24.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:24.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:24.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:23 smithi049 ceph-mon[40087]: osdmap e614: 12 total, 11 up, 11 in 2024-04-15T18:36:24.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:24.223 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:24.244 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:24.268 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:24.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:24.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:24.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:23 smithi082 ceph-mon[44206]: osdmap e614: 12 total, 11 up, 11 in 2024-04-15T18:36:24.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:24.305 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:24.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:24.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:24.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:24.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:24.395 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:24.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:24.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:24.442 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:24.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:24.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:24.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:24.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:24.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:24.603 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:24.676 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:24.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:24.721 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-15T18:36:24.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-15T18:36:24.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:24.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:25.058 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: pgmap v1600: 17 pgs: 17 active+clean; 3.8 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3747631795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.060 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:25.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: pgmap v1600: 17 pgs: 17 active+clean; 3.8 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3747631795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.163 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:25.190 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:25.212 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:25.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: pgmap v1600: 17 pgs: 17 active+clean; 3.8 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 770 op/s 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3747631795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:25.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:25.346 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:25.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:25.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:25.731 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:25.743 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:25.743 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:25.753 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:25.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:25.857 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:25.886 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:25.905 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:25.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:26.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:26.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.041 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.062 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:26.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:26.194 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:26.317 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:26.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:36:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:36:26.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:36:26.421 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:26.453 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.475 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:26.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:26.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:26.606 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.624 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:26.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:26.759 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.780 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:26.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:26.914 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:26.974 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:36:26.987 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:26.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:27.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:26 smithi049 ceph-mon[40087]: pgmap v1602: 17 pgs: 17 active+clean; 3.8 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 578 op/s 2024-04-15T18:36:27.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:36:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:26 smithi082 ceph-mon[44206]: pgmap v1602: 17 pgs: 17 active+clean; 3.8 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 578 op/s 2024-04-15T18:36:27.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:36:27.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:27.118 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:27.266 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:27.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:27.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:27.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:26 smithi159 ceph-mon[45336]: pgmap v1602: 17 pgs: 17 active+clean; 3.8 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 578 op/s 2024-04-15T18:36:27.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:36:27.398 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:27.419 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:27.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:27.441 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:27.526 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:27.558 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:27.579 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:27.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:27.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:27.713 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:28.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:27 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:28.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:36:28.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:27 smithi049 ceph-mon[40087]: osdmap e615: 12 total, 11 up, 11 in 2024-04-15T18:36:28.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:27 smithi049 ceph-mon[40087]: osdmap e616: 12 total, 11 up, 11 in 2024-04-15T18:36:28.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:27 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:28.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:36:28.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:27 smithi082 ceph-mon[44206]: osdmap e615: 12 total, 11 up, 11 in 2024-04-15T18:36:28.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:27 smithi082 ceph-mon[44206]: osdmap e616: 12 total, 11 up, 11 in 2024-04-15T18:36:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:27 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1747824323' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:36:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:27 smithi159 ceph-mon[45336]: osdmap e615: 12 total, 11 up, 11 in 2024-04-15T18:36:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:27 smithi159 ceph-mon[45336]: osdmap e616: 12 total, 11 up, 11 in 2024-04-15T18:36:28.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:36:29.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:28 smithi082 ceph-mon[44206]: pgmap v1604: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 356 op/s 2024-04-15T18:36:29.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:28 smithi049 ceph-mon[40087]: pgmap v1604: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 356 op/s 2024-04-15T18:36:29.393 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:29.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:29.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:28 smithi159 ceph-mon[45336]: pgmap v1604: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 356 op/s 2024-04-15T18:36:29.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:29.530 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:29.552 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:29.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:29.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:29.686 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:29.706 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:29.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:29.810 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:29.838 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:29.858 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:29.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:29.962 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:29.989 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.008 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:30.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:30.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.165 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:30.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:30.288 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:30.296 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:30.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:36:30.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:30.444 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.463 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:30.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:30.599 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.621 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:30.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:30.753 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.773 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:30.877 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:30.904 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:30.923 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:30.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:30.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:36:31.026 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:31.053 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:31.071 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:31.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:31.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:31.206 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:31.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:30 smithi049 ceph-mon[40087]: pgmap v1606: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 8 op/s 2024-04-15T18:36:31.227 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:31.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:31.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:30 smithi082 ceph-mon[44206]: pgmap v1606: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 8 op/s 2024-04-15T18:36:31.332 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:31.356 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:30 smithi159 ceph-mon[45336]: pgmap v1606: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 8 op/s 2024-04-15T18:36:31.359 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:31.377 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:31.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:31.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:31.513 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:31.535 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:31.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:31.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:31.670 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:31.843 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:31.868 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:31.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:31.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:32.034 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' removed 2024-04-15T18:36:32.040 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:32.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:36:32.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:36:32.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:36:32.669 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:33.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:33 smithi082 ceph-mon[44206]: pgmap v1607: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 6 op/s 2024-04-15T18:36:33.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:36:33.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:33 smithi082 ceph-mon[44206]: osdmap e617: 12 total, 11 up, 11 in 2024-04-15T18:36:33.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:33 smithi082 ceph-mon[44206]: osdmap e618: 12 total, 11 up, 11 in 2024-04-15T18:36:33.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:33 smithi159 ceph-mon[45336]: pgmap v1607: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 6 op/s 2024-04-15T18:36:33.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:36:33.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:33 smithi159 ceph-mon[45336]: osdmap e617: 12 total, 11 up, 11 in 2024-04-15T18:36:33.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:33 smithi159 ceph-mon[45336]: osdmap e618: 12 total, 11 up, 11 in 2024-04-15T18:36:33.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:33 smithi049 ceph-mon[40087]: pgmap v1607: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 6 op/s 2024-04-15T18:36:33.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1947469082' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:36:33.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:33 smithi049 ceph-mon[40087]: osdmap e617: 12 total, 11 up, 11 in 2024-04-15T18:36:33.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:33 smithi049 ceph-mon[40087]: osdmap e618: 12 total, 11 up, 11 in 2024-04-15T18:36:34.274 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:36:34.274 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:36:34.275 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-04-15T18:36:34.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-04-15T18:36:34.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:34.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:34.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:34 smithi049 ceph-mon[40087]: pgmap v1610: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:34.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:34 smithi049 ceph-mon[40087]: osdmap e619: 12 total, 11 up, 11 in 2024-04-15T18:36:34.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:34.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:34.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:34 smithi082 ceph-mon[44206]: pgmap v1610: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:34.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:34 smithi082 ceph-mon[44206]: osdmap e619: 12 total, 11 up, 11 in 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:34.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:34.855 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:34 smithi159 ceph-mon[45336]: pgmap v1610: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:34 smithi159 ceph-mon[45336]: osdmap e619: 12 total, 11 up, 11 in 2024-04-15T18:36:34.977 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:35.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:35.028 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:35.054 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:35.054 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:35.471 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:36:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3171218230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3171218230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3171218230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:36:35.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: pgmap v1612: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:36:36.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:36.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:36 smithi049 ceph-mon[40087]: osdmap e620: 12 total, 11 up, 11 in 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: pgmap v1612: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:36:36.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:36.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:36 smithi082 ceph-mon[44206]: osdmap e620: 12 total, 11 up, 11 in 2024-04-15T18:36:36.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: pgmap v1612: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/27367153' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:36.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:36 smithi159 ceph-mon[45336]: osdmap e620: 12 total, 11 up, 11 in 2024-04-15T18:36:37.201 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:37.230 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:37.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:37.340 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:37.361 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:37.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:37.495 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:37.515 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:37.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:37.646 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:37.666 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:37.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:37.801 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:37.822 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:37.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:37.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:36:37.954 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:37.973 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:37.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:38.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:38.103 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:38.121 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:38.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:38.225 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:38.257 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:38.273 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:36:38.274 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:36:38.274 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:36:38.274 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:36:38.274 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:36:38.278 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:38.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:38.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:38.411 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:38.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:38 smithi049 ceph-mon[40087]: pgmap v1614: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:38.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:38.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:38 smithi159 ceph-mon[45336]: pgmap v1614: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:38.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:38.742 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:38 smithi082 ceph-mon[44206]: pgmap v1614: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:38.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:38.764 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:38.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:38.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:38.900 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:38.921 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:38.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:38.941 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:39.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:39.055 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.075 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:39.179 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:39.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:36:39.208 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.227 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:39.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:39.363 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.383 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:39.487 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:39.516 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.536 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:39.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:39.668 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.742 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:39.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:36:39.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:39.884 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:39.941 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:39.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:40.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:40.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:40.102 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:40.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:40.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:40.236 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:40.485 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:40.485 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_5' created 2024-04-15T18:36:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:40.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:40.620 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:40 smithi082 ceph-mon[44206]: pgmap v1615: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-15T18:36:40.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:40 smithi049 ceph-mon[40087]: pgmap v1615: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:40.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-15T18:36:40.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:40 smithi159 ceph-mon[45336]: pgmap v1615: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:40.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-04-15T18:36:41.069 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:41.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:41.206 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:41.227 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:41.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:41.363 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:41.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:41.488 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:41.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-04-15T18:36:41.520 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:41.541 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:41.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:41.673 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:41.693 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:41.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-15T18:36:41.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:41 smithi049 ceph-mon[40087]: osdmap e621: 12 total, 11 up, 11 in 2024-04-15T18:36:41.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-15T18:36:41.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:41 smithi082 ceph-mon[44206]: osdmap e621: 12 total, 11 up, 11 in 2024-04-15T18:36:41.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:41.826 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:41.853 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:41.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/752256985' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-04-15T18:36:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:41 smithi159 ceph-mon[45336]: osdmap e621: 12 total, 11 up, 11 in 2024-04-15T18:36:41.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:41.960 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:41.998 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:42.006 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:42.009 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:42.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:42.052 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:42.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:42.099 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:42.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:42.149 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:42.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:42.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:42.224 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:42.250 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:42.252 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:42.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:42.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:42.283 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:42.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:42.389 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:42.410 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:42.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:42.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:42.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: pgmap v1617: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:42.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:36:42.529 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: osdmap e622: 12 total, 11 up, 11 in 2024-04-15T18:36:42.529 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3770994195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:42.529 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:42.529 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:42.543 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:42.563 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:42.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:42.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: pgmap v1617: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:42.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:36:42.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: osdmap e622: 12 total, 11 up, 11 in 2024-04-15T18:36:42.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3770994195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:42.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:42.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:42.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:42.697 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: pgmap v1617: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: osdmap e622: 12 total, 11 up, 11 in 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3770994195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:43 smithi049 ceph-mon[40087]: osdmap e623: 12 total, 11 up, 11 in 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:43 smithi082 ceph-mon[44206]: osdmap e623: 12 total, 11 up, 11 in 2024-04-15T18:36:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:43.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:43 smithi159 ceph-mon[45336]: osdmap e623: 12 total, 11 up, 11 in 2024-04-15T18:36:43.951 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:43.983 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.004 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:44.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:44.138 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.158 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:44.262 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:44.294 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.315 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:44.420 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:44.451 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.485 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:44.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:44.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:44.616 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.635 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:44.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:44 smithi159 ceph-mon[45336]: pgmap v1620: 17 pgs: 11 active+clean, 3 creating+peering, 3 unknown; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:44.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:44 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:44.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:44 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:44.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:44 smithi049 ceph-mon[40087]: pgmap v1620: 17 pgs: 11 active+clean, 3 creating+peering, 3 unknown; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:44.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:44 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:44.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:44 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:44.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:44.771 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:44 smithi082 ceph-mon[44206]: pgmap v1620: 17 pgs: 11 active+clean, 3 creating+peering, 3 unknown; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:44.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:44 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:44.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:44 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:44.792 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:44.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:44.926 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:44.946 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:44.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:45.050 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:45.079 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:45.098 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:45.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:45.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:45.229 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:45.249 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:45.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:45.353 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:45.379 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:45.398 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:45.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:45.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:45.534 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:45.553 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:45.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:36:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:36:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:45 smithi159 ceph-mon[45336]: osdmap e624: 12 total, 11 up, 11 in 2024-04-15T18:36:45.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:45.683 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:45.702 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:45.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:36:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:36:45.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:45 smithi082 ceph-mon[44206]: osdmap e624: 12 total, 11 up, 11 in 2024-04-15T18:36:45.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:45.831 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:36:45.831 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:36:45.832 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:45 smithi049 ceph-mon[40087]: osdmap e624: 12 total, 11 up, 11 in 2024-04-15T18:36:45.833 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:36:46.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:36:46.082 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:46.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:46.185 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:46.217 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:46.239 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:46.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:46.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:46.372 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:46.392 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:46.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:46.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:46.523 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:36:46.532 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:46.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:46 smithi159 ceph-mon[45336]: pgmap v1622: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:46.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:46.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:46 smithi049 ceph-mon[40087]: pgmap v1622: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:46.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:46.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:46 smithi082 ceph-mon[44206]: pgmap v1622: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:46.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:36:46.912 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:46.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:47.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:47.524 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-04-15T18:36:47.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:47 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:47.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:47.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:47 smithi082 ceph-mon[44206]: osdmap e625: 12 total, 11 up, 11 in 2024-04-15T18:36:47.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:36:47.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:47 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:47.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:47.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:47 smithi159 ceph-mon[45336]: osdmap e625: 12 total, 11 up, 11 in 2024-04-15T18:36:47.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:36:47.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:47 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:47.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2292795068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:36:47.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:47 smithi049 ceph-mon[40087]: osdmap e625: 12 total, 11 up, 11 in 2024-04-15T18:36:47.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:36:48.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:36:48.380 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:48.380 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:48.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:48 smithi049 ceph-mon[40087]: pgmap v1624: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:48.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:36:48.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:48 smithi049 ceph-mon[40087]: osdmap e626: 12 total, 11 up, 11 in 2024-04-15T18:36:48.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:48 smithi082 ceph-mon[44206]: pgmap v1624: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:48.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:36:48.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:48 smithi082 ceph-mon[44206]: osdmap e626: 12 total, 11 up, 11 in 2024-04-15T18:36:48.895 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:48.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:48 smithi159 ceph-mon[45336]: pgmap v1624: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:48.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4275931847' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:36:48.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:48 smithi159 ceph-mon[45336]: osdmap e626: 12 total, 11 up, 11 in 2024-04-15T18:36:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:48.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:49.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:49.033 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.054 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:49.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:49.187 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.207 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:49.312 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:49.344 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.366 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:49.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:36:49.469 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:49.496 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:49.501 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.522 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:49.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:49.654 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:49 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:36:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.677 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:49.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:49 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:36:49.699 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-04-15T18:36:49.701 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:36:49.701 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:36:49.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:36:49.756 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:36:49.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:36:49.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_149998 2024-04-15T18:36:49.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:36:49.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:36:49.782 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:49.811 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:49.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:49.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:49 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:36:49.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:49.981 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:50.337 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:50.343 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:50.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:50.449 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:50.477 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:50.499 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:50.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:50.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:50.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:50 smithi049 ceph-mon[40087]: pgmap v1626: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:50.645 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:50.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 2221 2205 137.799 137.812 0.017987 0.00707456 2024-04-15T18:36:50.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:50 smithi082 ceph-mon[44206]: pgmap v1626: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:50.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:50 smithi159 ceph-mon[45336]: pgmap v1626: 17 pgs: 17 active+clean; 577 KiB data, 4.2 GiB used, 979 GiB / 983 GiB avail 2024-04-15T18:36:51.098 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:51.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:51.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:51.230 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:51.249 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:51.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:51.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:51.393 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:51.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 3506 3490 109.052 80.3125 0.00563415 0.00912387 2024-04-15T18:36:52.752 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:52.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 4868 4852 101.074 85.125 0.00184059 0.00985367 2024-04-15T18:36:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:52.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:52.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:52.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:52 smithi159 ceph-mon[45336]: pgmap v1627: 17 pgs: 17 active+clean; 27 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.8 MiB/s wr, 60 op/s 2024-04-15T18:36:52.941 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:52.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:52 smithi049 ceph-mon[40087]: pgmap v1627: 17 pgs: 17 active+clean; 27 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.8 MiB/s wr, 60 op/s 2024-04-15T18:36:52.967 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:52.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:52 smithi082 ceph-mon[44206]: pgmap v1627: 17 pgs: 17 active+clean; 27 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.8 MiB/s wr, 60 op/s 2024-04-15T18:36:53.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:53.110 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:53.139 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:53.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:53.197 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:53.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:53.281 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:53.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:53.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:53.549 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:53.573 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:53.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:53.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:36:53.724 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:53.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 6652 6636 103.679 111.5 0.00677266 0.00962331 2024-04-15T18:36:54.185 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:36:54.185 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:36:54.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 8556 8540 106.741 119 0.0193352 0.00935502 2024-04-15T18:36:54.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:54 smithi159 ceph-mon[45336]: pgmap v1628: 17 pgs: 17 active+clean; 206 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 26 MiB/s wr, 411 op/s 2024-04-15T18:36:54.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:54 smithi049 ceph-mon[40087]: pgmap v1628: 17 pgs: 17 active+clean; 206 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 26 MiB/s wr, 411 op/s 2024-04-15T18:36:55.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:54 smithi082 ceph-mon[44206]: pgmap v1628: 17 pgs: 17 active+clean; 206 MiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 26 MiB/s wr, 411 op/s 2024-04-15T18:36:55.486 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:36:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:55.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:36:55.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:55.625 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:55.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 9832 9816 102.242 79.75 0.00398934 0.00976876 2024-04-15T18:36:56.408 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:56.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:56.514 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:56.542 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:56.562 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:56.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:56.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:56.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:56.714 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:56.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:56 smithi049 ceph-mon[40087]: pgmap v1629: 17 pgs: 17 active+clean; 309 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 35 MiB/s wr, 553 op/s 2024-04-15T18:36:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:56.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:56.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 11766 11750 104.903 120.875 0.01818 0.00950476 2024-04-15T18:36:56.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:36:56.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:56.860 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:56 smithi159 ceph-mon[45336]: pgmap v1629: 17 pgs: 17 active+clean; 309 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 35 MiB/s wr, 553 op/s 2024-04-15T18:36:56.863 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:56.884 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:56.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:56.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:36:57.027 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:56 smithi082 ceph-mon[44206]: pgmap v1629: 17 pgs: 17 active+clean; 309 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 35 MiB/s wr, 553 op/s 2024-04-15T18:36:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:36:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:36:57.237 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:36:57.264 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.284 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:36:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:36:57.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:57.415 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.435 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:57.501 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:36:57.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:36:57.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:36:57.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:36:57.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:36:57.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:36:57.528 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:36:57.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:57.566 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:36:57.575 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.598 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:57.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:36:57.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:36:57.672 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:36:57.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:36:57.688 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:36:57.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:36:57.732 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.755 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:36:57.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 12996 12980 101.399 76.875 0.0143019 0.00985072 2024-04-15T18:36:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:36:57.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:57.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2400173870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:57.886 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:57.904 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:57.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:57.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2400173870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:58.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:36:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2400173870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:36:58.039 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:58.063 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:36:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:58.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:36:58.167 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:36:58.203 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:58.227 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:36:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:58.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:36:58.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:58.364 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:58.503 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:58.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:58.600 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.600 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.600 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:36:58.625 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:36:58.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:58.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: pgmap v1630: 17 pgs: 17 active+clean; 403 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3066734281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:58.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:58 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:58.664 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:36:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:58.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:36:58.757 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 15 14543 14528 100.881 96.75 0.00916817 0.00990112 2024-04-15T18:36:58.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:36:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: pgmap v1630: 17 pgs: 17 active+clean; 403 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3066734281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:58.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:58 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:58.906 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:36:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:58.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:36:59.012 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:36:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: pgmap v1630: 17 pgs: 17 active+clean; 403 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 40 MiB/s wr, 644 op/s 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3066734281' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:59.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:58 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:36:59.040 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:59.061 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:36:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:59.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:36:59.165 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:36:59.204 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:59.228 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:36:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:59.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:36:59.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:59.373 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:36:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:36:59.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:36:59.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:36:59.539 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:36:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:59 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:59 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:59.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:36:59 smithi049 ceph-mon[40087]: osdmap e627: 12 total, 11 up, 11 in 2024-04-15T18:36:59.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 15 16149 16134 100.83 100.375 0.0229695 0.00990697 2024-04-15T18:36:59.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:59 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:36:59.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:59 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:36:59.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:36:59 smithi159 ceph-mon[45336]: osdmap e627: 12 total, 11 up, 11 in 2024-04-15T18:37:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:59 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:59 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:00.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:36:59 smithi082 ceph-mon[44206]: osdmap e627: 12 total, 11 up, 11 in 2024-04-15T18:37:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:37:00.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:37:00.490 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:00.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:00.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:00.622 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:00.644 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:00.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:00.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:00 smithi049 ceph-mon[40087]: pgmap v1632: 17 pgs: 17 active+clean; 720 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:37:00.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:00.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 17280 17264 98.083 70.625 0.0471705 0.0101202 2024-04-15T18:37:00.775 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:00.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:00 smithi159 ceph-mon[45336]: pgmap v1632: 17 pgs: 17 active+clean; 720 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:37:00.976 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:00.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:00 smithi082 ceph-mon[44206]: pgmap v1632: 17 pgs: 17 active+clean; 720 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:37:01.080 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:01.113 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.137 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:01.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:01.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:01.270 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.294 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:01.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:01.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:01.428 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.448 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:01.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:01.553 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:01.587 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.613 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:01.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:01.631 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:37:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:37:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:37:01.631 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:37:01.632 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:37:01.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:01.750 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 18555 18539 96.5496 79.6875 0.0132511 0.0103384 2024-04-15T18:37:01.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:01.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:01.878 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:01.925 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:01.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:37:02.382 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:02.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:02.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:37:02.461 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:02.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:37:02.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:02.515 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:02.538 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:02.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:02 smithi049 ceph-mon[40087]: pgmap v1633: 17 pgs: 17 active+clean; 891 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:02.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:02.669 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:02.690 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:02.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:02.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 20289 20273 97.4586 108.375 0.0109002 0.010247 2024-04-15T18:37:02.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:02 smithi082 ceph-mon[44206]: pgmap v1633: 17 pgs: 17 active+clean; 891 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:02.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:02.821 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-04-15T18:37:02.821 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-04-15T18:37:02.822 DEBUG:teuthology.orchestra.run.smithi082:osd.6> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6.service 2024-04-15T18:37:02.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl start ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6 2024-04-15T18:37:02.841 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:02.852 INFO:journalctl@ceph.osd.6.smithi082.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:37:02.865 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:02.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:02.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:02 smithi159 ceph-mon[45336]: pgmap v1633: 17 pgs: 17 active+clean; 891 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:02.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:03.007 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:03.030 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:03.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:03.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:03.277 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:02 smithi082 systemd[1]: Starting Ceph osd.6 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:37:03.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:03.738 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:03.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:03.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 21542 21526 96.0906 78.3125 0.00343204 0.0103767 2024-04-15T18:37:03.843 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:03.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:03.910 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:03.985 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:04.012 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:04.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:04.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:04.158 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:04.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 22512 22496 93.7259 60.625 0.0150535 0.0106594 2024-04-15T18:37:04.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:04 smithi082 ceph-mon[44206]: pgmap v1634: 17 pgs: 17 active+clean; 1.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T18:37:04.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:04 smithi082 bash[153484]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-15T18:37:04.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:04 smithi159 ceph-mon[45336]: pgmap v1634: 17 pgs: 17 active+clean; 1.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-15T18:37:04.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:04 smithi049 ceph-mon[40087]: pgmap v1634: 17 pgs: 17 active+clean; 1.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-04-15T18:37:05.093 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:05.095 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:05.116 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:05.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:05.220 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:05.250 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:05.271 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:05.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:05.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:05.471 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:05.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:05.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:05.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:05 smithi049 ceph-mon[40087]: osdmap e628: 12 total, 11 up, 11 in 2024-04-15T18:37:05.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 23775 23759 92.8007 78.9375 0.00177774 0.0107365 2024-04-15T18:37:05.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:05.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:05.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:05 smithi159 ceph-mon[45336]: osdmap e628: 12 total, 11 up, 11 in 2024-04-15T18:37:05.948 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:05 smithi082 ceph-osd[153913]: -- 172.21.15.82:0/3503773914 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fbdfa91680 con 0x55fbdfa5ac00 2024-04-15T18:37:05.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:05.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:05.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:05 smithi082 ceph-mon[44206]: osdmap e628: 12 total, 11 up, 11 in 2024-04-15T18:37:05.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:37:06.226 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:06.277 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:05 smithi082 bash[153875]: 00463963543c12e5e126a56eccb1a2c3c67b0b1259e0a834cf83cbde1f09b28e 2024-04-15T18:37:06.277 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:05 smithi082 systemd[1]: Started Ceph osd.6 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:37:06.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 15 25102 25087 92.2237 83 0.00815378 0.010836 2024-04-15T18:37:06.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:06 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:06.414+0000 7f0704c417c0 -1 Falling back to public interface 2024-04-15T18:37:06.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:06 smithi082 ceph-mon[44206]: pgmap v1636: 17 pgs: 17 active+clean; 1.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:37:06.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:06 smithi049 ceph-mon[40087]: pgmap v1636: 17 pgs: 17 active+clean; 1.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:37:06.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:06 smithi159 ceph-mon[45336]: pgmap v1636: 17 pgs: 17 active+clean; 1.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:37:07.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 26908 26892 93.3671 112.812 0.00607343 0.010702 2024-04-15T18:37:08.284 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-04-15T18:37:08.284 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-04-15T18:37:08.284 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-04-15T18:37:08.285 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-04-15T18:37:08.646 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:08 smithi082 ceph-mon[44206]: pgmap v1637: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:37:08.646 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:08.654 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:37:08.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 28401 28385 93.3638 93.3125 0.00235937 0.0106973 2024-04-15T18:37:08.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:08 smithi159 ceph-mon[45336]: pgmap v1637: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:37:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:08 smithi049 ceph-mon[40087]: pgmap v1637: 17 pgs: 17 active+clean; 1.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-04-15T18:37:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-15T18:37:09.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-04-15T18:37:09.041 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:09.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:09.150 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:09.188 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:09.213 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:09.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:09.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:09.372 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:09.632 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:09.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:09.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:09.758 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:37:09.769587+0000 min lat: 0.00107291 max lat: 0.452874 avg lat: 0.0106825 2024-04-15T18:37:09.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:37:09.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 29949 29933 93.5328 96.75 0.00657157 0.0106825 2024-04-15T18:37:09.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:09.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-mon[44206]: osdmap e629: 12 total, 11 up, 11 in 2024-04-15T18:37:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-mon[44206]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:09.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:09.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.526+0000 7f0704c417c0 -1 osd.6 561 log_to_monitors true 2024-04-15T18:37:09.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.686+0000 7f06f1a24700 -1 osd.6 561 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 562, got 0 bytes 2024-04-15T18:37:09.778 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 563, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 562, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 563, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 564, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 565, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 566, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 567, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 568, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df86c700 -1 osd.6 570 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 562, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 563, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 564, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 565, got 0 bytes 2024-04-15T18:37:09.779 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 566, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 567, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 568, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e2872700 -1 osd.6 570 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 562, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 563, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 564, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 565, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 566, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 567, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 568, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06e086e700 -1 osd.6 570 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 564, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 565, got 0 bytes 2024-04-15T18:37:09.780 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 566, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 567, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 568, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06df06b700 -1 osd.6 570 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 562, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 563, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 564, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 565, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 566, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 567, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 568, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.691+0000 7f06de069700 -1 osd.6 570 failed to load OSD map for epoch 569, got 0 bytes 2024-04-15T18:37:09.781 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:37:09 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:37:09.707+0000 7f06f6099700 -1 osd.6 630 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:37:09.782 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:09.808 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:09.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:09.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:09.941 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:09.962 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:09.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:09 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:09.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:09.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:09 smithi049 ceph-mon[40087]: osdmap e629: 12 total, 11 up, 11 in 2024-04-15T18:37:09.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:09 smithi049 ceph-mon[40087]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:09.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:09 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:09.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:10.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:10.104 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:09 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:10.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/369979483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:10.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:09 smithi159 ceph-mon[45336]: osdmap e629: 12 total, 11 up, 11 in 2024-04-15T18:37:10.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:09 smithi159 ceph-mon[45336]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:10.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:09 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T18:37:10.149 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:10.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:10.296 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.320 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:10.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:10.452 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.474 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:10.578 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:10.612 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.633 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:10.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:10.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 31452 31436 93.5517 93.9375 0.00511634 0.0106804 2024-04-15T18:37:10.774 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: pgmap v1639: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:37:10.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:37:10.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: osdmap e630: 12 total, 11 up, 11 in 2024-04-15T18:37:10.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:10.798 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:10.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: pgmap v1639: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:37:10.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:37:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: osdmap e630: 12 total, 11 up, 11 in 2024-04-15T18:37:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:10.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:10.933 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:10.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:10.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: pgmap v1639: 17 pgs: 17 active+clean; 1.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: from='osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: osdmap e630: 12 total, 11 up, 11 in 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-04-15T18:37:10.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:11.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:11.087 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:11.106 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:11.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:11.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:11.241 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:11.467 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:37:11.467 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:37:11.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:37:11.467 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head, version: 402'5230, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:60e0f001:::benchmark_data_smithi159_116570_object85135:head@402'5230, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 579.38145982599997, 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66078538499999995, 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:37:11.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093316+0000", 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.093367+0000", 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0549e-05 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752462+0000", 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.65909545999999997 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752479+0000", 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6592e-05 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752556+0000", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6841000000000005e-05 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.752564+0000", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6589999999999996e-06 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.754101+0000", 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015365310000000001 2024-04-15T18:37:11.471 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.a 436/405 [PushOp(5:54042545:::benchmark_data_smithi159_116570_object78839:head, version: 397'4917, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:54042545:::benchmark_data_smithi159_116570_object78839:head@397'4917, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 578.75023490900003, 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50450683799999996, 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724540+0000", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724541+0000", 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7700000000000004e-07 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:32.724582+0000", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1350999999999998e-05 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227396+0000", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.502813128 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227405+0000", 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9379999999999998e-06 2024-04-15T18:37:11.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227470+0000", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5637999999999997e-05 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.227479+0000", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4700000000000002e-06 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.229047+0000", 2024-04-15T18:37:11.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001568636 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:6189473b:::benchmark_data_smithi159_116570_object88044:head, version: 403'5400, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6189473b:::benchmark_data_smithi159_116570_object88044:head@403'5400, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 577.969219641, 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48358990099999999, 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505556+0000", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505557+0000", 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.55e-06 2024-04-15T18:37:11.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.505641+0000", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3617999999999998e-05 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987415+0000", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48177457400000001 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987426+0000", 2024-04-15T18:37:11.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0428000000000001e-05 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987508+0000", 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2164999999999997e-05 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.987517+0000", 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5420000000000005e-06 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:33.989146+0000", 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001629024 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPush(5.6 436/405 [PushOp(5:62260e07:::benchmark_data_smithi159_116570_object92807:head, version: 403'5695, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:62260e07:::benchmark_data_smithi159_116570_object92807:head@403'5695, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 576.33199327099999, 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49781294999999998, 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.480 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142782+0000", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142783+0000", 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3300000000000002e-07 2024-04-15T18:37:11.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.142843+0000", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0627000000000001e-05 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638705+0000", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49586129000000001 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638724+0000", 2024-04-15T18:37:11.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9579999999999999e-05 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638840+0000", 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011596599999999999 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.638857+0000", 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6864e-05 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:35.640595+0000", 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00173799 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] ondisk+write+known_if_redirected e447)", 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.737278027, 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55780170699999998, 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737497+0000", 2024-04-15T18:37:11.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737499+0000", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2920000000000001e-06 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737515+0000", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737531+0000", 2024-04-15T18:37:11.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6053e-05 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737542+0000", 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0835999999999999e-05 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737608+0000", 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6070999999999998e-05 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737764+0000", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156816 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.737886+0000", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121797 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.740403+0000", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002517039 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.741178+0000", 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077443900000000003 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295224+0000", 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.55404577600000005 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295259+0000", 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5099999999999999e-05 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295299+0000", 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0435000000000002e-05 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:115172 5.a 5:5b6cba54:::benchmark_data_smithi159_116570_object22012:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e447)", 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.68786729199996, 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50835932699999997, 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.490 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for ondisk", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 115172 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786908+0000", 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786909+0000", 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786912+0000", 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7390000000000001e-06 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786921+0000", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2730000000000002e-06 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786928+0000", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1709999999999996e-06 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.786991+0000", 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.321e-05 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for ondisk", 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:51.787066+0000", 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4807000000000004e-05 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:27:52.295267+0000", 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50820147699999996 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.494 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25447.0:126776 5.a 5:5a87c32d:::benchmark_data_smithi159_116570_object33616:head [delete] snapc 0=[] ondisk+write+known_if_redirected e480)", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 479.76753538299999, 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.477931355, 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25447", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 126776 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707240+0000", 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707242+0000", 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.39e-06 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707255+0000", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2709e-05 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707287+0000", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2403000000000001e-05 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707306+0000", 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8539e-05 2024-04-15T18:37:11.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707479+0000", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017332900000000001 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707663+0000", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018317 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 8,10", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.707820+0000", 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015760200000000001 2024-04-15T18:37:11.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.709796+0000", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019761599999999998 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:11.711733+0000", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019369490000000001 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185060+0000", 2024-04-15T18:37:11.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47332694600000003 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185114+0000", 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4221999999999997e-05 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:29:12.185171+0000", 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6935999999999998e-05 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26224.0:179753 6.0 6:0844f8ba:::benchmark_data_smithi159_139342_object89874:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.097813827000003, 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60003534400000003, 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26224", 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 179753 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376963+0000", 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-04-15T18:37:11.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376968+0000", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8640000000000004e-06 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376978+0000", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.064e-05 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376984+0000", 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2249999999999997e-06 2024-04-15T18:37:11.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377476+0000", 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049176799999999996 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377618+0000", 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014151600000000001 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-04-15T18:37:11.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377745+0000", 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012707899999999999 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.383798+0000", 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060528409999999998 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.387084+0000", 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032860659999999998 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976917+0000", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.58983302100000001 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976961+0000", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3986000000000002e-05 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976997+0000", 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6151e-05 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26224.0:179753 6.0 e611/541 ondisk, result = 0)", 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.089174114000002, 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59140211200000004, 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:37:11.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385602+0000", 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.035e-06 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385608+0000", 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6829999999999996e-06 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976844+0000", 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59123575299999998 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976853+0000", 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2690000000000005e-06 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976903+0000", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0364000000000001e-05 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976911+0000", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.515e-06 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.977003+0000", 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2492999999999997e-05 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.510 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:13058 7.2 e626/621 7:417488d0:::benchmark_data_smithi159_149998_object13057:head v 626'835, mlcod=626'833)", 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:57.817302+0000", 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 13.657473403999999, 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.151709327, 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.817302+0000", 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.817302+0000", 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.817303+0000", 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.44e-07 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.817378+0000", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5329999999999999e-05 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.968299+0000", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15092138499999999 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.968307+0000", 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4510000000000003e-06 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.968357+0000", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0015000000000001e-05 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.968380+0000", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2722999999999999e-05 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.968972+0000", 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00059284400000000003 2024-04-15T18:37:11.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:57.969011+0000", 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8934999999999999e-05 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:17271 7.8 7:1e91ebe9:::benchmark_data_smithi159_149998_object17270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e627)", 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:00.698010+0000", 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.776765833000001, 2024-04-15T18:37:11.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13365169499999999, 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 17271 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698010+0000", 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698010+0000", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698010+0000", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-07 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698086+0000", 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5550999999999995e-05 2024-04-15T18:37:11.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698096+0000", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0417999999999999e-05 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698102+0000", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7230000000000001e-06 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698142+0000", 2024-04-15T18:37:11.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0253999999999998e-05 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698220+0000", 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7656000000000003e-05 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698309+0000", 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8979000000000003e-05 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.698977+0000", 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066863700000000003 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.699425+0000", 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00044792999999999998 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.831594+0000", 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.132169075 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.831632+0000", 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7032000000000001e-05 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:00.831661+0000", 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9689999999999999e-05 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:20978 7.3 7:c2ae995e:::benchmark_data_smithi159_149998_object20977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e627)", 2024-04-15T18:37:11.521 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 8.2682586100000002, 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24318234699999999, 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 20978 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206518+0000", 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206580+0000", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2403000000000005e-05 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206591+0000", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899e-05 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206597+0000", 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0929999999999999e-06 2024-04-15T18:37:11.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206678+0000", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0079000000000004e-05 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206844+0000", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000166534 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207012+0000", 2024-04-15T18:37:11.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167856 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.208175+0000", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001162672 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.208725+0000", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00055022600000000001 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449599+0000", 2024-04-15T18:37:11.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24087378400000001 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449656+0000", 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7376000000000001e-05 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449699+0000", 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3081999999999998e-05 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:21004 7.c e627/621 7:31cddcce:::benchmark_data_smithi159_149998_object21003:head v 627'1288, mlcod=627'1285)", 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.222091+0000", 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 8.2526840149999998, 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19256242900000001, 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.222091+0000", 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.222091+0000", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.222092+0000", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0599999999999999e-07 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.222175+0000", 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2960999999999994e-05 2024-04-15T18:37:11.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.413734+0000", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.191559065 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.413743+0000", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3769999999999996e-06 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.413811+0000", 2024-04-15T18:37:11.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8711e-05 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.413826+0000", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4832e-05 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.414623+0000", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079643699999999999 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.414654+0000", 2024-04-15T18:37:11.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1239999999999999e-05 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:22406 7.8 7:11ca545a:::benchmark_data_smithi159_149998_object22405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e627)", 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.515036+0000", 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.9597395090000003, 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20469189500000001, 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.532 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 22406 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515036+0000", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515036+0000", 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515037+0000", 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2299999999999999e-07 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515166+0000", 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129135 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515181+0000", 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5296999999999999e-05 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515190+0000", 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5590000000000006e-06 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515244+0000", 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4441000000000003e-05 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515340+0000", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5630000000000004e-05 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.515439+0000", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8688000000000001e-05 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516512+0000", 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001073472 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516894+0000", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038187099999999999 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719623+0000", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.202729558 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719685+0000", 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1739e-05 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719728+0000", 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2582e-05 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:22406 7.8 e627/621 ondisk, result = 0)", 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.516690+0000", 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.958084919, 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20304507199999999, 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516690+0000", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516690+0000", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516691+0000", 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9999999999999996e-07 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.516700+0000", 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7490000000000004e-06 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719556+0000", 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20285642600000001 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719562+0000", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8019999999999997e-06 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719608+0000", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6217999999999997e-05 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719616+0000", 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7889999999999995e-06 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.719736+0000", 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119288 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:24093 7.3 7:cde12f3f:::benchmark_data_smithi159_149998_object24092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e628)", 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.040620+0000", 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4341549650000003, 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22287336799999999, 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 24093 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040620+0000", 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040620+0000", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040621+0000", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4099999999999997e-07 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040670+0000", 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9039000000000003e-05 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040681+0000", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0759e-05 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040687+0000", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9440000000000002e-06 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040730+0000", 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2837000000000003e-05 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040906+0000", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017625599999999999 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041013+0000", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010666399999999999 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041913+0000", 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00090000799999999999 2024-04-15T18:37:11.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.042164+0000", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025087500000000001 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263363+0000", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22119961399999999 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263446+0000", 2024-04-15T18:37:11.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2356999999999998e-05 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263494+0000", 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8074000000000001e-05 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:24093 7.3 e628/621 ondisk, result = 0)", 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.041952+0000", 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4328233880000001, 2024-04-15T18:37:11.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.221548141, 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041952+0000", 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041952+0000", 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041953+0000", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0900000000000001e-07 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.041957+0000", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9979999999999997e-06 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.262468+0000", 2024-04-15T18:37:11.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22051186 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.262476+0000", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1099999999999997e-06 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263353+0000", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00087724400000000001 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263360+0000", 2024-04-15T18:37:11.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9099999999999999e-06 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263500+0000", 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014040999999999999 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:24102 7.3 7:c1a7580c:::benchmark_data_smithi159_149998_object24101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e628)", 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.049629+0000", 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4251458609999998, 2024-04-15T18:37:11.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21486764, 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 24102 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049629+0000", 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049629+0000", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049630+0000", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5900000000000005e-07 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049710+0000", 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9494999999999997e-05 2024-04-15T18:37:11.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049722+0000", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1953000000000001e-05 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049730+0000", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3149999999999992e-06 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049787+0000", 2024-04-15T18:37:11.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7006000000000003e-05 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049890+0000", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010286899999999999 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.049988+0000", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7683999999999995e-05 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.050975+0000", 2024-04-15T18:37:11.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00098707599999999993 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.051212+0000", 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023716699999999999 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264390+0000", 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21317778600000001 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264455+0000", 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5364999999999995e-05 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264497+0000", 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1965000000000002e-05 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:24102 7.3 e628/621 ondisk, result = 0)", 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.051144+0000", 2024-04-15T18:37:11.558 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4236311170000002, 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21336110999999999, 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.051144+0000", 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.051144+0000", 2024-04-15T18:37:11.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.051145+0000", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-07 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.051149+0000", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1019999999999999e-06 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263727+0000", 2024-04-15T18:37:11.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21257810799999999 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.263737+0000", 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7690000000000008e-06 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264369+0000", 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063191799999999998 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264376+0000", 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2010000000000002e-06 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264505+0000", 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129262 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:24110 7.2 e628/621 7:444ea4df:::benchmark_data_smithi159_149998_object24109:head v 628'1478, mlcod=628'1475)", 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.060942+0000", 2024-04-15T18:37:11.562 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4138334239999999, 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20599904399999999, 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.060942+0000", 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.060942+0000", 2024-04-15T18:37:11.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.060943+0000", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.061034+0000", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1384000000000005e-05 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.265634+0000", 2024-04-15T18:37:11.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204600315 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.265641+0000", 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3940000000000001e-06 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.265688+0000", 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7228000000000003e-05 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:37:11.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.265710+0000", 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1841e-05 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266881+0000", 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011714399999999999 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266941+0000", 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9744000000000003e-05 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:11.567 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:11.567 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:37:11.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 32158 32142 91.3049 44.125 0.00958908 0.0108788 2024-04-15T18:37:11.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:37:11.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:37:11.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:37:11.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:37:11.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:37:11.859 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:37:11.899 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:37:11.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:11 smithi049 ceph-mon[40087]: osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271] boot 2024-04-15T18:37:11.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:11 smithi049 ceph-mon[40087]: osdmap e631: 12 total, 12 up, 11 in 2024-04-15T18:37:11.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:11.940 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:37:11.978 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:37:12.018 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:37:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:11 smithi082 ceph-mon[44206]: osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271] boot 2024-04-15T18:37:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:11 smithi082 ceph-mon[44206]: osdmap e631: 12 total, 12 up, 11 in 2024-04-15T18:37:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:12.058 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:37:12.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:37:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:37:12.105 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:37:12.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:11 smithi159 ceph-mon[45336]: osd.6 [v2:172.21.15.82:6816/256106271,v1:172.21.15.82:6817/256106271] boot 2024-04-15T18:37:12.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:11 smithi159 ceph-mon[45336]: osdmap e631: 12 total, 12 up, 11 in 2024-04-15T18:37:12.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T18:37:12.391 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:12.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:12.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:37:12.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:12.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:12.546 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:12.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:12.682 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:12.706 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: pgmap v1642: 17 pgs: 17 active+clean; 1.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/602632638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osdmap e632: 12 total, 12 up, 11 in 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:12.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:12.720 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:12.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:37:12.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 33563 33547 91.1527 87.8125 0.00221905 0.0109571 2024-04-15T18:37:12.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:12.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:12.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: pgmap v1642: 17 pgs: 17 active+clean; 1.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/602632638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osdmap e632: 12 total, 12 up, 11 in 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: pgmap v1642: 17 pgs: 17 active+clean; 1.7 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/602632638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osdmap e632: 12 total, 12 up, 11 in 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.505 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:13.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:13.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:13.640 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:13.660 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:13.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:13.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 34576 34560 89.9925 63.3125 0.0147421 0.0110824 2024-04-15T18:37:13.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:13.791 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:13.897 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:13 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:37:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:13 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:13 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:14.240 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:14.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:14.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:14.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:14.418 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:14.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:14.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:14.701 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:14.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:14.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 35968 35952 89.8725 87 0.0227897 0.0111186 2024-04-15T18:37:14.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:14.807 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:14.839 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:14.957 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:37:14.957 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:37:14.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:14 smithi049 ceph-mon[40087]: pgmap v1644: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:37:14.983 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:14 smithi082 ceph-mon[44206]: pgmap v1644: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:37:15.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:14 smithi159 ceph-mon[45336]: pgmap v1644: 17 pgs: 17 active+clean; 2.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:37:15.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:37:15.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:15.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:15.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:15.631 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:15.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 37238 37222 89.4685 79.375 0.00878461 0.0111686 2024-04-15T18:37:16.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 38027 38011 87.9809 49.3125 0.0203365 0.0113502 2024-04-15T18:37:16.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:16 smithi049 ceph-mon[40087]: pgmap v1645: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-15T18:37:16.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:16.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:17.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:16 smithi082 ceph-mon[44206]: pgmap v1645: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-15T18:37:17.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:17.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:17.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:16 smithi159 ceph-mon[45336]: pgmap v1645: 17 pgs: 17 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-04-15T18:37:17.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:17.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:17.759 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:17.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 39232 39216 87.5282 75.3125 0.0145504 0.0114159 2024-04-15T18:37:18.694 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:37:18.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 40440 40424 87.1133 75.5 0.0402643 0.0114691 2024-04-15T18:37:18.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:18.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:18.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:37:18.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:18.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:18 smithi049 ceph-mon[40087]: pgmap v1646: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:37:18.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2130064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:18.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:18 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:18.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:19.000 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:19.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:19.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:18 smithi082 ceph-mon[44206]: pgmap v1646: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:37:19.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2130064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:19.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:18 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:19.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:19.130 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:18 smithi159 ceph-mon[45336]: pgmap v1646: 17 pgs: 17 active+clean; 2.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:37:19.130 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2130064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:19.130 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:18 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:19.133 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:19.155 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:19.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:19.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:19.299 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: osdmap e633: 12 total, 12 up, 11 in 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:19.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:19 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:19.740 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:19.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 41697 41681 86.8281 78.5625 0.00379616 0.0115112 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: osdmap e633: 12 total, 12 up, 11 in 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:20.001 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:19 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: osdmap e633: 12 total, 12 up, 11 in 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:20.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:19 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:20.753 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:20.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:20.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 42907 42891 86.4665 75.625 0.00298393 0.0115583 2024-04-15T18:37:20.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:20.931 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:20.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:20 smithi049 ceph-mon[40087]: pgmap v1648: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:20.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:20 smithi049 ceph-mon[40087]: osdmap e634: 12 total, 12 up, 11 in 2024-04-15T18:37:20.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:20.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:21.006 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:37:21.006 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:37:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:20 smithi082 ceph-mon[44206]: pgmap v1648: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:20 smithi082 ceph-mon[44206]: osdmap e634: 12 total, 12 up, 11 in 2024-04-15T18:37:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:21.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:21.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:20 smithi159 ceph-mon[45336]: pgmap v1648: 17 pgs: 17 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:21.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:20 smithi159 ceph-mon[45336]: osdmap e634: 12 total, 12 up, 11 in 2024-04-15T18:37:21.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:21.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:21.691 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:21.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:21.697 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:37:21.697 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-15T18:37:21.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:37:21.711 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:37:21.753 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:37:21.759 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 44006 43990 85.9107 68.6875 0.0201581 0.0116328 2024-04-15T18:37:21.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:21.796 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:37:21.824 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:21.832 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:37:21.847 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:21.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:21.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:37:21.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:37:21.912 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:37:21.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:37:21.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:21.963 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:37:21.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:21 smithi049 ceph-mon[40087]: osdmap e635: 12 total, 12 up, 11 in 2024-04-15T18:37:21.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/646807954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:21.976 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:21 smithi159 ceph-mon[45336]: osdmap e635: 12 total, 12 up, 11 in 2024-04-15T18:37:21.976 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/646807954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:21.978 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:21.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:37:22.007 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:22.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:22.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:21 smithi082 ceph-mon[44206]: osdmap e635: 12 total, 12 up, 11 in 2024-04-15T18:37:22.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/646807954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:22.037 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:37:22.076 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:37:22.081 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:37:22.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:37:22.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:22.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:22.397 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:22.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:22.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:22.549 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:22.658 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:37:22.659 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:37:22.659 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:37:22.659 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:37:22.659 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:37:22.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 45334 45318 85.8223 83 0.0152929 0.011645 2024-04-15T18:37:22.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:37:22.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: pgmap v1651: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:37:22.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:22 smithi049 ceph-mon[40087]: osdmap e636: 12 total, 12 up, 11 in 2024-04-15T18:37:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: pgmap v1651: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:37:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:22 smithi082 ceph-mon[44206]: osdmap e636: 12 total, 12 up, 11 in 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: pgmap v1651: 17 pgs: 17 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:22 smithi159 ceph-mon[45336]: osdmap e636: 12 total, 12 up, 11 in 2024-04-15T18:37:23.384 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:37:23.504 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:23.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:23.611 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:23.648 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:23.674 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:23.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:23.695 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:23.723 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:23.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:23.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 15 46500 46485 85.4432 72.9375 0.0294967 0.0116983 2024-04-15T18:37:23.778 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:23.812 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:23.838 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:23.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:23.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:23.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:23.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:23.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:23.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:23.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:23.977 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:24.001 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:24.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:24.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:24.143 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:24.407 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:24.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:24.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:24.554 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:24.664 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:24.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:24.673 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:24.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:24.674 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:24.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 47801 47785 85.3232 81.25 0.0483441 0.01171 2024-04-15T18:37:24.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:24.816 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:24.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:24.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:24.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:24 smithi049 ceph-mon[40087]: pgmap v1653: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:37:24.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:24 smithi049 ceph-mon[40087]: osdmap e637: 12 total, 12 up, 11 in 2024-04-15T18:37:24.973 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:24 smithi082 ceph-mon[44206]: pgmap v1653: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:37:24.973 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:24 smithi082 ceph-mon[44206]: osdmap e637: 12 total, 12 up, 11 in 2024-04-15T18:37:24.975 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:24.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:25.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:25.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:25.137 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:25.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:24 smithi159 ceph-mon[45336]: pgmap v1653: 17 pgs: 17 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-04-15T18:37:25.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:24 smithi159 ceph-mon[45336]: osdmap e637: 12 total, 12 up, 11 in 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.598 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:113653 5.f 5:fa10d0de:::benchmark_data_smithi159_116570_object20493:head [delete] snapc 0=[] ondisk+write+known_if_redirected e438)", 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 587.36099840999998, 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44737796299999999, 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 113653 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244938+0000", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244940+0000", 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5770000000000001e-06 2024-04-15T18:37:25.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244948+0000", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1540000000000002e-06 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244966+0000", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7779999999999999e-05 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.244974+0000", 2024-04-15T18:37:25.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4440000000000008e-06 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245031+0000", 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245129+0000", 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8061999999999997e-05 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.602 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.245207+0000", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7355999999999996e-05 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.247755+0000", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025486670000000001 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.248470+0000", 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071462499999999998 2024-04-15T18:37:25.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692202+0000", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44373232600000001 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692262+0000", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9703000000000001e-05 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:27:38.692316+0000", 2024-04-15T18:37:25.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4274999999999998e-05 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:118580 5.3 e459/415 5:c1d7d9b0:::benchmark_data_smithi159_116570_object25420:head v 459'7541, mlcod=459'7538)", 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 546.88818172100002, 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.48049194000000001, 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717755+0000", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717756+0000", 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3499999999999995e-07 2024-04-15T18:37:25.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:18.717761+0000", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5629999999999999e-06 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193762+0000", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47600057200000001 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193767+0000", 2024-04-15T18:37:25.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0919999999999997e-06 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193812+0000", 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5049999999999997e-05 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.193833+0000", 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0786000000000001e-05 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.608 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198189+0000", 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043559719999999996 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:19.198247+0000", 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.817e-05 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:119061 5.7 5:e4a9cef3:::benchmark_data_smithi159_116570_object25901:head [delete] snapc 0=[] ondisk+write+known_if_redirected e460)", 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 542.99478295500001, 2024-04-15T18:37:25.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52903251299999998, 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 119061 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611154+0000", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9599999999999998e-07 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611159+0000", 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4669999999999998e-06 2024-04-15T18:37:25.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611170+0000", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0614e-05 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611176+0000", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0249999999999999e-06 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611214+0000", 2024-04-15T18:37:25.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8059999999999998e-05 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611297+0000", 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3330000000000003e-05 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.611364+0000", 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6617000000000003e-05 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.613743+0000", 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002379459 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614653+0000", 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091005100000000005 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140066+0000", 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52541264300000001 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140152+0000", 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6450000000000001e-05 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140186+0000", 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3900999999999999e-05 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.615 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:119061 5.7 e460/396 ondisk, result = 0)", 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 542.99134155900003, 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52559624999999999, 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614595+0000", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6899999999999999e-07 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:22.614598+0000", 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3099999999999999e-06 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.139929+0000", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.52533111200000004 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.139940+0000", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1453e-05 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140035+0000", 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5049000000000004e-05 2024-04-15T18:37:25.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140048+0000", 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3239e-05 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:28:23.140191+0000", 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014281799999999999 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:127441 5.f 5:f29729c2:::benchmark_data_smithi159_116570_object34281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e481)", 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 490.420095903, 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.509987313, 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 127441 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185841+0000", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185842+0000", 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.088e-06 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185846+0000", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5889999999999996e-06 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185857+0000", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0915e-05 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185863+0000", 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.981e-06 2024-04-15T18:37:25.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185908+0000", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4372999999999997e-05 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.185994+0000", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6490999999999996e-05 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.186065+0000", 2024-04-15T18:37:25.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0646000000000001e-05 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.189580+0000", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035155519999999999 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.476339+0000", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286758654 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695753+0000", 2024-04-15T18:37:25.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.219414315 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695794+0000", 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0277000000000001e-05 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:15.695828+0000", 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4431999999999998e-05 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:129236 5.3 e486/415 5:ce6b8309:::benchmark_data_smithi159_116570_object36076:head v 486'8209, mlcod=484'8206)", 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 479.20625437400003, 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44633631600000001, 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399682+0000", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8e-07 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.399686+0000", 2024-04-15T18:37:25.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5980000000000001e-06 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843210+0000", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.44352421400000003 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843216+0000", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.767e-06 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843268+0000", 2024-04-15T18:37:25.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2008000000000001e-05 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.843284+0000", 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5656000000000001e-05 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.845981+0000", 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026974550000000001 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:26.846018+0000", 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7438000000000003e-05 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.25447.0:129784 5.f 5:fae35ede:::benchmark_data_smithi159_116570_object36624:head [delete] snapc 0=[] ondisk+write+known_if_redirected e486)", 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 476.30569429100001, 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57517218699999995, 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.630 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.25447", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/30035999", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 129784 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300242+0000", 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300244+0000", 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.266e-06 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300249+0000", 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.096e-06 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300266+0000", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7793999999999999e-05 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300275+0000", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.602e-06 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300326+0000", 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1465000000000003e-05 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300428+0000", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101933 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 4,11", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.300510+0000", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1563999999999994e-05 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304070+0000", 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035599609999999999 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304862+0000", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00079191600000000004 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875350+0000", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.570488459 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875384+0000", 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4087000000000001e-05 2024-04-15T18:37:25.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875414+0000", 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0043999999999999e-05 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.25447.0:129784 5.f e486/393 ondisk, result = 0)", 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 476.30127608599997, 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57075858000000002, 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.636 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304660+0000", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304661+0000", 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5899999999999995e-07 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.304666+0000", 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2459999999999997e-06 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875294+0000", 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.57062857700000003 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875300+0000", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2090000000000002e-06 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875342+0000", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1758000000000002e-05 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875347+0000", 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8409999999999999e-06 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:29:29.875419+0000", 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2089999999999996e-05 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.25447.0:144515 5.6 e508/437 5:63eb13b9:::benchmark_data_smithi159_116570_object51355:head v 508'8892, mlcod=507'8889)", 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 403.00674404300003, 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.474313188, 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.640 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599193+0000", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7999999999999995e-07 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:42.599198+0000", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6739999999999996e-06 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070635+0000", 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.47143691900000001 2024-04-15T18:37:25.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070639+0000", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.729e-06 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070689+0000", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9104000000000003e-05 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.070706+0000", 2024-04-15T18:37:25.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7028e-05 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.073477+0000", 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0027715420000000001 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:30:43.073506+0000", 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8612e-05 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:164354 6.d 6:bbcdb5bb:::benchmark_data_smithi159_139342_object74475:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.887760264999997, 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29154140200000001, 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 164354 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.645 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.646 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718177+0000", 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718181+0000", 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6820000000000001e-06 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718192+0000", 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0951000000000001e-05 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.647 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718197+0000", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.587e-06 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718252+0000", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5090999999999998e-05 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718368+0000", 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115615 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.648 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718486+0000", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117946 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.722722+0000", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042362509999999999 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723060+0000", 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033780300000000002 2024-04-15T18:37:25.649 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009629+0000", 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28656950199999998 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009670+0000", 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0249e-05 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.650 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009718+0000", 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8075000000000003e-05 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26224.0:164354 6.d e606/541 ondisk, result = 0)", 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 87.882875385000006, 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286662904, 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:37:25.651 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723062+0000", 2024-04-15T18:37:25.652 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.82e-07 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723065+0000", 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4290000000000001e-06 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009573+0000", 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286507593 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.653 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009576+0000", 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.501e-06 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009619+0000", 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2441999999999998e-05 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009626+0000", 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.058e-06 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.654 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009724+0000", 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8399000000000004e-05 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26224.0:177238 6.4 e609/541 6:2ec8682d:::benchmark_data_smithi159_139342_object87359:head v 609'10914, mlcod=609'10912)", 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 67.582737274999999, 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.2949041, 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.655 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023200+0000", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3399999999999998e-07 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023206+0000", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9259999999999997e-06 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313521+0000", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29031515699999999 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313528+0000", 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8569999999999997e-06 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313582+0000", 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3565e-05 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313607+0000", 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4990999999999998e-05 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.658 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.318066+0000", 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044594719999999999 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.318103+0000", 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7397999999999997e-05 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.659 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:179749 6.d 6:bfc4ccdc:::benchmark_data_smithi159_139342_object89870:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 64.237480042000001, 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.75209580899999995, 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179749 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.660 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368458+0000", 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.483e-06 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.661 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368463+0000", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0189999999999997e-06 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368476+0000", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2588e-05 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368483+0000", 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4599999999999997e-06 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.662 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368535+0000", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1928999999999997e-05 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368651+0000", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116334 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368760+0000", 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108328 2024-04-15T18:37:25.663 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.373250+0000", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044905420000000001 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.383059+0000", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0098083690000000008 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120474+0000", 2024-04-15T18:37:25.664 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.73741578100000005 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120520+0000", 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5677999999999999e-05 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120552+0000", 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2298000000000001e-05 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.665 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31772 7.0 7:002aa88c:::benchmark_data_smithi159_149998_object31771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.584466256000001, 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32090146899999999, 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31772 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.666 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021471+0000", 2024-04-15T18:37:25.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021582+0000", 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110575 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021593+0000", 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.075e-05 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021599+0000", 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3060000000000003e-06 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021642+0000", 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2583999999999998e-05 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021735+0000", 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2876999999999999e-05 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021822+0000", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7751999999999996e-05 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.022676+0000", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085419300000000005 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.023361+0000", 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00068498799999999998 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342304+0000", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31894291299999999 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342340+0000", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5759000000000001e-05 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342372+0000", 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1615000000000001e-05 2024-04-15T18:37:25.671 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26545.0:31792 7.b e631/621 7:d043e990:::benchmark_data_smithi159_149998_object31791:head v 631'1903, mlcod=631'1900)", 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.557794038000001, 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31352247700000002, 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.23e-07 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048237+0000", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3766999999999999e-05 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360616+0000", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31237881899999997 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360620+0000", 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.33e-06 2024-04-15T18:37:25.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360666+0000", 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6100000000000002e-05 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360686+0000", 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0154000000000001e-05 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.361637+0000", 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095048699999999997 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.361665+0000", 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8096999999999999e-05 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26545.0:45889 7.b e636/621 7:db21d6d0:::benchmark_data_smithi159_149998_object45888:head v 636'2779, mlcod=636'2775)", 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 2.434672038, 2024-04-15T18:37:25.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.200374936, 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9299999999999998e-07 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171359+0000", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3683000000000004e-05 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369880+0000", 2024-04-15T18:37:25.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19852134900000001 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369887+0000", 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9380000000000003e-06 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369944+0000", 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7318999999999998e-05 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369969+0000", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4663000000000001e-05 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.371598+0000", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001628675 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.371639+0000", 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1715999999999998e-05 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.680 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:48075 7.0 7:0971b1ff:::benchmark_data_smithi159_149998_object48074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.50417882199999997, 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20932758800000001, 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 48075 2024-04-15T18:37:25.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101763+0000", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0880000000000002e-06 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101788+0000", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5174999999999999e-05 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101804+0000", 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5716e-05 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101829+0000", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5181000000000002e-05 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101888+0000", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9389999999999999e-05 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101985+0000", 2024-04-15T18:37:25.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6218999999999995e-05 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.102075+0000", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0828999999999994e-05 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103588+0000", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001512698 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103879+0000", 2024-04-15T18:37:25.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029104499999999998 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310986+0000", 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20710657299999999 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311043+0000", 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7800999999999998e-05 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311085+0000", 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1872999999999997e-05 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:48075 7.0 e637/621 ondisk, result = 0)", 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.502091709, 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.207248614, 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:37:25.687 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:37:25.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.84e-07 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103847+0000", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9350000000000001e-06 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310924+0000", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20707672799999999 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310931+0000", 2024-04-15T18:37:25.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8929999999999999e-06 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310973+0000", 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2160999999999998e-05 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310981+0000", 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0949999999999996e-06 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311093+0000", 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011241799999999999 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:48080 7.0 7:02c5fcc6:::benchmark_data_smithi159_149998_object48079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.134410+0000", 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.47152674500000002, 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.176836619, 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:37:25.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 48080 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134410+0000", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134410+0000", 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134411+0000", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.850000000000001e-07 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134431+0000", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9775999999999998e-05 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134441+0000", 2024-04-15T18:37:25.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0261999999999999e-05 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134446+0000", 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3589999999999999e-06 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134524+0000", 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8039000000000003e-05 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134608+0000", 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3529000000000006e-05 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.134696+0000", 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8315999999999999e-05 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.135738+0000", 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00104149 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.136332+0000", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00059400999999999996 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311136+0000", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17480414399999999 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311199+0000", 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3052999999999994e-05 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311246+0000", 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7655999999999999e-05 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:48080 7.0 e637/621 ondisk, result = 0)", 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.136689+0000", 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.46924721899999999, 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17456565099999999, 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:37:25.697 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.136689+0000", 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.136689+0000", 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.136690+0000", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8599999999999999e-07 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.136692+0000", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3700000000000002e-06 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310944+0000", 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17425152299999999 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310946+0000", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.678e-06 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311120+0000", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017409700000000001 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311130+0000", 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1740000000000006e-06 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:37:25.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311255+0000", 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012542299999999999 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:37:25.701 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:37:25.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 48545 48529 84.2447 46.5 0.0169656 0.0118635 2024-04-15T18:37:26.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:37:26.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:26.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:26.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:37:26.402 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:26.432 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:26.453 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:26.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:26.557 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:26.589 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:26.613 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:26.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:26.621 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:26.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:26.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 49791 49775 84.0724 77.875 0.00257695 0.011887 2024-04-15T18:37:26.768 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:26.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:26 smithi159 ceph-mon[45336]: pgmap v1655: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:26.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:26 smithi049 ceph-mon[40087]: pgmap v1655: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:27.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:26 smithi082 ceph-mon[44206]: pgmap v1655: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:37:27.129 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:27.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:27.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:27.280 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:27.310 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:27.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:27.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:27.447 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:27.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 15 51213 51198 84.2002 88.9375 0.0048754 0.0118693 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: pgmap v1656: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: osdmap e638: 12 total, 12 up, 11 in 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:28.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:28 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 52574 52558 84.2205 85 0.0594745 0.0118647 2024-04-15T18:37:28.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: pgmap v1656: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s 2024-04-15T18:37:28.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: osdmap e638: 12 total, 12 up, 11 in 2024-04-15T18:37:28.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:28.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:28.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:28 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: pgmap v1656: 17 pgs: 17 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: osdmap e638: 12 total, 12 up, 11 in 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:28.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:28 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:29.200 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:29.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:29.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:29.357 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:29.405 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:29.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:29.512 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:29.541 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:29.564 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:29.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:29.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:29.695 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:29.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:37:29.771234+0000 min lat: 0.00107291 max lat: 0.452874 avg lat: 0.0119081 2024-04-15T18:37:29.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:37:29.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 53728 53712 83.918 72.125 0.0190756 0.0119081 2024-04-15T18:37:29.895 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:29.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:30.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:30.031 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:30.052 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:30.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:30.062 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:37:30.062 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:37:30.062 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:37:30.062 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:37:30.108 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:30.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:30.200 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:30.485 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:37:30.593 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:30.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:30.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:30.745 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:30 smithi049 ceph-mon[40087]: pgmap v1658: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:30.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2090858242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:30.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:30 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:30.749 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:30.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 54904 54888 83.6637 73.5 0.00855409 0.011945 2024-04-15T18:37:30.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:30 smithi082 ceph-mon[44206]: pgmap v1658: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:30.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2090858242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:30.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:30 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:30.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:30 smithi159 ceph-mon[45336]: pgmap v1658: 17 pgs: 17 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:37:30.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2090858242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:30.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:30 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:37:31.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:37:31.351 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:31.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:37:31.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:31.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:37:31.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: osdmap e639: 12 total, 12 up, 11 in 2024-04-15T18:37:31.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:37:31.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:37:31.687 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:31 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:37:31.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 55800 55784 83.005 56 0.00419476 0.0120414 2024-04-15T18:37:31.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:37:31.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:31.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:37:31.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: osdmap e639: 12 total, 12 up, 11 in 2024-04-15T18:37:31.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:37:31.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:37:31.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:31 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:37:31.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: osdmap e639: 12 total, 12 up, 11 in 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:37:31.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:31 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:37:32.511 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:32.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:32.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:32.654 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:32 smithi049 ceph-mon[40087]: osdmap e640: 12 total, 12 up, 11 in 2024-04-15T18:37:32.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 57160 57144 83.0512 85 0.0221911 0.0120334 2024-04-15T18:37:32.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:32.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:32.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:32.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:32.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:32 smithi082 ceph-mon[44206]: osdmap e640: 12 total, 12 up, 11 in 2024-04-15T18:37:32.856 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:32.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:32.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:37:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:37:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:37:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:37:32.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:32 smithi159 ceph-mon[45336]: osdmap e640: 12 total, 12 up, 11 in 2024-04-15T18:37:32.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:32.990 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.141 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:33.246 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:33.276 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.300 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:33.404 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:33.429 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:37:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:37:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:37:33.429 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:37:33.429 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:37:33.433 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.453 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 640, 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:37:32.475114+0000", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:37:10.676318+0000", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:34:03.394588+0000", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:37:33.468 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 30, 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:37:33.469 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 7, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "jewel", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:37:33.470 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:37:33.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:37:33.472 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:37:33.473 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:37:33.474 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:37:33.475 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 7, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_5", 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:36:40.461125+0000", 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:37:33.477 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:37:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "626", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:37:33.479 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:37:33.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:37:33.481 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.8700000047683716, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8700000047683716, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1000000238418579, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.059999942779541, 2024-04-15T18:37:33.482 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.059999942779541, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.483 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 400, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.488 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.489 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.490 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:37:33.493 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:37:33.494 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:37:33.495 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.496 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:37:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 415, 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:37:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:37:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:37:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:37:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:37:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-04-15T18:37:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 541, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:37:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:37:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:37:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.542 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.543 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:37:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 94, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 621, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:37:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:37:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:37:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:37:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 4245248647 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/4245248647", 2024-04-15T18:37:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/4245248647", 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/4245248647", 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/4245248647", 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:44.371179+0000", 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:21.210382+0000", 2024-04-15T18:37:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:20.228372+0000", 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:37:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:37:04.602702+0000", 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:32.801065+0000", 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:37:20.293692+0000", 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:37:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:37:10.503920+0000", 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:37:22.969332+0000", 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:44.386727+0000", 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:21.025992+0000", 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:37:18.703017+0000", 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:37:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:36:22.547410+0000", 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:37:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:37:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:37:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:37:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:37:33.566 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:37:33.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:33.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1962128724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:37:33.604 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.631 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:33.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:37:33.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:37:33.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:33.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:37:33.760 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 58154 58138 82.5755 62.125 0.00605308 0.0121048 2024-04-15T18:37:33.770 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1962128724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:37:33.772 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1962128724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:37:33.794 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:37:33.794 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:33.832 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:37:33.870 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:37:33.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:33.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:37:33.931 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:33.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:37:33.956 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:33.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:33.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:37:34.029 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:37:34.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:37:34.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:37:34.091 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:34.106 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:37:34.108 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:37:34.109 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:37:34.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:37:34.359 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:37:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:34.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:37:34.379 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:37:34.447 INFO:tasks.thrashosds.thrasher:j is {'epoch': 640, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:37:32.475114+0000', 'last_up_change': '2024-04-15T18:37:10.676318+0000', 'last_in_change': '2024-04-15T18:34:03.394588+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-04-15T18:36:40.461125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '626', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8700000047683716, 'score_stable': 1.8700000047683716, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 400, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 621, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 415, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 621, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 541, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 94, 'up_thru': 621, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 4245248647}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 4245248647}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 4245248647}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 4245248647}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 4245248647}]}, 'public_addr': '172.21.15.159:6825/4245248647', 'cluster_addr': '172.21.15.159:6827/4245248647', 'heartbeat_back_addr': '172.21.15.159:6831/4245248647', 'heartbeat_front_addr': '172.21.15.159:6829/4245248647', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:44.371179+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:21.210382+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:20.228372+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:37:04.602702+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:32.801065+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:37:20.293692+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-15T18:37:10.503920+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:37:22.969332+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:44.386727+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:21.025992+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:37:18.703017+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:36:22.547410+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:37:34.447 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-15T18:37:34.463 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: pgmap v1662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1010387278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.516 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:34.651 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:34.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:34.726 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:37:34.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:34.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 59184 59168 82.1709 64.375 0.00309155 0.0121634 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: pgmap v1662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1010387278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.804 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:34.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: pgmap v1662: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-04-15T18:37:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1010387278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:34.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.173 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:35.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:35.279 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:35.313 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:35.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.564 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:35.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:35.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:35.731 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:35.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 60451 60435 82.1059 79.1875 0.012797 0.0121733 2024-04-15T18:37:35.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:35.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:35.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:35.941 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:36.146 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:36.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:36.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:36.289 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:36.313 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:36.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:36.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:36.448 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:36.473 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:36.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:36.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:36.612 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:36.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:36 smithi159 ceph-mon[45336]: pgmap v1663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:37:36.711 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:36.713 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:36 smithi082 ceph-mon[44206]: pgmap v1663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:37:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:36.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:36.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 61675 61659 81.9865 76.5 0.0111807 0.0121888 2024-04-15T18:37:36.774 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:37:36.774 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:37:36.815 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:37:36.846 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:36.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:36 smithi049 ceph-mon[40087]: pgmap v1663: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:37:37.047 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:37:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:37.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:37:37.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:37.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:37:37.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:37.368 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:37.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:37.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:37.523 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:37.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:37:37.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 62567 62551 81.4399 55.75 0.0124658 0.0122723 2024-04-15T18:37:38.080 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:38.083 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:38.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:38.129 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:38.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:38.241 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:38.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:38.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:38.491 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:38.512 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:38.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:38.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:38 smithi049 ceph-mon[40087]: pgmap v1664: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:37:38.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:38 smithi049 ceph-mon[40087]: osdmap e641: 12 total, 12 up, 11 in 2024-04-15T18:37:38.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:38.645 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:38.666 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:38.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:38.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 63546 63530 81.0265 61.1875 0.00297572 0.0123333 2024-04-15T18:37:38.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:38 smithi082 ceph-mon[44206]: pgmap v1664: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:37:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:38 smithi082 ceph-mon[44206]: osdmap e641: 12 total, 12 up, 11 in 2024-04-15T18:37:38.805 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:38 smithi159 ceph-mon[45336]: pgmap v1664: 17 pgs: 17 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-15T18:37:38.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:38 smithi159 ceph-mon[45336]: osdmap e641: 12 total, 12 up, 11 in 2024-04-15T18:37:39.448 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 11, 3, 2] out_osds: [6] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 11, 3, 2, 5, 6] 2024-04-15T18:37:39.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:37:39.449 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-15T18:37:39.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd in 6 2024-04-15T18:37:39.570 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:39.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:39.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:39.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:39.723 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:39.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:39.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 64406 64390 80.4808 53.75 0.00260754 0.0124112 2024-04-15T18:37:39.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:39 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:39.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:39 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:39.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:39.853 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:39.874 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:39.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:39.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:39 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:39.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:39 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:39.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:39 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:39.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:39 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:39.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:40.016 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:40.049 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:40.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:40.155 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:40.182 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:40.204 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:40.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:40.309 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:40.344 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:40.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:40.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:40.476 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:40.490 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: pgmap v1666: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-15T18:37:40.490 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: osdmap e642: 12 total, 12 up, 11 in 2024-04-15T18:37:40.490 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2862938982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:40.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:40.504 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:40.513 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:37:40.524 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:40.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:40.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:40.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: pgmap v1666: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-15T18:37:40.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: osdmap e642: 12 total, 12 up, 11 in 2024-04-15T18:37:40.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2862938982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:40.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:40.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 15 65469 65454 80.2066 66.5 0.0366446 0.0124615 2024-04-15T18:37:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:40.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: pgmap v1666: 17 pgs: 17 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-04-15T18:37:40.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: osdmap e642: 12 total, 12 up, 11 in 2024-04-15T18:37:40.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2862938982' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:37:40.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:40.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:40.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:40.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:40.985 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:41.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:41 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:41.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:41 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:41 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:41 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:41.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:41 smithi049 ceph-mon[40087]: osdmap e643: 12 total, 12 up, 11 in 2024-04-15T18:37:41.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 66737 66721 80.1869 79.1875 0.00516326 0.0124651 2024-04-15T18:37:41.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:41 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:41.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:41 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:41.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:41 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:41.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:41 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:41.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:41 smithi082 ceph-mon[44206]: osdmap e643: 12 total, 12 up, 11 in 2024-04-15T18:37:41.845 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:41.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:41 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:41 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:41 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:41 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:37:41.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:41 smithi159 ceph-mon[45336]: osdmap e643: 12 total, 12 up, 11 in 2024-04-15T18:37:41.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:41.990 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:42.018 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:42.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:42.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:42.172 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:42.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:42 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:42.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:42 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:42.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:42 smithi049 ceph-mon[40087]: pgmap v1669: 17 pgs: 17 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:37:42.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:42 smithi049 ceph-mon[40087]: osdmap e644: 12 total, 12 up, 11 in 2024-04-15T18:37:42.689 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:42 smithi049 ceph-mon[40087]: osdmap e645: 12 total, 12 up, 11 in 2024-04-15T18:37:42.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 67877 67861 80.0182 71.25 0.00973579 0.012492 2024-04-15T18:37:42.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:42 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:42.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:42 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:42.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:42 smithi082 ceph-mon[44206]: pgmap v1669: 17 pgs: 17 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:37:42.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:42 smithi082 ceph-mon[44206]: osdmap e644: 12 total, 12 up, 11 in 2024-04-15T18:37:42.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:42 smithi082 ceph-mon[44206]: osdmap e645: 12 total, 12 up, 11 in 2024-04-15T18:37:42.843 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:42.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:42.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:42 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:42.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:42 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:42.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:42 smithi159 ceph-mon[45336]: pgmap v1669: 17 pgs: 17 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-04-15T18:37:42.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:42 smithi159 ceph-mon[45336]: osdmap e644: 12 total, 12 up, 11 in 2024-04-15T18:37:42.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:42 smithi159 ceph-mon[45336]: osdmap e645: 12 total, 12 up, 11 in 2024-04-15T18:37:42.949 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:42.981 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:43.006 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:43.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:43.029 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:37:43.029 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:37:43.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:43.160 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:43.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 69004 68988 79.8406 70.4375 0.00967906 0.0125186 2024-04-15T18:37:44.530 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:44.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:44.637 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:44.665 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:44.677 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:44.686 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:44.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:44.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:44 smithi049 ceph-mon[40087]: pgmap v1672: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:37:44.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:37:44.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 16 69886 69870 79.3912 55.125 0.0209274 0.0125908 2024-04-15T18:37:44.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:44 smithi082 ceph-mon[44206]: pgmap v1672: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:37:44.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:44.800 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:37:44.826 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:44 smithi159 ceph-mon[45336]: pgmap v1672: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:37:44.830 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:44.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:37:44.859 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:44.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:44.883 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:37:44.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:37:44.963 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:44.967 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:37:44.994 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:37:45.023 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:45.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:37:45.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:37:45.118 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:37:45.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:37:45.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:37:45.159 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:37:45.178 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:37:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:37:45.197 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:37:45.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:37:45.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:37:45.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:37:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:37:45.282 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:45.309 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.331 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:45.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:45.472 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.498 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:45.605 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:45.640 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.662 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:45.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 70904 70888 79.1096 63.625 0.00579239 0.0126352 2024-04-15T18:37:45.768 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/751651506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/751651506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:45.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:45.815 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:45.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:45.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/751651506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:37:45.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:45.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:45.969 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:46.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:37:46.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 71995 71979 78.9179 68.1875 0.00860902 0.0126654 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:46 smithi159 ceph-mon[45336]: pgmap v1673: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:46.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:46 smithi049 ceph-mon[40087]: pgmap v1673: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:37:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:37:47.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:46 smithi082 ceph-mon[44206]: pgmap v1673: 17 pgs: 17 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-04-15T18:37:47.129 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:47.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:47.265 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:47.288 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:47.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:47.422 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:47.457 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:47.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:47.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:47.613 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:47.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:47.742 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:47.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 73019 73003 78.6606 64 0.0369619 0.0127071 2024-04-15T18:37:47.763 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:47.861 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:47.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:37:47.872 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:47.894 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:47.914 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:37:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:47.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:37:48.018 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:37:48.046 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.067 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:37:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:48.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:37:48.171 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:48.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.232 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:48.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:48.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:48.384 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.419 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:48.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:48.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:48.553 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.574 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:48.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:48.678 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:48.708 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.736 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:48.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:48.761 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 74020 74004 78.3876 62.5625 0.00599583 0.0127502 2024-04-15T18:37:48.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:48.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:48 smithi049 ceph-mon[40087]: pgmap v1674: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s 2024-04-15T18:37:48.889 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:48.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:48 smithi159 ceph-mon[45336]: pgmap v1674: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s 2024-04-15T18:37:49.024 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:48 smithi082 ceph-mon[44206]: pgmap v1674: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s 2024-04-15T18:37:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:49.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:49.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:37:49.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:49.162 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:49.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:49.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:49.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:49.323 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:49.349 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:49.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:49.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:49.489 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:49.511 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:49.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:49.616 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:49.670 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:49.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:49.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:37:49.773047+0000 min lat: 0.00107291 max lat: 0.452874 avg lat: 0.0127835 2024-04-15T18:37:49.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:37:49.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 15 75088 75073 78.1943 66.8125 0.0147894 0.0127835 2024-04-15T18:37:49.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:49.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:37:49.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:37:49.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:49 smithi049 ceph-mon[40087]: osdmap e646: 12 total, 12 up, 11 in 2024-04-15T18:37:49.826 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:49.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:37:49.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:37:49.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:49 smithi159 ceph-mon[45336]: osdmap e646: 12 total, 12 up, 11 in 2024-04-15T18:37:50.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:37:50.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:37:50.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:49 smithi082 ceph-mon[44206]: osdmap e646: 12 total, 12 up, 11 in 2024-04-15T18:37:50.698 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:50.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:50.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 76195 76179 78.0455 69.125 0.0205168 0.0128061 2024-04-15T18:37:50.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:50.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:50 smithi049 ceph-mon[40087]: pgmap v1676: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:37:50.841 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:50.873 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:50.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:50.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:50 smithi159 ceph-mon[45336]: pgmap v1676: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:37:50.984 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:51.009 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:50 smithi082 ceph-mon[44206]: pgmap v1676: 17 pgs: 17 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-04-15T18:37:51.032 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:51.033 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:51.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:51.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:51.163 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:51.185 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:51.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:51.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:51.324 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:51.364 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:51.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:51.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:51.558 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:51.597 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:51.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:51.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:51.738 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:51.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 77241 77225 77.8411 65.375 0.0222482 0.0128409 2024-04-15T18:37:51.763 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:51.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:51.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:51.926 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:52.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:51 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:37:51.995+0000 7f8015e7e700 -1 mon.a@0(leader).osd e646 definitely_dead 0 2024-04-15T18:37:52.485 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.6. 2024-04-15T18:37:52.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 77986 77970 77.3445 46.5625 0.220919 0.0128779 2024-04-15T18:37:52.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:52 smithi049 ceph-mon[40087]: pgmap v1677: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:37:52.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1222042041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:52.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:52.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-15T18:37:52.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:52 smithi049 ceph-mon[40087]: osdmap e647: 12 total, 12 up, 12 in 2024-04-15T18:37:52.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:52 smithi159 ceph-mon[45336]: pgmap v1677: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:37:52.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1222042041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:52.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:52.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-15T18:37:52.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:52 smithi159 ceph-mon[45336]: osdmap e647: 12 total, 12 up, 12 in 2024-04-15T18:37:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:52 smithi082 ceph-mon[44206]: pgmap v1677: 17 pgs: 17 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-04-15T18:37:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1222042041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-15T18:37:53.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-15T18:37:53.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:52 smithi082 ceph-mon[44206]: osdmap e647: 12 total, 12 up, 12 in 2024-04-15T18:37:53.086 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:37:53.086 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:37:53.086 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:37:53.086 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:37:53.087 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:37:53.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 77998 77982 76.1476 0.75 0.00326668 0.0128908 2024-04-15T18:37:54.139 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-15T18:37:54.459 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:54.497 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:54.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:54.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:37:54.507 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:37:54.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:37:54.631 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:54.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:54 smithi049 ceph-mon[40087]: pgmap v1679: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:37:54.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:54 smithi049 ceph-mon[40087]: osdmap e648: 12 total, 12 up, 12 in 2024-04-15T18:37:54.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:54.741 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:37:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:54.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:37:54.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 78068 78052 75.0434 4.375 0.0109082 0.0133188 2024-04-15T18:37:54.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:54 smithi082 ceph-mon[44206]: pgmap v1679: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:37:54.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:54 smithi082 ceph-mon[44206]: osdmap e648: 12 total, 12 up, 12 in 2024-04-15T18:37:54.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:54.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:54.890 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:54.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:54 smithi159 ceph-mon[45336]: pgmap v1679: 17 pgs: 17 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:37:54.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:54 smithi159 ceph-mon[45336]: osdmap e648: 12 total, 12 up, 12 in 2024-04-15T18:37:54.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:37:54.913 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:54.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:55.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:55.047 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.068 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:55.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:55.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:55.202 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.226 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:55.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:55.332 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:55.369 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.395 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:55.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:55.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:55.528 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.547 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:55.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:55.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:37:55.684 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.708 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:37:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:55.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:37:55.762 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 78870 78854 74.6658 50.125 0.0144246 0.0133876 2024-04-15T18:37:55.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:55 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:55.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:55.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:55 smithi082 ceph-mon[44206]: osdmap e649: 12 total, 12 up, 12 in 2024-04-15T18:37:55.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:37:55.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:55 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:55.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:55.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:55 smithi049 ceph-mon[40087]: osdmap e649: 12 total, 12 up, 12 in 2024-04-15T18:37:55.863 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:55.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:55 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:37:55.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3014880804' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:37:55.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:55 smithi159 ceph-mon[45336]: osdmap e649: 12 total, 12 up, 12 in 2024-04-15T18:37:56.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 79560 79544 74.195 43.125 0.00641204 0.0134709 2024-04-15T18:37:56.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:56 smithi082 ceph-mon[44206]: 7.a scrub starts 2024-04-15T18:37:56.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:56 smithi082 ceph-mon[44206]: pgmap v1682: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 4771/232662 objects misplaced (2.051%); 235 KiB/s, 3 objects/s recovering 2024-04-15T18:37:56.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:56 smithi082 ceph-mon[44206]: osdmap e650: 12 total, 12 up, 12 in 2024-04-15T18:37:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:37:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:56.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:37:56.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:56 smithi049 ceph-mon[40087]: 7.a scrub starts 2024-04-15T18:37:56.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:56 smithi049 ceph-mon[40087]: pgmap v1682: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 4771/232662 objects misplaced (2.051%); 235 KiB/s, 3 objects/s recovering 2024-04-15T18:37:56.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:56 smithi049 ceph-mon[40087]: osdmap e650: 12 total, 12 up, 12 in 2024-04-15T18:37:56.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:56 smithi159 ceph-mon[45336]: 7.a scrub starts 2024-04-15T18:37:56.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:56 smithi159 ceph-mon[45336]: pgmap v1682: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 4771/232662 objects misplaced (2.051%); 235 KiB/s, 3 objects/s recovering 2024-04-15T18:37:56.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:56 smithi159 ceph-mon[45336]: osdmap e650: 12 total, 12 up, 12 in 2024-04-15T18:37:56.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:37:56.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:56.977 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:37:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:56.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:37:57.081 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:57.112 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:57.138 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:57.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:57.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:57.314 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:57.424 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:37:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:57.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:37:57.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:57.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:57 smithi049 ceph-mon[40087]: osdmap e651: 12 total, 12 up, 12 in 2024-04-15T18:37:57.568 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:57.590 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:57.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:57.694 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:37:57.722 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:57.744 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:37:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:57.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:37:57.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 80232 80216 73.7215 42 0.00717654 0.013557 2024-04-15T18:37:57.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:57 smithi082 ceph-mon[44206]: osdmap e651: 12 total, 12 up, 12 in 2024-04-15T18:37:57.848 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:37:57.873 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:57 smithi159 ceph-mon[45336]: osdmap e651: 12 total, 12 up, 12 in 2024-04-15T18:37:57.875 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:57.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:37:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:57.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:37:57.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:37:58.005 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:37:58.035 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:58.056 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:37:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:58.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:37:58.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:37:58.190 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:58.219 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:37:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:58.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:37:58.323 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:37:58.351 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:58.374 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:37:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:58.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:37:58.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:37:58.552 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:58.598 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:37:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:37:58.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:37:58.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:58 smithi159 ceph-mon[45336]: 7.a scrub ok 2024-04-15T18:37:58.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:58 smithi159 ceph-mon[45336]: 7.3 scrub starts 2024-04-15T18:37:58.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:58 smithi159 ceph-mon[45336]: pgmap v1685: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 2/235335 objects degraded (0.001%); 14259/235335 objects misplaced (6.059%); 3.5 MiB/s, 54 objects/s recovering 2024-04-15T18:37:58.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:58 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 2/235335 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:37:58.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:37:58.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 15 80898 80883 73.2572 41.6875 0.0288568 0.0136445 2024-04-15T18:37:58.769 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:58 smithi049 ceph-mon[40087]: 7.a scrub ok 2024-04-15T18:37:58.769 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:58 smithi049 ceph-mon[40087]: 7.3 scrub starts 2024-04-15T18:37:58.769 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:58 smithi049 ceph-mon[40087]: pgmap v1685: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 2/235335 objects degraded (0.001%); 14259/235335 objects misplaced (6.059%); 3.5 MiB/s, 54 objects/s recovering 2024-04-15T18:37:58.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:58 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 2/235335 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:37:58.772 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:37:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:58 smithi082 ceph-mon[44206]: 7.a scrub ok 2024-04-15T18:37:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:58 smithi082 ceph-mon[44206]: 7.3 scrub starts 2024-04-15T18:37:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:58 smithi082 ceph-mon[44206]: pgmap v1685: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 2/235335 objects degraded (0.001%); 14259/235335 objects misplaced (6.059%); 3.5 MiB/s, 54 objects/s recovering 2024-04-15T18:37:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:58 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 2/235335 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:37:59.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 11, 3, 2, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 11, 3, 2, 5, 6] 2024-04-15T18:37:59.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:37:59.140 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-04-15T18:37:59.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:37:59.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:37:59.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 81674 81658 72.9026 48.4375 0.0269456 0.0137097 2024-04-15T18:37:59.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:59 smithi159 ceph-mon[45336]: 7.0 scrub starts 2024-04-15T18:37:59.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:37:59 smithi159 ceph-mon[45336]: 7.3 scrub ok 2024-04-15T18:37:59.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:59 smithi049 ceph-mon[40087]: 7.0 scrub starts 2024-04-15T18:37:59.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:37:59 smithi049 ceph-mon[40087]: 7.3 scrub ok 2024-04-15T18:38:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:59 smithi082 ceph-mon[44206]: 7.0 scrub starts 2024-04-15T18:38:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:37:59 smithi082 ceph-mon[44206]: 7.3 scrub ok 2024-04-15T18:38:00.451 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:38:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:00.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:38:00.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:00.640 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:00.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:00 smithi049 ceph-mon[40087]: 7.0 scrub ok 2024-04-15T18:38:00.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:00 smithi049 ceph-mon[40087]: pgmap v1686: 17 pgs: 2 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 469 op/s; 2/238947 objects degraded (0.001%); 14259/238947 objects misplaced (5.967%); 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:38:00.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 82401 82385 72.5157 45.4375 0.0392482 0.0137842 2024-04-15T18:38:00.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:00 smithi159 ceph-mon[45336]: 7.0 scrub ok 2024-04-15T18:38:00.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:00 smithi159 ceph-mon[45336]: pgmap v1686: 17 pgs: 2 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 469 op/s; 2/238947 objects degraded (0.001%); 14259/238947 objects misplaced (5.967%); 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:38:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:00 smithi082 ceph-mon[44206]: 7.0 scrub ok 2024-04-15T18:38:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:00 smithi082 ceph-mon[44206]: pgmap v1686: 17 pgs: 2 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 469 op/s; 2/238947 objects degraded (0.001%); 14259/238947 objects misplaced (5.967%); 2.3 MiB/s, 36 objects/s recovering 2024-04-15T18:38:01.419 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:01.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:01.524 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:01.553 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:01.575 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:01.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:01.679 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:38:01.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:01 smithi049 ceph-mon[40087]: 7.4 scrub starts 2024-04-15T18:38:01.724 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:01.748 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:38:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:01.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:38:01.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 83140 83124 72.15 46.1875 0.0211511 0.0138537 2024-04-15T18:38:01.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:01.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:01 smithi159 ceph-mon[45336]: 7.4 scrub starts 2024-04-15T18:38:02.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:01 smithi082 ceph-mon[44206]: 7.4 scrub starts 2024-04-15T18:38:02.267 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:02.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:02.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:02.402 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:02.421 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:02.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:02.525 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:02.551 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:02.573 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:02.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:02.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:38:02.705 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:02 smithi082 ceph-mon[44206]: 7.1 scrub starts 2024-04-15T18:38:02.705 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:02 smithi082 ceph-mon[44206]: pgmap v1687: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 2/242286 objects degraded (0.001%); 14083/242286 objects misplaced (5.813%); 3.6 MiB/s, 56 objects/s recovering 2024-04-15T18:38:02.705 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:02 smithi082 ceph-mon[44206]: 7.4 scrub ok 2024-04-15T18:38:02.706 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:02 smithi082 ceph-mon[44206]: 7.8 scrub starts 2024-04-15T18:38:02.709 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:02.730 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:38:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:02.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:38:02.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 83681 83665 71.6248 33.8125 0.0450432 0.0139551 2024-04-15T18:38:02.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:02.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:02 smithi049 ceph-mon[40087]: 7.1 scrub starts 2024-04-15T18:38:02.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:02 smithi049 ceph-mon[40087]: pgmap v1687: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 2/242286 objects degraded (0.001%); 14083/242286 objects misplaced (5.813%); 3.6 MiB/s, 56 objects/s recovering 2024-04-15T18:38:02.885 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:02 smithi049 ceph-mon[40087]: 7.4 scrub ok 2024-04-15T18:38:02.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:02 smithi049 ceph-mon[40087]: 7.8 scrub starts 2024-04-15T18:38:02.892 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:02.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:02 smithi159 ceph-mon[45336]: 7.1 scrub starts 2024-04-15T18:38:02.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:02 smithi159 ceph-mon[45336]: pgmap v1687: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 561 op/s; 2/242286 objects degraded (0.001%); 14083/242286 objects misplaced (5.813%); 3.6 MiB/s, 56 objects/s recovering 2024-04-15T18:38:02.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:02 smithi159 ceph-mon[45336]: 7.4 scrub ok 2024-04-15T18:38:02.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:02 smithi159 ceph-mon[45336]: 7.8 scrub starts 2024-04-15T18:38:02.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:38:02.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:38:03.034 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:38:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:03.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:38:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:03.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:03.146 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:38:03.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:38:03.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:38:03.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:38:03.237 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:03.260 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:38:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:03.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:38:03.273 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:38:03.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:38:03.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:38:03.389 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:03.409 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:38:03.410 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:38:03.410 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:38:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:03.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:38:03.457 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:38:03.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:38:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:38:03.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:38:03.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:38:03.567 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:03.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 84287 84271 71.1687 37.875 0.00730537 0.0140441 2024-04-15T18:38:03.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/436317686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:03.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/436317686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:03.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:03.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/436317686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:04.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 85118 85102 70.9122 51.9375 0.0871486 0.0140957 2024-04-15T18:38:04.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: 7.1 scrub ok 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: pgmap v1688: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 2/249267 objects degraded (0.001%); 13707/249267 objects misplaced (5.499%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:38:04.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/249267 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:04.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:04 smithi159 ceph-mon[45336]: 7.8 scrub ok 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: 7.1 scrub ok 2024-04-15T18:38:04.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:04.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: pgmap v1688: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 2/249267 objects degraded (0.001%); 13707/249267 objects misplaced (5.499%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:38:04.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/249267 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:04.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:04 smithi049 ceph-mon[40087]: 7.8 scrub ok 2024-04-15T18:38:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: 7.1 scrub ok 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: pgmap v1688: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 2/249267 objects degraded (0.001%); 13707/249267 objects misplaced (5.499%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/249267 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:05.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:04 smithi082 ceph-mon[44206]: 7.8 scrub ok 2024-04-15T18:38:05.145 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:38:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:05.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:38:05.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:05.327 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:05.368 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:05.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:05.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:38:05.529 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:05.610 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:38:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:05.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:38:05.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:05.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 85789 85773 70.5309 41.9375 0.0473377 0.0141721 2024-04-15T18:38:05.785 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:05.824 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:05.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:05.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:05.981 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:06.005 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:06.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:06.109 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:06.136 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:06.165 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:06.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:06.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:06.597 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:38:06.598 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:38:06.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 86531 86515 70.2172 46.375 0.00413455 0.0142349 2024-04-15T18:38:06.787 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:06.791 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:06.796 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:06.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:06.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:06 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:38:06.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:06 smithi159 ceph-mon[45336]: pgmap v1689: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 624 op/s; 2/251358 objects degraded (0.001%); 13528/251358 objects misplaced (5.382%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:06.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:06 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:38:06.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:06.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:06 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:38:06.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:06 smithi049 ceph-mon[40087]: pgmap v1689: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 624 op/s; 2/251358 objects degraded (0.001%); 13528/251358 objects misplaced (5.382%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:06.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:06 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:38:06.976 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:07.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:06 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:38:07.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:06 smithi082 ceph-mon[44206]: pgmap v1689: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 624 op/s; 2/251358 objects degraded (0.001%); 13528/251358 objects misplaced (5.382%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:07.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:06 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:38:07.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 87278 87262 69.9155 46.6875 0.0293196 0.0142972 2024-04-15T18:38:07.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:07.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:07.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:08.004 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:08.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:08.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:08.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:08.178 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:08.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:38:08.763 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 88022 88006 69.619 46.5 0.0260803 0.0143579 2024-04-15T18:38:08.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:08 smithi159 ceph-mon[45336]: 7.6 scrub starts 2024-04-15T18:38:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:08 smithi159 ceph-mon[45336]: pgmap v1690: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 577 op/s; 2/252609 objects degraded (0.001%); 13528/252609 objects misplaced (5.355%); 5.3 MiB/s, 84 objects/s recovering 2024-04-15T18:38:08.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:08 smithi049 ceph-mon[40087]: 7.6 scrub starts 2024-04-15T18:38:08.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:08 smithi049 ceph-mon[40087]: pgmap v1690: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 577 op/s; 2/252609 objects degraded (0.001%); 13528/252609 objects misplaced (5.355%); 5.3 MiB/s, 84 objects/s recovering 2024-04-15T18:38:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:08 smithi082 ceph-mon[44206]: 7.6 scrub starts 2024-04-15T18:38:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:08 smithi082 ceph-mon[44206]: pgmap v1690: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 577 op/s; 2/252609 objects degraded (0.001%); 13528/252609 objects misplaced (5.355%); 5.3 MiB/s, 84 objects/s recovering 2024-04-15T18:38:09.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:38:09.775707+0000 min lat: 0.00107291 max lat: 2.21458 avg lat: 0.0144213 2024-04-15T18:38:09.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:38:09.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 88751 88735 69.3175 45.5625 0.00730627 0.0144213 2024-04-15T18:38:09.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:09 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/259995 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:10.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:09 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/259995 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:10.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:09 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/259995 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:10.201 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:10.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:10.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:38:10.308 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:38:10.334 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:10.356 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:38:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:10.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:38:10.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:10.496 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:10.520 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:10.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:10.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:10.764 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 89495 89479 69.0357 46.5 0.0540608 0.0144793 2024-04-15T18:38:10.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:10 smithi159 ceph-mon[45336]: pgmap v1691: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s; 2/259995 objects degraded (0.001%); 13148/259995 objects misplaced (5.057%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:10.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:10 smithi159 ceph-mon[45336]: 7.6 scrub ok 2024-04-15T18:38:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:10 smithi049 ceph-mon[40087]: pgmap v1691: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s; 2/259995 objects degraded (0.001%); 13148/259995 objects misplaced (5.057%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:10 smithi049 ceph-mon[40087]: 7.6 scrub ok 2024-04-15T18:38:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:10 smithi082 ceph-mon[44206]: pgmap v1691: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s; 2/259995 objects degraded (0.001%); 13148/259995 objects misplaced (5.057%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:38:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:10 smithi082 ceph-mon[44206]: 7.6 scrub ok 2024-04-15T18:38:11.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 90218 90202 68.7448 45.1875 0.00407932 0.0145398 2024-04-15T18:38:12.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:12.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 90940 90924 68.4602 45.125 0.0103724 0.0146013 2024-04-15T18:38:12.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:12.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:12.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:12.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: pgmap v1692: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 2/263442 objects degraded (0.001%); 12942/263442 objects misplaced (4.913%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:38:12.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: osdmap e652: 12 total, 12 up, 12 in 2024-04-15T18:38:12.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:12 smithi049 ceph-mon[40087]: osdmap e653: 12 total, 12 up, 12 in 2024-04-15T18:38:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: pgmap v1692: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 2/263442 objects degraded (0.001%); 12942/263442 objects misplaced (4.913%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: osdmap e652: 12 total, 12 up, 12 in 2024-04-15T18:38:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:12 smithi082 ceph-mon[44206]: osdmap e653: 12 total, 12 up, 12 in 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: pgmap v1692: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 680 op/s; 2/263442 objects degraded (0.001%); 12942/263442 objects misplaced (4.913%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: osdmap e652: 12 total, 12 up, 12 in 2024-04-15T18:38:13.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:12 smithi159 ceph-mon[45336]: osdmap e653: 12 total, 12 up, 12 in 2024-04-15T18:38:13.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 91661 91645 68.1816 45.0625 0.0221075 0.0146591 2024-04-15T18:38:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:14.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:14.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 92396 92380 67.9199 45.9375 0.0184784 0.0147178 2024-04-15T18:38:15.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:14 smithi082 ceph-mon[44206]: pgmap v1695: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 816 op/s; 2/270960 objects degraded (0.001%); 12570/270960 objects misplaced (4.639%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:38:15.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:14 smithi082 ceph-mon[44206]: osdmap e654: 12 total, 12 up, 12 in 2024-04-15T18:38:15.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:14 smithi082 ceph-mon[44206]: 7.e scrub starts 2024-04-15T18:38:15.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:14 smithi159 ceph-mon[45336]: pgmap v1695: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 816 op/s; 2/270960 objects degraded (0.001%); 12570/270960 objects misplaced (4.639%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:38:15.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:14 smithi159 ceph-mon[45336]: osdmap e654: 12 total, 12 up, 12 in 2024-04-15T18:38:15.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:14 smithi159 ceph-mon[45336]: 7.e scrub starts 2024-04-15T18:38:15.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:14 smithi049 ceph-mon[40087]: pgmap v1695: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 816 op/s; 2/270960 objects degraded (0.001%); 12570/270960 objects misplaced (4.639%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:38:15.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:14 smithi049 ceph-mon[40087]: osdmap e654: 12 total, 12 up, 12 in 2024-04-15T18:38:15.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:14 smithi049 ceph-mon[40087]: 7.e scrub starts 2024-04-15T18:38:15.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 93058 93042 67.6112 41.375 0.0731483 0.0147777 2024-04-15T18:38:16.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:15 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/273720 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:16.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:15 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/273720 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:15 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/273720 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:16.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 93820 93804 67.3814 47.625 0.0122181 0.0148353 2024-04-15T18:38:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:16 smithi159 ceph-mon[45336]: 7.2 scrub starts 2024-04-15T18:38:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:16 smithi159 ceph-mon[45336]: pgmap v1697: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 762 op/s; 2/273720 objects degraded (0.001%); 12336/273720 objects misplaced (4.507%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:16 smithi159 ceph-mon[45336]: 7.e scrub ok 2024-04-15T18:38:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:16 smithi049 ceph-mon[40087]: 7.2 scrub starts 2024-04-15T18:38:17.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:16 smithi049 ceph-mon[40087]: pgmap v1697: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 762 op/s; 2/273720 objects degraded (0.001%); 12336/273720 objects misplaced (4.507%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:17.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:16 smithi049 ceph-mon[40087]: 7.e scrub ok 2024-04-15T18:38:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:16 smithi082 ceph-mon[44206]: 7.2 scrub starts 2024-04-15T18:38:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:16 smithi082 ceph-mon[44206]: pgmap v1697: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 762 op/s; 2/273720 objects degraded (0.001%); 12336/273720 objects misplaced (4.507%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:16 smithi082 ceph-mon[44206]: 7.e scrub ok 2024-04-15T18:38:17.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 94588 94572 67.1612 48 0.0145353 0.0148836 2024-04-15T18:38:18.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:17 smithi159 ceph-mon[45336]: 7.2 scrub ok 2024-04-15T18:38:18.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:17 smithi159 ceph-mon[45336]: osdmap e655: 12 total, 12 up, 12 in 2024-04-15T18:38:18.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:17 smithi049 ceph-mon[40087]: 7.2 scrub ok 2024-04-15T18:38:18.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:17 smithi049 ceph-mon[40087]: osdmap e655: 12 total, 12 up, 12 in 2024-04-15T18:38:18.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:17 smithi082 ceph-mon[44206]: 7.2 scrub ok 2024-04-15T18:38:18.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:17 smithi082 ceph-mon[44206]: osdmap e655: 12 total, 12 up, 12 in 2024-04-15T18:38:18.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 95289 95273 66.8988 43.8125 0.0125608 0.0149393 2024-04-15T18:38:18.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:18 smithi049 ceph-mon[40087]: pgmap v1698: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 2/274341 objects degraded (0.001%); 12336/274341 objects misplaced (4.497%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:38:19.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:19.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:19.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:18 smithi159 ceph-mon[45336]: pgmap v1698: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 2/274341 objects degraded (0.001%); 12336/274341 objects misplaced (4.497%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:38:19.201 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:18 smithi082 ceph-mon[44206]: pgmap v1698: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 605 op/s; 2/274341 objects degraded (0.001%); 12336/274341 objects misplaced (4.497%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:38:19.765 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 12 95997 95985 66.6498 44.5 0.0201061 0.0149958 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.7557 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 95997 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 66.1095 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 23.4819 2024-04-15T18:38:20.766 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 137.812 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0.75 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 1057 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 375.71 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 2205 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 12 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0150385 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.0332457 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 2.21458 2024-04-15T18:38:20.767 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00107291 2024-04-15T18:38:20.785 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:38:20.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:20 smithi049 ceph-mon[40087]: pgmap v1700: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s; 2/281757 objects degraded (0.001%); 11941/281757 objects misplaced (4.238%); 8.9 MiB/s, 143 objects/s recovering 2024-04-15T18:38:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:20 smithi159 ceph-mon[45336]: pgmap v1700: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s; 2/281757 objects degraded (0.001%); 11941/281757 objects misplaced (4.238%); 8.9 MiB/s, 143 objects/s recovering 2024-04-15T18:38:21.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:20 smithi082 ceph-mon[44206]: pgmap v1700: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 872 op/s; 2/281757 objects degraded (0.001%); 11941/281757 objects misplaced (4.238%); 8.9 MiB/s, 143 objects/s recovering 2024-04-15T18:38:21.912 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:21.913 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:22.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:21 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/284823 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:22.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:21 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/284823 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:22.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:21 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/284823 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:22.697 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:22.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:22.802 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:22.829 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:22.851 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:22.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:22.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:22.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:22 smithi049 ceph-mon[40087]: pgmap v1701: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2/284823 objects degraded (0.001%); 11775/284823 objects misplaced (4.134%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:38:22.987 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:23.014 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:23.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:23.117 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:22 smithi159 ceph-mon[45336]: pgmap v1701: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2/284823 objects degraded (0.001%); 11775/284823 objects misplaced (4.134%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:38:23.147 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:23.171 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:23.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:23.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:23.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:22 smithi082 ceph-mon[44206]: pgmap v1701: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2/284823 objects degraded (0.001%); 11775/284823 objects misplaced (4.134%); 6.2 MiB/s, 99 objects/s recovering 2024-04-15T18:38:23.388 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:23.520 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:38:23.524 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:38:23.920 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:38:24.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2445673011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:23 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:24.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:24.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2445673011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:23 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:24.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:24.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2445673011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:23 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:38:24.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:24.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:24 smithi049 ceph-mon[40087]: pgmap v1702: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 689 op/s; 2/284853 objects degraded (0.001%); 11333/284853 objects misplaced (3.979%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:38:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:24 smithi049 ceph-mon[40087]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:38:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:24 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:38:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:24 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:38:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:24 smithi049 ceph-mon[40087]: osdmap e656: 12 total, 12 up, 12 in 2024-04-15T18:38:25.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:24 smithi082 ceph-mon[44206]: pgmap v1702: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 689 op/s; 2/284853 objects degraded (0.001%); 11333/284853 objects misplaced (3.979%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:38:25.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:24 smithi082 ceph-mon[44206]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:38:25.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:24 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:38:25.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:24 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:38:25.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:24 smithi082 ceph-mon[44206]: osdmap e656: 12 total, 12 up, 12 in 2024-04-15T18:38:25.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:24 smithi159 ceph-mon[45336]: pgmap v1702: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 689 op/s; 2/284853 objects degraded (0.001%); 11333/284853 objects misplaced (3.979%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:38:25.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:24 smithi159 ceph-mon[45336]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:38:25.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:24 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:38:25.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:24 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:38:25.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:24 smithi159 ceph-mon[45336]: osdmap e656: 12 total, 12 up, 12 in 2024-04-15T18:38:26.483 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:26.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:26.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:38:26.617 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:26.638 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:38:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:26.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:38:26.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:26.802 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:26 smithi049 ceph-mon[40087]: pgmap v1704: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 36 MiB/s wr, 800 op/s; 2/282432 objects degraded (0.001%); 11112/282432 objects misplaced (3.934%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:38:27.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:26 smithi082 ceph-mon[44206]: pgmap v1704: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 36 MiB/s wr, 800 op/s; 2/282432 objects degraded (0.001%); 11112/282432 objects misplaced (3.934%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:38:27.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:26 smithi159 ceph-mon[45336]: pgmap v1704: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 36 MiB/s wr, 800 op/s; 2/282432 objects degraded (0.001%); 11112/282432 objects misplaced (3.934%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:38:28.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/282432 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:28.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:27 smithi049 ceph-mon[40087]: osdmap e657: 12 total, 12 up, 12 in 2024-04-15T18:38:28.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/282432 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:28.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:27 smithi082 ceph-mon[44206]: osdmap e657: 12 total, 12 up, 12 in 2024-04-15T18:38:28.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/282432 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:28.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:27 smithi159 ceph-mon[45336]: osdmap e657: 12 total, 12 up, 12 in 2024-04-15T18:38:28.825 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:28.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:28.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:38:28.994 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:29.087 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:38:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:29.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:38:29.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:28 smithi049 ceph-mon[40087]: pgmap v1706: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 16 MiB/s wr, 522 op/s; 2/281697 objects degraded (0.001%); 11112/281697 objects misplaced (3.945%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:38:29.198 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:29.226 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:28 smithi159 ceph-mon[45336]: pgmap v1706: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 16 MiB/s wr, 522 op/s; 2/281697 objects degraded (0.001%); 11112/281697 objects misplaced (3.945%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:38:29.228 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:29.256 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:29.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:29.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:28 smithi082 ceph-mon[44206]: pgmap v1706: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 16 MiB/s wr, 522 op/s; 2/281697 objects degraded (0.001%); 11112/281697 objects misplaced (3.945%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:38:29.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:29.400 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:30.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:38:30.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:38:30.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:38:30.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:38:30.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:38:30.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:38:30.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:38:30.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:38:30.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:38:30.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:38:30.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:38:30.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:38:31.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:38:32.476 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:32.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:32.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:32.635 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:32.658 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:32.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:32 smithi049 ceph-mon[40087]: pgmap v1708: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 569 op/s; 2/271197 objects degraded (0.001%); 10365/271197 objects misplaced (3.822%); 7.6 MiB/s, 120 objects/s recovering 2024-04-15T18:38:32.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/271197 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:32.763 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:38:32.798 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:32.823 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:38:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:32.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:38:32.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:32 smithi159 ceph-mon[45336]: pgmap v1708: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 569 op/s; 2/271197 objects degraded (0.001%); 10365/271197 objects misplaced (3.822%); 7.6 MiB/s, 120 objects/s recovering 2024-04-15T18:38:32.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/271197 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:32.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:38:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:32 smithi082 ceph-mon[44206]: pgmap v1708: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 127 B/s wr, 569 op/s; 2/271197 objects degraded (0.001%); 10365/271197 objects misplaced (3.822%); 7.6 MiB/s, 120 objects/s recovering 2024-04-15T18:38:33.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/271197 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:33.202 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:38:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:33.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:38:33.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:33.348 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:33.371 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:33.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:33.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:33.536 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:33.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:33.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:33.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:33.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:33.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:33 smithi082 ceph-mon[44206]: osdmap e658: 12 total, 12 up, 12 in 2024-04-15T18:38:33.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:33.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:33.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:33 smithi159 ceph-mon[45336]: osdmap e658: 12 total, 12 up, 12 in 2024-04-15T18:38:33.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:33.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:33.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:33 smithi049 ceph-mon[40087]: osdmap e658: 12 total, 12 up, 12 in 2024-04-15T18:38:34.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:34 smithi082 ceph-mon[44206]: pgmap v1710: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 2/263640 objects degraded (0.001%); 9904/263640 objects misplaced (3.757%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:38:34.810 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1702,"stamp":"2024-04-15T18:38:23.497831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6223168032,"num_objects":94951,"num_object_clones":0,"num_object_copies":284853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":11333,"num_objects_unfound":0,"num_objects_dirty":94951,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":97226,"num_write_kb":6146746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3155,"num_bytes_recovered":209551072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97184,"ondisk_log_size":97184,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24017192,"kb_used_data":18865672,"kb_used_omap":287,"kb_used_meta":5151072,"kb_avail":1100924632,"statfs":{"total":1151940427776,"available":1127346823168,"internally_reserved":0,"allocated":19318448128,"data_stored":19313525348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":294247,"internal_metadata":5274698393},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":303497216,"num_objects":4631,"num_object_clones":0,"num_object_copies":13893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1237,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6727,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":81068032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.758623"},"pg_stats":[{"pgid":"7.f","version":"655'6110","reported_seq":6187,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:19.724487+0000","last_change":"2024-04-15T18:36:41.540300+0000","last_active":"2024-04-15T18:38:19.724487+0000","last_peered":"2024-04-15T18:38:19.724487+0000","last_clean":"2024-04-15T18:38:19.724487+0000","last_became_active":"2024-04-15T18:36:41.539961+0000","last_became_peered":"2024-04-15T18:36:41.539961+0000","last_unstale":"2024-04-15T18:38:19.724487+0000","last_undegraded":"2024-04-15T18:38:19.724487+0000","last_fullsized":"2024-04-15T18:38:19.724487+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":6110,"log_dups_size":0,"ondisk_log_size":6110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400424960,"num_objects":6110,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"655'6039","reported_seq":6181,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.484276+0000","last_change":"2024-04-15T18:38:15.813470+0000","last_active":"2024-04-15T18:38:22.484276+0000","last_peered":"2024-04-15T18:38:22.484276+0000","last_clean":"2024-04-15T18:38:22.484276+0000","last_became_active":"2024-04-15T18:36:41.504308+0000","last_became_peered":"2024-04-15T18:36:41.504308+0000","last_unstale":"2024-04-15T18:38:22.484276+0000","last_undegraded":"2024-04-15T18:38:22.484276+0000","last_fullsized":"2024-04-15T18:38:22.484276+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'5787","last_scrub_stamp":"2024-04-15T18:38:15.813069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:15.813069+0000","objects_scrubbed":5740,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:33.385038+0000","scrub_duration":1687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"655'5985","reported_seq":8032,"reported_epoch":655,"state":"active+recovering+undersized+remapped","last_fresh":"2024-04-15T18:38:20.432525+0000","last_change":"2024-04-15T18:37:54.720909+0000","last_active":"2024-04-15T18:38:20.432525+0000","last_peered":"2024-04-15T18:38:20.432525+0000","last_clean":"2024-04-15T18:37:52.468421+0000","last_became_active":"2024-04-15T18:37:54.672216+0000","last_became_peered":"2024-04-15T18:37:54.672216+0000","last_unstale":"2024-04-15T18:38:20.432525+0000","last_undegraded":"2024-04-15T18:38:20.432525+0000","last_fullsized":"2024-04-15T18:37:54.489679+0000","mapping_epoch":648,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":5985,"log_dups_size":0,"ondisk_log_size":5985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392232960,"num_objects":5985,"num_object_clones":0,"num_object_copies":17955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3810,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5985,"num_write_kb":383040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":983,"num_bytes_recovered":64421888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":2175},{"shards":"0,1,3","objects":3810}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"655'6078","reported_seq":8156,"reported_epoch":655,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:38:22.503486+0000","last_change":"2024-04-15T18:37:54.710439+0000","last_active":"2024-04-15T18:38:22.503486+0000","last_peered":"2024-04-15T18:38:22.503486+0000","last_clean":"2024-04-15T18:37:52.465486+0000","last_became_active":"2024-04-15T18:37:54.660953+0000","last_became_peered":"2024-04-15T18:37:54.660953+0000","last_unstale":"2024-04-15T18:38:22.503486+0000","last_undegraded":"2024-04-15T18:37:54.530980+0000","last_fullsized":"2024-04-15T18:37:54.499953+0000","mapping_epoch":648,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":6078,"log_dups_size":0,"ondisk_log_size":6078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384040960,"num_objects":5860,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3825,"num_objects_unfound":0,"num_objects_dirty":5860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6078,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":65536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[3,11],"avail_no_missing":["3","2","11"],"object_location_counts":[{"shards":"2,3,11","objects":3825},{"shards":"3,11","objects":2034}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"655'5984","reported_seq":6062,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.507161+0000","last_change":"2024-04-15T18:36:41.505306+0000","last_active":"2024-04-15T18:38:22.507161+0000","last_peered":"2024-04-15T18:38:22.507161+0000","last_clean":"2024-04-15T18:38:22.507161+0000","last_became_active":"2024-04-15T18:36:41.504965+0000","last_became_peered":"2024-04-15T18:36:41.504965+0000","last_unstale":"2024-04-15T18:38:22.507161+0000","last_undegraded":"2024-04-15T18:38:22.507161+0000","last_fullsized":"2024-04-15T18:38:22.507161+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":5984,"log_dups_size":0,"ondisk_log_size":5984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381026304,"num_objects":5814,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5984,"num_write_kb":377536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"655'6040","reported_seq":6175,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.483228+0000","last_change":"2024-04-15T18:37:56.593403+0000","last_active":"2024-04-15T18:38:22.483228+0000","last_peered":"2024-04-15T18:38:22.483228+0000","last_clean":"2024-04-15T18:38:22.483228+0000","last_became_active":"2024-04-15T18:36:41.506579+0000","last_became_peered":"2024-04-15T18:36:41.506579+0000","last_unstale":"2024-04-15T18:38:22.483228+0000","last_undegraded":"2024-04-15T18:38:22.483228+0000","last_fullsized":"2024-04-15T18:38:22.483228+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'4903","last_scrub_stamp":"2024-04-15T18:37:56.593107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:37:56.593107+0000","objects_scrubbed":4877,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:26.186165+0000","scrub_duration":1334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385875968,"num_objects":5888,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":381696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.9","version":"655'6059","reported_seq":6137,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:19.754557+0000","last_change":"2024-04-15T18:36:41.540120+0000","last_active":"2024-04-15T18:38:19.754557+0000","last_peered":"2024-04-15T18:38:19.754557+0000","last_clean":"2024-04-15T18:38:19.754557+0000","last_became_active":"2024-04-15T18:36:41.539744+0000","last_became_peered":"2024-04-15T18:36:41.539744+0000","last_unstale":"2024-04-15T18:38:19.754557+0000","last_undegraded":"2024-04-15T18:38:19.754557+0000","last_fullsized":"2024-04-15T18:38:19.754557+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":6059,"log_dups_size":0,"ondisk_log_size":6059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397082624,"num_objects":6059,"num_object_clones":0,"num_object_copies":18177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6059,"num_write_kb":387776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"655'6068","reported_seq":6205,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.490685+0000","last_change":"2024-04-15T18:38:03.684464+0000","last_active":"2024-04-15T18:38:22.490685+0000","last_peered":"2024-04-15T18:38:22.490685+0000","last_clean":"2024-04-15T18:38:22.490685+0000","last_became_active":"2024-04-15T18:36:41.503582+0000","last_became_peered":"2024-04-15T18:36:41.503582+0000","last_unstale":"2024-04-15T18:38:22.490685+0000","last_undegraded":"2024-04-15T18:38:22.490685+0000","last_fullsized":"2024-04-15T18:38:22.490685+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5265","last_scrub_stamp":"2024-04-15T18:38:03.684179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:03.684179+0000","objects_scrubbed":5242,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:25.433445+0000","scrub_duration":1546,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386007040,"num_objects":5890,"num_object_clones":0,"num_object_copies":17670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6068,"num_write_kb":382656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"655'6116","reported_seq":8511,"reported_epoch":655,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:38:22.518435+0000","last_change":"2024-04-15T18:37:54.706535+0000","last_active":"2024-04-15T18:38:22.518435+0000","last_peered":"2024-04-15T18:38:22.518435+0000","last_clean":"2024-04-15T18:37:52.458773+0000","last_became_active":"2024-04-15T18:37:54.665797+0000","last_became_peered":"2024-04-15T18:37:54.665797+0000","last_unstale":"2024-04-15T18:38:22.518435+0000","last_undegraded":"2024-04-15T18:37:54.537600+0000","last_fullsized":"2024-04-15T18:37:54.499970+0000","mapping_epoch":648,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":6116,"log_dups_size":0,"ondisk_log_size":6116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388759552,"num_objects":5932,"num_object_clones":0,"num_object_copies":17796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3698,"num_objects_unfound":0,"num_objects_dirty":5932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":75890688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[3,8],"avail_no_missing":["3","1","8"],"object_location_counts":[{"shards":"1,3,8","objects":3698},{"shards":"3,8","objects":2233}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"655'5999","reported_seq":6076,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:19.746261+0000","last_change":"2024-04-15T18:36:41.540133+0000","last_active":"2024-04-15T18:38:19.746261+0000","last_peered":"2024-04-15T18:38:19.746261+0000","last_clean":"2024-04-15T18:38:19.746261+0000","last_became_active":"2024-04-15T18:36:41.539815+0000","last_became_peered":"2024-04-15T18:36:41.539815+0000","last_unstale":"2024-04-15T18:38:19.746261+0000","last_undegraded":"2024-04-15T18:38:19.746261+0000","last_fullsized":"2024-04-15T18:38:19.746261+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:36:40.483753+0000","objects_scrubbed":0,"log_size":5999,"log_dups_size":0,"ondisk_log_size":5999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393150464,"num_objects":5999,"num_object_clones":0,"num_object_copies":17997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5999,"num_write_kb":383936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"655'6214","reported_seq":6352,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.581041+0000","last_change":"2024-04-15T18:38:01.733547+0000","last_active":"2024-04-15T18:38:22.581041+0000","last_peered":"2024-04-15T18:38:22.581041+0000","last_clean":"2024-04-15T18:38:22.581041+0000","last_became_active":"2024-04-15T18:36:41.539742+0000","last_became_peered":"2024-04-15T18:36:41.539742+0000","last_unstale":"2024-04-15T18:38:22.581041+0000","last_undegraded":"2024-04-15T18:38:22.581041+0000","last_fullsized":"2024-04-15T18:38:22.581041+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5307","last_scrub_stamp":"2024-04-15T18:38:01.733256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:01.733256+0000","objects_scrubbed":5286,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:24.729773+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395182080,"num_objects":6030,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6214,"num_write_kb":391808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"655'6019","reported_seq":6155,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.469112+0000","last_change":"2024-04-15T18:37:58.764935+0000","last_active":"2024-04-15T18:38:22.469112+0000","last_peered":"2024-04-15T18:38:22.469112+0000","last_clean":"2024-04-15T18:38:22.469112+0000","last_became_active":"2024-04-15T18:36:41.503519+0000","last_became_peered":"2024-04-15T18:36:41.503519+0000","last_unstale":"2024-04-15T18:38:22.469112+0000","last_undegraded":"2024-04-15T18:38:22.469112+0000","last_fullsized":"2024-04-15T18:38:22.469112+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5032","last_scrub_stamp":"2024-04-15T18:37:58.764742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:37:58.764742+0000","objects_scrubbed":4982,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:03.041642+0000","scrub_duration":1555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382926848,"num_objects":5843,"num_object_clones":0,"num_object_copies":17529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":379584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"655'6144","reported_seq":6288,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:23.118628+0000","last_change":"2024-04-15T18:38:16.379973+0000","last_active":"2024-04-15T18:38:23.118628+0000","last_peered":"2024-04-15T18:38:23.118628+0000","last_clean":"2024-04-15T18:38:23.118628+0000","last_became_active":"2024-04-15T18:36:41.505208+0000","last_became_peered":"2024-04-15T18:36:41.505208+0000","last_unstale":"2024-04-15T18:38:23.118628+0000","last_undegraded":"2024-04-15T18:38:23.118628+0000","last_fullsized":"2024-04-15T18:38:23.118628+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'5850","last_scrub_stamp":"2024-04-15T18:38:16.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:16.379689+0000","objects_scrubbed":5821,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:38.594247+0000","scrub_duration":1565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386400256,"num_objects":5896,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":385280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2],"acting":[10,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"655'6019","reported_seq":6158,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.558441+0000","last_change":"2024-04-15T18:38:03.401985+0000","last_active":"2024-04-15T18:38:22.558441+0000","last_peered":"2024-04-15T18:38:22.558441+0000","last_clean":"2024-04-15T18:38:22.558441+0000","last_became_active":"2024-04-15T18:36:41.505014+0000","last_became_peered":"2024-04-15T18:36:41.505014+0000","last_unstale":"2024-04-15T18:38:22.558441+0000","last_undegraded":"2024-04-15T18:38:22.558441+0000","last_fullsized":"2024-04-15T18:38:22.558441+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5214","last_scrub_stamp":"2024-04-15T18:38:03.401892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:03.401892+0000","objects_scrubbed":5176,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:04.256370+0000","scrub_duration":1956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382140416,"num_objects":5831,"num_object_clones":0,"num_object_copies":17493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"655'6026","reported_seq":6162,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:22.813071+0000","last_change":"2024-04-15T18:37:58.690558+0000","last_active":"2024-04-15T18:38:22.813071+0000","last_peered":"2024-04-15T18:38:22.813071+0000","last_clean":"2024-04-15T18:38:22.813071+0000","last_became_active":"2024-04-15T18:36:41.505722+0000","last_became_peered":"2024-04-15T18:36:41.505722+0000","last_unstale":"2024-04-15T18:38:22.813071+0000","last_undegraded":"2024-04-15T18:38:22.813071+0000","last_fullsized":"2024-04-15T18:38:22.813071+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5023","last_scrub_stamp":"2024-04-15T18:37:58.690464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:37:58.690464+0000","objects_scrubbed":5006,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:02.754362+0000","scrub_duration":978,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383123456,"num_objects":5846,"num_object_clones":0,"num_object_copies":17538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6026,"num_write_kb":379904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"655'6145","reported_seq":6287,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:21.308391+0000","last_change":"2024-04-15T18:38:09.578956+0000","last_active":"2024-04-15T18:38:21.308391+0000","last_peered":"2024-04-15T18:38:21.308391+0000","last_clean":"2024-04-15T18:38:21.308391+0000","last_became_active":"2024-04-15T18:36:41.539618+0000","last_became_peered":"2024-04-15T18:36:41.539618+0000","last_unstale":"2024-04-15T18:38:21.308391+0000","last_undegraded":"2024-04-15T18:38:21.308391+0000","last_fullsized":"2024-04-15T18:38:21.308391+0000","mapping_epoch":621,"log_start":"0'0","ondisk_log_start":"0'0","created":621,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"651'5663","last_scrub_stamp":"2024-04-15T18:38:09.578656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:36:40.483753+0000","last_clean_scrub_stamp":"2024-04-15T18:38:09.578656+0000","objects_scrubbed":5606,"log_size":6145,"log_dups_size":0,"ondisk_log_size":6145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:27.937241+0000","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399179776,"num_objects":6091,"num_object_clones":0,"num_object_copies":18273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1639,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-04-15T18:38:17.510745+0000","last_change":"2024-04-15T18:38:05.598869+0000","last_active":"2024-04-15T18:38:17.510745+0000","last_peered":"2024-04-15T18:38:17.510745+0000","last_clean":"2024-04-15T18:38:17.510745+0000","last_became_active":"2024-04-15T18:37:56.526550+0000","last_became_peered":"2024-04-15T18:37:56.526550+0000","last_unstale":"2024-04-15T18:38:17.510745+0000","last_undegraded":"2024-04-15T18:38:17.510745+0000","last_fullsized":"2024-04-15T18:38:17.510745+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:38:05.598729+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:38:05.598729+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:39:27.142015+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6222577664,"num_objects":94949,"num_object_clones":0,"num_object_copies":284847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":11333,"num_objects_unfound":0,"num_objects_dirty":94949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97045,"num_write_kb":6143808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3141,"num_bytes_recovered":205848576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18892787712,"data_stored":18892783640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97045,"ondisk_log_size":97045,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926174,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2470324,"kb_used_data":1903276,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":91274828,"statfs":{"total":95995035648,"available":93465423872,"internally_reserved":0,"allocated":1948954624,"data_stored":1948541736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":580647364},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515756,"kb_used_data":1127716,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92229396,"statfs":{"total":95995035648,"available":94442901504,"internally_reserved":0,"allocated":1154781184,"data_stored":1154371868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2387116,"kb_used_data":1896420,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91358036,"statfs":{"total":95995035648,"available":93550628864,"internally_reserved":0,"allocated":1941934080,"data_stored":1941524764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874604,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503404,"kb_used_data":1165412,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92241748,"statfs":{"total":95995035648,"available":94455549952,"internally_reserved":0,"allocated":1193381888,"data_stored":1192972572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212340,"kb_used_data":775788,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92532812,"statfs":{"total":95995035648,"available":94753599488,"internally_reserved":0,"allocated":794406912,"data_stored":793994044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":447019466},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876336,"kb_used_data":795304,"kb_used_omap":111,"kb_used_meta":80912,"kb_avail":92868816,"statfs":{"total":95995035648,"available":95097667584,"internally_reserved":0,"allocated":814391296,"data_stored":813982188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":82854555},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575252,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967792,"kb_used_data":1449320,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91777360,"statfs":{"total":95995035648,"available":93980016640,"internally_reserved":0,"allocated":1484103680,"data_stored":1483690292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738773,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2898604,"kb_used_data":2330148,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":90846548,"statfs":{"total":95995035648,"available":93026865152,"internally_reserved":0,"allocated":2386071552,"data_stored":2385662236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194024,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2531244,"kb_used_data":2197988,"kb_used_omap":114,"kb_used_meta":333133,"kb_avail":91213908,"statfs":{"total":95995035648,"available":93403041792,"internally_reserved":0,"allocated":2250739712,"data_stored":2250330396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":341128433},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609154,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3527596,"kb_used_data":2965476,"kb_used_omap":44,"kb_used_meta":562067,"kb_avail":90217556,"statfs":{"total":95995035648,"available":92382777344,"internally_reserved":0,"allocated":3036647424,"data_stored":3036238108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":575557162},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986560,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2038572,"kb_used_data":1511780,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91706580,"statfs":{"total":95995035648,"available":93907537920,"internally_reserved":0,"allocated":1548062720,"data_stored":1547653404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088108,"kb_used_data":747044,"kb_used_omap":4,"kb_used_meta":341051,"kb_avail":92657044,"statfs":{"total":95995035648,"available":94880813056,"internally_reserved":0,"allocated":764973056,"data_stored":764563740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":349236929},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2385051648,"data_stored":2385051648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1481510912,"data_stored":1481506840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2244149248,"data_stored":2244149248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3019833344,"data_stored":3019833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1546256384,"data_stored":1546256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":763559936,"data_stored":763559936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":426115072,"data_stored":426115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":789118976,"data_stored":789118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1192296448,"data_stored":1192296448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1937768448,"data_stored":1937768448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1148256256,"data_stored":1148256256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1958871040,"data_stored":1958871040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:38:34.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-04-15T18:38:34.870 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:34 smithi049 ceph-mon[40087]: pgmap v1710: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 2/263640 objects degraded (0.001%); 9904/263640 objects misplaced (3.757%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:38:34.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:34 smithi159 ceph-mon[45336]: pgmap v1710: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 2/263640 objects degraded (0.001%); 9904/263640 objects misplaced (3.757%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:38:35.702 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:35.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:35.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:35.896 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:38:36.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:38:36.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:38:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:38:36.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:38:36.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:38:36.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:38:36.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:36 smithi049 ceph-mon[40087]: pgmap v1711: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2/260739 objects degraded (0.001%); 9693/260739 objects misplaced (3.718%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:38:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:36 smithi159 ceph-mon[45336]: pgmap v1711: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2/260739 objects degraded (0.001%); 9693/260739 objects misplaced (3.718%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:38:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:36 smithi082 ceph-mon[44206]: pgmap v1711: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 2/260739 objects degraded (0.001%); 9693/260739 objects misplaced (3.718%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:38:37.482 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:37.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:37.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:37.645 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:37.806 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:37.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:37.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:37.996 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/260739 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:38.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/260739 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:38.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/260739 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:38.584 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:38.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:38.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:38.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:38.721 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:38.743 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:38.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:38.768 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:38:38.768 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:38:38.820 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:38:38.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:38.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:38 smithi049 ceph-mon[40087]: pgmap v1712: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/259305 objects degraded (0.001%); 9693/259305 objects misplaced (3.738%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:38:38.877 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:38.897 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:38.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:38.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:38:38.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:38:38.985 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:38:39.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:38 smithi082 ceph-mon[44206]: pgmap v1712: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/259305 objects degraded (0.001%); 9693/259305 objects misplaced (3.738%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:38:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1142712183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:39.029 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:39.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:38:39.050 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:39.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:39.086 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:38:39.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:38:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:38 smithi159 ceph-mon[45336]: pgmap v1712: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/259305 objects degraded (0.001%); 9693/259305 objects misplaced (3.738%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:38:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1142712183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:39.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:38:39.172 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:38:39.182 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:39.202 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:38:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:39.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:38:39.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1142712183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:38:39.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:38:39.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:38:39.307 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:39.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:38:39.335 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:39.354 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:39.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:39.438 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:38:39.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:38:39.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:38:39.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:38:39.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:38:39.439 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:38:39.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:39.529 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:40.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:39 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:39 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:39 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:40.706 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:40.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:40.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:38:40.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:40.862 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:38:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:40.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:38:40.969 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:38:40.997 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:41.018 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:38:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:41.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:38:41.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:41.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:41.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:40 smithi049 ceph-mon[40087]: pgmap v1713: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 2/252252 objects degraded (0.001%); 9241/252252 objects misplaced (3.663%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:38:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:40 smithi159 ceph-mon[45336]: pgmap v1713: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 2/252252 objects degraded (0.001%); 9241/252252 objects misplaced (3.663%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:38:41.204 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:41.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:38:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:40 smithi082 ceph-mon[44206]: pgmap v1713: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 2/252252 objects degraded (0.001%); 9241/252252 objects misplaced (3.663%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:38:43.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:42 smithi159 ceph-mon[45336]: pgmap v1714: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/248787 objects degraded (0.001%); 8991/248787 objects misplaced (3.614%); 8.6 MiB/s, 137 objects/s recovering 2024-04-15T18:38:43.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/248787 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:43.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:42 smithi049 ceph-mon[40087]: pgmap v1714: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/248787 objects degraded (0.001%); 8991/248787 objects misplaced (3.614%); 8.6 MiB/s, 137 objects/s recovering 2024-04-15T18:38:43.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/248787 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:43.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:42 smithi082 ceph-mon[44206]: pgmap v1714: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2/248787 objects degraded (0.001%); 8991/248787 objects misplaced (3.614%); 8.6 MiB/s, 137 objects/s recovering 2024-04-15T18:38:43.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/248787 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:44.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:38:44.722 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:44.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:44.828 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:44.861 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:44.890 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:44.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:44.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:44 smithi049 ceph-mon[40087]: pgmap v1715: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 2/241149 objects degraded (0.001%); 8537/241149 objects misplaced (3.540%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:38:44.993 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:45.023 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:45.045 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:45.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:45.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:44 smithi159 ceph-mon[45336]: pgmap v1715: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 2/241149 objects degraded (0.001%); 8537/241149 objects misplaced (3.540%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:38:45.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:45.203 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:45.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:44 smithi082 ceph-mon[44206]: pgmap v1715: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 2/241149 objects degraded (0.001%); 8537/241149 objects misplaced (3.540%); 7.8 MiB/s, 125 objects/s recovering 2024-04-15T18:38:46.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:38:47.149 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:47.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:47.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:46 smithi049 ceph-mon[40087]: pgmap v1716: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 2/238428 objects degraded (0.001%); 8270/238428 objects misplaced (3.469%); 8.5 MiB/s, 136 objects/s recovering 2024-04-15T18:38:47.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:47.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:47.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:47.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:47.262 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:46 smithi082 ceph-mon[44206]: pgmap v1716: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 2/238428 objects degraded (0.001%); 8270/238428 objects misplaced (3.469%); 8.5 MiB/s, 136 objects/s recovering 2024-04-15T18:38:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:47.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:46 smithi159 ceph-mon[45336]: pgmap v1716: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 700 op/s; 2/238428 objects degraded (0.001%); 8270/238428 objects misplaced (3.469%); 8.5 MiB/s, 136 objects/s recovering 2024-04-15T18:38:47.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:47.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:47.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:47.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:47.290 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:47.313 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:47.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:47.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:47.450 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:47.482 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:47.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:47.585 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:47.614 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:47.638 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:47.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:47.741 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:47.770 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:47.795 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:47.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:47.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:38:48.044 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.079 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:38:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:38:48.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:47 smithi159 ceph-mon[45336]: osdmap e659: 12 total, 12 up, 12 in 2024-04-15T18:38:48.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/238428 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:48.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:47 smithi049 ceph-mon[40087]: osdmap e659: 12 total, 12 up, 12 in 2024-04-15T18:38:48.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/238428 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:48.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:38:48.222 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:38:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:38:48.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:47 smithi082 ceph-mon[44206]: osdmap e659: 12 total, 12 up, 12 in 2024-04-15T18:38:48.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/238428 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:48.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:48.382 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.401 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:48.505 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:48.533 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.552 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:48.613 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:48.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:38:48.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:48.682 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.706 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:48.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:48.838 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.859 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:48.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:48.964 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:48.990 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:48.997 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:38:49.013 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:49.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:49.120 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:49.154 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:49.179 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:49.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: pgmap v1718: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2/237009 objects degraded (0.001%); 8270/237009 objects misplaced (3.489%); 8.9 MiB/s, 142 objects/s recovering 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: osdmap e660: 12 total, 12 up, 12 in 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1240003218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:48 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: pgmap v1718: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2/237009 objects degraded (0.001%); 8270/237009 objects misplaced (3.489%); 8.9 MiB/s, 142 objects/s recovering 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: osdmap e660: 12 total, 12 up, 12 in 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1240003218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:48 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.283 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:38:49.310 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:49.332 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:38:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:49.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:38:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: pgmap v1718: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 2/237009 objects degraded (0.001%); 8270/237009 objects misplaced (3.489%); 8.9 MiB/s, 142 objects/s recovering 2024-04-15T18:38:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: osdmap e660: 12 total, 12 up, 12 in 2024-04-15T18:38:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1240003218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:48 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:38:49.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:38:49.487 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:50.002 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 7 pgp_num to 16 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: osdmap e661: 12 total, 12 up, 12 in 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1051167949' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:50.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: osdmap e661: 12 total, 12 up, 12 in 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1051167949' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:50.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:50.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:50.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:50.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:38:50.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:38:50.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: osdmap e661: 12 total, 12 up, 12 in 2024-04-15T18:38:50.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1051167949' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:50.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-15T18:38:51.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:50 smithi049 ceph-mon[40087]: pgmap v1721: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2/230235 objects degraded (0.001%); 7721/230235 objects misplaced (3.354%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:51.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:51 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:38:51.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:51 smithi049 ceph-mon[40087]: osdmap e662: 12 total, 12 up, 12 in 2024-04-15T18:38:51.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:50 smithi082 ceph-mon[44206]: pgmap v1721: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2/230235 objects degraded (0.001%); 7721/230235 objects misplaced (3.354%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:51.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:38:51.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:50 smithi082 ceph-mon[44206]: osdmap e662: 12 total, 12 up, 12 in 2024-04-15T18:38:51.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:50 smithi159 ceph-mon[45336]: pgmap v1721: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 2/230235 objects degraded (0.001%); 7721/230235 objects misplaced (3.354%); 8.5 MiB/s, 135 objects/s recovering 2024-04-15T18:38:51.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-04-15T18:38:51.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:50 smithi159 ceph-mon[45336]: osdmap e662: 12 total, 12 up, 12 in 2024-04-15T18:38:52.568 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:38:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:52.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:38:52.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:38:52.705 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:52.740 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:38:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:52.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:38:52.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:38:52.893 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:52.918 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:38:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:38:53.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:38:53.053 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:53.082 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:38:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:53.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:38:53.186 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:53.216 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:53.240 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:53.241 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:53 smithi159 ceph-mon[45336]: pgmap v1723: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 2/226944 objects degraded (0.001%); 7451/226944 objects misplaced (3.283%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:38:53.241 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/226944 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:53.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:53.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:53 smithi082 ceph-mon[44206]: pgmap v1723: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 2/226944 objects degraded (0.001%); 7451/226944 objects misplaced (3.283%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:38:53.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/226944 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:53.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:38:53.378 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:53.402 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:38:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:53.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:38:53.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:53 smithi049 ceph-mon[40087]: pgmap v1723: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 2/226944 objects degraded (0.001%); 7451/226944 objects misplaced (3.283%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:38:53.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/226944 objects degraded (0.001%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:38:53.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:38:53.651 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:55.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:55 smithi082 ceph-mon[44206]: pgmap v1724: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s; 2/220053 objects degraded (0.001%); 6881/220053 objects misplaced (3.127%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:38:55.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:55 smithi159 ceph-mon[45336]: pgmap v1724: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s; 2/220053 objects degraded (0.001%); 6881/220053 objects misplaced (3.127%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:38:55.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:55 smithi049 ceph-mon[40087]: pgmap v1724: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 941 op/s; 2/220053 objects degraded (0.001%); 6881/220053 objects misplaced (3.127%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:38:56.914 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:38:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:56.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:38:57.022 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:38:57.052 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:57.082 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:38:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:57.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:38:57.187 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:38:57.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:57 smithi049 ceph-mon[40087]: pgmap v1725: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2/217377 objects degraded (0.001%); 6595/217377 objects misplaced (3.034%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:38:57.217 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:57.240 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:38:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:57.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:38:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:57 smithi082 ceph-mon[44206]: pgmap v1725: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2/217377 objects degraded (0.001%); 6595/217377 objects misplaced (3.034%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:38:57.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:38:57.377 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:57.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:57 smithi159 ceph-mon[45336]: pgmap v1725: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2/217377 objects degraded (0.001%); 6595/217377 objects misplaced (3.034%); 9.3 MiB/s, 148 objects/s recovering 2024-04-15T18:38:57.401 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:38:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:57.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:38:57.507 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:38:57.544 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:57.569 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:38:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:57.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:38:57.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:38:57.706 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:57.735 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:38:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:38:57.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:38:57.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:38:57.905 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:38:58.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/217377 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:38:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/217377 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:38:58.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/217377 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:38:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:59 smithi159 ceph-mon[45336]: pgmap v1726: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 2/216780 objects degraded (0.001%); 6595/216780 objects misplaced (3.042%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:38:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:38:59.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:38:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:38:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:59 smithi049 ceph-mon[40087]: pgmap v1726: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 2/216780 objects degraded (0.001%); 6595/216780 objects misplaced (3.042%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:38:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:38:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:38:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:38:59.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:59 smithi082 ceph-mon[44206]: pgmap v1726: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 2/216780 objects degraded (0.001%); 6595/216780 objects misplaced (3.042%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:38:59.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:38:59.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:38:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:00.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:00 smithi159 ceph-mon[45336]: osdmap e663: 12 total, 12 up, 12 in 2024-04-15T18:39:00.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:00 smithi049 ceph-mon[40087]: osdmap e663: 12 total, 12 up, 12 in 2024-04-15T18:39:00.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:00 smithi082 ceph-mon[44206]: osdmap e663: 12 total, 12 up, 12 in 2024-04-15T18:39:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:01 smithi159 ceph-mon[45336]: pgmap v1728: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 2/209388 objects degraded (0.001%); 6009/209388 objects misplaced (2.870%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:39:01.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:01 smithi049 ceph-mon[40087]: pgmap v1728: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 2/209388 objects degraded (0.001%); 6009/209388 objects misplaced (2.870%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:39:01.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:01 smithi082 ceph-mon[44206]: pgmap v1728: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 2/209388 objects degraded (0.001%); 6009/209388 objects misplaced (2.870%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:39:01.602 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:01.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:01.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:01.776 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:01.924 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 11, 3, 2, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 11, 3, 2, 5, 6] 2024-04-15T18:39:01.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 0 1 2024-04-15T18:39:03.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:03.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:03 smithi159 ceph-mon[45336]: pgmap v1729: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2/206064 objects degraded (0.001%); 5599/206064 objects misplaced (2.717%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:39:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:03 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/206064 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:03.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:03.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:03 smithi082 ceph-mon[44206]: pgmap v1729: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2/206064 objects degraded (0.001%); 5599/206064 objects misplaced (2.717%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:39:03.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:03 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/206064 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:03.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:03.455 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:03.457 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:03 smithi049 ceph-mon[40087]: pgmap v1729: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2/206064 objects degraded (0.001%); 5599/206064 objects misplaced (2.717%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:39:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:03 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/206064 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:03.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:03.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:03.481 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:03.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:03.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:03.676 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:03.698 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:39:03.699 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:39:03.699 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:39:03.699 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:39:03.699 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:39:03.725 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:03.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:03.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:39:03.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:03.863 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:03.888 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:03.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:03.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:04.061 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:04.097 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:04.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:04.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:04.237 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:04.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:04.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:04.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:04.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:04 smithi159 ceph-mon[45336]: osdmap e664: 12 total, 12 up, 12 in 2024-04-15T18:39:04.397 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:04.421 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:04.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:04.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:04 smithi049 ceph-mon[40087]: osdmap e664: 12 total, 12 up, 12 in 2024-04-15T18:39:04.525 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:04.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:04 smithi082 ceph-mon[44206]: osdmap e664: 12 total, 12 up, 12 in 2024-04-15T18:39:04.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:39:04.555 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:04.578 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:04.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:04.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:04.788 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:04.843 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:04.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:04.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:39:04.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:04.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:39:04.976 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:05.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:05.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:05.004 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:39:05.071 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:39:05.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:05.122 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:39:05.135 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:05.157 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:05.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:05.162 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:39:05.211 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:39:05.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:39:05.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:05 smithi082 ceph-mon[44206]: pgmap v1731: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 2/198762 objects degraded (0.001%); 4946/198762 objects misplaced (2.488%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:39:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:05 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1607313935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.312 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:39:05.329 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:05.361 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:39:05.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:05 smithi159 ceph-mon[45336]: pgmap v1731: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 2/198762 objects degraded (0.001%); 4946/198762 objects misplaced (2.488%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:39:05.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:05 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1607313935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:05.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.406 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:39:05.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:39:05.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:05 smithi049 ceph-mon[40087]: pgmap v1731: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 2/198762 objects degraded (0.001%); 4946/198762 objects misplaced (2.488%); 13 MiB/s, 206 objects/s recovering 2024-04-15T18:39:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:05 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1607313935' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:05.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:39:05.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:39:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:39:06.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:06.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:07 smithi159 ceph-mon[45336]: pgmap v1732: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 2/196812 objects degraded (0.001%); 4946/196812 objects misplaced (2.513%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:39:07.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:07 smithi049 ceph-mon[40087]: pgmap v1732: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 2/196812 objects degraded (0.001%); 4946/196812 objects misplaced (2.513%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:39:07.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:07 smithi082 ceph-mon[44206]: pgmap v1732: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 2/196812 objects degraded (0.001%); 4946/196812 objects misplaced (2.513%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:39:08.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/196812 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:08.490 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:08.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:08.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/196812 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:08.600 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:08.630 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/196812 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:08.634 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:08.656 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:08.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:08.760 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:08.792 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:08.816 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:08.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:08.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:08.998 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:09.031 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:09.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:09.145 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:09.193 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:09.224 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:09.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:09.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:09.359 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:09.381 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:09.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:09.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:09 smithi049 ceph-mon[40087]: pgmap v1733: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 2/194715 objects degraded (0.001%); 4613/194715 objects misplaced (2.369%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:39:09.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:09.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:09 smithi082 ceph-mon[44206]: pgmap v1733: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 2/194715 objects degraded (0.001%); 4613/194715 objects misplaced (2.369%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:39:09.538 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:09.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:09 smithi159 ceph-mon[45336]: pgmap v1733: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 2/194715 objects degraded (0.001%); 4613/194715 objects misplaced (2.369%); 10 MiB/s, 164 objects/s recovering 2024-04-15T18:39:09.991 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:10.294 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:10.294 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:10.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:10.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:10.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:10.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:11.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:11 smithi049 ceph-mon[40087]: pgmap v1734: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2/188193 objects degraded (0.001%); 4009/188193 objects misplaced (2.130%); 12 MiB/s, 199 objects/s recovering 2024-04-15T18:39:11.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:11 smithi049 ceph-mon[40087]: osdmap e665: 12 total, 12 up, 12 in 2024-04-15T18:39:11.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:11 smithi082 ceph-mon[44206]: pgmap v1734: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2/188193 objects degraded (0.001%); 4009/188193 objects misplaced (2.130%); 12 MiB/s, 199 objects/s recovering 2024-04-15T18:39:11.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:11 smithi082 ceph-mon[44206]: osdmap e665: 12 total, 12 up, 12 in 2024-04-15T18:39:11.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:11 smithi159 ceph-mon[45336]: pgmap v1734: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2/188193 objects degraded (0.001%); 4009/188193 objects misplaced (2.130%); 12 MiB/s, 199 objects/s recovering 2024-04-15T18:39:11.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:11 smithi159 ceph-mon[45336]: osdmap e665: 12 total, 12 up, 12 in 2024-04-15T18:39:12.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:39:12.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2550983262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2550983262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2550983262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:39:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-04-15T18:39:13.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: pgmap v1736: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2/184857 objects degraded (0.001%); 3645/184857 objects misplaced (1.972%); 9.7 MiB/s, 154 objects/s recovering 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: osdmap e666: 12 total, 12 up, 12 in 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/184857 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1335204416,config=1059521453,health=3847179634,kv=3777394421}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=1943755843,logm=4102195298}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938137330}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308503458}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513567493}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3532219774}) 2024-04-15T18:39:13.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192102911}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721886646}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299234505}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981160889}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778731974}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316041021}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575601395}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598921453}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281611576}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231098784}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121267125}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822089940}) 2024-04-15T18:39:13.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577523723}) 2024-04-15T18:39:13.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: pgmap v1736: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2/184857 objects degraded (0.001%); 3645/184857 objects misplaced (1.972%); 9.7 MiB/s, 154 objects/s recovering 2024-04-15T18:39:13.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: osdmap e666: 12 total, 12 up, 12 in 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/184857 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1335204416,config=1059521453,health=3847179634,kv=3777394421}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=1943755843,logm=4102195298}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938137330}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308503458}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513567493}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3532219774}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192102911}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721886646}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299234505}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981160889}) 2024-04-15T18:39:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778731974}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316041021}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575601395}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598921453}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281611576}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231098784}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121267125}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822089940}) 2024-04-15T18:39:13.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577523723}) 2024-04-15T18:39:13.564 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:13.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:13.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: pgmap v1736: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2/184857 objects degraded (0.001%); 3645/184857 objects misplaced (1.972%); 9.7 MiB/s, 154 objects/s recovering 2024-04-15T18:39:13.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:39:13.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: osdmap e666: 12 total, 12 up, 12 in 2024-04-15T18:39:13.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/184857 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1335204416,config=1059521453,health=3847179634,kv=3777394421}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {kv=10,logm=90} crc {kv=1943755843,logm=4102195298}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3938137330}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308503458}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1513567493}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3532219774}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3192102911}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721886646}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299234505}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2981160889}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3778731974}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316041021}) 2024-04-15T18:39:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575601395}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598921453}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281611576}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231098784}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1121267125}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3822089940}) 2024-04-15T18:39:13.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3577523723}) 2024-04-15T18:39:13.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:13.722 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:14.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 1 1 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268523890}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258203279}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199773392}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2229663073}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042237879}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989509040}) 2024-04-15T18:39:14.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161961520}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491181625}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651439655}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012530616}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996137379}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255480777}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4109404934}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977703207}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278259068}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010210748}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655799119}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=577414837}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1963537297}) 2024-04-15T18:39:14.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2544330787}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136870292}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1805933758}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259673896}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1983432963}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575351792}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562130758}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2786626171}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3463137585}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2453369805}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51100850}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764528633}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906057167}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772897079}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=154016240}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941950799}) 2024-04-15T18:39:14.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833471855}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885424910}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3294719643}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243350917}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377731929}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588016979}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2145856997}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520297145}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885750906}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459707390}) 2024-04-15T18:39:14.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2719587245}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177642179}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701528454}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179440159}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771792539}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531043980}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799292436}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776446695}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289360373}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1991385321}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095439913}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762723938}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440033241}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353216879}) 2024-04-15T18:39:14.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=64} crc {logm=3966321569,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1182993650,mon_config_key=389852272}) 2024-04-15T18:39:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3409274780}) 2024-04-15T18:39:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2291512894}) 2024-04-15T18:39:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=23,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=59} crc {mon_config_key=1353955235,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=3231712577,osdmap=339076014}) 2024-04-15T18:39:14.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3762049970}) 2024-04-15T18:39:14.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268523890}) 2024-04-15T18:39:14.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258203279}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199773392}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2229663073}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042237879}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989509040}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161961520}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491181625}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651439655}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012530616}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996137379}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255480777}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4109404934}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977703207}) 2024-04-15T18:39:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278259068}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010210748}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655799119}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=577414837}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1963537297}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2544330787}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136870292}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1805933758}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259673896}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1983432963}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575351792}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562130758}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2786626171}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3463137585}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2453369805}) 2024-04-15T18:39:14.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51100850}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764528633}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906057167}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772897079}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=154016240}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941950799}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833471855}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885424910}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3294719643}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243350917}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377731929}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588016979}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2145856997}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520297145}) 2024-04-15T18:39:14.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885750906}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459707390}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2719587245}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177642179}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701528454}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179440159}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771792539}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531043980}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799292436}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776446695}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289360373}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1991385321}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095439913}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762723938}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440033241}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353216879}) 2024-04-15T18:39:14.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=64} crc {logm=3966321569,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1182993650,mon_config_key=389852272}) 2024-04-15T18:39:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3409274780}) 2024-04-15T18:39:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2291512894}) 2024-04-15T18:39:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=23,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=59} crc {mon_config_key=1353955235,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=3231712577,osdmap=339076014}) 2024-04-15T18:39:14.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3762049970}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2268523890}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3258203279}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4199773392}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2229663073}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042237879}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3989509040}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161961520}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491181625}) 2024-04-15T18:39:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651439655}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012530616}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996137379}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255480777}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4109404934}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977703207}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1278259068}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010210748}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655799119}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=577414837}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1963537297}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2544330787}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=136870292}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1805933758}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259673896}) 2024-04-15T18:39:14.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1983432963}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575351792}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=562130758}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2786626171}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3463137585}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2453369805}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=51100850}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2764528633}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906057167}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772897079}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=154016240}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941950799}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3833471855}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885424910}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3294719643}) 2024-04-15T18:39:14.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243350917}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1377731929}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=588016979}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2145856997}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520297145}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885750906}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3459707390}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2719587245}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177642179}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3701528454}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179440159}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771792539}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531043980}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799292436}) 2024-04-15T18:39:14.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776446695}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289360373}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1991385321}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095439913}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762723938}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440033241}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2353216879}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=64} crc {logm=3966321569,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=1182993650,mon_config_key=389852272}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3409274780}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2291512894}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=23,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=59} crc {mon_config_key=1353955235,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=3231712577,osdmap=339076014}) 2024-04-15T18:39:14.650 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=3762049970}) 2024-04-15T18:39:15.299 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:15.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:15.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:15 smithi049 ceph-mon[40087]: pgmap v1738: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/177048 objects degraded (0.001%); 3090/177048 objects misplaced (1.745%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:15.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:15.434 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:15.456 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:15.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:15.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:15 smithi082 ceph-mon[44206]: pgmap v1738: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/177048 objects degraded (0.001%); 3090/177048 objects misplaced (1.745%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:15.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:15.629 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:15.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:15 smithi159 ceph-mon[45336]: pgmap v1738: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/177048 objects degraded (0.001%); 3090/177048 objects misplaced (1.745%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:15.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:15.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:15.833 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:15.857 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:15.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:15.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:15.991 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:16.016 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:16.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:16.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:16.189 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:17.280 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:39:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:17 smithi082 ceph-mon[44206]: pgmap v1739: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/174954 objects degraded (0.001%); 3090/174954 objects misplaced (1.766%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:39:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:17 smithi082 ceph-mon[44206]: osdmap e667: 12 total, 12 up, 12 in 2024-04-15T18:39:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:17.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:17.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:17.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:17 smithi159 ceph-mon[45336]: pgmap v1739: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/174954 objects degraded (0.001%); 3090/174954 objects misplaced (1.766%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:39:17.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:17 smithi159 ceph-mon[45336]: osdmap e667: 12 total, 12 up, 12 in 2024-04-15T18:39:17.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:17.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:17 smithi049 ceph-mon[40087]: pgmap v1739: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s; 2/174954 objects degraded (0.001%); 3090/174954 objects misplaced (1.766%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:39:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:17 smithi049 ceph-mon[40087]: osdmap e667: 12 total, 12 up, 12 in 2024-04-15T18:39:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: osdmap e668: 12 total, 12 up, 12 in 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:18.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:18 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/174954 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:18.599 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: osdmap e668: 12 total, 12 up, 12 in 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:18.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:18 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/174954 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:18.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1021398470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: osdmap e668: 12 total, 12 up, 12 in 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:18.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:18 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/174954 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:18.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:18.738 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:18.762 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:18.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:18.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:18.875 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:18.893 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:18.917 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:18.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:19.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:19.067 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:19.121 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:19.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:19.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:19.283 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:19.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:19 smithi049 ceph-mon[40087]: pgmap v1742: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 2/173529 objects degraded (0.001%); 2760/173529 objects misplaced (1.591%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:39:19.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:19 smithi049 ceph-mon[40087]: osdmap e669: 12 total, 12 up, 12 in 2024-04-15T18:39:19.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:19 smithi082 ceph-mon[44206]: pgmap v1742: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 2/173529 objects degraded (0.001%); 2760/173529 objects misplaced (1.591%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:39:19.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:19 smithi082 ceph-mon[44206]: osdmap e669: 12 total, 12 up, 12 in 2024-04-15T18:39:19.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:19 smithi159 ceph-mon[45336]: pgmap v1742: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 2/173529 objects degraded (0.001%); 2760/173529 objects misplaced (1.591%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:39:19.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:19 smithi159 ceph-mon[45336]: osdmap e669: 12 total, 12 up, 12 in 2024-04-15T18:39:20.246 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:20.247 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:20.460 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:21.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:39:21.168 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:21.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:21.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:21.305 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:21.344 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:21.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:21.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:21.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:21 smithi049 ceph-mon[40087]: pgmap v1744: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2/166068 objects degraded (0.001%); 2023/166068 objects misplaced (1.218%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:39:21.496 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:21.501 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:21.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:21 smithi082 ceph-mon[44206]: pgmap v1744: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2/166068 objects degraded (0.001%); 2023/166068 objects misplaced (1.218%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:39:21.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:21 smithi159 ceph-mon[45336]: pgmap v1744: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2/166068 objects degraded (0.001%); 2023/166068 objects misplaced (1.218%); 11 MiB/s, 177 objects/s recovering 2024-04-15T18:39:21.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:39:22.377 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:22.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:22.481 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:22.513 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:22.545 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:22.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:22.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:22.686 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:23.040 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:23.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:23.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:23.210 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:23.410 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:23.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:23.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:23 smithi049 ceph-mon[40087]: pgmap v1745: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 2/162408 objects degraded (0.001%); 1701/162408 objects misplaced (1.047%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:23.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/162408 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:23.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:23.557 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:23.634 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:23.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:23.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:23 smithi159 ceph-mon[45336]: pgmap v1745: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 2/162408 objects degraded (0.001%); 1701/162408 objects misplaced (1.047%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:23.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/162408 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:23.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:23.766 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:23.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:23 smithi082 ceph-mon[44206]: pgmap v1745: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 2/162408 objects degraded (0.001%); 1701/162408 objects misplaced (1.047%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:39:23.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/162408 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:23.780 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:23.780 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:23.785 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:23.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:23.891 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:23.917 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:23.938 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:23.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:24.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:39:24.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:39:24.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:39:24.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:39:24.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:39:24.043 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:24.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:39:24.045 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:24.072 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:39:24.075 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:24.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:39:24.144 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:39:24.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:39:24.173 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:24.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:24.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:39:24.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:39:24.223 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:39:24.224 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:39:24.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:24.300 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-04-15T18:39:24.319 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: osdmap e670: 12 total, 12 up, 12 in 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1378442954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.487 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:39:24.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:39:24.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: osdmap e670: 12 total, 12 up, 12 in 2024-04-15T18:39:24.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1378442954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:24.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: osdmap e670: 12 total, 12 up, 12 in 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1378442954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:24.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:39:24.861 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: pgmap v1747: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 2/154923 objects degraded (0.001%); 865/154923 objects misplaced (0.558%); 19 MiB/s, 302 objects/s recovering 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:39:25.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:25 smithi082 ceph-mon[44206]: osdmap e671: 12 total, 12 up, 12 in 2024-04-15T18:39:25.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: pgmap v1747: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 2/154923 objects degraded (0.001%); 865/154923 objects misplaced (0.558%); 19 MiB/s, 302 objects/s recovering 2024-04-15T18:39:25.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:39:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:25 smithi159 ceph-mon[45336]: osdmap e671: 12 total, 12 up, 12 in 2024-04-15T18:39:25.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: pgmap v1747: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s; 2/154923 objects degraded (0.001%); 865/154923 objects misplaced (0.558%); 19 MiB/s, 302 objects/s recovering 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/537439530' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:39:25.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:25 smithi049 ceph-mon[40087]: osdmap e671: 12 total, 12 up, 12 in 2024-04-15T18:39:25.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 4 1 2024-04-15T18:39:26.658 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:39:26.658 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:39:26.658 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:39:26.658 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:39:26.658 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:39:26.693 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:26.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:26.729 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:26.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:26.825 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:26.846 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:26.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:26.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:26.981 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:27.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:27 smithi159 ceph-mon[45336]: pgmap v1749: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2/152985 objects degraded (0.001%); 865/152985 objects misplaced (0.565%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:39:27.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:27 smithi159 ceph-mon[45336]: osdmap e672: 12 total, 12 up, 12 in 2024-04-15T18:39:27.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:27 smithi049 ceph-mon[40087]: pgmap v1749: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2/152985 objects degraded (0.001%); 865/152985 objects misplaced (0.565%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:39:27.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:27 smithi049 ceph-mon[40087]: osdmap e672: 12 total, 12 up, 12 in 2024-04-15T18:39:27.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:27 smithi082 ceph-mon[44206]: pgmap v1749: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s; 2/152985 objects degraded (0.001%); 865/152985 objects misplaced (0.565%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:39:27.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:27 smithi082 ceph-mon[44206]: osdmap e672: 12 total, 12 up, 12 in 2024-04-15T18:39:28.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:39:28.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:28 smithi049 ceph-mon[40087]: osdmap e673: 12 total, 12 up, 12 in 2024-04-15T18:39:28.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2/152985 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:28.319 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:39:28.571 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:39:28.595 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:28 smithi082 ceph-mon[44206]: osdmap e673: 12 total, 12 up, 12 in 2024-04-15T18:39:28.596 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2/152985 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:28.596 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:39:28.627 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:39:28.630 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:28.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:28.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:28 smithi159 ceph-mon[45336]: osdmap e673: 12 total, 12 up, 12 in 2024-04-15T18:39:28.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2/152985 objects degraded (0.001%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:39:28.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:39:28.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:28.816 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:28.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:28.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:28.955 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:28.966 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:28.990 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:29.249 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:29.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:29.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:29.382 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:29.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:29.512 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:29 smithi082 ceph-mon[44206]: pgmap v1752: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 1/151500 objects degraded (0.001%); 607/151500 objects misplaced (0.401%); 3.8 MiB/s, 61 objects/s recovering 2024-04-15T18:39:29.513 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:29 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:29.513 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:39:29.513 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:29 smithi082 ceph-mon[44206]: osdmap e674: 12 total, 12 up, 12 in 2024-04-15T18:39:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:29.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:29.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:29.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:29 smithi159 ceph-mon[45336]: pgmap v1752: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 1/151500 objects degraded (0.001%); 607/151500 objects misplaced (0.401%); 3.8 MiB/s, 61 objects/s recovering 2024-04-15T18:39:29.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:29 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:29.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:39:29.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:29 smithi159 ceph-mon[45336]: osdmap e674: 12 total, 12 up, 12 in 2024-04-15T18:39:29.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:29 smithi049 ceph-mon[40087]: pgmap v1752: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 322 op/s; 1/151500 objects degraded (0.001%); 607/151500 objects misplaced (0.401%); 3.8 MiB/s, 61 objects/s recovering 2024-04-15T18:39:29.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:29 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:29.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4064191592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:39:29.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:29 smithi049 ceph-mon[40087]: osdmap e674: 12 total, 12 up, 12 in 2024-04-15T18:39:29.704 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:29.750 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:29.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:29.768 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:29.857 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:29.911 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:29.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:29.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:30.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:30.102 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:30.129 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:30.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:30.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:30.286 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:30.347 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:30 smithi159 ceph-mon[45336]: 7.2 scrub starts 2024-04-15T18:39:30.347 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1495449194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.347 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:30 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.348 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:30 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:30.348 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:30 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:30.362 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-04-15T18:39:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:30 smithi082 ceph-mon[44206]: 7.2 scrub starts 2024-04-15T18:39:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1495449194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:30 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:30 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:30 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:30.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:30 smithi049 ceph-mon[40087]: 7.2 scrub starts 2024-04-15T18:39:30.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1495449194' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:30 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:39:30.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:30 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:30.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:30 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:57366 6.e 6:7868e193:::benchmark_data_smithi159_139342_object57365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:39:30.951 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 354.392608833, 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.461560207, 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57366 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:39:30.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566092+0000", 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5719999999999999e-06 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566149+0000", 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7744999999999997e-05 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566162+0000", 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2685e-05 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566170+0000", 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0239999999999997e-06 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566220+0000", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9595000000000002e-05 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566225+0000", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567183+0000", 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095810800000000005 2024-04-15T18:39:30.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567310+0000", 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012681300000000001 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,2", 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567438+0000", 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000127935 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.569385+0000", 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001947465 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.570421+0000", 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010361540000000001 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027576+0000", 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45715493899999998 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027618+0000", 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1644999999999999e-05 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027650+0000", 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2159000000000002e-05 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.958 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:158265 6.e 6:7c3d7611:::benchmark_data_smithi159_139342_object68386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 223.51049088100001, 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46395664199999997, 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 158265 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448210+0000", 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.12e-06 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448221+0000", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0628999999999999e-05 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448248+0000", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7606999999999998e-05 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448263+0000", 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4593000000000001e-05 2024-04-15T18:39:30.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448391+0000", 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012803099999999999 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448516+0000", 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125313 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,2", 2024-04-15T18:39:30.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448581+0000", 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5152000000000002e-05 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.452334+0000", 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037521999999999998 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.453884+0000", 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015505219999999999 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912067+0000", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45818335399999999 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912117+0000", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9982999999999999e-05 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912165+0000", 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7138000000000002e-05 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.964 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:177229 6.6 6:6142b3eb:::benchmark_data_smithi159_139342_object87350:head [delete] snapc 0=[] ondisk+write+known_if_redirected e609)", 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 192.94641426499999, 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29316968399999999, 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:30.965 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 177229 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012286+0000", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.172e-06 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012305+0000", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8970999999999998e-05 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012317+0000", 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.185e-05 2024-04-15T18:39:30.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012325+0000", 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3860000000000007e-06 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012390+0000", 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4948000000000002e-05 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012534+0000", 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014438100000000001 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012665+0000", 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000130576 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.017780+0000", 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051149960000000001 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.028518+0000", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010737714000000001 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305326+0000", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.276808479 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305380+0000", 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3721999999999998e-05 2024-04-15T18:39:30.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305454+0000", 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4488999999999998e-05 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:179751 6.6 6:68808706:::benchmark_data_smithi159_139342_object89872:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 189.58170456400001, 2024-04-15T18:39:30.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25397364900000002, 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179751 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376995+0000", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2200000000000003e-07 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376997+0000", 2024-04-15T18:39:30.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.497e-06 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377007+0000", 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6509999999999996e-06 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377013+0000", 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6879999999999996e-06 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377058+0000", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5090999999999999e-05 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377180+0000", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012175700000000001 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377300+0000", 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012048999999999999 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.382334+0000", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.005033694 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.382479+0000", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014488700000000001 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630903+0000", 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24842419199999999 2024-04-15T18:39:30.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630939+0000", 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630968+0000", 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.851e-05 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31782 7.1 7:82fc4c0d:::benchmark_data_smithi159_149998_object31781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.92231590700001, 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31042012400000002, 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31782 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0600000000000001e-07 2024-04-15T18:39:30.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036412+0000", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8798000000000001e-05 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036418+0000", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7520000000000002e-06 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036421+0000", 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1470000000000001e-06 2024-04-15T18:39:30.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037561+0000", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011398739999999999 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037642+0000", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1481000000000001e-05 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037728+0000", 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5646e-05 2024-04-15T18:39:30.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038470+0000", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000741729 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038630+0000", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016056800000000001 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346734+0000", 2024-04-15T18:39:30.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30810352899999999 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346773+0000", 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8949999999999998e-05 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346803+0000", 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0343999999999999e-05 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31793 7.4 7:2fbec3a3:::benchmark_data_smithi159_149998_object31792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.91086223100001, 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31189694899999998, 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31793 2024-04-15T18:39:30.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047839+0000", 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0530000000000001e-06 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047884+0000", 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5325999999999999e-05 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047911+0000", 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7457000000000002e-05 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047932+0000", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0888999999999999e-05 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048077+0000", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144361 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048189+0000", 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112435 2024-04-15T18:39:30.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048271+0000", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1411999999999999e-05 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048888+0000", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000617872 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.049960+0000", 2024-04-15T18:39:30.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001071935 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359672+0000", 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30971173899999999 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359707+0000", 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4965000000000002e-05 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359734+0000", 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6505000000000001e-05 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31795 7.1 7:81918b20:::benchmark_data_smithi159_149998_object31794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.90742865799999, 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32741436800000001, 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31795 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:39:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051271+0000", 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1900000000000001e-07 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051305+0000", 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3918000000000003e-05 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051315+0000", 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0376e-05 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051323+0000", 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8329999999999994e-06 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051397+0000", 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4245999999999994e-05 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051487+0000", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0140000000000001e-05 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051573+0000", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5922999999999997e-05 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052330+0000", 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075673500000000002 2024-04-15T18:39:30.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.053094+0000", 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076432699999999998 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378621+0000", 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32552697800000002 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:30.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378663+0000", 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1357999999999999e-05 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378685+0000", 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1715000000000001e-05 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:31782 7.1 e631/621 ondisk, result = 0)", 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:39:30.996 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.90570353000001, 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29381337200000002, 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:39:30.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052996+0000", 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.41e-07 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052997+0000", 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.265e-06 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346692+0000", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29369515000000002 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346695+0000", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3160000000000001e-06 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346724+0000", 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8614000000000001e-05 2024-04-15T18:39:30.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346730+0000", 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346809+0000", 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8584999999999994e-05 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:32099 7.1 7:85bb4853:::benchmark_data_smithi159_149998_object32098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.563518+0000", 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 139.395181218, 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23326264299999999, 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 32099 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.001 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563518+0000", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563518+0000", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563518+0000", 2024-04-15T18:39:31.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.54e-07 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563582+0000", 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3806000000000005e-05 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563593+0000", 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0991e-05 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563601+0000", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8150000000000006e-06 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563651+0000", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0345e-05 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563778+0000", 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012611400000000001 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.563879+0000", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101048 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.564845+0000", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00096666700000000005 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.587417+0000", 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.022572048000000001 2024-04-15T18:39:31.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.796715+0000", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20929730699999999 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.796743+0000", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8809000000000001e-05 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.796780+0000", 2024-04-15T18:39:31.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6838999999999997e-05 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55371 7.1 7:830c46a9:::benchmark_data_smithi159_149998_object55370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.80799165400001, 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25659491200000001, 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.007 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55371 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:39:31.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150708+0000", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1909999999999999e-06 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150765+0000", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7090999999999999e-05 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150775+0000", 2024-04-15T18:39:31.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0016e-05 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150781+0000", 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5960000000000003e-06 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150813+0000", 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2066000000000001e-05 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150906+0000", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3035999999999995e-05 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150998+0000", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699000000000004e-05 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152490+0000", 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014920739999999999 2024-04-15T18:39:31.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152917+0000", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042668799999999999 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407183+0000", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25426654300000001 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407248+0000", 2024-04-15T18:39:31.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4925999999999995e-05 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407302+0000", 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3986000000000001e-05 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55375 7.1 7:8b006e6f:::benchmark_data_smithi159_149998_object55374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:39:31.013 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.805050643, 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28137763399999999, 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55375 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153649+0000", 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8899999999999998e-07 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153727+0000", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8459000000000002e-05 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153742+0000", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4202000000000001e-05 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153748+0000", 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3269999999999997e-06 2024-04-15T18:39:31.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153791+0000", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3341999999999999e-05 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153937+0000", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014536700000000001 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.154086+0000", 2024-04-15T18:39:31.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149674 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.155698+0000", 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016121659999999999 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.156469+0000", 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00077094200000000005 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434910+0000", 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27844058500000002 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434974+0000", 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3758999999999997e-05 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435026+0000", 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2021999999999998e-05 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55371 7.1 e639/621 ondisk, result = 0)", 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.157810+0000", 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.800889293, 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24950082900000001, 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.157810+0000", 2024-04-15T18:39:31.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.157810+0000", 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.157810+0000", 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.54e-07 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.157816+0000", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5010000000000004e-06 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407112+0000", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24929593899999999 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407118+0000", 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1480000000000003e-06 2024-04-15T18:39:31.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407167+0000", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9097000000000001e-05 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407176+0000", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.9430000000000006e-06 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407310+0000", 2024-04-15T18:39:31.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013434699999999999 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55403 7.1 7:8aca0ab0:::benchmark_data_smithi159_149998_object55402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.78222596800001, 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27940837899999998, 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.024 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55403 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:39:31.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176474+0000", 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3319999999999999e-06 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176505+0000", 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0821000000000002e-05 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176519+0000", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3957999999999999e-05 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176527+0000", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.79e-06 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176587+0000", 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9973000000000003e-05 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176743+0000", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000155814 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176908+0000", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165198 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.178583+0000", 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001675073 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179015+0000", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043260999999999999 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455781+0000", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27676578499999999 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455836+0000", 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4837999999999999e-05 2024-04-15T18:39:31.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455881+0000", 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5187e-05 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55375 7.1 e639/621 ondisk, result = 0)", 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.77936255500001, 2024-04-15T18:39:31.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25569712700000002, 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179337+0000", 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8100000000000002e-07 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179341+0000", 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2080000000000002e-06 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.032 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434841+0000", 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25549990900000003 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434848+0000", 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5019999999999998e-06 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434894+0000", 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6836999999999999e-05 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.033 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434904+0000", 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3519999999999996e-06 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435033+0000", 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012963800000000001 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.034 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55417 7.4 7:2aa6faee:::benchmark_data_smithi159_149998_object55416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.76836387900001, 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26001807999999998, 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55417 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.035 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190336+0000", 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3629999999999999e-06 2024-04-15T18:39:31.036 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190388+0000", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1354000000000003e-05 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190402+0000", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4353e-05 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190409+0000", 2024-04-15T18:39:31.037 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9110000000000004e-06 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190462+0000", 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3458999999999998e-05 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190560+0000", 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7677999999999996e-05 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:39:31.038 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190645+0000", 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4790000000000006e-05 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192252+0000", 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016068009999999999 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192667+0000", 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000415189 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.039 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450236+0000", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25756942999999999 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450301+0000", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5079000000000005e-05 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450353+0000", 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1672999999999998e-05 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.040 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55420 7.4 7:2ab922f1:::benchmark_data_smithi159_149998_object55419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.763977774, 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27355053699999998, 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.041 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55420 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.042 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194722+0000", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0699999999999996e-07 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194774+0000", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2434e-05 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194785+0000", 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0767e-05 2024-04-15T18:39:31.043 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194793+0000", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6389999999999992e-06 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194835+0000", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2561000000000001e-05 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194931+0000", 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5377999999999994e-05 2024-04-15T18:39:31.044 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.195017+0000", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6224000000000007e-05 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196339+0000", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013224020000000001 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196756+0000", 2024-04-15T18:39:31.045 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00041645199999999999 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468144+0000", 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27138785100000001 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468225+0000", 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1422999999999996e-05 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.046 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468272+0000", 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6499000000000003e-05 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55417 7.4 e639/621 ondisk, result = 0)", 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.761867933, 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25352939099999999, 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.047 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.048 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8000000000000001e-07 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196834+0000", 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3640000000000002e-06 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450171+0000", 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25333760599999999 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.049 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450176+0000", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0150000000000003e-06 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450220+0000", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3334000000000002e-05 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450230+0000", 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0484e-05 2024-04-15T18:39:31.050 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450360+0000", 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013020800000000001 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:78937 7.1 7:802f7ee1:::benchmark_data_smithi159_149998_object78936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e650)", 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:55.852405+0000", 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.106294141999996, 2024-04-15T18:39:31.051 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22801801499999999, 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 78937 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852405+0000", 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.052 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852405+0000", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852406+0000", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5600000000000001e-06 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852446+0000", 2024-04-15T18:39:31.053 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9388999999999998e-05 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852471+0000", 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.582e-05 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852480+0000", 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3939999999999996e-06 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.054 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852532+0000", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2175000000000001e-05 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852652+0000", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120393 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.852767+0000", 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000114988 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.055 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:55.854992+0000", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022249399999999999 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:56.065726+0000", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21073346000000001 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:56.080308+0000", 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014582014000000001 2024-04-15T18:39:31.056 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:56.080369+0000", 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1403999999999997e-05 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:56.080423+0000", 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3477999999999999e-05 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.057 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:124560 7.4 7:2a9d76e5:::benchmark_data_smithi159_149998_object28560:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 30.796762434000001, 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27331726099999998, 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124560 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.058 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161937+0000", 2024-04-15T18:39:31.059 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5700000000000001e-07 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161941+0000", 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0520000000000003e-06 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161950+0000", 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8769999999999999e-06 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.060 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161958+0000", 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3780000000000003e-06 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161994+0000", 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6823999999999998e-05 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162070+0000", 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5834999999999995e-05 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.061 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162135+0000", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4783000000000006e-05 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.165059+0000", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029242880000000001 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.168016+0000", 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029565440000000002 2024-04-15T18:39:31.062 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435166+0000", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26715059699999999 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435204+0000", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7344e-05 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435254+0000", 2024-04-15T18:39:31.063 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9882000000000001e-05 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:143366 7.4 7:21b697b5:::benchmark_data_smithi159_149998_object47366:head [delete] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4.0527165959999998, 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53450922700000003, 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:39:31.064 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 143366 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:39:31.065 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905984+0000", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9429999999999999e-06 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905995+0000", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0349999999999999e-05 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906022+0000", 2024-04-15T18:39:31.066 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7248999999999999e-05 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906037+0000", 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5150000000000001e-05 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906143+0000", 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106461 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:39:31.067 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906269+0000", 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012599200000000001 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906398+0000", 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012908 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.909755+0000", 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033568209999999999 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.068 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.910370+0000", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061510700000000005 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440411+0000", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53004094499999999 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440446+0000", 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4193000000000003e-05 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:39:31.069 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440491+0000", 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5936000000000002e-05 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:39:31.070 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:39:31.346 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:31.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:31.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:31.503 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:31.644 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: pgmap v1754: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 1/146358 objects degraded (0.001%); 240/146358 objects misplaced (0.164%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:39:31.644 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: 7.f scrub starts 2024-04-15T18:39:31.644 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: 7.6 scrub starts 2024-04-15T18:39:31.644 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: 7.2 scrub ok 2024-04-15T18:39:31.644 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: osdmap e675: 12 total, 12 up, 12 in 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: 7.6 scrub ok 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:31 smithi082 ceph-mon[44206]: 7.3 scrub starts 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: pgmap v1754: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 1/146358 objects degraded (0.001%); 240/146358 objects misplaced (0.164%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: 7.f scrub starts 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: 7.6 scrub starts 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: 7.2 scrub ok 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: osdmap e675: 12 total, 12 up, 12 in 2024-04-15T18:39:31.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: 7.6 scrub ok 2024-04-15T18:39:31.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:31 smithi159 ceph-mon[45336]: 7.3 scrub starts 2024-04-15T18:39:31.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:39:31.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: pgmap v1754: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 1/146358 objects degraded (0.001%); 240/146358 objects misplaced (0.164%); 7.5 MiB/s, 119 objects/s recovering 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: 7.f scrub starts 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: 7.6 scrub starts 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: 7.2 scrub ok 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: osdmap e675: 12 total, 12 up, 12 in 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: 7.6 scrub ok 2024-04-15T18:39:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:31 smithi049 ceph-mon[40087]: 7.3 scrub starts 2024-04-15T18:39:31.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:39:31.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 5 1 2024-04-15T18:39:31.942 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:32.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:32.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:32.307 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:32.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:32 smithi049 ceph-mon[40087]: 7.f scrub ok 2024-04-15T18:39:32.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:32 smithi049 ceph-mon[40087]: osdmap e676: 12 total, 12 up, 12 in 2024-04-15T18:39:32.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:32 smithi049 ceph-mon[40087]: 7.1 scrub starts 2024-04-15T18:39:32.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:32 smithi049 ceph-mon[40087]: 7.d scrub starts 2024-04-15T18:39:32.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:32 smithi049 ceph-mon[40087]: 7.3 scrub ok 2024-04-15T18:39:32.411 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:32.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:32.458 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:32.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:39:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:32 smithi082 ceph-mon[44206]: 7.f scrub ok 2024-04-15T18:39:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:32 smithi082 ceph-mon[44206]: osdmap e676: 12 total, 12 up, 12 in 2024-04-15T18:39:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:32 smithi082 ceph-mon[44206]: 7.1 scrub starts 2024-04-15T18:39:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:32 smithi082 ceph-mon[44206]: 7.d scrub starts 2024-04-15T18:39:32.495 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:32 smithi082 ceph-mon[44206]: 7.3 scrub ok 2024-04-15T18:39:32.497 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:32.517 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:32.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:39:32.557 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:32.576 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:32.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:32.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:39:32.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:39:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:32 smithi159 ceph-mon[45336]: 7.f scrub ok 2024-04-15T18:39:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:32 smithi159 ceph-mon[45336]: osdmap e676: 12 total, 12 up, 12 in 2024-04-15T18:39:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:32 smithi159 ceph-mon[45336]: 7.1 scrub starts 2024-04-15T18:39:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:32 smithi159 ceph-mon[45336]: 7.d scrub starts 2024-04-15T18:39:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:32 smithi159 ceph-mon[45336]: 7.3 scrub ok 2024-04-15T18:39:32.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:32.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:39:32.725 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:39:32.728 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:32.764 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:32.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:39:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:32.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:32.819 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:39:32.863 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:39:32.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:32.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:39:32.962 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:32.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:39:33.010 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:33.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:39:33.047 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:39:33.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:39:33.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:39:33.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:39:33.048 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:39:33.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: pgmap v1757: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 1/142704 objects degraded (0.001%); 214/142704 objects misplaced (0.150%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: 7.1 scrub ok 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osdmap e677: 12 total, 12 up, 12 in 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: 7.4 scrub starts 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4176825793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: 7.d scrub ok 2024-04-15T18:39:33.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1/142704 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: pgmap v1757: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 1/142704 objects degraded (0.001%); 214/142704 objects misplaced (0.150%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:39:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:39:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:39:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: 7.1 scrub ok 2024-04-15T18:39:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osdmap e677: 12 total, 12 up, 12 in 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: 7.4 scrub starts 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4176825793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: 7.d scrub ok 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1/142704 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: pgmap v1757: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 1/142704 objects degraded (0.001%); 214/142704 objects misplaced (0.150%); 5.8 MiB/s, 93 objects/s recovering 2024-04-15T18:39:33.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:39:33.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: 7.1 scrub ok 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osdmap e677: 12 total, 12 up, 12 in 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: 7.4 scrub starts 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4176825793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: 7.d scrub ok 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1/142704 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:33.969 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:33.969 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:34.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: 7.4 scrub ok 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: 7.8 scrub starts 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: 7.c scrub starts 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: 7.b scrub starts 2024-04-15T18:39:34.409 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:34 smithi049 ceph-mon[40087]: 7.e scrub starts 2024-04-15T18:39:34.670 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: 7.4 scrub ok 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: 7.8 scrub starts 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: 7.c scrub starts 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: 7.b scrub starts 2024-04-15T18:39:34.671 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:34 smithi082 ceph-mon[44206]: 7.e scrub starts 2024-04-15T18:39:34.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:39:34.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: 7.4 scrub ok 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: 7.8 scrub starts 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: 7.c scrub starts 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: 7.b scrub starts 2024-04-15T18:39:34.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:34 smithi159 ceph-mon[45336]: 7.e scrub starts 2024-04-15T18:39:35.079 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: pgmap v1759: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 1/137547 objects degraded (0.001%); 214/137547 objects misplaced (0.156%); 334 KiB/s, 5 objects/s recovering 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: 7.c scrub ok 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: 7.5 scrub starts 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: 7.b scrub ok 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:39:35.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:35 smithi082 ceph-mon[44206]: 7.8 scrub ok 2024-04-15T18:39:35.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: pgmap v1759: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 1/137547 objects degraded (0.001%); 214/137547 objects misplaced (0.156%); 334 KiB/s, 5 objects/s recovering 2024-04-15T18:39:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: 7.c scrub ok 2024-04-15T18:39:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: 7.5 scrub starts 2024-04-15T18:39:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: 7.b scrub ok 2024-04-15T18:39:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:39:35.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:35 smithi049 ceph-mon[40087]: 7.8 scrub ok 2024-04-15T18:39:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: pgmap v1759: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 1/137547 objects degraded (0.001%); 214/137547 objects misplaced (0.156%); 334 KiB/s, 5 objects/s recovering 2024-04-15T18:39:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: 7.c scrub ok 2024-04-15T18:39:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: 7.5 scrub starts 2024-04-15T18:39:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: 7.b scrub ok 2024-04-15T18:39:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:39:35.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:35 smithi159 ceph-mon[45336]: 7.8 scrub ok 2024-04-15T18:39:36.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:36 smithi082 ceph-mon[44206]: 7.0 scrub starts 2024-04-15T18:39:36.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:36 smithi082 ceph-mon[44206]: 7.5 scrub ok 2024-04-15T18:39:36.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:36 smithi082 ceph-mon[44206]: 7.0 scrub ok 2024-04-15T18:39:36.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:36 smithi082 ceph-mon[44206]: 7.e scrub ok 2024-04-15T18:39:36.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:36 smithi159 ceph-mon[45336]: 7.0 scrub starts 2024-04-15T18:39:36.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:36 smithi159 ceph-mon[45336]: 7.5 scrub ok 2024-04-15T18:39:36.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:36 smithi159 ceph-mon[45336]: 7.0 scrub ok 2024-04-15T18:39:36.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:36 smithi159 ceph-mon[45336]: 7.e scrub ok 2024-04-15T18:39:36.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:36 smithi049 ceph-mon[40087]: 7.0 scrub starts 2024-04-15T18:39:36.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:36 smithi049 ceph-mon[40087]: 7.5 scrub ok 2024-04-15T18:39:36.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:36 smithi049 ceph-mon[40087]: 7.0 scrub ok 2024-04-15T18:39:36.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:36 smithi049 ceph-mon[40087]: 7.e scrub ok 2024-04-15T18:39:37.432 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:37 smithi082 ceph-mon[44206]: pgmap v1760: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 1/135741 objects degraded (0.001%); 214/135741 objects misplaced (0.158%); 277 KiB/s, 4 objects/s recovering 2024-04-15T18:39:37.432 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:37 smithi082 ceph-mon[44206]: 7.a scrub starts 2024-04-15T18:39:37.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:37 smithi049 ceph-mon[40087]: pgmap v1760: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 1/135741 objects degraded (0.001%); 214/135741 objects misplaced (0.158%); 277 KiB/s, 4 objects/s recovering 2024-04-15T18:39:37.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:37 smithi049 ceph-mon[40087]: 7.a scrub starts 2024-04-15T18:39:37.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:37 smithi159 ceph-mon[45336]: pgmap v1760: 17 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 1/135741 objects degraded (0.001%); 214/135741 objects misplaced (0.158%); 277 KiB/s, 4 objects/s recovering 2024-04-15T18:39:37.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:37 smithi159 ceph-mon[45336]: 7.a scrub starts 2024-04-15T18:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:38.118 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:38.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: 7.a scrub ok 2024-04-15T18:39:38.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:39:38.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:39:38.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: 7.9 scrub starts 2024-04-15T18:39:38.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: osdmap e678: 12 total, 12 up, 12 in 2024-04-15T18:39:38.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:38 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: 7.a scrub ok 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: 7.9 scrub starts 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: osdmap e678: 12 total, 12 up, 12 in 2024-04-15T18:39:38.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:38 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:38.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: 7.a scrub ok 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: 7.9 scrub starts 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: osdmap e678: 12 total, 12 up, 12 in 2024-04-15T18:39:38.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:38 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:39:39.071 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:39.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:39.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:39.240 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:39.241 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:39:39.273 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:39.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:39.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: pgmap v1762: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 2.2 MiB/s, 35 objects/s recovering 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: 7.9 scrub ok 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: 7.7 scrub starts 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:39.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:39 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: pgmap v1762: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 2.2 MiB/s, 35 objects/s recovering 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: 7.9 scrub ok 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: 7.7 scrub starts 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:39.698 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:39 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:39.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: pgmap v1762: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 2.2 MiB/s, 35 objects/s recovering 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: 7.9 scrub ok 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: 7.7 scrub starts 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/135741 objects degraded (0.001%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:39.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:39 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:40.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:40.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:40.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:40.520 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:40 smithi082 ceph-mon[44206]: 7.7 scrub ok 2024-04-15T18:39:40.570 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:40 smithi159 ceph-mon[45336]: 7.7 scrub ok 2024-04-15T18:39:40.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:40 smithi049 ceph-mon[40087]: 7.7 scrub ok 2024-04-15T18:39:40.856 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:40.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:40.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:39:40.993 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:41.019 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:39:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:41.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:39:41.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:39:41.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:41.163 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:41.164 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:41.166 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:41.189 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:41.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:41.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:41.326 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:41.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:41 smithi049 ceph-mon[40087]: pgmap v1763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:41.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:41 smithi082 ceph-mon[44206]: pgmap v1763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:41 smithi159 ceph-mon[45336]: pgmap v1763: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:42.177 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:42.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:42.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:42.322 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:42.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:42 smithi082 ceph-mon[44206]: from='client.26744 172.21.15.49:0/3145965957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3080587060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:42.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:42 smithi082 ceph-mon[44206]: from='client.26744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.454 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:42 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:42.459 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-04-15T18:39:42.467 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:39:42.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:42 smithi159 ceph-mon[45336]: from='client.26744 172.21.15.49:0/3145965957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3080587060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:42.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:42 smithi159 ceph-mon[45336]: from='client.26744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.637 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:42 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:42.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:42 smithi049 ceph-mon[40087]: from='client.26744 172.21.15.49:0/3145965957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3080587060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:42.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:42 smithi049 ceph-mon[40087]: from='client.26744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:39:42.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:42 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:39:43.605 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:43.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:43.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: pgmap v1764: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:39:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:39:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:39:43.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='client.26744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: osdmap e679: 12 total, 12 up, 12 in 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:39:43.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:43 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: pgmap v1764: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='client.26744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:43.726 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: osdmap e679: 12 total, 12 up, 12 in 2024-04-15T18:39:43.727 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.727 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.727 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:39:43.727 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:39:43.727 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:43 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.750 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: pgmap v1764: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 1.5 MiB/s, 23 objects/s recovering 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='client.26744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:39:43.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: osdmap e679: 12 total, 12 up, 12 in 2024-04-15T18:39:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:39:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:39:43.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:43 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:39:44.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 7 1 2024-04-15T18:39:44.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3620456614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:44.837 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:39:44.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:39:44.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:39:44.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:39:44.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3620456614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:39:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:39:44.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:39:44.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:39:44.909 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:44.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:44.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3620456614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:44.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:39:45.014 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:45.049 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:45.072 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:45.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:45.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:45.216 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:45.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:45.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:45.356 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:39:45.358 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:45.385 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:45.405 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:45.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:45.442 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:45.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:45.538 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:45.560 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: pgmap v1766: 17 pgs: 17 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.642 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:45.730 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: pgmap v1766: 17 pgs: 17 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.879 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:45.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: pgmap v1766: 17 pgs: 17 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-15T18:39:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.891 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:45.992 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:46.025 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.049 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:46.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:46.181 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.201 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:46.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:39:46.335 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.357 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:39:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:39:46.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:46.495 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.518 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:46.626 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:46.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:46 smithi159 ceph-mon[45336]: pgmap v1767: 17 pgs: 17 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-04-15T18:39:46.670 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.696 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:46.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:46 smithi082 ceph-mon[44206]: pgmap v1767: 17 pgs: 17 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-04-15T18:39:46.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:46.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:46.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:46 smithi049 ceph-mon[40087]: pgmap v1767: 17 pgs: 17 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-04-15T18:39:46.851 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:46.960 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:46.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:47.075 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:47.113 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:47.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:39:47.361 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:157552 7.f 7:f8fc3e35:::benchmark_data_smithi159_149998_object61552:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.348716+0000", 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.019396219999999999, 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019426151999999999, 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:47.362 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157552 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348716+0000", 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348716+0000", 2024-04-15T18:39:47.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348716+0000", 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9200000000000002e-07 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348719+0000", 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4119999999999999e-06 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348724+0000", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5160000000000002e-06 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.348730+0000", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5600000000000001e-06 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349105+0000", 2024-04-15T18:39:47.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037543100000000002 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349234+0000", 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012928000000000001 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349349+0000", 2024-04-15T18:39:47.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114506 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.362784+0000", 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013435453999999999 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366677+0000", 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038927160000000001 2024-04-15T18:39:47.367 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:157554 7.f 7:f397e879:::benchmark_data_smithi159_149998_object61554:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.349007+0000", 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.019104844999999999, 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019225269999999999, 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:39:47.368 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157554 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349007+0000", 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349007+0000", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349008+0000", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0610000000000001e-06 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349013+0000", 2024-04-15T18:39:47.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1970000000000004e-06 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349025+0000", 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2013e-05 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.349034+0000", 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3280000000000006e-06 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353414+0000", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043806360000000003 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353507+0000", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2338999999999998e-05 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353609+0000", 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102366 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.362791+0000", 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091819699999999994 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366845+0000", 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040538079999999999 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:157556 7.f 7:fd43dc5c:::benchmark_data_smithi159_149998_object61556:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.354258+0000", 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.013854119, 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014058342999999999, 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157556 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.374 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354258+0000", 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354258+0000", 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354259+0000", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.291e-06 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354265+0000", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0329999999999997e-06 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354280+0000", 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4603999999999999e-05 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354288+0000", 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5639999999999997e-06 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.357699+0000", 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034103599999999999 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.357838+0000", 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013981399999999999 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.357974+0000", 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013573 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366690+0000", 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087162660000000003 2024-04-15T18:39:47.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:157557 7.f 7:f487cbf6:::benchmark_data_smithi159_149998_object61557:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.354303+0000", 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.013808548, 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014103152000000001, 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.379 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157557 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354303+0000", 2024-04-15T18:39:47.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354303+0000", 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354304+0000", 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-04-15T18:39:47.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354307+0000", 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7599999999999998e-06 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354315+0000", 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.354323+0000", 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2819999999999999e-06 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.362836+0000", 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085130829999999994 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.362970+0000", 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013465300000000001 2024-04-15T18:39:47.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.363089+0000", 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366696+0000", 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036069380000000001 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.384 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:157562 7.7 e679/670 7:e215f8f1:::benchmark_data_smithi159_149998_object61562:head v 679'9835, mlcod=679'9833)", 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.358212+0000", 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.009899929, 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010288633, 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358212+0000", 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358212+0000", 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358213+0000", 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3399999999999998e-07 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358219+0000", 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5409999999999997e-06 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358237+0000", 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7550999999999998e-05 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.358243+0000", 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8930000000000002e-06 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366491+0000", 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082482509999999998 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.366512+0000", 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1118999999999999e-05 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:157564 7.f 7:f057a892:::benchmark_data_smithi159_149998_object61564:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.367197+0000", 2024-04-15T18:39:47.389 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.00091541300000000001, 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001363319, 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 157564 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367197+0000", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367197+0000", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367197+0000", 2024-04-15T18:39:47.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1699999999999997e-07 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367202+0000", 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1799999999999998e-06 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367212+0000", 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0514e-05 2024-04-15T18:39:47.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367219+0000", 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6900000000000003e-06 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367272+0000", 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3050000000000002e-05 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:47.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367420+0000", 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014826300000000001 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.367559+0000", 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013839200000000001 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:157552 7.f e679/621)", 2024-04-15T18:39:47.394 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.353656+0000", 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.014455727, 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014972668, 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "queued for pg", 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353656+0000", 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353656+0000", 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353657+0000", 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2300000000000001e-07 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.353664+0000", 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.43e-06 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368427+0000", 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014762898 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368434+0000", 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8630000000000002e-06 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.397 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:157554 7.f e679/621)", 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.368388+0000", 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4294967295.9997244, 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028861499999999999, 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "queued for pg", 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368388+0000", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368388+0000", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368388+0000", 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9999999999999997e-07 2024-04-15T18:39:47.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368396+0000", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.396e-06 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368449+0000", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3322999999999998e-05 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368453+0000", 2024-04-15T18:39:47.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9700000000000001e-06 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:157567 7.9 e679/621)", 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:47.368407+0000", 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4294967295.9997048, 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031390299999999998, 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:47.401 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "queued for pg", 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368407+0000", 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368407+0000", 2024-04-15T18:39:47.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368407+0000", 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0400000000000002e-07 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368411+0000", 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.721e-06 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:47.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368463+0000", 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1861999999999999e-05 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:47.368468+0000", 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7759999999999997e-06 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 9 2024-04-15T18:39:47.404 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:39:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:47.406 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:47.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:47.507 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:47.536 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:47.556 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:47.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:47.663 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:47.706 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:47.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:39:48.096 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:48.566 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:48.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:48.574 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-04-15T18:39:48.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:48.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:48 smithi049 ceph-mon[40087]: pgmap v1768: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s 2024-04-15T18:39:48.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/759496800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:48.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:48 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:48.719 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:48.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:48 smithi159 ceph-mon[45336]: pgmap v1768: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s 2024-04-15T18:39:48.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/759496800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:48.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:48 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:48 smithi082 ceph-mon[44206]: pgmap v1768: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s 2024-04-15T18:39:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/759496800' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:48 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:39:49.108 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:39:49.238 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:49.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:49.342 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:49.382 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:49.408 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:49.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:49.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:49.562 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:49.586 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:49.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:49.631 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:39:49.640 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:49.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:39:49.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:49 smithi159 ceph-mon[45336]: osdmap e680: 12 total, 12 up, 12 in 2024-04-15T18:39:49.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:49.749 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:49.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:39:49.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:49 smithi049 ceph-mon[40087]: osdmap e680: 12 total, 12 up, 12 in 2024-04-15T18:39:50.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:39:50.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:49 smithi082 ceph-mon[44206]: osdmap e680: 12 total, 12 up, 12 in 2024-04-15T18:39:50.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:50 smithi159 ceph-mon[45336]: pgmap v1770: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s 2024-04-15T18:39:50.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:50 smithi049 ceph-mon[40087]: pgmap v1770: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s 2024-04-15T18:39:51.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:50 smithi082 ceph-mon[44206]: pgmap v1770: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 703 op/s 2024-04-15T18:39:51.527 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:51.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:51.560 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:39:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:39:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:39:51.560 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:39:51.560 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:39:51.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 8 1 2024-04-15T18:39:51.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:51.757 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:52.408 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:52.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:52.512 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:39:52.543 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:52.567 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:39:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:52.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:39:52.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:39:52.700 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:52.721 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:39:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:52.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:39:52.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:52.854 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:52 smithi159 ceph-mon[45336]: pgmap v1771: 17 pgs: 17 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:39:52.862 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:52.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:52.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:39:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:52.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:52.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:52 smithi049 ceph-mon[40087]: pgmap v1771: 17 pgs: 17 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:39:53.006 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:53.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:52 smithi082 ceph-mon[44206]: pgmap v1771: 17 pgs: 17 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 722 op/s 2024-04-15T18:39:53.049 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:53.082 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:53.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:53.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:39:53.246 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:53.315 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:39:53.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:39:53.618 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:39:53.838 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:39:53.876 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:54.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:39:54.610 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:54 smithi159 ceph-mon[45336]: pgmap v1772: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:39:54.610 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:54 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:54.610 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:39:54.610 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:54 smithi159 ceph-mon[45336]: osdmap e681: 12 total, 12 up, 12 in 2024-04-15T18:39:54.638 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:39:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:54.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:39:54.752 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:39:54.795 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:54.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:54 smithi049 ceph-mon[40087]: pgmap v1772: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:39:54.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:54 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:54.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:39:54.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:54 smithi049 ceph-mon[40087]: osdmap e681: 12 total, 12 up, 12 in 2024-04-15T18:39:55.000 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:39:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:55.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:39:55.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:54 smithi082 ceph-mon[44206]: pgmap v1772: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s 2024-04-15T18:39:55.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:54 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:39:55.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3795782816' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:39:55.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:54 smithi082 ceph-mon[44206]: osdmap e681: 12 total, 12 up, 12 in 2024-04-15T18:39:55.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:55.448 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:55.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:55.556 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:55.582 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:55.606 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:55.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:55.673 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:39:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.25447.0:186306 5.e e536/358 5:78abf885:::benchmark_data_smithi159_116570_object93146:head v 536'11692, mlcod=536'11687)", 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 462.57692472500003, 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36219522999999998, 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.103992+0000", 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1799999999999999e-07 2024-04-15T18:39:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.104003+0000", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0264e-05 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462410+0000", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35840707900000002 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462420+0000", 2024-04-15T18:39:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0335e-05 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462477+0000", 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7283e-05 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.462498+0000", 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1155999999999999e-05 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466146+0000", 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036480150000000001 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:13.466187+0000", 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0880000000000002e-05 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:24786 6.1 6:8e595635:::benchmark_data_smithi159_139342_object24785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e550)", 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:32:55.852428+0000", 2024-04-15T18:39:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 419.828488483, 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22942001100000001, 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 24786 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852428+0000", 2024-04-15T18:39:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852428+0000", 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852429+0000", 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2799999999999995e-07 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852446+0000", 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6307000000000002e-05 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852456+0000", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0248000000000001e-05 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852461+0000", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7289999999999997e-06 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852536+0000", 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4826999999999994e-05 2024-04-15T18:39:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852617+0000", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1071999999999999e-05 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.852704+0000", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6842999999999995e-05 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.853452+0000", 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074760699999999996 2024-04-15T18:39:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:55.853781+0000", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032868900000000001 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:56.081715+0000", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.227934265 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:56.081772+0000", 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6959999999999997e-05 2024-04-15T18:39:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:32:56.081848+0000", 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6636000000000003e-05 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:89863 6.9 e578/541 6:91fd2ab3:::benchmark_data_smithi159_139342_object89862:head v 578'5537, mlcod=578'5534)", 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:34:08.945439+0000", 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 346.735477753, 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.376267413, 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945439+0000", 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945439+0000", 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945440+0000", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6000000000000003e-07 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945463+0000", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2711e-05 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.319379+0000", 2024-04-15T18:39:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37391612899999999 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.319385+0000", 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0279999999999997e-06 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.319426+0000", 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1610999999999999e-05 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.319442+0000", 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5811000000000002e-05 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321672+0000", 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022302820000000001 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321707+0000", 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4081000000000001e-05 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:124618 6.1 6:800b1472:::benchmark_data_smithi159_139342_object34739:head [delete] snapc 0=[] ondisk+write+known_if_redirected e587)", 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:34:58.388075+0000", 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 297.292842124, 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.284461087, 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 124618 2024-04-15T18:39:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388075+0000", 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388075+0000", 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388076+0000", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.283e-06 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388081+0000", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2429999999999996e-06 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388093+0000", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1245e-05 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388098+0000", 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9390000000000002e-06 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388154+0000", 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5282999999999999e-05 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388291+0000", 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000136878 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.388412+0000", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000121109 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.395021+0000", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0066088600000000003 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.395213+0000", 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019270100000000001 2024-04-15T18:39:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.672426+0000", 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27721298 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.672483+0000", 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6291e-05 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:58.672536+0000", 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3275000000000001e-05 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(6.3 595/575 [PushOp(6:c99fda26:::benchmark_data_smithi159_139342_object61024:head, version: 564'3856, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:c99fda26:::benchmark_data_smithi159_139342_object61024:head@564'3856, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:23.087046+0000", 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 272.59387093100003, 2024-04-15T18:39:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.342224311, 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.087046+0000", 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.087046+0000", 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.087047+0000", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6199999999999996e-07 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.087090+0000", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3466e-05 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.426496+0000", 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33940607 2024-04-15T18:39:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.426532+0000", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5324999999999998e-05 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.426716+0000", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000184554 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.426758+0000", 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2098999999999998e-05 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.429270+0000", 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025119349999999999 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(6.c 600/575 [PushOp(6:37268a98:::benchmark_data_smithi159_139342_object76663:head, version: 567'4860, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:37268a98:::benchmark_data_smithi159_139342_object76663:head@567'4860, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:47.275745+0000", 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 248.405171798, 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33299212700000003, 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:39:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.275745+0000", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.275745+0000", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.275746+0000", 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4399999999999998e-07 2024-04-15T18:39:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.275782+0000", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5781999999999997e-05 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.605849+0000", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33006723599999999 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.605878+0000", 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.851e-05 2024-04-15T18:39:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.605985+0000", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010772000000000001 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.605995+0000", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9019999999999994e-06 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.608737+0000", 2024-04-15T18:39:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027420330000000001 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:177249 6.9 e609/541 6:9e4639b2:::benchmark_data_smithi159_139342_object87370:head v 609'10922, mlcod=609'10919)", 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:18.051509+0000", 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 217.629408101, 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25966589499999998, 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051509+0000", 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051509+0000", 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051510+0000", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9100000000000003e-07 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051515+0000", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.699e-06 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.306079+0000", 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25456421899999998 2024-04-15T18:39:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.306085+0000", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.045e-06 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.306130+0000", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5337000000000003e-05 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.306150+0000", 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9945999999999998e-05 2024-04-15T18:39:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.311130+0000", 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049797390000000004 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.311175+0000", 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5118999999999999e-05 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:179741 6.9 e611/541 6:91fd2ab3:::benchmark_data_smithi159_139342_object89862:head v 611'11074, mlcod=611'11071)", 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:21.359163+0000", 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 214.32175359499999, 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.91243897600000001, 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.359163+0000", 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.359163+0000", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.359164+0000", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0299999999999999e-07 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.359170+0000", 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5880000000000003e-06 2024-04-15T18:39:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.267448+0000", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.90827760800000001 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.267463+0000", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.465e-05 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.267689+0000", 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000226293 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.267736+0000", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6649e-05 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.271572+0000", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003836426 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.271602+0000", 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0159e-05 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:179752 6.3 e611/600 6:c7acc4bf:::benchmark_data_smithi159_139342_object89873:head v 611'11296, mlcod=611'11293)", 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:21.377391+0000", 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 214.30352583499999, 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.50231203199999996, 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.377391+0000", 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.377391+0000", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.377392+0000", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0950000000000001e-06 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.377401+0000", 2024-04-15T18:39:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5199999999999997e-06 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.876387+0000", 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.49898641500000002 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.876397+0000", 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5510000000000005e-06 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.876483+0000", 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6415000000000005e-05 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.876512+0000", 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9019000000000001e-05 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.879656+0000", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031438479999999999 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.879703+0000", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7169000000000002e-05 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:17276 7.f 7:f9e2ce40:::benchmark_data_smithi159_149998_object17275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e627)", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:00.705743+0000", 2024-04-15T18:39:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 174.97517347499999, 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23113477599999999, 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 17276 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705743+0000", 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705743+0000", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705745+0000", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705767+0000", 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1999999999999999e-05 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705781+0000", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4481000000000001e-05 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705790+0000", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4759999999999998e-06 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705843+0000", 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3149000000000002e-05 2024-04-15T18:39:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.705936+0000", 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3546000000000002e-05 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706025+0000", 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8590000000000004e-05 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706800+0000", 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00077483100000000002 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.707115+0000", 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031515999999999998 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.936810+0000", 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229694752 2024-04-15T18:39:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.936848+0000", 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8062000000000003e-05 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.936878+0000", 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0374000000000001e-05 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:24092 7.7 7:e7b2c8a2:::benchmark_data_smithi159_149998_object24091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e628)", 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:06.040419+0000", 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 169.64049768999999, 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23182048999999999, 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 24092 2024-04-15T18:39:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040419+0000", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040419+0000", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040420+0000", 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4700000000000001e-07 2024-04-15T18:39:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040438+0000", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8097999999999999e-05 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040447+0000", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000003e-06 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040454+0000", 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1049999999999997e-06 2024-04-15T18:39:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040493+0000", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9768999999999998e-05 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040572+0000", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8214999999999996e-05 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,3", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.040657+0000", 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4926000000000006e-05 2024-04-15T18:39:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.041394+0000", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00073729799999999995 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.042035+0000", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00064080099999999998 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272142+0000", 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.230107492 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272197+0000", 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4561999999999997e-05 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272240+0000", 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2936999999999999e-05 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:31794 7.f 7:f52c522c:::benchmark_data_smithi159_149998_object31793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.049375+0000", 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 164.63154218599999, 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.290680253, 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 31794 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049375+0000", 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049375+0000", 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049376+0000", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6400000000000001e-07 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049393+0000", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7598999999999999e-05 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049404+0000", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0719e-05 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049410+0000", 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8109999999999999e-06 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049455+0000", 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4972999999999998e-05 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049538+0000", 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3199000000000001e-05 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049626+0000", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7686e-05 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.050608+0000", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00098203000000000006 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.050959+0000", 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035170500000000001 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.339988+0000", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28902913800000002 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.340024+0000", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726999999999999e-05 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.340055+0000", 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0802000000000001e-05 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:48068 7.7 7:ee2c02a9:::benchmark_data_smithi159_149998_object48067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:25.082001+0000", 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.59891603, 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.217554897, 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 48068 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082001+0000", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082001+0000", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082002+0000", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6000000000000002e-07 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082043+0000", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.172e-05 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082055+0000", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1698e-05 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082061+0000", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8119999999999996e-06 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082104+0000", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2914999999999998e-05 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082191+0000", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6724000000000005e-05 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,3", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.082281+0000", 2024-04-15T18:39:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0593000000000005e-05 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.083650+0000", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013688330000000001 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.084606+0000", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00095574600000000005 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.299477+0000", 2024-04-15T18:39:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.214871753 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.299512+0000", 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4289000000000003e-05 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:25.299556+0000", 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3954e-05 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:110637 7.f 7:f23ca5e6:::benchmark_data_smithi159_149998_object14637:head [delete] snapc 0=[] ondisk+write+known_if_redirected e658)", 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:40.605294+0000", 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 75.075623070999995, 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20617919900000001, 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 110637 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605294+0000", 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605294+0000", 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605295+0000", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.206e-06 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605300+0000", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0089999999999998e-06 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605312+0000", 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2208000000000001e-05 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.605319+0000", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8399999999999997e-06 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606011+0000", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069243600000000005 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606123+0000", 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011195400000000001 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606233+0000", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109566 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.608731+0000", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024984429999999999 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.612506+0000", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003774215 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.811402+0000", 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.198896293 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.811440+0000", 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7800999999999999e-05 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.811473+0000", 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3228000000000001e-05 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:116748 7.f 7:f69146b6:::benchmark_data_smithi159_149998_object20748:head [delete] snapc 0=[] ondisk+write+known_if_redirected e661)", 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:49.150140+0000", 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.530776474000007, 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27330465100000001, 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 116748 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150140+0000", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150140+0000", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150142+0000", 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1489999999999999e-06 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150147+0000", 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7300000000000002e-06 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150159+0000", 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1401e-05 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150165+0000", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5300000000000002e-06 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150208+0000", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3302e-05 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150306+0000", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7720999999999995e-05 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.150433+0000", 2024-04-15T18:39:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012679100000000001 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.154309+0000", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038764490000000001 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.154706+0000", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039618999999999999 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423369+0000", 2024-04-15T18:39:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26866287900000002 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423409+0000", 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0077000000000003e-05 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423445+0000", 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6431999999999999e-05 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:116748 7.f e661/621 ondisk, result = 0)", 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:49.159869+0000", 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.521047944000003, 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26358101299999998, 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.159869+0000", 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.159869+0000", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.159869+0000", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.29e-07 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.159871+0000", 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.751e-06 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423297+0000", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26342594400000002 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423303+0000", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8549999999999998e-06 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423349+0000", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5840000000000002e-05 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423355+0000", 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8589999999999999e-06 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:49.423450+0000", 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4535000000000002e-05 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:158030 7.f 7:f91d6d13:::benchmark_data_smithi159_149998_object62030:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:48.042375+0000", 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.6385413619999998, 2024-04-15T18:39:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26175157399999999, 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158030 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042375+0000", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042375+0000", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042376+0000", 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4499999999999996e-07 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042381+0000", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5229999999999997e-06 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042399+0000", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7861999999999999e-05 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042407+0000", 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0669999999999992e-06 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.047608+0000", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052013679999999996 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.047734+0000", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012577000000000001 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.047856+0000", 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000121975 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.059540+0000", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011684445999999999 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.059589+0000", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9088000000000002e-05 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304056+0000", 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.244466717 2024-04-15T18:39:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304096+0000", 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9672999999999998e-05 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304127+0000", 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1340000000000001e-05 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:158032 7.f 7:f6dd5e24:::benchmark_data_smithi159_149998_object62032:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:48.042569+0000", 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.6383477759999998, 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.261732572, 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158032 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042569+0000", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042569+0000", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042571+0000", 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2989999999999999e-06 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042577+0000", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2379999999999997e-06 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042589+0000", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2203e-05 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.042597+0000", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8560000000000007e-06 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.051460+0000", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088633370000000006 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.051579+0000", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118904 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.051653+0000", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3725000000000003e-05 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.062964+0000", 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011311811 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.063318+0000", 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035348999999999998 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304183+0000", 2024-04-15T18:39:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24086470500000001 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304242+0000", 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8854000000000003e-05 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304302+0000", 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0149999999999998e-05 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:158039 7.f 7:f915d6d5:::benchmark_data_smithi159_149998_object62039:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:48.053352+0000", 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.6275648900000004, 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25110414599999997, 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158039 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053352+0000", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053352+0000", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053354+0000", 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6130000000000001e-06 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053370+0000", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6668e-05 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053384+0000", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3328e-05 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.053391+0000", 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7600000000000002e-06 2024-04-15T18:39:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.055587+0000", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021960539999999998 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.055710+0000", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012286799999999999 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.055798+0000", 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7663000000000004e-05 2024-04-15T18:39:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.062973+0000", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071748530000000001 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.063475+0000", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050183900000000002 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304354+0000", 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24087969000000001 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304417+0000", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2438000000000001e-05 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304456+0000", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9372000000000002e-05 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:158041 7.f 7:ff8a3680:::benchmark_data_smithi159_149998_object62041:head [delete] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:48.056379+0000", 2024-04-15T18:39:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.6245381029999999, 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24823803699999999, 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158041 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056379+0000", 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056379+0000", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056380+0000", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.206e-06 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056385+0000", 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.746e-06 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056397+0000", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2182e-05 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.056405+0000", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6520000000000006e-06 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.059615+0000", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032107870000000001 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:39:55.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.059717+0000", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102052 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,9", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.059796+0000", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8467000000000006e-05 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.063095+0000", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032993739999999999 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.128746+0000", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.065650820999999998 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304511+0000", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17576450800000001 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304572+0000", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1118000000000007e-05 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:48.304617+0000", 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5123999999999997e-05 2024-04-15T18:39:55.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:39:55.782 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:39:55.782 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:39:55.782 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:55.810 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:55.834 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:55.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:55.941 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:39:55.987 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:56.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:39:56.451 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:39:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:56.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:39:56.556 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:56.593 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:56.626 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:56.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:56.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:39:56.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:56.779 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:39:56.810 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:56.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:56 smithi159 ceph-mon[45336]: pgmap v1774: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:39:56.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:39:56.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:39:56.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:39:56.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:39:56.941 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:56.946 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:39:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:56.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:56.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:56 smithi049 ceph-mon[40087]: pgmap v1774: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:39:57.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:39:57.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:56 smithi082 ceph-mon[44206]: pgmap v1774: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s 2024-04-15T18:39:57.032 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:39:57.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:39:57.067 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:39:57.098 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:57.126 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:39:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:57.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:39:57.131 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:39:57.171 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:39:57.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:39:57.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:39:57.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:39:57.295 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:39:57.295 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:39:57.295 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:39:57.295 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:39:57.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:39:57.307 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:57.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1962877682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:57.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.644 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1962877682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:57.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:57.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1962877682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.389 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:39:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:58.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:39:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:39:58.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:39:58.550 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:58.662 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: pgmap v1775: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:39:58.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:39:58.831 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:39:58.831 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:39:58.854 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.854 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.854 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.854 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.855 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.855 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:58.855 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: pgmap v1775: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:39:58.855 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: pgmap v1775: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s 2024-04-15T18:39:59.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:39:59.122 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:39:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:59.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:39:59.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:39:59.260 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:59.280 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:39:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:59.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:39:59.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:39:59.453 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:39:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:59.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:39:59.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:39:59.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:39:59 smithi049 ceph-mon[40087]: osdmap e682: 12 total, 12 up, 12 in 2024-04-15T18:39:59.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 9 1 2024-04-15T18:39:59.793 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:39:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:39:59.876 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:39:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:39:59.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:39:59.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:40:00.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:39:59 smithi082 ceph-mon[44206]: osdmap e682: 12 total, 12 up, 12 in 2024-04-15T18:40:00.060 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:00.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1217436017' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:40:00.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:39:59 smithi159 ceph-mon[45336]: osdmap e682: 12 total, 12 up, 12 in 2024-04-15T18:40:00.183 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:00.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:00.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:40:00.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:40:00.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:00.322 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:00.344 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:00.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:00.452 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:00.485 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:00.524 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:00.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:00.628 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:00.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:00.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:00 smithi049 ceph-mon[40087]: pgmap v1777: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:00.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:00.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:00.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:00 smithi049 ceph-mon[40087]: osdmap e683: 12 total, 12 up, 12 in 2024-04-15T18:40:00.665 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:00 smithi049 ceph-mon[40087]: overall HEALTH_WARN noscrub flag(s) set 2024-04-15T18:40:00.695 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:00 smithi159 ceph-mon[45336]: pgmap v1777: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:00 smithi159 ceph-mon[45336]: osdmap e683: 12 total, 12 up, 12 in 2024-04-15T18:40:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:00 smithi159 ceph-mon[45336]: overall HEALTH_WARN noscrub flag(s) set 2024-04-15T18:40:00.908 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:00.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:01.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:00 smithi082 ceph-mon[44206]: pgmap v1777: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:00 smithi082 ceph-mon[44206]: osdmap e683: 12 total, 12 up, 12 in 2024-04-15T18:40:01.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:00 smithi082 ceph-mon[44206]: overall HEALTH_WARN noscrub flag(s) set 2024-04-15T18:40:01.066 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:01.193 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:01.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:01.204 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:01.273 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:01.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:01.381 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:40:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:01.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:01.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:01.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:01.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:01.955 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:01.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:01.977 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:01.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:02.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:02.111 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:02.132 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:02.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:02.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:02.292 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:02.541 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:02.541 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:02.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:02 smithi159 ceph-mon[45336]: pgmap v1779: 17 pgs: 17 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s 2024-04-15T18:40:02.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:02 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:02.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:02.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:02 smithi159 ceph-mon[45336]: osdmap e684: 12 total, 12 up, 12 in 2024-04-15T18:40:02.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:02 smithi159 ceph-mon[45336]: osdmap e685: 12 total, 12 up, 12 in 2024-04-15T18:40:02.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:02 smithi049 ceph-mon[40087]: pgmap v1779: 17 pgs: 17 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s 2024-04-15T18:40:02.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:02 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:02.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:02.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:02 smithi049 ceph-mon[40087]: osdmap e684: 12 total, 12 up, 12 in 2024-04-15T18:40:02.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:02 smithi049 ceph-mon[40087]: osdmap e685: 12 total, 12 up, 12 in 2024-04-15T18:40:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:02 smithi082 ceph-mon[44206]: pgmap v1779: 17 pgs: 17 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s 2024-04-15T18:40:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:02 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1827780520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:02 smithi082 ceph-mon[44206]: osdmap e684: 12 total, 12 up, 12 in 2024-04-15T18:40:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:02 smithi082 ceph-mon[44206]: osdmap e685: 12 total, 12 up, 12 in 2024-04-15T18:40:03.554 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:03.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:03.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:03.708 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:03.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:03.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:40:03.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:03.926 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:04.246 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:04.757 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:04 smithi159 ceph-mon[45336]: pgmap v1782: 17 pgs: 17 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s 2024-04-15T18:40:04.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:04 smithi049 ceph-mon[40087]: pgmap v1782: 17 pgs: 17 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s 2024-04-15T18:40:04.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:40:05.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:04 smithi082 ceph-mon[44206]: pgmap v1782: 17 pgs: 17 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s 2024-04-15T18:40:05.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:05.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:05.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:05.275 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:05.309 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:05.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:05.413 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:05.460 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:05.912 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:05.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:06.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:06.084 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:06.518 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:06.519 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:06.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:06 smithi159 ceph-mon[45336]: pgmap v1783: 17 pgs: 17 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-04-15T18:40:06.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:06 smithi049 ceph-mon[40087]: pgmap v1783: 17 pgs: 17 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-04-15T18:40:07.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:06 smithi082 ceph-mon[44206]: pgmap v1783: 17 pgs: 17 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-04-15T18:40:07.058 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:07.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:07.164 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:07.175 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:07.195 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:07.218 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:40:07.220 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:07.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:07.244 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:40:07.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:40:07.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:07.336 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:40:07.355 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:07.376 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:07.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:07.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:40:07.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:40:07.480 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:07.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:40:07.510 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:07.540 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:40:07.543 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:07.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:07.585 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:40:07.615 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:07.623 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:40:07.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:07.667 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:40:07.703 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:07.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:40:07.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:40:07.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:40:07.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:40:07.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:40:07.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:40:07.752 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:40:07.753 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:40:07.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3393230920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:07.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:07 smithi159 ceph-mon[45336]: osdmap e686: 12 total, 12 up, 12 in 2024-04-15T18:40:07.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1273493475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:07.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:07 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:07.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3393230920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:07.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:07 smithi049 ceph-mon[40087]: osdmap e686: 12 total, 12 up, 12 in 2024-04-15T18:40:07.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1273493475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:07 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:08.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3393230920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:08.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:07 smithi082 ceph-mon[44206]: osdmap e686: 12 total, 12 up, 12 in 2024-04-15T18:40:08.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1273493475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:08.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:07 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:40:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:08.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:08.215 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:08.243 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:08.265 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:08.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:08.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:08.416 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:08.575 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-04-15T18:40:08.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: pgmap v1785: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:40:08.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:08 smithi159 ceph-mon[45336]: osdmap e687: 12 total, 12 up, 12 in 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:08.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: pgmap v1785: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:40:08.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:08 smithi049 ceph-mon[40087]: osdmap e687: 12 total, 12 up, 12 in 2024-04-15T18:40:08.979 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:08.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: pgmap v1785: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:40:09.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:08 smithi082 ceph-mon[44206]: osdmap e687: 12 total, 12 up, 12 in 2024-04-15T18:40:09.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:09.151 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:09.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 10 1 2024-04-15T18:40:09.816 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:09.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:10.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:10.203 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:173932 7.e e688/621 7:701eddc2:::benchmark_data_smithi159_149998_object77932:head v 688'10789, mlcod=688'10786)", 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:40:10.197856+0000", 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.016267996999999999, 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016290841, 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:10.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197856+0000", 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197856+0000", 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197856+0000", 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-04-15T18:40:10.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197864+0000", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9449999999999993e-06 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197900+0000", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5709e-05 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.197907+0000", 2024-04-15T18:40:10.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3989999999999998e-06 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.200577+0000", 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002669584 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.200596+0000", 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9199e-05 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:173933 7.e e688/621 7:74a5da31:::benchmark_data_smithi159_149998_object77933:head v 688'10790, mlcod=688'10786)", 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:40:10.201740+0000", 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.012383471, 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012466193, 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:10.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201740+0000", 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201740+0000", 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201741+0000", 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201751+0000", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8349999999999999e-06 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201785+0000", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4227999999999998e-05 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.201793+0000", 2024-04-15T18:40:10.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8310000000000001e-06 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.204842+0000", 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030485360000000001 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.204861+0000", 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9556999999999999e-05 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:173934 7.e e688/621 7:7701a19d:::benchmark_data_smithi159_149998_object77934:head v 688'10791, mlcod=688'10786)", 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:40:10.205892+0000", 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0082316769999999997, 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0083847550000000007, 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205892+0000", 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205892+0000", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205893+0000", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.09e-07 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205901+0000", 2024-04-15T18:40:10.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4570000000000005e-06 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205930+0000", 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9155000000000001e-05 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.205936+0000", 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9610000000000002e-06 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:10.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.208834+0000", 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028980270000000001 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.208859+0000", 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4192000000000001e-05 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:173936 7.e e688/621 7:74116ed2:::benchmark_data_smithi159_149998_object77936:head v 688'10792, mlcod=688'10786)", 2024-04-15T18:40:10.215 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:40:10.212654+0000", 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0014700900000000001, 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016739610000000001, 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212654+0000", 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:10.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212654+0000", 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212654+0000", 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1400000000000001e-07 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212657+0000", 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2710000000000001e-06 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212715+0000", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8217999999999997e-05 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.212721+0000", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.716e-06 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.213221+0000", 2024-04-15T18:40:10.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049972999999999999 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:40:10.213241+0000", 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0681000000000001e-05 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 4 2024-04-15T18:40:10.219 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:40:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:10.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:10.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:10.688 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:10.722 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:10.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:10.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:10.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:10 smithi159 ceph-mon[45336]: pgmap v1787: 17 pgs: 17 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:40:10.884 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:10 smithi159 ceph-mon[45336]: osdmap e688: 12 total, 12 up, 12 in 2024-04-15T18:40:10.918 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:40:10.934 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:10.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:10.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:10.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:10 smithi049 ceph-mon[40087]: pgmap v1787: 17 pgs: 17 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:40:10.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:10 smithi049 ceph-mon[40087]: osdmap e688: 12 total, 12 up, 12 in 2024-04-15T18:40:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:10.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:11.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:40:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:10 smithi082 ceph-mon[44206]: pgmap v1787: 17 pgs: 17 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s 2024-04-15T18:40:11.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:10 smithi082 ceph-mon[44206]: osdmap e688: 12 total, 12 up, 12 in 2024-04-15T18:40:11.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:11.075 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:11.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:11.200 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:11.236 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:11.265 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:11.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:11.296 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:11.370 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:11.408 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:11.540 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:11.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:11.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:11.681 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:11.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:11.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:11.843 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:11.844 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:11.844 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:11 smithi159 ceph-mon[45336]: osdmap e689: 12 total, 12 up, 12 in 2024-04-15T18:40:11.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:11.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:11.942 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:11.943 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:11 smithi049 ceph-mon[40087]: osdmap e689: 12 total, 12 up, 12 in 2024-04-15T18:40:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:12.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:11 smithi082 ceph-mon[44206]: osdmap e689: 12 total, 12 up, 12 in 2024-04-15T18:40:12.565 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:40:12.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: pgmap v1790: 17 pgs: 17 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: osdmap e690: 12 total, 12 up, 12 in 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/792197818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: osdmap e691: 12 total, 12 up, 12 in 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:12.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:12.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: pgmap v1790: 17 pgs: 17 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: osdmap e690: 12 total, 12 up, 12 in 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/792197818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:13.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: osdmap e691: 12 total, 12 up, 12 in 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:13.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: pgmap v1790: 17 pgs: 17 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: osdmap e690: 12 total, 12 up, 12 in 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/792197818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:13.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:13.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: osdmap e691: 12 total, 12 up, 12 in 2024-04-15T18:40:13.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:13.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:13.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:13.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:13.146 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:13.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:13.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:13.293 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:13.574 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:13.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:13.690 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:13.726 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:13.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:13.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:14.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:14.103 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:14.104 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:14.129 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:14.278 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:14.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:14.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:14.413 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:14.435 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:14.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:40:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:40:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:40:14.459 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:40:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:40:14.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:14.587 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:14.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:14.773 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:14.774 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:14 smithi082 ceph-mon[44206]: pgmap v1793: 17 pgs: 17 active+clean; 1.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-04-15T18:40:14.820 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:14.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:14 smithi049 ceph-mon[40087]: pgmap v1793: 17 pgs: 17 active+clean; 1.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-04-15T18:40:15.109 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:14 smithi159 ceph-mon[45336]: pgmap v1793: 17 pgs: 17 active+clean; 1.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-04-15T18:40:15.149 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:40:15.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:15.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:15.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:40:15.674 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:15.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:15.763 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:15.819 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-04-15T18:40:15.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1142191451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:15.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:15.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1142191451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:15.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1142191451' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: pgmap v1794: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: osdmap e692: 12 total, 12 up, 12 in 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:16.872 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:16 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:16.947 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:16.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:17.059 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:17.108 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:17.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: pgmap v1794: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s 2024-04-15T18:40:17.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:40:17.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: osdmap e692: 12 total, 12 up, 12 in 2024-04-15T18:40:17.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:17.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:17.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:16 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.212 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:17.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:17.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: pgmap v1794: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 702 op/s 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: osdmap e692: 12 total, 12 up, 12 in 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:16 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:40:17.318 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:17.357 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:17.386 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:17.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:17.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:17.553 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:141747 6.f e595/541 6:fa5b3ac3:::benchmark_data_smithi159_139342_object51868:head v 595'8933, mlcod=595'8930)", 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 294.49620418400002, 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.256791241, 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.566 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068247+0000", 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068248+0000", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1030000000000001e-06 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.068258+0000", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.679e-06 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321692+0000", 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25343376699999998 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321699+0000", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7170000000000002e-06 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321769+0000", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0734000000000004e-05 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.321790+0000", 2024-04-15T18:40:17.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0996999999999999e-05 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.325000+0000", 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032095650000000002 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:23.325039+0000", 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8679e-05 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:179749 6.d e611/541 6:bfc4ccdc:::benchmark_data_smithi159_139342_object89870:head v 611'11403, mlcod=611'11399)", 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:21.368887+0000", 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 236.19556443600001, 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75125385200000006, 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.368887+0000", 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.368887+0000", 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.368889+0000", 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4389999999999999e-06 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.368896+0000", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7849999999999992e-06 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.116487+0000", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74759027700000003 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.116493+0000", 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6000000000000003e-06 2024-04-15T18:40:17.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.116559+0000", 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5877000000000006e-05 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.116586+0000", 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7087e-05 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.120102+0000", 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035158009999999998 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.120141+0000", 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8986000000000002e-05 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:21021 7.1 e627/621 7:887b7031:::benchmark_data_smithi159_149998_object21020:head v 627'1309, mlcod=627'1306)", 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:03.235135+0000", 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 194.329316589, 2024-04-15T18:40:17.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21029441900000001, 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.235135+0000", 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.235135+0000", 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.235136+0000", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7999999999999995e-07 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.235212+0000", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6223999999999994e-05 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.444234+0000", 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209022284 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.444243+0000", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8400000000000001e-06 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.444305+0000", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2490999999999995e-05 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.444329+0000", 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3195999999999999e-05 2024-04-15T18:40:17.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.445383+0000", 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010538990000000001 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:03.445429+0000", 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6805000000000003e-05 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.579 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:22376 7.1 e627/621 7:87ef7b77:::benchmark_data_smithi159_149998_object22375:head v 627'1400, mlcod=627'1397)", 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:04.484236+0000", 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 193.080215522, 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21327723000000001, 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.484236+0000", 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.484236+0000", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.484237+0000", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.049e-06 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.484314+0000", 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6668999999999994e-05 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.696395+0000", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21208086300000001 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.696402+0000", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.99e-06 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.696463+0000", 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1760000000000005e-05 2024-04-15T18:40:17.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.696485+0000", 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2116e-05 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.697474+0000", 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009880519999999999 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:04.697513+0000", 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9731000000000003e-05 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:24110 7.2 7:444ea4df:::benchmark_data_smithi159_149998_object24109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e628)", 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:06.060594+0000", 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 191.50385773799999, 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21164812699999999, 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.584 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 24110 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060594+0000", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060594+0000", 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060595+0000", 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0449999999999999e-06 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060615+0000", 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0508000000000001e-05 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060627+0000", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1555e-05 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060633+0000", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0610000000000001e-06 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060680+0000", 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6882999999999998e-05 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060772+0000", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1918999999999999e-05 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.060860+0000", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7835000000000002e-05 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.061632+0000", 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00077220800000000003 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.267213+0000", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20558168199999999 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272128+0000", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049143549999999996 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272194+0000", 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5976000000000007e-05 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272242+0000", 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8099999999999997e-05 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:24110 7.2 e628/621 ondisk, result = 0)", 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:06.061959+0000", 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 191.50249269899999, 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.210298761, 2024-04-15T18:40:17.590 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.061959+0000", 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.061959+0000", 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.061959+0000", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3900000000000004e-07 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.061963+0000", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7079999999999999e-06 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272056+0000", 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21009233599999999 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272061+0000", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9440000000000002e-06 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272111+0000", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9843999999999999e-05 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272119+0000", 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0949999999999996e-06 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:06.272258+0000", 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000138195 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:31756 7.2 7:45054463:::benchmark_data_smithi159_149998_object31755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.003935+0000", 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 186.560516675, 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35040733699999999, 2024-04-15T18:40:17.594 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 31756 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003935+0000", 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003935+0000", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003937+0000", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8390000000000001e-06 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003959+0000", 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2631000000000001e-05 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003975+0000", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5789000000000001e-05 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.003984+0000", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6000000000000007e-06 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.004033+0000", 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8813999999999997e-05 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.004136+0000", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103161 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.004226+0000", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0078000000000001e-05 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.005376+0000", 2024-04-15T18:40:17.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001150723 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.006083+0000", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070607199999999997 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.354263+0000", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34818064799999998 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.354313+0000", 2024-04-15T18:40:17.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9591e-05 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.354342+0000", 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9391e-05 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:31795 7.1 e631/621 7:81918b20:::benchmark_data_smithi159_149998_object31794:head v 631'1993, mlcod=631'1989)", 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.051698+0000", 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 186.512753409, 2024-04-15T18:40:17.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29486970000000001, 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.051698+0000", 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.051698+0000", 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.051699+0000", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0189999999999999e-06 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.051773+0000", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3586999999999999e-05 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.345375+0000", 2024-04-15T18:40:17.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29360263399999997 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.345403+0000", 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7379e-05 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.345464+0000", 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1555000000000003e-05 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.345494+0000", 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9781000000000002e-05 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.346532+0000", 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001037811 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.346568+0000", 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5933999999999998e-05 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.604 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:32145 7.2 7:425e92d5:::benchmark_data_smithi159_149998_object32144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.600537+0000", 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 185.96391462700001, 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.15707522600000001, 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 32145 2024-04-15T18:40:17.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600537+0000", 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600537+0000", 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600539+0000", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6199999999999999e-06 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600561+0000", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2639000000000001e-05 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600576+0000", 2024-04-15T18:40:17.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5099999999999999e-05 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600584+0000", 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5140000000000004e-06 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600635+0000", 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0917000000000001e-05 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600738+0000", 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010373400000000001 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.600830+0000", 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1990000000000005e-05 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.601973+0000", 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011430120000000001 2024-04-15T18:40:17.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.606667+0000", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046933260000000003 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.757544+0000", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.15087745899999999 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.757583+0000", 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9076999999999999e-05 2024-04-15T18:40:17.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.757612+0000", 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8838000000000001e-05 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:55432 7.e e639/621 7:70f993cc:::benchmark_data_smithi159_149998_object55431:head v 639'3437, mlcod=639'3435)", 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:31.227208+0000", 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 166.33724386399999, 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16209928600000001, 2024-04-15T18:40:17.611 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.227208+0000", 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.227208+0000", 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.227212+0000", 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4059999999999998e-06 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.227285+0000", 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2669000000000005e-05 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.613 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.387725+0000", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16044059299999999 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.387733+0000", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8769999999999993e-06 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.387785+0000", 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1888000000000002e-05 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.614 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.387807+0000", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2325999999999999e-05 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.389272+0000", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0014644110000000001 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:31.389307+0000", 2024-04-15T18:40:17.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5116000000000001e-05 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:81711 7.2 7:476a6143:::benchmark_data_smithi159_149998_object81710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:59.821543+0000", 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 137.74290846599999, 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.175008776, 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.616 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 81711 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821543+0000", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821543+0000", 2024-04-15T18:40:17.617 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821544+0000", 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7899999999999997e-07 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821563+0000", 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9244000000000001e-05 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821573+0000", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8549999999999997e-06 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821579+0000", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6699999999999999e-06 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821620+0000", 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0890000000000003e-05 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821758+0000", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000138341 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.821881+0000", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012318 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.823638+0000", 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001757254 2024-04-15T18:40:17.620 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.825171+0000", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015327450000000001 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.996444+0000", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.171272386 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.996506+0000", 2024-04-15T18:40:17.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2681999999999994e-05 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:59.996552+0000", 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5649999999999998e-05 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:121667 7.2 7:402e22ac:::benchmark_data_smithi159_149998_object25667:head [delete] snapc 0=[] ondisk+write+known_if_redirected e662)", 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:56.136198+0000", 2024-04-15T18:40:17.622 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 81.428253984999998, 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17157430700000001, 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 121667 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.623 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136198+0000", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136198+0000", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136199+0000", 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9399999999999993e-07 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136203+0000", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6879999999999998e-06 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136215+0000", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1273e-05 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136220+0000", 2024-04-15T18:40:17.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7579999999999998e-06 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136265+0000", 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4935000000000003e-05 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136399+0000", 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013418300000000001 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.136541+0000", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014161499999999999 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.139505+0000", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029640679999999998 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.140188+0000", 2024-04-15T18:40:17.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068326599999999997 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.307587+0000", 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.167398505 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.307721+0000", 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013458200000000001 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.628 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:56.307772+0000", 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0439999999999998e-05 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:123988 7.2 7:475be767:::benchmark_data_smithi159_149998_object27988:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:59.260735+0000", 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.303716382000005, 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20781688700000001, 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.629 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 123988 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260735+0000", 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.630 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260735+0000", 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260736+0000", 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.24e-06 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260742+0000", 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6459999999999998e-06 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260754+0000", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2075999999999999e-05 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260761+0000", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0330000000000003e-06 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260807+0000", 2024-04-15T18:40:17.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5361999999999997e-05 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.260896+0000", 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9299e-05 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.261003+0000", 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010678100000000001 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.263943+0000", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029404779999999998 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.264396+0000", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00045246700000000002 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.468487+0000", 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.204091367 2024-04-15T18:40:17.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.468522+0000", 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4820000000000002e-05 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:59.468552+0000", 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0318e-05 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:143340 7.2 7:4c09de55:::benchmark_data_smithi159_149998_object47340:head [delete] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 50.806582648000003, 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.343721899, 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 143340 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.636 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757871+0000", 2024-04-15T18:40:17.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.689e-06 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757878+0000", 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2830000000000004e-06 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757893+0000", 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5038000000000001e-05 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757901+0000", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6909999999999997e-06 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757952+0000", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.758042+0000", 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0453999999999998e-05 2024-04-15T18:40:17.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.758114+0000", 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2093000000000002e-05 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.761392+0000", 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032774060000000001 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763286+0000", 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018943740000000001 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101516+0000", 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33822963499999997 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101556+0000", 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0268000000000001e-05 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101591+0000", 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4824999999999999e-05 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:143340 7.2 e672/621 ondisk, result = 0)", 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 50.801347200999999, 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33849178099999999, 2024-04-15T18:40:17.642 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763105+0000", 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7700000000000005e-07 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763108+0000", 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.506e-06 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101426+0000", 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33831763300000001 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101431+0000", 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2469999999999999e-06 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101503+0000", 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1979999999999999e-05 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101511+0000", 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7770000000000002e-06 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101596+0000", 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5161000000000007e-05 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:145668 7.2 7:487c5f29:::benchmark_data_smithi159_149998_object49668:head [delete] snapc 0=[] ondisk+write+known_if_redirected e675)", 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:30.437012+0000", 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.127439993000003, 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.174589467, 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 145668 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437012+0000", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437012+0000", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437013+0000", 2024-04-15T18:40:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1930000000000001e-06 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437018+0000", 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7230000000000001e-06 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437030+0000", 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1463e-05 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437037+0000", 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7329999999999997e-06 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437087+0000", 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0583e-05 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437169+0000", 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1317999999999996e-05 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.437231+0000", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2681000000000005e-05 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.441256+0000", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040245400000000001 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.445102+0000", 2024-04-15T18:40:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038461680000000001 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.611535+0000", 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.166432725 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.611571+0000", 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6087999999999997e-05 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:30.611601+0000", 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0252000000000001e-05 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "replica_scrub(pg: 7.1,from:0'0,to:675'9066,epoch:676/621,start:7:8923feff:::benchmark_data_smithi159_149998_object80750:0,end:7:89aa5995:::benchmark_data_smithi159_149998_object63737:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.818205503999998, 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.40023066699999998, 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746247+0000", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8100000000000003e-07 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746253+0000", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5989999999999998e-06 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146191+0000", 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39993743199999998 2024-04-15T18:40:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146210+0000", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9602999999999998e-05 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146395+0000", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018475300000000001 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146422+0000", 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7135999999999999e-05 2024-04-15T18:40:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146477+0000", 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4562999999999999e-05 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:146597 7.2 7:40f54c50:::benchmark_data_smithi159_149998_object50597:head [delete] snapc 0=[] ondisk+write+known_if_redirected e676)", 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.796584093999996, 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33914392500000001, 2024-04-15T18:40:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 146597 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767869+0000", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.637e-06 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767875+0000", 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9209999999999997e-06 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767889+0000", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3542e-05 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767895+0000", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9500000000000004e-06 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767943+0000", 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7043999999999999e-05 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.768031+0000", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8159999999999996e-05 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.768102+0000", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1759000000000002e-05 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.771302+0000", 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031990529999999999 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.772670+0000", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013685870000000001 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.106933+0000", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33426301200000003 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.106975+0000", 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1527999999999999e-05 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.107011+0000", 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6732e-05 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:162650 7.2 7:49129aad:::benchmark_data_smithi159_149998_object66650:head [delete] snapc 0=[] ondisk+write+known_if_redirected e681)", 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:54.426382+0000", 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 23.138069714, 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16104043900000001, 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 162650 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426382+0000", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426382+0000", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426383+0000", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4389999999999999e-06 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426389+0000", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8019999999999997e-06 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426402+0000", 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3288999999999999e-05 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.426411+0000", 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3939999999999996e-06 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.429599+0000", 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031884750000000001 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.429684+0000", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5198000000000007e-05 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.429759+0000", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5005000000000004e-05 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.433652+0000", 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038929770000000002 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.437068+0000", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034159329999999999 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.587343+0000", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.150274502 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.587392+0000", 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9302000000000003e-05 2024-04-15T18:40:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.587422+0000", 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0122999999999999e-05 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:163036 7.1 e681/621 7:836073c8:::benchmark_data_smithi159_149998_object67036:head v 681'10123, mlcod=681'10120)", 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:54.962861+0000", 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.60159032, 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16561689299999999, 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:40:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.962861+0000", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.962861+0000", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:40:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.962862+0000", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:54.962871+0000", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2250000000000006e-06 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.124609+0000", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.161738033 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:40:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.124616+0000", 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6270000000000002e-06 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.124668+0000", 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1768000000000003e-05 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.124691+0000", 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2982000000000001e-05 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.128429+0000", 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037386540000000001 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:55.128478+0000", 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8723000000000001e-05 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:40:17.673 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:40:17.773 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:17.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:17.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 11 1 2024-04-15T18:40:17.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:17.964 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:18.107 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:18.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:18.159 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:18.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:18.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:40:18.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:40:18.266 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:18.305 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:40:18.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:40:18.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:40:18.433 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:40:18.436 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:18.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:18.466 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:40:18.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:40:18.540 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:40:18.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:18.571 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:18.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:40:18.597 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:18.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:18.618 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:18.629 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:40:18.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:40:18.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:18 smithi049 ceph-mon[40087]: osdmap e693: 12 total, 12 up, 12 in 2024-04-15T18:40:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:18 smithi049 ceph-mon[40087]: pgmap v1796: 17 pgs: 17 active+clean; 929 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-04-15T18:40:18.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1332869111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:18.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:18.722 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:40:18.733 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:40:18.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:40:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:40:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:40:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:40:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:40:18.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:40:18.887 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:18.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:18.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:18 smithi159 ceph-mon[45336]: osdmap e693: 12 total, 12 up, 12 in 2024-04-15T18:40:18.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:18 smithi159 ceph-mon[45336]: pgmap v1796: 17 pgs: 17 active+clean; 929 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-04-15T18:40:18.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1332869111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:18.891 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:18.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:19.015 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:18 smithi082 ceph-mon[44206]: osdmap e693: 12 total, 12 up, 12 in 2024-04-15T18:40:19.015 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:18 smithi082 ceph-mon[44206]: pgmap v1796: 17 pgs: 17 active+clean; 929 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s 2024-04-15T18:40:19.015 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1332869111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:19.015 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.017 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:19.041 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:19.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:19.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:19.175 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:19.197 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:19.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:19.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:19.346 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:19.449 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:19.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:19.554 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:19.574 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.575 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.594 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.955 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:19.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:19.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:19.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:20.063 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:20.104 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:20.240 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:20.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:20.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:20.384 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:20.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:20.512 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:20.527 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:20.527 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:20.542 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:20.563 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:20.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:20.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:20.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:20.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:20 smithi049 ceph-mon[40087]: pgmap v1798: 17 pgs: 17 active+clean; 850 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-04-15T18:40:20.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:20 smithi049 ceph-mon[40087]: osdmap e694: 12 total, 12 up, 12 in 2024-04-15T18:40:20.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:20.697 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:20.718 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:20.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:20.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:20.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:20.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:20.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:20 smithi159 ceph-mon[45336]: pgmap v1798: 17 pgs: 17 active+clean; 850 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-04-15T18:40:20.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:20 smithi159 ceph-mon[45336]: osdmap e694: 12 total, 12 up, 12 in 2024-04-15T18:40:20.883 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:20.980 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:20.985 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:20.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:20 smithi082 ceph-mon[44206]: pgmap v1798: 17 pgs: 17 active+clean; 850 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 649 op/s 2024-04-15T18:40:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:20 smithi082 ceph-mon[44206]: osdmap e694: 12 total, 12 up, 12 in 2024-04-15T18:40:21.092 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:21.120 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:21.142 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:21.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:21.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:40:21.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:21.275 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:21.297 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:21.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:21.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:21.441 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:21.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:21.576 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:21.599 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:40:21.832 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:21.900 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:21.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:22.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:22.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:22.040 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:22.060 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:22.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:22.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:22.204 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:22.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: osdmap e695: 12 total, 12 up, 12 in 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:22.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:22.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:22 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: osdmap e695: 12 total, 12 up, 12 in 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:22.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:22.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:22.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:22.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:22.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:22 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1251144921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: osdmap e695: 12 total, 12 up, 12 in 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:40:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:40:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:40:23.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:22 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:40:23.063 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:23.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:23.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:23.195 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:23.217 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:23.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:23.321 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:23.354 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:23.381 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:23.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:23.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:23.545 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:23.823 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:23.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:23.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:23.959 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:23.984 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:23.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:24.088 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:24.122 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:24.162 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:24.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:24.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:24.323 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:24.347 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:24.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:24.377 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:24.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:24.497 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:24.497 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:24.513 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:24.546 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:24.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:24.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:24.679 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:24.700 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:24.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:24.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:24.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:24 smithi049 ceph-mon[40087]: pgmap v1802: 17 pgs: 17 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s 2024-04-15T18:40:24.844 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:24.855 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:24 smithi159 ceph-mon[45336]: pgmap v1802: 17 pgs: 17 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s 2024-04-15T18:40:24.953 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:24.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:25.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:24 smithi082 ceph-mon[44206]: pgmap v1802: 17 pgs: 17 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s 2024-04-15T18:40:25.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:25.110 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:25.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:40:25.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:40:25.587 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:40:25.896 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:25.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:26.010 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:26.039 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.063 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:26.168 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:26.196 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.218 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:26.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:26.349 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.370 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:26.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:26.506 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.526 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:26.550 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:26.615 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-04-15T18:40:26.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:26.659 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.682 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:26.789 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:26.824 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:26.850 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:26.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:26.858 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:26 smithi159 ceph-mon[45336]: pgmap v1803: 17 pgs: 17 active+clean; 594 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s 2024-04-15T18:40:26.858 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3448999978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:26.858 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:26 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:26.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:26.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:26 smithi049 ceph-mon[40087]: pgmap v1803: 17 pgs: 17 active+clean; 594 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s 2024-04-15T18:40:26.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3448999978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:26.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:26 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:26.989 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:27.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:26 smithi082 ceph-mon[44206]: pgmap v1803: 17 pgs: 17 active+clean; 594 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s 2024-04-15T18:40:27.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3448999978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:27.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:26 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:40:27.341 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:27.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:27.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:27.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 3 1 2024-04-15T18:40:27.511 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:27.573 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:27.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:27.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:27.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:27 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:40:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:27 smithi049 ceph-mon[40087]: osdmap e696: 12 total, 12 up, 12 in 2024-04-15T18:40:27.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:27 smithi049 ceph-mon[40087]: osdmap e697: 12 total, 12 up, 12 in 2024-04-15T18:40:27.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:27.841 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:27.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:27.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:27 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:40:27.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:27 smithi159 ceph-mon[45336]: osdmap e696: 12 total, 12 up, 12 in 2024-04-15T18:40:27.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:27 smithi159 ceph-mon[45336]: osdmap e697: 12 total, 12 up, 12 in 2024-04-15T18:40:27.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:27.964 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:27.964 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:27 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:40:28.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:27 smithi082 ceph-mon[44206]: osdmap e696: 12 total, 12 up, 12 in 2024-04-15T18:40:28.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:27 smithi082 ceph-mon[44206]: osdmap e697: 12 total, 12 up, 12 in 2024-04-15T18:40:28.034 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:28.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:28.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:28.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:40:28.408 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:28.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:40:28.501 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:40:28.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:40:28.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:40:28.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:40:28.631 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:28.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:28.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:40:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:28 smithi049 ceph-mon[40087]: pgmap v1805: 17 pgs: 17 active+clean; 497 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s 2024-04-15T18:40:28.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2984275502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:28.715 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:40:28.740 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:28.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:40:28.781 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:28.998 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:40:29.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:28 smithi082 ceph-mon[44206]: pgmap v1805: 17 pgs: 17 active+clean; 497 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s 2024-04-15T18:40:29.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2984275502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:29.044 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:29.046 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:28 smithi159 ceph-mon[45336]: pgmap v1805: 17 pgs: 17 active+clean; 497 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s 2024-04-15T18:40:29.046 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2984275502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:29.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:29.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:40:29.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:40:29.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:40:29.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:40:29.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:29.210 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:29.421 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:29.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:29.532 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:29.568 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:29.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:29.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:29.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:29 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.761 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:29.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:29 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:29 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.121 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:30.159 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.181 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:30.286 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:30.313 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.333 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:30.364 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:30.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:30.472 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.496 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:30.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:30.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:30.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:30.785 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.825 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:30.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:30 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.863 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:30 smithi049 ceph-mon[40087]: pgmap v1807: 17 pgs: 17 active+clean; 416 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-04-15T18:40:30.929 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:30.959 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.959 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.959 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:30 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:30.959 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:30 smithi082 ceph-mon[44206]: pgmap v1807: 17 pgs: 17 active+clean; 416 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-04-15T18:40:30.961 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:30.982 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:30.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:31.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:31.135 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:30 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:30 smithi159 ceph-mon[45336]: pgmap v1807: 17 pgs: 17 active+clean; 416 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-04-15T18:40:31.709 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:31.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:31.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:31.869 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:31.943 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:31.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:31.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:31.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:32.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:32.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:32.060 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:32.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:32.086 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:32.089 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:32.125 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:32.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:32.232 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:32.261 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:32.261 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:32.285 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:32.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:32.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:32.439 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:32.554 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:32.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:32.590 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:40:32.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:32.687 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:32.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: pgmap v1808: 17 pgs: 17 active+clean; 320 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s 2024-04-15T18:40:32.691 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: osdmap e698: 12 total, 12 up, 12 in 2024-04-15T18:40:32.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1430201104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:32.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:32.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:32.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:40:32.692 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:32 smithi049 ceph-mon[40087]: osdmap e699: 12 total, 12 up, 12 in 2024-04-15T18:40:32.710 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:32.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:32.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:32.844 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:32.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:32.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:32.975 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: pgmap v1808: 17 pgs: 17 active+clean; 320 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: osdmap e698: 12 total, 12 up, 12 in 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1430201104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:32.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:40:33.000 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:32 smithi082 ceph-mon[44206]: osdmap e699: 12 total, 12 up, 12 in 2024-04-15T18:40:33.001 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.024 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:33.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: pgmap v1808: 17 pgs: 17 active+clean; 320 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: osdmap e698: 12 total, 12 up, 12 in 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1430201104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:40:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:32 smithi159 ceph-mon[45336]: osdmap e699: 12 total, 12 up, 12 in 2024-04-15T18:40:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:40:33.310 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:33.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:33.449 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.474 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:33.578 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:33.607 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.628 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:33.704 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:33.705 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:33.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:33.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:33.737 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:33.766 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.785 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:33.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:33.922 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:33.947 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:33.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:34.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:34.082 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:34.102 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:34.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:34.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:34.245 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:34.271 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:34.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:34.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:34.408 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:34.434 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:34.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:34.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:34.584 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:34.704 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:34 smithi082 ceph-mon[44206]: pgmap v1811: 17 pgs: 17 active+clean; 195 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-04-15T18:40:34.704 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:34 smithi082 ceph-mon[44206]: osdmap e700: 12 total, 12 up, 12 in 2024-04-15T18:40:34.705 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:34 smithi159 ceph-mon[45336]: pgmap v1811: 17 pgs: 17 active+clean; 195 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-04-15T18:40:34.705 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:34 smithi159 ceph-mon[45336]: osdmap e700: 12 total, 12 up, 12 in 2024-04-15T18:40:34.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:34 smithi049 ceph-mon[40087]: pgmap v1811: 17 pgs: 17 active+clean; 195 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-04-15T18:40:34.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:34 smithi049 ceph-mon[40087]: osdmap e700: 12 total, 12 up, 12 in 2024-04-15T18:40:35.355 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:35.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:35.463 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:35.501 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:35.527 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:35.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:35.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:35.684 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:40:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:40:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:40:36.253 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:40:36.301 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 95997 objects 2024-04-15T18:40:36.301 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :135.517 2024-04-15T18:40:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:36.720 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-04-15T18:40:36.744 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:36.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:36.848 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:36.878 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:36.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:36.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:36 smithi049 ceph-mon[40087]: pgmap v1813: 17 pgs: 17 active+clean; 151 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:40:37.003 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:36 smithi082 ceph-mon[44206]: pgmap v1813: 17 pgs: 17 active+clean; 151 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:40:37.037 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.058 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:36 smithi159 ceph-mon[45336]: pgmap v1813: 17 pgs: 17 active+clean; 151 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-04-15T18:40:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:40:37.162 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:37.194 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.214 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:37.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:37.351 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.379 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:37.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:40:37.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:37.517 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.538 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:37.642 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:37.671 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.690 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:37.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:37.820 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:40:37.820 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:37 smithi159 ceph-mon[45336]: osdmap e701: 12 total, 12 up, 12 in 2024-04-15T18:40:37.822 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:37.840 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:37.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:37.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:37.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:40:37.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:37 smithi049 ceph-mon[40087]: osdmap e701: 12 total, 12 up, 12 in 2024-04-15T18:40:37.974 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3050761879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:40:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:37 smithi082 ceph-mon[44206]: osdmap e701: 12 total, 12 up, 12 in 2024-04-15T18:40:38.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:40:38.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:38 smithi049 ceph-mon[40087]: pgmap v1815: 17 pgs: 17 active+clean; 64 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:40:38.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:38.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:38 smithi082 ceph-mon[44206]: pgmap v1815: 17 pgs: 17 active+clean; 64 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:40:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:38 smithi159 ceph-mon[45336]: pgmap v1815: 17 pgs: 17 active+clean; 64 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s 2024-04-15T18:40:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:39.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 2 1 2024-04-15T18:40:39.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:39.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:39.596 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:39.628 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:39.649 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:39.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:39.753 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:39.782 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:39.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:39.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:39.834 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:39.874 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:39.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:39 smithi159 ceph-mon[45336]: osdmap e702: 12 total, 12 up, 12 in 2024-04-15T18:40:39.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:39.946 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:40:39.959 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:39.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:39.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:40:39.997 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:40:40.021 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:40:40.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:39 smithi082 ceph-mon[44206]: osdmap e702: 12 total, 12 up, 12 in 2024-04-15T18:40:40.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:40:40.064 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:40.070 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:40:40.092 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:40.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:39 smithi049 ceph-mon[40087]: osdmap e702: 12 total, 12 up, 12 in 2024-04-15T18:40:40.096 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:40:40.111 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:40.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:40.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:40:40.154 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:40:40.179 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:40:40.203 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:40:40.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:40.228 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:40:40.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:40:40.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:40:40.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:40:40.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:40:40.230 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:40:40.242 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:40.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:40.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:40.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:40.399 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:40.600 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:40.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:40.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:40.737 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:40.757 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:40.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:40.861 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:40.889 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:40.908 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:40.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:41.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: pgmap v1817: 17 pgs: 17 active+clean; 9.5 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/274617688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:40 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: pgmap v1817: 17 pgs: 17 active+clean; 9.5 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/274617688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:40 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.041 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: pgmap v1817: 17 pgs: 17 active+clean; 9.5 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/274617688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:40 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.550 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-04-15T18:40:41.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-04-15T18:40:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:41.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:41.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:41.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:41.750 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:41.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:41.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:41.886 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:41.908 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:41.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:41.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:42.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.041 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:42.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:42.432 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:42.694 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:42.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:42.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:42.827 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:42.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:42 smithi049 ceph-mon[40087]: pgmap v1818: 17 pgs: 17 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s 2024-04-15T18:40:43.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:42 smithi082 ceph-mon[44206]: pgmap v1818: 17 pgs: 17 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s 2024-04-15T18:40:43.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:42 smithi159 ceph-mon[45336]: pgmap v1818: 17 pgs: 17 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s 2024-04-15T18:40:43.211 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:43.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:40:44.749 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:40:44.750 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:40:45.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:44 smithi082 ceph-mon[44206]: pgmap v1819: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s 2024-04-15T18:40:45.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:44 smithi159 ceph-mon[45336]: pgmap v1819: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s 2024-04-15T18:40:45.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:44 smithi049 ceph-mon[40087]: pgmap v1819: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 300 op/s 2024-04-15T18:40:45.278 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:45.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:45.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:45.415 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:45.437 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:45.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:45.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:45.571 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:45.590 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:45.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:45.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:45.725 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:45.750 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:40:45.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:46.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:46.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:40:46.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:46 smithi049 ceph-mon[40087]: pgmap v1820: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s 2024-04-15T18:40:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:46 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:46.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:46 smithi049 ceph-mon[40087]: osdmap e703: 12 total, 12 up, 12 in 2024-04-15T18:40:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:46 smithi082 ceph-mon[44206]: pgmap v1820: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s 2024-04-15T18:40:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:46 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:47.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:46 smithi082 ceph-mon[44206]: osdmap e703: 12 total, 12 up, 12 in 2024-04-15T18:40:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:46 smithi159 ceph-mon[45336]: pgmap v1820: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 167 op/s 2024-04-15T18:40:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:46 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3121028103' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:40:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:46 smithi159 ceph-mon[45336]: osdmap e703: 12 total, 12 up, 12 in 2024-04-15T18:40:47.405 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:47.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:47.510 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:47.542 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:47.563 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:47.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:47.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:47.698 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:47.718 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:47.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:47.755 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-04-15T18:40:47.822 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:47.854 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:47.875 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:47.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:47.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:47.999 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:48.006 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:40:48.008 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:48.027 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:48.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:48.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:48.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:40:48.158 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:40:48.161 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:48.215 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:48.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:48.319 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:48.350 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:48.371 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:48.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:48.475 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:40:48.504 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:48.524 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:40:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:48.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:40:48.628 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:48.660 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:48.682 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:48.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:48.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:48.814 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:48 smithi082 ceph-mon[44206]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s 2024-04-15T18:40:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:40:49.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:48 smithi082 ceph-mon[44206]: osdmap e704: 12 total, 12 up, 12 in 2024-04-15T18:40:49.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:48 smithi159 ceph-mon[45336]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s 2024-04-15T18:40:49.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:40:49.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:48 smithi159 ceph-mon[45336]: osdmap e704: 12 total, 12 up, 12 in 2024-04-15T18:40:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:48 smithi049 ceph-mon[40087]: pgmap v1822: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 16 op/s 2024-04-15T18:40:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/997135655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:40:49.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:48 smithi049 ceph-mon[40087]: osdmap e704: 12 total, 12 up, 12 in 2024-04-15T18:40:49.762 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_5' removed 2024-04-15T18:40:49.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:40:50.014 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:40:50.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:40:50.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 6 1 2024-04-15T18:40:50.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:40:50.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:40:50.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:50.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:50.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:51.016 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:50 smithi049 ceph-mon[40087]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:51.016 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:40:51.016 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:50 smithi049 ceph-mon[40087]: osdmap e705: 12 total, 12 up, 12 in 2024-04-15T18:40:51.021 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:51.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:50 smithi082 ceph-mon[44206]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:51.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:40:51.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:50 smithi082 ceph-mon[44206]: osdmap e705: 12 total, 12 up, 12 in 2024-04-15T18:40:51.056 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:40:51.079 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:40:51.103 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:40:51.127 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:40:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:50 smithi159 ceph-mon[45336]: pgmap v1824: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/742708946' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:40:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:50 smithi159 ceph-mon[45336]: osdmap e705: 12 total, 12 up, 12 in 2024-04-15T18:40:51.152 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:40:51.155 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:51.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:51.177 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:40:51.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:40:51.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:51.286 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:51.305 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:51.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:51.408 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:51.439 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:51.459 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:51.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:51.517 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:40:51.517 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:40:51.518 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-04-15T18:40:51.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-04-15T18:40:51.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:51.627 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:51.751 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:51.753 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:51.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:51.855 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:51.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:51.905 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:51.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:52.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:52.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4229697732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:52.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.031 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:40:52.041 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:52.055 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:40:52.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:52.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:52.079 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:40:52.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:40:52.128 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:40:52.130 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:40:52.131 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:40:52.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4229697732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:52.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4229697732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:40:52.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.163 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:40:52.190 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:52.210 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:40:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:52.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:40:52.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:40:52.341 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:52.359 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:40:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:52.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:40:52.463 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:52.495 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:52.515 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:52.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:52.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:52.650 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: pgmap v1826: 1 pgs: 1 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:52.856 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: pgmap v1826: 1 pgs: 1 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: pgmap v1826: 1 pgs: 1 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.402 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:53.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:53.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:40:53.539 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:53.560 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:40:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:53.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:40:53.665 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:40:53.696 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:53.717 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:40:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:53.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:40:53.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:53.853 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:53.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:53 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:40:53.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:53 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:40:53.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:53 smithi159 ceph-mon[45336]: osdmap e706: 12 total, 12 up, 12 in 2024-04-15T18:40:53.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:53.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:53 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:40:53.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:53 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:40:53.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:53 smithi049 ceph-mon[40087]: osdmap e706: 12 total, 12 up, 12 in 2024-04-15T18:40:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:40:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:53 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:40:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:53 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:40:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:53 smithi082 ceph-mon[44206]: osdmap e706: 12 total, 12 up, 12 in 2024-04-15T18:40:54.704 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:54.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:54.809 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:40:54.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:54.863 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:40:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:54.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:40:54.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:40:54.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:54 smithi049 ceph-mon[40087]: pgmap v1828: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:54.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:40:54.998 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:55.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:54 smithi159 ceph-mon[45336]: pgmap v1828: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:55.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:54 smithi082 ceph-mon[44206]: pgmap v1828: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:55.793 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:40:55.841 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:55.852 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:40:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:55.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:40:55.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:40:55.991 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:56.013 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:40:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:56.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:40:56.117 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:56.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:56.146 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:40:56.149 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:56.170 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:56.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:56.274 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:56.304 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:56.325 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:56.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:56.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:40:56.460 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:56.793 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-04-15T18:40:57.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:56 smithi159 ceph-mon[45336]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:57.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:56 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:57.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:57.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:56 smithi159 ceph-mon[45336]: osdmap e707: 12 total, 12 up, 12 in 2024-04-15T18:40:57.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:40:57.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:56 smithi049 ceph-mon[40087]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:57.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:56 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:57.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:57.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:56 smithi049 ceph-mon[40087]: osdmap e707: 12 total, 12 up, 12 in 2024-04-15T18:40:57.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:40:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:56 smithi082 ceph-mon[44206]: pgmap v1829: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:56 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:40:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1038503515' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:40:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:56 smithi082 ceph-mon[44206]: osdmap e707: 12 total, 12 up, 12 in 2024-04-15T18:40:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:26879 6.2 e550/541 6:4ba4b07f:::benchmark_data_smithi159_139342_object26878:head v 550'1710, mlcod=550'1708)", 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 479.79961453700002, 2024-04-15T18:40:57.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22823505999999999, 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751559+0000", 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751560+0000", 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9199999999999999e-07 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.751592+0000", 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2632000000000002e-05 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978825+0000", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22723285100000001 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978831+0000", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.57e-06 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978884+0000", 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3667e-05 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.978900+0000", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5863e-05 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.979741+0000", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000840907 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:57.979794+0000", 2024-04-15T18:40:57.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2677999999999999e-05 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:34121 6.4 e552/541 6:26243d18:::benchmark_data_smithi159_139342_object34120:head v 552'2060, mlcod=552'2057)", 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 474.61437222900003, 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.259943007, 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.547 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936801+0000", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936802+0000", 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:02.936821+0000", 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9222999999999999e-05 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195530+0000", 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25870855500000001 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195535+0000", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.57e-06 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195587+0000", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1397000000000003e-05 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.195603+0000", 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6858999999999999e-05 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.196691+0000", 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010880200000000001 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:03.196744+0000", 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2673999999999997e-05 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:39450 6.0 e555/541 6:0b29b07c:::benchmark_data_smithi159_139342_object39449:head v 555'2492, mlcod=555'2488)", 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:20.721870+0000", 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 456.82930369899998, 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35120601800000001, 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.721870+0000", 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.721870+0000", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.721871+0000", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.029e-06 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.721899+0000", 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7818e-05 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.071782+0000", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34988371000000001 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.071790+0000", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6599999999999995e-06 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.071845+0000", 2024-04-15T18:40:57.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5056999999999998e-05 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.071869+0000", 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3870999999999999e-05 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.073045+0000", 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011757790000000001 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.073076+0000", 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1093999999999997e-05 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:39495 6.6 e555/541 6:6c77d314:::benchmark_data_smithi159_139342_object39494:head v 555'2433, mlcod=555'2431)", 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:20.823680+0000", 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 456.72749358800002, 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.222429397, 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.556 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.823680+0000", 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.823680+0000", 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.823680+0000", 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6499999999999999e-07 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.823758+0000", 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7632000000000005e-05 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.044786+0000", 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22102836200000001 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.044792+0000", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4650000000000002e-06 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.044856+0000", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4245999999999994e-05 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.044884+0000", 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8014999999999999e-05 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.046074+0000", 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001189954 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.046109+0000", 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5058000000000002e-05 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.560 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:87294 6.4 e573/541 6:2033d07a:::benchmark_data_smithi159_139342_object87293:head v 573'5388, mlcod=573'5385)", 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:34:03.360106+0000", 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 414.19106690799998, 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30683557900000002, 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.360106+0000", 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.360106+0000", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.360107+0000", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6499999999999999e-07 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.360128+0000", 2024-04-15T18:40:57.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0420000000000001e-05 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.663219+0000", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30309129099999998 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.663224+0000", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.293e-06 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.663297+0000", 2024-04-15T18:40:57.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2758999999999999e-05 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.663311+0000", 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4535999999999999e-05 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.666894+0000", 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003583021 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.666942+0000", 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7593999999999999e-05 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:87365 6.7 e574/541 6:eaf62e9d:::benchmark_data_smithi159_139342_object87364:head v 574'5449, mlcod=573'5446)", 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:34:03.716541+0000", 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 413.83463268899999, 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24473499600000001, 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.565 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.716541+0000", 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.716541+0000", 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.716543+0000", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1569999999999998e-06 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.716626+0000", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3363999999999997e-05 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.958813+0000", 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24218699199999999 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.958823+0000", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0162e-05 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.958891+0000", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7978000000000005e-05 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.958912+0000", 2024-04-15T18:40:57.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0435e-05 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.961237+0000", 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023250739999999999 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:03.961276+0000", 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8834000000000001e-05 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:124617 6.7 e587/541 6:e126c0da:::benchmark_data_smithi159_139342_object34738:head v 587'7730, mlcod=587'7726)", 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:34:58.388910+0000", 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 359.16226317100001, 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21652708500000001, 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.388910+0000", 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.388910+0000", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.388911+0000", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1299999999999998e-07 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.388919+0000", 2024-04-15T18:40:57.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4159999999999998e-06 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.599236+0000", 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21031775599999999 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.599243+0000", 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1720000000000001e-06 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.599330+0000", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6504999999999999e-05 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.599349+0000", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8621999999999999e-05 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.605376+0000", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060275219999999996 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:34:58.605437+0000", 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1179000000000005e-05 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:130650 6.6 e589/541 6:6d96bfc8:::benchmark_data_smithi159_139342_object40771:head v 589'8149, mlcod=589'8143)", 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:35:07.040978+0000", 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 350.51019534800002, 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23905739300000001, 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.040978+0000", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.040978+0000", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.040979+0000", 2024-04-15T18:40:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000003e-07 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.040984+0000", 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4639999999999997e-06 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.271930+0000", 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23094595700000001 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.271939+0000", 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0189999999999995e-06 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.272007+0000", 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7878000000000003e-05 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.272030+0000", 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3382000000000001e-05 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.279989+0000", 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079582969999999996 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:07.280035+0000", 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6822e-05 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:177237 6.b e609/541 6:d369e0e6:::benchmark_data_smithi159_139342_object87358:head v 609'11144, mlcod=609'11140)", 2024-04-15T18:40:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:18.022026+0000", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 279.52914696800002, 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27716159400000001, 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.022026+0000", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.022026+0000", 2024-04-15T18:40:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.022027+0000", 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5300000000000002e-07 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.022035+0000", 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6839999999999996e-06 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.290663+0000", 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26862771000000002 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.290674+0000", 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.152e-05 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.290797+0000", 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012249599999999999 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.290820+0000", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3382000000000001e-05 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.299135+0000", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0083145119999999996 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:18.299188+0000", 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3337000000000001e-05 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:179751 6.6 e611/541 6:68808706:::benchmark_data_smithi159_139342_object89872:head v 611'11288, mlcod=611'11286)", 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:21.377374+0000", 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 276.17379922999999, 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25312864600000001, 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377374+0000", 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377374+0000", 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377375+0000", 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3900000000000004e-07 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377382+0000", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6019999999999996e-06 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.626800+0000", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249418798 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.626811+0000", 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0708e-05 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.626873+0000", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1953999999999997e-05 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.626896+0000", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2739e-05 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.630466+0000", 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035707120000000002 2024-04-15T18:40:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.630503+0000", 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6294000000000002e-05 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:20978 7.3 e627/621 7:c2ae995e:::benchmark_data_smithi159_149998_object20977:head v 627'1333, mlcod=627'1330)", 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.207104+0000", 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 234.34406921499999, 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.242235272, 2024-04-15T18:40:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207104+0000", 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207104+0000", 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207105+0000", 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7000000000000004e-07 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207128+0000", 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2796e-05 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.447898+0000", 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24077046899999999 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.447905+0000", 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.255e-06 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.447972+0000", 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7061e-05 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.447994+0000", 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1067e-05 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449291+0000", 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012976419999999999 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449339+0000", 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8211999999999999e-05 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:21010 7.d e627/621 7:b6ab6c0b:::benchmark_data_smithi159_149998_object21009:head v 627'1289, mlcod=627'1285)", 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.225562+0000", 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 234.32561182200001, 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21344311399999999, 2024-04-15T18:40:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.225562+0000", 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.225562+0000", 2024-04-15T18:40:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.225562+0000", 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.09e-07 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.225585+0000", 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3003999999999999e-05 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.437924+0000", 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21233808300000001 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.437930+0000", 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3550000000000001e-06 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.437982+0000", 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2154000000000002e-05 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.438005+0000", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2459e-05 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.438958+0000", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095392800000000005 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.439005+0000", 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6221999999999999e-05 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:21015 7.d e627/621 7:b4aad474:::benchmark_data_smithi159_149998_object21014:head v 627'1290, mlcod=627'1286)", 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.228999+0000", 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 234.322174674, 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21144742499999999, 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.228999+0000", 2024-04-15T18:40:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.228999+0000", 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.228999+0000", 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1600000000000001e-07 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.229021+0000", 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1889999999999999e-05 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.439434+0000", 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21041233400000001 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.439441+0000", 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.165e-06 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.439489+0000", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8303000000000002e-05 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.439513+0000", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3901999999999999e-05 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.440393+0000", 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00087964399999999996 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.440446+0000", 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3470999999999997e-05 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:24092 7.7 e628/621 7:e7b2c8a2:::benchmark_data_smithi159_149998_object24091:head v 628'1507, mlcod=628'1504)", 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.040799+0000", 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 231.51037404100001, 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.230942811, 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040799+0000", 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040799+0000", 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040800+0000", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0100000000000004e-07 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.040867+0000", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6594000000000006e-05 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.270645+0000", 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.229778018 2024-04-15T18:40:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.270652+0000", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.905e-06 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.270697+0000", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5846000000000001e-05 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.270711+0000", 2024-04-15T18:40:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3983e-05 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.271703+0000", 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00099188000000000011 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.271742+0000", 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8884000000000002e-05 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31772 7.0 e631/621 7:002aa88c:::benchmark_data_smithi159_149998_object31771:head v 631'1969, mlcod=631'1966)", 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.021900+0000", 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 226.52927377099999, 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32026865100000002, 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.021900+0000", 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.021900+0000", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.021900+0000", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7199999999999998e-07 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.022011+0000", 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110698 2024-04-15T18:40:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.340961+0000", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31895016799999998 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.340970+0000", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3440000000000007e-06 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.341031+0000", 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0374000000000002e-05 2024-04-15T18:40:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.341051+0000", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0633e-05 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.342130+0000", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001078702 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.342168+0000", 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8059999999999998e-05 2024-04-15T18:40:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:55403 7.1 e639/621 7:8aca0ab0:::benchmark_data_smithi159_149998_object55402:head v 639'3471, mlcod=639'3466)", 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:31.177014+0000", 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 206.37415921499999, 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.230033562, 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.177014+0000", 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.177014+0000", 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.177014+0000", 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5400000000000002e-07 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.177143+0000", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128307 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.405391+0000", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.228248601 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.405397+0000", 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2020000000000003e-06 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.405469+0000", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2285000000000003e-05 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.405485+0000", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6518999999999999e-05 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.407014+0000", 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015283059999999999 2024-04-15T18:40:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.407048+0000", 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3988000000000001e-05 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:78937 7.1 e650/621 7:802f7ee1:::benchmark_data_smithi159_149998_object78936:head v 650'4897, mlcod=650'4893)", 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:55.852875+0000", 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 181.69829849600001, 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21695633, 2024-04-15T18:40:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:55.852875+0000", 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:55.852875+0000", 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:55.852875+0000", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.13e-07 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:55.852960+0000", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4970999999999997e-05 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.063090+0000", 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21012971699999999 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.063097+0000", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3939999999999998e-06 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.063146+0000", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8899000000000001e-05 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.063162+0000", 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5965999999999999e-05 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.069777+0000", 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066148170000000003 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:56.069831+0000", 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4052999999999999e-05 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:116782 7.1 e661/621 7:897aa864:::benchmark_data_smithi159_149998_object20782:head v 661'7226, mlcod=661'7224)", 2024-04-15T18:40:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 128.35770048800001, 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22267819999999999, 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193474+0000", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4900000000000005e-07 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193480+0000", 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9850000000000003e-06 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408763+0000", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.215283789 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408779+0000", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5925000000000001e-05 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408841+0000", 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1650999999999996e-05 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408864+0000", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3337e-05 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.416103+0000", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072389539999999997 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.416151+0000", 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7809999999999998e-05 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(7.c 670/648 [PushReplyOp(7:329499ac:::benchmark_data_smithi159_149998_object70303:head)])", 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 93.863428236000004, 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41282126800000002, 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:40:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687746+0000", 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6499999999999999e-07 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687750+0000", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7500000000000003e-06 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099805+0000", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41205476099999999 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099816+0000", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1031e-05 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099882+0000", 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6072e-05 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.100566+0000", 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00068408900000000003 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:146592 7.0 e676/621 7:0ce3da3a:::benchmark_data_smithi159_149998_object50592:head v 676'9102, mlcod=676'9099)", 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:40:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 85.82755349, 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37473794599999999, 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:40:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723621+0000", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4099999999999997e-07 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723626+0000", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.485e-06 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094806+0000", 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37117979200000001 2024-04-15T18:40:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094816+0000", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0121e-05 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094906+0000", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9442000000000002e-05 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094928+0000", 2024-04-15T18:40:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2731999999999999e-05 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.098320+0000", 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033914890000000001 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.098358+0000", 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7944000000000001e-05 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:40:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:40:57.631 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:40:57.784 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-04-15T18:40:57.785 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-04-15T18:40:57.785 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:40:57.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:40:57.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:40:57.851 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:57 smithi049 ceph-mon[40087]: osdmap e708: 12 total, 12 up, 12 in 2024-04-15T18:40:58.111 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:40:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:58.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:40:58.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:40:58.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:57 smithi159 ceph-mon[45336]: osdmap e708: 12 total, 12 up, 12 in 2024-04-15T18:40:58.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:40:58.243 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:58.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2355859561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:40:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:57 smithi082 ceph-mon[44206]: osdmap e708: 12 total, 12 up, 12 in 2024-04-15T18:40:58.537 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:58.920 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:40:58.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:40:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:40:58 smithi159 ceph-mon[45336]: pgmap v1832: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:59.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:40:58 smithi049 ceph-mon[40087]: pgmap v1832: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:40:58 smithi082 ceph-mon[44206]: pgmap v1832: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:40:59.492 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:40:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:59.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:40:59.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:59.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:59.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:59.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:40:59.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:40:59.785 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:59.805 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:40:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:59.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:40:59.909 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:40:59.941 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:40:59.962 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:40:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:40:59.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:00.067 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:00.099 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:00.120 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:00.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:00.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:00.252 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:00.450 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:00.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:00.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:00.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:41:00.584 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:00.684 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:00.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:00.788 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:00.820 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:00.840 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:00.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:00.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:00.972 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:01.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:00 smithi159 ceph-mon[45336]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:01.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:00 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:41:01.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:00 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:41:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:00 smithi049 ceph-mon[40087]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:00 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:41:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:00 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:41:01.216 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:01.220 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:01.220 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:01.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:01.265 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:01.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:00 smithi082 ceph-mon[44206]: pgmap v1833: 1 pgs: 1 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:00 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:41:01.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:00 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:41:01.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:01.354 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:01.502 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:01.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:01.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:01.638 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:01.660 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:01.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:01.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:01.795 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:01.814 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_6' created 2024-04-15T18:41:01.819 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:01.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:01.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:01.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:01.976 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:01.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:02.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-15T18:41:02.081 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:02.109 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-15T18:41:02.111 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:02.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:02.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:02.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-04-15T18:41:02.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:02.265 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:02.284 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:02.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:02.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:02.417 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:02.436 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:02.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:02.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:02.572 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:02.594 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:02.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:02.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:02.731 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:03.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-15T18:41:03.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: osdmap e709: 12 total, 12 up, 12 in 2024-04-15T18:41:03.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:41:03.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:41:03.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:41:03.019 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:41:03.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-15T18:41:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: osdmap e709: 12 total, 12 up, 12 in 2024-04-15T18:41:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:41:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:41:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:41:03.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3976622328' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: osdmap e709: 12 total, 12 up, 12 in 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:41:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:41:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:03 smithi049 ceph-mon[40087]: osdmap e710: 12 total, 12 up, 12 in 2024-04-15T18:41:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:41:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:41:03.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:41:04.014 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:04.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:04.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:03 smithi159 ceph-mon[45336]: osdmap e710: 12 total, 12 up, 12 in 2024-04-15T18:41:04.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:41:04.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:41:04.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:41:04.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:04.203 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:03 smithi082 ceph-mon[44206]: osdmap e710: 12 total, 12 up, 12 in 2024-04-15T18:41:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:41:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:41:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:41:04.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:04.342 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:04.363 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:04.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:04.500 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:04.521 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:04.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:04.657 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:04.679 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:04.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:04.812 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:04.832 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:04.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:04.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:04.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:04 smithi049 ceph-mon[40087]: pgmap v1837: 17 pgs: 4 creating+peering, 6 unknown, 7 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:04.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:04 smithi049 ceph-mon[40087]: osdmap e711: 12 total, 12 up, 12 in 2024-04-15T18:41:04.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:04 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:41:04.968 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:41:04.968 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:41:04.968 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:41:04.968 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:41:04.969 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:05.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:04 smithi082 ceph-mon[44206]: pgmap v1837: 17 pgs: 4 creating+peering, 6 unknown, 7 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:04 smithi082 ceph-mon[44206]: osdmap e711: 12 total, 12 up, 12 in 2024-04-15T18:41:05.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:04 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:41:05.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:04 smithi159 ceph-mon[45336]: pgmap v1837: 17 pgs: 4 creating+peering, 6 unknown, 7 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:05.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:04 smithi159 ceph-mon[45336]: osdmap e711: 12 total, 12 up, 12 in 2024-04-15T18:41:05.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:04 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:41:05.319 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:05.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:05.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-04-15T18:41:05.423 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:05.455 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:05.476 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:05.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:05.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:05.608 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:05.681 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:05.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:05.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:05.818 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:05.839 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:05.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:05.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:05.972 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:05.991 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:05.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:06.095 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:06.128 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:06.149 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:06.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:06.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:41:06.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:06.287 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:06.308 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:06.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:06.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:06.440 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:07.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:06 smithi049 ceph-mon[40087]: pgmap v1839: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:07.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/532852997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:07.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:06 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:06 smithi159 ceph-mon[45336]: pgmap v1839: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:07.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/532852997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:07.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:06 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:06 smithi082 ceph-mon[44206]: pgmap v1839: 17 pgs: 4 creating+peering, 13 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:07.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/532852997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:07.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:06 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.662 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:41:07.662 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":711,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:41:03.825931+0000","last_up_change":"2024-04-15T18:37:10.676318+0000","last_in_change":"2024-04-15T18:37:52.482645+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":10,"score_stable":10,"optimal_score":0.30000001192092896,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}},{"pool":8,"pool_name":"unique_pool_6","create_time":"2024-04-15T18:41:00.938258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"711","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1.1000000238418579,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":0.83333325386047363,"average_primary_affinity":1,"average_primary_affinity_weighted":0.82999998331069946}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":671,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":709,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":307,"up_from":361,"up_thru":709,"down_at":308,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6819","nonce":2410603388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6821","nonce":2410603388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6825","nonce":2410603388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6823","nonce":2410603388}]},"public_addr":"172.21.15.49:6819/2410603388","cluster_addr":"172.21.15.49:6821/2410603388","heartbeat_back_addr":"172.21.15.49:6825/2410603388","heartbeat_front_addr":"172.21.15.49:6823/2410603388","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":26,"last_clean_end":177,"up_from":226,"up_thru":675,"down_at":178,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6827","nonce":1036848877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6829","nonce":1036848877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6833","nonce":1036848877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6831","nonce":1036848877}]},"public_addr":"172.21.15.49:6827/1036848877","cluster_addr":"172.21.15.49:6829/1036848877","heartbeat_back_addr":"172.21.15.49:6833/1036848877","heartbeat_front_addr":"172.21.15.49:6831/1036848877","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":621,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":363,"up_from":376,"up_thru":709,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6809","nonce":1630882481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6811","nonce":1630882481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6815","nonce":1630882481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6813","nonce":1630882481}]},"public_addr":"172.21.15.82:6809/1630882481","cluster_addr":"172.21.15.82:6811/1630882481","heartbeat_back_addr":"172.21.15.82:6815/1630882481","heartbeat_front_addr":"172.21.15.82:6813/1630882481","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":47,"last_clean_end":561,"up_from":631,"up_thru":676,"down_at":562,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6817","nonce":256106271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6819","nonce":256106271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6823","nonce":256106271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6821","nonce":256106271}]},"public_addr":"172.21.15.82:6817/256106271","cluster_addr":"172.21.15.82:6819/256106271","heartbeat_back_addr":"172.21.15.82:6823/256106271","heartbeat_front_addr":"172.21.15.82:6821/256106271","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":650,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":709,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":709,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":709,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":94,"up_thru":709,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6825","nonce":4245248647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6827","nonce":4245248647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6831","nonce":4245248647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":4245248647},{"type":"v1","addr":"172.21.15.159:6829","nonce":4245248647}]},"public_addr":"172.21.15.159:6825/4245248647","cluster_addr":"172.21.15.159:6827/4245248647","heartbeat_back_addr":"172.21.15.159:6831/4245248647","heartbeat_front_addr":"172.21.15.159:6829/4245248647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:52.322987+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:11.686909+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:10.403116+0000","dead_epoch":307},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:37.921930+0000","dead_epoch":177},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:19.312877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:39:59.591340+0000","dead_epoch":363},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:32.797708+0000","dead_epoch":561},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:09.102066+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:30.896001+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:12.003750+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:07.471399+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:40:12.132056+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:41:07.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:07.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:07.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:07.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:07.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:07.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:07.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1568577569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:08.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1568577569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:08.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1568577569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:41:08.838 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:41:09.098 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:09.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:08 smithi049 ceph-mon[40087]: pgmap v1840: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:09.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:08 smithi049 ceph-mon[40087]: osdmap e712: 12 total, 12 up, 12 in 2024-04-15T18:41:09.099 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:09.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:08 smithi159 ceph-mon[45336]: pgmap v1840: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:08 smithi159 ceph-mon[45336]: osdmap e712: 12 total, 12 up, 12 in 2024-04-15T18:41:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:09.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:09.234 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:09.254 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:09.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:09.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:08 smithi082 ceph-mon[44206]: pgmap v1840: 17 pgs: 3 creating+peering, 14 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:08 smithi082 ceph-mon[44206]: osdmap e712: 12 total, 12 up, 12 in 2024-04-15T18:41:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:09.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:09.738 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:09.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:09.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:09.869 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:10.064 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:09 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:10.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:10.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:09 smithi049 ceph-mon[40087]: osdmap e713: 12 total, 12 up, 12 in 2024-04-15T18:41:10.065 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:10.065 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:10.067 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:10.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:10.103 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:41:10.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:09 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:10.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:10.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:09 smithi159 ceph-mon[45336]: osdmap e713: 12 total, 12 up, 12 in 2024-04-15T18:41:10.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:10.200 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:09 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3582627131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:09 smithi082 ceph-mon[44206]: osdmap e713: 12 total, 12 up, 12 in 2024-04-15T18:41:10.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:41:10.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:10 smithi049 ceph-mon[40087]: pgmap v1843: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:10.949 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:10.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:11.053 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:11.085 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:11.106 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:11.108 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:10 smithi159 ceph-mon[45336]: pgmap v1843: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:11.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:11.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:11.237 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:11.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:10 smithi082 ceph-mon[44206]: pgmap v1843: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:11.386 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:11.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:11.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:11.518 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:11.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:41:11.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:11.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:11.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:11.999 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:12.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:12.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:12.583 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:12.658 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:12.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:12.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:12.795 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:13.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:12 smithi159 ceph-mon[45336]: pgmap v1844: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:13.201 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:13.201 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:12 smithi049 ceph-mon[40087]: pgmap v1844: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:13.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:12 smithi082 ceph-mon[44206]: pgmap v1844: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:13.844 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:13.848 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:13.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:13.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:13.986 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.008 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:14.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:14.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:14.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.165 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:14.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:14.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:14.299 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.319 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:14.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:14.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:14.456 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.477 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:14.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:14.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:41:14.581 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:14.611 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.631 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:14.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:14.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:14.767 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:14.850 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-04-15T18:41:15.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:14 smithi159 ceph-mon[45336]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:15.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:41:15.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:14 smithi049 ceph-mon[40087]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:15.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:41:15.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:14 smithi082 ceph-mon[44206]: pgmap v1845: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:15.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:41:15.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:41:15.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:15 smithi049 ceph-mon[40087]: osdmap e714: 12 total, 12 up, 12 in 2024-04-15T18:41:16.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:41:16.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:15 smithi159 ceph-mon[45336]: osdmap e714: 12 total, 12 up, 12 in 2024-04-15T18:41:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1642243698' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:41:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:15 smithi082 ceph-mon[44206]: osdmap e714: 12 total, 12 up, 12 in 2024-04-15T18:41:16.621 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:16.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:16.726 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:16.760 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:16.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:16.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:16.864 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: osdmap e715: 12 total, 12 up, 12 in 2024-04-15T18:41:16.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:41:16.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:16.916 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:41:17.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: osdmap e715: 12 total, 12 up, 12 in 2024-04-15T18:41:17.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: pgmap v1847: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: osdmap e715: 12 total, 12 up, 12 in 2024-04-15T18:41:17.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:41:18.079 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:18.094 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:18.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:18.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:18.131 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: osdmap e716: 12 total, 12 up, 12 in 2024-04-15T18:41:18.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:17 smithi159 ceph-mon[45336]: osdmap e717: 12 total, 12 up, 12 in 2024-04-15T18:41:18.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:18.179 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:18.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:18.205 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: osdmap e716: 12 total, 12 up, 12 in 2024-04-15T18:41:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:17 smithi049 ceph-mon[40087]: osdmap e717: 12 total, 12 up, 12 in 2024-04-15T18:41:18.230 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:18.230 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:18.235 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-04-15T18:41:18.237 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:41:18.238 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:41:18.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:18.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:18.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:18.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:18.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/484299736' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:41:18.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: osdmap e716: 12 total, 12 up, 12 in 2024-04-15T18:41:18.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:17 smithi082 ceph-mon[44206]: osdmap e717: 12 total, 12 up, 12 in 2024-04-15T18:41:18.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:18.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:18.331 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:18.356 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:41:18.356 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:41:18.356 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_162467 2024-04-15T18:41:18.357 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:18.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:41:18.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:41:18.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:18.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:18.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:18.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:18.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:18.359 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:18.579 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:18.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:18.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:18.716 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:18.793 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:18.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:18.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:18.925 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:18.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:18.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:19.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:19.079 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1962257628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:18 smithi159 ceph-mon[45336]: osdmap e718: 12 total, 12 up, 12 in 2024-04-15T18:41:19.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:19.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1962257628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:18 smithi049 ceph-mon[40087]: osdmap e718: 12 total, 12 up, 12 in 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: pgmap v1850: 17 pgs: 17 active+clean; 577 KiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1962257628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:18 smithi082 ceph-mon[44206]: osdmap e718: 12 total, 12 up, 12 in 2024-04-15T18:41:19.334 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:19.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:19.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 2010 1994 124.618 124.625 0.00627075 0.00793552 2024-04-15T18:41:19.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:19.469 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:19.490 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:19.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:19.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:19.630 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:19.656 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:19.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:19.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:19.790 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.240 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:20.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:20.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:20.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 3825 3809 119.02 113.438 0.00607634 0.00828148 2024-04-15T18:41:20.371 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:20.389 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:20.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:20.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:20.540 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:20.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:20.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:20.673 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:20.727 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:20.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:20.831 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:20.858 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:20.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:20 smithi049 ceph-mon[40087]: pgmap v1853: 17 pgs: 17 active+clean; 1.3 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s wr, 2 op/s 2024-04-15T18:41:20.876 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:20.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:20.882 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:41:20.882 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:41:20.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:41:20.882 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26224.0:179753 6.0 6:0844f8ba:::benchmark_data_smithi159_139342_object89874:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 299.514699253, 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60003534400000003, 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26224", 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 179753 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.883 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376962+0000", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376963+0000", 2024-04-15T18:41:20.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376968+0000", 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8640000000000004e-06 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376978+0000", 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.064e-05 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.376984+0000", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2249999999999997e-06 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377476+0000", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049176799999999996 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377618+0000", 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014151600000000001 2024-04-15T18:41:20.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.377745+0000", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012707899999999999 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.383798+0000", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060528409999999998 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.387084+0000", 2024-04-15T18:41:20.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032860659999999998 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976917+0000", 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.58983302100000001 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976961+0000", 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3986000000000002e-05 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976997+0000", 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6151e-05 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26224.0:179753 6.0 e611/541 ondisk, result = 0)", 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 299.50605954000002, 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59140211200000004, 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:41:20.889 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385601+0000", 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385602+0000", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.035e-06 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.385608+0000", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6829999999999996e-06 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976844+0000", 2024-04-15T18:41:20.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59123575299999998 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976853+0000", 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2690000000000005e-06 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976903+0000", 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0364000000000001e-05 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.976911+0000", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.515e-06 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:36:21.977003+0000", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2492999999999997e-05 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:20978 7.3 7:c2ae995e:::benchmark_data_smithi159_149998_object20977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e627)", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:41:20.893 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 257.685144036, 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24318234699999999, 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 20978 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206517+0000", 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206518+0000", 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206580+0000", 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2403000000000005e-05 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206591+0000", 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899e-05 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206597+0000", 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0929999999999999e-06 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206678+0000", 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0079000000000004e-05 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.206844+0000", 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000166534 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.207012+0000", 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000167856 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.208175+0000", 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001162672 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.208725+0000", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00055022600000000001 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449599+0000", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24087378400000001 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449656+0000", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7376000000000001e-05 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:03.449699+0000", 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3081999999999998e-05 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31756 7.2 e631/621 7:45054463:::benchmark_data_smithi159_149998_object31755:head v 631'1956, mlcod=631'1954)", 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.004330+0000", 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.88733097799999, 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34978704599999999, 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.004330+0000", 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.004330+0000", 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.004331+0000", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4300000000000002e-07 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.004444+0000", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113366 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.352544+0000", 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34810039700000001 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.352552+0000", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6599999999999995e-06 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.352620+0000", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7951e-05 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.352646+0000", 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6339999999999999e-05 2024-04-15T18:41:20.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354073+0000", 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00142628 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354117+0000", 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4209000000000003e-05 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:31761 7.e 7:75e61d4c:::benchmark_data_smithi159_149998_object31760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.009469+0000", 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.88219222800001, 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.349855623, 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 31761 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009469+0000", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009469+0000", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009470+0000", 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.099e-06 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009511+0000", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1579999999999998e-05 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009525+0000", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3409e-05 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009533+0000", 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3389999999999993e-06 2024-04-15T18:41:20.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009604+0000", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1390000000000006e-05 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009695+0000", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0445000000000006e-05 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.009798+0000", 2024-04-15T18:41:20.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010333 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.010898+0000", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010999779999999999 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.017813+0000", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0069148309999999998 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359225+0000", 2024-04-15T18:41:20.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34141200500000002 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359280+0000", 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5124999999999998e-05 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359324+0000", 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4091999999999997e-05 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:31761 7.e e631/621 ondisk, result = 0)", 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.012087+0000", 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.87957345999999, 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34724340599999998, 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.012087+0000", 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.012087+0000", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.012088+0000", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6899999999999997e-07 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.012091+0000", 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.241e-06 2024-04-15T18:41:20.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359146+0000", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34705438599999999 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359151+0000", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5339999999999999e-06 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359209+0000", 2024-04-15T18:41:20.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7627999999999998e-05 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359216+0000", 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7389999999999999e-06 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359331+0000", 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011430900000000001 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:31779 7.8 7:15328945:::benchmark_data_smithi159_149998_object31778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.029118+0000", 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.86254252099999, 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32716872000000002, 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 31779 2024-04-15T18:41:20.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029118+0000", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029118+0000", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029120+0000", 2024-04-15T18:41:20.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.717e-06 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029164+0000", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3648e-05 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029181+0000", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6884999999999998e-05 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029187+0000", 2024-04-15T18:41:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8850000000000001e-06 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029251+0000", 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3668000000000001e-05 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029360+0000", 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010934600000000001 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.029458+0000", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7571999999999993e-05 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.030740+0000", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012821180000000001 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.031145+0000", 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040531699999999998 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356190+0000", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32504480800000002 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356240+0000", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9379999999999998e-05 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356287+0000", 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7376000000000002e-05 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.920 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:31779 7.8 e631/621 ondisk, result = 0)", 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.030958+0000", 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.860702737, 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32533682600000002, 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.030958+0000", 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.030958+0000", 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.030959+0000", 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.09e-07 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.030963+0000", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0049999999999998e-06 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354784+0000", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32382116799999999 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354792+0000", 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7270000000000007e-06 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356175+0000", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001382936 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356183+0000", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1689999999999992e-06 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.356295+0000", 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112012 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.924 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:31785 7.e 7:78b1b8f5:::benchmark_data_smithi159_149998_object31784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.040995+0000", 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.85066602099999, 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34008148399999999, 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.925 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 31785 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.040995+0000", 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.040995+0000", 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.040996+0000", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.477e-06 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041033+0000", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6878999999999997e-05 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041050+0000", 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6901999999999998e-05 2024-04-15T18:41:20.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041057+0000", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000002e-06 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041106+0000", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9597e-05 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041219+0000", 2024-04-15T18:41:20.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112991 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.041374+0000", 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015475600000000001 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.042468+0000", 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010938160000000001 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.117135+0000", 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.074667258 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.381003+0000", 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26386832500000001 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.381042+0000", 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9131000000000002e-05 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.381076+0000", 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3822000000000002e-05 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:31785 7.e e631/621 ondisk, result = 0)", 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.042845+0000", 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.848815903, 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33823868899999998, 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.931 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.042845+0000", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.042845+0000", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.042846+0000", 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6000000000000003e-07 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.042849+0000", 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8149999999999999e-06 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.380934+0000", 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33808438800000001 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.380939+0000", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3380000000000004e-06 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.380989+0000", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9693e-05 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.380997+0000", 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4649999999999994e-06 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.381084+0000", 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6329999999999995e-05 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31790 7.c e631/621 7:31db844e:::benchmark_data_smithi159_149998_object31789:head v 631'1978, mlcod=631'1973)", 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.045819+0000", 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.845841659, 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30993954299999998, 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.935 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.045819+0000", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.045819+0000", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.045820+0000", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0200000000000001e-07 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.045944+0000", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124003 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.353787+0000", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30784317 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.353794+0000", 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0840000000000003e-06 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.353861+0000", 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7008000000000006e-05 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.353886+0000", 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4553999999999999e-05 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355724+0000", 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018386979999999999 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355759+0000", 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4224000000000003e-05 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.939 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31797 7.c e631/621 7:32fe5542:::benchmark_data_smithi159_149998_object31796:head v 631'1979, mlcod=631'1974)", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.055890+0000", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 249.83577105699999, 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30002665000000001, 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.055890+0000", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.055890+0000", 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.055891+0000", 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1699999999999997e-07 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.055983+0000", 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2527000000000004e-05 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355034+0000", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29905053199999998 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355038+0000", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3630000000000004e-06 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355091+0000", 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2719999999999997e-05 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355112+0000", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1100000000000001e-05 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355891+0000", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077912499999999998 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.355916+0000", 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5366000000000001e-05 2024-04-15T18:41:20.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:55389 7.8 7:1b3fac0c:::benchmark_data_smithi159_149998_object55388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:31.165186+0000", 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 229.72647459999999, 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24157884299999999, 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.944 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 55389 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165186+0000", 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165186+0000", 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165187+0000", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.28e-06 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165235+0000", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7886000000000002e-05 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165249+0000", 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3372e-05 2024-04-15T18:41:20.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165257+0000", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1890000000000007e-06 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165321+0000", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3922000000000002e-05 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165484+0000", 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000163232 2024-04-15T18:41:20.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.165652+0000", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016750700000000001 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.167460+0000", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018087940000000001 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.169850+0000", 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023893069999999998 2024-04-15T18:41:20.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.406697+0000", 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23684746200000001 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.406733+0000", 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5688000000000001e-05 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.406765+0000", 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2203999999999999e-05 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:77914 7.c e646/621)", 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:52.465936+0000", 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 208.42572519500001, 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27896208900000002, 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:41:20.950 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.465936+0000", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.465936+0000", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.465936+0000", 2024-04-15T18:41:20.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4700000000000001e-07 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.466029+0000", 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2590999999999995e-05 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.744598+0000", 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27856857899999998 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.744612+0000", 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4025999999999999e-05 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.744856+0000", 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000244498 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.744898+0000", 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1647999999999998e-05 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "rep_scrubmap(7.8 e621 from shard 4)", 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:02.372149+0000", 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 198.51951177699999, 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18307280200000001, 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.372149+0000", 2024-04-15T18:41:20.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.372149+0000", 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.372150+0000", 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-06 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.372179+0000", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8719000000000001e-05 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.554970+0000", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18279124199999999 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.554976+0000", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5450000000000003e-06 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.555019+0000", 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3383000000000001e-05 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.555031+0000", 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1705000000000001e-05 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.555222+0000", 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019088800000000001 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:146597 7.2 e676/621 7:40f54c50:::benchmark_data_smithi159_149998_object50597:head v 676'9150, mlcod=676'9147)", 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:31.768190+0000", 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 109.123470369, 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33841615400000002, 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.768190+0000", 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.768190+0000", 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.768191+0000", 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.768200+0000", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9509999999999995e-06 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.103196+0000", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33499582300000003 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.103202+0000", 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2650000000000002e-06 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.103266+0000", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4109999999999994e-05 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.103286+0000", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0000000000000002e-05 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.106573+0000", 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003287192 2024-04-15T18:41:20.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.106607+0000", 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3071999999999998e-05 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:161854 7.3 7:cd884343:::benchmark_data_smithi159_149998_object65854:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:53.294361+0000", 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.597299996999993, 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22073303999999999, 2024-04-15T18:41:20.962 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 161854 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294361+0000", 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294361+0000", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294361+0000", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6800000000000001e-07 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294365+0000", 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2720000000000002e-06 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294373+0000", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9190000000000005e-06 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294380+0000", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.263e-06 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.294942+0000", 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056239099999999998 2024-04-15T18:41:20.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.295067+0000", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124759 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.295150+0000", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3257999999999995e-05 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.299117+0000", 2024-04-15T18:41:20.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039665289999999999 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.299838+0000", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072121900000000005 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.514982+0000", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21514462100000001 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.515044+0000", 2024-04-15T18:41:20.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1702e-05 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.515094+0000", 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9638999999999997e-05 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:161859 7.3 7:c193a2a5:::benchmark_data_smithi159_149998_object65859:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:53.300449+0000", 2024-04-15T18:41:20.968 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.591211541000007, 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223654358, 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 161859 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300449+0000", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300449+0000", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300450+0000", 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5400000000000001e-07 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300453+0000", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4249999999999998e-06 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300461+0000", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8290000000000008e-06 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300467+0000", 2024-04-15T18:41:20.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9009999999999999e-06 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300515+0000", 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8442999999999997e-05 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300636+0000", 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012012 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.300756+0000", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012035 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.305628+0000", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048720580000000003 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.309492+0000", 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038638380000000001 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.523999+0000", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.214507317 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.524056+0000", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6815999999999999e-05 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.524104+0000", 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7607e-05 2024-04-15T18:41:20.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:161881 7.3 7:ca72be45:::benchmark_data_smithi159_149998_object65881:head [delete] snapc 0=[] ondisk+write+known_if_redirected e680)", 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:53.334925+0000", 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.556735931000006, 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.200959045, 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:20.975 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 161881 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334925+0000", 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334925+0000", 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334927+0000", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9259999999999999e-06 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334933+0000", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8300000000000001e-06 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334947+0000", 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4644000000000001e-05 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.334956+0000", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8480000000000007e-06 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.335020+0000", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3756000000000004e-05 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.335160+0000", 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000140209 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.335289+0000", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129018 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.339407+0000", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041182229999999998 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.339549+0000", 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014149599999999999 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.535755+0000", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.196206626 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.535824+0000", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8399999999999996e-05 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:53.535884+0000", 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0068999999999997e-05 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.980 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:1365 8.a e718/709 8:5f468201:::benchmark_data_smithi159_162467_object1364:head v 718'71, mlcod=718'66)", 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:41:18.911971+0000", 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.9796892879999999, 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18274380500000001, 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:41:20.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:18.911971+0000", 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:18.911971+0000", 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:18.911972+0000", 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1200000000000003e-07 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:18.912031+0000", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8455000000000002e-05 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094113+0000", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18208260600000001 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094119+0000", 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0510000000000002e-06 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094179+0000", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9456999999999997e-05 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094212+0000", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3599999999999997e-05 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094667+0000", 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045472699999999998 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:41:20.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:41:19.094715+0000", 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8297000000000002e-05 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:41:20.985 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:41:20.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:21.013 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.034 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:21.138 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:20 smithi159 ceph-mon[45336]: pgmap v1853: 17 pgs: 17 active+clean; 1.3 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s wr, 2 op/s 2024-04-15T18:41:21.171 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.194 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:20 smithi082 ceph-mon[44206]: pgmap v1853: 17 pgs: 17 active+clean; 1.3 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s wr, 2 op/s 2024-04-15T18:41:21.299 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:21.338 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:41:21.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 4840 4824 100.491 63.4375 0.00800279 0.00988449 2024-04-15T18:41:21.359 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:21.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:21.493 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.520 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:21.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:21.658 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.718 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:21.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:41:21.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:41:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:21.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:41:21.860 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:21.883 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:41:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:21.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:41:21.987 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:22.022 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:22.046 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:22.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:22.099 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:22.151 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:22.189 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:22.219 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:22.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:22.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:41:22.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 7178 7162 111.896 146.125 0.00184708 0.00892166 2024-04-15T18:41:22.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:22.386 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:41:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:22.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:41:22.490 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:22.521 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:22.648 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:22.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:22.753 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:22.788 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:22.814 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:22.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:22.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:22.951 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:22 smithi082 ceph-mon[44206]: pgmap v1854: 17 pgs: 17 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s 2024-04-15T18:41:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:22 smithi082 ceph-mon[44206]: osdmap e719: 12 total, 12 up, 12 in 2024-04-15T18:41:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:22 smithi159 ceph-mon[45336]: pgmap v1854: 17 pgs: 17 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s 2024-04-15T18:41:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:22 smithi159 ceph-mon[45336]: osdmap e719: 12 total, 12 up, 12 in 2024-04-15T18:41:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:22 smithi049 ceph-mon[40087]: pgmap v1854: 17 pgs: 17 active+clean; 74 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s 2024-04-15T18:41:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:22 smithi049 ceph-mon[40087]: osdmap e719: 12 total, 12 up, 12 in 2024-04-15T18:41:23.222 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:23.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 8942 8926 111.566 110.25 0.00328466 0.00895097 2024-04-15T18:41:23.504 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:23.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:23.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:23.648 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:23.753 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:23.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:23.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:23.893 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:23.920 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:23.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:24.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:24.056 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.079 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:24.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:24.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:24.183 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:24.219 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.245 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:24.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:24.357 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 10769 10753 112.001 114.188 0.0477941 0.00889977 2024-04-15T18:41:24.387 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:24.415 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:24.418 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:24.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:24.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:24.480 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:24.480 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:24.523 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:24.559 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.587 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:24.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:24.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:24.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:24.698 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:24.698 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:24.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.752 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:24.857 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: pgmap v1856: 17 pgs: 17 active+clean; 394 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osdmap e720: 12 total, 12 up, 12 in 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3822202700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:24.876 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.877 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:24.896 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:24.922 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:24.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:25.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:41:25.056 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: pgmap v1856: 17 pgs: 17 active+clean; 394 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osdmap e720: 12 total, 12 up, 12 in 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3822202700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.154 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.154 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.154 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.154 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.155 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:41:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:25.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:41:25.183 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:41:25.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: pgmap v1856: 17 pgs: 17 active+clean; 394 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:41:25.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osdmap e720: 12 total, 12 up, 12 in 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3822202700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:25.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:25.291 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:25.311 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:25.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:25.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 12117 12101 108.035 84.25 0.00698519 0.0091769 2024-04-15T18:41:25.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:25.442 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:25.463 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:25.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:25.498 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:25.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:26.010 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.010 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.010 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.011 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.011 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:26.011 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:26.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:26.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 13334 13318 104.038 76.0625 0.033341 0.00958094 2024-04-15T18:41:27.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:26 smithi082 ceph-mon[44206]: pgmap v1858: 17 pgs: 17 active+clean; 598 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:41:27.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:26 smithi082 ceph-mon[44206]: osdmap e721: 12 total, 12 up, 12 in 2024-04-15T18:41:27.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:26 smithi159 ceph-mon[45336]: pgmap v1858: 17 pgs: 17 active+clean; 598 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:41:27.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:26 smithi159 ceph-mon[45336]: osdmap e721: 12 total, 12 up, 12 in 2024-04-15T18:41:27.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:26 smithi049 ceph-mon[40087]: pgmap v1858: 17 pgs: 17 active+clean; 598 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-04-15T18:41:27.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:26 smithi049 ceph-mon[40087]: osdmap e721: 12 total, 12 up, 12 in 2024-04-15T18:41:27.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 14407 14391 99.9289 67.0625 0.00283809 0.00999646 2024-04-15T18:41:27.388 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:27.389 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:27.447 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:27.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:27.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:27.592 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.018 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:41:28.044 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:28.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:28.182 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.205 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:28.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:28.337 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:28.340 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 15 15728 15713 98.1978 82.625 0.037861 0.0101573 2024-04-15T18:41:28.363 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:28.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:28.503 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.528 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:28.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:28.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:28 smithi159 ceph-mon[45336]: pgmap v1860: 17 pgs: 17 active+clean; 623 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:28.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:28 smithi159 ceph-mon[45336]: osdmap e722: 12 total, 12 up, 12 in 2024-04-15T18:41:28.563 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3165840230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.564 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:28 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:28.670 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.698 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:28 smithi049 ceph-mon[40087]: pgmap v1860: 17 pgs: 17 active+clean; 623 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:28 smithi049 ceph-mon[40087]: osdmap e722: 12 total, 12 up, 12 in 2024-04-15T18:41:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3165840230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:28 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.802 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:28.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:28.870 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:28.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:28.899 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:28 smithi082 ceph-mon[44206]: pgmap v1860: 17 pgs: 17 active+clean; 623 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:28.899 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:28 smithi082 ceph-mon[44206]: osdmap e722: 12 total, 12 up, 12 in 2024-04-15T18:41:28.899 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3165840230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.899 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:28 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:28.901 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:41:28.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:29.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:29.028 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:29.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:29.133 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:29.168 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:29.189 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:29.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:29.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:29.322 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:29.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 17326 17310 98.3439 99.8125 0.0142633 0.0101574 2024-04-15T18:41:29.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:41:29.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:29 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:29.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:29 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:29.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:29 smithi049 ceph-mon[40087]: osdmap e723: 12 total, 12 up, 12 in 2024-04-15T18:41:29.625 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:29 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:29.626 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:29 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:29.626 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:29 smithi082 ceph-mon[44206]: osdmap e723: 12 total, 12 up, 12 in 2024-04-15T18:41:29.673 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:29.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:29.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:29.811 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:29.819 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:29.837 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:29.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:29.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:29 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:29.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:29 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:29.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:29 smithi159 ceph-mon[45336]: osdmap e723: 12 total, 12 up, 12 in 2024-04-15T18:41:29.945 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:29.974 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:29.992 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:29.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:30.096 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:30.103 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:41:30.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:41:30.125 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:30.231 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:30.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:30.335 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:30.335 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:30.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:30.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 18386 18370 95.6688 66.25 0.0224528 0.0104192 2024-04-15T18:41:30.366 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:30.983 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:30 smithi082 ceph-mon[44206]: pgmap v1863: 17 pgs: 17 active+clean; 872 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:31.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:41:31.015 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:31.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:30 smithi159 ceph-mon[45336]: pgmap v1863: 17 pgs: 17 active+clean; 872 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:31.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:30 smithi049 ceph-mon[40087]: pgmap v1863: 17 pgs: 17 active+clean; 872 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:31.155 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:31.324 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:31.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 20102 20086 96.559 107.25 0.00806556 0.010347 2024-04-15T18:41:31.662 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:31.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:31.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:41:31.767 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:31.796 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:32.245 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:32.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 21345 21329 95.2106 77.6875 0.00303432 0.0104974 2024-04-15T18:41:32.547 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:32.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:32.652 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:32.686 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:32.707 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:32.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:32.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:32 smithi082 ceph-mon[44206]: pgmap v1864: 17 pgs: 17 active+clean; 1020 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-15T18:41:33.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:32 smithi082 ceph-mon[44206]: osdmap e724: 12 total, 12 up, 12 in 2024-04-15T18:41:33.109 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:33.109 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:33.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:32 smithi159 ceph-mon[45336]: pgmap v1864: 17 pgs: 17 active+clean; 1020 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-15T18:41:33.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:32 smithi159 ceph-mon[45336]: osdmap e724: 12 total, 12 up, 12 in 2024-04-15T18:41:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:32 smithi049 ceph-mon[40087]: pgmap v1864: 17 pgs: 17 active+clean; 1020 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-04-15T18:41:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:32 smithi049 ceph-mon[40087]: osdmap e724: 12 total, 12 up, 12 in 2024-04-15T18:41:33.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 22408 22392 93.2917 66.4375 0.0109319 0.01066 2024-04-15T18:41:33.776 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:33.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:33.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:41:33.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:33.924 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:33.963 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:33.963 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:33.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:34.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:34.102 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:34.179 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:34.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:34.234 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:34.261 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:34.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:34.267 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:34.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:34.324 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:34.354 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:34.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 23878 23862 93.2028 91.875 0.0301001 0.0107205 2024-04-15T18:41:34.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:34.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:34.401 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:34.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:34.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:34.472 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:34.503 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:34.534 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:34.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:34.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:34.535 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: pgmap v1866: 17 pgs: 17 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1334405954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: pgmap v1866: 17 pgs: 17 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1334405954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 25187 25171 92.5324 81.8125 0.0305713 0.0107988 2024-04-15T18:41:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: pgmap v1866: 17 pgs: 17 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-04-15T18:41:35.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1334405954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:35.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:35.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:41:36.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:35 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:41:36.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:35 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:41:36.331 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:41:36.331 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:41:36.331 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:41:36.331 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:41:36.331 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:41:36.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 15 26644 26629 92.4536 91.125 0.0133484 0.0108086 2024-04-15T18:41:36.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:41:36.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:35 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:41:36.792 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:36.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:36.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:36.941 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:36 smithi082 ceph-mon[44206]: pgmap v1867: 17 pgs: 17 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-15T18:41:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:36 smithi082 ceph-mon[44206]: osdmap e725: 12 total, 12 up, 12 in 2024-04-15T18:41:37.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:41:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:36 smithi049 ceph-mon[40087]: pgmap v1867: 17 pgs: 17 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-15T18:41:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:36 smithi049 ceph-mon[40087]: osdmap e725: 12 total, 12 up, 12 in 2024-04-15T18:41:37.355 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:41:37.358 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 27556 27540 90.5841 56.9375 0.00242343 0.0110241 2024-04-15T18:41:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:36 smithi159 ceph-mon[45336]: pgmap v1867: 17 pgs: 17 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-04-15T18:41:37.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:36 smithi159 ceph-mon[45336]: osdmap e725: 12 total, 12 up, 12 in 2024-04-15T18:41:37.396 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:37.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:37.502 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:37.536 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:37.570 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:41:37.614 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:37.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:37.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:37.752 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:37.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:41:38.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:41:38.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:37 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:38.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:41:38.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:37 smithi049 ceph-mon[40087]: osdmap e726: 12 total, 12 up, 12 in 2024-04-15T18:41:38.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:41:38.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:37 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:38.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:41:38.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:37 smithi082 ceph-mon[44206]: osdmap e726: 12 total, 12 up, 12 in 2024-04-15T18:41:38.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:41:38.370727+0000 min lat: 0.00103313 max lat: 0.61683 avg lat: 0.0110241 2024-04-15T18:41:38.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:41:38.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 29028 29012 90.6545 92 0.00258365 0.0110241 2024-04-15T18:41:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:41:38.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:37 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:41:38.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3353549860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:41:38.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:37 smithi159 ceph-mon[45336]: osdmap e726: 12 total, 12 up, 12 in 2024-04-15T18:41:38.993 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:38 smithi082 ceph-mon[44206]: pgmap v1869: 17 pgs: 17 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:41:39.104 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:39.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:39.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:39.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:38 smithi049 ceph-mon[40087]: pgmap v1869: 17 pgs: 17 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:41:39.241 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:39.264 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:39.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:39.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 30513 30497 90.7569 92.8125 0.0022635 0.0110095 2024-04-15T18:41:39.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:39.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:38 smithi159 ceph-mon[45336]: pgmap v1869: 17 pgs: 17 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-04-15T18:41:39.414 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:39.418 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:41:39.418 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77931 7.7 7.ff006b77 (undecoded) ondisk+retry+write+known_if_redirected e647)", 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.492891+0000", 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.93421785300001, 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.99671738200000004, 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.419 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77931 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492891+0000", 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492891+0000", 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.420 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492891+0000", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0500000000000004e-07 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492939+0000", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8248999999999998e-05 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492947+0000", 2024-04-15T18:41:39.421 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2699999999999999e-06 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.492952+0000", 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2689999999999999e-06 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.493005+0000", 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3149999999999998e-05 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.493014+0000", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4250000000000004e-06 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489593+0000", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.996578874 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489608+0000", 2024-04-15T18:41:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.464e-05 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77949 7.d 7.2db84f2d (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.497493+0000", 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.929614899, 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.99212266599999999, 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.424 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77949 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497493+0000", 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497493+0000", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497494+0000", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0300000000000001e-06 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497543+0000", 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8848999999999999e-05 2024-04-15T18:41:39.426 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497554+0000", 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0213e-05 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497559+0000", 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5539999999999997e-06 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.427 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497603+0000", 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4292000000000002e-05 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.497609+0000", 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.054e-06 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489606+0000", 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.99199608699999997 2024-04-15T18:41:39.428 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489616+0000", 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0587e-05 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77957 7.7 7.51fe96b7 (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.504318+0000", 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.922790429, 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.98532516400000003, 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.429 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77957 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504318+0000", 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.430 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504318+0000", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504319+0000", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.387e-06 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504368+0000", 2024-04-15T18:41:39.431 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8205999999999999e-05 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504382+0000", 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.433e-05 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504389+0000", 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0559999999999999e-06 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.432 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504435+0000", 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6567999999999999e-05 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.504443+0000", 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5989999999999996e-06 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489636+0000", 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.98519308299999997 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.433 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489643+0000", 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9349999999999997e-06 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77962 7.7 7.9f7f9687 (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.509947+0000", 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.91716090400001, 2024-04-15T18:41:39.434 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.97974002299999996, 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77962 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.509947+0000", 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.435 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.509947+0000", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.509949+0000", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.336e-06 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.510012+0000", 2024-04-15T18:41:39.436 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3662000000000001e-05 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.510028+0000", 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5133000000000001e-05 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.510038+0000", 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0759e-05 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.437 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.510098+0000", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9848e-05 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.510109+0000", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.049e-05 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489675+0000", 2024-04-15T18:41:39.438 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.97956635599999997 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489687+0000", 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2439000000000001e-05 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77967 7.7 7.958c5537 (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.514492+0000", 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.91261615100001, 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.97521882400000004, 2024-04-15T18:41:39.439 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77967 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514492+0000", 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.440 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514492+0000", 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514493+0000", 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2669999999999999e-06 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.441 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514563+0000", 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9920999999999997e-05 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514574+0000", 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1089e-05 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514581+0000", 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0410000000000003e-06 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.442 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514624+0000", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3886e-05 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.514630+0000", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0769999999999996e-06 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489703+0000", 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.975072402 2024-04-15T18:41:39.443 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489711+0000", 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1410000000000005e-06 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77975 7.7 7.cbb12c77 (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.524661+0000", 2024-04-15T18:41:39.444 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.902447631, 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.96509872799999996, 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77975 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524661+0000", 2024-04-15T18:41:39.445 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524661+0000", 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524662+0000", 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3900000000000003e-07 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.446 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524707+0000", 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5631999999999999e-05 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524718+0000", 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1028e-05 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524725+0000", 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6730000000000003e-06 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.447 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524780+0000", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4814000000000001e-05 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524788+0000", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4919999999999993e-06 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489747+0000", 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.96495866500000005 2024-04-15T18:41:39.448 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489759+0000", 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2485e-05 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77976 7.d 7.b825d4bd (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.524840+0000", 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.90226842300001, 2024-04-15T18:41:39.449 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.96492205600000003, 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77976 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524840+0000", 2024-04-15T18:41:39.450 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524840+0000", 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524840+0000", 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1799999999999995e-07 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.451 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524903+0000", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2359000000000004e-05 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524910+0000", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0299999999999996e-06 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524915+0000", 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0649999999999998e-06 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.452 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524960+0000", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5108000000000002e-05 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.524967+0000", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9700000000000002e-06 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489753+0000", 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.96478587500000001 2024-04-15T18:41:39.453 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489762+0000", 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1309999999999995e-06 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77986 7.7 7.c088b2d7 (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:52.744177+0000", 2024-04-15T18:41:39.454 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.6829309, 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.74564108399999995, 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77986 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.455 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744177+0000", 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744177+0000", 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.456 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744179+0000", 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4330000000000001e-06 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744227+0000", 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8569000000000002e-05 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744240+0000", 2024-04-15T18:41:39.457 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2099000000000001e-05 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744246+0000", 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4359999999999999e-06 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744295+0000", 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8535000000000002e-05 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.458 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:52.744304+0000", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7249999999999992e-06 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489810+0000", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.74550554999999996 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489819+0000", 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7369999999999995e-06 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:77998 7.d 7.4564824d (undecoded) ondisk+write+known_if_redirected e647)", 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:53.170261+0000", 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 226.25684719399999, 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31954211100000002, 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.460 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 77998 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170261+0000", 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170261+0000", 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.461 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170263+0000", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.688e-06 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170355+0000", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2231000000000005e-05 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170374+0000", 2024-04-15T18:41:39.462 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8691000000000001e-05 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170385+0000", 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1455e-05 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170472+0000", 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7213999999999995e-05 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for peered", 2024-04-15T18:41:39.463 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.170482+0000", 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8079999999999999e-06 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489795+0000", 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.319312282 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:53.489803+0000", 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7420000000000003e-06 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.464 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(7.d 651/648 [PushOp(7:b2a19f42:::benchmark_data_smithi159_149998_object3260:head, version: 626'182, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:b2a19f42:::benchmark_data_smithi159_149998_object3260:head@626'182, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:59.811669+0000", 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 219.615439568, 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15088143400000001, 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.465 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.811669+0000", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.811669+0000", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.811670+0000", 2024-04-15T18:41:39.466 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8400000000000003e-07 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.811705+0000", 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5271000000000002e-05 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.961311+0000", 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14960595700000001 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.467 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.961339+0000", 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7708999999999998e-05 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.961538+0000", 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019937500000000001 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.468 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.961547+0000", 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2699999999999993e-06 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:59.962550+0000", 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010029679999999999 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(7.7 662/648 [PushOp(7:eca93316:::benchmark_data_smithi159_149998_object47196:head, version: 637'2948, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:eca93316:::benchmark_data_smithi159_149998_object47196:head@637'2948, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:41:39.469 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:38:56.115467+0000", 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 163.31164090799999, 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.152500149, 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.115467+0000", 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.470 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.115467+0000", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.115469+0000", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.567e-06 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.115524+0000", 2024-04-15T18:41:39.471 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4559999999999999e-05 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.265518+0000", 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14999475000000001 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.265534+0000", 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5634e-05 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.472 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.265591+0000", 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7250000000000002e-05 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.265600+0000", 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8610000000000004e-06 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.267968+0000", 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023675269999999999 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.473 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(7.c 662/648 [PushOp(7:318879cb:::benchmark_data_smithi159_149998_object41827:head, version: 634'2603, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:318879cb:::benchmark_data_smithi159_149998_object41827:head@634'2603, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:38:56.124465+0000", 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 163.30264303600001, 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14470509500000001, 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.474 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.124465+0000", 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.124465+0000", 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.124466+0000", 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8299999999999997e-07 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.475 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.124502+0000", 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5794999999999999e-05 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.267137+0000", 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14263530899999999 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.476 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.267152+0000", 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5339999999999999e-05 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.267222+0000", 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9500999999999997e-05 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.267231+0000", 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4620000000000002e-06 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.477 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:38:56.269170+0000", 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001939105 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(7.d 670/648 [PushOp(7:bf907c0e:::benchmark_data_smithi159_149998_object76331:head, version: 646'4710, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:bf907c0e:::benchmark_data_smithi159_149998_object76331:head@646'4710, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:23.959807+0000", 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.46730099800001, 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.204998181, 2024-04-15T18:41:39.478 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:23.959807+0000", 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:23.959807+0000", 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.479 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:23.959808+0000", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7599999999999996e-07 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:23.959844+0000", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5630999999999998e-05 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.162498+0000", 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.202654366 2024-04-15T18:41:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.162509+0000", 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0917000000000001e-05 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.162574+0000", 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5203000000000006e-05 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.481 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.162583+0000", 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1390000000000001e-06 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.164806+0000", 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022221490000000001 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.482 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:141980 7.7 7:ef7377c7:::benchmark_data_smithi159_149998_object45980:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:24.780674+0000", 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.646434744, 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.181199791, 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 141980 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.483 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780674+0000", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780674+0000", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780675+0000", 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.274e-06 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.484 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780683+0000", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9799999999999998e-06 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780700+0000", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7292999999999999e-05 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.780709+0000", 2024-04-15T18:41:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2369999999999998e-06 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.782966+0000", 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022563050000000001 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.783061+0000", 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5346000000000005e-05 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.486 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.783135+0000", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3858999999999999e-05 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.786041+0000", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029063499999999998 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.787714+0000", 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016729049999999999 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.487 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.961770+0000", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17405541299999999 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.961825+0000", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5547999999999997e-05 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.961873+0000", 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8281000000000001e-05 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.488 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:142005 7.7 7:e833adb5:::benchmark_data_smithi159_149998_object46005:head [delete] snapc 0=[] ondisk+write+known_if_redirected e671)", 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:24.813486+0000", 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.61362228199999, 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15085410699999999, 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.489 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 142005 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813486+0000", 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813486+0000", 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.490 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813487+0000", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4139999999999999e-06 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813495+0000", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9039999999999992e-06 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813510+0000", 2024-04-15T18:41:39.491 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5034e-05 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813519+0000", 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1340000000000004e-06 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813579+0000", 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0149000000000003e-05 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.492 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813691+0000", 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112268 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.813794+0000", 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103191 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.817869+0000", 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040751240000000003 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.493 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.818929+0000", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010599489999999999 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.964278+0000", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14534850299999999 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.964308+0000", 2024-04-15T18:41:39.494 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0620000000000002e-05 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:24.964340+0000", 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1816999999999997e-05 2024-04-15T18:41:39.495 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.497 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.497 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.497 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:177926 7.d 7:b0f7583d:::benchmark_data_smithi159_149998_object81926:head [delete] snapc 0=[] ondisk+write+known_if_redirected e692)", 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:16.037664+0000", 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.389444570999999, 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16431611500000001, 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.498 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 177926 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037664+0000", 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037664+0000", 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.499 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037665+0000", 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.237e-06 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037670+0000", 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1399999999999999e-06 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.500 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037681+0000", 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0543000000000001e-05 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037687+0000", 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9009999999999999e-06 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037743+0000", 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6379999999999999e-05 2024-04-15T18:41:39.501 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.037890+0000", 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000147035 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.038045+0000", 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015499200000000001 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:41:39.502 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.041467+0000", 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034219490000000001 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.048720+0000", 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0072532289999999999 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201903+0000", 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15318300000000001 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.503 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201938+0000", 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4749999999999998e-05 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201980+0000", 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1959000000000003e-05 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.504 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:177926 7.d e692/672 ondisk, result = 0)", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:16.045248+0000", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 83.381860601, 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15673673699999999, 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.045248+0000", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.505 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.045248+0000", 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.045249+0000", 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.158e-06 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.045255+0000", 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4819999999999999e-06 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.506 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201848+0000", 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.156592958 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201855+0000", 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.319e-06 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.507 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201895+0000", 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0120000000000002e-05 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201900+0000", 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0039999999999999e-06 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:16.201984+0000", 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4696000000000004e-05 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.508 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:179281 7.d 7:b94c71dc:::benchmark_data_smithi159_149998_object83281:head [delete] snapc 0=[] ondisk+write+known_if_redirected e693)", 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:17.921003+0000", 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.506105040999998, 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16404941000000001, 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179281 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.509 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921003+0000", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921003+0000", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921004+0000", 2024-04-15T18:41:39.510 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5900000000000005e-07 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921009+0000", 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8400000000000002e-06 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921021+0000", 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1861e-05 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.511 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921027+0000", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.699e-06 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921067+0000", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0105999999999999e-05 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921176+0000", 2024-04-15T18:41:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010904399999999999 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.921286+0000", 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110099 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.924680+0000", 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033935879999999999 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.513 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.933042+0000", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0083623699999999992 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.084971+0000", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.151929119 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.085007+0000", 2024-04-15T18:41:39.514 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6235999999999997e-05 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.085053+0000", 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5488999999999998e-05 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:179281 7.d e693/672 ondisk, result = 0)", 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:17.932136+0000", 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 81.494972418000003, 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15292224099999999, 2024-04-15T18:41:39.515 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.932136+0000", 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.932136+0000", 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.516 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.932137+0000", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1299999999999999e-07 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:17.932141+0000", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8439999999999997e-06 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.084932+0000", 2024-04-15T18:41:39.517 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15279077599999999 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.084938+0000", 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2549999999999997e-06 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.084964+0000", 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6846e-05 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.518 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.084968+0000", 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5580000000000001e-06 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:18.085058+0000", 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0248999999999996e-05 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:191961 7.d 7:b225a1f7:::benchmark_data_smithi159_149998_object95961:head [delete] snapc 0=[] ondisk+write+known_if_redirected e700)", 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:35.873474+0000", 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.553634432999999, 2024-04-15T18:41:39.519 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43930228199999999, 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 191961 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873474+0000", 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.520 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873474+0000", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873475+0000", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.540000000000001e-07 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873480+0000", 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8629999999999999e-06 2024-04-15T18:41:39.521 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873491+0000", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1316e-05 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873498+0000", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9419999999999998e-06 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873553+0000", 2024-04-15T18:41:39.522 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5399000000000003e-05 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873678+0000", 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012418800000000001 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.873803+0000", 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012491600000000001 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:41:39.523 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.876986+0000", 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003183136 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.879073+0000", 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020876039999999998 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.312671+0000", 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.43359777199999999 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.524 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.312746+0000", 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5277000000000004e-05 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.312776+0000", 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9915000000000001e-05 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:41:39.525 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:41:39.526 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:39.527 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:41:39.528 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:41:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:39.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:39.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:39.744 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:39.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:39.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:39.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:39.916 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:39.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:39.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:40.004 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:40.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:40.081 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:40.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:39 smithi159 ceph-mon[45336]: from='client.26915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:40.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:39 smithi082 ceph-mon[44206]: from='client.26915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:40.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:40.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:40.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:41:40.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:39 smithi049 ceph-mon[40087]: from='client.26915 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:40.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:40.326 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:40.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 32295 32279 91.6937 111.375 0.00812694 0.0108996 2024-04-15T18:41:40.533 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:40.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:40.578 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1870,"stamp":"2024-04-15T18:41:37.576025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1557201440,"num_objects":23754,"num_object_clones":0,"num_object_copies":71262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23754,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":23933,"num_write_kb":1523066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23891,"ondisk_log_size":23891,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10396728,"kb_used_data":4716292,"kb_used_omap":287,"kb_used_meta":5680032,"kb_avail":1114545096,"statfs":{"total":1151940427776,"available":1141294178304,"internally_reserved":0,"allocated":4829483008,"data_stored":4825114846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":294247,"internal_metadata":5816353433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":487325696,"num_objects":7436,"num_object_clones":0,"num_object_copies":22308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7436,"num_write_kb":475904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.006286"},"pg_stats":[{"pgid":"8.f","version":"724'1584","reported_seq":1623,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:34.894709+0000","last_change":"2024-04-15T18:41:02.832876+0000","last_active":"2024-04-15T18:41:34.894709+0000","last_peered":"2024-04-15T18:41:34.894709+0000","last_clean":"2024-04-15T18:41:34.894709+0000","last_became_active":"2024-04-15T18:41:02.832590+0000","last_became_peered":"2024-04-15T18:41:02.832590+0000","last_unstale":"2024-04-15T18:41:34.894709+0000","last_undegraded":"2024-04-15T18:41:34.894709+0000","last_fullsized":"2024-04-15T18:41:34.894709+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:30.026180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"725'1738","reported_seq":1777,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-04-15T18:41:37.258040+0000","last_change":"2024-04-15T18:41:02.884857+0000","last_active":"2024-04-15T18:41:37.258040+0000","last_peered":"2024-04-15T18:41:37.258040+0000","last_clean":"2024-04-15T18:41:37.258040+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:41:37.258040+0000","last_undegraded":"2024-04-15T18:41:37.258040+0000","last_fullsized":"2024-04-15T18:41:37.258040+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1738,"log_dups_size":0,"ondisk_log_size":1738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:25.318932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1738,"num_write_kb":111232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"724'1518","reported_seq":1557,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:34.894016+0000","last_change":"2024-04-15T18:41:02.832815+0000","last_active":"2024-04-15T18:41:34.894016+0000","last_peered":"2024-04-15T18:41:34.894016+0000","last_clean":"2024-04-15T18:41:34.894016+0000","last_became_active":"2024-04-15T18:41:02.832496+0000","last_became_peered":"2024-04-15T18:41:02.832496+0000","last_unstale":"2024-04-15T18:41:34.894016+0000","last_undegraded":"2024-04-15T18:41:34.894016+0000","last_fullsized":"2024-04-15T18:41:34.894016+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:04.892005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1518,"num_write_kb":97152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"725'1767","reported_seq":1808,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-04-15T18:41:37.565731+0000","last_change":"2024-04-15T18:41:02.835975+0000","last_active":"2024-04-15T18:41:37.565731+0000","last_peered":"2024-04-15T18:41:37.565731+0000","last_clean":"2024-04-15T18:41:37.565731+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:41:37.565731+0000","last_undegraded":"2024-04-15T18:41:37.565731+0000","last_fullsized":"2024-04-15T18:41:37.565731+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1767,"log_dups_size":0,"ondisk_log_size":1767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:13.946789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115802112,"num_objects":1767,"num_object_clones":0,"num_object_copies":5301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1767,"num_write_kb":113088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"724'1557","reported_seq":1596,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:35.330278+0000","last_change":"2024-04-15T18:41:02.837566+0000","last_active":"2024-04-15T18:41:35.330278+0000","last_peered":"2024-04-15T18:41:35.330278+0000","last_clean":"2024-04-15T18:41:35.330278+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:41:35.330278+0000","last_undegraded":"2024-04-15T18:41:35.330278+0000","last_fullsized":"2024-04-15T18:41:35.330278+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:12.482320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102039552,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1557,"num_write_kb":99648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"724'1566","reported_seq":1606,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:35.325057+0000","last_change":"2024-04-15T18:41:02.832540+0000","last_active":"2024-04-15T18:41:35.325057+0000","last_peered":"2024-04-15T18:41:35.325057+0000","last_clean":"2024-04-15T18:41:35.325057+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:41:35.325057+0000","last_undegraded":"2024-04-15T18:41:35.325057+0000","last_fullsized":"2024-04-15T18:41:35.325057+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:06.502180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102629376,"num_objects":1566,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1566,"num_write_kb":100224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"724'1369","reported_seq":1407,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:32.840207+0000","last_change":"2024-04-15T18:41:02.836328+0000","last_active":"2024-04-15T18:41:32.840207+0000","last_peered":"2024-04-15T18:41:32.840207+0000","last_clean":"2024-04-15T18:41:32.840207+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:41:32.840207+0000","last_undegraded":"2024-04-15T18:41:32.840207+0000","last_fullsized":"2024-04-15T18:41:32.840207+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:19.394707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1369,"num_write_kb":87616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"724'1432","reported_seq":1472,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:32.588681+0000","last_change":"2024-04-15T18:41:02.835362+0000","last_active":"2024-04-15T18:41:32.588681+0000","last_peered":"2024-04-15T18:41:32.588681+0000","last_clean":"2024-04-15T18:41:32.588681+0000","last_became_active":"2024-04-15T18:41:02.835074+0000","last_became_peered":"2024-04-15T18:41:02.835074+0000","last_unstale":"2024-04-15T18:41:32.588681+0000","last_undegraded":"2024-04-15T18:41:32.588681+0000","last_fullsized":"2024-04-15T18:41:32.588681+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1432,"log_dups_size":0,"ondisk_log_size":1432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:10.433823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93847552,"num_objects":1432,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":91648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"724'1559","reported_seq":1597,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:34.900816+0000","last_change":"2024-04-15T18:41:02.833708+0000","last_active":"2024-04-15T18:41:34.900816+0000","last_peered":"2024-04-15T18:41:34.900816+0000","last_clean":"2024-04-15T18:41:34.900816+0000","last_became_active":"2024-04-15T18:41:02.833455+0000","last_became_peered":"2024-04-15T18:41:02.833455+0000","last_unstale":"2024-04-15T18:41:34.900816+0000","last_undegraded":"2024-04-15T18:41:34.900816+0000","last_fullsized":"2024-04-15T18:41:34.900816+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:03.750844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102170624,"num_objects":1559,"num_object_clones":0,"num_object_copies":4677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1559,"num_write_kb":99776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"724'1322","reported_seq":1360,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:33.280908+0000","last_change":"2024-04-15T18:41:02.834566+0000","last_active":"2024-04-15T18:41:33.280908+0000","last_peered":"2024-04-15T18:41:33.280908+0000","last_clean":"2024-04-15T18:41:33.280908+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:41:33.280908+0000","last_undegraded":"2024-04-15T18:41:33.280908+0000","last_fullsized":"2024-04-15T18:41:33.280908+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1322,"log_dups_size":0,"ondisk_log_size":1322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:03.614389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1322,"num_write_kb":84608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"724'1288","reported_seq":1327,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:32.729826+0000","last_change":"2024-04-15T18:41:02.835990+0000","last_active":"2024-04-15T18:41:32.729826+0000","last_peered":"2024-04-15T18:41:32.729826+0000","last_clean":"2024-04-15T18:41:32.729826+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:41:32.729826+0000","last_undegraded":"2024-04-15T18:41:32.729826+0000","last_fullsized":"2024-04-15T18:41:32.729826+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:18.671045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":82432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"724'1495","reported_seq":1533,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:34.911597+0000","last_change":"2024-04-15T18:41:02.834723+0000","last_active":"2024-04-15T18:41:34.911597+0000","last_peered":"2024-04-15T18:41:34.911597+0000","last_clean":"2024-04-15T18:41:34.911597+0000","last_became_active":"2024-04-15T18:41:02.834497+0000","last_became_peered":"2024-04-15T18:41:02.834497+0000","last_unstale":"2024-04-15T18:41:34.911597+0000","last_undegraded":"2024-04-15T18:41:34.911597+0000","last_fullsized":"2024-04-15T18:41:34.911597+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:21.549776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97976320,"num_objects":1495,"num_object_clones":0,"num_object_copies":4485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":95680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"724'1455","reported_seq":1492,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:33.305540+0000","last_change":"2024-04-15T18:41:02.884780+0000","last_active":"2024-04-15T18:41:33.305540+0000","last_peered":"2024-04-15T18:41:33.305540+0000","last_clean":"2024-04-15T18:41:33.305540+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:41:33.305540+0000","last_undegraded":"2024-04-15T18:41:33.305540+0000","last_fullsized":"2024-04-15T18:41:33.305540+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:29.461596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95354880,"num_objects":1455,"num_object_clones":0,"num_object_copies":4365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1455,"num_write_kb":93120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"724'1309","reported_seq":1348,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:32.583452+0000","last_change":"2024-04-15T18:41:02.836317+0000","last_active":"2024-04-15T18:41:32.583452+0000","last_peered":"2024-04-15T18:41:32.583452+0000","last_clean":"2024-04-15T18:41:32.583452+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:41:32.583452+0000","last_undegraded":"2024-04-15T18:41:32.583452+0000","last_fullsized":"2024-04-15T18:41:32.583452+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:15.816154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85786624,"num_objects":1309,"num_object_clones":0,"num_object_copies":3927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1309,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"724'1451","reported_seq":1490,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:32.844016+0000","last_change":"2024-04-15T18:41:02.836371+0000","last_active":"2024-04-15T18:41:32.844016+0000","last_peered":"2024-04-15T18:41:32.844016+0000","last_clean":"2024-04-15T18:41:32.844016+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:41:32.844016+0000","last_undegraded":"2024-04-15T18:41:32.844016+0000","last_fullsized":"2024-04-15T18:41:32.844016+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1451,"log_dups_size":0,"ondisk_log_size":1451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:07.557097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":4353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1451,"num_write_kb":92864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"724'1342","reported_seq":1381,"reported_epoch":724,"state":"active+clean","last_fresh":"2024-04-15T18:41:33.286378+0000","last_change":"2024-04-15T18:41:02.884642+0000","last_active":"2024-04-15T18:41:33.286378+0000","last_peered":"2024-04-15T18:41:33.286378+0000","last_clean":"2024-04-15T18:41:33.286378+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:41:33.286378+0000","last_undegraded":"2024-04-15T18:41:33.286378+0000","last_fullsized":"2024-04-15T18:41:33.286378+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":1342,"log_dups_size":0,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:04.966017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":85888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1794,"reported_epoch":725,"state":"active+clean","last_fresh":"2024-04-15T18:41:35.953442+0000","last_change":"2024-04-15T18:41:00.313929+0000","last_active":"2024-04-15T18:41:35.953442+0000","last_peered":"2024-04-15T18:41:35.953442+0000","last_clean":"2024-04-15T18:41:35.953442+0000","last_became_active":"2024-04-15T18:37:56.526550+0000","last_became_peered":"2024-04-15T18:37:56.526550+0000","last_unstale":"2024-04-15T18:41:35.953442+0000","last_undegraded":"2024-04-15T18:41:35.953442+0000","last_fullsized":"2024-04-15T18:41:35.953442+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:28.620159+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1556611072,"num_objects":23752,"num_object_clones":0,"num_object_copies":71256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23752,"num_write_kb":1520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4754898944,"data_stored":4754898944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23752,"ondisk_log_size":23752,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115432,"kb_used_data":492832,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92629720,"statfs":{"total":95995035648,"available":94852833280,"internally_reserved":0,"allocated":504659968,"data_stored":504293638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":637532612},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187851,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956384,"kb_used_data":528152,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92788768,"statfs":{"total":95995035648,"available":95015698432,"internally_reserved":0,"allocated":540827648,"data_stored":540464890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514898,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1214636,"kb_used_data":660068,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92530516,"statfs":{"total":95995035648,"available":94751248384,"internally_reserved":0,"allocated":675909632,"data_stored":675542376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874643,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":670500,"kb_used_data":288088,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93074652,"statfs":{"total":95995035648,"available":95308443648,"internally_reserved":0,"allocated":295002112,"data_stored":294639354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168984,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663416,"kb_used_data":212972,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93081736,"statfs":{"total":95995035648,"available":95315697664,"internally_reserved":0,"allocated":218083328,"data_stored":217712520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":461240778},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176876,"kb_used_data":1440,"kb_used_omap":111,"kb_used_meta":175312,"kb_avail":93568276,"statfs":{"total":95995035648,"available":95813914624,"internally_reserved":0,"allocated":1474560,"data_stored":1108258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":179520155},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575290,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137364,"kb_used_data":578636,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92607788,"statfs":{"total":95995035648,"available":94830374912,"internally_reserved":0,"allocated":592523264,"data_stored":592165054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913248,"kb_used_data":304536,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92831904,"statfs":{"total":95995035648,"available":95059869696,"internally_reserved":0,"allocated":311844864,"data_stored":311482106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194063,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907308,"kb_used_data":542116,"kb_used_omap":114,"kb_used_meta":365069,"kb_avail":92837844,"statfs":{"total":95995035648,"available":95065952256,"internally_reserved":0,"allocated":555126784,"data_stored":554759528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":373830897},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609192,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942804,"kb_used_data":326540,"kb_used_omap":44,"kb_used_meta":616211,"kb_avail":92802348,"statfs":{"total":95995035648,"available":95029604352,"internally_reserved":0,"allocated":334376960,"data_stored":334018750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":631000618},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986599,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897444,"kb_used_data":347032,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92847708,"statfs":{"total":95995035648,"available":95076052992,"internally_reserved":0,"allocated":355360768,"data_stored":354998010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703493,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801316,"kb_used_data":433880,"kb_used_omap":4,"kb_used_meta":367419,"kb_avail":92943836,"statfs":{"total":95995035648,"available":95174488064,"internally_reserved":0,"allocated":444293120,"data_stored":443930362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":376237761},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315555840,"data_stored":315555840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":530710528,"data_stored":530710528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":564330496,"data_stored":564330496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277282816,"data_stored":277282816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":355532800,"data_stored":355532800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":444203008,"data_stored":444203008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":218693632,"data_stored":218693632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":309919744,"data_stored":309919744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":683606016,"data_stored":683606016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":547028992,"data_stored":547028992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":508035072,"data_stored":508035072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:41:40.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:41:40.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:40.704 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:40.789 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:40.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:40.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:40.930 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:40.960 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:40.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:41.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:41.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:40 smithi049 ceph-mon[40087]: pgmap v1871: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:41:41.098 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:40 smithi082 ceph-mon[44206]: pgmap v1871: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:41:41.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 33708 33692 91.5462 88.3125 0.0159867 0.0108899 2024-04-15T18:41:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:40 smithi159 ceph-mon[45336]: pgmap v1871: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:41:42.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:41:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:42.362 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 35186 35170 91.5804 92.375 0.0454624 0.0109138 2024-04-15T18:41:43.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:42 smithi049 ceph-mon[40087]: pgmap v1872: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:41:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:42 smithi082 ceph-mon[44206]: pgmap v1872: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:41:43.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 36729 36713 91.7744 96.4375 0.0320307 0.0108733 2024-04-15T18:41:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:42 smithi159 ceph-mon[45336]: pgmap v1872: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-04-15T18:41:43.455 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:43.457 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:43.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:43.560 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:43.605 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:43.633 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:43.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:43.738 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:43.772 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:43.796 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:43.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:43.806 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:43.900 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:43.937 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:43.961 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:43.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:44.066 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:44.100 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.122 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:44.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:44.269 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.343 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:44.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 38056 38040 91.4343 82.9375 0.0568353 0.0109276 2024-04-15T18:41:44.448 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:44.484 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.508 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:44.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:41:44.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.667 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:41:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:41:44.718 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:44.748 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:44.771 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:44.780 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:44.812 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:44.812 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.838 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:44.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:44.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:44.907 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:44.938 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:44.942 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:44.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:44.972 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:44.994 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:44.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:45.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:45.034 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:45.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:45.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:45.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:45.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:45.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:45.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:45.099 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:45.127 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:45.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:44 smithi159 ceph-mon[45336]: pgmap v1873: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-15T18:41:45.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3491761599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:45.149 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:45.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:45.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:44 smithi049 ceph-mon[40087]: pgmap v1873: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-15T18:41:45.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3491761599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:45.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:45.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:44 smithi082 ceph-mon[44206]: pgmap v1873: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-04-15T18:41:45.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3491761599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:45.285 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:45.309 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:45.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:45.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 39344 39328 91.029 80.5 0.0138686 0.0109781 2024-04-15T18:41:45.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:45.441 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:45.464 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:45.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:45.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:45.604 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 15 40854 40839 91.1505 94.4375 0.00611673 0.0109638 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:47.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:47.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:46 smithi049 ceph-mon[40087]: pgmap v1874: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-15T18:41:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:46 smithi082 ceph-mon[44206]: pgmap v1874: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-15T18:41:47.359 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 41935 41919 90.3346 67.5 0.00257923 0.0110159 2024-04-15T18:41:47.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:46 smithi159 ceph-mon[45336]: pgmap v1874: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-04-15T18:41:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:47.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:47.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:48.017 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.043 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:48.148 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:41:48.177 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.199 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:41:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:41:48.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:48.343 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 43390 43374 90.3538 90.9375 0.0213544 0.0110614 2024-04-15T18:41:48.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:48.475 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:48.510 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.540 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:48.645 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:48.672 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.697 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:48.803 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:48.842 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:48.873 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:48.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:48.978 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:49.016 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.047 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:49.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:48 smithi049 ceph-mon[40087]: pgmap v1875: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:41:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:48 smithi159 ceph-mon[45336]: pgmap v1875: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:41:49.152 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:49.180 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.204 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:49.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:41:49.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:48 smithi082 ceph-mon[44206]: pgmap v1875: 17 pgs: 17 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:41:49.308 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:49.334 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.353 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:49.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 44691 44675 90.0619 81.3125 0.0895527 0.0110929 2024-04-15T18:41:49.458 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:49.487 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.507 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:49.610 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:41:49.610 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:41:49.612 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:49.637 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:49.760 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:41:49.796 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:49.826 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:41:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:49.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:41:49.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:49.978 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:50.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:50.338 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:41:50.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 45842 45826 89.4953 71.9375 0.0043444 0.0111681 2024-04-15T18:41:50.373 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:50.396 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:41:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:50.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:41:50.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:50.542 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:50.659 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:41:50.663 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:41:51.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:50 smithi049 ceph-mon[40087]: pgmap v1876: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:50 smithi082 ceph-mon[44206]: pgmap v1876: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:51.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 47256 47240 89.4612 88.375 0.00199315 0.0111655 2024-04-15T18:41:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:50 smithi159 ceph-mon[45336]: pgmap v1876: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:41:51.544 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:51.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:41:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:51.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:51.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:51.678 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:51.700 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:51.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:51.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:51.833 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:51.852 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:51.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:51.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:51 smithi082 ceph-mon[44206]: from='client.17127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:51.956 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:51.992 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.015 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:52.119 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:52.149 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.173 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:52.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:51 smithi049 ceph-mon[40087]: from='client.17127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:52.276 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:52.278 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:52.287 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1876,"stamp":"2024-04-15T18:41:49.573145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2707161632,"num_objects":41301,"num_object_clones":0,"num_object_copies":123903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41301,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":41480,"num_write_kb":2646074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41438,"ondisk_log_size":41438,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13821628,"kb_used_data":7987096,"kb_used_omap":287,"kb_used_meta":5834144,"kb_avail":1111120196,"statfs":{"total":1151940427776,"available":1137787080704,"internally_reserved":0,"allocated":8178786304,"data_stored":8174313988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":294247,"internal_metadata":5974164121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1149960192,"num_objects":17547,"num_object_clones":0,"num_object_copies":52641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17547,"num_write_kb":1123008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.997119"},"pg_stats":[{"pgid":"8.f","version":"726'2483","reported_seq":2524,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:44.908952+0000","last_change":"2024-04-15T18:41:02.832876+0000","last_active":"2024-04-15T18:41:44.908952+0000","last_peered":"2024-04-15T18:41:44.908952+0000","last_clean":"2024-04-15T18:41:44.908952+0000","last_became_active":"2024-04-15T18:41:02.832590+0000","last_became_peered":"2024-04-15T18:41:02.832590+0000","last_unstale":"2024-04-15T18:41:44.908952+0000","last_undegraded":"2024-04-15T18:41:44.908952+0000","last_fullsized":"2024-04-15T18:41:44.908952+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:30.026180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162725888,"num_objects":2483,"num_object_clones":0,"num_object_copies":7449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":158912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.e","version":"726'2655","reported_seq":2697,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.235290+0000","last_change":"2024-04-15T18:41:02.884857+0000","last_active":"2024-04-15T18:41:47.235290+0000","last_peered":"2024-04-15T18:41:47.235290+0000","last_clean":"2024-04-15T18:41:47.235290+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:41:47.235290+0000","last_undegraded":"2024-04-15T18:41:47.235290+0000","last_fullsized":"2024-04-15T18:41:47.235290+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2655,"log_dups_size":0,"ondisk_log_size":2655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:25.318932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":7965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":169920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"726'2435","reported_seq":2477,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:44.887898+0000","last_change":"2024-04-15T18:41:02.832815+0000","last_active":"2024-04-15T18:41:44.887898+0000","last_peered":"2024-04-15T18:41:44.887898+0000","last_clean":"2024-04-15T18:41:44.887898+0000","last_became_active":"2024-04-15T18:41:02.832496+0000","last_became_peered":"2024-04-15T18:41:02.832496+0000","last_unstale":"2024-04-15T18:41:44.887898+0000","last_undegraded":"2024-04-15T18:41:44.887898+0000","last_fullsized":"2024-04-15T18:41:44.887898+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2435,"log_dups_size":0,"ondisk_log_size":2435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:04.892005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159580160,"num_objects":2435,"num_object_clones":0,"num_object_copies":7305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2435,"num_write_kb":155840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"726'2646","reported_seq":2689,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.543483+0000","last_change":"2024-04-15T18:41:02.835975+0000","last_active":"2024-04-15T18:41:47.543483+0000","last_peered":"2024-04-15T18:41:47.543483+0000","last_clean":"2024-04-15T18:41:47.543483+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:41:47.543483+0000","last_undegraded":"2024-04-15T18:41:47.543483+0000","last_fullsized":"2024-04-15T18:41:47.543483+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2646,"log_dups_size":0,"ondisk_log_size":2646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:13.946789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173408256,"num_objects":2646,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2646,"num_write_kb":169344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"726'2466","reported_seq":2510,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:45.308848+0000","last_change":"2024-04-15T18:41:02.837566+0000","last_active":"2024-04-15T18:41:45.308848+0000","last_peered":"2024-04-15T18:41:45.308848+0000","last_clean":"2024-04-15T18:41:45.308848+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:41:45.308848+0000","last_undegraded":"2024-04-15T18:41:45.308848+0000","last_fullsized":"2024-04-15T18:41:45.308848+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2466,"log_dups_size":0,"ondisk_log_size":2466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:12.482320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":157824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"726'2440","reported_seq":2484,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:45.306367+0000","last_change":"2024-04-15T18:41:02.832540+0000","last_active":"2024-04-15T18:41:45.306367+0000","last_peered":"2024-04-15T18:41:45.306367+0000","last_clean":"2024-04-15T18:41:45.306367+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:41:45.306367+0000","last_undegraded":"2024-04-15T18:41:45.306367+0000","last_fullsized":"2024-04-15T18:41:45.306367+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2440,"log_dups_size":0,"ondisk_log_size":2440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:06.502180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159907840,"num_objects":2440,"num_object_clones":0,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2440,"num_write_kb":156160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"726'2648","reported_seq":2690,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.877714+0000","last_change":"2024-04-15T18:41:02.836328+0000","last_active":"2024-04-15T18:41:47.877714+0000","last_peered":"2024-04-15T18:41:47.877714+0000","last_clean":"2024-04-15T18:41:47.877714+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:41:47.877714+0000","last_undegraded":"2024-04-15T18:41:47.877714+0000","last_fullsized":"2024-04-15T18:41:47.877714+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2648,"log_dups_size":0,"ondisk_log_size":2648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:19.394707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173539328,"num_objects":2648,"num_object_clones":0,"num_object_copies":7944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2648,"num_write_kb":169472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"726'2708","reported_seq":2752,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.588625+0000","last_change":"2024-04-15T18:41:02.835362+0000","last_active":"2024-04-15T18:41:47.588625+0000","last_peered":"2024-04-15T18:41:47.588625+0000","last_clean":"2024-04-15T18:41:47.588625+0000","last_became_active":"2024-04-15T18:41:02.835074+0000","last_became_peered":"2024-04-15T18:41:02.835074+0000","last_unstale":"2024-04-15T18:41:47.588625+0000","last_undegraded":"2024-04-15T18:41:47.588625+0000","last_fullsized":"2024-04-15T18:41:47.588625+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:10.433823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177471488,"num_objects":2708,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2708,"num_write_kb":173312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"726'2457","reported_seq":2499,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:44.911585+0000","last_change":"2024-04-15T18:41:02.833708+0000","last_active":"2024-04-15T18:41:44.911585+0000","last_peered":"2024-04-15T18:41:44.911585+0000","last_clean":"2024-04-15T18:41:44.911585+0000","last_became_active":"2024-04-15T18:41:02.833455+0000","last_became_peered":"2024-04-15T18:41:02.833455+0000","last_unstale":"2024-04-15T18:41:44.911585+0000","last_undegraded":"2024-04-15T18:41:44.911585+0000","last_fullsized":"2024-04-15T18:41:44.911585+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2457,"log_dups_size":0,"ondisk_log_size":2457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:03.750844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161021952,"num_objects":2457,"num_object_clones":0,"num_object_copies":7371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2457,"num_write_kb":157248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"726'2618","reported_seq":2661,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:48.386797+0000","last_change":"2024-04-15T18:41:02.834566+0000","last_active":"2024-04-15T18:41:48.386797+0000","last_peered":"2024-04-15T18:41:48.386797+0000","last_clean":"2024-04-15T18:41:48.386797+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:41:48.386797+0000","last_undegraded":"2024-04-15T18:41:48.386797+0000","last_fullsized":"2024-04-15T18:41:48.386797+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2618,"log_dups_size":0,"ondisk_log_size":2618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:03.614389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171573248,"num_objects":2618,"num_object_clones":0,"num_object_copies":7854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"726'2567","reported_seq":2610,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.862522+0000","last_change":"2024-04-15T18:41:02.835990+0000","last_active":"2024-04-15T18:41:47.862522+0000","last_peered":"2024-04-15T18:41:47.862522+0000","last_clean":"2024-04-15T18:41:47.862522+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:41:47.862522+0000","last_undegraded":"2024-04-15T18:41:47.862522+0000","last_fullsized":"2024-04-15T18:41:47.862522+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2567,"log_dups_size":0,"ondisk_log_size":2567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:18.671045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":7701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":164288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"726'2433","reported_seq":2476,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:44.900666+0000","last_change":"2024-04-15T18:41:02.834723+0000","last_active":"2024-04-15T18:41:44.900666+0000","last_peered":"2024-04-15T18:41:44.900666+0000","last_clean":"2024-04-15T18:41:44.900666+0000","last_became_active":"2024-04-15T18:41:02.834497+0000","last_became_peered":"2024-04-15T18:41:02.834497+0000","last_unstale":"2024-04-15T18:41:44.900666+0000","last_undegraded":"2024-04-15T18:41:44.900666+0000","last_fullsized":"2024-04-15T18:41:44.900666+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2433,"log_dups_size":0,"ondisk_log_size":2433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:21.549776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449088,"num_objects":2433,"num_object_clones":0,"num_object_copies":7299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":155712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.2","version":"726'2787","reported_seq":2830,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:48.393619+0000","last_change":"2024-04-15T18:41:02.884780+0000","last_active":"2024-04-15T18:41:48.393619+0000","last_peered":"2024-04-15T18:41:48.393619+0000","last_clean":"2024-04-15T18:41:48.393619+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:41:48.393619+0000","last_undegraded":"2024-04-15T18:41:48.393619+0000","last_fullsized":"2024-04-15T18:41:48.393619+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:29.461596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182648832,"num_objects":2787,"num_object_clones":0,"num_object_copies":8361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":178368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"726'2621","reported_seq":2665,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.588593+0000","last_change":"2024-04-15T18:41:02.836317+0000","last_active":"2024-04-15T18:41:47.588593+0000","last_peered":"2024-04-15T18:41:47.588593+0000","last_clean":"2024-04-15T18:41:47.588593+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:41:47.588593+0000","last_undegraded":"2024-04-15T18:41:47.588593+0000","last_fullsized":"2024-04-15T18:41:47.588593+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2621,"log_dups_size":0,"ondisk_log_size":2621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:15.816154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171769856,"num_objects":2621,"num_object_clones":0,"num_object_copies":7863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":167744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"726'2667","reported_seq":2710,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:47.867832+0000","last_change":"2024-04-15T18:41:02.836371+0000","last_active":"2024-04-15T18:41:47.867832+0000","last_peered":"2024-04-15T18:41:47.867832+0000","last_clean":"2024-04-15T18:41:47.867832+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:41:47.867832+0000","last_undegraded":"2024-04-15T18:41:47.867832+0000","last_fullsized":"2024-04-15T18:41:47.867832+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2667,"log_dups_size":0,"ondisk_log_size":2667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:07.557097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174784512,"num_objects":2667,"num_object_clones":0,"num_object_copies":8001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":170688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"726'2668","reported_seq":2711,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:48.390581+0000","last_change":"2024-04-15T18:41:02.884642+0000","last_active":"2024-04-15T18:41:48.390581+0000","last_peered":"2024-04-15T18:41:48.390581+0000","last_clean":"2024-04-15T18:41:48.390581+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:41:48.390581+0000","last_undegraded":"2024-04-15T18:41:48.390581+0000","last_fullsized":"2024-04-15T18:41:48.390581+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:04.966017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2668,"num_write_kb":170752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1796,"reported_epoch":726,"state":"active+clean","last_fresh":"2024-04-15T18:41:37.582287+0000","last_change":"2024-04-15T18:41:00.313929+0000","last_active":"2024-04-15T18:41:37.582287+0000","last_peered":"2024-04-15T18:41:37.582287+0000","last_clean":"2024-04-15T18:41:37.582287+0000","last_became_active":"2024-04-15T18:37:56.526550+0000","last_became_peered":"2024-04-15T18:37:56.526550+0000","last_unstale":"2024-04-15T18:41:37.582287+0000","last_undegraded":"2024-04-15T18:41:37.582287+0000","last_fullsized":"2024-04-15T18:41:37.582287+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:42:28.620159+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2706571264,"num_objects":41299,"num_object_clones":0,"num_object_copies":123897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41299,"num_write_kb":2643136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192917504,"data_stored":8192917504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41299,"ondisk_log_size":41299,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3702496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":94,"seq":403726926215,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1403968,"kb_used_data":781368,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92341184,"statfs":{"total":95995035648,"available":94557372416,"internally_reserved":0,"allocated":800120832,"data_stored":799745456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":637532612},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460472,"kb_used_data":1018352,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":92284680,"statfs":{"total":95995035648,"available":94499512320,"internally_reserved":0,"allocated":1042792448,"data_stored":1042420644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":452721088},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514900,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1609848,"kb_used_data":996976,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92135304,"statfs":{"total":95995035648,"available":94346551296,"internally_reserved":0,"allocated":1020903424,"data_stored":1020531620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":627571136},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874645,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838968,"kb_used_data":460720,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92906184,"statfs":{"total":95995035648,"available":95135932416,"internally_reserved":0,"allocated":471777280,"data_stored":471405476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518168986,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787712,"kb_used_data":327544,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92957440,"statfs":{"total":95995035648,"available":95188418560,"internally_reserved":0,"allocated":335405056,"data_stored":335029700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":471202250},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176900,"kb_used_data":1464,"kb_used_omap":111,"kb_used_meta":175312,"kb_avail":93568252,"statfs":{"total":95995035648,"available":95813890048,"internally_reserved":0,"allocated":1499136,"data_stored":1123788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":179520155},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575293,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619192,"kb_used_data":1046576,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92125960,"statfs":{"total":95995035648,"available":94336983040,"internally_reserved":0,"allocated":1071693824,"data_stored":1071322020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089976,"kb_used_data":472944,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92655176,"statfs":{"total":95995035648,"available":94878900224,"internally_reserved":0,"allocated":484294656,"data_stored":483922852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194065,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1196536,"kb_used_data":831344,"kb_used_omap":114,"kb_used_meta":365069,"kb_avail":92548616,"statfs":{"total":95995035648,"available":94769782784,"internally_reserved":0,"allocated":851296256,"data_stored":850924452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":373830897},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609195,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210936,"kb_used_data":540528,"kb_used_omap":44,"kb_used_meta":670355,"kb_avail":92534216,"statfs":{"total":95995035648,"available":94755037184,"internally_reserved":0,"allocated":553500672,"data_stored":553128868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":686444074},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986602,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216760,"kb_used_data":662192,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92528392,"statfs":{"total":95995035648,"available":94749073408,"internally_reserved":0,"allocated":678084608,"data_stored":677712804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703496,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210360,"kb_used_data":847088,"kb_used_omap":4,"kb_used_meta":363259,"kb_avail":92534792,"statfs":{"total":95995035648,"available":94755627008,"internally_reserved":0,"allocated":867418112,"data_stored":867046308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":371977921},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":489160704,"data_stored":489160704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1043464192,"data_stored":1043464192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":864092160,"data_stored":864092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":522846208,"data_stored":522846208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":691929088,"data_stored":691929088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":867172352,"data_stored":867172352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":336986112,"data_stored":336986112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":483590144,"data_stored":483590144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1027342336,"data_stored":1027342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1061027840,"data_stored":1061027840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":805306368,"data_stored":805306368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:41:52.288 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:41:52.303 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.323 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:52.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 48494 48478 89.1055 77.375 0.00290286 0.0112152 2024-04-15T18:41:52.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:51 smithi159 ceph-mon[45336]: from='client.17127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:41:52.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:52.455 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.477 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:52.583 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:41:52.622 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.652 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:41:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:41:52.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:52.785 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.807 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:52.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:52.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:52.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:52.973 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:41:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:52 smithi049 ceph-mon[40087]: pgmap v1877: 17 pgs: 17 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:41:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3881756106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:52 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:52 smithi082 ceph-mon[44206]: pgmap v1877: 17 pgs: 17 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:41:53.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3881756106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:52 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.296 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:53.313 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:41:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:53.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:41:53.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 49790 49774 88.8737 81 0.00181324 0.0112306 2024-04-15T18:41:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:52 smithi159 ceph-mon[45336]: pgmap v1877: 17 pgs: 17 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.40k op/s 2024-04-15T18:41:53.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3881756106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:52 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:41:53.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:41:53.449 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:53.529 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:41:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:53.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:41:53.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:53.677 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:53.709 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:53.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:53.813 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:53.841 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:53.861 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:53.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:53.952 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:53 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:53.952 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:53 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:41:53.952 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:53 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:53.952 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:53 smithi159 ceph-mon[45336]: osdmap e727: 12 total, 12 up, 12 in 2024-04-15T18:41:53.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:54.005 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:53 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:53 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:41:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:53 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:53 smithi049 ceph-mon[40087]: osdmap e727: 12 total, 12 up, 12 in 2024-04-15T18:41:54.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:53 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:41:54.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:53 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:41:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:53 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:41:54.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:53 smithi082 ceph-mon[44206]: osdmap e727: 12 total, 12 up, 12 in 2024-04-15T18:41:54.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 51109 51093 88.6947 82.4375 0.00602507 0.0112688 2024-04-15T18:41:54.862 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:54.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:54.968 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:55.021 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.055 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:55.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:55.190 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.212 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:55.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:54 smithi049 ceph-mon[40087]: pgmap v1879: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:41:55.215 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:54 smithi082 ceph-mon[44206]: pgmap v1879: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:41:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:55.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:55.352 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 52308 52292 88.3227 74.9375 0.00384298 0.0113144 2024-04-15T18:41:55.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:54 smithi159 ceph-mon[45336]: pgmap v1879: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-04-15T18:41:55.398 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:55.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:41:55.534 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.554 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:41:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:41:55.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:55.685 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.704 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:55.807 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:55.835 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:55.854 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:55.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:55.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:55.986 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:56.009 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:56.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:56.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:56.152 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:56.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:41:56.230 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:56.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:56.334 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:56.360 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 53300 53284 87.6299 62 0.00570564 0.0114063 2024-04-15T18:41:56.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:56.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:56.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:56.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:41:56.496 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:41:56.515 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:56.528 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:41:56.536 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:41:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:56.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:41:56.560 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:41:56.604 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:41:56.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:41:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:41:56.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:41:56.644 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:41:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:41:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:41:56.645 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:41:56.645 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:41:56.673 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:41:56.694 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:56.711 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:41:57.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:56 smithi049 ceph-mon[40087]: pgmap v1880: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:41:57.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/212760135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:57.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:56 smithi082 ceph-mon[44206]: pgmap v1880: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:41:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/212760135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 54484 54468 87.2802 74 0.0032427 0.011421 2024-04-15T18:41:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:56 smithi159 ceph-mon[45336]: pgmap v1880: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-04-15T18:41:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/212760135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:41:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:41:57.584 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:41:57.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:41:57.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:41:57.700 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:41:57.707 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:41:57.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:41:57.709 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osdmap e728: 12 total, 12 up, 12 in 2024-04-15T18:41:57.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:57.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:57 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osdmap e728: 12 total, 12 up, 12 in 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:57 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:41:58.372781+0000 min lat: 0.00103313 max lat: 0.61683 avg lat: 0.0114465 2024-04-15T18:41:58.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:41:58.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 55907 55891 87.3214 88.9375 0.012528 0.0114465 2024-04-15T18:41:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osdmap e728: 12 total, 12 up, 12 in 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:57 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:58.452 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:41:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:58.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:41:58.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:41:58.590 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:58.616 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:41:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:58.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:41:58.721 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:58.758 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:58.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:58.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:58.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:41:58.919 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:58.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:41:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:58.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:41:59.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:41:59.075 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:59.096 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:41:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:41:59.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:41:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:58 smithi159 ceph-mon[45336]: pgmap v1881: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:41:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:58 smithi049 ceph-mon[40087]: pgmap v1881: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.237 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:41:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:58 smithi082 ceph-mon[44206]: pgmap v1881: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-04-15T18:41:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:41:59.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 57103 57087 87.0146 74.75 0.00790906 0.0114875 2024-04-15T18:42:00.041 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:00.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:00.185 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:00.208 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:00.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:00.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:41:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:00.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:00.343 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:00.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 58461 58445 86.9635 84.875 0.0100745 0.0114938 2024-04-15T18:42:00.365 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:00.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:00.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:41:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:00.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:00.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:41:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:00.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:00.499 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:00.520 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:00.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:42:00.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:42:00.662 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:00.692 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:42:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:42:00.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:00.825 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:00.847 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:00.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:00.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:42:01.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:01.034 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:42:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:01.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:42:01.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:01.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:00 smithi082 ceph-mon[44206]: pgmap v1883: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:01.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:00 smithi082 ceph-mon[44206]: osdmap e729: 12 total, 12 up, 12 in 2024-04-15T18:42:01.172 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:01.193 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:01.195 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:01.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:00 smithi049 ceph-mon[40087]: pgmap v1883: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:00 smithi049 ceph-mon[40087]: osdmap e729: 12 total, 12 up, 12 in 2024-04-15T18:42:01.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:01.324 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:01.346 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:01.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:01.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 59708 59692 86.7534 77.9375 0.002971 0.0115168 2024-04-15T18:42:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:00 smithi159 ceph-mon[45336]: pgmap v1883: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:00 smithi159 ceph-mon[45336]: osdmap e729: 12 total, 12 up, 12 in 2024-04-15T18:42:01.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:01.497 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:01.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:01.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:02.001 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:42:02.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:42:02.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:02 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:02.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:02 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:02.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:02 smithi049 ceph-mon[40087]: osdmap e730: 12 total, 12 up, 12 in 2024-04-15T18:42:02.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3790074368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:02.117 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:02 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:02 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:02 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:02 smithi082 ceph-mon[44206]: osdmap e730: 12 total, 12 up, 12 in 2024-04-15T18:42:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3790074368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:02.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:02 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:02.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 60901 60885 86.4763 74.5625 0.019135 0.0115585 2024-04-15T18:42:02.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:02 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:02.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:02 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:02.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:02 smithi159 ceph-mon[45336]: osdmap e730: 12 total, 12 up, 12 in 2024-04-15T18:42:02.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3790074368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:02.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:02 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:42:03.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:03 smithi049 ceph-mon[40087]: pgmap v1886: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:42:03.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:03 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:03.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:03 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:42:03.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:03 smithi049 ceph-mon[40087]: osdmap e731: 12 total, 12 up, 12 in 2024-04-15T18:42:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:03 smithi049 ceph-mon[40087]: osdmap e732: 12 total, 12 up, 12 in 2024-04-15T18:42:03.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:03 smithi082 ceph-mon[44206]: pgmap v1886: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:42:03.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:03 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:03.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:03 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:42:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:03 smithi082 ceph-mon[44206]: osdmap e731: 12 total, 12 up, 12 in 2024-04-15T18:42:03.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:03 smithi082 ceph-mon[44206]: osdmap e732: 12 total, 12 up, 12 in 2024-04-15T18:42:03.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 62113 62097 86.2378 75.75 0.00460144 0.0115905 2024-04-15T18:42:03.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:03 smithi159 ceph-mon[45336]: pgmap v1886: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-04-15T18:42:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:03 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:03 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:42:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:03 smithi159 ceph-mon[45336]: osdmap e731: 12 total, 12 up, 12 in 2024-04-15T18:42:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:03 smithi159 ceph-mon[45336]: osdmap e732: 12 total, 12 up, 12 in 2024-04-15T18:42:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:04 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:04.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 63097 63081 85.6999 61.5 0.00699513 0.0116635 2024-04-15T18:42:04.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:04 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:04.375 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:04.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:04.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:04 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:04.484 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:42:04.529 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:04.558 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:42:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:04.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:42:04.668 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:42:04.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:04.728 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:42:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:04.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:42:04.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:04.871 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:04.893 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:04.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:04.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:05.024 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:05.065 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:05.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:05 smithi159 ceph-mon[45336]: pgmap v1889: 17 pgs: 17 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-04-15T18:42:05.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:05.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:05.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:05 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:42:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:05 smithi082 ceph-mon[44206]: pgmap v1889: 17 pgs: 17 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-04-15T18:42:05.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:05.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:05.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:05 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:42:05.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:05 smithi049 ceph-mon[40087]: pgmap v1889: 17 pgs: 17 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-04-15T18:42:05.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:05.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:05.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:05 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:42:05.332 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:05.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:05.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 64375 64359 85.5758 79.875 0.024537 0.0116802 2024-04-15T18:42:05.438 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:05.475 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:05.503 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:05.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:05.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:05.759 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:05.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:05.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:05.897 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:05.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:05.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:06.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:42:06.066 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:06.102 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:42:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:06.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:42:06.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:06.243 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:06.272 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:06.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:06.361 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 65690 65674 85.5051 82.1875 0.00930349 0.0116901 2024-04-15T18:42:06.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:06.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:42:06.409 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:06.432 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:06.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:06.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:06.577 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:06.831 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:06.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:06.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:06.965 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:06.986 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:06.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:07.057 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:42:07.057 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:42:07.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:42:07.057 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:42:07.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25447.0:186314 5.b e536/409 5:db471f5e:::benchmark_data_smithi159_116570_object93154:head v 536'11820, mlcod=536'11817)", 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 593.95132298600004, 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98939341700000005, 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113789+0000", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113790+0000", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.066e-06 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:13.113796+0000", 2024-04-15T18:42:07.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6450000000000002e-06 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098795+0000", 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98499919199999997 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098805+0000", 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9539999999999999e-06 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098871+0000", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5586000000000005e-05 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.098893+0000", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2595999999999998e-05 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.103141+0000", 2024-04-15T18:42:07.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042477979999999997 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:32:14.103183+0000", 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1579999999999998e-05 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:39501 6.a e555/541 6:57bd5a22:::benchmark_data_smithi159_139342_object39500:head v 555'2399, mlcod=555'2396)", 2024-04-15T18:42:07.062 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23247315100002, 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270015279, 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832639+0000", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832640+0000", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6700000000000003e-07 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:20.832700+0000", 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0361e-05 2024-04-15T18:42:07.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101287+0000", 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26858680499999998 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101294+0000", 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4719999999999998e-06 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101355+0000", 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0955999999999997e-05 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.101379+0000", 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3892000000000001e-05 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.102607+0000", 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001227749 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:33:21.102654+0000", 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7477e-05 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26224.0:158190 6.9 e600/541 6:9d4dbcde:::benchmark_data_smithi159_139342_object68311:head v 600'9712, mlcod=600'9707)", 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:35:47.132986+0000", 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 379.93212605299999, 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24760269700000001, 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.067 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.132986+0000", 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.132986+0000", 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.132987+0000", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8800000000000002e-07 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.132994+0000", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7070000000000003e-06 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.376815+0000", 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24382183199999999 2024-04-15T18:42:07.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.376831+0000", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5287000000000001e-05 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.376944+0000", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113157 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.376985+0000", 2024-04-15T18:42:07.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1375000000000003e-05 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.380551+0000", 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003565431 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:35:47.380589+0000", 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8120000000000001e-05 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.071 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:22409 7.6 e627/621 7:637759b3:::benchmark_data_smithi159_149998_object22408:head v 627'1465, mlcod=627'1461)", 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 302.54589664299999, 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21426896200000001, 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519217+0000", 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.099e-06 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519266+0000", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8946999999999997e-05 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732196+0000", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21293002699999999 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732203+0000", 2024-04-15T18:42:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2450000000000001e-06 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732297+0000", 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4067000000000006e-05 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732318+0000", 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1378000000000001e-05 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.733441+0000", 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011223870000000001 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.733485+0000", 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3812e-05 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:22411 7.5 e627/621 7:ae9a4933:::benchmark_data_smithi159_149998_object22410:head v 627'1406, mlcod=627'1403)", 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:42:07.076 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 302.54196151000002, 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.180674529, 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4000000000000001e-07 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523218+0000", 2024-04-15T18:42:07.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6638999999999997e-05 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702813+0000", 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17959461600000001 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702818+0000", 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3040000000000004e-06 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702936+0000", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011765300000000001 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702962+0000", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5967999999999999e-05 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.703798+0000", 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00083654400000000005 2024-04-15T18:42:07.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.703825+0000", 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7064999999999999e-05 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:24097 7.6 e628/621 7:64a751cc:::benchmark_data_smithi159_149998_object24096:head v 628'1566, mlcod=628'1563)", 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.020501954, 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22156325900000001, 2024-04-15T18:42:07.081 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044611+0000", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5899999999999996e-07 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044679+0000", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8415000000000001e-05 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264832+0000", 2024-04-15T18:42:07.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22015254400000001 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264839+0000", 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5619999999999998e-06 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264890+0000", 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0745000000000003e-05 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264908+0000", 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7991000000000001e-05 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266132+0000", 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001223657 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266174+0000", 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1686e-05 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31783 7.9 e631/621 7:999e18b7:::benchmark_data_smithi159_149998_object31782:head v 631'2061, mlcod=631'2059)", 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.02578087000001, 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30491139699999997, 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039332+0000", 2024-04-15T18:42:07.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039377+0000", 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4564000000000002e-05 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343028+0000", 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.303650851 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343040+0000", 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2043e-05 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343095+0000", 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5083000000000001e-05 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343111+0000", 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6446000000000001e-05 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.344208+0000", 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010970000000000001 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.344243+0000", 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4449000000000002e-05 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.090 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31793 7.4 e631/621 7:2fbec3a3:::benchmark_data_smithi159_149998_object31792:head v 631'2016, mlcod=631'2013)", 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.01675965599998, 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31115801599999998, 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048444+0000", 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1067000000000001e-05 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358065+0000", 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30962077900000001 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358072+0000", 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8680000000000001e-06 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358119+0000", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7188999999999998e-05 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358133+0000", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3766e-05 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359477+0000", 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013445880000000001 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359511+0000", 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3061000000000001e-05 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:55420 7.4 e639/621 7:2ab922f1:::benchmark_data_smithi159_149998_object55419:head v 639'3531, mlcod=639'3527)", 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 275.870013744, 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25497385900000002, 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195099+0000", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9100000000000004e-07 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195196+0000", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6490000000000006e-05 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448347+0000", 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25315195600000001 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448355+0000", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9580000000000007e-06 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448408+0000", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2417000000000003e-05 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448434+0000", 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5846e-05 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.450038+0000", 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001604329 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.450072+0000", 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4272e-05 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77949 7.d 7:b4f21db4:::benchmark_data_smithi159_149998_object77948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 252.55068055999999, 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18121273600000001, 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77949 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514434+0000", 2024-04-15T18:42:07.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.367e-06 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515468+0000", 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001033876 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515487+0000", 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8987e-05 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515499+0000", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1762e-05 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516470+0000", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00097108399999999999 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516482+0000", 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1817999999999999e-05 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.679949+0000", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16346764499999999 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.681990+0000", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002041036 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.682168+0000", 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017791199999999999 2024-04-15T18:42:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690088+0000", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079198110000000006 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690146+0000", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8103999999999998e-05 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.692919+0000", 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027726740000000001 2024-04-15T18:42:07.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.692975+0000", 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6691999999999999e-05 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695644+0000", 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026689679999999999 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77976 7.d 7:bd2ba41d:::benchmark_data_smithi159_149998_object77975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 252.549588511, 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18580798200000001, 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77976 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.107 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515525+0000", 2024-04-15T18:42:07.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5000000000000001e-07 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515594+0000", 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9438999999999997e-05 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515604+0000", 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0032e-05 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515611+0000", 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8700000000000003e-06 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516508+0000", 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00089666500000000005 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516514+0000", 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4069999999999998e-06 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687324+0000", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17081004 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687484+0000", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016032500000000001 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687654+0000", 2024-04-15T18:42:07.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016935999999999999 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695721+0000", 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080673120000000001 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695914+0000", 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019322799999999999 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.698278+0000", 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023638140000000001 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.698326+0000", 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7848e-05 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.701332+0000", 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030056919999999999 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.113 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77998 7.d 7:b24126a2:::benchmark_data_smithi159_149998_object77997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 252.54948985999999, 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.207862504, 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.114 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77998 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515623+0000", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000003e-07 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515686+0000", 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3291000000000001e-05 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515695+0000", 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1900000000000001e-06 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515702+0000", 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7279999999999998e-06 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516689+0000", 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0009865570000000001 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516698+0000", 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7920000000000004e-06 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690233+0000", 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17353442899999999 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690357+0000", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123984 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690464+0000", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010748500000000001 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695895+0000", 2024-04-15T18:42:07.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054309470000000002 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.701394+0000", 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054990689999999997 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.720843+0000", 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019448213999999998 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.720907+0000", 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3930999999999995e-05 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.723485+0000", 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002578313 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:83282 7.d 7:b94c71dc:::benchmark_data_smithi159_149998_object83281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e651)", 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:01.985798+0000", 2024-04-15T18:42:07.121 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 245.07931373100001, 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17033372099999999, 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 83282 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985798+0000", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985798+0000", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985800+0000", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5060000000000001e-06 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985862+0000", 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2031000000000003e-05 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985873+0000", 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1029000000000001e-05 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.985880+0000", 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.81e-06 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.989016+0000", 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031364180000000002 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.989148+0000", 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131426 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:01.989264+0000", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116778 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.005072+0000", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015807180000000001 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.005142+0000", 2024-04-15T18:42:07.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0685000000000005e-05 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.153260+0000", 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14811728599999999 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.153329+0000", 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9318000000000003e-05 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.156132+0000", 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028032539999999998 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.129 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:83519 7.f e651/621 7:f547c4e5:::benchmark_data_smithi159_149998_object83518:head v 651'5282, mlcod=651'5280)", 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 244.676525975, 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.190204965, 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388587+0000", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4600000000000003e-07 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388688+0000", 2024-04-15T18:42:07.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100833 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572910+0000", 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.184222577 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572918+0000", 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3000000000000004e-06 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572971+0000", 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3060999999999999e-05 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572993+0000", 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2608000000000001e-05 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.578738+0000", 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057440590000000001 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.578791+0000", 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3681000000000003e-05 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:116803 7.5 e661/621 7:a4d27c37:::benchmark_data_smithi159_149998_object20803:head v 661'7308, mlcod=661'7306)", 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 197.819090106, 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20000991300000001, 2024-04-15T18:42:07.134 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:42:07.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246023+0000", 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.322e-06 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246031+0000", 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1969999999999999e-06 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.428454+0000", 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18242346000000001 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.428461+0000", 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9519999999999997e-06 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.429519+0000", 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00105795 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.429540+0000", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0656999999999999e-05 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.445995+0000", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016454933000000001 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.446032+0000", 2024-04-15T18:42:07.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7441999999999998e-05 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:124560 7.4 e663/621 7:2a9d76e5:::benchmark_data_smithi159_149998_object28560:head v 663'7929, mlcod=663'7926)", 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 186.90291295, 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27284343, 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162200+0000", 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7599999999999996e-07 2024-04-15T18:42:07.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162213+0000", 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2889e-05 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431402+0000", 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26918940000000002 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431409+0000", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8619999999999997e-06 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431461+0000", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2212000000000001e-05 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431476+0000", 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5014999999999999e-05 2024-04-15T18:42:07.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.434997+0000", 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035202620000000001 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.435043+0000", 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6013999999999997e-05 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(7.d 670/648 [PushReplyOp(7:bbf5c71a:::benchmark_data_smithi159_149998_object76520:head)])", 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:24.198899+0000", 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 162.86621311499999, 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16758324899999999, 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.198899+0000", 2024-04-15T18:42:07.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.198899+0000", 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.198900+0000", 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9500000000000002e-07 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.198904+0000", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5009999999999998e-06 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.364047+0000", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16514273900000001 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.364056+0000", 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1600000000000004e-06 2024-04-15T18:42:07.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.366258+0000", 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022015110000000002 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.366482+0000", 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022474300000000001 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:143179 7.d e671/648)", 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 160.74726800900001, 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31904452599999999, 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317845+0000", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2099999999999995e-07 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317852+0000", 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9020000000000002e-06 2024-04-15T18:42:07.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636785+0000", 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.318932893 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636790+0000", 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4029999999999998e-06 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636870+0000", 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9237e-05 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636889+0000", 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9170000000000001e-05 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:143366 7.4 e672/621 7:21b697b5:::benchmark_data_smithi159_149998_object47366:head v 672'9144, mlcod=672'9142)", 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:42:07.151 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 160.15852362199999, 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53369277100000001, 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:42:07.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906590+0000", 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.717e-06 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906604+0000", 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3473e-05 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435564+0000", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52895973699999999 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435570+0000", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7490000000000001e-06 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435679+0000", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010908600000000001 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435700+0000", 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0897e-05 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.440236+0000", 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045354540000000004 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.440281+0000", 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5658000000000002e-05 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:191961 7.d e700/672 7:b225a1f7:::benchmark_data_smithi159_149998_object95961:head v 700'11971, mlcod=700'11968)", 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 91.191167664000005, 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43827476300000001, 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:42:07.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9999999999999998e-07 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873949+0000", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2289999999999997e-06 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308026+0000", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43407702100000001 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308036+0000", 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4180000000000003e-06 2024-04-15T18:42:07.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308109+0000", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3162000000000002e-05 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308139+0000", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0298e-05 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312172+0000", 2024-04-15T18:42:07.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004033043 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312219+0000", 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7191999999999998e-05 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:42:07.160 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:42:07.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:07.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:07 smithi049 ceph-mon[40087]: pgmap v1890: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:42:07.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:07.240 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:07.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:07.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:07 smithi082 ceph-mon[44206]: pgmap v1890: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:42:07.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:07.362 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 16 66743 66727 85.1016 65.8125 0.0071035 0.0117457 2024-04-15T18:42:07.386 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:07 smithi159 ceph-mon[45336]: pgmap v1890: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:42:08.363 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 67685 67669 84.5757 58.875 0.00527901 0.011819 2024-04-15T18:42:08.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:42:08.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:42:08.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:42:08.411 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:42:08.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:42:08.449 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:08.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:08.480 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:42:08.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:42:08.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:42:08.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:08.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:42:08.593 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:08.621 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:42:08.641 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:08.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:08.672 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:42:08.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:42:08.711 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:42:08.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:42:08.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:08.797 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:08.829 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:08.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:08.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:42:08.980 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:09.008 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:42:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:09.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:42:09.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:09.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: pgmap v1891: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3920644843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:09 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.181 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:09.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:09.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: pgmap v1891: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-15T18:42:09.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:09.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3920644843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:09 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: pgmap v1891: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3920644843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.318 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:09 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:09.340 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:09.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:42:09.363 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 15 68724 68709 84.1917 65 0.03118 0.0118714 2024-04-15T18:42:09.481 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:09.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:09.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:09.640 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:09.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:09.784 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:09.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:09.919 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:10.051 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:10.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:10.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:10.186 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:10.208 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:10.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:10.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:10 smithi049 ceph-mon[40087]: osdmap e733: 12 total, 12 up, 12 in 2024-04-15T18:42:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:10 smithi082 ceph-mon[44206]: osdmap e733: 12 total, 12 up, 12 in 2024-04-15T18:42:10.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:10.346 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:10.363 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 69881 69865 83.962 72.25 0.00335382 0.0119042 2024-04-15T18:42:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:10.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:10 smithi159 ceph-mon[45336]: osdmap e733: 12 total, 12 up, 12 in 2024-04-15T18:42:10.702 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:10.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:10.724 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:10.809 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:10.851 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:10.880 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:10.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:10.986 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:11.026 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.054 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:11.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:11 smithi159 ceph-mon[45336]: pgmap v1893: 17 pgs: 17 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:11.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:11.194 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.216 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:11.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:11 smithi082 ceph-mon[44206]: pgmap v1893: 17 pgs: 17 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:11.320 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:11.353 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.363 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 70970 70954 83.662 68.0625 0.0377408 0.011945 2024-04-15T18:42:11.374 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:11.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:11 smithi049 ceph-mon[40087]: pgmap v1893: 17 pgs: 17 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-04-15T18:42:11.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:11.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:11.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:11.769 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:11.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:42:11.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:11.943 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:11.968 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:11.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:12.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:12.129 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:12.289 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-04-15T18:42:12.289 INFO:tasks.thrashosds.thrasher:Killing osd 11, live_osds are [0, 1, 4, 7, 8, 9, 10, 11, 3, 2, 5, 6] 2024-04-15T18:42:12.290 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-15T18:42:12.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11 2024-04-15T18:42:12.363 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 16 71972 71956 83.2723 62.625 0.00834951 0.0120023 2024-04-15T18:42:12.645 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:42:12 smithi159 systemd[1]: Stopping Ceph osd.11 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:42:13.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 15 73064 73049 83.0002 68.3125 0.0207004 0.0120355 2024-04-15T18:42:13.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:13 smithi049 ceph-mon[40087]: pgmap v1894: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:42:13.385 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:13.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:13 smithi159 ceph-mon[45336]: pgmap v1894: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:42:13.489 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:13.516 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:13 smithi082 ceph-mon[44206]: pgmap v1894: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:42:13.517 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:13.539 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:13.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:13.643 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:13.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:13.694 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:13.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:13.800 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:13.851 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:13.913 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:13.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:14.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:14.046 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:14.067 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:14.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:14.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:14.214 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:14.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 74161 74145 82.7412 68.5 0.0191147 0.0120802 2024-04-15T18:42:14.394 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:42:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:42:14.144+0000 7fde28365700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:42:14.394 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:42:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:42:14.144+0000 7fde28365700 -1 osd.11 733 *** Got signal Terminated *** 2024-04-15T18:42:14.395 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:42:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[73765]: 2024-04-15T18:42:14.144+0000 7fde28365700 -1 osd.11 733 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:42:14.484 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:14.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:14.504 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:14.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:14.615 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:42:14.615 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:42:14.628 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:15.053 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:42:15.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 74835 74819 82.0286 42.125 0.318983 0.0121362 2024-04-15T18:42:15.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:15 smithi159 ceph-mon[45336]: pgmap v1895: 17 pgs: 17 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:42:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:15 smithi159 ceph-mon[45336]: osd.11 marked itself down and dead 2024-04-15T18:42:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1274651204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:15.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:15 smithi049 ceph-mon[40087]: pgmap v1895: 17 pgs: 17 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:42:15.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:15 smithi049 ceph-mon[40087]: osd.11 marked itself down and dead 2024-04-15T18:42:15.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1274651204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:15.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:15.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:15 smithi082 ceph-mon[44206]: pgmap v1895: 17 pgs: 17 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-04-15T18:42:15.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:15 smithi082 ceph-mon[44206]: osd.11 marked itself down and dead 2024-04-15T18:42:15.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1274651204' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:15.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:16.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:42:16.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 75246 75230 81.0572 25.6875 0.00451506 0.012331 2024-04-15T18:42:16.395 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:42:16 smithi159 bash[164745]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11 2024-04-15T18:42:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:16 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:16 smithi159 ceph-mon[45336]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:42:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:16.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:16 smithi159 ceph-mon[45336]: osdmap e734: 12 total, 11 up, 12 in 2024-04-15T18:42:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:16 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:16 smithi049 ceph-mon[40087]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:42:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:16 smithi049 ceph-mon[40087]: osdmap e734: 12 total, 11 up, 12 in 2024-04-15T18:42:16.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:16 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:16.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:16 smithi082 ceph-mon[44206]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-15T18:42:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:16 smithi082 ceph-mon[44206]: osdmap e734: 12 total, 11 up, 12 in 2024-04-15T18:42:16.680 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:16.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:16.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:16.830 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:17.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:17 smithi049 ceph-mon[40087]: pgmap v1897: 17 pgs: 4 stale+active+clean, 13 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-15T18:42:17.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:17 smithi049 ceph-mon[40087]: osdmap e735: 12 total, 11 up, 12 in 2024-04-15T18:42:17.316 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:17.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 76060 76044 80.5455 50.875 0.0103733 0.0124026 2024-04-15T18:42:17.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:17 smithi159 ceph-mon[45336]: pgmap v1897: 17 pgs: 4 stale+active+clean, 13 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-15T18:42:17.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:17 smithi159 ceph-mon[45336]: osdmap e735: 12 total, 11 up, 12 in 2024-04-15T18:42:17.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:17 smithi082 ceph-mon[44206]: pgmap v1897: 17 pgs: 4 stale+active+clean, 13 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-04-15T18:42:17.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:17 smithi082 ceph-mon[44206]: osdmap e735: 12 total, 11 up, 12 in 2024-04-15T18:42:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:17.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:17.889 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:17.935 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:18.126 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11.service' 2024-04-15T18:42:18.145 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:18.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:18.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:42:18.181 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-15T18:42:18.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd down 11 2024-04-15T18:42:18.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:18.279 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:18.299 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:18.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:18.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:42:18.376046+0000 min lat: 0.00103313 max lat: 1.0412 avg lat: 0.0124261 2024-04-15T18:42:18.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:42:18.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 77244 77228 80.4364 74 0.00361914 0.0124261 2024-04-15T18:42:18.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:18.441 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:18.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:42:18.581 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:18 smithi082 ceph-mon[44206]: pgmap v1899: 17 pgs: 3 active+undersized+degraded, 2 stale+active+clean, 12 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s; 9634/221244 objects degraded (4.354%) 2024-04-15T18:42:18.581 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:18 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 9634/221244 objects degraded (4.354%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:18.600 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:18.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:18.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:18.748 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:18.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:18 smithi159 ceph-mon[45336]: pgmap v1899: 17 pgs: 3 active+undersized+degraded, 2 stale+active+clean, 12 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s; 9634/221244 objects degraded (4.354%) 2024-04-15T18:42:18.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:18 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 9634/221244 objects degraded (4.354%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:18.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:18 smithi049 ceph-mon[40087]: pgmap v1899: 17 pgs: 3 active+undersized+degraded, 2 stale+active+clean, 12 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s; 9634/221244 objects degraded (4.354%) 2024-04-15T18:42:18.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:18 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 9634/221244 objects degraded (4.354%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:19.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 78431 78415 80.3338 74.1875 0.00372431 0.0124426 2024-04-15T18:42:19.910 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:19.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:42:20.001 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:42:20.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:20.030 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:42:20.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:42:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:20.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:42:20.112 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:20.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:20.143 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:42:20.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:42:20.181 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:42:20.182 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:42:20.232 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:20.261 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:20.281 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:20.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:20.364 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 79552 79536 80.1674 70.0625 0.00364782 0.0124684 2024-04-15T18:42:20.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:20.431 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:20.462 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:20.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:20.567 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:20.606 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:20.631 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:20.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:20.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:20.787 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:20.834 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:20.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: pgmap v1900: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 24125/229014 objects degraded (10.534%) 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osdmap e736: 12 total, 11 up, 12 in 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/844029780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:20.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: pgmap v1900: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 24125/229014 objects degraded (10.534%) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osdmap e736: 12 total, 11 up, 12 in 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/844029780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:20.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:20.990 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: pgmap v1900: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 24125/229014 objects degraded (10.534%) 2024-04-15T18:42:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osdmap e736: 12 total, 11 up, 12 in 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/844029780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 80853 80837 80.1855 81.3125 0.0184122 0.0124648 2024-04-15T18:42:21.677 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:21.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:21.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:21.812 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:21.832 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:21.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:42:21.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:42:21.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:21 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:21 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:21.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:21 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:21.965 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:21.984 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:21.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:22.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:22.126 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:22.206 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:22.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:22.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:22.338 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:22.356 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:22.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:22.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 81762 81746 79.8202 56.8125 0.00635744 0.0125196 2024-04-15T18:42:22.371 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:22.371 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:22.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:22.490 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:22.513 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:22.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:22.562 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:42:22.562 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:42:22.562 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:42:22.562 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:42:22.562 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:42:22.591 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:22 smithi082 ceph-mon[44206]: pgmap v1902: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 685 op/s; 24457/231741 objects degraded (10.554%) 2024-04-15T18:42:22.621 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:22.647 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:22.668 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:22.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:22.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:22.818 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:22.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:22.868 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:22 smithi049 ceph-mon[40087]: pgmap v1902: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 685 op/s; 24457/231741 objects degraded (10.554%) 2024-04-15T18:42:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:22.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:22.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:22 smithi159 ceph-mon[45336]: pgmap v1902: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 685 op/s; 24457/231741 objects degraded (10.554%) 2024-04-15T18:42:22.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:22.999 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:23.022 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:23.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:23.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:23.156 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:23.177 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:23.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:23.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:23.319 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:23.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 82809 82793 79.5989 65.4375 0.00343966 0.0125576 2024-04-15T18:42:23.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:42:23.794 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 25816/245313 objects degraded (10.524%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:23.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:42:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 25816/245313 objects degraded (10.524%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 25816/245313 objects degraded (10.524%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:24.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 83788 83772 79.3199 61.1875 0.00601165 0.0126017 2024-04-15T18:42:24.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:24 smithi049 ceph-mon[40087]: pgmap v1903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 25816/245313 objects degraded (10.524%) 2024-04-15T18:42:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:24.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:24 smithi082 ceph-mon[44206]: pgmap v1903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 25816/245313 objects degraded (10.524%) 2024-04-15T18:42:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:24 smithi159 ceph-mon[45336]: pgmap v1903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 25816/245313 objects degraded (10.524%) 2024-04-15T18:42:25.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:25.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:25.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 84861 84845 79.1368 67.0625 0.0158757 0.0126313 2024-04-15T18:42:25.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:25 smithi049 ceph-mon[40087]: osdmap e737: 12 total, 11 up, 12 in 2024-04-15T18:42:25.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:25.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:25.857 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:25.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:25.963 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:25.993 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:25 smithi159 ceph-mon[45336]: osdmap e737: 12 total, 11 up, 12 in 2024-04-15T18:42:25.993 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:25.993 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:25.996 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.020 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:26.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:25 smithi082 ceph-mon[44206]: osdmap e737: 12 total, 11 up, 12 in 2024-04-15T18:42:26.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:26.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:26.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:26.156 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:26.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:26.320 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.346 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:26.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 85893 85877 78.9215 64.5 0.00464172 0.0126659 2024-04-15T18:42:26.452 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:26.482 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.506 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:26.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:26.640 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.662 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:26.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:26 smithi049 ceph-mon[40087]: pgmap v1905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 25816/247467 objects degraded (10.432%) 2024-04-15T18:42:26.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:26 smithi049 ceph-mon[40087]: osdmap e738: 12 total, 11 up, 12 in 2024-04-15T18:42:26.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:26.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.817 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:26.923 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:26.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:26 smithi082 ceph-mon[44206]: pgmap v1905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 25816/247467 objects degraded (10.432%) 2024-04-15T18:42:26.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:26 smithi082 ceph-mon[44206]: osdmap e738: 12 total, 11 up, 12 in 2024-04-15T18:42:26.950 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:26.969 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:26.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:27.065 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:27.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:27.097 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:26 smithi159 ceph-mon[45336]: pgmap v1905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 25816/247467 objects degraded (10.432%) 2024-04-15T18:42:27.097 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:26 smithi159 ceph-mon[45336]: osdmap e738: 12 total, 11 up, 12 in 2024-04-15T18:42:27.099 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:27.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:27.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:27.257 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:27.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 86415 86399 78.2505 32.625 0.00330615 0.0127198 2024-04-15T18:42:27.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:27.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:27.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:27 smithi049 ceph-mon[40087]: osdmap e739: 12 total, 11 up, 12 in 2024-04-15T18:42:27.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:42:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:27 smithi082 ceph-mon[44206]: osdmap e739: 12 total, 11 up, 12 in 2024-04-15T18:42:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:42:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:27 smithi159 ceph-mon[45336]: osdmap e739: 12 total, 11 up, 12 in 2024-04-15T18:42:28.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:42:28.245 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:28.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 87580 87564 78.1727 72.8125 0.027696 0.012787 2024-04-15T18:42:28.640 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:42:28.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:28 smithi049 ceph-mon[40087]: pgmap v1908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 26423/251070 objects degraded (10.524%) 2024-04-15T18:42:28.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:28 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:28.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:42:28.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:28 smithi049 ceph-mon[40087]: osdmap e740: 12 total, 11 up, 12 in 2024-04-15T18:42:28.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11 is already down. 2024-04-15T18:42:28.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:28 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:42:28.720+0000 7f8015e7e700 -1 mon.a@0(leader).osd e740 definitely_dead 0 2024-04-15T18:42:29.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:28 smithi082 ceph-mon[44206]: pgmap v1908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 26423/251070 objects degraded (10.524%) 2024-04-15T18:42:29.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:28 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:29.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:42:29.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:28 smithi082 ceph-mon[44206]: osdmap e740: 12 total, 11 up, 12 in 2024-04-15T18:42:29.109 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:29.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:28 smithi159 ceph-mon[45336]: pgmap v1908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 26423/251070 objects degraded (10.524%) 2024-04-15T18:42:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:28 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:42:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1045861270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:42:29.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:28 smithi159 ceph-mon[45336]: osdmap e740: 12 total, 11 up, 12 in 2024-04-15T18:42:29.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:42:29.269 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:29.329 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:42:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:29.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:42:29.365 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 88648 88632 78.0117 66.75 0.0148138 0.0128129 2024-04-15T18:42:29.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:29.468 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:29.490 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:29.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:29.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:29.649 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:29.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/419423251' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["11"]}]: dispatch 2024-04-15T18:42:29.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/419423251' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["11"]}]: dispatch 2024-04-15T18:42:29.919 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 4, 5, 7, 8, 9, 10, 11, 3, 2, 6] 2024-04-15T18:42:29.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd out 11 2024-04-15T18:42:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/419423251' entity='client.admin' cmd=[{"prefix": "osd down", "ids": ["11"]}]: dispatch 2024-04-15T18:42:30.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 16 89804 89788 77.9316 72.25 0.00490514 0.0128264 2024-04-15T18:42:30.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:30 smithi049 ceph-mon[40087]: 8.1 scrub starts 2024-04-15T18:42:30.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:30 smithi049 ceph-mon[40087]: pgmap v1910: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 27295/260103 objects degraded (10.494%) 2024-04-15T18:42:30.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:30 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 27295/260103 objects degraded (10.494%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:31.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:30 smithi082 ceph-mon[44206]: 8.1 scrub starts 2024-04-15T18:42:31.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:30 smithi082 ceph-mon[44206]: pgmap v1910: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 27295/260103 objects degraded (10.494%) 2024-04-15T18:42:31.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:30 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 27295/260103 objects degraded (10.494%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:31.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:30 smithi159 ceph-mon[45336]: 8.1 scrub starts 2024-04-15T18:42:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:30 smithi159 ceph-mon[45336]: pgmap v1910: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 27295/260103 objects degraded (10.494%) 2024-04-15T18:42:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:30 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 27295/260103 objects degraded (10.494%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:31.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 90834 90818 77.7458 64.375 0.00399036 0.012857 2024-04-15T18:42:31.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:31 smithi049 ceph-mon[40087]: 8.1 scrub ok 2024-04-15T18:42:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:31 smithi049 ceph-mon[40087]: 8.a scrub starts 2024-04-15T18:42:31.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:31 smithi049 ceph-mon[40087]: 8.e scrub starts 2024-04-15T18:42:32.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:31 smithi082 ceph-mon[44206]: 8.1 scrub ok 2024-04-15T18:42:32.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:31 smithi082 ceph-mon[44206]: 8.a scrub starts 2024-04-15T18:42:32.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:31 smithi082 ceph-mon[44206]: 8.e scrub starts 2024-04-15T18:42:32.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:31 smithi159 ceph-mon[45336]: 8.1 scrub ok 2024-04-15T18:42:32.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:31 smithi159 ceph-mon[45336]: 8.a scrub starts 2024-04-15T18:42:32.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:31 smithi159 ceph-mon[45336]: 8.e scrub starts 2024-04-15T18:42:32.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 91575 91559 77.321 46.3125 0.0160697 0.012927 2024-04-15T18:42:32.411 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:32.416 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:32.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:32.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:32.548 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:32.566 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:32.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:32.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:32.710 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:32.739 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:32.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:32.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:32.877 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:32.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:32 smithi159 ceph-mon[45336]: pgmap v1911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 27627/262908 objects degraded (10.508%) 2024-04-15T18:42:32.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:32 smithi159 ceph-mon[45336]: 8.b scrub starts 2024-04-15T18:42:32.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:32 smithi159 ceph-mon[45336]: 8.e scrub ok 2024-04-15T18:42:32.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:32 smithi159 ceph-mon[45336]: 8.a scrub ok 2024-04-15T18:42:32.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:32 smithi159 ceph-mon[45336]: 8.b scrub ok 2024-04-15T18:42:32.899 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:32.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:32.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:32 smithi049 ceph-mon[40087]: pgmap v1911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 27627/262908 objects degraded (10.508%) 2024-04-15T18:42:32.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:32 smithi049 ceph-mon[40087]: 8.b scrub starts 2024-04-15T18:42:32.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:32 smithi049 ceph-mon[40087]: 8.e scrub ok 2024-04-15T18:42:32.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:32 smithi049 ceph-mon[40087]: 8.a scrub ok 2024-04-15T18:42:32.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:32 smithi049 ceph-mon[40087]: 8.b scrub ok 2024-04-15T18:42:33.005 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:32 smithi082 ceph-mon[44206]: pgmap v1911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 27627/262908 objects degraded (10.508%) 2024-04-15T18:42:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:32 smithi082 ceph-mon[44206]: 8.b scrub starts 2024-04-15T18:42:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:32 smithi082 ceph-mon[44206]: 8.e scrub ok 2024-04-15T18:42:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:32 smithi082 ceph-mon[44206]: 8.a scrub ok 2024-04-15T18:42:33.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:32 smithi082 ceph-mon[44206]: 8.b scrub ok 2024-04-15T18:42:33.038 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.061 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:33.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:33.205 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.232 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:33.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:33.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 92434 92418 77.0059 53.6875 0.0252199 0.0129809 2024-04-15T18:42:33.404 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.433 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:33.544 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:33.573 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.596 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:33.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:33 smithi049 ceph-mon[40087]: 8.5 scrub starts 2024-04-15T18:42:33.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:33 smithi049 ceph-mon[40087]: 8.c scrub starts 2024-04-15T18:42:33.729 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.754 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:33.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:33.903 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:33.930 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:33.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:34.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:33 smithi082 ceph-mon[44206]: 8.5 scrub starts 2024-04-15T18:42:34.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:33 smithi082 ceph-mon[44206]: 8.c scrub starts 2024-04-15T18:42:34.034 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:34.071 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.097 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:34.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:42:34.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:33 smithi159 ceph-mon[45336]: 8.5 scrub starts 2024-04-15T18:42:34.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:33 smithi159 ceph-mon[45336]: 8.c scrub starts 2024-04-15T18:42:34.204 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:34.234 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.261 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:34.366 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:34.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 93417 93401 76.801 61.4375 0.0269763 0.0130158 2024-04-15T18:42:34.399 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.420 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:34.525 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:34.556 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.579 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:34.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:34.713 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.747 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:34.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:34.891 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:34.913 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:34.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:34.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:34 smithi049 ceph-mon[40087]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 28795/274776 objects degraded (10.479%) 2024-04-15T18:42:34.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:34 smithi049 ceph-mon[40087]: 8.0 scrub starts 2024-04-15T18:42:34.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:34 smithi049 ceph-mon[40087]: 8.9 scrub starts 2024-04-15T18:42:35.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:35.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:34 smithi082 ceph-mon[44206]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 28795/274776 objects degraded (10.479%) 2024-04-15T18:42:35.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:34 smithi082 ceph-mon[44206]: 8.0 scrub starts 2024-04-15T18:42:35.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:34 smithi082 ceph-mon[44206]: 8.9 scrub starts 2024-04-15T18:42:35.044 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:35.067 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:35.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:35.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:34 smithi159 ceph-mon[45336]: pgmap v1912: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 28795/274776 objects degraded (10.479%) 2024-04-15T18:42:35.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:34 smithi159 ceph-mon[45336]: 8.0 scrub starts 2024-04-15T18:42:35.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:34 smithi159 ceph-mon[45336]: 8.9 scrub starts 2024-04-15T18:42:35.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:35.212 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:35.244 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:35.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:35.353 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:35.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 94277 94261 76.5015 53.75 0.0139647 0.0130669 2024-04-15T18:42:35.390 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:35.414 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:35.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:35.519 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:35.556 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:35.582 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:35.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:35.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:35.733 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:35.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: 8.5 scrub ok 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: 8.2 scrub starts 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: osdmap e741: 12 total, 11 up, 12 in 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: 8.4 scrub starts 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: 8.0 scrub ok 2024-04-15T18:42:35.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:35 smithi159 ceph-mon[45336]: 8.c scrub ok 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: 8.5 scrub ok 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: 8.2 scrub starts 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: osdmap e741: 12 total, 11 up, 12 in 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: 8.4 scrub starts 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: 8.0 scrub ok 2024-04-15T18:42:36.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:35 smithi082 ceph-mon[44206]: 8.c scrub ok 2024-04-15T18:42:36.096 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:36.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: 8.5 scrub ok 2024-04-15T18:42:36.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:36.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:36.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: 8.2 scrub starts 2024-04-15T18:42:36.100 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: osdmap e741: 12 total, 11 up, 12 in 2024-04-15T18:42:36.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: 8.4 scrub starts 2024-04-15T18:42:36.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: 8.0 scrub ok 2024-04-15T18:42:36.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:35 smithi049 ceph-mon[40087]: 8.c scrub ok 2024-04-15T18:42:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:36.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:36.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:36.234 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:36.260 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:36.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:36.366 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:36.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 16 95247 95231 76.2979 60.625 0.0221143 0.0131013 2024-04-15T18:42:36.398 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:36.427 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:36.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:36.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:36.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:42:36.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:42:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:36.623 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:42:36.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:42:36.709 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:42:36.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:42:36.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: 8.9 scrub ok 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: pgmap v1914: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 28795/276423 objects degraded (10.417%) 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 28795/276423 objects degraded (10.417%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: 8.8 scrub starts 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: 8.4 scrub ok 2024-04-15T18:42:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2991124035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:36.833 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:42:36.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:42:36.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:42:36.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:42:36.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:42:36.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:42:36.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:42:36.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:42:36.936 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:42:36.936 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:42:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: 8.9 scrub ok 2024-04-15T18:42:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: pgmap v1914: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 28795/276423 objects degraded (10.417%) 2024-04-15T18:42:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 28795/276423 objects degraded (10.417%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: 8.8 scrub starts 2024-04-15T18:42:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: 8.4 scrub ok 2024-04-15T18:42:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2991124035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: 8.9 scrub ok 2024-04-15T18:42:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: pgmap v1914: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 28795/276423 objects degraded (10.417%) 2024-04-15T18:42:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 28795/276423 objects degraded (10.417%), 6 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: 8.8 scrub starts 2024-04-15T18:42:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: 8.4 scrub ok 2024-04-15T18:42:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2991124035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:37.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:36 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:42:36.836+0000 7f8015e7e700 -1 mon.a@0(leader).osd e741 definitely_dead 0 2024-04-15T18:42:37.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 96343 96327 76.1992 68.5 0.00765184 0.0131178 2024-04-15T18:42:37.742 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.11. 2024-04-15T18:42:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: 8.2 scrub ok 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: Client client.admin marked osd.11 out, after it was down for 21 seconds 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:37 smithi082 ceph-mon[44206]: 8.8 scrub ok 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: 8.2 scrub ok 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: Client client.admin marked osd.11 out, after it was down for 21 seconds 2024-04-15T18:42:38.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:37 smithi049 ceph-mon[40087]: 8.8 scrub ok 2024-04-15T18:42:38.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: 8.2 scrub ok 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: Client client.admin marked osd.11 out, after it was down for 21 seconds 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:37 smithi159 ceph-mon[45336]: 8.8 scrub ok 2024-04-15T18:42:38.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:42:38.378350+0000 min lat: 0.00103313 max lat: 1.0412 avg lat: 0.0131623 2024-04-15T18:42:38.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:42:38.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 96739 96723 75.556 24.75 0.457999 0.0131623 2024-04-15T18:42:38.536 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:38.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:38.641 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:38.672 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:38.691 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:38.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:38.795 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:38.823 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:38.842 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:38.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:38.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:38.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:38 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:38.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:38 smithi049 ceph-mon[40087]: pgmap v1915: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 29856/282207 objects degraded (10.579%) 2024-04-15T18:42:38.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:38 smithi049 ceph-mon[40087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:42:38.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:38 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-15T18:42:38.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:38 smithi049 ceph-mon[40087]: osdmap e742: 12 total, 11 up, 11 in 2024-04-15T18:42:38.976 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:39.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:38 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:39.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:38 smithi082 ceph-mon[44206]: pgmap v1915: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 29856/282207 objects degraded (10.579%) 2024-04-15T18:42:39.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:38 smithi082 ceph-mon[44206]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:42:39.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:38 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-15T18:42:39.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:38 smithi082 ceph-mon[44206]: osdmap e742: 12 total, 11 up, 11 in 2024-04-15T18:42:39.076 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:39.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:39.085 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.11 2024-04-15T18:42:39.085 INFO:tasks.thrashosds.thrasher:remote for osd 11 is ubuntu@smithi159.front.sepia.ceph.com 2024-04-15T18:42:39.086 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.11 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --op list-pgs 2024-04-15T18:42:39.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:38 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:39.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:38 smithi159 ceph-mon[45336]: pgmap v1915: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 29856/282207 objects degraded (10.579%) 2024-04-15T18:42:39.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:38 smithi159 ceph-mon[45336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-15T18:42:39.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:38 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/368411032' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-04-15T18:42:39.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:38 smithi159 ceph-mon[45336]: osdmap e742: 12 total, 11 up, 11 in 2024-04-15T18:42:39.180 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:39.209 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:39.227 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:39.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:39.306 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.11/config 2024-04-15T18:42:39.331 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:39.357 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:39.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 96743 96727 74.6263 0.25 0.00437422 0.0131698 2024-04-15T18:42:39.514 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:42:39.514 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:42:39.556 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:39.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:39.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:39.688 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:39.810 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:39 smithi159 ceph-mon[45336]: osdmap e743: 12 total, 11 up, 11 in 2024-04-15T18:42:39.810 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:39 smithi159 ceph-mon[45336]: osdmap e744: 12 total, 11 up, 11 in 2024-04-15T18:42:39.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:39 smithi049 ceph-mon[40087]: osdmap e743: 12 total, 11 up, 11 in 2024-04-15T18:42:39.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:39 smithi049 ceph-mon[40087]: osdmap e744: 12 total, 11 up, 11 in 2024-04-15T18:42:40.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:39 smithi082 ceph-mon[44206]: osdmap e743: 12 total, 11 up, 11 in 2024-04-15T18:42:40.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:39 smithi082 ceph-mon[44206]: osdmap e744: 12 total, 11 up, 11 in 2024-04-15T18:42:40.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 97046 97030 73.9472 18.9375 0.0176234 0.0135165 2024-04-15T18:42:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:42:40.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:42:40.753 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:40.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:40.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:40.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:40.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:40.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:40 smithi049 ceph-mon[40087]: pgmap v1918: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 46 MiB/s wr, 738 op/s; 24129/288063 objects degraded (8.376%) 2024-04-15T18:42:40.934 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:41.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:41.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:41.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:40 smithi082 ceph-mon[44206]: pgmap v1918: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 46 MiB/s wr, 738 op/s; 24129/288063 objects degraded (8.376%) 2024-04-15T18:42:41.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:41.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:41.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:40 smithi159 ceph-mon[45336]: pgmap v1918: 17 pgs: 1 peering, 5 active+undersized+degraded, 11 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 46 MiB/s wr, 738 op/s; 24129/288063 objects degraded (8.376%) 2024-04-15T18:42:41.290 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:41.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:41.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:41.338 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:42:41.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 97725 97709 73.5675 42.4375 0.0101676 0.0135871 2024-04-15T18:42:41.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:41.447 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:41.475 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:41.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:41.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:42:41.617 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:41.708 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:42:41.768 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: osdmap e745: 12 total, 11 up, 11 in 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:41.769 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:41 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:42:41.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: osdmap e745: 12 total, 11 up, 11 in 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:41.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:41 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:42:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: osdmap e745: 12 total, 11 up, 11 in 2024-04-15T18:42:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:42:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:42:41.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:42:41.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:42:41.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:42:41.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:42:41.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:41 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:42:42.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 98309 98293 73.1262 36.5 0.0346089 0.0136679 2024-04-15T18:42:42.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:42:42.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 24160/289647 objects degraded (8.341%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:42.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:42 smithi159 ceph-mon[45336]: osdmap e746: 12 total, 11 up, 11 in 2024-04-15T18:42:43.139 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 24160/289647 objects degraded (8.341%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:43.140 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:42 smithi082 ceph-mon[44206]: osdmap e746: 12 total, 11 up, 11 in 2024-04-15T18:42:43.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 24160/289647 objects degraded (8.341%), 4 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:43.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:42 smithi049 ceph-mon[40087]: osdmap e746: 12 total, 11 up, 11 in 2024-04-15T18:42:43.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 99008 98992 72.7798 43.6875 0.00574438 0.0137346 2024-04-15T18:42:44.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 15 99651 99636 72.4016 40.25 0.0307494 0.0138052 2024-04-15T18:42:44.394 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:42:44.394 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:42:45.122 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:44 smithi082 ceph-mon[44206]: pgmap v1923: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 29 MiB/s wr, 465 op/s; 30011/294831 objects degraded (10.179%); 4.6 MiB/s, 71 objects/s recovering 2024-04-15T18:42:45.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:44 smithi159 ceph-mon[45336]: pgmap v1923: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 29 MiB/s wr, 465 op/s; 30011/294831 objects degraded (10.179%); 4.6 MiB/s, 71 objects/s recovering 2024-04-15T18:42:45.158 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:42:45.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:44 smithi049 ceph-mon[40087]: pgmap v1923: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 29 MiB/s wr, 465 op/s; 30011/294831 objects degraded (10.179%); 4.6 MiB/s, 71 objects/s recovering 2024-04-15T18:42:45.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 100352 100336 72.0722 43.75 0.00541707 0.0138688 2024-04-15T18:42:45.512 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:42:45.765 INFO:teuthology.orchestra.run.smithi159.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:42:45.786 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:42:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:45.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:42:45.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:42:45.931 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:46.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 100983 100967 71.7013 39.4375 0.0227334 0.0139403 2024-04-15T18:42:46.977 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:42:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:46.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:42:47.067 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:46 smithi049 ceph-mon[40087]: pgmap v1924: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 28 MiB/s wr, 445 op/s; 30011/296094 objects degraded (10.136%); 3.7 MiB/s, 57 objects/s recovering 2024-04-15T18:42:47.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:47.108 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:46 smithi159 ceph-mon[45336]: pgmap v1924: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 28 MiB/s wr, 445 op/s; 30011/296094 objects degraded (10.136%); 3.7 MiB/s, 57 objects/s recovering 2024-04-15T18:42:47.111 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:46 smithi082 ceph-mon[44206]: pgmap v1924: 17 pgs: 1 peering, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 28 MiB/s wr, 445 op/s; 30011/296094 objects degraded (10.136%); 3.7 MiB/s, 57 objects/s recovering 2024-04-15T18:42:47.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 101593 101577 71.324 38.125 0.0165263 0.0140149 2024-04-15T18:42:47.888 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:48.064 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:48.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:48.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 30011/296094 objects degraded (10.136%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:48.170 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:48.206 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:48.230 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:48.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:48.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 30011/296094 objects degraded (10.136%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:48.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:48.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 9 102302 102293 71.0287 44.75 0.0393047 0.0140729 2024-04-15T18:42:48.380 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:48.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 30011/296094 objects degraded (10.136%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:48.633 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:48.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:48.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:42:48.770 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:49.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:48 smithi082 ceph-mon[44206]: pgmap v1925: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 31 MiB/s wr, 499 op/s; 29404/299814 objects degraded (9.807%); 7.7 MiB/s, 121 objects/s recovering 2024-04-15T18:42:49.043 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:42:49.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:48 smithi049 ceph-mon[40087]: pgmap v1925: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 31 MiB/s wr, 499 op/s; 29404/299814 objects degraded (9.807%); 7.7 MiB/s, 121 objects/s recovering 2024-04-15T18:42:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:49.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:42:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:48 smithi159 ceph-mon[45336]: pgmap v1925: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 31 MiB/s wr, 499 op/s; 29404/299814 objects degraded (9.807%); 7.7 MiB/s, 121 objects/s recovering 2024-04-15T18:42:49.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:49.195 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:49.243 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:49.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:49.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:42:49.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.7904 2024-04-15T18:42:49.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 102302 2024-04-15T18:42:49.367 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 70.4246 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 23.3388 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 146.125 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0.25 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 1126 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 373.421 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 2338 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 4 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0140872 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.0335635 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 2.33684 2024-04-15T18:42:49.368 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00103313 2024-04-15T18:42:49.389 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:42:49.398 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:49.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:42:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:49.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:42:49.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:49.585 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:49.853 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:42:49.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1061808732' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:49.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:49 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:49.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1061808732' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:49.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:49 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1061808732' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:50.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:49 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:42:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:50 smithi159 ceph-mon[45336]: pgmap v1926: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 29252/304737 objects degraded (9.599%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:42:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:50 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:42:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:51.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:50 smithi159 ceph-mon[45336]: osdmap e747: 12 total, 11 up, 11 in 2024-04-15T18:42:51.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:50 smithi049 ceph-mon[40087]: pgmap v1926: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 29252/304737 objects degraded (9.599%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:42:51.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:50 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:42:51.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:51.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:50 smithi049 ceph-mon[40087]: osdmap e747: 12 total, 11 up, 11 in 2024-04-15T18:42:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:50 smithi082 ceph-mon[44206]: pgmap v1926: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 29252/304737 objects degraded (9.599%); 7.9 MiB/s, 125 objects/s recovering 2024-04-15T18:42:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:50 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:42:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:42:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:50 smithi082 ceph-mon[44206]: osdmap e747: 12 total, 11 up, 11 in 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:8.3 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:1.0 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:8.d 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:8.6 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:8.7 2024-04-15T18:42:51.329 INFO:teuthology.orchestra.run.smithi159.stdout:8.f 2024-04-15T18:42:52.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:42:52.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:42:52.181 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:42:52.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:42:52.274 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:42:52.322 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:42:52.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:42:52.404 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:42:52.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.11 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --op export --pgid 8.f --file /var/log/ceph/exp.8.f.11 2024-04-15T18:42:52.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:42:52.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:42:52.519 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:42:52.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:42:52.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:42:52.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:42:52.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:42:52.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:42:52.528 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:42:52.679 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.11/config 2024-04-15T18:42:52.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:42:53.105 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:53.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: pgmap v1928: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 104 B/s rd, 31 MiB/s wr, 516 op/s; 29023/305283 objects degraded (9.507%); 7.7 MiB/s, 122 objects/s recovering 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osdmap e748: 12 total, 11 up, 11 in 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3437894321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.109 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.110 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.110 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.110 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.110 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.110 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 29023/305283 objects degraded (9.507%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:53.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: pgmap v1928: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 104 B/s rd, 31 MiB/s wr, 516 op/s; 29023/305283 objects degraded (9.507%); 7.7 MiB/s, 122 objects/s recovering 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osdmap e748: 12 total, 11 up, 11 in 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3437894321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 29023/305283 objects degraded (9.507%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:53.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:42:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: pgmap v1928: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 104 B/s rd, 31 MiB/s wr, 516 op/s; 29023/305283 objects degraded (9.507%); 7.7 MiB/s, 122 objects/s recovering 2024-04-15T18:42:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osdmap e748: 12 total, 11 up, 11 in 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3437894321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 29023/305283 objects degraded (9.507%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:53.239 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:53.284 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:42:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:53.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:42:53.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:53.420 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:53.499 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:53.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:53.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:42:53.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:53.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.857 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:53 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:53.948 INFO:teuthology.orchestra.run.smithi159.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:42:54.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:53 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.106 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:42:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:54.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:42:54.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:42:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:53 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:42:54.249 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:54.274 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:42:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:54.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:42:54.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:54.408 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:54.455 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:54.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:54.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:42:54.610 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:54.732 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:42:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:42:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:42:54.732 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:42:54.732 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:42:54.855 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:54 smithi049 ceph-mon[40087]: pgmap v1930: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 28 MiB/s wr, 691 op/s; 28365/301131 objects degraded (9.419%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:42:55.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:54 smithi082 ceph-mon[44206]: pgmap v1930: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 28 MiB/s wr, 691 op/s; 28365/301131 objects degraded (9.419%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:42:55.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:54 smithi159 ceph-mon[45336]: pgmap v1930: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 28 MiB/s wr, 691 op/s; 28365/301131 objects degraded (9.419%); 13 MiB/s, 205 objects/s recovering 2024-04-15T18:42:55.511 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:42:55.805 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:42:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:55.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:42:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:42:55.833 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:42:55.886 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:42:55.913 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:42:55.926 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:55.955 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:55.987 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:42:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:55.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:42:56.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:56.128 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:56.159 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:42:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:56.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:42:56.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:42:56.325 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:57.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:56 smithi159 ceph-mon[45336]: pgmap v1931: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 587 op/s; 28365/299919 objects degraded (9.458%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:42:57.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:56 smithi049 ceph-mon[40087]: pgmap v1931: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 587 op/s; 28365/299919 objects degraded (9.458%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:42:57.258 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:56 smithi082 ceph-mon[44206]: pgmap v1931: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.1 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 127 B/s rd, 18 MiB/s wr, 587 op/s; 28365/299919 objects degraded (9.458%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:42:57.533 INFO:teuthology.orchestra.run.smithi159.stderr:Exporting 8.f info 8.f( v 733'4786 (0'0,733'4786] local-lis/les=709/710 n=4786 ec=709/709 lis/c=709/709 les/c/f=710/710/0 sis=709) 2024-04-15T18:42:57.563 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0007692:::benchmark_data_smithi159_162467_object9074:head# 2024-04-15T18:42:57.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f001dccb:::benchmark_data_smithi159_162467_object15227:head# 2024-04-15T18:42:57.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0027798:::benchmark_data_smithi159_162467_object35147:head# 2024-04-15T18:42:57.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0027d97:::benchmark_data_smithi159_162467_object10916:head# 2024-04-15T18:42:57.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f002cb85:::benchmark_data_smithi159_162467_object14155:head# 2024-04-15T18:42:57.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0042912:::benchmark_data_smithi159_162467_object27820:head# 2024-04-15T18:42:57.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0047f8b:::benchmark_data_smithi159_162467_object45337:head# 2024-04-15T18:42:57.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0057d3e:::benchmark_data_smithi159_162467_object70432:head# 2024-04-15T18:42:57.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f005c5a8:::benchmark_data_smithi159_162467_object43033:head# 2024-04-15T18:42:57.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0087387:::benchmark_data_smithi159_162467_object57032:head# 2024-04-15T18:42:57.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00ae780:::benchmark_data_smithi159_162467_object11625:head# 2024-04-15T18:42:57.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00b31a4:::benchmark_data_smithi159_162467_object57881:head# 2024-04-15T18:42:57.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00d7f15:::benchmark_data_smithi159_162467_object67870:head# 2024-04-15T18:42:57.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00d9e8c:::benchmark_data_smithi159_162467_object54960:head# 2024-04-15T18:42:57.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00dec60:::benchmark_data_smithi159_162467_object16009:head# 2024-04-15T18:42:57.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00e4858:::benchmark_data_smithi159_162467_object20615:head# 2024-04-15T18:42:57.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00e5e4b:::benchmark_data_smithi159_162467_object37445:head# 2024-04-15T18:42:57.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f00f66c0:::benchmark_data_smithi159_162467_object51718:head# 2024-04-15T18:42:57.573 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01000e3:::benchmark_data_smithi159_162467_object24788:head# 2024-04-15T18:42:57.574 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01055f4:::benchmark_data_smithi159_162467_object69981:head# 2024-04-15T18:42:57.576 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f010848b:::benchmark_data_smithi159_162467_object853:head# 2024-04-15T18:42:57.576 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01091af:::benchmark_data_smithi159_162467_object18491:head# 2024-04-15T18:42:57.577 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f011c6f5:::benchmark_data_smithi159_162467_object35835:head# 2024-04-15T18:42:57.577 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f013beb8:::benchmark_data_smithi159_162467_object13473:head# 2024-04-15T18:42:57.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f015ee08:::benchmark_data_smithi159_162467_object21855:head# 2024-04-15T18:42:57.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01942e9:::benchmark_data_smithi159_162467_object64312:head# 2024-04-15T18:42:57.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0196995:::benchmark_data_smithi159_162467_object17922:head# 2024-04-15T18:42:57.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01b3fb6:::benchmark_data_smithi159_162467_object45217:head# 2024-04-15T18:42:57.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01c8eb3:::benchmark_data_smithi159_162467_object31976:head# 2024-04-15T18:42:57.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01d79ca:::benchmark_data_smithi159_162467_object51773:head# 2024-04-15T18:42:57.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f01f5826:::benchmark_data_smithi159_162467_object18169:head# 2024-04-15T18:42:57.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02055f5:::benchmark_data_smithi159_162467_object48741:head# 2024-04-15T18:42:57.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f022c6ad:::benchmark_data_smithi159_162467_object23553:head# 2024-04-15T18:42:57.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f022d587:::benchmark_data_smithi159_162467_object20678:head# 2024-04-15T18:42:57.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0232719:::benchmark_data_smithi159_162467_object68123:head# 2024-04-15T18:42:57.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0244418:::benchmark_data_smithi159_162467_object7959:head# 2024-04-15T18:42:57.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f025d4b3:::benchmark_data_smithi159_162467_object49961:head# 2024-04-15T18:42:57.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0270319:::benchmark_data_smithi159_162467_object21445:head# 2024-04-15T18:42:57.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f028cd10:::benchmark_data_smithi159_162467_object70399:head# 2024-04-15T18:42:57.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02a1aa8:::benchmark_data_smithi159_162467_object6800:head# 2024-04-15T18:42:57.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02b7a1f:::benchmark_data_smithi159_162467_object5104:head# 2024-04-15T18:42:57.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02c2362:::benchmark_data_smithi159_162467_object16055:head# 2024-04-15T18:42:57.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02c2614:::benchmark_data_smithi159_162467_object10875:head# 2024-04-15T18:42:57.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02c4e85:::benchmark_data_smithi159_162467_object42070:head# 2024-04-15T18:42:57.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02d1ebb:::benchmark_data_smithi159_162467_object63181:head# 2024-04-15T18:42:57.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02d491d:::benchmark_data_smithi159_162467_object63617:head# 2024-04-15T18:42:57.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02d9720:::benchmark_data_smithi159_162467_object1568:head# 2024-04-15T18:42:57.589 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02d9cd6:::benchmark_data_smithi159_162467_object49020:head# 2024-04-15T18:42:57.589 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f02f4199:::benchmark_data_smithi159_162467_object73767:head# 2024-04-15T18:42:57.590 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f030d14a:::benchmark_data_smithi159_162467_object23976:head# 2024-04-15T18:42:57.590 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0323cc5:::benchmark_data_smithi159_162467_object25903:head# 2024-04-15T18:42:57.590 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0330516:::benchmark_data_smithi159_162467_object4906:head# 2024-04-15T18:42:57.591 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0330e28:::benchmark_data_smithi159_162467_object19783:head# 2024-04-15T18:42:57.591 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f033eb3d:::benchmark_data_smithi159_162467_object23916:head# 2024-04-15T18:42:57.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f034406b:::benchmark_data_smithi159_162467_object33293:head# 2024-04-15T18:42:57.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f036e1da:::benchmark_data_smithi159_162467_object40130:head# 2024-04-15T18:42:57.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f037b763:::benchmark_data_smithi159_162467_object46990:head# 2024-04-15T18:42:57.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f037eda8:::benchmark_data_smithi159_162467_object73925:head# 2024-04-15T18:42:57.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0384cce:::benchmark_data_smithi159_162467_object12622:head# 2024-04-15T18:42:57.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03c2c3a:::benchmark_data_smithi159_162467_object74647:head# 2024-04-15T18:42:57.594 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03c4fcf:::benchmark_data_smithi159_162467_object43536:head# 2024-04-15T18:42:57.594 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03c59b6:::benchmark_data_smithi159_162467_object30833:head# 2024-04-15T18:42:57.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03d2284:::benchmark_data_smithi159_162467_object33618:head# 2024-04-15T18:42:57.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03d4b9a:::benchmark_data_smithi159_162467_object42882:head# 2024-04-15T18:42:57.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03ea7f3:::benchmark_data_smithi159_162467_object19894:head# 2024-04-15T18:42:57.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f03ee77f:::benchmark_data_smithi159_162467_object66556:head# 2024-04-15T18:42:57.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0424b88:::benchmark_data_smithi159_162467_object56655:head# 2024-04-15T18:42:57.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0429e01:::benchmark_data_smithi159_162467_object2264:head# 2024-04-15T18:42:57.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0430d2d:::benchmark_data_smithi159_162467_object65364:head# 2024-04-15T18:42:57.598 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04662de:::benchmark_data_smithi159_162467_object25988:head# 2024-04-15T18:42:57.598 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04692f4:::benchmark_data_smithi159_162467_object46683:head# 2024-04-15T18:42:57.598 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04698e7:::benchmark_data_smithi159_162467_object34509:head# 2024-04-15T18:42:57.599 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04759df:::benchmark_data_smithi159_162467_object56521:head# 2024-04-15T18:42:57.599 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0484001:::benchmark_data_smithi159_162467_object28912:head# 2024-04-15T18:42:57.600 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0485e6d:::benchmark_data_smithi159_162467_object16489:head# 2024-04-15T18:42:57.600 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04918a1:::benchmark_data_smithi159_162467_object29182:head# 2024-04-15T18:42:57.600 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04982a5:::benchmark_data_smithi159_162467_object19540:head# 2024-04-15T18:42:57.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f049b6ed:::benchmark_data_smithi159_162467_object70866:head# 2024-04-15T18:42:57.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04a15d9:::benchmark_data_smithi159_162467_object20493:head# 2024-04-15T18:42:57.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04a5e7e:::benchmark_data_smithi159_162467_object42137:head# 2024-04-15T18:42:57.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04b64ef:::benchmark_data_smithi159_162467_object58783:head# 2024-04-15T18:42:57.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04bdd72:::benchmark_data_smithi159_162467_object15721:head# 2024-04-15T18:42:57.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04d15a5:::benchmark_data_smithi159_162467_object29707:head# 2024-04-15T18:42:57.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04d193c:::benchmark_data_smithi159_162467_object31030:head# 2024-04-15T18:42:57.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04d6cdf:::benchmark_data_smithi159_162467_object20441:head# 2024-04-15T18:42:57.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04db0ec:::benchmark_data_smithi159_162467_object17363:head# 2024-04-15T18:42:57.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f04f4b26:::benchmark_data_smithi159_162467_object43800:head# 2024-04-15T18:42:57.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05119a4:::benchmark_data_smithi159_162467_object69011:head# 2024-04-15T18:42:57.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f051260d:::benchmark_data_smithi159_162467_object56232:head# 2024-04-15T18:42:57.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0538203:::benchmark_data_smithi159_162467_object33302:head# 2024-04-15T18:42:57.611 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0540d4a:::benchmark_data_smithi159_162467_object38271:head# 2024-04-15T18:42:57.611 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0549831:::benchmark_data_smithi159_162467_object61874:head# 2024-04-15T18:42:57.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f055ed10:::benchmark_data_smithi159_162467_object45599:head# 2024-04-15T18:42:57.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f056de60:::benchmark_data_smithi159_162467_object64044:head# 2024-04-15T18:42:57.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0575bbd:::benchmark_data_smithi159_162467_object5290:head# 2024-04-15T18:42:57.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f058f8e0:::benchmark_data_smithi159_162467_object10530:head# 2024-04-15T18:42:57.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05ab18e:::benchmark_data_smithi159_162467_object45966:head# 2024-04-15T18:42:57.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05aea4e:::benchmark_data_smithi159_162467_object37871:head# 2024-04-15T18:42:57.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05c4116:::benchmark_data_smithi159_162467_object49350:head# 2024-04-15T18:42:57.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05d3782:::benchmark_data_smithi159_162467_object62447:head# 2024-04-15T18:42:57.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f05d46f0:::benchmark_data_smithi159_162467_object62442:head# 2024-04-15T18:42:57.616 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f060da17:::benchmark_data_smithi159_162467_object9817:head# 2024-04-15T18:42:57.616 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f060ed1c:::benchmark_data_smithi159_162467_object15206:head# 2024-04-15T18:42:57.617 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0610acb:::benchmark_data_smithi159_162467_object51135:head# 2024-04-15T18:42:57.617 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06120f4:::benchmark_data_smithi159_162467_object10314:head# 2024-04-15T18:42:57.617 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0629262:::benchmark_data_smithi159_162467_object68578:head# 2024-04-15T18:42:57.618 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0633c3e:::benchmark_data_smithi159_162467_object50601:head# 2024-04-15T18:42:57.618 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f063e2cf:::benchmark_data_smithi159_162467_object16241:head# 2024-04-15T18:42:57.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f063f01e:::benchmark_data_smithi159_162467_object44282:head# 2024-04-15T18:42:57.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06467a1:::benchmark_data_smithi159_162467_object61524:head# 2024-04-15T18:42:57.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06686f6:::benchmark_data_smithi159_162467_object42891:head# 2024-04-15T18:42:57.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0674e00:::benchmark_data_smithi159_162467_object56812:head# 2024-04-15T18:42:57.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06821da:::benchmark_data_smithi159_162467_object18653:head# 2024-04-15T18:42:57.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0696ed4:::benchmark_data_smithi159_162467_object40456:head# 2024-04-15T18:42:57.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f069f940:::benchmark_data_smithi159_162467_object34827:head# 2024-04-15T18:42:57.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06a28ec:::benchmark_data_smithi159_162467_object6757:head# 2024-04-15T18:42:57.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06e2e98:::benchmark_data_smithi159_162467_object50688:head# 2024-04-15T18:42:57.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06fe90b:::benchmark_data_smithi159_162467_object27108:head# 2024-04-15T18:42:57.623 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f06fee52:::benchmark_data_smithi159_162467_object68996:head# 2024-04-15T18:42:57.625 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0715e97:::benchmark_data_smithi159_162467_object26372:head# 2024-04-15T18:42:57.625 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f073b0ed:::benchmark_data_smithi159_162467_object5023:head# 2024-04-15T18:42:57.626 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f073fcd8:::benchmark_data_smithi159_162467_object68195:head# 2024-04-15T18:42:57.626 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0746f0c:::benchmark_data_smithi159_162467_object6573:head# 2024-04-15T18:42:57.629 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07560e0:::benchmark_data_smithi159_162467_object40632:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f078c4f0:::benchmark_data_smithi159_162467_object22718:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07a72c4:::benchmark_data_smithi159_162467_object30361:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07b4937:::benchmark_data_smithi159_162467_object69517:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07b99e0:::benchmark_data_smithi159_162467_object47996:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07d82f1:::benchmark_data_smithi159_162467_object20604:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f07e4a58:::benchmark_data_smithi159_162467_object57737:head# 2024-04-15T18:42:57.630 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0803b3c:::benchmark_data_smithi159_162467_object44411:head# 2024-04-15T18:42:57.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f080fe8e:::benchmark_data_smithi159_162467_object61391:head# 2024-04-15T18:42:57.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0819869:::benchmark_data_smithi159_162467_object52555:head# 2024-04-15T18:42:57.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f082df34:::benchmark_data_smithi159_162467_object3861:head# 2024-04-15T18:42:57.632 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f083ec22:::benchmark_data_smithi159_162467_object60184:head# 2024-04-15T18:42:57.634 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0844369:::benchmark_data_smithi159_162467_object42937:head# 2024-04-15T18:42:57.637 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0851cde:::benchmark_data_smithi159_162467_object396:head# 2024-04-15T18:42:57.638 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0859815:::benchmark_data_smithi159_162467_object49850:head# 2024-04-15T18:42:57.638 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f086090a:::benchmark_data_smithi159_162467_object25603:head# 2024-04-15T18:42:57.639 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0873728:::benchmark_data_smithi159_162467_object21563:head# 2024-04-15T18:42:57.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f087e281:::benchmark_data_smithi159_162467_object58270:head# 2024-04-15T18:42:57.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0894a8c:::benchmark_data_smithi159_162467_object51614:head# 2024-04-15T18:42:57.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f089d7d1:::benchmark_data_smithi159_162467_object69211:head# 2024-04-15T18:42:57.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08a21e8:::benchmark_data_smithi159_162467_object14068:head# 2024-04-15T18:42:57.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08bb5a5:::benchmark_data_smithi159_162467_object28522:head# 2024-04-15T18:42:57.643 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08d26c7:::benchmark_data_smithi159_162467_object5949:head# 2024-04-15T18:42:57.643 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08d64e7:::benchmark_data_smithi159_162467_object63061:head# 2024-04-15T18:42:57.644 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08dc0b4:::benchmark_data_smithi159_162467_object9170:head# 2024-04-15T18:42:57.644 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08f5481:::benchmark_data_smithi159_162467_object57099:head# 2024-04-15T18:42:57.644 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08fb8a2:::benchmark_data_smithi159_162467_object26269:head# 2024-04-15T18:42:57.645 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08fbedc:::benchmark_data_smithi159_162467_object8190:head# 2024-04-15T18:42:57.645 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08ffcc4:::benchmark_data_smithi159_162467_object69674:head# 2024-04-15T18:42:57.646 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f08ffe12:::benchmark_data_smithi159_162467_object6292:head# 2024-04-15T18:42:57.646 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09117d3:::benchmark_data_smithi159_162467_object3725:head# 2024-04-15T18:42:57.647 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0912013:::benchmark_data_smithi159_162467_object56225:head# 2024-04-15T18:42:57.647 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0917a77:::benchmark_data_smithi159_162467_object23531:head# 2024-04-15T18:42:57.647 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f092f4a7:::benchmark_data_smithi159_162467_object47403:head# 2024-04-15T18:42:57.648 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09378e3:::benchmark_data_smithi159_162467_object7137:head# 2024-04-15T18:42:57.648 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09568ba:::benchmark_data_smithi159_162467_object31709:head# 2024-04-15T18:42:57.649 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0957137:::benchmark_data_smithi159_162467_object12612:head# 2024-04-15T18:42:57.649 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f095a28f:::benchmark_data_smithi159_162467_object6610:head# 2024-04-15T18:42:57.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f095f60a:::benchmark_data_smithi159_162467_object25704:head# 2024-04-15T18:42:57.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0960333:::benchmark_data_smithi159_162467_object1812:head# 2024-04-15T18:42:57.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09697c9:::benchmark_data_smithi159_162467_object47853:head# 2024-04-15T18:42:57.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f097055e:::benchmark_data_smithi159_162467_object8308:head# 2024-04-15T18:42:57.652 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0975400:::benchmark_data_smithi159_162467_object50568:head# 2024-04-15T18:42:57.652 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f099d937:::benchmark_data_smithi159_162467_object4289:head# 2024-04-15T18:42:57.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f099fb87:::benchmark_data_smithi159_162467_object8450:head# 2024-04-15T18:42:57.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09a30d6:::benchmark_data_smithi159_162467_object62273:head# 2024-04-15T18:42:57.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09a7f0d:::benchmark_data_smithi159_162467_object63179:head# 2024-04-15T18:42:57.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09acea7:::benchmark_data_smithi159_162467_object8436:head# 2024-04-15T18:42:57.655 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09c2148:::benchmark_data_smithi159_162467_object10178:head# 2024-04-15T18:42:57.655 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09dcb20:::benchmark_data_smithi159_162467_object61719:head# 2024-04-15T18:42:57.655 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09e9f6e:::benchmark_data_smithi159_162467_object59835:head# 2024-04-15T18:42:57.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f09ed140:::benchmark_data_smithi159_162467_object31652:head# 2024-04-15T18:42:57.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a0955e:::benchmark_data_smithi159_162467_object66413:head# 2024-04-15T18:42:57.658 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a33436:::benchmark_data_smithi159_162467_object24356:head# 2024-04-15T18:42:57.658 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a3834a:::benchmark_data_smithi159_162467_object41176:head# 2024-04-15T18:42:57.658 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a41ea4:::benchmark_data_smithi159_162467_object33315:head# 2024-04-15T18:42:57.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a46563:::benchmark_data_smithi159_162467_object33491:head# 2024-04-15T18:42:57.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a605c1:::benchmark_data_smithi159_162467_object38303:head# 2024-04-15T18:42:57.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a7a3f6:::benchmark_data_smithi159_162467_object41746:head# 2024-04-15T18:42:57.660 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a8fe82:::benchmark_data_smithi159_162467_object20469:head# 2024-04-15T18:42:57.660 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a927ac:::benchmark_data_smithi159_162467_object12618:head# 2024-04-15T18:42:57.660 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0a9b0db:::benchmark_data_smithi159_162467_object650:head# 2024-04-15T18:42:57.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0aa3735:::benchmark_data_smithi159_162467_object12300:head# 2024-04-15T18:42:57.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0aaa1d7:::benchmark_data_smithi159_162467_object22963:head# 2024-04-15T18:42:57.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0aadd45:::benchmark_data_smithi159_162467_object4793:head# 2024-04-15T18:42:57.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ab29b7:::benchmark_data_smithi159_162467_object46267:head# 2024-04-15T18:42:57.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ad2b13:::benchmark_data_smithi159_162467_object24608:head# 2024-04-15T18:42:57.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ae946e:::benchmark_data_smithi159_162467_object1878:head# 2024-04-15T18:42:57.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0af6217:::benchmark_data_smithi159_162467_object68287:head# 2024-04-15T18:42:57.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0af6728:::benchmark_data_smithi159_162467_object35193:head# 2024-04-15T18:42:57.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b02023:::benchmark_data_smithi159_162467_object3144:head# 2024-04-15T18:42:57.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b16573:::benchmark_data_smithi159_162467_object36888:head# 2024-04-15T18:42:57.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b4dcc1:::benchmark_data_smithi159_162467_object42028:head# 2024-04-15T18:42:57.666 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b4de97:::benchmark_data_smithi159_162467_object28278:head# 2024-04-15T18:42:57.666 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b512d1:::benchmark_data_smithi159_162467_object19847:head# 2024-04-15T18:42:57.666 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b602f8:::benchmark_data_smithi159_162467_object30633:head# 2024-04-15T18:42:57.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b6e24c:::benchmark_data_smithi159_162467_object56561:head# 2024-04-15T18:42:57.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b7124c:::benchmark_data_smithi159_162467_object30799:head# 2024-04-15T18:42:57.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b7ea75:::benchmark_data_smithi159_162467_object22885:head# 2024-04-15T18:42:57.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b83154:::benchmark_data_smithi159_162467_object28287:head# 2024-04-15T18:42:57.669 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b9359e:::benchmark_data_smithi159_162467_object17896:head# 2024-04-15T18:42:57.669 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0b97ebe:::benchmark_data_smithi159_162467_object52026:head# 2024-04-15T18:42:57.670 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0bb368d:::benchmark_data_smithi159_162467_object39406:head# 2024-04-15T18:42:57.673 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0bb3ba8:::benchmark_data_smithi159_162467_object37051:head# 2024-04-15T18:42:57.674 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0bb417b:::benchmark_data_smithi159_162467_object70689:head# 2024-04-15T18:42:57.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0be2020:::benchmark_data_smithi159_162467_object36874:head# 2024-04-15T18:42:57.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0be213f:::benchmark_data_smithi159_162467_object44933:head# 2024-04-15T18:42:57.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0be44fc:::benchmark_data_smithi159_162467_object55777:head# 2024-04-15T18:42:57.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0be5c5c:::benchmark_data_smithi159_162467_object55521:head# 2024-04-15T18:42:57.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0be9736:::benchmark_data_smithi159_162467_object13604:head# 2024-04-15T18:42:57.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c05db6:::benchmark_data_smithi159_162467_object47619:head# 2024-04-15T18:42:57.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c08d57:::benchmark_data_smithi159_162467_object51478:head# 2024-04-15T18:42:57.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c11324:::benchmark_data_smithi159_162467_object59206:head# 2024-04-15T18:42:57.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c1635f:::benchmark_data_smithi159_162467_object64199:head# 2024-04-15T18:42:57.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c2d19a:::benchmark_data_smithi159_162467_object7652:head# 2024-04-15T18:42:57.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c3d755:::benchmark_data_smithi159_162467_object44766:head# 2024-04-15T18:42:57.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c41107:::benchmark_data_smithi159_162467_object69349:head# 2024-04-15T18:42:57.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c699cd:::benchmark_data_smithi159_162467_object38140:head# 2024-04-15T18:42:57.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0c772fd:::benchmark_data_smithi159_162467_object47197:head# 2024-04-15T18:42:57.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ca55a9:::benchmark_data_smithi159_162467_object48742:head# 2024-04-15T18:42:57.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cb1ff3:::benchmark_data_smithi159_162467_object42273:head# 2024-04-15T18:42:57.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cb64ca:::benchmark_data_smithi159_162467_object71274:head# 2024-04-15T18:42:57.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cba3d9:::benchmark_data_smithi159_162467_object41335:head# 2024-04-15T18:42:57.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cbe7f0:::benchmark_data_smithi159_162467_object3775:head# 2024-04-15T18:42:57.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cc502b:::benchmark_data_smithi159_162467_object24913:head# 2024-04-15T18:42:57.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ce282c:::benchmark_data_smithi159_162467_object51886:head# 2024-04-15T18:42:57.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cf039e:::benchmark_data_smithi159_162467_object34809:head# 2024-04-15T18:42:57.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cf1286:::benchmark_data_smithi159_162467_object74522:head# 2024-04-15T18:42:57.684 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0cf34d9:::benchmark_data_smithi159_162467_object11872:head# 2024-04-15T18:42:57.684 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d0280c:::benchmark_data_smithi159_162467_object58632:head# 2024-04-15T18:42:57.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d07f6b:::benchmark_data_smithi159_162467_object47211:head# 2024-04-15T18:42:57.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d1b934:::benchmark_data_smithi159_162467_object18605:head# 2024-04-15T18:42:57.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d1ce72:::benchmark_data_smithi159_162467_object51624:head# 2024-04-15T18:42:57.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d27b81:::benchmark_data_smithi159_162467_object62357:head# 2024-04-15T18:42:57.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d450da:::benchmark_data_smithi159_162467_object24944:head# 2024-04-15T18:42:57.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d5c408:::benchmark_data_smithi159_162467_object8038:head# 2024-04-15T18:42:57.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d6493e:::benchmark_data_smithi159_162467_object11477:head# 2024-04-15T18:42:57.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d6eb93:::benchmark_data_smithi159_162467_object36239:head# 2024-04-15T18:42:57.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d821f7:::benchmark_data_smithi159_162467_object30669:head# 2024-04-15T18:42:57.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d8b557:::benchmark_data_smithi159_162467_object10637:head# 2024-04-15T18:42:57.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d9a15e:::benchmark_data_smithi159_162467_object50096:head# 2024-04-15T18:42:57.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0d9ea58:::benchmark_data_smithi159_162467_object5801:head# 2024-04-15T18:42:57.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0da09c5:::benchmark_data_smithi159_162467_object62651:head# 2024-04-15T18:42:57.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0dafe78:::benchmark_data_smithi159_162467_object70993:head# 2024-04-15T18:42:57.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0dbb8a5:::benchmark_data_smithi159_162467_object54143:head# 2024-04-15T18:42:57.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0dc3b1f:::benchmark_data_smithi159_162467_object46304:head# 2024-04-15T18:42:57.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0dd9562:::benchmark_data_smithi159_162467_object41088:head# 2024-04-15T18:42:57.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ddb074:::benchmark_data_smithi159_162467_object70053:head# 2024-04-15T18:42:57.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0de1091:::benchmark_data_smithi159_162467_object70342:head# 2024-04-15T18:42:57.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0de4623:::benchmark_data_smithi159_162467_object39546:head# 2024-04-15T18:42:57.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e10f3a:::benchmark_data_smithi159_162467_object1271:head# 2024-04-15T18:42:57.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e1aee4:::benchmark_data_smithi159_162467_object43235:head# 2024-04-15T18:42:57.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e26e25:::benchmark_data_smithi159_162467_object40828:head# 2024-04-15T18:42:57.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e4b7ca:::benchmark_data_smithi159_162467_object14758:head# 2024-04-15T18:42:57.697 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e63bd9:::benchmark_data_smithi159_162467_object27424:head# 2024-04-15T18:42:57.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e86aa9:::benchmark_data_smithi159_162467_object11515:head# 2024-04-15T18:42:57.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e89f04:::benchmark_data_smithi159_162467_object60477:head# 2024-04-15T18:42:57.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e8dd80:::benchmark_data_smithi159_162467_object46294:head# 2024-04-15T18:42:57.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e976c7:::benchmark_data_smithi159_162467_object5004:head# 2024-04-15T18:42:57.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0e9addb:::benchmark_data_smithi159_162467_object64498:head# 2024-04-15T18:42:57.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ed4d78:::benchmark_data_smithi159_162467_object37471:head# 2024-04-15T18:42:57.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ef03e7:::benchmark_data_smithi159_162467_object71324:head# 2024-04-15T18:42:57.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ef8bc0:::benchmark_data_smithi159_162467_object10618:head# 2024-04-15T18:42:57.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0eff920:::benchmark_data_smithi159_162467_object20327:head# 2024-04-15T18:42:57.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f08de8:::benchmark_data_smithi159_162467_object15418:head# 2024-04-15T18:42:57.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f16bb7:::benchmark_data_smithi159_162467_object29187:head# 2024-04-15T18:42:57.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f2d535:::benchmark_data_smithi159_162467_object16510:head# 2024-04-15T18:42:57.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f4d71f:::benchmark_data_smithi159_162467_object56967:head# 2024-04-15T18:42:57.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f5dd19:::benchmark_data_smithi159_162467_object8101:head# 2024-04-15T18:42:57.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f5e1e8:::benchmark_data_smithi159_162467_object16462:head# 2024-04-15T18:42:57.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f63751:::benchmark_data_smithi159_162467_object33421:head# 2024-04-15T18:42:57.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f656ea:::benchmark_data_smithi159_162467_object52704:head# 2024-04-15T18:42:57.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f77848:::benchmark_data_smithi159_162467_object19444:head# 2024-04-15T18:42:57.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f810e4:::benchmark_data_smithi159_162467_object2649:head# 2024-04-15T18:42:57.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0f93ca7:::benchmark_data_smithi159_162467_object66653:head# 2024-04-15T18:42:57.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fa27a2:::benchmark_data_smithi159_162467_object6325:head# 2024-04-15T18:42:57.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fb89a1:::benchmark_data_smithi159_162467_object63969:head# 2024-04-15T18:42:57.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fc5832:::benchmark_data_smithi159_162467_object46832:head# 2024-04-15T18:42:57.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fdc895:::benchmark_data_smithi159_162467_object28191:head# 2024-04-15T18:42:57.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fdf4dd:::benchmark_data_smithi159_162467_object5423:head# 2024-04-15T18:42:57.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0fed877:::benchmark_data_smithi159_162467_object2224:head# 2024-04-15T18:42:57.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f0ffb23b:::benchmark_data_smithi159_162467_object11896:head# 2024-04-15T18:42:57.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f100a8b4:::benchmark_data_smithi159_162467_object8565:head# 2024-04-15T18:42:57.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f100f22b:::benchmark_data_smithi159_162467_object50386:head# 2024-04-15T18:42:57.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1053937:::benchmark_data_smithi159_162467_object43369:head# 2024-04-15T18:42:57.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f105c061:::benchmark_data_smithi159_162467_object61257:head# 2024-04-15T18:42:57.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f105d816:::benchmark_data_smithi159_162467_object55489:head# 2024-04-15T18:42:57.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f105ffd5:::benchmark_data_smithi159_162467_object72255:head# 2024-04-15T18:42:57.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1060b09:::benchmark_data_smithi159_162467_object59447:head# 2024-04-15T18:42:57.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f106a26d:::benchmark_data_smithi159_162467_object47526:head# 2024-04-15T18:42:57.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f106c204:::benchmark_data_smithi159_162467_object35833:head# 2024-04-15T18:42:57.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f108343c:::benchmark_data_smithi159_162467_object72572:head# 2024-04-15T18:42:57.711 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f108506a:::benchmark_data_smithi159_162467_object48436:head# 2024-04-15T18:42:57.711 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f108759d:::benchmark_data_smithi159_162467_object23476:head# 2024-04-15T18:42:57.711 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1098cd9:::benchmark_data_smithi159_162467_object57197:head# 2024-04-15T18:42:57.712 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10b2cc9:::benchmark_data_smithi159_162467_object14322:head# 2024-04-15T18:42:57.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10b4014:::benchmark_data_smithi159_162467_object68399:head# 2024-04-15T18:42:57.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10bc48b:::benchmark_data_smithi159_162467_object57338:head# 2024-04-15T18:42:57.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10bf21e:::benchmark_data_smithi159_162467_object4357:head# 2024-04-15T18:42:57.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10c97b6:::benchmark_data_smithi159_162467_object45541:head# 2024-04-15T18:42:57.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10d7f6a:::benchmark_data_smithi159_162467_object69032:head# 2024-04-15T18:42:57.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10de642:::benchmark_data_smithi159_162467_object54579:head# 2024-04-15T18:42:57.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f10e37e4:::benchmark_data_smithi159_162467_object27892:head# 2024-04-15T18:42:57.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f110f5c5:::benchmark_data_smithi159_162467_object9129:head# 2024-04-15T18:42:57.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1115c15:::benchmark_data_smithi159_162467_object67139:head# 2024-04-15T18:42:57.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1124338:::benchmark_data_smithi159_162467_object44826:head# 2024-04-15T18:42:57.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f115c9d0:::benchmark_data_smithi159_162467_object70962:head# 2024-04-15T18:42:57.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f115eb6e:::benchmark_data_smithi159_162467_object7900:head# 2024-04-15T18:42:57.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f115edb5:::benchmark_data_smithi159_162467_object20922:head# 2024-04-15T18:42:57.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f115f837:::benchmark_data_smithi159_162467_object27768:head# 2024-04-15T18:42:57.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1173a99:::benchmark_data_smithi159_162467_object24816:head# 2024-04-15T18:42:57.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f119c5d9:::benchmark_data_smithi159_162467_object15178:head# 2024-04-15T18:42:57.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f119da25:::benchmark_data_smithi159_162467_object61485:head# 2024-04-15T18:42:57.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f11e2040:::benchmark_data_smithi159_162467_object17721:head# 2024-04-15T18:42:57.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f11eb54a:::benchmark_data_smithi159_162467_object25111:head# 2024-04-15T18:42:57.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1207b3c:::benchmark_data_smithi159_162467_object62906:head# 2024-04-15T18:42:57.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1210fa6:::benchmark_data_smithi159_162467_object8124:head# 2024-04-15T18:42:57.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12141b3:::benchmark_data_smithi159_162467_object8895:head# 2024-04-15T18:42:57.725 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f121db97:::benchmark_data_smithi159_162467_object24562:head# 2024-04-15T18:42:57.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f124ca36:::benchmark_data_smithi159_162467_object35727:head# 2024-04-15T18:42:57.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1251329:::benchmark_data_smithi159_162467_object51590:head# 2024-04-15T18:42:57.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1269c9c:::benchmark_data_smithi159_162467_object34087:head# 2024-04-15T18:42:57.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f128aae3:::benchmark_data_smithi159_162467_object54748:head# 2024-04-15T18:42:57.728 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1291883:::benchmark_data_smithi159_162467_object74134:head# 2024-04-15T18:42:57.728 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12994d9:::benchmark_data_smithi159_162467_object28110:head# 2024-04-15T18:42:57.729 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12a6d91:::benchmark_data_smithi159_162467_object28676:head# 2024-04-15T18:42:57.729 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12ab793:::benchmark_data_smithi159_162467_object65229:head# 2024-04-15T18:42:57.729 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12c228f:::benchmark_data_smithi159_162467_object63395:head# 2024-04-15T18:42:57.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12d5860:::benchmark_data_smithi159_162467_object1784:head# 2024-04-15T18:42:57.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f12ed092:::benchmark_data_smithi159_162467_object11246:head# 2024-04-15T18:42:57.731 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f132f249:::benchmark_data_smithi159_162467_object9975:head# 2024-04-15T18:42:57.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1333316:::benchmark_data_smithi159_162467_object36712:head# 2024-04-15T18:42:57.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1349c21:::benchmark_data_smithi159_162467_object45899:head# 2024-04-15T18:42:57.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f135e75b:::benchmark_data_smithi159_162467_object54876:head# 2024-04-15T18:42:57.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f136e305:::benchmark_data_smithi159_162467_object65191:head# 2024-04-15T18:42:57.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f136fe7c:::benchmark_data_smithi159_162467_object2512:head# 2024-04-15T18:42:57.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1377789:::benchmark_data_smithi159_162467_object20762:head# 2024-04-15T18:42:57.738 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f139080b:::benchmark_data_smithi159_162467_object14793:head# 2024-04-15T18:42:57.738 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1399cc9:::benchmark_data_smithi159_162467_object17739:head# 2024-04-15T18:42:57.739 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13beb02:::benchmark_data_smithi159_162467_object67967:head# 2024-04-15T18:42:57.739 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13c8898:::benchmark_data_smithi159_162467_object55325:head# 2024-04-15T18:42:57.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13d68c1:::benchmark_data_smithi159_162467_object30691:head# 2024-04-15T18:42:57.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13d8f86:::benchmark_data_smithi159_162467_object69565:head# 2024-04-15T18:42:57.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13e1f4a:::benchmark_data_smithi159_162467_object33264:head# 2024-04-15T18:42:57.743 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f13f1c3f:::benchmark_data_smithi159_162467_object44281:head# 2024-04-15T18:42:57.743 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14017f1:::benchmark_data_smithi159_162467_object35869:head# 2024-04-15T18:42:57.744 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1414f75:::benchmark_data_smithi159_162467_object33140:head# 2024-04-15T18:42:57.744 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f143d5c7:::benchmark_data_smithi159_162467_object24379:head# 2024-04-15T18:42:57.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1451190:::benchmark_data_smithi159_162467_object50123:head# 2024-04-15T18:42:57.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f145a4fa:::benchmark_data_smithi159_162467_object18118:head# 2024-04-15T18:42:57.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f145a78b:::benchmark_data_smithi159_162467_object46894:head# 2024-04-15T18:42:57.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f146520f:::benchmark_data_smithi159_162467_object64959:head# 2024-04-15T18:42:57.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1472ec9:::benchmark_data_smithi159_162467_object54279:head# 2024-04-15T18:42:57.747 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14759c6:::benchmark_data_smithi159_162467_object2189:head# 2024-04-15T18:42:57.747 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14b2494:::benchmark_data_smithi159_162467_object19677:head# 2024-04-15T18:42:57.748 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14c09c6:::benchmark_data_smithi159_162467_object70235:head# 2024-04-15T18:42:57.748 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14c130d:::benchmark_data_smithi159_162467_object53778:head# 2024-04-15T18:42:57.748 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14c7d4a:::benchmark_data_smithi159_162467_object48989:head# 2024-04-15T18:42:57.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14c7e78:::benchmark_data_smithi159_162467_object47700:head# 2024-04-15T18:42:57.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14d21fa:::benchmark_data_smithi159_162467_object56746:head# 2024-04-15T18:42:57.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14e4292:::benchmark_data_smithi159_162467_object18240:head# 2024-04-15T18:42:57.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14ef2be:::benchmark_data_smithi159_162467_object2042:head# 2024-04-15T18:42:57.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14f5efc:::benchmark_data_smithi159_162467_object53239:head# 2024-04-15T18:42:57.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f14ff2cb:::benchmark_data_smithi159_162467_object9788:head# 2024-04-15T18:42:57.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f150f8d7:::benchmark_data_smithi159_162467_object264:head# 2024-04-15T18:42:57.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1512361:::benchmark_data_smithi159_162467_object3290:head# 2024-04-15T18:42:57.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f151da88:::benchmark_data_smithi159_162467_object3251:head# 2024-04-15T18:42:57.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1527f92:::benchmark_data_smithi159_162467_object61819:head# 2024-04-15T18:42:57.753 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1533bfc:::benchmark_data_smithi159_162467_object41066:head# 2024-04-15T18:42:57.753 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f154c95f:::benchmark_data_smithi159_162467_object49259:head# 2024-04-15T18:42:57.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1566d01:::benchmark_data_smithi159_162467_object21927:head# 2024-04-15T18:42:57.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1569571:::benchmark_data_smithi159_162467_object28508:head# 2024-04-15T18:42:57.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f156d61d:::benchmark_data_smithi159_162467_object14172:head# 2024-04-15T18:42:57.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1571f12:::benchmark_data_smithi159_162467_object49902:head# 2024-04-15T18:42:57.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1571f96:::benchmark_data_smithi159_162467_object66947:head# 2024-04-15T18:42:57.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f158e755:::benchmark_data_smithi159_162467_object6589:head# 2024-04-15T18:42:57.756 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1596e4e:::benchmark_data_smithi159_162467_object22188:head# 2024-04-15T18:42:57.756 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15986a5:::benchmark_data_smithi159_162467_object55535:head# 2024-04-15T18:42:57.757 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15a5190:::benchmark_data_smithi159_162467_object72132:head# 2024-04-15T18:42:57.757 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15a7f6f:::benchmark_data_smithi159_162467_object40563:head# 2024-04-15T18:42:57.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15b4747:::benchmark_data_smithi159_162467_object8430:head# 2024-04-15T18:42:57.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15b73f0:::benchmark_data_smithi159_162467_object11976:head# 2024-04-15T18:42:57.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15cc247:::benchmark_data_smithi159_162467_object38568:head# 2024-04-15T18:42:57.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15d0252:::benchmark_data_smithi159_162467_object12446:head# 2024-04-15T18:42:57.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15d609d:::benchmark_data_smithi159_162467_object58550:head# 2024-04-15T18:42:57.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15e550d:::benchmark_data_smithi159_162467_object52193:head# 2024-04-15T18:42:57.760 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f15f739d:::benchmark_data_smithi159_162467_object6143:head# 2024-04-15T18:42:57.760 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f16159a4:::benchmark_data_smithi159_162467_object65593:head# 2024-04-15T18:42:57.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f161d733:::benchmark_data_smithi159_162467_object37301:head# 2024-04-15T18:42:57.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1620b73:::benchmark_data_smithi159_162467_object37973:head# 2024-04-15T18:42:57.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1640b58:::benchmark_data_smithi159_162467_object70396:head# 2024-04-15T18:42:57.762 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1641704:::benchmark_data_smithi159_162467_object64503:head# 2024-04-15T18:42:57.762 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f164243f:::benchmark_data_smithi159_162467_object46391:head# 2024-04-15T18:42:57.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f165199c:::benchmark_data_smithi159_162467_object47210:head# 2024-04-15T18:42:57.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1654a28:::benchmark_data_smithi159_162467_object1096:head# 2024-04-15T18:42:57.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f166baf5:::benchmark_data_smithi159_162467_object70337:head# 2024-04-15T18:42:57.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f167649a:::benchmark_data_smithi159_162467_object73318:head# 2024-04-15T18:42:57.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1677287:::benchmark_data_smithi159_162467_object42078:head# 2024-04-15T18:42:57.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f16b271b:::benchmark_data_smithi159_162467_object10011:head# 2024-04-15T18:42:57.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f16c8eee:::benchmark_data_smithi159_162467_object32031:head# 2024-04-15T18:42:57.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f16cfbbc:::benchmark_data_smithi159_162467_object46111:head# 2024-04-15T18:42:57.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f16cfc83:::benchmark_data_smithi159_162467_object17374:head# 2024-04-15T18:42:57.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1708651:::benchmark_data_smithi159_162467_object27883:head# 2024-04-15T18:42:57.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1710361:::benchmark_data_smithi159_162467_object21415:head# 2024-04-15T18:42:57.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1714f9a:::benchmark_data_smithi159_162467_object16330:head# 2024-04-15T18:42:57.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17209d2:::benchmark_data_smithi159_162467_object12114:head# 2024-04-15T18:42:57.774 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f172d266:::benchmark_data_smithi159_162467_object9790:head# 2024-04-15T18:42:57.774 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f173aa7b:::benchmark_data_smithi159_162467_object28743:head# 2024-04-15T18:42:57.775 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f174e253:::benchmark_data_smithi159_162467_object19138:head# 2024-04-15T18:42:57.775 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f176a0ad:::benchmark_data_smithi159_162467_object53104:head# 2024-04-15T18:42:57.775 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f176b1f2:::benchmark_data_smithi159_162467_object21884:head# 2024-04-15T18:42:57.776 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1775b95:::benchmark_data_smithi159_162467_object11819:head# 2024-04-15T18:42:57.776 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1785e28:::benchmark_data_smithi159_162467_object46110:head# 2024-04-15T18:42:57.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17890a7:::benchmark_data_smithi159_162467_object70116:head# 2024-04-15T18:42:57.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f178b592:::benchmark_data_smithi159_162467_object18298:head# 2024-04-15T18:42:57.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f178bd3e:::benchmark_data_smithi159_162467_object55740:head# 2024-04-15T18:42:57.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17931b6:::benchmark_data_smithi159_162467_object5649:head# 2024-04-15T18:42:57.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17abb59:::benchmark_data_smithi159_162467_object62537:head# 2024-04-15T18:42:57.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17d9941:::benchmark_data_smithi159_162467_object36376:head# 2024-04-15T18:42:57.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f17dfd48:::benchmark_data_smithi159_162467_object45815:head# 2024-04-15T18:42:57.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f180b07e:::benchmark_data_smithi159_162467_object25122:head# 2024-04-15T18:42:57.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f180ffe4:::benchmark_data_smithi159_162467_object14080:head# 2024-04-15T18:42:57.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f182eb33:::benchmark_data_smithi159_162467_object4893:head# 2024-04-15T18:42:57.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f182fa28:::benchmark_data_smithi159_162467_object9162:head# 2024-04-15T18:42:57.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f183bae6:::benchmark_data_smithi159_162467_object21122:head# 2024-04-15T18:42:57.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1854a55:::benchmark_data_smithi159_162467_object72096:head# 2024-04-15T18:42:57.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1867e8e:::benchmark_data_smithi159_162467_object3785:head# 2024-04-15T18:42:57.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f186f1fe:::benchmark_data_smithi159_162467_object5732:head# 2024-04-15T18:42:57.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1871587:::benchmark_data_smithi159_162467_object7981:head# 2024-04-15T18:42:57.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f18a1a15:::benchmark_data_smithi159_162467_object19038:head# 2024-04-15T18:42:57.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f18a8c31:::benchmark_data_smithi159_162467_object25906:head# 2024-04-15T18:42:57.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f18df34e:::benchmark_data_smithi159_162467_object22437:head# 2024-04-15T18:42:57.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f18e3d8a:::benchmark_data_smithi159_162467_object56764:head# 2024-04-15T18:42:57.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f190155b:::benchmark_data_smithi159_162467_object5985:head# 2024-04-15T18:42:57.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1910822:::benchmark_data_smithi159_162467_object18757:head# 2024-04-15T18:42:57.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f191c678:::benchmark_data_smithi159_162467_object57944:head# 2024-04-15T18:42:57.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f192ebb4:::benchmark_data_smithi159_162467_object69824:head# 2024-04-15T18:42:57.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19318e4:::benchmark_data_smithi159_162467_object38013:head# 2024-04-15T18:42:57.791 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1944090:::benchmark_data_smithi159_162467_object54907:head# 2024-04-15T18:42:57.791 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f194dbb9:::benchmark_data_smithi159_162467_object54034:head# 2024-04-15T18:42:57.792 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f194eba6:::benchmark_data_smithi159_162467_object54236:head# 2024-04-15T18:42:57.792 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f195d649:::benchmark_data_smithi159_162467_object53183:head# 2024-04-15T18:42:57.793 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1960d29:::benchmark_data_smithi159_162467_object6120:head# 2024-04-15T18:42:57.793 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f196545e:::benchmark_data_smithi159_162467_object58448:head# 2024-04-15T18:42:57.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f199bbab:::benchmark_data_smithi159_162467_object43722:head# 2024-04-15T18:42:57.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19a3142:::benchmark_data_smithi159_162467_object32655:head# 2024-04-15T18:42:57.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19a9d5b:::benchmark_data_smithi159_162467_object8764:head# 2024-04-15T18:42:57.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19aafbd:::benchmark_data_smithi159_162467_object65663:head# 2024-04-15T18:42:57.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19ab83c:::benchmark_data_smithi159_162467_object26607:head# 2024-04-15T18:42:57.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19b82a8:::benchmark_data_smithi159_162467_object49612:head# 2024-04-15T18:42:57.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19bf61a:::benchmark_data_smithi159_162467_object71016:head# 2024-04-15T18:42:57.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19c9cb7:::benchmark_data_smithi159_162467_object40775:head# 2024-04-15T18:42:57.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19d7d6a:::benchmark_data_smithi159_162467_object31874:head# 2024-04-15T18:42:57.798 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19f229d:::benchmark_data_smithi159_162467_object71307:head# 2024-04-15T18:42:57.799 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f19ff6e9:::benchmark_data_smithi159_162467_object33374:head# 2024-04-15T18:42:57.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a00756:::benchmark_data_smithi159_162467_object72454:head# 2024-04-15T18:42:57.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a05012:::benchmark_data_smithi159_162467_object57411:head# 2024-04-15T18:42:57.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a14d95:::benchmark_data_smithi159_162467_object16790:head# 2024-04-15T18:42:57.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a23f33:::benchmark_data_smithi159_162467_object66833:head# 2024-04-15T18:42:57.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a29ff4:::benchmark_data_smithi159_162467_object41698:head# 2024-04-15T18:42:57.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a66af3:::benchmark_data_smithi159_162467_object15969:head# 2024-04-15T18:42:57.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a68cc1:::benchmark_data_smithi159_162467_object10863:head# 2024-04-15T18:42:57.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a8afe3:::benchmark_data_smithi159_162467_object53219:head# 2024-04-15T18:42:57.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a8dd1d:::benchmark_data_smithi159_162467_object19257:head# 2024-04-15T18:42:57.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a8fadc:::benchmark_data_smithi159_162467_object42686:head# 2024-04-15T18:42:57.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1a98f63:::benchmark_data_smithi159_162467_object71170:head# 2024-04-15T18:42:57.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ab2093:::benchmark_data_smithi159_162467_object29735:head# 2024-04-15T18:42:57.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ab5733:::benchmark_data_smithi159_162467_object72126:head# 2024-04-15T18:42:57.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ad1820:::benchmark_data_smithi159_162467_object48074:head# 2024-04-15T18:42:57.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ad2af9:::benchmark_data_smithi159_162467_object9730:head# 2024-04-15T18:42:57.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ad447a:::benchmark_data_smithi159_162467_object7124:head# 2024-04-15T18:42:57.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ae1c22:::benchmark_data_smithi159_162467_object43535:head# 2024-04-15T18:42:57.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1af17fd:::benchmark_data_smithi159_162467_object16278:head# 2024-04-15T18:42:57.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b0457e:::benchmark_data_smithi159_162467_object22696:head# 2024-04-15T18:42:57.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b123a4:::benchmark_data_smithi159_162467_object74801:head# 2024-04-15T18:42:57.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b21d40:::benchmark_data_smithi159_162467_object55629:head# 2024-04-15T18:42:57.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b4b3b1:::benchmark_data_smithi159_162467_object50677:head# 2024-04-15T18:42:57.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b76afb:::benchmark_data_smithi159_162467_object28143:head# 2024-04-15T18:42:57.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b7731a:::benchmark_data_smithi159_162467_object18215:head# 2024-04-15T18:42:57.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b959c5:::benchmark_data_smithi159_162467_object50353:head# 2024-04-15T18:42:57.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1b9d97b:::benchmark_data_smithi159_162467_object58613:head# 2024-04-15T18:42:57.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bb075c:::benchmark_data_smithi159_162467_object49969:head# 2024-04-15T18:42:57.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bc4728:::benchmark_data_smithi159_162467_object61038:head# 2024-04-15T18:42:57.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bc67ca:::benchmark_data_smithi159_162467_object45118:head# 2024-04-15T18:42:57.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bda84e:::benchmark_data_smithi159_162467_object44330:head# 2024-04-15T18:42:57.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bdc6ee:::benchmark_data_smithi159_162467_object55556:head# 2024-04-15T18:42:57.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1bf1977:::benchmark_data_smithi159_162467_object44947:head# 2024-04-15T18:42:57.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c03fc0:::benchmark_data_smithi159_162467_object34215:head# 2024-04-15T18:42:57.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c05213:::benchmark_data_smithi159_162467_object8971:head# 2024-04-15T18:42:57.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c0ccfa:::benchmark_data_smithi159_162467_object23092:head# 2024-04-15T18:42:57.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c14649:::benchmark_data_smithi159_162467_object45184:head# 2024-04-15T18:42:57.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c15698:::benchmark_data_smithi159_162467_object16467:head# 2024-04-15T18:42:57.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c1609a:::benchmark_data_smithi159_162467_object52132:head# 2024-04-15T18:42:57.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c2e399:::benchmark_data_smithi159_162467_object48050:head# 2024-04-15T18:42:57.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c3bb9b:::benchmark_data_smithi159_162467_object40531:head# 2024-04-15T18:42:57.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c4e0a0:::benchmark_data_smithi159_162467_object34588:head# 2024-04-15T18:42:57.822 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c5d989:::benchmark_data_smithi159_162467_object49579:head# 2024-04-15T18:42:57.822 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c7049a:::benchmark_data_smithi159_162467_object68545:head# 2024-04-15T18:42:57.823 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c76e3b:::benchmark_data_smithi159_162467_object32582:head# 2024-04-15T18:42:57.823 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1c8aa0d:::benchmark_data_smithi159_162467_object10076:head# 2024-04-15T18:42:57.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ca490a:::benchmark_data_smithi159_162467_object12085:head# 2024-04-15T18:42:57.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cba416:::benchmark_data_smithi159_162467_object55991:head# 2024-04-15T18:42:57.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cc2a93:::benchmark_data_smithi159_162467_object42011:head# 2024-04-15T18:42:57.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cd0a96:::benchmark_data_smithi159_162467_object63146:head# 2024-04-15T18:42:57.826 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cd16c6:::benchmark_data_smithi159_162467_object66396:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cddce6:::benchmark_data_smithi159_162467_object55942:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cf4fa0:::benchmark_data_smithi159_162467_object42726:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1cf78c9:::benchmark_data_smithi159_162467_object2007:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d08ca4:::benchmark_data_smithi159_162467_object65543:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d1040b:::benchmark_data_smithi159_162467_object19456:head# 2024-04-15T18:42:57.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d2c0ab:::benchmark_data_smithi159_162467_object2331:head# 2024-04-15T18:42:57.832 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d350f7:::benchmark_data_smithi159_162467_object4329:head# 2024-04-15T18:42:57.832 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d3fb93:::benchmark_data_smithi159_162467_object72051:head# 2024-04-15T18:42:57.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:42:57.835 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d43adc:::benchmark_data_smithi159_162467_object21572:head# 2024-04-15T18:42:57.835 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d48608:::benchmark_data_smithi159_162467_object5850:head# 2024-04-15T18:42:57.835 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d8e69b:::benchmark_data_smithi159_162467_object39597:head# 2024-04-15T18:42:57.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d97add:::benchmark_data_smithi159_162467_object39272:head# 2024-04-15T18:42:57.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1d983b8:::benchmark_data_smithi159_162467_object3308:head# 2024-04-15T18:42:57.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1da002d:::benchmark_data_smithi159_162467_object13965:head# 2024-04-15T18:42:57.837 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1dcd5e9:::benchmark_data_smithi159_162467_object54015:head# 2024-04-15T18:42:57.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1dcdb6a:::benchmark_data_smithi159_162467_object1131:head# 2024-04-15T18:42:57.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1dd8c97:::benchmark_data_smithi159_162467_object66013:head# 2024-04-15T18:42:57.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e0846d:::benchmark_data_smithi159_162467_object21760:head# 2024-04-15T18:42:57.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e0e4c0:::benchmark_data_smithi159_162467_object41179:head# 2024-04-15T18:42:57.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e127d9:::benchmark_data_smithi159_162467_object26225:head# 2024-04-15T18:42:57.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e155db:::benchmark_data_smithi159_162467_object63135:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e16640:::benchmark_data_smithi159_162467_object62415:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e173cd:::benchmark_data_smithi159_162467_object52373:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e1890b:::benchmark_data_smithi159_162467_object43438:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e3706a:::benchmark_data_smithi159_162467_object72577:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e410f8:::benchmark_data_smithi159_162467_object8580:head# 2024-04-15T18:42:57.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e6cab1:::benchmark_data_smithi159_162467_object62802:head# 2024-04-15T18:42:57.846 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e722b4:::benchmark_data_smithi159_162467_object73600:head# 2024-04-15T18:42:57.846 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e7cb59:::benchmark_data_smithi159_162467_object65295:head# 2024-04-15T18:42:57.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e7ded0:::benchmark_data_smithi159_162467_object16031:head# 2024-04-15T18:42:57.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1e9b7ac:::benchmark_data_smithi159_162467_object64551:head# 2024-04-15T18:42:57.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1eaf97e:::benchmark_data_smithi159_162467_object42573:head# 2024-04-15T18:42:57.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ecfc07:::benchmark_data_smithi159_162467_object2861:head# 2024-04-15T18:42:57.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ed8406:::benchmark_data_smithi159_162467_object42022:head# 2024-04-15T18:42:57.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ee63a1:::benchmark_data_smithi159_162467_object29427:head# 2024-04-15T18:42:57.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ef5b72:::benchmark_data_smithi159_162467_object51008:head# 2024-04-15T18:42:57.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f0dd26:::benchmark_data_smithi159_162467_object53907:head# 2024-04-15T18:42:57.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f114ab:::benchmark_data_smithi159_162467_object34145:head# 2024-04-15T18:42:57.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f1bb40:::benchmark_data_smithi159_162467_object67126:head# 2024-04-15T18:42:57.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f1c55b:::benchmark_data_smithi159_162467_object41758:head# 2024-04-15T18:42:57.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f289aa:::benchmark_data_smithi159_162467_object17957:head# 2024-04-15T18:42:57.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f4689c:::benchmark_data_smithi159_162467_object29478:head# 2024-04-15T18:42:57.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f4cc9a:::benchmark_data_smithi159_162467_object34978:head# 2024-04-15T18:42:57.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f51cd7:::benchmark_data_smithi159_162467_object34471:head# 2024-04-15T18:42:57.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f7835e:::benchmark_data_smithi159_162467_object12747:head# 2024-04-15T18:42:57.854 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f9ab10:::benchmark_data_smithi159_162467_object54234:head# 2024-04-15T18:42:57.854 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1f9c3c3:::benchmark_data_smithi159_162467_object25651:head# 2024-04-15T18:42:57.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1fb2a9b:::benchmark_data_smithi159_162467_object15532:head# 2024-04-15T18:42:57.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1fc5bd2:::benchmark_data_smithi159_162467_object2067:head# 2024-04-15T18:42:57.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1fd0e2d:::benchmark_data_smithi159_162467_object70907:head# 2024-04-15T18:42:57.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1fe5550:::benchmark_data_smithi159_162467_object55652:head# 2024-04-15T18:42:57.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ff2ed5:::benchmark_data_smithi159_162467_object49647:head# 2024-04-15T18:42:57.857 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ff30b8:::benchmark_data_smithi159_162467_object68936:head# 2024-04-15T18:42:57.857 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f1ffd76f:::benchmark_data_smithi159_162467_object19491:head# 2024-04-15T18:42:57.858 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2009938:::benchmark_data_smithi159_162467_object52891:head# 2024-04-15T18:42:57.858 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f200b2f6:::benchmark_data_smithi159_162467_object12705:head# 2024-04-15T18:42:57.860 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2019e03:::benchmark_data_smithi159_162467_object15611:head# 2024-04-15T18:42:57.861 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f201be6c:::benchmark_data_smithi159_162467_object45533:head# 2024-04-15T18:42:57.861 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f203c047:::benchmark_data_smithi159_162467_object29292:head# 2024-04-15T18:42:57.861 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f203fcc7:::benchmark_data_smithi159_162467_object30023:head# 2024-04-15T18:42:57.862 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2045c00:::benchmark_data_smithi159_162467_object42667:head# 2024-04-15T18:42:57.862 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2053803:::benchmark_data_smithi159_162467_object15387:head# 2024-04-15T18:42:57.863 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2055e20:::benchmark_data_smithi159_162467_object26723:head# 2024-04-15T18:42:57.863 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2070ef0:::benchmark_data_smithi159_162467_object35298:head# 2024-04-15T18:42:57.864 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f207f197:::benchmark_data_smithi159_162467_object37866:head# 2024-04-15T18:42:57.864 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2081425:::benchmark_data_smithi159_162467_object18992:head# 2024-04-15T18:42:57.865 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f208c250:::benchmark_data_smithi159_162467_object14027:head# 2024-04-15T18:42:57.865 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2091efa:::benchmark_data_smithi159_162467_object73419:head# 2024-04-15T18:42:57.865 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2092277:::benchmark_data_smithi159_162467_object30564:head# 2024-04-15T18:42:57.866 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2095994:::benchmark_data_smithi159_162467_object8001:head# 2024-04-15T18:42:57.866 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f209bcaa:::benchmark_data_smithi159_162467_object38744:head# 2024-04-15T18:42:57.867 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f209c4ac:::benchmark_data_smithi159_162467_object31859:head# 2024-04-15T18:42:57.867 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f209fea3:::benchmark_data_smithi159_162467_object25490:head# 2024-04-15T18:42:57.868 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20a84de:::benchmark_data_smithi159_162467_object33231:head# 2024-04-15T18:42:57.868 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20abb49:::benchmark_data_smithi159_162467_object3952:head# 2024-04-15T18:42:57.870 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20afded:::benchmark_data_smithi159_162467_object42195:head# 2024-04-15T18:42:57.873 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20bd8c3:::benchmark_data_smithi159_162467_object1647:head# 2024-04-15T18:42:57.874 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20c3819:::benchmark_data_smithi159_162467_object15101:head# 2024-04-15T18:42:57.874 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f20d89be:::benchmark_data_smithi159_162467_object17076:head# 2024-04-15T18:42:57.876 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2113204:::benchmark_data_smithi159_162467_object26100:head# 2024-04-15T18:42:57.876 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:42:57.877 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:42:57.877 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 28365/299919 objects degraded (9.458%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:57.877 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f212b0c3:::benchmark_data_smithi159_162467_object42628:head# 2024-04-15T18:42:57.877 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2130e15:::benchmark_data_smithi159_162467_object48639:head# 2024-04-15T18:42:57.877 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f213cf47:::benchmark_data_smithi159_162467_object27682:head# 2024-04-15T18:42:57.878 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f214b74e:::benchmark_data_smithi159_162467_object48465:head# 2024-04-15T18:42:57.878 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f214f0ca:::benchmark_data_smithi159_162467_object68806:head# 2024-04-15T18:42:57.879 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2154838:::benchmark_data_smithi159_162467_object1918:head# 2024-04-15T18:42:57.886 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f218de75:::benchmark_data_smithi159_162467_object73244:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f219155c:::benchmark_data_smithi159_162467_object10089:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21a37a3:::benchmark_data_smithi159_162467_object70632:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21a6afc:::benchmark_data_smithi159_162467_object23268:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21acc5f:::benchmark_data_smithi159_162467_object6642:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21affc4:::benchmark_data_smithi159_162467_object30568:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21be023:::benchmark_data_smithi159_162467_object64156:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21c2fb4:::benchmark_data_smithi159_162467_object21761:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21c3a55:::benchmark_data_smithi159_162467_object23846:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21cf15f:::benchmark_data_smithi159_162467_object9436:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21edc7d:::benchmark_data_smithi159_162467_object67391:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21f0ecd:::benchmark_data_smithi159_162467_object73696:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f21f4ce8:::benchmark_data_smithi159_162467_object18389:head# 2024-04-15T18:42:57.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f220152e:::benchmark_data_smithi159_162467_object20631:head# 2024-04-15T18:42:57.888 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2203771:::benchmark_data_smithi159_162467_object73391:head# 2024-04-15T18:42:57.888 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2207a3a:::benchmark_data_smithi159_162467_object53097:head# 2024-04-15T18:42:57.888 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22095dd:::benchmark_data_smithi159_162467_object63947:head# 2024-04-15T18:42:57.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2228c1d:::benchmark_data_smithi159_162467_object44038:head# 2024-04-15T18:42:57.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2233e33:::benchmark_data_smithi159_162467_object62897:head# 2024-04-15T18:42:57.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2236c76:::benchmark_data_smithi159_162467_object55699:head# 2024-04-15T18:42:57.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2238508:::benchmark_data_smithi159_162467_object5958:head# 2024-04-15T18:42:57.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22431a8:::benchmark_data_smithi159_162467_object59679:head# 2024-04-15T18:42:57.890 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f224ae3d:::benchmark_data_smithi159_162467_object1983:head# 2024-04-15T18:42:57.890 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22544d4:::benchmark_data_smithi159_162467_object10816:head# 2024-04-15T18:42:57.890 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f225593c:::benchmark_data_smithi159_162467_object16803:head# 2024-04-15T18:42:57.891 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22579b9:::benchmark_data_smithi159_162467_object39572:head# 2024-04-15T18:42:57.891 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2268834:::benchmark_data_smithi159_162467_object31650:head# 2024-04-15T18:42:57.895 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22724d2:::benchmark_data_smithi159_162467_object65345:head# 2024-04-15T18:42:57.895 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2273085:::benchmark_data_smithi159_162467_object63238:head# 2024-04-15T18:42:57.896 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2275760:::benchmark_data_smithi159_162467_object37476:head# 2024-04-15T18:42:57.896 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f228173d:::benchmark_data_smithi159_162467_object71761:head# 2024-04-15T18:42:57.896 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22823b0:::benchmark_data_smithi159_162467_object48909:head# 2024-04-15T18:42:57.897 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2286198:::benchmark_data_smithi159_162467_object31605:head# 2024-04-15T18:42:57.897 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f228b80d:::benchmark_data_smithi159_162467_object73933:head# 2024-04-15T18:42:57.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f229cb44:::benchmark_data_smithi159_162467_object21712:head# 2024-04-15T18:42:57.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22c3aa9:::benchmark_data_smithi159_162467_object20435:head# 2024-04-15T18:42:57.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22c89a5:::benchmark_data_smithi159_162467_object60916:head# 2024-04-15T18:42:57.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22e408f:::benchmark_data_smithi159_162467_object10666:head# 2024-04-15T18:42:57.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f22efbf8:::benchmark_data_smithi159_162467_object70333:head# 2024-04-15T18:42:57.900 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23092cb:::benchmark_data_smithi159_162467_object68106:head# 2024-04-15T18:42:57.900 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23240f9:::benchmark_data_smithi159_162467_object48374:head# 2024-04-15T18:42:57.901 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23401db:::benchmark_data_smithi159_162467_object14250:head# 2024-04-15T18:42:57.901 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f234c9d0:::benchmark_data_smithi159_162467_object37361:head# 2024-04-15T18:42:57.902 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23803e4:::benchmark_data_smithi159_162467_object17939:head# 2024-04-15T18:42:57.904 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23877a5:::benchmark_data_smithi159_162467_object72914:head# 2024-04-15T18:42:57.905 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f238966b:::benchmark_data_smithi159_162467_object8831:head# 2024-04-15T18:42:57.905 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2397b5e:::benchmark_data_smithi159_162467_object33660:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23ac60f:::benchmark_data_smithi159_162467_object43818:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23b1e97:::benchmark_data_smithi159_162467_object34622:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23b3c8a:::benchmark_data_smithi159_162467_object67713:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23c519f:::benchmark_data_smithi159_162467_object28614:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23cdde9:::benchmark_data_smithi159_162467_object30572:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23dafe6:::benchmark_data_smithi159_162467_object20773:head# 2024-04-15T18:42:57.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f23e4696:::benchmark_data_smithi159_162467_object2607:head# 2024-04-15T18:42:57.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2408447:::benchmark_data_smithi159_162467_object20505:head# 2024-04-15T18:42:57.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f242b7a0:::benchmark_data_smithi159_162467_object21024:head# 2024-04-15T18:42:57.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2442592:::benchmark_data_smithi159_162467_object10007:head# 2024-04-15T18:42:57.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f244b39a:::benchmark_data_smithi159_162467_object57568:head# 2024-04-15T18:42:57.911 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2471bbd:::benchmark_data_smithi159_162467_object36256:head# 2024-04-15T18:42:57.912 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2471fc2:::benchmark_data_smithi159_162467_object53246:head# 2024-04-15T18:42:57.912 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f247a96b:::benchmark_data_smithi159_162467_object29373:head# 2024-04-15T18:42:57.912 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f247c424:::benchmark_data_smithi159_162467_object7978:head# 2024-04-15T18:42:57.913 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2481cfb:::benchmark_data_smithi159_162467_object31068:head# 2024-04-15T18:42:57.913 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2485022:::benchmark_data_smithi159_162467_object37586:head# 2024-04-15T18:42:57.914 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24850ff:::benchmark_data_smithi159_162467_object1752:head# 2024-04-15T18:42:57.915 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f248e85f:::benchmark_data_smithi159_162467_object73839:head# 2024-04-15T18:42:57.915 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2499239:::benchmark_data_smithi159_162467_object32213:head# 2024-04-15T18:42:57.916 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24b8816:::benchmark_data_smithi159_162467_object69108:head# 2024-04-15T18:42:57.916 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24c5c42:::benchmark_data_smithi159_162467_object7943:head# 2024-04-15T18:42:57.917 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24cc62a:::benchmark_data_smithi159_162467_object43366:head# 2024-04-15T18:42:57.918 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24df08c:::benchmark_data_smithi159_162467_object23634:head# 2024-04-15T18:42:57.918 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24e1ece:::benchmark_data_smithi159_162467_object16593:head# 2024-04-15T18:42:57.919 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24e3ed6:::benchmark_data_smithi159_162467_object59550:head# 2024-04-15T18:42:57.920 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f24f43eb:::benchmark_data_smithi159_162467_object48926:head# 2024-04-15T18:42:57.920 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2518fef:::benchmark_data_smithi159_162467_object11895:head# 2024-04-15T18:42:57.921 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f252b063:::benchmark_data_smithi159_162467_object19037:head# 2024-04-15T18:42:57.921 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f252e80c:::benchmark_data_smithi159_162467_object37606:head# 2024-04-15T18:42:57.924 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f254d572:::benchmark_data_smithi159_162467_object58669:head# 2024-04-15T18:42:57.925 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f255adae:::benchmark_data_smithi159_162467_object25628:head# 2024-04-15T18:42:57.925 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2575eb8:::benchmark_data_smithi159_162467_object14921:head# 2024-04-15T18:42:57.926 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25773ae:::benchmark_data_smithi159_162467_object16347:head# 2024-04-15T18:42:57.926 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2590cdb:::benchmark_data_smithi159_162467_object8025:head# 2024-04-15T18:42:57.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25c4bf8:::benchmark_data_smithi159_162467_object52006:head# 2024-04-15T18:42:57.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25d25fd:::benchmark_data_smithi159_162467_object11366:head# 2024-04-15T18:42:57.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25d9876:::benchmark_data_smithi159_162467_object41202:head# 2024-04-15T18:42:57.928 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25e33dd:::benchmark_data_smithi159_162467_object40922:head# 2024-04-15T18:42:57.928 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f25f97c4:::benchmark_data_smithi159_162467_object68093:head# 2024-04-15T18:42:57.929 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2602e16:::benchmark_data_smithi159_162467_object64295:head# 2024-04-15T18:42:57.929 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f260a9bb:::benchmark_data_smithi159_162467_object70000:head# 2024-04-15T18:42:57.930 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2624383:::benchmark_data_smithi159_162467_object9402:head# 2024-04-15T18:42:57.930 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26292f6:::benchmark_data_smithi159_162467_object49367:head# 2024-04-15T18:42:57.931 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f262cc1d:::benchmark_data_smithi159_162467_object31936:head# 2024-04-15T18:42:57.931 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2630a6f:::benchmark_data_smithi159_162467_object40172:head# 2024-04-15T18:42:57.932 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2636c83:::benchmark_data_smithi159_162467_object67045:head# 2024-04-15T18:42:57.932 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f263cbb3:::benchmark_data_smithi159_162467_object29240:head# 2024-04-15T18:42:57.933 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f263d400:::benchmark_data_smithi159_162467_object16370:head# 2024-04-15T18:42:57.933 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f264a94d:::benchmark_data_smithi159_162467_object70269:head# 2024-04-15T18:42:57.933 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2658f8c:::benchmark_data_smithi159_162467_object7252:head# 2024-04-15T18:42:57.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2662e81:::benchmark_data_smithi159_162467_object58698:head# 2024-04-15T18:42:57.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f266d2fa:::benchmark_data_smithi159_162467_object25042:head# 2024-04-15T18:42:57.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f267259b:::benchmark_data_smithi159_162467_object26888:head# 2024-04-15T18:42:57.935 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f267db66:::benchmark_data_smithi159_162467_object41332:head# 2024-04-15T18:42:57.936 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f267fd9c:::benchmark_data_smithi159_162467_object62998:head# 2024-04-15T18:42:57.936 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2689534:::benchmark_data_smithi159_162467_object18523:head# 2024-04-15T18:42:57.936 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f268ae50:::benchmark_data_smithi159_162467_object3682:head# 2024-04-15T18:42:57.937 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26959dc:::benchmark_data_smithi159_162467_object57886:head# 2024-04-15T18:42:57.937 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26bc838:::benchmark_data_smithi159_162467_object30935:head# 2024-04-15T18:42:57.938 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26d09c8:::benchmark_data_smithi159_162467_object14273:head# 2024-04-15T18:42:57.938 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26e9356:::benchmark_data_smithi159_162467_object59364:head# 2024-04-15T18:42:57.939 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f26f0013:::benchmark_data_smithi159_162467_object70753:head# 2024-04-15T18:42:57.939 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27018b4:::benchmark_data_smithi159_162467_object4406:head# 2024-04-15T18:42:57.939 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2706ea6:::benchmark_data_smithi159_162467_object44261:head# 2024-04-15T18:42:57.940 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27098d5:::benchmark_data_smithi159_162467_object66742:head# 2024-04-15T18:42:57.940 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27124bc:::benchmark_data_smithi159_162467_object8382:head# 2024-04-15T18:42:57.941 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f274225b:::benchmark_data_smithi159_162467_object44249:head# 2024-04-15T18:42:57.941 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f274b4ec:::benchmark_data_smithi159_162467_object56217:head# 2024-04-15T18:42:57.942 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f275c5d6:::benchmark_data_smithi159_162467_object10948:head# 2024-04-15T18:42:57.942 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2769459:::benchmark_data_smithi159_162467_object25960:head# 2024-04-15T18:42:57.942 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2777431:::benchmark_data_smithi159_162467_object8829:head# 2024-04-15T18:42:57.943 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2778195:::benchmark_data_smithi159_162467_object51697:head# 2024-04-15T18:42:57.943 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f278ce51:::benchmark_data_smithi159_162467_object26801:head# 2024-04-15T18:42:57.944 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2792311:::benchmark_data_smithi159_162467_object56902:head# 2024-04-15T18:42:57.944 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f279bdcd:::benchmark_data_smithi159_162467_object67910:head# 2024-04-15T18:42:57.945 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f279cd2d:::benchmark_data_smithi159_162467_object8615:head# 2024-04-15T18:42:57.945 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27b7970:::benchmark_data_smithi159_162467_object51807:head# 2024-04-15T18:42:57.946 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27bb612:::benchmark_data_smithi159_162467_object54531:head# 2024-04-15T18:42:57.946 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27ce395:::benchmark_data_smithi159_162467_object35026:head# 2024-04-15T18:42:57.946 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27f4911:::benchmark_data_smithi159_162467_object5160:head# 2024-04-15T18:42:57.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27f7b87:::benchmark_data_smithi159_162467_object42512:head# 2024-04-15T18:42:57.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f27fcd47:::benchmark_data_smithi159_162467_object67799:head# 2024-04-15T18:42:57.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28090c4:::benchmark_data_smithi159_162467_object67558:head# 2024-04-15T18:42:57.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2812d8c:::benchmark_data_smithi159_162467_object41681:head# 2024-04-15T18:42:57.949 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f281cf22:::benchmark_data_smithi159_162467_object48023:head# 2024-04-15T18:42:57.949 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f283ae53:::benchmark_data_smithi159_162467_object71290:head# 2024-04-15T18:42:57.950 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2876f4a:::benchmark_data_smithi159_162467_object48929:head# 2024-04-15T18:42:57.950 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f289f0f6:::benchmark_data_smithi159_162467_object48416:head# 2024-04-15T18:42:57.950 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28a3f37:::benchmark_data_smithi159_162467_object52531:head# 2024-04-15T18:42:57.951 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28b15b5:::benchmark_data_smithi159_162467_object36932:head# 2024-04-15T18:42:57.951 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28bb2d0:::benchmark_data_smithi159_162467_object72595:head# 2024-04-15T18:42:57.951 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28bba51:::benchmark_data_smithi159_162467_object62300:head# 2024-04-15T18:42:57.952 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28ee051:::benchmark_data_smithi159_162467_object19288:head# 2024-04-15T18:42:57.954 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f28fca23:::benchmark_data_smithi159_162467_object43180:head# 2024-04-15T18:42:57.954 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2919bac:::benchmark_data_smithi159_162467_object31766:head# 2024-04-15T18:42:57.955 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f293758e:::benchmark_data_smithi159_162467_object385:head# 2024-04-15T18:42:57.957 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2937827:::benchmark_data_smithi159_162467_object61465:head# 2024-04-15T18:42:57.957 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f293879f:::benchmark_data_smithi159_162467_object4062:head# 2024-04-15T18:42:57.958 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f293e974:::benchmark_data_smithi159_162467_object65550:head# 2024-04-15T18:42:57.958 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f294182b:::benchmark_data_smithi159_162467_object47151:head# 2024-04-15T18:42:57.959 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2977c12:::benchmark_data_smithi159_162467_object52455:head# 2024-04-15T18:42:57.959 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2996e93:::benchmark_data_smithi159_162467_object36948:head# 2024-04-15T18:42:57.959 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29a6245:::benchmark_data_smithi159_162467_object7616:head# 2024-04-15T18:42:57.960 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29a7679:::benchmark_data_smithi159_162467_object56642:head# 2024-04-15T18:42:57.960 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29b7094:::benchmark_data_smithi159_162467_object52661:head# 2024-04-15T18:42:57.961 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29d8974:::benchmark_data_smithi159_162467_object69240:head# 2024-04-15T18:42:57.961 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29e8524:::benchmark_data_smithi159_162467_object58000:head# 2024-04-15T18:42:57.962 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29ef07a:::benchmark_data_smithi159_162467_object30248:head# 2024-04-15T18:42:57.962 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29ef21c:::benchmark_data_smithi159_162467_object2625:head# 2024-04-15T18:42:57.962 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29fa01f:::benchmark_data_smithi159_162467_object43434:head# 2024-04-15T18:42:57.963 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f29fb707:::benchmark_data_smithi159_162467_object54074:head# 2024-04-15T18:42:57.963 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a0f87f:::benchmark_data_smithi159_162467_object40848:head# 2024-04-15T18:42:57.963 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a29c66:::benchmark_data_smithi159_162467_object55565:head# 2024-04-15T18:42:57.964 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a4c1e1:::benchmark_data_smithi159_162467_object54277:head# 2024-04-15T18:42:57.964 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a4cf5e:::benchmark_data_smithi159_162467_object48782:head# 2024-04-15T18:42:57.965 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a53560:::benchmark_data_smithi159_162467_object35382:head# 2024-04-15T18:42:57.965 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a56652:::benchmark_data_smithi159_162467_object52540:head# 2024-04-15T18:42:57.966 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a7020e:::benchmark_data_smithi159_162467_object29449:head# 2024-04-15T18:42:57.966 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2a7cb09:::benchmark_data_smithi159_162467_object13031:head# 2024-04-15T18:42:57.967 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2aae6de:::benchmark_data_smithi159_162467_object27547:head# 2024-04-15T18:42:57.967 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ab18b3:::benchmark_data_smithi159_162467_object41407:head# 2024-04-15T18:42:57.967 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2adb979:::benchmark_data_smithi159_162467_object38967:head# 2024-04-15T18:42:57.968 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ae2f6b:::benchmark_data_smithi159_162467_object15285:head# 2024-04-15T18:42:57.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2af099c:::benchmark_data_smithi159_162467_object12652:head# 2024-04-15T18:42:57.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2af187e:::benchmark_data_smithi159_162467_object39887:head# 2024-04-15T18:42:57.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b06c84:::benchmark_data_smithi159_162467_object49937:head# 2024-04-15T18:42:57.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b09209:::benchmark_data_smithi159_162467_object42517:head# 2024-04-15T18:42:57.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b148b5:::benchmark_data_smithi159_162467_object1291:head# 2024-04-15T18:42:57.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b43f19:::benchmark_data_smithi159_162467_object5666:head# 2024-04-15T18:42:57.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b5b507:::benchmark_data_smithi159_162467_object32277:head# 2024-04-15T18:42:57.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b5fef8:::benchmark_data_smithi159_162467_object15908:head# 2024-04-15T18:42:57.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b87065:::benchmark_data_smithi159_162467_object31562:head# 2024-04-15T18:42:57.972 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b8a775:::benchmark_data_smithi159_162467_object40270:head# 2024-04-15T18:42:57.972 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b8e48d:::benchmark_data_smithi159_162467_object30281:head# 2024-04-15T18:42:57.973 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2b9ea07:::benchmark_data_smithi159_162467_object57305:head# 2024-04-15T18:42:57.973 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ba7615:::benchmark_data_smithi159_162467_object3428:head# 2024-04-15T18:42:57.974 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bb2de1:::benchmark_data_smithi159_162467_object6951:head# 2024-04-15T18:42:57.974 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bb40c2:::benchmark_data_smithi159_162467_object49984:head# 2024-04-15T18:42:57.974 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bb9bf5:::benchmark_data_smithi159_162467_object24758:head# 2024-04-15T18:42:57.975 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bbf9af:::benchmark_data_smithi159_162467_object58685:head# 2024-04-15T18:42:57.975 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2be2749:::benchmark_data_smithi159_162467_object62222:head# 2024-04-15T18:42:57.976 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2be8588:::benchmark_data_smithi159_162467_object53624:head# 2024-04-15T18:42:57.976 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bea2aa:::benchmark_data_smithi159_162467_object58517:head# 2024-04-15T18:42:57.976 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2bee4b6:::benchmark_data_smithi159_162467_object19767:head# 2024-04-15T18:42:57.977 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c0150c:::benchmark_data_smithi159_162467_object46285:head# 2024-04-15T18:42:57.977 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c083c4:::benchmark_data_smithi159_162467_object15502:head# 2024-04-15T18:42:57.978 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c1f1f1:::benchmark_data_smithi159_162467_object26226:head# 2024-04-15T18:42:57.978 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c25c27:::benchmark_data_smithi159_162467_object11555:head# 2024-04-15T18:42:57.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c27b5f:::benchmark_data_smithi159_162467_object25992:head# 2024-04-15T18:42:57.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c2a718:::benchmark_data_smithi159_162467_object71661:head# 2024-04-15T18:42:57.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c45f53:::benchmark_data_smithi159_162467_object17340:head# 2024-04-15T18:42:57.980 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c4dfe8:::benchmark_data_smithi159_162467_object25784:head# 2024-04-15T18:42:57.980 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c55ca1:::benchmark_data_smithi159_162467_object46203:head# 2024-04-15T18:42:57.981 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c62dbe:::benchmark_data_smithi159_162467_object14087:head# 2024-04-15T18:42:57.981 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c6a17a:::benchmark_data_smithi159_162467_object2379:head# 2024-04-15T18:42:57.982 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c8aacf:::benchmark_data_smithi159_162467_object9850:head# 2024-04-15T18:42:57.982 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c952c1:::benchmark_data_smithi159_162467_object26958:head# 2024-04-15T18:42:57.982 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2c965f8:::benchmark_data_smithi159_162467_object23053:head# 2024-04-15T18:42:57.983 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2cc1778:::benchmark_data_smithi159_162467_object14562:head# 2024-04-15T18:42:57.983 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2cc49df:::benchmark_data_smithi159_162467_object11748:head# 2024-04-15T18:42:57.983 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2cd140a:::benchmark_data_smithi159_162467_object65123:head# 2024-04-15T18:42:57.984 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2cd2eb4:::benchmark_data_smithi159_162467_object30204:head# 2024-04-15T18:42:57.984 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2cf6e43:::benchmark_data_smithi159_162467_object53151:head# 2024-04-15T18:42:57.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d1204a:::benchmark_data_smithi159_162467_object55243:head# 2024-04-15T18:42:57.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d1faec:::benchmark_data_smithi159_162467_object70477:head# 2024-04-15T18:42:57.986 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d24eae:::benchmark_data_smithi159_162467_object48056:head# 2024-04-15T18:42:57.986 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d27d28:::benchmark_data_smithi159_162467_object22814:head# 2024-04-15T18:42:57.987 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d49b7a:::benchmark_data_smithi159_162467_object68642:head# 2024-04-15T18:42:57.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d4d561:::benchmark_data_smithi159_162467_object9354:head# 2024-04-15T18:42:57.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d4fc7e:::benchmark_data_smithi159_162467_object32267:head# 2024-04-15T18:42:57.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d5e6f1:::benchmark_data_smithi159_162467_object5289:head# 2024-04-15T18:42:57.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d6a1b5:::benchmark_data_smithi159_162467_object58161:head# 2024-04-15T18:42:57.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2d91786:::benchmark_data_smithi159_162467_object12928:head# 2024-04-15T18:42:57.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2da2ef6:::benchmark_data_smithi159_162467_object36279:head# 2024-04-15T18:42:57.990 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2dac05a:::benchmark_data_smithi159_162467_object23062:head# 2024-04-15T18:42:57.990 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2db20ce:::benchmark_data_smithi159_162467_object55632:head# 2024-04-15T18:42:57.991 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2db642c:::benchmark_data_smithi159_162467_object1602:head# 2024-04-15T18:42:57.991 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2db66b4:::benchmark_data_smithi159_162467_object69865:head# 2024-04-15T18:42:57.992 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2dbf4fe:::benchmark_data_smithi159_162467_object65671:head# 2024-04-15T18:42:57.992 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2dc3138:::benchmark_data_smithi159_162467_object64157:head# 2024-04-15T18:42:57.993 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2de8ef2:::benchmark_data_smithi159_162467_object17718:head# 2024-04-15T18:42:57.993 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2df32ae:::benchmark_data_smithi159_162467_object61124:head# 2024-04-15T18:42:57.993 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2dfd0cb:::benchmark_data_smithi159_162467_object33079:head# 2024-04-15T18:42:57.994 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e0031b:::benchmark_data_smithi159_162467_object68974:head# 2024-04-15T18:42:57.994 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e42c3c:::benchmark_data_smithi159_162467_object43687:head# 2024-04-15T18:42:57.995 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e43c7b:::benchmark_data_smithi159_162467_object29100:head# 2024-04-15T18:42:57.995 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e48a17:::benchmark_data_smithi159_162467_object4175:head# 2024-04-15T18:42:57.995 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e5c614:::benchmark_data_smithi159_162467_object640:head# 2024-04-15T18:42:57.996 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e728a9:::benchmark_data_smithi159_162467_object70058:head# 2024-04-15T18:42:57.996 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e7853f:::benchmark_data_smithi159_162467_object56491:head# 2024-04-15T18:42:57.996 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e7f2b7:::benchmark_data_smithi159_162467_object19126:head# 2024-04-15T18:42:57.997 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e88a67:::benchmark_data_smithi159_162467_object455:head# 2024-04-15T18:42:57.997 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e8ac43:::benchmark_data_smithi159_162467_object17912:head# 2024-04-15T18:42:57.998 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e8c946:::benchmark_data_smithi159_162467_object28919:head# 2024-04-15T18:42:57.998 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2e9a3d9:::benchmark_data_smithi159_162467_object28783:head# 2024-04-15T18:42:57.998 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ea3a85:::benchmark_data_smithi159_162467_object37585:head# 2024-04-15T18:42:57.999 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2eb77a3:::benchmark_data_smithi159_162467_object7341:head# 2024-04-15T18:42:58.000 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ecb479:::benchmark_data_smithi159_162467_object21315:head# 2024-04-15T18:42:58.000 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2edbf51:::benchmark_data_smithi159_162467_object25078:head# 2024-04-15T18:42:58.000 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ef8af3:::benchmark_data_smithi159_162467_object8498:head# 2024-04-15T18:42:58.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f27617:::benchmark_data_smithi159_162467_object52366:head# 2024-04-15T18:42:58.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f38190:::benchmark_data_smithi159_162467_object57053:head# 2024-04-15T18:42:58.002 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f4251b:::benchmark_data_smithi159_162467_object20526:head# 2024-04-15T18:42:58.002 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f4c395:::benchmark_data_smithi159_162467_object40108:head# 2024-04-15T18:42:58.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f57505:::benchmark_data_smithi159_162467_object11790:head# 2024-04-15T18:42:58.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f800e6:::benchmark_data_smithi159_162467_object17060:head# 2024-04-15T18:42:58.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f8400d:::benchmark_data_smithi159_162467_object39445:head# 2024-04-15T18:42:58.004 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2f948a3:::benchmark_data_smithi159_162467_object30679:head# 2024-04-15T18:42:58.004 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fab0ae:::benchmark_data_smithi159_162467_object15001:head# 2024-04-15T18:42:58.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fab2e2:::benchmark_data_smithi159_162467_object25469:head# 2024-04-15T18:42:58.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2faee4c:::benchmark_data_smithi159_162467_object68263:head# 2024-04-15T18:42:58.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fb66e4:::benchmark_data_smithi159_162467_object46776:head# 2024-04-15T18:42:58.006 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fbf0de:::benchmark_data_smithi159_162467_object73060:head# 2024-04-15T18:42:58.006 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fbf33c:::benchmark_data_smithi159_162467_object10484:head# 2024-04-15T18:42:58.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fc7f53:::benchmark_data_smithi159_162467_object6259:head# 2024-04-15T18:42:58.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fce48b:::benchmark_data_smithi159_162467_object30046:head# 2024-04-15T18:42:58.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fe12ee:::benchmark_data_smithi159_162467_object62699:head# 2024-04-15T18:42:58.008 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2fea33b:::benchmark_data_smithi159_162467_object62636:head# 2024-04-15T18:42:58.008 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2feed63:::benchmark_data_smithi159_162467_object59086:head# 2024-04-15T18:42:58.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f2ffed37:::benchmark_data_smithi159_162467_object47304:head# 2024-04-15T18:42:58.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f30082b7:::benchmark_data_smithi159_162467_object44986:head# 2024-04-15T18:42:58.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f30160df:::benchmark_data_smithi159_162467_object60742:head# 2024-04-15T18:42:58.010 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3026671:::benchmark_data_smithi159_162467_object32424:head# 2024-04-15T18:42:58.010 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3027a97:::benchmark_data_smithi159_162467_object62246:head# 2024-04-15T18:42:58.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f305fad4:::benchmark_data_smithi159_162467_object7213:head# 2024-04-15T18:42:58.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f306b1cd:::benchmark_data_smithi159_162467_object54058:head# 2024-04-15T18:42:58.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3082e46:::benchmark_data_smithi159_162467_object33214:head# 2024-04-15T18:42:58.012 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3085290:::benchmark_data_smithi159_162467_object8370:head# 2024-04-15T18:42:58.012 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f308b39c:::benchmark_data_smithi159_162467_object71741:head# 2024-04-15T18:42:58.013 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f30b4254:::benchmark_data_smithi159_162467_object20097:head# 2024-04-15T18:42:58.013 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f30db67b:::benchmark_data_smithi159_162467_object71806:head# 2024-04-15T18:42:58.013 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3106b69:::benchmark_data_smithi159_162467_object6245:head# 2024-04-15T18:42:58.014 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f312c0c5:::benchmark_data_smithi159_162467_object73402:head# 2024-04-15T18:42:58.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f314cc13:::benchmark_data_smithi159_162467_object19368:head# 2024-04-15T18:42:58.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3151fcd:::benchmark_data_smithi159_162467_object38134:head# 2024-04-15T18:42:58.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f315622e:::benchmark_data_smithi159_162467_object70223:head# 2024-04-15T18:42:58.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f315770e:::benchmark_data_smithi159_162467_object35652:head# 2024-04-15T18:42:58.016 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f315bac2:::benchmark_data_smithi159_162467_object32019:head# 2024-04-15T18:42:58.016 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f316a44d:::benchmark_data_smithi159_162467_object6997:head# 2024-04-15T18:42:58.017 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f317343b:::benchmark_data_smithi159_162467_object33932:head# 2024-04-15T18:42:58.017 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f317e656:::benchmark_data_smithi159_162467_object74652:head# 2024-04-15T18:42:58.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f318230e:::benchmark_data_smithi159_162467_object5007:head# 2024-04-15T18:42:58.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f31840e3:::benchmark_data_smithi159_162467_object8698:head# 2024-04-15T18:42:58.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f318d78e:::benchmark_data_smithi159_162467_object29787:head# 2024-04-15T18:42:58.019 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3192c8f:::benchmark_data_smithi159_162467_object2943:head# 2024-04-15T18:42:58.019 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3193896:::benchmark_data_smithi159_162467_object8363:head# 2024-04-15T18:42:58.019 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3194f1a:::benchmark_data_smithi159_162467_object10876:head# 2024-04-15T18:42:58.020 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f31ea49b:::benchmark_data_smithi159_162467_object33980:head# 2024-04-15T18:42:58.020 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f31f2132:::benchmark_data_smithi159_162467_object54846:head# 2024-04-15T18:42:58.021 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3203984:::benchmark_data_smithi159_162467_object28965:head# 2024-04-15T18:42:58.021 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f320ad53:::benchmark_data_smithi159_162467_object30:head# 2024-04-15T18:42:58.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f320be24:::benchmark_data_smithi159_162467_object58951:head# 2024-04-15T18:42:58.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3212609:::benchmark_data_smithi159_162467_object54832:head# 2024-04-15T18:42:58.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3236f24:::benchmark_data_smithi159_162467_object18729:head# 2024-04-15T18:42:58.023 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3252b0d:::benchmark_data_smithi159_162467_object64298:head# 2024-04-15T18:42:58.023 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f325711d:::benchmark_data_smithi159_162467_object40684:head# 2024-04-15T18:42:58.024 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f326a4ec:::benchmark_data_smithi159_162467_object65340:head# 2024-04-15T18:42:58.024 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f327c9cf:::benchmark_data_smithi159_162467_object16650:head# 2024-04-15T18:42:58.025 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32923e5:::benchmark_data_smithi159_162467_object2016:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32977cb:::benchmark_data_smithi159_162467_object10736:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32a8be8:::benchmark_data_smithi159_162467_object38122:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32afb0f:::benchmark_data_smithi159_162467_object2371:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32b14d0:::benchmark_data_smithi159_162467_object61034:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32c04d4:::benchmark_data_smithi159_162467_object35895:head# 2024-04-15T18:42:58.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f32d1705:::benchmark_data_smithi159_162467_object48867:head# 2024-04-15T18:42:58.029 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3307629:::benchmark_data_smithi159_162467_object2223:head# 2024-04-15T18:42:58.029 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f330f9ed:::benchmark_data_smithi159_162467_object5000:head# 2024-04-15T18:42:58.030 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f331c7cf:::benchmark_data_smithi159_162467_object69396:head# 2024-04-15T18:42:58.030 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f331cd78:::benchmark_data_smithi159_162467_object61427:head# 2024-04-15T18:42:58.030 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3328a02:::benchmark_data_smithi159_162467_object49063:head# 2024-04-15T18:42:58.031 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f332b1f3:::benchmark_data_smithi159_162467_object26290:head# 2024-04-15T18:42:58.031 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f332c565:::benchmark_data_smithi159_162467_object36361:head# 2024-04-15T18:42:58.031 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f334513b:::benchmark_data_smithi159_162467_object65268:head# 2024-04-15T18:42:58.032 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f33494da:::benchmark_data_smithi159_162467_object38249:head# 2024-04-15T18:42:58.032 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f334a39b:::benchmark_data_smithi159_162467_object9458:head# 2024-04-15T18:42:58.033 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f334af9e:::benchmark_data_smithi159_162467_object21593:head# 2024-04-15T18:42:58.033 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f335d54d:::benchmark_data_smithi159_162467_object29780:head# 2024-04-15T18:42:58.034 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f335f7fb:::benchmark_data_smithi159_162467_object52461:head# 2024-04-15T18:42:58.034 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f336c34d:::benchmark_data_smithi159_162467_object45820:head# 2024-04-15T18:42:58.035 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f339ed61:::benchmark_data_smithi159_162467_object39356:head# 2024-04-15T18:42:58.035 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f33ab726:::benchmark_data_smithi159_162467_object6008:head# 2024-04-15T18:42:58.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f33ad1ad:::benchmark_data_smithi159_162467_object17763:head# 2024-04-15T18:42:58.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f33bbf49:::benchmark_data_smithi159_162467_object68816:head# 2024-04-15T18:42:58.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f33de772:::benchmark_data_smithi159_162467_object5333:head# 2024-04-15T18:42:58.037 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3416cd2:::benchmark_data_smithi159_162467_object35995:head# 2024-04-15T18:42:58.037 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3438325:::benchmark_data_smithi159_162467_object44273:head# 2024-04-15T18:42:58.038 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3443a64:::benchmark_data_smithi159_162467_object65917:head# 2024-04-15T18:42:58.038 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3444a4e:::benchmark_data_smithi159_162467_object71106:head# 2024-04-15T18:42:58.039 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f344b1d2:::benchmark_data_smithi159_162467_object5781:head# 2024-04-15T18:42:58.039 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f344ef50:::benchmark_data_smithi159_162467_object23056:head# 2024-04-15T18:42:58.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3459793:::benchmark_data_smithi159_162467_object5131:head# 2024-04-15T18:42:58.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f345f8d2:::benchmark_data_smithi159_162467_object22872:head# 2024-04-15T18:42:58.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3464d70:::benchmark_data_smithi159_162467_object5133:head# 2024-04-15T18:42:58.043 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34883bf:::benchmark_data_smithi159_162467_object74115:head# 2024-04-15T18:42:58.045 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34a9742:::benchmark_data_smithi159_162467_object58592:head# 2024-04-15T18:42:58.046 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34acd5a:::benchmark_data_smithi159_162467_object66579:head# 2024-04-15T18:42:58.046 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34ad09b:::benchmark_data_smithi159_162467_object15770:head# 2024-04-15T18:42:58.047 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34b37fa:::benchmark_data_smithi159_162467_object74356:head# 2024-04-15T18:42:58.047 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34c0553:::benchmark_data_smithi159_162467_object69019:head# 2024-04-15T18:42:58.047 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34d2b77:::benchmark_data_smithi159_162467_object54954:head# 2024-04-15T18:42:58.048 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34d2bf3:::benchmark_data_smithi159_162467_object27018:head# 2024-04-15T18:42:58.048 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f34e1d88:::benchmark_data_smithi159_162467_object44121:head# 2024-04-15T18:42:58.049 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f350192d:::benchmark_data_smithi159_162467_object27908:head# 2024-04-15T18:42:58.049 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f350271e:::benchmark_data_smithi159_162467_object55409:head# 2024-04-15T18:42:58.049 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3509665:::benchmark_data_smithi159_162467_object59477:head# 2024-04-15T18:42:58.050 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f350aa05:::benchmark_data_smithi159_162467_object19695:head# 2024-04-15T18:42:58.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f350ccf4:::benchmark_data_smithi159_162467_object32201:head# 2024-04-15T18:42:58.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f351f3cc:::benchmark_data_smithi159_162467_object68892:head# 2024-04-15T18:42:58.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3534400:::benchmark_data_smithi159_162467_object11697:head# 2024-04-15T18:42:58.052 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f354eddf:::benchmark_data_smithi159_162467_object30064:head# 2024-04-15T18:42:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f355092a:::benchmark_data_smithi159_162467_object12952:head# 2024-04-15T18:42:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f355238d:::benchmark_data_smithi159_162467_object72790:head# 2024-04-15T18:42:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f355d5d6:::benchmark_data_smithi159_162467_object72980:head# 2024-04-15T18:42:58.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f357a72e:::benchmark_data_smithi159_162467_object6612:head# 2024-04-15T18:42:58.055 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f357bc06:::benchmark_data_smithi159_162467_object35038:head# 2024-04-15T18:42:58.055 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3586556:::benchmark_data_smithi159_162467_object36700:head# 2024-04-15T18:42:58.056 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f359bdab:::benchmark_data_smithi159_162467_object46117:head# 2024-04-15T18:42:58.056 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f359e055:::benchmark_data_smithi159_162467_object56964:head# 2024-04-15T18:42:58.056 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35a32a1:::benchmark_data_smithi159_162467_object42909:head# 2024-04-15T18:42:58.057 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35a4b18:::benchmark_data_smithi159_162467_object44312:head# 2024-04-15T18:42:58.057 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35abfb8:::benchmark_data_smithi159_162467_object34485:head# 2024-04-15T18:42:58.058 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35b0c6f:::benchmark_data_smithi159_162467_object22286:head# 2024-04-15T18:42:58.058 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35b7637:::benchmark_data_smithi159_162467_object64780:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35c49e2:::benchmark_data_smithi159_162467_object40977:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35c8336:::benchmark_data_smithi159_162467_object11334:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35d29ed:::benchmark_data_smithi159_162467_object1700:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35f03b2:::benchmark_data_smithi159_162467_object18536:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f35fade8:::benchmark_data_smithi159_162467_object30955:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f36096d9:::benchmark_data_smithi159_162467_object10201:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f361146c:::benchmark_data_smithi159_162467_object41940:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3634acf:::benchmark_data_smithi159_162467_object47013:head# 2024-04-15T18:42:58.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3644429:::benchmark_data_smithi159_162467_object8686:head# 2024-04-15T18:42:58.063 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f364e708:::benchmark_data_smithi159_162467_object25364:head# 2024-04-15T18:42:58.063 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f365eb54:::benchmark_data_smithi159_162467_object9750:head# 2024-04-15T18:42:58.063 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f36663e4:::benchmark_data_smithi159_162467_object17158:head# 2024-04-15T18:42:58.064 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3668ddf:::benchmark_data_smithi159_162467_object62298:head# 2024-04-15T18:42:58.064 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f367abd9:::benchmark_data_smithi159_162467_object20265:head# 2024-04-15T18:42:58.065 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f367c98b:::benchmark_data_smithi159_162467_object60878:head# 2024-04-15T18:42:58.065 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f36a1339:::benchmark_data_smithi159_162467_object24267:head# 2024-04-15T18:42:58.066 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f36aff9a:::benchmark_data_smithi159_162467_object57429:head# 2024-04-15T18:42:58.066 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f36fca99:::benchmark_data_smithi159_162467_object21241:head# 2024-04-15T18:42:58.066 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f370c19e:::benchmark_data_smithi159_162467_object71744:head# 2024-04-15T18:42:58.067 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37478c0:::benchmark_data_smithi159_162467_object68794:head# 2024-04-15T18:42:58.067 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f375cdc9:::benchmark_data_smithi159_162467_object30888:head# 2024-04-15T18:42:58.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f375ce7b:::benchmark_data_smithi159_162467_object68291:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f376005c:::benchmark_data_smithi159_162467_object44363:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3760810:::benchmark_data_smithi159_162467_object68802:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3761d17:::benchmark_data_smithi159_162467_object40955:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37633c3:::benchmark_data_smithi159_162467_object40992:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f376b723:::benchmark_data_smithi159_162467_object26076:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3777091:::benchmark_data_smithi159_162467_object3013:head# 2024-04-15T18:42:58.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f377821a:::benchmark_data_smithi159_162467_object10623:head# 2024-04-15T18:42:58.072 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f377add1:::benchmark_data_smithi159_162467_object48813:head# 2024-04-15T18:42:58.072 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f377e7ea:::benchmark_data_smithi159_162467_object74636:head# 2024-04-15T18:42:58.072 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3786816:::benchmark_data_smithi159_162467_object20300:head# 2024-04-15T18:42:58.073 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3789350:::benchmark_data_smithi159_162467_object54157:head# 2024-04-15T18:42:58.073 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37a0075:::benchmark_data_smithi159_162467_object43432:head# 2024-04-15T18:42:58.074 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37ab5e5:::benchmark_data_smithi159_162467_object10616:head# 2024-04-15T18:42:58.074 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37c4139:::benchmark_data_smithi159_162467_object62744:head# 2024-04-15T18:42:58.074 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37cd0b5:::benchmark_data_smithi159_162467_object35742:head# 2024-04-15T18:42:58.075 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37d623d:::benchmark_data_smithi159_162467_object12942:head# 2024-04-15T18:42:58.075 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37d7c63:::benchmark_data_smithi159_162467_object14624:head# 2024-04-15T18:42:58.076 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f37fc91a:::benchmark_data_smithi159_162467_object41184:head# 2024-04-15T18:42:58.076 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f380b1ac:::benchmark_data_smithi159_162467_object62850:head# 2024-04-15T18:42:58.076 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f380b413:::benchmark_data_smithi159_162467_object8591:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3810a90:::benchmark_data_smithi159_162467_object2725:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3829918:::benchmark_data_smithi159_162467_object20619:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38359fe:::benchmark_data_smithi159_162467_object64741:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f383996b:::benchmark_data_smithi159_162467_object52795:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f383e9ae:::benchmark_data_smithi159_162467_object27421:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3841e0b:::benchmark_data_smithi159_162467_object9659:head# 2024-04-15T18:42:58.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f384cd56:::benchmark_data_smithi159_162467_object35702:head# 2024-04-15T18:42:58.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f384f7c7:::benchmark_data_smithi159_162467_object53424:head# 2024-04-15T18:42:58.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3851b45:::benchmark_data_smithi159_162467_object67911:head# 2024-04-15T18:42:58.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f385a2d4:::benchmark_data_smithi159_162467_object19851:head# 2024-04-15T18:42:58.082 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f385db62:::benchmark_data_smithi159_162467_object60059:head# 2024-04-15T18:42:58.083 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f386978e:::benchmark_data_smithi159_162467_object60727:head# 2024-04-15T18:42:58.083 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f386cfd2:::benchmark_data_smithi159_162467_object29507:head# 2024-04-15T18:42:58.084 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38723e3:::benchmark_data_smithi159_162467_object31017:head# 2024-04-15T18:42:58.084 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38791ac:::benchmark_data_smithi159_162467_object53965:head# 2024-04-15T18:42:58.084 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3890ff9:::benchmark_data_smithi159_162467_object74398:head# 2024-04-15T18:42:58.085 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3891003:::benchmark_data_smithi159_162467_object55505:head# 2024-04-15T18:42:58.085 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f389a351:::benchmark_data_smithi159_162467_object57474:head# 2024-04-15T18:42:58.086 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38b04fc:::benchmark_data_smithi159_162467_object12310:head# 2024-04-15T18:42:58.086 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38b19af:::benchmark_data_smithi159_162467_object48825:head# 2024-04-15T18:42:58.089 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38e2cac:::benchmark_data_smithi159_162467_object61961:head# 2024-04-15T18:42:58.089 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38e5837:::benchmark_data_smithi159_162467_object32453:head# 2024-04-15T18:42:58.090 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f38f574d:::benchmark_data_smithi159_162467_object30247:head# 2024-04-15T18:42:58.090 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f390ce09:::benchmark_data_smithi159_162467_object34369:head# 2024-04-15T18:42:58.091 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f390ce4a:::benchmark_data_smithi159_162467_object29408:head# 2024-04-15T18:42:58.091 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3925eaa:::benchmark_data_smithi159_162467_object46430:head# 2024-04-15T18:42:58.091 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39263de:::benchmark_data_smithi159_162467_object66688:head# 2024-04-15T18:42:58.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f392cf3f:::benchmark_data_smithi159_162467_object28671:head# 2024-04-15T18:42:58.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f393619b:::benchmark_data_smithi159_162467_object46952:head# 2024-04-15T18:42:58.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f393a07d:::benchmark_data_smithi159_162467_object4173:head# 2024-04-15T18:42:58.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3941a14:::benchmark_data_smithi159_162467_object26295:head# 2024-04-15T18:42:58.094 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3944c16:::benchmark_data_smithi159_162467_object43413:head# 2024-04-15T18:42:58.094 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3951e5d:::benchmark_data_smithi159_162467_object34859:head# 2024-04-15T18:42:58.094 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39529cd:::benchmark_data_smithi159_162467_object59205:head# 2024-04-15T18:42:58.095 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39558f0:::benchmark_data_smithi159_162467_object69972:head# 2024-04-15T18:42:58.095 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f395d18b:::benchmark_data_smithi159_162467_object31717:head# 2024-04-15T18:42:58.096 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3971031:::benchmark_data_smithi159_162467_object2498:head# 2024-04-15T18:42:58.098 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f397cebe:::benchmark_data_smithi159_162467_object44504:head# 2024-04-15T18:42:58.099 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39826b8:::benchmark_data_smithi159_162467_object45651:head# 2024-04-15T18:42:58.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39a14ae:::benchmark_data_smithi159_162467_object30159:head# 2024-04-15T18:42:58.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39a5482:::benchmark_data_smithi159_162467_object37800:head# 2024-04-15T18:42:58.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39b48a4:::benchmark_data_smithi159_162467_object29796:head# 2024-04-15T18:42:58.102 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39be05e:::benchmark_data_smithi159_162467_object48846:head# 2024-04-15T18:42:58.102 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39be845:::benchmark_data_smithi159_162467_object3792:head# 2024-04-15T18:42:58.103 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39bea67:::benchmark_data_smithi159_162467_object59065:head# 2024-04-15T18:42:58.103 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39cb3c9:::benchmark_data_smithi159_162467_object63440:head# 2024-04-15T18:42:58.104 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:42:58.104 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39e1d43:::benchmark_data_smithi159_162467_object66894:head# 2024-04-15T18:42:58.104 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39f104c:::benchmark_data_smithi159_162467_object72321:head# 2024-04-15T18:42:58.105 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f39fae05:::benchmark_data_smithi159_162467_object25152:head# 2024-04-15T18:42:58.105 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a03815:::benchmark_data_smithi159_162467_object52945:head# 2024-04-15T18:42:58.105 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a0775e:::benchmark_data_smithi159_162467_object7675:head# 2024-04-15T18:42:58.106 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a353d2:::benchmark_data_smithi159_162467_object66582:head# 2024-04-15T18:42:58.106 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a3ab33:::benchmark_data_smithi159_162467_object20760:head# 2024-04-15T18:42:58.106 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a58dbc:::benchmark_data_smithi159_162467_object25982:head# 2024-04-15T18:42:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:58.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:42:58.108 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a77f40:::benchmark_data_smithi159_162467_object38510:head# 2024-04-15T18:42:58.108 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a7f59d:::benchmark_data_smithi159_162467_object21119:head# 2024-04-15T18:42:58.108 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a89593:::benchmark_data_smithi159_162467_object4457:head# 2024-04-15T18:42:58.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a97dda:::benchmark_data_smithi159_162467_object22510:head# 2024-04-15T18:42:58.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3a9e0d1:::benchmark_data_smithi159_162467_object51951:head# 2024-04-15T18:42:58.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3aadcc3:::benchmark_data_smithi159_162467_object52242:head# 2024-04-15T18:42:58.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3aaee5b:::benchmark_data_smithi159_162467_object54977:head# 2024-04-15T18:42:58.110 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ab0e1f:::benchmark_data_smithi159_162467_object48623:head# 2024-04-15T18:42:58.110 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ac19fb:::benchmark_data_smithi159_162467_object17373:head# 2024-04-15T18:42:58.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ac32b5:::benchmark_data_smithi159_162467_object23021:head# 2024-04-15T18:42:58.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ad9965:::benchmark_data_smithi159_162467_object53463:head# 2024-04-15T18:42:58.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3adbab2:::benchmark_data_smithi159_162467_object69315:head# 2024-04-15T18:42:58.112 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ae77e8:::benchmark_data_smithi159_162467_object9578:head# 2024-04-15T18:42:58.112 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3aea2b8:::benchmark_data_smithi159_162467_object39003:head# 2024-04-15T18:42:58.113 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3af8d02:::benchmark_data_smithi159_162467_object49127:head# 2024-04-15T18:42:58.113 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3afa726:::benchmark_data_smithi159_162467_object54530:head# 2024-04-15T18:42:58.114 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b03fe2:::benchmark_data_smithi159_162467_object72033:head# 2024-04-15T18:42:58.114 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b06b75:::benchmark_data_smithi159_162467_object31136:head# 2024-04-15T18:42:58.114 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b070e8:::benchmark_data_smithi159_162467_object59366:head# 2024-04-15T18:42:58.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b165f3:::benchmark_data_smithi159_162467_object660:head# 2024-04-15T18:42:58.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b18168:::benchmark_data_smithi159_162467_object39433:head# 2024-04-15T18:42:58.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b259e2:::benchmark_data_smithi159_162467_object68334:head# 2024-04-15T18:42:58.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b36f15:::benchmark_data_smithi159_162467_object29929:head# 2024-04-15T18:42:58.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b3b37a:::benchmark_data_smithi159_162467_object28680:head# 2024-04-15T18:42:58.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b3cec1:::benchmark_data_smithi159_162467_object53265:head# 2024-04-15T18:42:58.117 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b40e47:::benchmark_data_smithi159_162467_object50426:head# 2024-04-15T18:42:58.117 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b42f19:::benchmark_data_smithi159_162467_object31653:head# 2024-04-15T18:42:58.118 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b48086:::benchmark_data_smithi159_162467_object74078:head# 2024-04-15T18:42:58.119 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b562f9:::benchmark_data_smithi159_162467_object11901:head# 2024-04-15T18:42:58.119 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b5b3e0:::benchmark_data_smithi159_162467_object63075:head# 2024-04-15T18:42:58.119 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b5ed84:::benchmark_data_smithi159_162467_object72517:head# 2024-04-15T18:42:58.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b64b00:::benchmark_data_smithi159_162467_object27952:head# 2024-04-15T18:42:58.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b6ba05:::benchmark_data_smithi159_162467_object7518:head# 2024-04-15T18:42:58.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b72ac1:::benchmark_data_smithi159_162467_object60685:head# 2024-04-15T18:42:58.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3b7d4b4:::benchmark_data_smithi159_162467_object65288:head# 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:57366 6.e 6:7868e193:::benchmark_data_smithi159_139342_object57365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e562)", 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 561.56256606299996, 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.461560207, 2024-04-15T18:42:58.122 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 57366 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566090+0000", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566092+0000", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5719999999999999e-06 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566149+0000", 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7744999999999997e-05 2024-04-15T18:42:58.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566162+0000", 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2685e-05 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566170+0000", 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0239999999999997e-06 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566220+0000", 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9595000000000002e-05 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op must wait for map", 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.566225+0000", 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567183+0000", 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095810800000000005 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567310+0000", 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012681300000000001 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,2", 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.567438+0000", 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000127935 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.569385+0000", 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001947465 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:36.570421+0000", 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010361540000000001 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027576+0000", 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45715493899999998 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027618+0000", 2024-04-15T18:42:58.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1644999999999999e-05 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:33:37.027650+0000", 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2159000000000002e-05 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:158265 6.e 6:7c3d7611:::benchmark_data_smithi159_139342_object68386:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:42:58.130 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 430.68044811099998, 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.46395664199999997, 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 158265 2024-04-15T18:42:58.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448208+0000", 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448210+0000", 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.12e-06 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448221+0000", 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0628999999999999e-05 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448248+0000", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7606999999999998e-05 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448263+0000", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4593000000000001e-05 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448391+0000", 2024-04-15T18:42:58.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012803099999999999 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448516+0000", 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125313 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,2", 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.448581+0000", 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5152000000000002e-05 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.452334+0000", 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037521999999999998 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.453884+0000", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015505219999999999 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912067+0000", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.45818335399999999 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912117+0000", 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9982999999999999e-05 2024-04-15T18:42:58.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:47.912165+0000", 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7138000000000002e-05 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:177229 6.6 6:6142b3eb:::benchmark_data_smithi159_139342_object87350:head [delete] snapc 0=[] ondisk+write+known_if_redirected e609)", 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 400.11637149500001, 2024-04-15T18:42:58.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29316968399999999, 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 177229 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:42:58.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012285+0000", 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012286+0000", 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.172e-06 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012305+0000", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8970999999999998e-05 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012317+0000", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.185e-05 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012325+0000", 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3860000000000007e-06 2024-04-15T18:42:58.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012390+0000", 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4948000000000002e-05 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012534+0000", 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014438100000000001 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.012665+0000", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000130576 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.017780+0000", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051149960000000001 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.028518+0000", 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010737714000000001 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305326+0000", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.276808479 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305380+0000", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3721999999999998e-05 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.305454+0000", 2024-04-15T18:42:58.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4488999999999998e-05 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:179751 6.6 6:68808706:::benchmark_data_smithi159_139342_object89872:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 396.75166179399997, 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25397364900000002, 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.145 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179751 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376994+0000", 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376995+0000", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2200000000000003e-07 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.376997+0000", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.497e-06 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377007+0000", 2024-04-15T18:42:58.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6509999999999996e-06 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377013+0000", 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6879999999999996e-06 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377058+0000", 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5090999999999999e-05 2024-04-15T18:42:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377180+0000", 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012175700000000001 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.377300+0000", 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012048999999999999 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.382334+0000", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.005033694 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.382479+0000", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014488700000000001 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630903+0000", 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24842419199999999 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630939+0000", 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6470000000000001e-05 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.630968+0000", 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.851e-05 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31782 7.1 7:82fc4c0d:::benchmark_data_smithi159_149998_object31781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 347.09227313700001, 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31042012400000002, 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31782 2024-04-15T18:42:58.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.153 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0600000000000001e-07 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036412+0000", 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8798000000000001e-05 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036418+0000", 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7520000000000002e-06 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036421+0000", 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1470000000000001e-06 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037561+0000", 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011398739999999999 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037642+0000", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1481000000000001e-05 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037728+0000", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5646e-05 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038470+0000", 2024-04-15T18:42:58.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000741729 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038630+0000", 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016056800000000001 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346734+0000", 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30810352899999999 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346773+0000", 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8949999999999998e-05 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346803+0000", 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0343999999999999e-05 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.158 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31793 7.4 7:2fbec3a3:::benchmark_data_smithi159_149998_object31792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 347.08081946099998, 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31189694899999998, 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31793 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.159 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047839+0000", 2024-04-15T18:42:58.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0530000000000001e-06 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047884+0000", 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5325999999999999e-05 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047911+0000", 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7457000000000002e-05 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047932+0000", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0888999999999999e-05 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048077+0000", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144361 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048189+0000", 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112435 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048271+0000", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1411999999999999e-05 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048888+0000", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000617872 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.049960+0000", 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001071935 2024-04-15T18:42:58.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359672+0000", 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30971173899999999 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359707+0000", 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4965000000000002e-05 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359734+0000", 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6505000000000001e-05 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31795 7.1 7:81918b20:::benchmark_data_smithi159_149998_object31794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 347.07738588799998, 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32741436800000001, 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.165 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31795 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051271+0000", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1900000000000001e-07 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051305+0000", 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3918000000000003e-05 2024-04-15T18:42:58.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051315+0000", 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0376e-05 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051323+0000", 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8329999999999994e-06 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051397+0000", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4245999999999994e-05 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051487+0000", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0140000000000001e-05 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051573+0000", 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5922999999999997e-05 2024-04-15T18:42:58.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052330+0000", 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075673500000000002 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.053094+0000", 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076432699999999998 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378621+0000", 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32552697800000002 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378663+0000", 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1357999999999999e-05 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378685+0000", 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1715000000000001e-05 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:31782 7.1 e631/621 ondisk, result = 0)", 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 347.07566076000001, 2024-04-15T18:42:58.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29381337200000002, 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052996+0000", 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.41e-07 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052997+0000", 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.265e-06 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346692+0000", 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29369515000000002 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346695+0000", 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3160000000000001e-06 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346724+0000", 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8614000000000001e-05 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346730+0000", 2024-04-15T18:42:58.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346809+0000", 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8584999999999994e-05 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55371 7.1 7:830c46a9:::benchmark_data_smithi159_149998_object55370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:42:58.177 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.977948884, 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25659491200000001, 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55371 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150708+0000", 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1909999999999999e-06 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150765+0000", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7090999999999999e-05 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150775+0000", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0016e-05 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150781+0000", 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5960000000000003e-06 2024-04-15T18:42:58.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150813+0000", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2066000000000001e-05 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150906+0000", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3035999999999995e-05 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150998+0000", 2024-04-15T18:42:58.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699000000000004e-05 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152490+0000", 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014920739999999999 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152917+0000", 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042668799999999999 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407183+0000", 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25426654300000001 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407248+0000", 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4925999999999995e-05 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407302+0000", 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3986000000000001e-05 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55375 7.1 7:8b006e6f:::benchmark_data_smithi159_149998_object55374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.97500787299998, 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28137763399999999, 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.184 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55375 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153649+0000", 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8899999999999998e-07 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153727+0000", 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8459000000000002e-05 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153742+0000", 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4202000000000001e-05 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153748+0000", 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3269999999999997e-06 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153791+0000", 2024-04-15T18:42:58.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3341999999999999e-05 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153937+0000", 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014536700000000001 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.154086+0000", 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149674 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.155698+0000", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016121659999999999 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.156469+0000", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00077094200000000005 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434910+0000", 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27844058500000002 2024-04-15T18:42:58.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434974+0000", 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3758999999999997e-05 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435026+0000", 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2021999999999998e-05 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.190 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55403 7.1 7:8aca0ab0:::benchmark_data_smithi159_149998_object55402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.952183198, 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27940837899999998, 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55403 2024-04-15T18:42:58.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176474+0000", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3319999999999999e-06 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176505+0000", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0821000000000002e-05 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176519+0000", 2024-04-15T18:42:58.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3957999999999999e-05 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176527+0000", 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.79e-06 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176587+0000", 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9973000000000003e-05 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176743+0000", 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000155814 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176908+0000", 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165198 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.178583+0000", 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001675073 2024-04-15T18:42:58.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179015+0000", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043260999999999999 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455781+0000", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27676578499999999 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455836+0000", 2024-04-15T18:42:58.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4837999999999999e-05 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455881+0000", 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5187e-05 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55375 7.1 e639/621 ondisk, result = 0)", 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.949319785, 2024-04-15T18:42:58.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25569712700000002, 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:42:58.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179337+0000", 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8100000000000002e-07 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179341+0000", 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2080000000000002e-06 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434841+0000", 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25549990900000003 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434848+0000", 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5019999999999998e-06 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434894+0000", 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6836999999999999e-05 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434904+0000", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3519999999999996e-06 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435033+0000", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012963800000000001 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55417 7.4 7:2aa6faee:::benchmark_data_smithi159_149998_object55416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:42:58.201 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.93832110900001, 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26001807999999998, 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55417 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190336+0000", 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3629999999999999e-06 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190388+0000", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1354000000000003e-05 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190402+0000", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4353e-05 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190409+0000", 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9110000000000004e-06 2024-04-15T18:42:58.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190462+0000", 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3458999999999998e-05 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190560+0000", 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7677999999999996e-05 2024-04-15T18:42:58.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190645+0000", 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4790000000000006e-05 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192252+0000", 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016068009999999999 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192667+0000", 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000415189 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450236+0000", 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25756942999999999 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450301+0000", 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5079000000000005e-05 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450353+0000", 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1672999999999998e-05 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55420 7.4 7:2ab922f1:::benchmark_data_smithi159_149998_object55419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:42:58.208 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.93393500399998, 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27355053699999998, 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55420 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.209 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194722+0000", 2024-04-15T18:42:58.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0699999999999996e-07 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194774+0000", 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2434e-05 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194785+0000", 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0767e-05 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194793+0000", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6389999999999992e-06 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194835+0000", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2561000000000001e-05 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194931+0000", 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5377999999999994e-05 2024-04-15T18:42:58.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.195017+0000", 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6224000000000007e-05 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196339+0000", 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013224020000000001 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196756+0000", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00041645199999999999 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468144+0000", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27138785100000001 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468225+0000", 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1422999999999996e-05 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468272+0000", 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6499000000000003e-05 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55417 7.4 e639/621 ondisk, result = 0)", 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 326.93182516299998, 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25352939099999999, 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.215 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196831+0000", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8000000000000001e-07 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196834+0000", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3640000000000002e-06 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450171+0000", 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25333760599999999 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450176+0000", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0150000000000003e-06 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450220+0000", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3334000000000002e-05 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450230+0000", 2024-04-15T18:42:58.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0484e-05 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450360+0000", 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013020800000000001 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.219 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:124560 7.4 7:2a9d76e5:::benchmark_data_smithi159_149998_object28560:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 237.96671966400001, 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27331726099999998, 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124560 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161937+0000", 2024-04-15T18:42:58.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5700000000000001e-07 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161941+0000", 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0520000000000003e-06 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161950+0000", 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8769999999999999e-06 2024-04-15T18:42:58.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161958+0000", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3780000000000003e-06 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161994+0000", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6823999999999998e-05 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162070+0000", 2024-04-15T18:42:58.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5834999999999995e-05 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162135+0000", 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4783000000000006e-05 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.165059+0000", 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029242880000000001 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.168016+0000", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029565440000000002 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435166+0000", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26715059699999999 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435204+0000", 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7344e-05 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435254+0000", 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9882000000000001e-05 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:143366 7.4 7:21b697b5:::benchmark_data_smithi159_149998_object47366:head [delete] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 211.222673826, 2024-04-15T18:42:58.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53450922700000003, 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 143366 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905984+0000", 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9429999999999999e-06 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905995+0000", 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0349999999999999e-05 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906022+0000", 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7248999999999999e-05 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906037+0000", 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5150000000000001e-05 2024-04-15T18:42:58.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906143+0000", 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106461 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906269+0000", 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012599200000000001 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:42:58.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906398+0000", 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012908 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.909755+0000", 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033568209999999999 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.910370+0000", 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061510700000000005 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440411+0000", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53004094499999999 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440446+0000", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4193000000000003e-05 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440491+0000", 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5936000000000002e-05 2024-04-15T18:42:58.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDScrubReserve(7.3,RELEASE,e:676,from:2,reservation_nonce:0)", 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 206.34099922799999, 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30058055299999997, 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8300000000000001e-07 2024-04-15T18:42:58.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787659+0000", 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.886e-06 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088016+0000", 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30035709100000002 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088030+0000", 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4496e-05 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088171+0000", 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140505 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088209+0000", 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8426000000000001e-05 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088237+0000", 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7966e-05 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:191990 7.1 7:816ba94e:::benchmark_data_smithi159_149998_object95990:head [delete] snapc 0=[] ondisk+write+known_if_redirected e700)", 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 142.22614435899999, 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.272141566, 2024-04-15T18:42:58.237 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 191990 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902514+0000", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9269999999999999e-06 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902520+0000", 2024-04-15T18:42:58.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1419999999999998e-06 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902535+0000", 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5251e-05 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902543+0000", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8790000000000003e-06 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902601+0000", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8553e-05 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902725+0000", 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012331500000000001 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902841+0000", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011651500000000001 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907458+0000", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046170109999999999 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907670+0000", 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212017 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174581+0000", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26691115799999998 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174620+0000", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8198000000000003e-05 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174653+0000", 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3599999999999997e-05 2024-04-15T18:42:58.243 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:191990 7.1 e700/621 ondisk, result = 0)", 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 142.221406551, 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26740929600000002, 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:42:58.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907250+0000", 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3900000000000004e-07 2024-04-15T18:42:58.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907256+0000", 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5400000000000003e-06 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174508+0000", 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26725248200000001 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:42:58.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174514+0000", 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7400000000000001e-06 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174569+0000", 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5276999999999999e-05 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174577+0000", 2024-04-15T18:42:58.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8979999999999996e-06 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174659+0000", 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1520000000000006e-05 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:42:58.248 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:42:58.248 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:58.250 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:42:58.250 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:42:58.250 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 28365/299919 objects degraded (9.458%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3baaaa5:::benchmark_data_smithi159_162467_object54589:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bc382e:::benchmark_data_smithi159_162467_object8930:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bcd6d4:::benchmark_data_smithi159_162467_object62228:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bd6497:::benchmark_data_smithi159_162467_object69486:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bdb0c5:::benchmark_data_smithi159_162467_object74085:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bdf458:::benchmark_data_smithi159_162467_object53063:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3be0e68:::benchmark_data_smithi159_162467_object22575:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bec68f:::benchmark_data_smithi159_162467_object18195:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bf2e0f:::benchmark_data_smithi159_162467_object32456:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bf75b6:::benchmark_data_smithi159_162467_object34499:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bfb6a0:::benchmark_data_smithi159_162467_object48455:head# 2024-04-15T18:42:58.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3bfdda6:::benchmark_data_smithi159_162467_object59459:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c0acc1:::benchmark_data_smithi159_162467_object39508:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c1e7f1:::benchmark_data_smithi159_162467_object51304:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c24fcb:::benchmark_data_smithi159_162467_object49878:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c3c991:::benchmark_data_smithi159_162467_object38965:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c4c749:::benchmark_data_smithi159_162467_object8090:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c54af0:::benchmark_data_smithi159_162467_object35094:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c55ed3:::benchmark_data_smithi159_162467_object4:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c609ab:::benchmark_data_smithi159_162467_object1180:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c69164:::benchmark_data_smithi159_162467_object63022:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c71e47:::benchmark_data_smithi159_162467_object33256:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c7e452:::benchmark_data_smithi159_162467_object70933:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c80a95:::benchmark_data_smithi159_162467_object8172:head# 2024-04-15T18:42:58.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c855c6:::benchmark_data_smithi159_162467_object16385:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c8d1c3:::benchmark_data_smithi159_162467_object649:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c973cf:::benchmark_data_smithi159_162467_object21194:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c99bc8:::benchmark_data_smithi159_162467_object16815:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3c9b090:::benchmark_data_smithi159_162467_object3875:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ca19d8:::benchmark_data_smithi159_162467_object72419:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ca6382:::benchmark_data_smithi159_162467_object49439:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3cbdd52:::benchmark_data_smithi159_162467_object8220:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3cc2136:::benchmark_data_smithi159_162467_object74453:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3cc40fe:::benchmark_data_smithi159_162467_object22565:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3cdc193:::benchmark_data_smithi159_162467_object25756:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ce9a3c:::benchmark_data_smithi159_162467_object36514:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ce9b80:::benchmark_data_smithi159_162467_object44500:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3cef398:::benchmark_data_smithi159_162467_object58414:head# 2024-04-15T18:42:58.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3d2c0eb:::benchmark_data_smithi159_162467_object26201:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3d64244:::benchmark_data_smithi159_162467_object65366:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3d7e245:::benchmark_data_smithi159_162467_object21390:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3d9a5a7:::benchmark_data_smithi159_162467_object59009:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3da3bff:::benchmark_data_smithi159_162467_object3910:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3db302e:::benchmark_data_smithi159_162467_object69210:head# 2024-04-15T18:42:58.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3db9dbe:::benchmark_data_smithi159_162467_object37116:head# 2024-04-15T18:42:58.258 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:42:58.258 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:42:58.258 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 28365/299919 objects degraded (9.458%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3dc62f2:::benchmark_data_smithi159_162467_object65932:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3dde39d:::benchmark_data_smithi159_162467_object23907:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ddf2f6:::benchmark_data_smithi159_162467_object33804:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3de1f71:::benchmark_data_smithi159_162467_object31513:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3dfc86d:::benchmark_data_smithi159_162467_object21009:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e0a25a:::benchmark_data_smithi159_162467_object24283:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e10cbb:::benchmark_data_smithi159_162467_object46807:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e1387f:::benchmark_data_smithi159_162467_object70395:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e1b8c9:::benchmark_data_smithi159_162467_object23792:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e27d33:::benchmark_data_smithi159_162467_object73684:head# 2024-04-15T18:42:58.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e3ba7d:::benchmark_data_smithi159_162467_object4101:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e50a3e:::benchmark_data_smithi159_162467_object69456:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e565d3:::benchmark_data_smithi159_162467_object42108:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e8a5f4:::benchmark_data_smithi159_162467_object46915:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e95f89:::benchmark_data_smithi159_162467_object3901:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3e987a1:::benchmark_data_smithi159_162467_object54805:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3eaa719:::benchmark_data_smithi159_162467_object61052:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3eaec3a:::benchmark_data_smithi159_162467_object33297:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3eb1e14:::benchmark_data_smithi159_162467_object62568:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f00d9d:::benchmark_data_smithi159_162467_object45610:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f0a95f:::benchmark_data_smithi159_162467_object15616:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f1c24a:::benchmark_data_smithi159_162467_object2770:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f206d6:::benchmark_data_smithi159_162467_object43562:head# 2024-04-15T18:42:58.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f3bed8:::benchmark_data_smithi159_162467_object23930:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f3c10d:::benchmark_data_smithi159_162467_object72920:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f50c7a:::benchmark_data_smithi159_162467_object35706:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f51a10:::benchmark_data_smithi159_162467_object41111:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f51d10:::benchmark_data_smithi159_162467_object9818:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f6d9f3:::benchmark_data_smithi159_162467_object58182:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f8293a:::benchmark_data_smithi159_162467_object56185:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f88cb9:::benchmark_data_smithi159_162467_object72205:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f90b4a:::benchmark_data_smithi159_162467_object24443:head# 2024-04-15T18:42:58.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f93a04:::benchmark_data_smithi159_162467_object43537:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f9889b:::benchmark_data_smithi159_162467_object42574:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3f9e194:::benchmark_data_smithi159_162467_object54964:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3fb7b18:::benchmark_data_smithi159_162467_object63481:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3fc149e:::benchmark_data_smithi159_162467_object10332:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3fcd53d:::benchmark_data_smithi159_162467_object49665:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3fd6dda:::benchmark_data_smithi159_162467_object67284:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3fd8748:::benchmark_data_smithi159_162467_object16160:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f3ffc47a:::benchmark_data_smithi159_162467_object73137:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f400868d:::benchmark_data_smithi159_162467_object64927:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f400b006:::benchmark_data_smithi159_162467_object65871:head# 2024-04-15T18:42:58.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f400e4b0:::benchmark_data_smithi159_162467_object66363:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4010aa1:::benchmark_data_smithi159_162467_object3802:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4023bb3:::benchmark_data_smithi159_162467_object57915:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f402c623:::benchmark_data_smithi159_162467_object5088:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f403587f:::benchmark_data_smithi159_162467_object40135:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f404807b:::benchmark_data_smithi159_162467_object61828:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f405417b:::benchmark_data_smithi159_162467_object33464:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4074c55:::benchmark_data_smithi159_162467_object68135:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4074f92:::benchmark_data_smithi159_162467_object48224:head# 2024-04-15T18:42:58.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f407b8e6:::benchmark_data_smithi159_162467_object68567:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f408d6b0:::benchmark_data_smithi159_162467_object13555:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4093b6b:::benchmark_data_smithi159_162467_object21640:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f409df71:::benchmark_data_smithi159_162467_object45866:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40a30ad:::benchmark_data_smithi159_162467_object21127:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40ba16c:::benchmark_data_smithi159_162467_object72694:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40d797d:::benchmark_data_smithi159_162467_object39425:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40e54a2:::benchmark_data_smithi159_162467_object60126:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40eabd4:::benchmark_data_smithi159_162467_object20316:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40f0ae6:::benchmark_data_smithi159_162467_object19880:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f40fb71a:::benchmark_data_smithi159_162467_object35925:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4104e45:::benchmark_data_smithi159_162467_object7217:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4105983:::benchmark_data_smithi159_162467_object7780:head# 2024-04-15T18:42:58.287 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4106fb7:::benchmark_data_smithi159_162467_object35488:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4118419:::benchmark_data_smithi159_162467_object6485:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41355e8:::benchmark_data_smithi159_162467_object72473:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f413de9a:::benchmark_data_smithi159_162467_object69255:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4142f8f:::benchmark_data_smithi159_162467_object48953:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4143506:::benchmark_data_smithi159_162467_object32043:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f414fc99:::benchmark_data_smithi159_162467_object46189:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f415ba49:::benchmark_data_smithi159_162467_object1113:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4165eb8:::benchmark_data_smithi159_162467_object61593:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41747e3:::benchmark_data_smithi159_162467_object9258:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4199819:::benchmark_data_smithi159_162467_object24724:head# 2024-04-15T18:42:58.288 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41a2f2c:::benchmark_data_smithi159_162467_object43126:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41aa1ac:::benchmark_data_smithi159_162467_object68166:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41b1e16:::benchmark_data_smithi159_162467_object15771:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41b2d38:::benchmark_data_smithi159_162467_object28155:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41bcc3a:::benchmark_data_smithi159_162467_object65759:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41c6260:::benchmark_data_smithi159_162467_object35370:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41eabda:::benchmark_data_smithi159_162467_object36114:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41ef5f8:::benchmark_data_smithi159_162467_object842:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41f0fde:::benchmark_data_smithi159_162467_object44346:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41f6b5f:::benchmark_data_smithi159_162467_object66677:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41f82bd:::benchmark_data_smithi159_162467_object4143:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41fc88d:::benchmark_data_smithi159_162467_object28822:head# 2024-04-15T18:42:58.289 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f41fcd21:::benchmark_data_smithi159_162467_object50369:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4208395:::benchmark_data_smithi159_162467_object36737:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42259dd:::benchmark_data_smithi159_162467_object43259:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f422f9ff:::benchmark_data_smithi159_162467_object37107:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f423657a:::benchmark_data_smithi159_162467_object21559:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4237889:::benchmark_data_smithi159_162467_object26155:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42379ae:::benchmark_data_smithi159_162467_object25801:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f424cf31:::benchmark_data_smithi159_162467_object60901:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4254aa4:::benchmark_data_smithi159_162467_object8953:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4256269:::benchmark_data_smithi159_162467_object15129:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4262991:::benchmark_data_smithi159_162467_object27785:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4264624:::benchmark_data_smithi159_162467_object74613:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f428aad9:::benchmark_data_smithi159_162467_object57050:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4299111:::benchmark_data_smithi159_162467_object45268:head# 2024-04-15T18:42:58.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42b145b:::benchmark_data_smithi159_162467_object21019:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42b3457:::benchmark_data_smithi159_162467_object8942:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42bcbd9:::benchmark_data_smithi159_162467_object25402:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42c8f77:::benchmark_data_smithi159_162467_object36843:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42e89e7:::benchmark_data_smithi159_162467_object37765:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f42ec127:::benchmark_data_smithi159_162467_object66220:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4306ba4:::benchmark_data_smithi159_162467_object20311:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4306f8e:::benchmark_data_smithi159_162467_object50692:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4306fc7:::benchmark_data_smithi159_162467_object37146:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f432bc12:::benchmark_data_smithi159_162467_object66422:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f434286d:::benchmark_data_smithi159_162467_object6833:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f434d671:::benchmark_data_smithi159_162467_object74592:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43703d7:::benchmark_data_smithi159_162467_object23593:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43760c1:::benchmark_data_smithi159_162467_object54886:head# 2024-04-15T18:42:58.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4382ec7:::benchmark_data_smithi159_162467_object57579:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43a7a6d:::benchmark_data_smithi159_162467_object18842:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43a83b0:::benchmark_data_smithi159_162467_object54615:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43baf63:::benchmark_data_smithi159_162467_object42426:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43c238e:::benchmark_data_smithi159_162467_object30473:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43cddb6:::benchmark_data_smithi159_162467_object26251:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43cf885:::benchmark_data_smithi159_162467_object51559:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43d1288:::benchmark_data_smithi159_162467_object52274:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43f62e1:::benchmark_data_smithi159_162467_object67146:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f43fe77a:::benchmark_data_smithi159_162467_object52446:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f440fc59:::benchmark_data_smithi159_162467_object66417:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4410c7a:::benchmark_data_smithi159_162467_object55728:head# 2024-04-15T18:42:58.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4412ea3:::benchmark_data_smithi159_162467_object73974:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f441d61a:::benchmark_data_smithi159_162467_object36023:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f443d4a5:::benchmark_data_smithi159_162467_object52735:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44477f8:::benchmark_data_smithi159_162467_object29669:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f444e495:::benchmark_data_smithi159_162467_object53879:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44688f0:::benchmark_data_smithi159_162467_object51676:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4473f43:::benchmark_data_smithi159_162467_object28272:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4484b7c:::benchmark_data_smithi159_162467_object30299:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44995f5:::benchmark_data_smithi159_162467_object35396:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f449c91a:::benchmark_data_smithi159_162467_object16078:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44ab884:::benchmark_data_smithi159_162467_object31897:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44cb027:::benchmark_data_smithi159_162467_object49525:head# 2024-04-15T18:42:58.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44defaa:::benchmark_data_smithi159_162467_object65870:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44ea7df:::benchmark_data_smithi159_162467_object6592:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f44f23cf:::benchmark_data_smithi159_162467_object72659:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4531986:::benchmark_data_smithi159_162467_object27605:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45376c6:::benchmark_data_smithi159_162467_object71461:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45455c5:::benchmark_data_smithi159_162467_object30841:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f456173f:::benchmark_data_smithi159_162467_object45691:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4570db7:::benchmark_data_smithi159_162467_object39696:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f458f5df:::benchmark_data_smithi159_162467_object808:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f459d048:::benchmark_data_smithi159_162467_object14299:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45b2c6b:::benchmark_data_smithi159_162467_object5359:head# 2024-04-15T18:42:58.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45b5b46:::benchmark_data_smithi159_162467_object11091:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45bbf49:::benchmark_data_smithi159_162467_object16412:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45c3d2c:::benchmark_data_smithi159_162467_object63942:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45c6f33:::benchmark_data_smithi159_162467_object56291:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45cde3d:::benchmark_data_smithi159_162467_object63615:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f45da16a:::benchmark_data_smithi159_162467_object71692:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f460aff3:::benchmark_data_smithi159_162467_object71112:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4634264:::benchmark_data_smithi159_162467_object57814:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f465c8fc:::benchmark_data_smithi159_162467_object62876:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46664e4:::benchmark_data_smithi159_162467_object12077:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46715f2:::benchmark_data_smithi159_162467_object65497:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46780da:::benchmark_data_smithi159_162467_object64138:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46804b9:::benchmark_data_smithi159_162467_object60678:head# 2024-04-15T18:42:58.295 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f469a11c:::benchmark_data_smithi159_162467_object55373:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46a96fd:::benchmark_data_smithi159_162467_object27556:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46b02e7:::benchmark_data_smithi159_162467_object61371:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46bd07d:::benchmark_data_smithi159_162467_object28523:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46cb242:::benchmark_data_smithi159_162467_object52742:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46cbcf4:::benchmark_data_smithi159_162467_object10632:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46d7068:::benchmark_data_smithi159_162467_object4343:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46e3c3a:::benchmark_data_smithi159_162467_object70970:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46ec5ad:::benchmark_data_smithi159_162467_object42930:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f46ff327:::benchmark_data_smithi159_162467_object10092:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47097b2:::benchmark_data_smithi159_162467_object18493:head# 2024-04-15T18:42:58.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f472c7a9:::benchmark_data_smithi159_162467_object27356:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4740a4a:::benchmark_data_smithi159_162467_object24024:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47470ef:::benchmark_data_smithi159_162467_object51632:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f476621c:::benchmark_data_smithi159_162467_object46202:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47731de:::benchmark_data_smithi159_162467_object31219:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f478f8a5:::benchmark_data_smithi159_162467_object19545:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f478fef9:::benchmark_data_smithi159_162467_object34299:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f479f4c8:::benchmark_data_smithi159_162467_object2591:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47aa3e6:::benchmark_data_smithi159_162467_object14700:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47c4cc8:::benchmark_data_smithi159_162467_object42466:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47c86a2:::benchmark_data_smithi159_162467_object30789:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47c96a3:::benchmark_data_smithi159_162467_object35047:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47eb6d9:::benchmark_data_smithi159_162467_object62443:head# 2024-04-15T18:42:58.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f47fafed:::benchmark_data_smithi159_162467_object72353:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48069fb:::benchmark_data_smithi159_162467_object31903:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f480e4a1:::benchmark_data_smithi159_162467_object24738:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f481522f:::benchmark_data_smithi159_162467_object65668:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f481b7d1:::benchmark_data_smithi159_162467_object61627:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f482c814:::benchmark_data_smithi159_162467_object53526:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f483007f:::benchmark_data_smithi159_162467_object51898:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f483060b:::benchmark_data_smithi159_162467_object33426:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f483f1a2:::benchmark_data_smithi159_162467_object1521:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f483ffee:::benchmark_data_smithi159_162467_object17489:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48431ac:::benchmark_data_smithi159_162467_object10915:head# 2024-04-15T18:42:58.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48604b3:::benchmark_data_smithi159_162467_object40584:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4861bbc:::benchmark_data_smithi159_162467_object34473:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4863e34:::benchmark_data_smithi159_162467_object53176:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f488ac35:::benchmark_data_smithi159_162467_object3005:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48b00a3:::benchmark_data_smithi159_162467_object55232:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48cda7b:::benchmark_data_smithi159_162467_object73412:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f48f657f:::benchmark_data_smithi159_162467_object4927:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f491b109:::benchmark_data_smithi159_162467_object69332:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f493f7ea:::benchmark_data_smithi159_162467_object46933:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4980352:::benchmark_data_smithi159_162467_object17548:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f498fafd:::benchmark_data_smithi159_162467_object6983:head# 2024-04-15T18:42:58.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4997281:::benchmark_data_smithi159_162467_object21756:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f499a399:::benchmark_data_smithi159_162467_object59980:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f499b8b3:::benchmark_data_smithi159_162467_object6876:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49a51e7:::benchmark_data_smithi159_162467_object53162:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49ad463:::benchmark_data_smithi159_162467_object49094:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49c3eec:::benchmark_data_smithi159_162467_object50989:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49cb340:::benchmark_data_smithi159_162467_object9:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49d3723:::benchmark_data_smithi159_162467_object46621:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f49da034:::benchmark_data_smithi159_162467_object26397:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a04f97:::benchmark_data_smithi159_162467_object54191:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a11488:::benchmark_data_smithi159_162467_object19626:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a153dc:::benchmark_data_smithi159_162467_object11356:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a26a4c:::benchmark_data_smithi159_162467_object17861:head# 2024-04-15T18:42:58.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a27a98:::benchmark_data_smithi159_162467_object65227:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a289a3:::benchmark_data_smithi159_162467_object5614:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a298e5:::benchmark_data_smithi159_162467_object67717:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a2ea0a:::benchmark_data_smithi159_162467_object25116:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a43f8e:::benchmark_data_smithi159_162467_object56901:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a51bfa:::benchmark_data_smithi159_162467_object62539:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a5da51:::benchmark_data_smithi159_162467_object18931:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a62551:::benchmark_data_smithi159_162467_object13231:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a8129f:::benchmark_data_smithi159_162467_object63485:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a943c4:::benchmark_data_smithi159_162467_object24810:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a9c3f1:::benchmark_data_smithi159_162467_object52112:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4a9c7bb:::benchmark_data_smithi159_162467_object67282:head# 2024-04-15T18:42:58.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4aa9cb1:::benchmark_data_smithi159_162467_object69619:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ab24c2:::benchmark_data_smithi159_162467_object33343:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4aedb69:::benchmark_data_smithi159_162467_object36608:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b0ed75:::benchmark_data_smithi159_162467_object64350:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b1b74a:::benchmark_data_smithi159_162467_object4974:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b2532a:::benchmark_data_smithi159_162467_object32913:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b3c2a0:::benchmark_data_smithi159_162467_object59111:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b466a5:::benchmark_data_smithi159_162467_object67475:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b4d298:::benchmark_data_smithi159_162467_object67834:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b5399d:::benchmark_data_smithi159_162467_object41048:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b54861:::benchmark_data_smithi159_162467_object62100:head# 2024-04-15T18:42:58.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b67fc7:::benchmark_data_smithi159_162467_object39283:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b68a36:::benchmark_data_smithi159_162467_object3364:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b8ca7f:::benchmark_data_smithi159_162467_object7716:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b8cc5f:::benchmark_data_smithi159_162467_object71032:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4b90748:::benchmark_data_smithi159_162467_object57558:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ba470e:::benchmark_data_smithi159_162467_object33600:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4bf372b:::benchmark_data_smithi159_162467_object20047:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c0f0af:::benchmark_data_smithi159_162467_object2117:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c16dd6:::benchmark_data_smithi159_162467_object6451:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c1864d:::benchmark_data_smithi159_162467_object67247:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c29b60:::benchmark_data_smithi159_162467_object25249:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c3603d:::benchmark_data_smithi159_162467_object3454:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c428cb:::benchmark_data_smithi159_162467_object73292:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c42a1a:::benchmark_data_smithi159_162467_object34395:head# 2024-04-15T18:42:58.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c60c0e:::benchmark_data_smithi159_162467_object3488:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c6a712:::benchmark_data_smithi159_162467_object7393:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c710b5:::benchmark_data_smithi159_162467_object42025:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c76255:::benchmark_data_smithi159_162467_object21762:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c7ae8c:::benchmark_data_smithi159_162467_object68277:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c81bfc:::benchmark_data_smithi159_162467_object70467:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c850d2:::benchmark_data_smithi159_162467_object47409:head# 2024-04-15T18:42:58.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c8e13d:::benchmark_data_smithi159_162467_object53352:head# 2024-04-15T18:42:58.310 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c8f84d:::benchmark_data_smithi159_162467_object48395:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c97a8e:::benchmark_data_smithi159_162467_object871:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4c9fa60:::benchmark_data_smithi159_162467_object20575:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4cad362:::benchmark_data_smithi159_162467_object35076:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4cc77c0:::benchmark_data_smithi159_162467_object50455:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ce812c:::benchmark_data_smithi159_162467_object13495:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4cedbcf:::benchmark_data_smithi159_162467_object38304:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4cf7cd0:::benchmark_data_smithi159_162467_object7591:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d081e0:::benchmark_data_smithi159_162467_object13655:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d09f36:::benchmark_data_smithi159_162467_object15603:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d15576:::benchmark_data_smithi159_162467_object54699:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d1a772:::benchmark_data_smithi159_162467_object61738:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d24ff7:::benchmark_data_smithi159_162467_object40580:head# 2024-04-15T18:42:58.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d3fb18:::benchmark_data_smithi159_162467_object14652:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d40b26:::benchmark_data_smithi159_162467_object33220:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d90bda:::benchmark_data_smithi159_162467_object34544:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d981a2:::benchmark_data_smithi159_162467_object24696:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4d9cf04:::benchmark_data_smithi159_162467_object52292:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dadcb8:::benchmark_data_smithi159_162467_object47301:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dbee66:::benchmark_data_smithi159_162467_object16517:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dc0cf3:::benchmark_data_smithi159_162467_object51904:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4de7737:::benchmark_data_smithi159_162467_object35552:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dec761:::benchmark_data_smithi159_162467_object19149:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dee69e:::benchmark_data_smithi159_162467_object625:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4dfdc1f:::benchmark_data_smithi159_162467_object20009:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e2ba17:::benchmark_data_smithi159_162467_object68976:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e41b8d:::benchmark_data_smithi159_162467_object39668:head# 2024-04-15T18:42:58.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e438cb:::benchmark_data_smithi159_162467_object47085:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e4a7df:::benchmark_data_smithi159_162467_object30639:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e4c2f3:::benchmark_data_smithi159_162467_object57144:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e54742:::benchmark_data_smithi159_162467_object34003:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e56f4e:::benchmark_data_smithi159_162467_object34345:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e67025:::benchmark_data_smithi159_162467_object8174:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e75b7a:::benchmark_data_smithi159_162467_object24307:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e8e203:::benchmark_data_smithi159_162467_object13827:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4e99be3:::benchmark_data_smithi159_162467_object53640:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ea3397:::benchmark_data_smithi159_162467_object20496:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ea9a22:::benchmark_data_smithi159_162467_object4867:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ec6307:::benchmark_data_smithi159_162467_object12439:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ed9235:::benchmark_data_smithi159_162467_object8929:head# 2024-04-15T18:42:58.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ed9f2b:::benchmark_data_smithi159_162467_object45561:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4edcc0b:::benchmark_data_smithi159_162467_object38537:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4eddc35:::benchmark_data_smithi159_162467_object30982:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4edf28d:::benchmark_data_smithi159_162467_object36206:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ee0125:::benchmark_data_smithi159_162467_object42153:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ee0943:::benchmark_data_smithi159_162467_object29093:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ee69f0:::benchmark_data_smithi159_162467_object31359:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4ef441c:::benchmark_data_smithi159_162467_object15963:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4efe966:::benchmark_data_smithi159_162467_object64407:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f213f8:::benchmark_data_smithi159_162467_object14909:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f28ff6:::benchmark_data_smithi159_162467_object64272:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f341df:::benchmark_data_smithi159_162467_object67825:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f37744:::benchmark_data_smithi159_162467_object31352:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f3cf94:::benchmark_data_smithi159_162467_object55036:head# 2024-04-15T18:42:58.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f53f4c:::benchmark_data_smithi159_162467_object57441:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f57aa1:::benchmark_data_smithi159_162467_object61136:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f5afd0:::benchmark_data_smithi159_162467_object27093:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f65817:::benchmark_data_smithi159_162467_object67097:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f816c8:::benchmark_data_smithi159_162467_object51003:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f8238d:::benchmark_data_smithi159_162467_object47469:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4f99ed4:::benchmark_data_smithi159_162467_object50234:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fa8fa0:::benchmark_data_smithi159_162467_object63046:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fa9306:::benchmark_data_smithi159_162467_object52434:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4faf3e6:::benchmark_data_smithi159_162467_object15313:head# 2024-04-15T18:42:58.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fb43e2:::benchmark_data_smithi159_162467_object3099:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fbdba9:::benchmark_data_smithi159_162467_object38268:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fbfe27:::benchmark_data_smithi159_162467_object23965:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fcd07c:::benchmark_data_smithi159_162467_object63099:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fcd4f9:::benchmark_data_smithi159_162467_object24719:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fd6d99:::benchmark_data_smithi159_162467_object64966:head# 2024-04-15T18:42:58.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fe6207:::benchmark_data_smithi159_162467_object35986:head# 2024-04-15T18:42:58.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f4fee1c0:::benchmark_data_smithi159_162467_object17295:head# 2024-04-15T18:42:58.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50046ec:::benchmark_data_smithi159_162467_object28602:head# 2024-04-15T18:42:58.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5005df7:::benchmark_data_smithi159_162467_object71276:head# 2024-04-15T18:42:58.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5005dfd:::benchmark_data_smithi159_162467_object11429:head# 2024-04-15T18:42:58.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5015527:::benchmark_data_smithi159_162467_object45692:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5018371:::benchmark_data_smithi159_162467_object59751:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5022b06:::benchmark_data_smithi159_162467_object43453:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5042e3d:::benchmark_data_smithi159_162467_object12037:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5051b5c:::benchmark_data_smithi159_162467_object65466:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f506f86a:::benchmark_data_smithi159_162467_object63742:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f507ff9d:::benchmark_data_smithi159_162467_object18259:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5081bd3:::benchmark_data_smithi159_162467_object57611:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50823e7:::benchmark_data_smithi159_162467_object64233:head# 2024-04-15T18:42:58.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f509b5e1:::benchmark_data_smithi159_162467_object24573:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50a8dcd:::benchmark_data_smithi159_162467_object15674:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50b9631:::benchmark_data_smithi159_162467_object38703:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50c7a94:::benchmark_data_smithi159_162467_object9089:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50d5b50:::benchmark_data_smithi159_162467_object47579:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50e412d:::benchmark_data_smithi159_162467_object39700:head# 2024-04-15T18:42:58.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50ebfd0:::benchmark_data_smithi159_162467_object32912:head# 2024-04-15T18:42:58.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50f10d4:::benchmark_data_smithi159_162467_object39978:head# 2024-04-15T18:42:58.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50f23c8:::benchmark_data_smithi159_162467_object9199:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50f68e7:::benchmark_data_smithi159_162467_object29085:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f50f751d:::benchmark_data_smithi159_162467_object20599:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51240a4:::benchmark_data_smithi159_162467_object53298:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5125942:::benchmark_data_smithi159_162467_object571:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51288df:::benchmark_data_smithi159_162467_object72421:head# 2024-04-15T18:42:58.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f512cfdb:::benchmark_data_smithi159_162467_object13482:head# 2024-04-15T18:42:58.325 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:58.325 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51463c1:::benchmark_data_smithi159_162467_object49634:head# 2024-04-15T18:42:58.325 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51587e4:::benchmark_data_smithi159_162467_object1307:head# 2024-04-15T18:42:58.325 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5167c3b:::benchmark_data_smithi159_162467_object32662:head# 2024-04-15T18:42:58.326 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f516e611:::benchmark_data_smithi159_162467_object31670:head# 2024-04-15T18:42:58.326 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5179827:::benchmark_data_smithi159_162467_object21334:head# 2024-04-15T18:42:58.327 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51a52a1:::benchmark_data_smithi159_162467_object73707:head# 2024-04-15T18:42:58.327 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51b68d5:::benchmark_data_smithi159_162467_object60633:head# 2024-04-15T18:42:58.328 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51c4a1c:::benchmark_data_smithi159_162467_object7743:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f51d1af7:::benchmark_data_smithi159_162467_object46181:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5208667:::benchmark_data_smithi159_162467_object12205:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f520fec6:::benchmark_data_smithi159_162467_object55599:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f521725b:::benchmark_data_smithi159_162467_object31729:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5219ab2:::benchmark_data_smithi159_162467_object66082:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f523dac5:::benchmark_data_smithi159_162467_object3858:head# 2024-04-15T18:42:58.331 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52411a0:::benchmark_data_smithi159_162467_object1213:head# 2024-04-15T18:42:58.332 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f524e7e3:::benchmark_data_smithi159_162467_object10753:head# 2024-04-15T18:42:58.332 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52671fd:::benchmark_data_smithi159_162467_object54755:head# 2024-04-15T18:42:58.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f526acdd:::benchmark_data_smithi159_162467_object64486:head# 2024-04-15T18:42:58.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f526e030:::benchmark_data_smithi159_162467_object10393:head# 2024-04-15T18:42:58.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f528068d:::benchmark_data_smithi159_162467_object20641:head# 2024-04-15T18:42:58.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52cee38:::benchmark_data_smithi159_162467_object22851:head# 2024-04-15T18:42:58.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52d2c3d:::benchmark_data_smithi159_162467_object59761:head# 2024-04-15T18:42:58.335 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52d82ab:::benchmark_data_smithi159_162467_object42799:head# 2024-04-15T18:42:58.335 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f52f35c0:::benchmark_data_smithi159_162467_object7776:head# 2024-04-15T18:42:58.336 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f530119b:::benchmark_data_smithi159_162467_object18157:head# 2024-04-15T18:42:58.336 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5318cfc:::benchmark_data_smithi159_162467_object5304:head# 2024-04-15T18:42:58.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f532a736:::benchmark_data_smithi159_162467_object21097:head# 2024-04-15T18:42:58.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f532f681:::benchmark_data_smithi159_162467_object10712:head# 2024-04-15T18:42:58.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5346198:::benchmark_data_smithi159_162467_object14694:head# 2024-04-15T18:42:58.338 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f534cb97:::benchmark_data_smithi159_162467_object71546:head# 2024-04-15T18:42:58.338 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f534fd1d:::benchmark_data_smithi159_162467_object68927:head# 2024-04-15T18:42:58.339 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5356cb2:::benchmark_data_smithi159_162467_object57401:head# 2024-04-15T18:42:58.341 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f535dd93:::benchmark_data_smithi159_162467_object21121:head# 2024-04-15T18:42:58.341 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5373bf3:::benchmark_data_smithi159_162467_object45529:head# 2024-04-15T18:42:58.341 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53779cb:::benchmark_data_smithi159_162467_object38270:head# 2024-04-15T18:42:58.341 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f537d1a7:::benchmark_data_smithi159_162467_object42703:head# 2024-04-15T18:42:58.342 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53826e1:::benchmark_data_smithi159_162467_object17422:head# 2024-04-15T18:42:58.342 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5384ae7:::benchmark_data_smithi159_162467_object25375:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53863b4:::benchmark_data_smithi159_162467_object21472:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f538ce28:::benchmark_data_smithi159_162467_object17467:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f538e566:::benchmark_data_smithi159_162467_object72278:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53b04b1:::benchmark_data_smithi159_162467_object16352:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53b44e3:::benchmark_data_smithi159_162467_object24600:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53bce43:::benchmark_data_smithi159_162467_object67851:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53d586a:::benchmark_data_smithi159_162467_object55751:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53df78a:::benchmark_data_smithi159_162467_object27996:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53e90c2:::benchmark_data_smithi159_162467_object54450:head# 2024-04-15T18:42:58.347 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:58.348 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53f5dbc:::benchmark_data_smithi159_162467_object17050:head# 2024-04-15T18:42:58.348 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53fbde6:::benchmark_data_smithi159_162467_object72375:head# 2024-04-15T18:42:58.348 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f53fd481:::benchmark_data_smithi159_162467_object44271:head# 2024-04-15T18:42:58.349 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5401756:::benchmark_data_smithi159_162467_object13914:head# 2024-04-15T18:42:58.349 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f540c9fe:::benchmark_data_smithi159_162467_object19507:head# 2024-04-15T18:42:58.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5441d6e:::benchmark_data_smithi159_162467_object1524:head# 2024-04-15T18:42:58.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f544ab65:::benchmark_data_smithi159_162467_object65581:head# 2024-04-15T18:42:58.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f546579a:::benchmark_data_smithi159_162467_object39048:head# 2024-04-15T18:42:58.352 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5479294:::benchmark_data_smithi159_162467_object6586:head# 2024-04-15T18:42:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:58.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:58.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f549997e:::benchmark_data_smithi159_162467_object74129:head# 2024-04-15T18:42:58.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54af2b0:::benchmark_data_smithi159_162467_object67240:head# 2024-04-15T18:42:58.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54b63c2:::benchmark_data_smithi159_162467_object29247:head# 2024-04-15T18:42:58.354 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54bd7f5:::benchmark_data_smithi159_162467_object18705:head# 2024-04-15T18:42:58.354 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54ca04e:::benchmark_data_smithi159_162467_object32116:head# 2024-04-15T18:42:58.354 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54d2be4:::benchmark_data_smithi159_162467_object40518:head# 2024-04-15T18:42:58.355 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54da7e2:::benchmark_data_smithi159_162467_object37274:head# 2024-04-15T18:42:58.358 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54e3893:::benchmark_data_smithi159_162467_object33571:head# 2024-04-15T18:42:58.358 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f54f4d04:::benchmark_data_smithi159_162467_object45560:head# 2024-04-15T18:42:58.359 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f552a907:::benchmark_data_smithi159_162467_object25317:head# 2024-04-15T18:42:58.359 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f552f17a:::benchmark_data_smithi159_162467_object44299:head# 2024-04-15T18:42:58.360 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5546e6e:::benchmark_data_smithi159_162467_object16505:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f554854d:::benchmark_data_smithi159_162467_object62705:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5555ec6:::benchmark_data_smithi159_162467_object43602:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5572e8f:::benchmark_data_smithi159_162467_object1565:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f557d318:::benchmark_data_smithi159_162467_object30022:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5587cbe:::benchmark_data_smithi159_162467_object59021:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55a4e30:::benchmark_data_smithi159_162467_object61962:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55b241f:::benchmark_data_smithi159_162467_object17061:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55c859d:::benchmark_data_smithi159_162467_object20785:head# 2024-04-15T18:42:58.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55c9506:::benchmark_data_smithi159_162467_object2177:head# 2024-04-15T18:42:58.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55d9697:::benchmark_data_smithi159_162467_object47899:head# 2024-04-15T18:42:58.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f55ea8e0:::benchmark_data_smithi159_162467_object42412:head# 2024-04-15T18:42:58.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5600ec3:::benchmark_data_smithi159_162467_object41444:head# 2024-04-15T18:42:58.366 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5613027:::benchmark_data_smithi159_162467_object7421:head# 2024-04-15T18:42:58.366 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56142e2:::benchmark_data_smithi159_162467_object60168:head# 2024-04-15T18:42:58.366 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5619ab5:::benchmark_data_smithi159_162467_object4658:head# 2024-04-15T18:42:58.367 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5621427:::benchmark_data_smithi159_162467_object21845:head# 2024-04-15T18:42:58.367 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f562fe2f:::benchmark_data_smithi159_162467_object29419:head# 2024-04-15T18:42:58.367 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5653d28:::benchmark_data_smithi159_162467_object14631:head# 2024-04-15T18:42:58.368 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f566bc2b:::benchmark_data_smithi159_162467_object13284:head# 2024-04-15T18:42:58.368 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5698eb5:::benchmark_data_smithi159_162467_object4590:head# 2024-04-15T18:42:58.369 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56a093f:::benchmark_data_smithi159_162467_object36364:head# 2024-04-15T18:42:58.369 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56b73b8:::benchmark_data_smithi159_162467_object18950:head# 2024-04-15T18:42:58.370 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56b8705:::benchmark_data_smithi159_162467_object10073:head# 2024-04-15T18:42:58.370 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56cf793:::benchmark_data_smithi159_162467_object74337:head# 2024-04-15T18:42:58.371 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56dbf6d:::benchmark_data_smithi159_162467_object12860:head# 2024-04-15T18:42:58.371 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56dfda2:::benchmark_data_smithi159_162467_object71182:head# 2024-04-15T18:42:58.372 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f56f9682:::benchmark_data_smithi159_162467_object10420:head# 2024-04-15T18:42:58.372 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f570cdfa:::benchmark_data_smithi159_162467_object21532:head# 2024-04-15T18:42:58.373 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f570d4b9:::benchmark_data_smithi159_162467_object26460:head# 2024-04-15T18:42:58.373 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f571ef39:::benchmark_data_smithi159_162467_object25590:head# 2024-04-15T18:42:58.373 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f571ff37:::benchmark_data_smithi159_162467_object29615:head# 2024-04-15T18:42:58.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5721b8f:::benchmark_data_smithi159_162467_object20825:head# 2024-04-15T18:42:58.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f572a547:::benchmark_data_smithi159_162467_object58813:head# 2024-04-15T18:42:58.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f574ba1b:::benchmark_data_smithi159_162467_object52602:head# 2024-04-15T18:42:58.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5754806:::benchmark_data_smithi159_162467_object37075:head# 2024-04-15T18:42:58.376 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57575ba:::benchmark_data_smithi159_162467_object18345:head# 2024-04-15T18:42:58.376 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f575ce93:::benchmark_data_smithi159_162467_object14464:head# 2024-04-15T18:42:58.377 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57698b4:::benchmark_data_smithi159_162467_object7071:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5776c5d:::benchmark_data_smithi159_162467_object40382:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f579cbc4:::benchmark_data_smithi159_162467_object23399:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57a1a5e:::benchmark_data_smithi159_162467_object25315:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57c4c02:::benchmark_data_smithi159_162467_object12162:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57c5f62:::benchmark_data_smithi159_162467_object72035:head# 2024-04-15T18:42:58.380 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57d0fd4:::benchmark_data_smithi159_162467_object66359:head# 2024-04-15T18:42:58.381 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57e8ec9:::benchmark_data_smithi159_162467_object39836:head# 2024-04-15T18:42:58.381 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57eca7d:::benchmark_data_smithi159_162467_object7061:head# 2024-04-15T18:42:58.381 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57fa332:::benchmark_data_smithi159_162467_object61246:head# 2024-04-15T18:42:58.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f57fcc5b:::benchmark_data_smithi159_162467_object28819:head# 2024-04-15T18:42:58.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f580af9c:::benchmark_data_smithi159_162467_object23191:head# 2024-04-15T18:42:58.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f580ca52:::benchmark_data_smithi159_162467_object21279:head# 2024-04-15T18:42:58.384 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f580f3aa:::benchmark_data_smithi159_162467_object25733:head# 2024-04-15T18:42:58.384 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58158f0:::benchmark_data_smithi159_162467_object32345:head# 2024-04-15T18:42:58.385 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5825d9b:::benchmark_data_smithi159_162467_object41677:head# 2024-04-15T18:42:58.385 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58460cc:::benchmark_data_smithi159_162467_object56077:head# 2024-04-15T18:42:58.386 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5872bbc:::benchmark_data_smithi159_162467_object41673:head# 2024-04-15T18:42:58.386 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f587c23a:::benchmark_data_smithi159_162467_object748:head# 2024-04-15T18:42:58.387 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f588d925:::benchmark_data_smithi159_162467_object29522:head# 2024-04-15T18:42:58.387 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58bfea5:::benchmark_data_smithi159_162467_object51435:head# 2024-04-15T18:42:58.388 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58d81f9:::benchmark_data_smithi159_162467_object10931:head# 2024-04-15T18:42:58.388 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58dcf6d:::benchmark_data_smithi159_162467_object71427:head# 2024-04-15T18:42:58.389 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58dfebe:::benchmark_data_smithi159_162467_object23421:head# 2024-04-15T18:42:58.389 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58e699b:::benchmark_data_smithi159_162467_object66782:head# 2024-04-15T18:42:58.390 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f58ea23d:::benchmark_data_smithi159_162467_object65574:head# 2024-04-15T18:42:58.390 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59186a9:::benchmark_data_smithi159_162467_object28439:head# 2024-04-15T18:42:58.390 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f592433f:::benchmark_data_smithi159_162467_object30040:head# 2024-04-15T18:42:58.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5926bc8:::benchmark_data_smithi159_162467_object48624:head# 2024-04-15T18:42:58.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59479f3:::benchmark_data_smithi159_162467_object56116:head# 2024-04-15T18:42:58.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59652e8:::benchmark_data_smithi159_162467_object53603:head# 2024-04-15T18:42:58.392 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5970e43:::benchmark_data_smithi159_162467_object24272:head# 2024-04-15T18:42:58.393 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5983658:::benchmark_data_smithi159_162467_object68377:head# 2024-04-15T18:42:58.395 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5984eae:::benchmark_data_smithi159_162467_object45015:head# 2024-04-15T18:42:58.395 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f598c5f7:::benchmark_data_smithi159_162467_object37785:head# 2024-04-15T18:42:58.396 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59a0a65:::benchmark_data_smithi159_162467_object39870:head# 2024-04-15T18:42:58.396 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59a696d:::benchmark_data_smithi159_162467_object11590:head# 2024-04-15T18:42:58.397 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59b5a40:::benchmark_data_smithi159_162467_object42816:head# 2024-04-15T18:42:58.397 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59b9273:::benchmark_data_smithi159_162467_object63133:head# 2024-04-15T18:42:58.398 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59df799:::benchmark_data_smithi159_162467_object61831:head# 2024-04-15T18:42:58.398 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59e2681:::benchmark_data_smithi159_162467_object6169:head# 2024-04-15T18:42:58.399 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f59f1072:::benchmark_data_smithi159_162467_object6114:head# 2024-04-15T18:42:58.400 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a0f33f:::benchmark_data_smithi159_162467_object16310:head# 2024-04-15T18:42:58.402 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a28e63:::benchmark_data_smithi159_162467_object48459:head# 2024-04-15T18:42:58.403 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a2b789:::benchmark_data_smithi159_162467_object65036:head# 2024-04-15T18:42:58.403 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a35dd7:::benchmark_data_smithi159_162467_object56924:head# 2024-04-15T18:42:58.403 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a3c3fe:::benchmark_data_smithi159_162467_object51365:head# 2024-04-15T18:42:58.404 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a3ebaf:::benchmark_data_smithi159_162467_object16059:head# 2024-04-15T18:42:58.404 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a4e3dd:::benchmark_data_smithi159_162467_object13396:head# 2024-04-15T18:42:58.405 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a67c3d:::benchmark_data_smithi159_162467_object1811:head# 2024-04-15T18:42:58.405 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a711d4:::benchmark_data_smithi159_162467_object36666:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a7132c:::benchmark_data_smithi159_162467_object33844:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a8040d:::benchmark_data_smithi159_162467_object31956:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a84d8a:::benchmark_data_smithi159_162467_object28416:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5a94fcd:::benchmark_data_smithi159_162467_object54492:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5abf04c:::benchmark_data_smithi159_162467_object65995:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ac54fc:::benchmark_data_smithi159_162467_object41403:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ac69cf:::benchmark_data_smithi159_162467_object71725:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ac6e7c:::benchmark_data_smithi159_162467_object6111:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ac817f:::benchmark_data_smithi159_162467_object26066:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5acf62f:::benchmark_data_smithi159_162467_object31247:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ae33fb:::benchmark_data_smithi159_162467_object35936:head# 2024-04-15T18:42:58.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ae921f:::benchmark_data_smithi159_162467_object42051:head# 2024-04-15T18:42:58.412 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5aff36b:::benchmark_data_smithi159_162467_object72266:head# 2024-04-15T18:42:58.412 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b14f14:::benchmark_data_smithi159_162467_object47273:head# 2024-04-15T18:42:58.412 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b15042:::benchmark_data_smithi159_162467_object60393:head# 2024-04-15T18:42:58.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b2ffa1:::benchmark_data_smithi159_162467_object8602:head# 2024-04-15T18:42:58.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b323dc:::benchmark_data_smithi159_162467_object68703:head# 2024-04-15T18:42:58.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b34e53:::benchmark_data_smithi159_162467_object36091:head# 2024-04-15T18:42:58.414 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b4072a:::benchmark_data_smithi159_162467_object14978:head# 2024-04-15T18:42:58.414 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b507a4:::benchmark_data_smithi159_162467_object54361:head# 2024-04-15T18:42:58.415 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b5d3ee:::benchmark_data_smithi159_162467_object9273:head# 2024-04-15T18:42:58.416 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b678d1:::benchmark_data_smithi159_162467_object61766:head# 2024-04-15T18:42:58.416 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b7dd1a:::benchmark_data_smithi159_162467_object52776:head# 2024-04-15T18:42:58.417 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b8de08:::benchmark_data_smithi159_162467_object53830:head# 2024-04-15T18:42:58.417 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5b8e9bd:::benchmark_data_smithi159_162467_object15792:head# 2024-04-15T18:42:58.422 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ba01ec:::benchmark_data_smithi159_162467_object44768:head# 2024-04-15T18:42:58.422 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ba0342:::benchmark_data_smithi159_162467_object54389:head# 2024-04-15T18:42:58.422 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ba0e1e:::benchmark_data_smithi159_162467_object38658:head# 2024-04-15T18:42:58.422 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ba54b0:::benchmark_data_smithi159_162467_object12250:head# 2024-04-15T18:42:58.422 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5bac263:::benchmark_data_smithi159_162467_object23701:head# 2024-04-15T18:42:58.423 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5bb797c:::benchmark_data_smithi159_162467_object59985:head# 2024-04-15T18:42:58.423 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5bbbab6:::benchmark_data_smithi159_162467_object53982:head# 2024-04-15T18:42:58.423 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5bc4d0c:::benchmark_data_smithi159_162467_object18783:head# 2024-04-15T18:42:58.423 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5bd1a01:::benchmark_data_smithi159_162467_object43774:head# 2024-04-15T18:42:58.424 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5be0f43:::benchmark_data_smithi159_162467_object22440:head# 2024-04-15T18:42:58.424 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5be7d10:::benchmark_data_smithi159_162467_object19266:head# 2024-04-15T18:42:58.424 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c03974:::benchmark_data_smithi159_162467_object12913:head# 2024-04-15T18:42:58.425 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c078aa:::benchmark_data_smithi159_162467_object66826:head# 2024-04-15T18:42:58.425 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c16ad8:::benchmark_data_smithi159_162467_object69552:head# 2024-04-15T18:42:58.426 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c235ce:::benchmark_data_smithi159_162467_object56418:head# 2024-04-15T18:42:58.426 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c3ca0b:::benchmark_data_smithi159_162467_object27948:head# 2024-04-15T18:42:58.427 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c3d112:::benchmark_data_smithi159_162467_object646:head# 2024-04-15T18:42:58.427 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c5512b:::benchmark_data_smithi159_162467_object50891:head# 2024-04-15T18:42:58.427 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c5bb4e:::benchmark_data_smithi159_162467_object1758:head# 2024-04-15T18:42:58.428 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c692c8:::benchmark_data_smithi159_162467_object37688:head# 2024-04-15T18:42:58.428 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c7d7c6:::benchmark_data_smithi159_162467_object69213:head# 2024-04-15T18:42:58.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c7f44d:::benchmark_data_smithi159_162467_object37803:head# 2024-04-15T18:42:58.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c800f5:::benchmark_data_smithi159_162467_object57038:head# 2024-04-15T18:42:58.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c8ad34:::benchmark_data_smithi159_162467_object64473:head# 2024-04-15T18:42:58.430 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5c9e90f:::benchmark_data_smithi159_162467_object29259:head# 2024-04-15T18:42:58.430 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5caf814:::benchmark_data_smithi159_162467_object37867:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5cc3f6a:::benchmark_data_smithi159_162467_object58528:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ccd464:::benchmark_data_smithi159_162467_object3422:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ccf912:::benchmark_data_smithi159_162467_object47686:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5cf2089:::benchmark_data_smithi159_162467_object58303:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5cf993b:::benchmark_data_smithi159_162467_object69850:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5cfdeee:::benchmark_data_smithi159_162467_object40830:head# 2024-04-15T18:42:58.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d00bd6:::benchmark_data_smithi159_162467_object60340:head# 2024-04-15T18:42:58.435 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d0d1b8:::benchmark_data_smithi159_162467_object58247:head# 2024-04-15T18:42:58.435 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d33c56:::benchmark_data_smithi159_162467_object72106:head# 2024-04-15T18:42:58.435 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d35cdd:::benchmark_data_smithi159_162467_object17024:head# 2024-04-15T18:42:58.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d39fea:::benchmark_data_smithi159_162467_object2348:head# 2024-04-15T18:42:58.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d3b50d:::benchmark_data_smithi159_162467_object10496:head# 2024-04-15T18:42:58.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d3faf1:::benchmark_data_smithi159_162467_object74559:head# 2024-04-15T18:42:58.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d644dc:::benchmark_data_smithi159_162467_object32264:head# 2024-04-15T18:42:58.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d6cc3e:::benchmark_data_smithi159_162467_object29474:head# 2024-04-15T18:42:58.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d8ca03:::benchmark_data_smithi159_162467_object52751:head# 2024-04-15T18:42:58.438 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5d9636a:::benchmark_data_smithi159_162467_object46299:head# 2024-04-15T18:42:58.438 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dbc50e:::benchmark_data_smithi159_162467_object66574:head# 2024-04-15T18:42:58.439 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dbc907:::benchmark_data_smithi159_162467_object71233:head# 2024-04-15T18:42:58.439 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dbf6ec:::benchmark_data_smithi159_162467_object32978:head# 2024-04-15T18:42:58.439 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dd96b7:::benchmark_data_smithi159_162467_object37027:head# 2024-04-15T18:42:58.440 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dfe3b7:::benchmark_data_smithi159_162467_object22103:head# 2024-04-15T18:42:58.440 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5dff6cb:::benchmark_data_smithi159_162467_object66026:head# 2024-04-15T18:42:58.440 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e06e0e:::benchmark_data_smithi159_162467_object57999:head# 2024-04-15T18:42:58.441 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e0cfa1:::benchmark_data_smithi159_162467_object53855:head# 2024-04-15T18:42:58.441 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e0e9db:::benchmark_data_smithi159_162467_object5145:head# 2024-04-15T18:42:58.442 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e121a0:::benchmark_data_smithi159_162467_object40539:head# 2024-04-15T18:42:58.442 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e14c41:::benchmark_data_smithi159_162467_object15105:head# 2024-04-15T18:42:58.442 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e22bd8:::benchmark_data_smithi159_162467_object41752:head# 2024-04-15T18:42:58.443 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e2ed78:::benchmark_data_smithi159_162467_object67491:head# 2024-04-15T18:42:58.443 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e2f950:::benchmark_data_smithi159_162467_object45745:head# 2024-04-15T18:42:58.444 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e356ac:::benchmark_data_smithi159_162467_object30416:head# 2024-04-15T18:42:58.444 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e5cb1d:::benchmark_data_smithi159_162467_object52543:head# 2024-04-15T18:42:58.445 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e66df2:::benchmark_data_smithi159_162467_object48492:head# 2024-04-15T18:42:58.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e680bd:::benchmark_data_smithi159_162467_object50338:head# 2024-04-15T18:42:58.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e6a72b:::benchmark_data_smithi159_162467_object9706:head# 2024-04-15T18:42:58.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e80225:::benchmark_data_smithi159_162467_object15806:head# 2024-04-15T18:42:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e952b0:::benchmark_data_smithi159_162467_object8284:head# 2024-04-15T18:42:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e99231:::benchmark_data_smithi159_162467_object15936:head# 2024-04-15T18:42:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e9c561:::benchmark_data_smithi159_162467_object50348:head# 2024-04-15T18:42:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5e9e68f:::benchmark_data_smithi159_162467_object11102:head# 2024-04-15T18:42:58.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ea67e8:::benchmark_data_smithi159_162467_object33652:head# 2024-04-15T18:42:58.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ec083f:::benchmark_data_smithi159_162467_object57237:head# 2024-04-15T18:42:58.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ec122f:::benchmark_data_smithi159_162467_object64559:head# 2024-04-15T18:42:58.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5eec1f6:::benchmark_data_smithi159_162467_object25643:head# 2024-04-15T18:42:58.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f0c134:::benchmark_data_smithi159_162467_object18117:head# 2024-04-15T18:42:58.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f1420d:::benchmark_data_smithi159_162467_object21906:head# 2024-04-15T18:42:58.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f1b343:::benchmark_data_smithi159_162467_object26065:head# 2024-04-15T18:42:58.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f235f3:::benchmark_data_smithi159_162467_object28207:head# 2024-04-15T18:42:58.452 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f3d210:::benchmark_data_smithi159_162467_object63693:head# 2024-04-15T18:42:58.453 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:42:58.454 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f47816:::benchmark_data_smithi159_162467_object51953:head# 2024-04-15T18:42:58.456 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f5b73b:::benchmark_data_smithi159_162467_object47062:head# 2024-04-15T18:42:58.456 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f60001:::benchmark_data_smithi159_162467_object67755:head# 2024-04-15T18:42:58.456 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f8ae03:::benchmark_data_smithi159_162467_object3439:head# 2024-04-15T18:42:58.456 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f91cac:::benchmark_data_smithi159_162467_object74193:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5f94d8b:::benchmark_data_smithi159_162467_object1327:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fa86b0:::benchmark_data_smithi159_162467_object31940:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fa96be:::benchmark_data_smithi159_162467_object28713:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5facc3a:::benchmark_data_smithi159_162467_object74188:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5faf206:::benchmark_data_smithi159_162467_object19363:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fc98b8:::benchmark_data_smithi159_162467_object35688:head# 2024-04-15T18:42:58.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fd2421:::benchmark_data_smithi159_162467_object16259:head# 2024-04-15T18:42:58.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fd9f02:::benchmark_data_smithi159_162467_object6394:head# 2024-04-15T18:42:58.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fe0fcf:::benchmark_data_smithi159_162467_object60977:head# 2024-04-15T18:42:58.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fe307b:::benchmark_data_smithi159_162467_object65844:head# 2024-04-15T18:42:58.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5fe532a:::benchmark_data_smithi159_162467_object6680:head# 2024-04-15T18:42:58.461 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ff4df1:::benchmark_data_smithi159_162467_object759:head# 2024-04-15T18:42:58.461 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f5ff7214:::benchmark_data_smithi159_162467_object11856:head# 2024-04-15T18:42:58.461 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6007742:::benchmark_data_smithi159_162467_object35718:head# 2024-04-15T18:42:58.462 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f601670f:::benchmark_data_smithi159_162467_object14819:head# 2024-04-15T18:42:58.462 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6029988:::benchmark_data_smithi159_162467_object47305:head# 2024-04-15T18:42:58.463 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f602cfe4:::benchmark_data_smithi159_162467_object66891:head# 2024-04-15T18:42:58.463 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f603688e:::benchmark_data_smithi159_162467_object63252:head# 2024-04-15T18:42:58.463 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6037a18:::benchmark_data_smithi159_162467_object52167:head# 2024-04-15T18:42:58.464 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f603e644:::benchmark_data_smithi159_162467_object405:head# 2024-04-15T18:42:58.464 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f605e199:::benchmark_data_smithi159_162467_object47193:head# 2024-04-15T18:42:58.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60632ec:::benchmark_data_smithi159_162467_object68301:head# 2024-04-15T18:42:58.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f606869d:::benchmark_data_smithi159_162467_object17608:head# 2024-04-15T18:42:58.466 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6069be4:::benchmark_data_smithi159_162467_object69931:head# 2024-04-15T18:42:58.466 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f606adf8:::benchmark_data_smithi159_162467_object48093:head# 2024-04-15T18:42:58.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f608c35c:::benchmark_data_smithi159_162467_object69207:head# 2024-04-15T18:42:58.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f609bc74:::benchmark_data_smithi159_162467_object7775:head# 2024-04-15T18:42:58.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f609ce6f:::benchmark_data_smithi159_162467_object11663:head# 2024-04-15T18:42:58.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60a32fa:::benchmark_data_smithi159_162467_object15255:head# 2024-04-15T18:42:58.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60ad938:::benchmark_data_smithi159_162467_object55378:head# 2024-04-15T18:42:58.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60e4c49:::benchmark_data_smithi159_162467_object10097:head# 2024-04-15T18:42:58.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60e9758:::benchmark_data_smithi159_162467_object35136:head# 2024-04-15T18:42:58.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f60f3cd3:::benchmark_data_smithi159_162467_object27305:head# 2024-04-15T18:42:58.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f610c24b:::benchmark_data_smithi159_162467_object26621:head# 2024-04-15T18:42:58.472 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f610d8e2:::benchmark_data_smithi159_162467_object27918:head# 2024-04-15T18:42:58.474 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f611132b:::benchmark_data_smithi159_162467_object20889:head# 2024-04-15T18:42:58.474 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6112579:::benchmark_data_smithi159_162467_object10222:head# 2024-04-15T18:42:58.474 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f611333a:::benchmark_data_smithi159_162467_object67128:head# 2024-04-15T18:42:58.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f612a827:::benchmark_data_smithi159_162467_object64661:head# 2024-04-15T18:42:58.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6131b6c:::benchmark_data_smithi159_162467_object53939:head# 2024-04-15T18:42:58.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f616640b:::benchmark_data_smithi159_162467_object39654:head# 2024-04-15T18:42:58.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f616a10f:::benchmark_data_smithi159_162467_object61018:head# 2024-04-15T18:42:58.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f616b2fb:::benchmark_data_smithi159_162467_object9744:head# 2024-04-15T18:42:58.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f616b5b8:::benchmark_data_smithi159_162467_object63819:head# 2024-04-15T18:42:58.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6173690:::benchmark_data_smithi159_162467_object13582:head# 2024-04-15T18:42:58.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6185fd5:::benchmark_data_smithi159_162467_object55986:head# 2024-04-15T18:42:58.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6189ba1:::benchmark_data_smithi159_162467_object21300:head# 2024-04-15T18:42:58.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f618d623:::benchmark_data_smithi159_162467_object50546:head# 2024-04-15T18:42:58.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f618f4de:::benchmark_data_smithi159_162467_object26725:head# 2024-04-15T18:42:58.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f619c877:::benchmark_data_smithi159_162467_object5244:head# 2024-04-15T18:42:58.478 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f619ca98:::benchmark_data_smithi159_162467_object11174:head# 2024-04-15T18:42:58.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61a2c64:::benchmark_data_smithi159_162467_object6168:head# 2024-04-15T18:42:58.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61ac4e4:::benchmark_data_smithi159_162467_object66023:head# 2024-04-15T18:42:58.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61c3215:::benchmark_data_smithi159_162467_object21735:head# 2024-04-15T18:42:58.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61d116b:::benchmark_data_smithi159_162467_object63766:head# 2024-04-15T18:42:58.480 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61d48ea:::benchmark_data_smithi159_162467_object12233:head# 2024-04-15T18:42:58.480 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61e6b1b:::benchmark_data_smithi159_162467_object19661:head# 2024-04-15T18:42:58.480 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61ec8cf:::benchmark_data_smithi159_162467_object66201:head# 2024-04-15T18:42:58.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f61fcd44:::benchmark_data_smithi159_162467_object31467:head# 2024-04-15T18:42:58.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62140d2:::benchmark_data_smithi159_162467_object22735:head# 2024-04-15T18:42:58.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f621b731:::benchmark_data_smithi159_162467_object46695:head# 2024-04-15T18:42:58.482 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6224d3e:::benchmark_data_smithi159_162467_object24837:head# 2024-04-15T18:42:58.482 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62299b0:::benchmark_data_smithi159_162467_object24414:head# 2024-04-15T18:42:58.483 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6232372:::benchmark_data_smithi159_162467_object39667:head# 2024-04-15T18:42:58.483 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6237912:::benchmark_data_smithi159_162467_object26535:head# 2024-04-15T18:42:58.484 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f624cede:::benchmark_data_smithi159_162467_object26436:head# 2024-04-15T18:42:58.484 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6268f7f:::benchmark_data_smithi159_162467_object47486:head# 2024-04-15T18:42:58.485 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:58.487 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f626e051:::benchmark_data_smithi159_162467_object53592:head# 2024-04-15T18:42:58.488 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6270cb7:::benchmark_data_smithi159_162467_object40937:head# 2024-04-15T18:42:58.488 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62794ce:::benchmark_data_smithi159_162467_object62957:head# 2024-04-15T18:42:58.489 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f628a4e8:::benchmark_data_smithi159_162467_object5416:head# 2024-04-15T18:42:58.489 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f629bfde:::benchmark_data_smithi159_162467_object45844:head# 2024-04-15T18:42:58.490 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62b554a:::benchmark_data_smithi159_162467_object26622:head# 2024-04-15T18:42:58.490 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62d3ccc:::benchmark_data_smithi159_162467_object49707:head# 2024-04-15T18:42:58.490 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62e5d8e:::benchmark_data_smithi159_162467_object32682:head# 2024-04-15T18:42:58.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f62fd572:::benchmark_data_smithi159_162467_object16314:head# 2024-04-15T18:42:58.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6303e2f:::benchmark_data_smithi159_162467_object63698:head# 2024-04-15T18:42:58.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f630a612:::benchmark_data_smithi159_162467_object26939:head# 2024-04-15T18:42:58.492 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6327559:::benchmark_data_smithi159_162467_object12949:head# 2024-04-15T18:42:58.492 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f632ca26:::benchmark_data_smithi159_162467_object53817:head# 2024-04-15T18:42:58.493 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f633162a:::benchmark_data_smithi159_162467_object60658:head# 2024-04-15T18:42:58.493 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6337e0d:::benchmark_data_smithi159_162467_object63611:head# 2024-04-15T18:42:58.494 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6351af2:::benchmark_data_smithi159_162467_object66599:head# 2024-04-15T18:42:58.494 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6354520:::benchmark_data_smithi159_162467_object39713:head# 2024-04-15T18:42:58.495 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63652a2:::benchmark_data_smithi159_162467_object25338:head# 2024-04-15T18:42:58.495 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63685a0:::benchmark_data_smithi159_162467_object23340:head# 2024-04-15T18:42:58.495 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f639c6d9:::benchmark_data_smithi159_162467_object18621:head# 2024-04-15T18:42:58.496 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63b97c2:::benchmark_data_smithi159_162467_object6615:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63be20e:::benchmark_data_smithi159_162467_object27241:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63d4d15:::benchmark_data_smithi159_162467_object47078:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63d9185:::benchmark_data_smithi159_162467_object61:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63e7fe6:::benchmark_data_smithi159_162467_object54052:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63f1ec3:::benchmark_data_smithi159_162467_object5312:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f63f808c:::benchmark_data_smithi159_162467_object23885:head# 2024-04-15T18:42:58.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64069d6:::benchmark_data_smithi159_162467_object351:head# 2024-04-15T18:42:58.500 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f640e920:::benchmark_data_smithi159_162467_object47685:head# 2024-04-15T18:42:58.500 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f641e61e:::benchmark_data_smithi159_162467_object16949:head# 2024-04-15T18:42:58.500 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6427e53:::benchmark_data_smithi159_162467_object23879:head# 2024-04-15T18:42:58.500 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f642bd57:::benchmark_data_smithi159_162467_object6831:head# 2024-04-15T18:42:58.501 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f646ac0f:::benchmark_data_smithi159_162467_object7648:head# 2024-04-15T18:42:58.501 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f646e53a:::benchmark_data_smithi159_162467_object60100:head# 2024-04-15T18:42:58.502 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6470806:::benchmark_data_smithi159_162467_object56829:head# 2024-04-15T18:42:58.502 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6477025:::benchmark_data_smithi159_162467_object15098:head# 2024-04-15T18:42:58.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f647947e:::benchmark_data_smithi159_162467_object24309:head# 2024-04-15T18:42:58.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f647b167:::benchmark_data_smithi159_162467_object67386:head# 2024-04-15T18:42:58.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f649765b:::benchmark_data_smithi159_162467_object72450:head# 2024-04-15T18:42:58.504 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6497dac:::benchmark_data_smithi159_162467_object21603:head# 2024-04-15T18:42:58.504 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64a8fd1:::benchmark_data_smithi159_162467_object40169:head# 2024-04-15T18:42:58.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64ab3f0:::benchmark_data_smithi159_162467_object44509:head# 2024-04-15T18:42:58.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64b62f9:::benchmark_data_smithi159_162467_object63592:head# 2024-04-15T18:42:58.506 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64bef0b:::benchmark_data_smithi159_162467_object73249:head# 2024-04-15T18:42:58.506 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64ce5ab:::benchmark_data_smithi159_162467_object73217:head# 2024-04-15T18:42:58.507 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64e6dca:::benchmark_data_smithi159_162467_object12470:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f64f7183:::benchmark_data_smithi159_162467_object65637:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6505646:::benchmark_data_smithi159_162467_object8666:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f651b6d9:::benchmark_data_smithi159_162467_object59671:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f651dac0:::benchmark_data_smithi159_162467_object23856:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f653c3cb:::benchmark_data_smithi159_162467_object16898:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f654e7d5:::benchmark_data_smithi159_162467_object45546:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6553378:::benchmark_data_smithi159_162467_object30759:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f655a9ec:::benchmark_data_smithi159_162467_object900:head# 2024-04-15T18:42:58.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f656eaaa:::benchmark_data_smithi159_162467_object27522:head# 2024-04-15T18:42:58.512 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:42:58.512 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65797a5:::benchmark_data_smithi159_162467_object11885:head# 2024-04-15T18:42:58.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65806bd:::benchmark_data_smithi159_162467_object44613:head# 2024-04-15T18:42:58.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f658cd1c:::benchmark_data_smithi159_162467_object70505:head# 2024-04-15T18:42:58.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6594f8a:::benchmark_data_smithi159_162467_object34767:head# 2024-04-15T18:42:58.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6598efa:::benchmark_data_smithi159_162467_object48725:head# 2024-04-15T18:42:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:58.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65e3c03:::benchmark_data_smithi159_162467_object53325:head# 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65ed08a:::benchmark_data_smithi159_162467_object46284:head# 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65f78e8:::benchmark_data_smithi159_162467_object9751:head# 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f65fc9d8:::benchmark_data_smithi159_162467_object17362:head# 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f660cda2:::benchmark_data_smithi159_162467_object58763:head# 2024-04-15T18:42:58.515 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6614fd6:::benchmark_data_smithi159_162467_object23103:head# 2024-04-15T18:42:58.516 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6615a50:::benchmark_data_smithi159_162467_object18663:head# 2024-04-15T18:42:58.516 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6618a5e:::benchmark_data_smithi159_162467_object69233:head# 2024-04-15T18:42:58.516 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6635208:::benchmark_data_smithi159_162467_object62757:head# 2024-04-15T18:42:58.516 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f664bee5:::benchmark_data_smithi159_162467_object11944:head# 2024-04-15T18:42:58.517 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f665ff3e:::benchmark_data_smithi159_162467_object46829:head# 2024-04-15T18:42:58.517 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6664146:::benchmark_data_smithi159_162467_object68948:head# 2024-04-15T18:42:58.518 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f666f629:::benchmark_data_smithi159_162467_object38137:head# 2024-04-15T18:42:58.518 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f66918e7:::benchmark_data_smithi159_162467_object68726:head# 2024-04-15T18:42:58.518 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6693ec7:::benchmark_data_smithi159_162467_object33362:head# 2024-04-15T18:42:58.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f669b62b:::benchmark_data_smithi159_162467_object67180:head# 2024-04-15T18:42:58.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f669c35b:::benchmark_data_smithi159_162467_object73577:head# 2024-04-15T18:42:58.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f66cbb60:::benchmark_data_smithi159_162467_object19554:head# 2024-04-15T18:42:58.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f66daed3:::benchmark_data_smithi159_162467_object29070:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f66e2da7:::benchmark_data_smithi159_162467_object14963:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6718f66:::benchmark_data_smithi159_162467_object39387:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f672771f:::benchmark_data_smithi159_162467_object23003:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67342ac:::benchmark_data_smithi159_162467_object3205:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67404c0:::benchmark_data_smithi159_162467_object64375:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67411d4:::benchmark_data_smithi159_162467_object65495:head# 2024-04-15T18:42:58.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6747f78:::benchmark_data_smithi159_162467_object24027:head# 2024-04-15T18:42:58.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6752c10:::benchmark_data_smithi159_162467_object4853:head# 2024-04-15T18:42:58.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f675fb55:::benchmark_data_smithi159_162467_object31206:head# 2024-04-15T18:42:58.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6764664:::benchmark_data_smithi159_162467_object39901:head# 2024-04-15T18:42:58.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f676c57c:::benchmark_data_smithi159_162467_object38115:head# 2024-04-15T18:42:58.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6770898:::benchmark_data_smithi159_162467_object65178:head# 2024-04-15T18:42:58.526 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6777d88:::benchmark_data_smithi159_162467_object6611:head# 2024-04-15T18:42:58.526 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f678c3f3:::benchmark_data_smithi159_162467_object51593:head# 2024-04-15T18:42:58.526 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6791bac:::benchmark_data_smithi159_162467_object19032:head# 2024-04-15T18:42:58.526 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67a1476:::benchmark_data_smithi159_162467_object29335:head# 2024-04-15T18:42:58.527 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67a2126:::benchmark_data_smithi159_162467_object71412:head# 2024-04-15T18:42:58.527 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67a83af:::benchmark_data_smithi159_162467_object45621:head# 2024-04-15T18:42:58.528 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67ca428:::benchmark_data_smithi159_162467_object55372:head# 2024-04-15T18:42:58.528 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67cb6a8:::benchmark_data_smithi159_162467_object19315:head# 2024-04-15T18:42:58.529 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67cbec2:::benchmark_data_smithi159_162467_object48580:head# 2024-04-15T18:42:58.529 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67ce64d:::benchmark_data_smithi159_162467_object40282:head# 2024-04-15T18:42:58.529 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67ceb1b:::benchmark_data_smithi159_162467_object57507:head# 2024-04-15T18:42:58.530 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67d01d3:::benchmark_data_smithi159_162467_object34616:head# 2024-04-15T18:42:58.530 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67dd41f:::benchmark_data_smithi159_162467_object50149:head# 2024-04-15T18:42:58.531 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67e21a2:::benchmark_data_smithi159_162467_object17311:head# 2024-04-15T18:42:58.531 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67e60ba:::benchmark_data_smithi159_162467_object27179:head# 2024-04-15T18:42:58.531 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67f0fa8:::benchmark_data_smithi159_162467_object61544:head# 2024-04-15T18:42:58.532 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f67f9e4f:::benchmark_data_smithi159_162467_object54768:head# 2024-04-15T18:42:58.532 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6800a35:::benchmark_data_smithi159_162467_object66799:head# 2024-04-15T18:42:58.533 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f680acfa:::benchmark_data_smithi159_162467_object66636:head# 2024-04-15T18:42:58.533 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f680b224:::benchmark_data_smithi159_162467_object29490:head# 2024-04-15T18:42:58.534 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68140af:::benchmark_data_smithi159_162467_object54071:head# 2024-04-15T18:42:58.534 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6818739:::benchmark_data_smithi159_162467_object40397:head# 2024-04-15T18:42:58.535 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6822ddf:::benchmark_data_smithi159_162467_object28402:head# 2024-04-15T18:42:58.535 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f682a366:::benchmark_data_smithi159_162467_object74279:head# 2024-04-15T18:42:58.535 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6854d67:::benchmark_data_smithi159_162467_object56953:head# 2024-04-15T18:42:58.536 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6863ac5:::benchmark_data_smithi159_162467_object70595:head# 2024-04-15T18:42:58.536 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6871fb4:::benchmark_data_smithi159_162467_object74602:head# 2024-04-15T18:42:58.536 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68721e5:::benchmark_data_smithi159_162467_object71709:head# 2024-04-15T18:42:58.537 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f687555c:::benchmark_data_smithi159_162467_object2986:head# 2024-04-15T18:42:58.537 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68783fa:::benchmark_data_smithi159_162467_object36686:head# 2024-04-15T18:42:58.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f687f539:::benchmark_data_smithi159_162467_object44159:head# 2024-04-15T18:42:58.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f688eef7:::benchmark_data_smithi159_162467_object45847:head# 2024-04-15T18:42:58.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f689a143:::benchmark_data_smithi159_162467_object13271:head# 2024-04-15T18:42:58.539 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68a0133:::benchmark_data_smithi159_162467_object53559:head# 2024-04-15T18:42:58.539 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68a1dcf:::benchmark_data_smithi159_162467_object23525:head# 2024-04-15T18:42:58.540 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68cc2f3:::benchmark_data_smithi159_162467_object3925:head# 2024-04-15T18:42:58.540 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68d545a:::benchmark_data_smithi159_162467_object521:head# 2024-04-15T18:42:58.541 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68ed53d:::benchmark_data_smithi159_162467_object9309:head# 2024-04-15T18:42:58.541 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68f1536:::benchmark_data_smithi159_162467_object74066:head# 2024-04-15T18:42:58.541 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68ff627:::benchmark_data_smithi159_162467_object9924:head# 2024-04-15T18:42:58.542 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f68ffed9:::benchmark_data_smithi159_162467_object3012:head# 2024-04-15T18:42:58.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69222fe:::benchmark_data_smithi159_162467_object46072:head# 2024-04-15T18:42:58.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69299d2:::benchmark_data_smithi159_162467_object28279:head# 2024-04-15T18:42:58.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f692e0d8:::benchmark_data_smithi159_162467_object39217:head# 2024-04-15T18:42:58.544 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6938252:::benchmark_data_smithi159_162467_object49461:head# 2024-04-15T18:42:58.544 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6954c50:::benchmark_data_smithi159_162467_object3872:head# 2024-04-15T18:42:58.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69561de:::benchmark_data_smithi159_162467_object56220:head# 2024-04-15T18:42:58.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f695cc3f:::benchmark_data_smithi159_162467_object27706:head# 2024-04-15T18:42:58.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6982ff2:::benchmark_data_smithi159_162467_object46022:head# 2024-04-15T18:42:58.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f698f2e8:::benchmark_data_smithi159_162467_object17553:head# 2024-04-15T18:42:58.546 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f699f4cd:::benchmark_data_smithi159_162467_object71730:head# 2024-04-15T18:42:58.546 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69a2f73:::benchmark_data_smithi159_162467_object47614:head# 2024-04-15T18:42:58.547 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69abc42:::benchmark_data_smithi159_162467_object32989:head# 2024-04-15T18:42:58.547 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69ae747:::benchmark_data_smithi159_162467_object50562:head# 2024-04-15T18:42:58.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69be367:::benchmark_data_smithi159_162467_object41965:head# 2024-04-15T18:42:58.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69c6fd6:::benchmark_data_smithi159_162467_object2132:head# 2024-04-15T18:42:58.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69e79c1:::benchmark_data_smithi159_162467_object73425:head# 2024-04-15T18:42:58.549 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f69fc67f:::benchmark_data_smithi159_162467_object17104:head# 2024-04-15T18:42:58.549 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a0e08b:::benchmark_data_smithi159_162467_object44746:head# 2024-04-15T18:42:58.550 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a20b2e:::benchmark_data_smithi159_162467_object42692:head# 2024-04-15T18:42:58.550 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a2c83f:::benchmark_data_smithi159_162467_object33690:head# 2024-04-15T18:42:58.551 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a44d1b:::benchmark_data_smithi159_162467_object53188:head# 2024-04-15T18:42:58.552 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a62fed:::benchmark_data_smithi159_162467_object68724:head# 2024-04-15T18:42:58.552 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a69b60:::benchmark_data_smithi159_162467_object28915:head# 2024-04-15T18:42:58.552 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a6bb41:::benchmark_data_smithi159_162467_object45756:head# 2024-04-15T18:42:58.552 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a74896:::benchmark_data_smithi159_162467_object12733:head# 2024-04-15T18:42:58.553 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a7ed89:::benchmark_data_smithi159_162467_object48827:head# 2024-04-15T18:42:58.553 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a98bbe:::benchmark_data_smithi159_162467_object27651:head# 2024-04-15T18:42:58.554 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6a9f3c0:::benchmark_data_smithi159_162467_object14073:head# 2024-04-15T18:42:58.554 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ab721d:::benchmark_data_smithi159_162467_object22310:head# 2024-04-15T18:42:58.555 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ac8b24:::benchmark_data_smithi159_162467_object50147:head# 2024-04-15T18:42:58.555 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ad28c9:::benchmark_data_smithi159_162467_object58742:head# 2024-04-15T18:42:58.555 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ad5529:::benchmark_data_smithi159_162467_object16461:head# 2024-04-15T18:42:58.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6adaf6a:::benchmark_data_smithi159_162467_object63701:head# 2024-04-15T18:42:58.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6adc81a:::benchmark_data_smithi159_162467_object39204:head# 2024-04-15T18:42:58.557 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6add549:::benchmark_data_smithi159_162467_object41732:head# 2024-04-15T18:42:58.557 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ae16db:::benchmark_data_smithi159_162467_object30246:head# 2024-04-15T18:42:58.557 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ae4e1e:::benchmark_data_smithi159_162467_object14714:head# 2024-04-15T18:42:58.558 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6af9b43:::benchmark_data_smithi159_162467_object69581:head# 2024-04-15T18:42:58.558 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b06c6b:::benchmark_data_smithi159_162467_object7044:head# 2024-04-15T18:42:58.559 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b0c902:::benchmark_data_smithi159_162467_object5854:head# 2024-04-15T18:42:58.559 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b285ae:::benchmark_data_smithi159_162467_object20149:head# 2024-04-15T18:42:58.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b37d44:::benchmark_data_smithi159_162467_object17493:head# 2024-04-15T18:42:58.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b4172c:::benchmark_data_smithi159_162467_object61085:head# 2024-04-15T18:42:58.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b6a845:::benchmark_data_smithi159_162467_object55347:head# 2024-04-15T18:42:58.561 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b8309e:::benchmark_data_smithi159_162467_object44205:head# 2024-04-15T18:42:58.561 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b8e2d6:::benchmark_data_smithi159_162467_object64202:head# 2024-04-15T18:42:58.561 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6b940fc:::benchmark_data_smithi159_162467_object8154:head# 2024-04-15T18:42:58.563 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bb6918:::benchmark_data_smithi159_162467_object3416:head# 2024-04-15T18:42:58.563 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bc31db:::benchmark_data_smithi159_162467_object55864:head# 2024-04-15T18:42:58.563 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bc759c:::benchmark_data_smithi159_162467_object54434:head# 2024-04-15T18:42:58.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bcba9f:::benchmark_data_smithi159_162467_object60369:head# 2024-04-15T18:42:58.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bd4f70:::benchmark_data_smithi159_162467_object11760:head# 2024-04-15T18:42:58.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6bddeb7:::benchmark_data_smithi159_162467_object56254:head# 2024-04-15T18:42:58.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c0d514:::benchmark_data_smithi159_162467_object28580:head# 2024-04-15T18:42:58.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c32efc:::benchmark_data_smithi159_162467_object69740:head# 2024-04-15T18:42:58.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c4e65d:::benchmark_data_smithi159_162467_object49330:head# 2024-04-15T18:42:58.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c5b97a:::benchmark_data_smithi159_162467_object57714:head# 2024-04-15T18:42:58.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c6a392:::benchmark_data_smithi159_162467_object71687:head# 2024-04-15T18:42:58.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c73d70:::benchmark_data_smithi159_162467_object31216:head# 2024-04-15T18:42:58.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c8886e:::benchmark_data_smithi159_162467_object12490:head# 2024-04-15T18:42:58.568 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c892ce:::benchmark_data_smithi159_162467_object2201:head# 2024-04-15T18:42:58.568 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c8e0f5:::benchmark_data_smithi159_162467_object54892:head# 2024-04-15T18:42:58.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c90454:::benchmark_data_smithi159_162467_object33818:head# 2024-04-15T18:42:58.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6c925c0:::benchmark_data_smithi159_162467_object464:head# 2024-04-15T18:42:58.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6cbf03a:::benchmark_data_smithi159_162467_object69920:head# 2024-04-15T18:42:58.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6cd8404:::benchmark_data_smithi159_162467_object29754:head# 2024-04-15T18:42:58.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6cdd3e1:::benchmark_data_smithi159_162467_object72117:head# 2024-04-15T18:42:58.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ce1191:::benchmark_data_smithi159_162467_object8900:head# 2024-04-15T18:42:58.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ce65cd:::benchmark_data_smithi159_162467_object64904:head# 2024-04-15T18:42:58.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6cfcb7d:::benchmark_data_smithi159_162467_object35740:head# 2024-04-15T18:42:58.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d04383:::benchmark_data_smithi159_162467_object74744:head# 2024-04-15T18:42:58.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d0ef25:::benchmark_data_smithi159_162467_object40917:head# 2024-04-15T18:42:58.573 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d15394:::benchmark_data_smithi159_162467_object1353:head# 2024-04-15T18:42:58.573 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d184f3:::benchmark_data_smithi159_162467_object74238:head# 2024-04-15T18:42:58.574 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d25607:::benchmark_data_smithi159_162467_object44481:head# 2024-04-15T18:42:58.574 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d257e4:::benchmark_data_smithi159_162467_object4363:head# 2024-04-15T18:42:58.575 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d27f56:::benchmark_data_smithi159_162467_object48622:head# 2024-04-15T18:42:58.575 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d29367:::benchmark_data_smithi159_162467_object11149:head# 2024-04-15T18:42:58.575 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d3897e:::benchmark_data_smithi159_162467_object42172:head# 2024-04-15T18:42:58.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d45052:::benchmark_data_smithi159_162467_object32136:head# 2024-04-15T18:42:58.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d5d481:::benchmark_data_smithi159_162467_object67751:head# 2024-04-15T18:42:58.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d6d039:::benchmark_data_smithi159_162467_object23825:head# 2024-04-15T18:42:58.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d78b40:::benchmark_data_smithi159_162467_object37591:head# 2024-04-15T18:42:58.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d79224:::benchmark_data_smithi159_162467_object5553:head# 2024-04-15T18:42:58.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d8173d:::benchmark_data_smithi159_162467_object71872:head# 2024-04-15T18:42:58.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d8b2b0:::benchmark_data_smithi159_162467_object50618:head# 2024-04-15T18:42:58.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d95ce5:::benchmark_data_smithi159_162467_object68733:head# 2024-04-15T18:42:58.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6d9a8dc:::benchmark_data_smithi159_162467_object62015:head# 2024-04-15T18:42:58.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6da0b0d:::benchmark_data_smithi159_162467_object3474:head# 2024-04-15T18:42:58.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6db668c:::benchmark_data_smithi159_162467_object22337:head# 2024-04-15T18:42:58.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6db992d:::benchmark_data_smithi159_162467_object28736:head# 2024-04-15T18:42:58.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6dbca0c:::benchmark_data_smithi159_162467_object35512:head# 2024-04-15T18:42:58.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6dc7824:::benchmark_data_smithi159_162467_object21840:head# 2024-04-15T18:42:58.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6dd7f7f:::benchmark_data_smithi159_162467_object72901:head# 2024-04-15T18:42:58.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ddbe39:::benchmark_data_smithi159_162467_object67691:head# 2024-04-15T18:42:58.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6de8927:::benchmark_data_smithi159_162467_object33442:head# 2024-04-15T18:42:58.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6e30a79:::benchmark_data_smithi159_162467_object14430:head# 2024-04-15T18:42:58.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6e34918:::benchmark_data_smithi159_162467_object60848:head# 2024-04-15T18:42:58.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6e6c7a0:::benchmark_data_smithi159_162467_object18554:head# 2024-04-15T18:42:58.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6e8773c:::benchmark_data_smithi159_162467_object69347:head# 2024-04-15T18:42:58.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6e89137:::benchmark_data_smithi159_162467_object66332:head# 2024-04-15T18:42:58.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ea1984:::benchmark_data_smithi159_162467_object28968:head# 2024-04-15T18:42:58.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ece448:::benchmark_data_smithi159_162467_object19860:head# 2024-04-15T18:42:58.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f28ef3:::benchmark_data_smithi159_162467_object22655:head# 2024-04-15T18:42:58.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f41838:::benchmark_data_smithi159_162467_object51005:head# 2024-04-15T18:42:58.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f542ea:::benchmark_data_smithi159_162467_object17113:head# 2024-04-15T18:42:58.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f5c438:::benchmark_data_smithi159_162467_object1677:head# 2024-04-15T18:42:58.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f5d5b6:::benchmark_data_smithi159_162467_object4910:head# 2024-04-15T18:42:58.589 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f6726c:::benchmark_data_smithi159_162467_object61960:head# 2024-04-15T18:42:58.589 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f70dfc:::benchmark_data_smithi159_162467_object12369:head# 2024-04-15T18:42:58.589 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f7fa74:::benchmark_data_smithi159_162467_object8357:head# 2024-04-15T18:42:58.590 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f94ba7:::benchmark_data_smithi159_162467_object31308:head# 2024-04-15T18:42:58.590 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6f96ffa:::benchmark_data_smithi159_162467_object66779:head# 2024-04-15T18:42:58.591 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fb5e36:::benchmark_data_smithi159_162467_object50219:head# 2024-04-15T18:42:58.591 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fb694d:::benchmark_data_smithi159_162467_object17448:head# 2024-04-15T18:42:58.591 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fb9dde:::benchmark_data_smithi159_162467_object72832:head# 2024-04-15T18:42:58.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fca01e:::benchmark_data_smithi159_162467_object31725:head# 2024-04-15T18:42:58.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fd037b:::benchmark_data_smithi159_162467_object28262:head# 2024-04-15T18:42:58.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fe32ea:::benchmark_data_smithi159_162467_object40127:head# 2024-04-15T18:42:58.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fe477e:::benchmark_data_smithi159_162467_object20440:head# 2024-04-15T18:42:58.594 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6fef370:::benchmark_data_smithi159_162467_object45865:head# 2024-04-15T18:42:58.594 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f6ff071f:::benchmark_data_smithi159_162467_object68388:head# 2024-04-15T18:42:58.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7004f70:::benchmark_data_smithi159_162467_object2653:head# 2024-04-15T18:42:58.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f700c0b9:::benchmark_data_smithi159_162467_object54631:head# 2024-04-15T18:42:58.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f700d268:::benchmark_data_smithi159_162467_object20976:head# 2024-04-15T18:42:58.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7036d06:::benchmark_data_smithi159_162467_object57236:head# 2024-04-15T18:42:58.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7040744:::benchmark_data_smithi159_162467_object8651:head# 2024-04-15T18:42:58.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7057f0a:::benchmark_data_smithi159_162467_object50177:head# 2024-04-15T18:42:58.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f706227a:::benchmark_data_smithi159_162467_object27946:head# 2024-04-15T18:42:58.598 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f706b8e8:::benchmark_data_smithi159_162467_object27201:head# 2024-04-15T18:42:58.599 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7074187:::benchmark_data_smithi159_162467_object1042:head# 2024-04-15T18:42:58.599 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f70752a4:::benchmark_data_smithi159_162467_object68575:head# 2024-04-15T18:42:58.600 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7077d6c:::benchmark_data_smithi159_162467_object24154:head# 2024-04-15T18:42:58.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f708597d:::benchmark_data_smithi159_162467_object28609:head# 2024-04-15T18:42:58.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f708e9d5:::benchmark_data_smithi159_162467_object65025:head# 2024-04-15T18:42:58.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f709e217:::benchmark_data_smithi159_162467_object50345:head# 2024-04-15T18:42:58.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f70ddf16:::benchmark_data_smithi159_162467_object22543:head# 2024-04-15T18:42:58.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f70e5f2e:::benchmark_data_smithi159_162467_object42312:head# 2024-04-15T18:42:58.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f70f2f72:::benchmark_data_smithi159_162467_object73360:head# 2024-04-15T18:42:58.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f70fca66:::benchmark_data_smithi159_162467_object47615:head# 2024-04-15T18:42:58.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f710eb73:::benchmark_data_smithi159_162467_object13324:head# 2024-04-15T18:42:58.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7125204:::benchmark_data_smithi159_162467_object8397:head# 2024-04-15T18:42:58.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f712593f:::benchmark_data_smithi159_162467_object17889:head# 2024-04-15T18:42:58.607 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71331a1:::benchmark_data_smithi159_162467_object8385:head# 2024-04-15T18:42:58.607 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f713ecb2:::benchmark_data_smithi159_162467_object7665:head# 2024-04-15T18:42:58.607 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7163c4e:::benchmark_data_smithi159_162467_object61227:head# 2024-04-15T18:42:58.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f716f775:::benchmark_data_smithi159_162467_object47399:head# 2024-04-15T18:42:58.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7175c4e:::benchmark_data_smithi159_162467_object45662:head# 2024-04-15T18:42:58.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7190be7:::benchmark_data_smithi159_162467_object69005:head# 2024-04-15T18:42:58.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7191e5b:::benchmark_data_smithi159_162467_object19482:head# 2024-04-15T18:42:58.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f719668a:::benchmark_data_smithi159_162467_object35992:head# 2024-04-15T18:42:58.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71b1ebb:::benchmark_data_smithi159_162467_object59731:head# 2024-04-15T18:42:58.611 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71d621d:::benchmark_data_smithi159_162467_object22029:head# 2024-04-15T18:42:58.611 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71d7b56:::benchmark_data_smithi159_162467_object59564:head# 2024-04-15T18:42:58.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71dbb23:::benchmark_data_smithi159_162467_object15497:head# 2024-04-15T18:42:58.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71ddfe8:::benchmark_data_smithi159_162467_object63681:head# 2024-04-15T18:42:58.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71edfa1:::benchmark_data_smithi159_162467_object33862:head# 2024-04-15T18:42:58.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71f085b:::benchmark_data_smithi159_162467_object7172:head# 2024-04-15T18:42:58.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71f534d:::benchmark_data_smithi159_162467_object58063:head# 2024-04-15T18:42:58.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71fa6b1:::benchmark_data_smithi159_162467_object40298:head# 2024-04-15T18:42:58.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f71fce7b:::benchmark_data_smithi159_162467_object1525:head# 2024-04-15T18:42:58.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:42:58.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7211cfe:::benchmark_data_smithi159_162467_object45414:head# 2024-04-15T18:42:58.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7213c24:::benchmark_data_smithi159_162467_object18690:head# 2024-04-15T18:42:58.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7214563:::benchmark_data_smithi159_162467_object36590:head# 2024-04-15T18:42:58.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f721979a:::benchmark_data_smithi159_162467_object69764:head# 2024-04-15T18:42:58.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7228253:::benchmark_data_smithi159_162467_object18353:head# 2024-04-15T18:42:58.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f722af15:::benchmark_data_smithi159_162467_object61366:head# 2024-04-15T18:42:58.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f722eb0d:::benchmark_data_smithi159_162467_object25584:head# 2024-04-15T18:42:58.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7231b7b:::benchmark_data_smithi159_162467_object16983:head# 2024-04-15T18:42:58.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7234e6b:::benchmark_data_smithi159_162467_object66172:head# 2024-04-15T18:42:58.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f723ce06:::benchmark_data_smithi159_162467_object42491:head# 2024-04-15T18:42:58.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f724d3e9:::benchmark_data_smithi159_162467_object55005:head# 2024-04-15T18:42:58.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f727341e:::benchmark_data_smithi159_162467_object71623:head# 2024-04-15T18:42:58.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f728828d:::benchmark_data_smithi159_162467_object16767:head# 2024-04-15T18:42:58.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f728b3c7:::benchmark_data_smithi159_162467_object29219:head# 2024-04-15T18:42:58.623 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72b4c8b:::benchmark_data_smithi159_162467_object36725:head# 2024-04-15T18:42:58.623 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72c1aff:::benchmark_data_smithi159_162467_object2197:head# 2024-04-15T18:42:58.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72c31a0:::benchmark_data_smithi159_162467_object17220:head# 2024-04-15T18:42:58.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72e2811:::benchmark_data_smithi159_162467_object19757:head# 2024-04-15T18:42:58.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72e49c0:::benchmark_data_smithi159_162467_object14192:head# 2024-04-15T18:42:58.625 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72e66fb:::benchmark_data_smithi159_162467_object31565:head# 2024-04-15T18:42:58.625 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f72f4d94:::benchmark_data_smithi159_162467_object58946:head# 2024-04-15T18:42:58.626 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7302337:::benchmark_data_smithi159_162467_object57117:head# 2024-04-15T18:42:58.626 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f730e2ae:::benchmark_data_smithi159_162467_object50510:head# 2024-04-15T18:42:58.626 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7346796:::benchmark_data_smithi159_162467_object12295:head# 2024-04-15T18:42:58.627 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f734b245:::benchmark_data_smithi159_162467_object39633:head# 2024-04-15T18:42:58.627 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73527f4:::benchmark_data_smithi159_162467_object7392:head# 2024-04-15T18:42:58.628 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f735ea82:::benchmark_data_smithi159_162467_object69455:head# 2024-04-15T18:42:58.628 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7378151:::benchmark_data_smithi159_162467_object30967:head# 2024-04-15T18:42:58.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7394440:::benchmark_data_smithi159_162467_object46951:head# 2024-04-15T18:42:58.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7397319:::benchmark_data_smithi159_162467_object62253:head# 2024-04-15T18:42:58.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73dc20e:::benchmark_data_smithi159_162467_object31033:head# 2024-04-15T18:42:58.632 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73decc4:::benchmark_data_smithi159_162467_object30641:head# 2024-04-15T18:42:58.633 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73e4145:::benchmark_data_smithi159_162467_object38353:head# 2024-04-15T18:42:58.633 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73eb084:::benchmark_data_smithi159_162467_object57824:head# 2024-04-15T18:42:58.635 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73ef3b7:::benchmark_data_smithi159_162467_object35402:head# 2024-04-15T18:42:58.636 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f73fb50c:::benchmark_data_smithi159_162467_object23894:head# 2024-04-15T18:42:58.637 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74027af:::benchmark_data_smithi159_162467_object69721:head# 2024-04-15T18:42:58.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74115c8:::benchmark_data_smithi159_162467_object65459:head# 2024-04-15T18:42:58.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7411641:::benchmark_data_smithi159_162467_object41784:head# 2024-04-15T18:42:58.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f741967a:::benchmark_data_smithi159_162467_object39153:head# 2024-04-15T18:42:58.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f741e07e:::benchmark_data_smithi159_162467_object48731:head# 2024-04-15T18:42:58.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7434145:::benchmark_data_smithi159_162467_object26424:head# 2024-04-15T18:42:58.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f743420f:::benchmark_data_smithi159_162467_object63144:head# 2024-04-15T18:42:58.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7435527:::benchmark_data_smithi159_162467_object52765:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f743697e:::benchmark_data_smithi159_162467_object6172:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74448d2:::benchmark_data_smithi159_162467_object52309:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74453a7:::benchmark_data_smithi159_162467_object13033:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f744a6ab:::benchmark_data_smithi159_162467_object9125:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f744b97b:::benchmark_data_smithi159_162467_object29817:head# 2024-04-15T18:42:58.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7451446:::benchmark_data_smithi159_162467_object830:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7451c30:::benchmark_data_smithi159_162467_object21987:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f747a739:::benchmark_data_smithi159_162467_object41196:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f747aa1e:::benchmark_data_smithi159_162467_object1850:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7487d9f:::benchmark_data_smithi159_162467_object29092:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74a57fe:::benchmark_data_smithi159_162467_object42965:head# 2024-04-15T18:42:58.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74b33a8:::benchmark_data_smithi159_162467_object14853:head# 2024-04-15T18:42:58.652 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74bab51:::benchmark_data_smithi159_162467_object71677:head# 2024-04-15T18:42:58.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74cc811:::benchmark_data_smithi159_162467_object48376:head# 2024-04-15T18:42:58.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74d9cb5:::benchmark_data_smithi159_162467_object32410:head# 2024-04-15T18:42:58.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74da7cc:::benchmark_data_smithi159_162467_object26253:head# 2024-04-15T18:42:58.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74dbde1:::benchmark_data_smithi159_162467_object52303:head# 2024-04-15T18:42:58.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74e4b0d:::benchmark_data_smithi159_162467_object9478:head# 2024-04-15T18:42:58.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f74f8f28:::benchmark_data_smithi159_162467_object65329:head# 2024-04-15T18:42:58.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7500736:::benchmark_data_smithi159_162467_object3874:head# 2024-04-15T18:42:58.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7505699:::benchmark_data_smithi159_162467_object38884:head# 2024-04-15T18:42:58.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7506d21:::benchmark_data_smithi159_162467_object12770:head# 2024-04-15T18:42:58.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f750a90b:::benchmark_data_smithi159_162467_object45978:head# 2024-04-15T18:42:58.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f751cc19:::benchmark_data_smithi159_162467_object21002:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7522a9a:::benchmark_data_smithi159_162467_object36180:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f752e33e:::benchmark_data_smithi159_162467_object1619:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7556ca1:::benchmark_data_smithi159_162467_object4268:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f755ec0f:::benchmark_data_smithi159_162467_object29024:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7568856:::benchmark_data_smithi159_162467_object26019:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7584ea3:::benchmark_data_smithi159_162467_object35379:head# 2024-04-15T18:42:58.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f758cfa5:::benchmark_data_smithi159_162467_object29526:head# 2024-04-15T18:42:58.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f758fbbd:::benchmark_data_smithi159_162467_object15096:head# 2024-04-15T18:42:58.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75957a6:::benchmark_data_smithi159_162467_object52864:head# 2024-04-15T18:42:58.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75af21d:::benchmark_data_smithi159_162467_object69707:head# 2024-04-15T18:42:58.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75ba727:::benchmark_data_smithi159_162467_object60128:head# 2024-04-15T18:42:58.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75c78a0:::benchmark_data_smithi159_162467_object27607:head# 2024-04-15T18:42:58.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75cb8e9:::benchmark_data_smithi159_162467_object23146:head# 2024-04-15T18:42:58.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f75dec8d:::benchmark_data_smithi159_162467_object23351:head# 2024-04-15T18:42:58.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7613b0f:::benchmark_data_smithi159_162467_object39932:head# 2024-04-15T18:42:58.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f761e626:::benchmark_data_smithi159_162467_object72942:head# 2024-04-15T18:42:58.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f762b43b:::benchmark_data_smithi159_162467_object24694:head# 2024-04-15T18:42:58.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f765a7be:::benchmark_data_smithi159_162467_object67647:head# 2024-04-15T18:42:58.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f765fec7:::benchmark_data_smithi159_162467_object18738:head# 2024-04-15T18:42:58.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7666f50:::benchmark_data_smithi159_162467_object61972:head# 2024-04-15T18:42:58.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f766ae00:::benchmark_data_smithi159_162467_object13125:head# 2024-04-15T18:42:58.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f766ed31:::benchmark_data_smithi159_162467_object3512:head# 2024-04-15T18:42:58.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f767af41:::benchmark_data_smithi159_162467_object40257:head# 2024-04-15T18:42:58.666 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f768c890:::benchmark_data_smithi159_162467_object41506:head# 2024-04-15T18:42:58.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7695730:::benchmark_data_smithi159_162467_object63843:head# 2024-04-15T18:42:58.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76a8739:::benchmark_data_smithi159_162467_object51249:head# 2024-04-15T18:42:58.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76afbea:::benchmark_data_smithi159_162467_object23738:head# 2024-04-15T18:42:58.669 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76b00df:::benchmark_data_smithi159_162467_object1231:head# 2024-04-15T18:42:58.670 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76c31d2:::benchmark_data_smithi159_162467_object7809:head# 2024-04-15T18:42:58.670 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76c8ca7:::benchmark_data_smithi159_162467_object1494:head# 2024-04-15T18:42:58.671 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76deb54:::benchmark_data_smithi159_162467_object70685:head# 2024-04-15T18:42:58.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76e128b:::benchmark_data_smithi159_162467_object74145:head# 2024-04-15T18:42:58.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76f337f:::benchmark_data_smithi159_162467_object40960:head# 2024-04-15T18:42:58.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f76fa617:::benchmark_data_smithi159_162467_object33089:head# 2024-04-15T18:42:58.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77000c5:::benchmark_data_smithi159_162467_object38909:head# 2024-04-15T18:42:58.673 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f770e5af:::benchmark_data_smithi159_162467_object57928:head# 2024-04-15T18:42:58.674 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:58.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7716886:::benchmark_data_smithi159_162467_object31185:head# 2024-04-15T18:42:58.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f771efd0:::benchmark_data_smithi159_162467_object65073:head# 2024-04-15T18:42:58.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f771f62a:::benchmark_data_smithi159_162467_object25118:head# 2024-04-15T18:42:58.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f772cf5d:::benchmark_data_smithi159_162467_object62799:head# 2024-04-15T18:42:58.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7737888:::benchmark_data_smithi159_162467_object5705:head# 2024-04-15T18:42:58.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7751367:::benchmark_data_smithi159_162467_object21951:head# 2024-04-15T18:42:58.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77515b3:::benchmark_data_smithi159_162467_object61017:head# 2024-04-15T18:42:58.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f775a163:::benchmark_data_smithi159_162467_object60155:head# 2024-04-15T18:42:58.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f776955e:::benchmark_data_smithi159_162467_object22526:head# 2024-04-15T18:42:58.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f778abf4:::benchmark_data_smithi159_162467_object59368:head# 2024-04-15T18:42:58.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77aa13c:::benchmark_data_smithi159_162467_object50183:head# 2024-04-15T18:42:58.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77b7dcd:::benchmark_data_smithi159_162467_object30406:head# 2024-04-15T18:42:58.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77c43f0:::benchmark_data_smithi159_162467_object22668:head# 2024-04-15T18:42:58.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77d251a:::benchmark_data_smithi159_162467_object36915:head# 2024-04-15T18:42:58.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f77f85bb:::benchmark_data_smithi159_162467_object13259:head# 2024-04-15T18:42:58.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f780de76:::benchmark_data_smithi159_162467_object11977:head# 2024-04-15T18:42:58.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7816044:::benchmark_data_smithi159_162467_object60245:head# 2024-04-15T18:42:58.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f781e015:::benchmark_data_smithi159_162467_object25513:head# 2024-04-15T18:42:58.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f781e8d8:::benchmark_data_smithi159_162467_object548:head# 2024-04-15T18:42:58.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7820e51:::benchmark_data_smithi159_162467_object32083:head# 2024-04-15T18:42:58.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f782ed9a:::benchmark_data_smithi159_162467_object39331:head# 2024-04-15T18:42:58.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7831d8b:::benchmark_data_smithi159_162467_object22458:head# 2024-04-15T18:42:58.684 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7833944:::benchmark_data_smithi159_162467_object2095:head# 2024-04-15T18:42:58.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f783bff9:::benchmark_data_smithi159_162467_object5578:head# 2024-04-15T18:42:58.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f78423ff:::benchmark_data_smithi159_162467_object67415:head# 2024-04-15T18:42:58.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f78438ed:::benchmark_data_smithi159_162467_object53809:head# 2024-04-15T18:42:58.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7869127:::benchmark_data_smithi159_162467_object69258:head# 2024-04-15T18:42:58.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7871cec:::benchmark_data_smithi159_162467_object341:head# 2024-04-15T18:42:58.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f787de91:::benchmark_data_smithi159_162467_object32524:head# 2024-04-15T18:42:58.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7880d02:::benchmark_data_smithi159_162467_object48159:head# 2024-04-15T18:42:58.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f788c724:::benchmark_data_smithi159_162467_object73935:head# 2024-04-15T18:42:58.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7891082:::benchmark_data_smithi159_162467_object28462:head# 2024-04-15T18:42:58.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f78bad19:::benchmark_data_smithi159_162467_object26023:head# 2024-04-15T18:42:58.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f78d82af:::benchmark_data_smithi159_162467_object21393:head# 2024-04-15T18:42:58.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f78eb5f6:::benchmark_data_smithi159_162467_object57561:head# 2024-04-15T18:42:58.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7944087:::benchmark_data_smithi159_162467_object52209:head# 2024-04-15T18:42:58.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79505a8:::benchmark_data_smithi159_162467_object14372:head# 2024-04-15T18:42:58.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7958aab:::benchmark_data_smithi159_162467_object10281:head# 2024-04-15T18:42:58.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f795da8d:::benchmark_data_smithi159_162467_object2453:head# 2024-04-15T18:42:58.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f795f356:::benchmark_data_smithi159_162467_object5997:head# 2024-04-15T18:42:58.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7960ef5:::benchmark_data_smithi159_162467_object52725:head# 2024-04-15T18:42:58.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79629b9:::benchmark_data_smithi159_162467_object23515:head# 2024-04-15T18:42:58.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f796d936:::benchmark_data_smithi159_162467_object38480:head# 2024-04-15T18:42:58.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79790b9:::benchmark_data_smithi159_162467_object70529:head# 2024-04-15T18:42:58.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79817e8:::benchmark_data_smithi159_162467_object18978:head# 2024-04-15T18:42:58.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7995947:::benchmark_data_smithi159_162467_object5001:head# 2024-04-15T18:42:58.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79a7fc8:::benchmark_data_smithi159_162467_object45503:head# 2024-04-15T18:42:58.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79b1894:::benchmark_data_smithi159_162467_object62155:head# 2024-04-15T18:42:58.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79b365f:::benchmark_data_smithi159_162467_object48140:head# 2024-04-15T18:42:58.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79ca467:::benchmark_data_smithi159_162467_object54609:head# 2024-04-15T18:42:58.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79ccf53:::benchmark_data_smithi159_162467_object18804:head# 2024-04-15T18:42:58.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79cede3:::benchmark_data_smithi159_162467_object62148:head# 2024-04-15T18:42:58.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79d7835:::benchmark_data_smithi159_162467_object31288:head# 2024-04-15T18:42:58.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79ea405:::benchmark_data_smithi159_162467_object20252:head# 2024-04-15T18:42:58.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f79fd821:::benchmark_data_smithi159_162467_object30120:head# 2024-04-15T18:42:58.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a0ad30:::benchmark_data_smithi159_162467_object9989:head# 2024-04-15T18:42:58.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a0de99:::benchmark_data_smithi159_162467_object27697:head# 2024-04-15T18:42:58.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a0fd04:::benchmark_data_smithi159_162467_object11119:head# 2024-04-15T18:42:58.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a15ece:::benchmark_data_smithi159_162467_object21591:head# 2024-04-15T18:42:58.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a28e0f:::benchmark_data_smithi159_162467_object64925:head# 2024-04-15T18:42:58.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a3e04c:::benchmark_data_smithi159_162467_object11322:head# 2024-04-15T18:42:58.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a5005f:::benchmark_data_smithi159_162467_object60730:head# 2024-04-15T18:42:58.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a56e1b:::benchmark_data_smithi159_162467_object27406:head# 2024-04-15T18:42:58.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a5cb8c:::benchmark_data_smithi159_162467_object33542:head# 2024-04-15T18:42:58.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a6226e:::benchmark_data_smithi159_162467_object62170:head# 2024-04-15T18:42:58.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a6e7ac:::benchmark_data_smithi159_162467_object62541:head# 2024-04-15T18:42:58.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a784d6:::benchmark_data_smithi159_162467_object57410:head# 2024-04-15T18:42:58.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a7a0cb:::benchmark_data_smithi159_162467_object43383:head# 2024-04-15T18:42:58.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a7fe7f:::benchmark_data_smithi159_162467_object33899:head# 2024-04-15T18:42:58.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7a8afb2:::benchmark_data_smithi159_162467_object26587:head# 2024-04-15T18:42:58.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7aa2aca:::benchmark_data_smithi159_162467_object55249:head# 2024-04-15T18:42:58.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7aa907e:::benchmark_data_smithi159_162467_object68673:head# 2024-04-15T18:42:58.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7abfa4d:::benchmark_data_smithi159_162467_object22082:head# 2024-04-15T18:42:58.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b06216:::benchmark_data_smithi159_162467_object1941:head# 2024-04-15T18:42:58.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b0757c:::benchmark_data_smithi159_162467_object34041:head# 2024-04-15T18:42:58.708 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:42:58.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b0ae63:::benchmark_data_smithi159_162467_object39167:head# 2024-04-15T18:42:58.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b0efb8:::benchmark_data_smithi159_162467_object45142:head# 2024-04-15T18:42:58.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b1c98b:::benchmark_data_smithi159_162467_object44415:head# 2024-04-15T18:42:58.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b29f74:::benchmark_data_smithi159_162467_object47657:head# 2024-04-15T18:42:58.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b2a5ad:::benchmark_data_smithi159_162467_object32269:head# 2024-04-15T18:42:58.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b354ba:::benchmark_data_smithi159_162467_object43256:head# 2024-04-15T18:42:58.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b57062:::benchmark_data_smithi159_162467_object11179:head# 2024-04-15T18:42:58.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b72e84:::benchmark_data_smithi159_162467_object547:head# 2024-04-15T18:42:58.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b73c22:::benchmark_data_smithi159_162467_object15205:head# 2024-04-15T18:42:58.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b74c78:::benchmark_data_smithi159_162467_object64223:head# 2024-04-15T18:42:58.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b754c5:::benchmark_data_smithi159_162467_object9481:head# 2024-04-15T18:42:58.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7b7ef5a:::benchmark_data_smithi159_162467_object61554:head# 2024-04-15T18:42:58.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bb878f:::benchmark_data_smithi159_162467_object8153:head# 2024-04-15T18:42:58.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bb8bba:::benchmark_data_smithi159_162467_object49207:head# 2024-04-15T18:42:58.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bbc9cf:::benchmark_data_smithi159_162467_object10685:head# 2024-04-15T18:42:58.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bc7aa2:::benchmark_data_smithi159_162467_object60604:head# 2024-04-15T18:42:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:42:58.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bd998c:::benchmark_data_smithi159_162467_object18892:head# 2024-04-15T18:42:58.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7be409e:::benchmark_data_smithi159_162467_object31129:head# 2024-04-15T18:42:58.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:42:58.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bf1ff4:::benchmark_data_smithi159_162467_object1516:head# 2024-04-15T18:42:58.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bf4df1:::benchmark_data_smithi159_162467_object46668:head# 2024-04-15T18:42:58.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bfa512:::benchmark_data_smithi159_162467_object2100:head# 2024-04-15T18:42:58.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7bffe88:::benchmark_data_smithi159_162467_object50327:head# 2024-04-15T18:42:58.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c2077a:::benchmark_data_smithi159_162467_object65034:head# 2024-04-15T18:42:58.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c231ac:::benchmark_data_smithi159_162467_object4364:head# 2024-04-15T18:42:58.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c3bfd2:::benchmark_data_smithi159_162467_object21828:head# 2024-04-15T18:42:58.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c3e458:::benchmark_data_smithi159_162467_object33945:head# 2024-04-15T18:42:58.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c40f70:::benchmark_data_smithi159_162467_object50318:head# 2024-04-15T18:42:58.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c5f4da:::benchmark_data_smithi159_162467_object64916:head# 2024-04-15T18:42:58.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c6af89:::benchmark_data_smithi159_162467_object40856:head# 2024-04-15T18:42:58.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c82e50:::benchmark_data_smithi159_162467_object62502:head# 2024-04-15T18:42:58.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c8ea5e:::benchmark_data_smithi159_162467_object29395:head# 2024-04-15T18:42:58.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7c959f0:::benchmark_data_smithi159_162467_object57771:head# 2024-04-15T18:42:58.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ca7af8:::benchmark_data_smithi159_162467_object39995:head# 2024-04-15T18:42:58.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7cd72e0:::benchmark_data_smithi159_162467_object28837:head# 2024-04-15T18:42:58.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ce1274:::benchmark_data_smithi159_162467_object21557:head# 2024-04-15T18:42:58.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7cf340e:::benchmark_data_smithi159_162467_object53076:head# 2024-04-15T18:42:58.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7cfe61c:::benchmark_data_smithi159_162467_object43300:head# 2024-04-15T18:42:58.724 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d0119d:::benchmark_data_smithi159_162467_object69748:head# 2024-04-15T18:42:58.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d1ebf5:::benchmark_data_smithi159_162467_object30339:head# 2024-04-15T18:42:58.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d22118:::benchmark_data_smithi159_162467_object34256:head# 2024-04-15T18:42:58.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d554e1:::benchmark_data_smithi159_162467_object50191:head# 2024-04-15T18:42:58.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d83f20:::benchmark_data_smithi159_162467_object51169:head# 2024-04-15T18:42:58.728 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d848d3:::benchmark_data_smithi159_162467_object26679:head# 2024-04-15T18:42:58.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7d84fc9:::benchmark_data_smithi159_162467_object20466:head# 2024-04-15T18:42:58.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7daa57b:::benchmark_data_smithi159_162467_object18944:head# 2024-04-15T18:42:58.732 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7dac0c1:::benchmark_data_smithi159_162467_object60622:head# 2024-04-15T18:42:58.732 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7dc5475:::benchmark_data_smithi159_162467_object28540:head# 2024-04-15T18:42:58.733 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7dcbb0c:::benchmark_data_smithi159_162467_object20457:head# 2024-04-15T18:42:58.733 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7dcea0b:::benchmark_data_smithi159_162467_object20868:head# 2024-04-15T18:42:58.733 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7dd7bd3:::benchmark_data_smithi159_162467_object72011:head# 2024-04-15T18:42:58.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7de0347:::benchmark_data_smithi159_162467_object14124:head# 2024-04-15T18:42:58.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7de6824:::benchmark_data_smithi159_162467_object42713:head# 2024-04-15T18:42:58.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e0906b:::benchmark_data_smithi159_162467_object30410:head# 2024-04-15T18:42:58.735 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e0d4bf:::benchmark_data_smithi159_162467_object23881:head# 2024-04-15T18:42:58.735 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e139b9:::benchmark_data_smithi159_162467_object1848:head# 2024-04-15T18:42:58.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e2b8f1:::benchmark_data_smithi159_162467_object62011:head# 2024-04-15T18:42:58.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e357c3:::benchmark_data_smithi159_162467_object45871:head# 2024-04-15T18:42:58.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e3c48d:::benchmark_data_smithi159_162467_object24186:head# 2024-04-15T18:42:58.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e54896:::benchmark_data_smithi159_162467_object10639:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e5834b:::benchmark_data_smithi159_162467_object6455:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e5dee6:::benchmark_data_smithi159_162467_object61121:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e600bc:::benchmark_data_smithi159_162467_object587:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e6384a:::benchmark_data_smithi159_162467_object61586:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e75c34:::benchmark_data_smithi159_162467_object26011:head# 2024-04-15T18:42:58.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7e7ce20:::benchmark_data_smithi159_162467_object25789:head# 2024-04-15T18:42:58.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ea2a30:::benchmark_data_smithi159_162467_object11693:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ed1306:::benchmark_data_smithi159_162467_object34800:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7edd8d2:::benchmark_data_smithi159_162467_object37269:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ede902:::benchmark_data_smithi159_162467_object6262:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ee740f:::benchmark_data_smithi159_162467_object18325:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7eefa09:::benchmark_data_smithi159_162467_object72281:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7efa781:::benchmark_data_smithi159_162467_object14841:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f00e01:::benchmark_data_smithi159_162467_object44143:head# 2024-04-15T18:42:58.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f02c57:::benchmark_data_smithi159_162467_object27154:head# 2024-04-15T18:42:58.747 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f39304:::benchmark_data_smithi159_162467_object62209:head# 2024-04-15T18:42:58.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f3b9ac:::benchmark_data_smithi159_162467_object48113:head# 2024-04-15T18:42:58.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f4543c:::benchmark_data_smithi159_162467_object52907:head# 2024-04-15T18:42:58.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f4a29c:::benchmark_data_smithi159_162467_object24194:head# 2024-04-15T18:42:58.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f65164:::benchmark_data_smithi159_162467_object24480:head# 2024-04-15T18:42:58.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f6a6af:::benchmark_data_smithi159_162467_object51372:head# 2024-04-15T18:42:58.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f8270c:::benchmark_data_smithi159_162467_object69336:head# 2024-04-15T18:42:58.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f82aab:::benchmark_data_smithi159_162467_object63801:head# 2024-04-15T18:42:58.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f83cd2:::benchmark_data_smithi159_162467_object35213:head# 2024-04-15T18:42:58.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f94ed1:::benchmark_data_smithi159_162467_object19446:head# 2024-04-15T18:42:58.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7f9be70:::benchmark_data_smithi159_162467_object42967:head# 2024-04-15T18:42:58.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7fafcce:::benchmark_data_smithi159_162467_object39127:head# 2024-04-15T18:42:58.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7fbafaf:::benchmark_data_smithi159_162467_object20081:head# 2024-04-15T18:42:58.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7fc7fbf:::benchmark_data_smithi159_162467_object33118:head# 2024-04-15T18:42:58.756 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7fd3b04:::benchmark_data_smithi159_162467_object44944:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7fe9a05:::benchmark_data_smithi159_162467_object20956:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f7ff1440:::benchmark_data_smithi159_162467_object56266:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8016939:::benchmark_data_smithi159_162467_object344:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8023e38:::benchmark_data_smithi159_162467_object72042:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f803c79d:::benchmark_data_smithi159_162467_object21236:head# 2024-04-15T18:42:58.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8046ff2:::benchmark_data_smithi159_162467_object4616:head# 2024-04-15T18:42:58.760 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f805e4ac:::benchmark_data_smithi159_162467_object60087:head# 2024-04-15T18:42:58.762 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8062515:::benchmark_data_smithi159_162467_object31534:head# 2024-04-15T18:42:58.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8066d16:::benchmark_data_smithi159_162467_object67837:head# 2024-04-15T18:42:58.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8071ada:::benchmark_data_smithi159_162467_object47316:head# 2024-04-15T18:42:58.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8078152:::benchmark_data_smithi159_162467_object30486:head# 2024-04-15T18:42:58.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8079bd2:::benchmark_data_smithi159_162467_object4977:head# 2024-04-15T18:42:58.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f807a1ef:::benchmark_data_smithi159_162467_object38391:head# 2024-04-15T18:42:58.765 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f807ba43:::benchmark_data_smithi159_162467_object25303:head# 2024-04-15T18:42:58.765 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8086f65:::benchmark_data_smithi159_162467_object64659:head# 2024-04-15T18:42:58.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80cc6be:::benchmark_data_smithi159_162467_object40386:head# 2024-04-15T18:42:58.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80cc917:::benchmark_data_smithi159_162467_object43769:head# 2024-04-15T18:42:58.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80def32:::benchmark_data_smithi159_162467_object30775:head# 2024-04-15T18:42:58.767 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80e48df:::benchmark_data_smithi159_162467_object65050:head# 2024-04-15T18:42:58.767 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80f9948:::benchmark_data_smithi159_162467_object13692:head# 2024-04-15T18:42:58.768 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f80fc848:::benchmark_data_smithi159_162467_object31917:head# 2024-04-15T18:42:58.768 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8110f35:::benchmark_data_smithi159_162467_object43645:head# 2024-04-15T18:42:58.769 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8112701:::benchmark_data_smithi159_162467_object28541:head# 2024-04-15T18:42:58.769 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81146eb:::benchmark_data_smithi159_162467_object2917:head# 2024-04-15T18:42:58.769 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f811b9bf:::benchmark_data_smithi159_162467_object60491:head# 2024-04-15T18:42:58.770 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81282b1:::benchmark_data_smithi159_162467_object50775:head# 2024-04-15T18:42:58.770 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f814011f:::benchmark_data_smithi159_162467_object125:head# 2024-04-15T18:42:58.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8170f6a:::benchmark_data_smithi159_162467_object60723:head# 2024-04-15T18:42:58.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8181dd5:::benchmark_data_smithi159_162467_object38167:head# 2024-04-15T18:42:58.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8186513:::benchmark_data_smithi159_162467_object51093:head# 2024-04-15T18:42:58.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f818c593:::benchmark_data_smithi159_162467_object59179:head# 2024-04-15T18:42:58.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f819c838:::benchmark_data_smithi159_162467_object58387:head# 2024-04-15T18:42:58.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81a0632:::benchmark_data_smithi159_162467_object14449:head# 2024-04-15T18:42:58.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81ae78e:::benchmark_data_smithi159_162467_object37963:head# 2024-04-15T18:42:58.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81b0ae7:::benchmark_data_smithi159_162467_object39924:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81b261f:::benchmark_data_smithi159_162467_object15135:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81ba34a:::benchmark_data_smithi159_162467_object63037:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81bc854:::benchmark_data_smithi159_162467_object18594:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81c4fb5:::benchmark_data_smithi159_162467_object64770:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81cb0d7:::benchmark_data_smithi159_162467_object12650:head# 2024-04-15T18:42:58.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81ecc59:::benchmark_data_smithi159_162467_object46042:head# 2024-04-15T18:42:58.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81f79bb:::benchmark_data_smithi159_162467_object65119:head# 2024-04-15T18:42:58.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f81f7bde:::benchmark_data_smithi159_162467_object42317:head# 2024-04-15T18:42:58.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8204e93:::benchmark_data_smithi159_162467_object64679:head# 2024-04-15T18:42:58.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82109e3:::benchmark_data_smithi159_162467_object62634:head# 2024-04-15T18:42:58.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8217e9d:::benchmark_data_smithi159_162467_object64933:head# 2024-04-15T18:42:58.779 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8224433:::benchmark_data_smithi159_162467_object45314:head# 2024-04-15T18:42:58.779 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82271fa:::benchmark_data_smithi159_162467_object55355:head# 2024-04-15T18:42:58.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f822a100:::benchmark_data_smithi159_162467_object14031:head# 2024-04-15T18:42:58.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f822a8e8:::benchmark_data_smithi159_162467_object72581:head# 2024-04-15T18:42:58.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f823d3f1:::benchmark_data_smithi159_162467_object63096:head# 2024-04-15T18:42:58.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8246ca0:::benchmark_data_smithi159_162467_object5297:head# 2024-04-15T18:42:58.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8248418:::benchmark_data_smithi159_162467_object42123:head# 2024-04-15T18:42:58.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82676f4:::benchmark_data_smithi159_162467_object26682:head# 2024-04-15T18:42:58.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8267b73:::benchmark_data_smithi159_162467_object64333:head# 2024-04-15T18:42:58.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82686b4:::benchmark_data_smithi159_162467_object30960:head# 2024-04-15T18:42:58.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8273a3a:::benchmark_data_smithi159_162467_object41204:head# 2024-04-15T18:42:58.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f827f78a:::benchmark_data_smithi159_162467_object30689:head# 2024-04-15T18:42:58.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8284d92:::benchmark_data_smithi159_162467_object27880:head# 2024-04-15T18:42:58.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82864b2:::benchmark_data_smithi159_162467_object57026:head# 2024-04-15T18:42:58.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f829553b:::benchmark_data_smithi159_162467_object12555:head# 2024-04-15T18:42:58.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8297a64:::benchmark_data_smithi159_162467_object23799:head# 2024-04-15T18:42:58.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8297ec3:::benchmark_data_smithi159_162467_object49091:head# 2024-04-15T18:42:58.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8298804:::benchmark_data_smithi159_162467_object13542:head# 2024-04-15T18:42:58.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f829d214:::benchmark_data_smithi159_162467_object42239:head# 2024-04-15T18:42:58.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82ae1ef:::benchmark_data_smithi159_162467_object4966:head# 2024-04-15T18:42:58.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82d11b6:::benchmark_data_smithi159_162467_object64225:head# 2024-04-15T18:42:58.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82e717c:::benchmark_data_smithi159_162467_object41276:head# 2024-04-15T18:42:58.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82eedae:::benchmark_data_smithi159_162467_object55253:head# 2024-04-15T18:42:58.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82f248a:::benchmark_data_smithi159_162467_object7945:head# 2024-04-15T18:42:58.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f82f7e09:::benchmark_data_smithi159_162467_object23282:head# 2024-04-15T18:42:58.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8317770:::benchmark_data_smithi159_162467_object57153:head# 2024-04-15T18:42:58.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f831d7a2:::benchmark_data_smithi159_162467_object42672:head# 2024-04-15T18:42:58.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83281ed:::benchmark_data_smithi159_162467_object18966:head# 2024-04-15T18:42:58.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83316ab:::benchmark_data_smithi159_162467_object10882:head# 2024-04-15T18:42:58.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8336599:::benchmark_data_smithi159_162467_object28639:head# 2024-04-15T18:42:58.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f834c59c:::benchmark_data_smithi159_162467_object6974:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f834e286:::benchmark_data_smithi159_162467_object19434:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8352785:::benchmark_data_smithi159_162467_object50404:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f836aa42:::benchmark_data_smithi159_162467_object45070:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f836e825:::benchmark_data_smithi159_162467_object59131:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f838ca56:::benchmark_data_smithi159_162467_object52124:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f839ff4c:::benchmark_data_smithi159_162467_object74450:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83b2f6a:::benchmark_data_smithi159_162467_object6823:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83d09db:::benchmark_data_smithi159_162467_object18196:head# 2024-04-15T18:42:58.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83e47f6:::benchmark_data_smithi159_162467_object12649:head# 2024-04-15T18:42:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83e58c7:::benchmark_data_smithi159_162467_object22898:head# 2024-04-15T18:42:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83edaaa:::benchmark_data_smithi159_162467_object45669:head# 2024-04-15T18:42:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f83f47cf:::benchmark_data_smithi159_162467_object17886:head# 2024-04-15T18:42:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84097f4:::benchmark_data_smithi159_162467_object24031:head# 2024-04-15T18:42:58.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f841a0e1:::benchmark_data_smithi159_162467_object38809:head# 2024-04-15T18:42:58.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f841d1ff:::benchmark_data_smithi159_162467_object36253:head# 2024-04-15T18:42:58.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f842053f:::benchmark_data_smithi159_162467_object55685:head# 2024-04-15T18:42:58.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8429a5a:::benchmark_data_smithi159_162467_object64682:head# 2024-04-15T18:42:58.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8445bc1:::benchmark_data_smithi159_162467_object45273:head# 2024-04-15T18:42:58.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f844a357:::benchmark_data_smithi159_162467_object20445:head# 2024-04-15T18:42:58.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f845af51:::benchmark_data_smithi159_162467_object56430:head# 2024-04-15T18:42:58.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8461a10:::benchmark_data_smithi159_162467_object43469:head# 2024-04-15T18:42:58.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8468302:::benchmark_data_smithi159_162467_object27203:head# 2024-04-15T18:42:58.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8469842:::benchmark_data_smithi159_162467_object62325:head# 2024-04-15T18:42:58.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84703f2:::benchmark_data_smithi159_162467_object29267:head# 2024-04-15T18:42:58.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84be26b:::benchmark_data_smithi159_162467_object42144:head# 2024-04-15T18:42:58.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84c1679:::benchmark_data_smithi159_162467_object39526:head# 2024-04-15T18:42:58.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84c4618:::benchmark_data_smithi159_162467_object64145:head# 2024-04-15T18:42:58.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84ddfb2:::benchmark_data_smithi159_162467_object2477:head# 2024-04-15T18:42:58.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f84fdf63:::benchmark_data_smithi159_162467_object28700:head# 2024-04-15T18:42:58.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8519833:::benchmark_data_smithi159_162467_object37281:head# 2024-04-15T18:42:58.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f851d71e:::benchmark_data_smithi159_162467_object28941:head# 2024-04-15T18:42:58.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8522514:::benchmark_data_smithi159_162467_object56636:head# 2024-04-15T18:42:58.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85287f2:::benchmark_data_smithi159_162467_object17944:head# 2024-04-15T18:42:58.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f853a625:::benchmark_data_smithi159_162467_object68589:head# 2024-04-15T18:42:58.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8547759:::benchmark_data_smithi159_162467_object57580:head# 2024-04-15T18:42:58.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85545f9:::benchmark_data_smithi159_162467_object35001:head# 2024-04-15T18:42:58.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8555ea4:::benchmark_data_smithi159_162467_object64090:head# 2024-04-15T18:42:58.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8558eee:::benchmark_data_smithi159_162467_object10602:head# 2024-04-15T18:42:58.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f856805c:::benchmark_data_smithi159_162467_object28699:head# 2024-04-15T18:42:58.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8575bf9:::benchmark_data_smithi159_162467_object999:head# 2024-04-15T18:42:58.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f857bf24:::benchmark_data_smithi159_162467_object38747:head# 2024-04-15T18:42:58.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f859513c:::benchmark_data_smithi159_162467_object58939:head# 2024-04-15T18:42:58.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f859c5ab:::benchmark_data_smithi159_162467_object73645:head# 2024-04-15T18:42:58.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85aa0b3:::benchmark_data_smithi159_162467_object51142:head# 2024-04-15T18:42:58.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85ac684:::benchmark_data_smithi159_162467_object10839:head# 2024-04-15T18:42:58.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85cca7e:::benchmark_data_smithi159_162467_object10728:head# 2024-04-15T18:42:58.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85d6906:::benchmark_data_smithi159_162467_object50599:head# 2024-04-15T18:42:58.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85dfe7f:::benchmark_data_smithi159_162467_object55980:head# 2024-04-15T18:42:58.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85e5031:::benchmark_data_smithi159_162467_object5941:head# 2024-04-15T18:42:58.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85e57ba:::benchmark_data_smithi159_162467_object55830:head# 2024-04-15T18:42:58.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f85e6a81:::benchmark_data_smithi159_162467_object10836:head# 2024-04-15T18:42:58.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f860a603:::benchmark_data_smithi159_162467_object27559:head# 2024-04-15T18:42:58.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f86165b9:::benchmark_data_smithi159_162467_object43553:head# 2024-04-15T18:42:58.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f862c68f:::benchmark_data_smithi159_162467_object33857:head# 2024-04-15T18:42:58.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8646723:::benchmark_data_smithi159_162467_object47531:head# 2024-04-15T18:42:58.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f864784b:::benchmark_data_smithi159_162467_object33769:head# 2024-04-15T18:42:58.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f865af2f:::benchmark_data_smithi159_162467_object33188:head# 2024-04-15T18:42:58.817 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:42:58.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f866e9ea:::benchmark_data_smithi159_162467_object15364:head# 2024-04-15T18:42:58.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8681135:::benchmark_data_smithi159_162467_object10525:head# 2024-04-15T18:42:58.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f86888cd:::benchmark_data_smithi159_162467_object2706:head# 2024-04-15T18:42:58.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f86ccefd:::benchmark_data_smithi159_162467_object12478:head# 2024-04-15T18:42:58.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8709748:::benchmark_data_smithi159_162467_object24408:head# 2024-04-15T18:42:58.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f870a740:::benchmark_data_smithi159_162467_object2051:head# 2024-04-15T18:42:58.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f870c99c:::benchmark_data_smithi159_162467_object37105:head# 2024-04-15T18:42:58.821 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8713a2c:::benchmark_data_smithi159_162467_object9294:head# 2024-04-15T18:42:58.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87147e0:::benchmark_data_smithi159_162467_object69708:head# 2024-04-15T18:42:58.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8719ecc:::benchmark_data_smithi159_162467_object34864:head# 2024-04-15T18:42:58.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8732915:::benchmark_data_smithi159_162467_object16172:head# 2024-04-15T18:42:58.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8741c06:::benchmark_data_smithi159_162467_object44323:head# 2024-04-15T18:42:58.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8741c43:::benchmark_data_smithi159_162467_object39492:head# 2024-04-15T18:42:58.826 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8742efd:::benchmark_data_smithi159_162467_object41740:head# 2024-04-15T18:42:58.826 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f874dd9c:::benchmark_data_smithi159_162467_object46841:head# 2024-04-15T18:42:58.827 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f876a025:::benchmark_data_smithi159_162467_object38533:head# 2024-04-15T18:42:58.827 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f879d61b:::benchmark_data_smithi159_162467_object8836:head# 2024-04-15T18:42:58.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87a76b4:::benchmark_data_smithi159_162467_object55859:head# 2024-04-15T18:42:58.830 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87b7272:::benchmark_data_smithi159_162467_object46421:head# 2024-04-15T18:42:58.830 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87baead:::benchmark_data_smithi159_162467_object35039:head# 2024-04-15T18:42:58.830 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87bb355:::benchmark_data_smithi159_162467_object56733:head# 2024-04-15T18:42:58.831 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87c4375:::benchmark_data_smithi159_162467_object46744:head# 2024-04-15T18:42:58.831 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87d026d:::benchmark_data_smithi159_162467_object39276:head# 2024-04-15T18:42:58.832 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87db4c5:::benchmark_data_smithi159_162467_object44113:head# 2024-04-15T18:42:58.835 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87e2411:::benchmark_data_smithi159_162467_object45575:head# 2024-04-15T18:42:58.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f87e55cc:::benchmark_data_smithi159_162467_object69845:head# 2024-04-15T18:42:58.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8805f96:::benchmark_data_smithi159_162467_object58055:head# 2024-04-15T18:42:58.839 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8810e84:::benchmark_data_smithi159_162467_object16057:head# 2024-04-15T18:42:58.839 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88147ed:::benchmark_data_smithi159_162467_object37235:head# 2024-04-15T18:42:58.840 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8822272:::benchmark_data_smithi159_162467_object69377:head# 2024-04-15T18:42:58.840 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8841cb3:::benchmark_data_smithi159_162467_object12401:head# 2024-04-15T18:42:58.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88588fb:::benchmark_data_smithi159_162467_object51011:head# 2024-04-15T18:42:58.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f886d350:::benchmark_data_smithi159_162467_object814:head# 2024-04-15T18:42:58.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88a455a:::benchmark_data_smithi159_162467_object35111:head# 2024-04-15T18:42:58.842 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88a4fe0:::benchmark_data_smithi159_162467_object62405:head# 2024-04-15T18:42:58.842 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88bb044:::benchmark_data_smithi159_162467_object19120:head# 2024-04-15T18:42:58.842 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88ccebc:::benchmark_data_smithi159_162467_object29400:head# 2024-04-15T18:42:58.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88d55e7:::benchmark_data_smithi159_162467_object67383:head# 2024-04-15T18:42:58.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88dd406:::benchmark_data_smithi159_162467_object9417:head# 2024-04-15T18:42:58.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88e2142:::benchmark_data_smithi159_162467_object48215:head# 2024-04-15T18:42:58.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88ecb30:::benchmark_data_smithi159_162467_object72468:head# 2024-04-15T18:42:58.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88f7b3a:::benchmark_data_smithi159_162467_object38945:head# 2024-04-15T18:42:58.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f88f8f6a:::benchmark_data_smithi159_162467_object26415:head# 2024-04-15T18:42:58.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8903de7:::benchmark_data_smithi159_162467_object73772:head# 2024-04-15T18:42:58.846 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f890da4f:::benchmark_data_smithi159_162467_object50850:head# 2024-04-15T18:42:58.846 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f891c98e:::benchmark_data_smithi159_162467_object33986:head# 2024-04-15T18:42:58.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8923d11:::benchmark_data_smithi159_162467_object52018:head# 2024-04-15T18:42:58.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f892e713:::benchmark_data_smithi159_162467_object65520:head# 2024-04-15T18:42:58.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f892fb93:::benchmark_data_smithi159_162467_object10976:head# 2024-04-15T18:42:58.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8932807:::benchmark_data_smithi159_162467_object30910:head# 2024-04-15T18:42:58.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89583a7:::benchmark_data_smithi159_162467_object62204:head# 2024-04-15T18:42:58.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8958b3b:::benchmark_data_smithi159_162467_object59371:head# 2024-04-15T18:42:58.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f895eb66:::benchmark_data_smithi159_162467_object26512:head# 2024-04-15T18:42:58.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f896627d:::benchmark_data_smithi159_162467_object37222:head# 2024-04-15T18:42:58.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f896e03e:::benchmark_data_smithi159_162467_object72677:head# 2024-04-15T18:42:58.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89740e9:::benchmark_data_smithi159_162467_object22925:head# 2024-04-15T18:42:58.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f898cabc:::benchmark_data_smithi159_162467_object52592:head# 2024-04-15T18:42:58.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f899ad33:::benchmark_data_smithi159_162467_object64754:head# 2024-04-15T18:42:58.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f899e34e:::benchmark_data_smithi159_162467_object11406:head# 2024-04-15T18:42:58.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89aa753:::benchmark_data_smithi159_162467_object58467:head# 2024-04-15T18:42:58.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89aca39:::benchmark_data_smithi159_162467_object60355:head# 2024-04-15T18:42:58.853 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:42:58.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89b7cf4:::benchmark_data_smithi159_162467_object48319:head# 2024-04-15T18:42:58.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89bd98f:::benchmark_data_smithi159_162467_object33631:head# 2024-04-15T18:42:58.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89c9ef2:::benchmark_data_smithi159_162467_object5210:head# 2024-04-15T18:42:58.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89e0911:::benchmark_data_smithi159_162467_object17055:head# 2024-04-15T18:42:58.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89ebd4f:::benchmark_data_smithi159_162467_object68777:head# 2024-04-15T18:42:58.859 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89ec5ef:::benchmark_data_smithi159_162467_object65182:head# 2024-04-15T18:42:58.862 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f89fadd9:::benchmark_data_smithi159_162467_object133:head# 2024-04-15T18:42:58.862 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a21883:::benchmark_data_smithi159_162467_object10263:head# 2024-04-15T18:42:58.863 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a230e4:::benchmark_data_smithi159_162467_object47011:head# 2024-04-15T18:42:58.865 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a2484f:::benchmark_data_smithi159_162467_object4265:head# 2024-04-15T18:42:58.865 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a27f8c:::benchmark_data_smithi159_162467_object62999:head# 2024-04-15T18:42:58.866 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a506e5:::benchmark_data_smithi159_162467_object37329:head# 2024-04-15T18:42:58.866 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a51c41:::benchmark_data_smithi159_162467_object10819:head# 2024-04-15T18:42:58.866 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a6ed77:::benchmark_data_smithi159_162467_object7643:head# 2024-04-15T18:42:58.867 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a7b81b:::benchmark_data_smithi159_162467_object39709:head# 2024-04-15T18:42:58.867 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a7d8b3:::benchmark_data_smithi159_162467_object62303:head# 2024-04-15T18:42:58.868 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a86769:::benchmark_data_smithi159_162467_object52406:head# 2024-04-15T18:42:58.868 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a88372:::benchmark_data_smithi159_162467_object13311:head# 2024-04-15T18:42:58.869 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a88455:::benchmark_data_smithi159_162467_object71716:head# 2024-04-15T18:42:58.869 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a8a17e:::benchmark_data_smithi159_162467_object8730:head# 2024-04-15T18:42:58.870 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8a9ac87:::benchmark_data_smithi159_162467_object72509:head# 2024-04-15T18:42:58.870 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8aa8c41:::benchmark_data_smithi159_162467_object53788:head# 2024-04-15T18:42:58.870 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ab32b7:::benchmark_data_smithi159_162467_object47763:head# 2024-04-15T18:42:58.871 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8aba79a:::benchmark_data_smithi159_162467_object46597:head# 2024-04-15T18:42:58.871 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ad260f:::benchmark_data_smithi159_162467_object27687:head# 2024-04-15T18:42:58.872 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8adfe8d:::benchmark_data_smithi159_162467_object33635:head# 2024-04-15T18:42:58.872 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8aea74c:::benchmark_data_smithi159_162467_object15697:head# 2024-04-15T18:42:58.873 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b04dfa:::benchmark_data_smithi159_162467_object10766:head# 2024-04-15T18:42:58.873 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b0578d:::benchmark_data_smithi159_162467_object3574:head# 2024-04-15T18:42:58.873 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b107db:::benchmark_data_smithi159_162467_object1375:head# 2024-04-15T18:42:58.874 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b33dfb:::benchmark_data_smithi159_162467_object67026:head# 2024-04-15T18:42:58.874 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b35681:::benchmark_data_smithi159_162467_object23097:head# 2024-04-15T18:42:58.876 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b4641b:::benchmark_data_smithi159_162467_object6579:head# 2024-04-15T18:42:58.877 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b47c60:::benchmark_data_smithi159_162467_object57253:head# 2024-04-15T18:42:58.877 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b525e7:::benchmark_data_smithi159_162467_object47178:head# 2024-04-15T18:42:58.878 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b62a36:::benchmark_data_smithi159_162467_object37376:head# 2024-04-15T18:42:58.878 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b6599b:::benchmark_data_smithi159_162467_object21048:head# 2024-04-15T18:42:58.878 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b72dc2:::benchmark_data_smithi159_162467_object70794:head# 2024-04-15T18:42:58.879 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b79739:::benchmark_data_smithi159_162467_object42893:head# 2024-04-15T18:42:58.879 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b7bb26:::benchmark_data_smithi159_162467_object63487:head# 2024-04-15T18:42:58.880 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b8006f:::benchmark_data_smithi159_162467_object38511:head# 2024-04-15T18:42:58.880 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b86745:::benchmark_data_smithi159_162467_object63067:head# 2024-04-15T18:42:58.881 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8b8d297:::benchmark_data_smithi159_162467_object51845:head# 2024-04-15T18:42:58.881 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8bb74b1:::benchmark_data_smithi159_162467_object28254:head# 2024-04-15T18:42:58.881 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8bb7580:::benchmark_data_smithi159_162467_object7326:head# 2024-04-15T18:42:58.883 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8bc7f7f:::benchmark_data_smithi159_162467_object20877:head# 2024-04-15T18:42:58.883 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8bcd4fe:::benchmark_data_smithi159_162467_object16418:head# 2024-04-15T18:42:58.884 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8bf24b2:::benchmark_data_smithi159_162467_object8164:head# 2024-04-15T18:42:58.884 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c2743f:::benchmark_data_smithi159_162467_object35434:head# 2024-04-15T18:42:58.885 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c2f4ee:::benchmark_data_smithi159_162467_object7065:head# 2024-04-15T18:42:58.885 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c37df5:::benchmark_data_smithi159_162467_object46387:head# 2024-04-15T18:42:58.886 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c38c0b:::benchmark_data_smithi159_162467_object15384:head# 2024-04-15T18:42:58.886 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c48238:::benchmark_data_smithi159_162467_object63654:head# 2024-04-15T18:42:58.886 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c4d306:::benchmark_data_smithi159_162467_object45543:head# 2024-04-15T18:42:58.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c70f55:::benchmark_data_smithi159_162467_object24243:head# 2024-04-15T18:42:58.887 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c736fe:::benchmark_data_smithi159_162467_object27191:head# 2024-04-15T18:42:58.888 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c82aab:::benchmark_data_smithi159_162467_object52622:head# 2024-04-15T18:42:58.888 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c88048:::benchmark_data_smithi159_162467_object36276:head# 2024-04-15T18:42:58.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8c9780c:::benchmark_data_smithi159_162467_object22494:head# 2024-04-15T18:42:58.889 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8cc13d0:::benchmark_data_smithi159_162467_object73133:head# 2024-04-15T18:42:58.890 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8cc1ce4:::benchmark_data_smithi159_162467_object37749:head# 2024-04-15T18:42:58.890 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8cca656:::benchmark_data_smithi159_162467_object66616:head# 2024-04-15T18:42:58.891 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ce4b15:::benchmark_data_smithi159_162467_object41019:head# 2024-04-15T18:42:58.891 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ce6397:::benchmark_data_smithi159_162467_object15459:head# 2024-04-15T18:42:58.891 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8cf8c04:::benchmark_data_smithi159_162467_object69033:head# 2024-04-15T18:42:58.892 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d27092:::benchmark_data_smithi159_162467_object14329:head# 2024-04-15T18:42:58.892 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d275b8:::benchmark_data_smithi159_162467_object55033:head# 2024-04-15T18:42:58.893 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d3380c:::benchmark_data_smithi159_162467_object51714:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d3d7e4:::benchmark_data_smithi159_162467_object55872:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d4cc44:::benchmark_data_smithi159_162467_object70071:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d54ba4:::benchmark_data_smithi159_162467_object4959:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d5f237:::benchmark_data_smithi159_162467_object26562:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d779eb:::benchmark_data_smithi159_162467_object60623:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d78f7d:::benchmark_data_smithi159_162467_object8715:head# 2024-04-15T18:42:58.898 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d79814:::benchmark_data_smithi159_162467_object54013:head# 2024-04-15T18:42:58.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8d8d31b:::benchmark_data_smithi159_162467_object35568:head# 2024-04-15T18:42:58.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8da9aca:::benchmark_data_smithi159_162467_object3921:head# 2024-04-15T18:42:58.899 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dad246:::benchmark_data_smithi159_162467_object69360:head# 2024-04-15T18:42:58.900 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8db1e61:::benchmark_data_smithi159_162467_object73135:head# 2024-04-15T18:42:58.900 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dc0154:::benchmark_data_smithi159_162467_object34791:head# 2024-04-15T18:42:58.901 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dc0e51:::benchmark_data_smithi159_162467_object74122:head# 2024-04-15T18:42:58.901 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dc2605:::benchmark_data_smithi159_162467_object62462:head# 2024-04-15T18:42:58.901 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dd6082:::benchmark_data_smithi159_162467_object58701:head# 2024-04-15T18:42:58.902 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dda072:::benchmark_data_smithi159_162467_object49874:head# 2024-04-15T18:42:58.903 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dea922:::benchmark_data_smithi159_162467_object32545:head# 2024-04-15T18:42:58.903 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dedc88:::benchmark_data_smithi159_162467_object58090:head# 2024-04-15T18:42:58.904 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8df1396:::benchmark_data_smithi159_162467_object57508:head# 2024-04-15T18:42:58.904 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8dffb0e:::benchmark_data_smithi159_162467_object57751:head# 2024-04-15T18:42:58.904 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e0ba58:::benchmark_data_smithi159_162467_object7810:head# 2024-04-15T18:42:58.905 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e1689d:::benchmark_data_smithi159_162467_object45445:head# 2024-04-15T18:42:58.905 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e2863c:::benchmark_data_smithi159_162467_object38680:head# 2024-04-15T18:42:58.906 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e3d055:::benchmark_data_smithi159_162467_object22858:head# 2024-04-15T18:42:58.906 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e3d664:::benchmark_data_smithi159_162467_object2839:head# 2024-04-15T18:42:58.907 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e5764b:::benchmark_data_smithi159_162467_object6454:head# 2024-04-15T18:42:58.907 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e59da6:::benchmark_data_smithi159_162467_object34992:head# 2024-04-15T18:42:58.908 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e7d004:::benchmark_data_smithi159_162467_object31458:head# 2024-04-15T18:42:58.908 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e861b7:::benchmark_data_smithi159_162467_object5509:head# 2024-04-15T18:42:58.908 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e89351:::benchmark_data_smithi159_162467_object19063:head# 2024-04-15T18:42:58.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8e9eb8a:::benchmark_data_smithi159_162467_object39573:head# 2024-04-15T18:42:58.909 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ead14c:::benchmark_data_smithi159_162467_object54870:head# 2024-04-15T18:42:58.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8eb17e3:::benchmark_data_smithi159_162467_object37887:head# 2024-04-15T18:42:58.910 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ebcac2:::benchmark_data_smithi159_162467_object47124:head# 2024-04-15T18:42:58.911 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ec4917:::benchmark_data_smithi159_162467_object21746:head# 2024-04-15T18:42:58.911 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8edca8a:::benchmark_data_smithi159_162467_object1028:head# 2024-04-15T18:42:58.912 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8ee0d88:::benchmark_data_smithi159_162467_object23070:head# 2024-04-15T18:42:58.912 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f39ebe:::benchmark_data_smithi159_162467_object40448:head# 2024-04-15T18:42:58.913 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f3bff6:::benchmark_data_smithi159_162467_object54682:head# 2024-04-15T18:42:58.913 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f4311b:::benchmark_data_smithi159_162467_object68368:head# 2024-04-15T18:42:58.913 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f4cc4b:::benchmark_data_smithi159_162467_object927:head# 2024-04-15T18:42:58.914 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f4fd4d:::benchmark_data_smithi159_162467_object67761:head# 2024-04-15T18:42:58.914 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f670d7:::benchmark_data_smithi159_162467_object4493:head# 2024-04-15T18:42:58.915 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f693f3:::benchmark_data_smithi159_162467_object43297:head# 2024-04-15T18:42:58.915 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f7646a:::benchmark_data_smithi159_162467_object34464:head# 2024-04-15T18:42:58.915 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f79e9f:::benchmark_data_smithi159_162467_object62248:head# 2024-04-15T18:42:58.916 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f83cfc:::benchmark_data_smithi159_162467_object55855:head# 2024-04-15T18:42:58.916 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8f9d768:::benchmark_data_smithi159_162467_object511:head# 2024-04-15T18:42:58.917 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8fb61a7:::benchmark_data_smithi159_162467_object24005:head# 2024-04-15T18:42:58.917 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8fc9c20:::benchmark_data_smithi159_162467_object47596:head# 2024-04-15T18:42:58.918 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8fd3c9f:::benchmark_data_smithi159_162467_object58064:head# 2024-04-15T18:42:58.919 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8fe42cd:::benchmark_data_smithi159_162467_object3683:head# 2024-04-15T18:42:58.921 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f8feadc7:::benchmark_data_smithi159_162467_object58216:head# 2024-04-15T18:42:58.921 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f903af4a:::benchmark_data_smithi159_162467_object66296:head# 2024-04-15T18:42:58.922 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f903e395:::benchmark_data_smithi159_162467_object593:head# 2024-04-15T18:42:58.922 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f905b938:::benchmark_data_smithi159_162467_object17438:head# 2024-04-15T18:42:58.923 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9082a18:::benchmark_data_smithi159_162467_object20838:head# 2024-04-15T18:42:58.923 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f908679b:::benchmark_data_smithi159_162467_object13064:head# 2024-04-15T18:42:58.924 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f909c17b:::benchmark_data_smithi159_162467_object58876:head# 2024-04-15T18:42:58.924 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90b479f:::benchmark_data_smithi159_162467_object39096:head# 2024-04-15T18:42:58.924 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90bf4ec:::benchmark_data_smithi159_162467_object61183:head# 2024-04-15T18:42:58.925 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90bf9d3:::benchmark_data_smithi159_162467_object8288:head# 2024-04-15T18:42:58.925 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90c8289:::benchmark_data_smithi159_162467_object5821:head# 2024-04-15T18:42:58.926 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90ca609:::benchmark_data_smithi159_162467_object39813:head# 2024-04-15T18:42:58.926 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90d2020:::benchmark_data_smithi159_162467_object17268:head# 2024-04-15T18:42:58.926 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90d7d9b:::benchmark_data_smithi159_162467_object34555:head# 2024-04-15T18:42:58.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90dc452:::benchmark_data_smithi159_162467_object71743:head# 2024-04-15T18:42:58.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f90ddefa:::benchmark_data_smithi159_162467_object32159:head# 2024-04-15T18:42:58.927 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9108385:::benchmark_data_smithi159_162467_object61792:head# 2024-04-15T18:42:58.928 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9114cea:::benchmark_data_smithi159_162467_object7196:head# 2024-04-15T18:42:58.928 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9128f12:::benchmark_data_smithi159_162467_object3023:head# 2024-04-15T18:42:58.929 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f912970f:::benchmark_data_smithi159_162467_object22344:head# 2024-04-15T18:42:58.929 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f912ffd9:::benchmark_data_smithi159_162467_object20031:head# 2024-04-15T18:42:58.929 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9134171:::benchmark_data_smithi159_162467_object43060:head# 2024-04-15T18:42:58.930 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91383eb:::benchmark_data_smithi159_162467_object52380:head# 2024-04-15T18:42:58.930 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f913b97e:::benchmark_data_smithi159_162467_object67666:head# 2024-04-15T18:42:58.931 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9153b7e:::benchmark_data_smithi159_162467_object52716:head# 2024-04-15T18:42:58.931 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f915933c:::benchmark_data_smithi159_162467_object52601:head# 2024-04-15T18:42:58.931 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9167164:::benchmark_data_smithi159_162467_object17539:head# 2024-04-15T18:42:58.932 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f917ba71:::benchmark_data_smithi159_162467_object63018:head# 2024-04-15T18:42:58.932 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9181b12:::benchmark_data_smithi159_162467_object11190:head# 2024-04-15T18:42:58.933 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91bc428:::benchmark_data_smithi159_162467_object59098:head# 2024-04-15T18:42:58.933 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91c3188:::benchmark_data_smithi159_162467_object8931:head# 2024-04-15T18:42:58.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91dd4cc:::benchmark_data_smithi159_162467_object24421:head# 2024-04-15T18:42:58.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91e14ef:::benchmark_data_smithi159_162467_object28129:head# 2024-04-15T18:42:58.934 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91eef24:::benchmark_data_smithi159_162467_object15606:head# 2024-04-15T18:42:58.935 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91f1a7f:::benchmark_data_smithi159_162467_object21289:head# 2024-04-15T18:42:58.935 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f91f3132:::benchmark_data_smithi159_162467_object29660:head# 2024-04-15T18:42:58.936 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f920df60:::benchmark_data_smithi159_162467_object43183:head# 2024-04-15T18:42:58.936 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f923b903:::benchmark_data_smithi159_162467_object42170:head# 2024-04-15T18:42:58.937 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92426ef:::benchmark_data_smithi159_162467_object28034:head# 2024-04-15T18:42:58.937 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f925656c:::benchmark_data_smithi159_162467_object59487:head# 2024-04-15T18:42:58.938 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92585da:::benchmark_data_smithi159_162467_object6256:head# 2024-04-15T18:42:58.938 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f925c060:::benchmark_data_smithi159_162467_object22837:head# 2024-04-15T18:42:58.939 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9268c5d:::benchmark_data_smithi159_162467_object68683:head# 2024-04-15T18:42:58.939 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f927967e:::benchmark_data_smithi159_162467_object512:head# 2024-04-15T18:42:58.940 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f927ef11:::benchmark_data_smithi159_162467_object29315:head# 2024-04-15T18:42:58.940 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9285325:::benchmark_data_smithi159_162467_object10428:head# 2024-04-15T18:42:58.940 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9297985:::benchmark_data_smithi159_162467_object37821:head# 2024-04-15T18:42:58.941 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92a0cfe:::benchmark_data_smithi159_162467_object12138:head# 2024-04-15T18:42:58.941 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92a2c9a:::benchmark_data_smithi159_162467_object33880:head# 2024-04-15T18:42:58.941 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92a3465:::benchmark_data_smithi159_162467_object70875:head# 2024-04-15T18:42:58.942 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92af738:::benchmark_data_smithi159_162467_object20409:head# 2024-04-15T18:42:58.942 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92afc15:::benchmark_data_smithi159_162467_object71818:head# 2024-04-15T18:42:58.943 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f92bb55a:::benchmark_data_smithi159_162467_object20417:head# 2024-04-15T18:42:58.943 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93208d5:::benchmark_data_smithi159_162467_object53629:head# 2024-04-15T18:42:58.944 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f933c0b0:::benchmark_data_smithi159_162467_object53609:head# 2024-04-15T18:42:58.944 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9340395:::benchmark_data_smithi159_162467_object15884:head# 2024-04-15T18:42:58.944 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f934e789:::benchmark_data_smithi159_162467_object13076:head# 2024-04-15T18:42:58.945 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9351328:::benchmark_data_smithi159_162467_object41453:head# 2024-04-15T18:42:58.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9351421:::benchmark_data_smithi159_162467_object72625:head# 2024-04-15T18:42:58.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f935b7e6:::benchmark_data_smithi159_162467_object55612:head# 2024-04-15T18:42:58.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9360f72:::benchmark_data_smithi159_162467_object50196:head# 2024-04-15T18:42:58.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9364092:::benchmark_data_smithi159_162467_object3942:head# 2024-04-15T18:42:58.947 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f937005b:::benchmark_data_smithi159_162467_object22534:head# 2024-04-15T18:42:58.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93865e0:::benchmark_data_smithi159_162467_object55272:head# 2024-04-15T18:42:58.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93899fe:::benchmark_data_smithi159_162467_object50904:head# 2024-04-15T18:42:58.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f938ec97:::benchmark_data_smithi159_162467_object55406:head# 2024-04-15T18:42:58.948 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9391f20:::benchmark_data_smithi159_162467_object15725:head# 2024-04-15T18:42:58.949 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f939e5e9:::benchmark_data_smithi159_162467_object34707:head# 2024-04-15T18:42:58.949 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93a150e:::benchmark_data_smithi159_162467_object36575:head# 2024-04-15T18:42:58.949 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93ae237:::benchmark_data_smithi159_162467_object34328:head# 2024-04-15T18:42:58.950 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93c286f:::benchmark_data_smithi159_162467_object71451:head# 2024-04-15T18:42:58.950 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93c4247:::benchmark_data_smithi159_162467_object67731:head# 2024-04-15T18:42:58.951 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f93d1db0:::benchmark_data_smithi159_162467_object2166:head# 2024-04-15T18:42:58.951 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f940ad09:::benchmark_data_smithi159_162467_object55679:head# 2024-04-15T18:42:58.952 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f942bbf3:::benchmark_data_smithi159_162467_object5908:head# 2024-04-15T18:42:58.952 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f942dbb0:::benchmark_data_smithi159_162467_object12581:head# 2024-04-15T18:42:58.953 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9438fd6:::benchmark_data_smithi159_162467_object12968:head# 2024-04-15T18:42:58.957 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f943bce1:::benchmark_data_smithi159_162467_object71202:head# 2024-04-15T18:42:58.957 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9451d46:::benchmark_data_smithi159_162467_object60648:head# 2024-04-15T18:42:58.958 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f945f5f3:::benchmark_data_smithi159_162467_object69254:head# 2024-04-15T18:42:58.958 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f946f506:::benchmark_data_smithi159_162467_object65198:head# 2024-04-15T18:42:58.959 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94716bc:::benchmark_data_smithi159_162467_object9835:head# 2024-04-15T18:42:58.959 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94829dc:::benchmark_data_smithi159_162467_object53627:head# 2024-04-15T18:42:58.960 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f949a2dc:::benchmark_data_smithi159_162467_object5796:head# 2024-04-15T18:42:58.960 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94a0e85:::benchmark_data_smithi159_162467_object2800:head# 2024-04-15T18:42:58.961 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94a958d:::benchmark_data_smithi159_162467_object59834:head# 2024-04-15T18:42:58.961 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94b54e6:::benchmark_data_smithi159_162467_object16997:head# 2024-04-15T18:42:58.962 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94cf127:::benchmark_data_smithi159_162467_object28665:head# 2024-04-15T18:42:58.962 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94d0c11:::benchmark_data_smithi159_162467_object62047:head# 2024-04-15T18:42:58.963 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94f5a62:::benchmark_data_smithi159_162467_object7357:head# 2024-04-15T18:42:58.963 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f94f74ac:::benchmark_data_smithi159_162467_object4349:head# 2024-04-15T18:42:58.964 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95001a1:::benchmark_data_smithi159_162467_object69735:head# 2024-04-15T18:42:58.964 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9505282:::benchmark_data_smithi159_162467_object41960:head# 2024-04-15T18:42:58.965 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9514fbb:::benchmark_data_smithi159_162467_object49347:head# 2024-04-15T18:42:58.965 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f954eb82:::benchmark_data_smithi159_162467_object33239:head# 2024-04-15T18:42:58.966 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f955df38:::benchmark_data_smithi159_162467_object55541:head# 2024-04-15T18:42:58.966 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f957dc26:::benchmark_data_smithi159_162467_object27719:head# 2024-04-15T18:42:58.966 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f958c495:::benchmark_data_smithi159_162467_object45923:head# 2024-04-15T18:42:58.967 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9593ad6:::benchmark_data_smithi159_162467_object69334:head# 2024-04-15T18:42:58.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9599d40:::benchmark_data_smithi159_162467_object9655:head# 2024-04-15T18:42:58.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95aed76:::benchmark_data_smithi159_162467_object17100:head# 2024-04-15T18:42:58.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95bc1ac:::benchmark_data_smithi159_162467_object71745:head# 2024-04-15T18:42:58.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95c81bf:::benchmark_data_smithi159_162467_object26053:head# 2024-04-15T18:42:58.969 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95c93f5:::benchmark_data_smithi159_162467_object59876:head# 2024-04-15T18:42:58.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95eb223:::benchmark_data_smithi159_162467_object45711:head# 2024-04-15T18:42:58.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95f3dcc:::benchmark_data_smithi159_162467_object53783:head# 2024-04-15T18:42:58.970 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f95f8f19:::benchmark_data_smithi159_162467_object29561:head# 2024-04-15T18:42:58.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9619edc:::benchmark_data_smithi159_162467_object71293:head# 2024-04-15T18:42:58.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f961d6d1:::benchmark_data_smithi159_162467_object59096:head# 2024-04-15T18:42:58.971 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96223e8:::benchmark_data_smithi159_162467_object24310:head# 2024-04-15T18:42:58.972 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9624055:::benchmark_data_smithi159_162467_object73378:head# 2024-04-15T18:42:58.972 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f962d442:::benchmark_data_smithi159_162467_object49691:head# 2024-04-15T18:42:58.973 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96337ef:::benchmark_data_smithi159_162467_object9933:head# 2024-04-15T18:42:58.973 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f964930b:::benchmark_data_smithi159_162467_object62581:head# 2024-04-15T18:42:58.974 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9657e79:::benchmark_data_smithi159_162467_object62670:head# 2024-04-15T18:42:58.974 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f965d5b9:::benchmark_data_smithi159_162467_object62672:head# 2024-04-15T18:42:58.975 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f965f10e:::benchmark_data_smithi159_162467_object7651:head# 2024-04-15T18:42:58.975 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9667c4f:::benchmark_data_smithi159_162467_object28842:head# 2024-04-15T18:42:58.975 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f968ce1a:::benchmark_data_smithi159_162467_object27329:head# 2024-04-15T18:42:58.976 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f969f4cf:::benchmark_data_smithi159_162467_object6969:head# 2024-04-15T18:42:58.976 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96a32f0:::benchmark_data_smithi159_162467_object47189:head# 2024-04-15T18:42:58.977 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96ac612:::benchmark_data_smithi159_162467_object41833:head# 2024-04-15T18:42:58.977 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96aec13:::benchmark_data_smithi159_162467_object30755:head# 2024-04-15T18:42:58.978 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96c5e2e:::benchmark_data_smithi159_162467_object52046:head# 2024-04-15T18:42:58.978 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96da3ee:::benchmark_data_smithi159_162467_object4304:head# 2024-04-15T18:42:58.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96e1a3c:::benchmark_data_smithi159_162467_object48393:head# 2024-04-15T18:42:58.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f96fa331:::benchmark_data_smithi159_162467_object908:head# 2024-04-15T18:42:58.979 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9703453:::benchmark_data_smithi159_162467_object31147:head# 2024-04-15T18:42:58.980 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f970afc6:::benchmark_data_smithi159_162467_object66957:head# 2024-04-15T18:42:58.980 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f970f958:::benchmark_data_smithi159_162467_object71100:head# 2024-04-15T18:42:58.984 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9717e67:::benchmark_data_smithi159_162467_object66018:head# 2024-04-15T18:42:58.984 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9719e5e:::benchmark_data_smithi159_162467_object20400:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f971e589:::benchmark_data_smithi159_162467_object50903:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9727c4d:::benchmark_data_smithi159_162467_object58832:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f972bddb:::benchmark_data_smithi159_162467_object70600:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f972dfa9:::benchmark_data_smithi159_162467_object16804:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f972f6be:::benchmark_data_smithi159_162467_object39009:head# 2024-04-15T18:42:58.985 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f973344e:::benchmark_data_smithi159_162467_object47890:head# 2024-04-15T18:42:58.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9734c7f:::benchmark_data_smithi159_162467_object5386:head# 2024-04-15T18:42:58.988 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9735ca7:::benchmark_data_smithi159_162467_object4967:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f973c9e1:::benchmark_data_smithi159_162467_object38970:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f973ef56:::benchmark_data_smithi159_162467_object12482:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f974d5d7:::benchmark_data_smithi159_162467_object9338:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f974f01d:::benchmark_data_smithi159_162467_object38368:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9760e32:::benchmark_data_smithi159_162467_object67572:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97610b3:::benchmark_data_smithi159_162467_object39592:head# 2024-04-15T18:42:58.989 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9769098:::benchmark_data_smithi159_162467_object108:head# 2024-04-15T18:42:58.991 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f976a691:::benchmark_data_smithi159_162467_object16739:head# 2024-04-15T18:42:58.992 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f978ae74:::benchmark_data_smithi159_162467_object30476:head# 2024-04-15T18:42:58.993 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9792080:::benchmark_data_smithi159_162467_object21416:head# 2024-04-15T18:42:58.996 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f979922e:::benchmark_data_smithi159_162467_object3816:head# 2024-04-15T18:42:58.996 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97abc1a:::benchmark_data_smithi159_162467_object42462:head# 2024-04-15T18:42:58.997 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97bfb67:::benchmark_data_smithi159_162467_object39107:head# 2024-04-15T18:42:58.997 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97ce137:::benchmark_data_smithi159_162467_object71820:head# 2024-04-15T18:42:58.998 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97d8c81:::benchmark_data_smithi159_162467_object31659:head# 2024-04-15T18:42:58.998 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f97ea7c2:::benchmark_data_smithi159_162467_object60136:head# 2024-04-15T18:42:58.999 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98068a1:::benchmark_data_smithi159_162467_object28486:head# 2024-04-15T18:42:58.999 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9829923:::benchmark_data_smithi159_162467_object26199:head# 2024-04-15T18:42:59.000 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9839b2a:::benchmark_data_smithi159_162467_object23690:head# 2024-04-15T18:42:59.000 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98495bd:::benchmark_data_smithi159_162467_object53946:head# 2024-04-15T18:42:59.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98597d8:::benchmark_data_smithi159_162467_object72556:head# 2024-04-15T18:42:59.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9865d97:::benchmark_data_smithi159_162467_object19232:head# 2024-04-15T18:42:59.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98970d3:::benchmark_data_smithi159_162467_object74794:head# 2024-04-15T18:42:59.001 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f989add8:::benchmark_data_smithi159_162467_object48051:head# 2024-04-15T18:42:59.002 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98d0751:::benchmark_data_smithi159_162467_object7342:head# 2024-04-15T18:42:59.002 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98d35af:::benchmark_data_smithi159_162467_object11157:head# 2024-04-15T18:42:59.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98d88fd:::benchmark_data_smithi159_162467_object31214:head# 2024-04-15T18:42:59.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98da52f:::benchmark_data_smithi159_162467_object1370:head# 2024-04-15T18:42:59.003 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f98e6c4c:::benchmark_data_smithi159_162467_object25827:head# 2024-04-15T18:42:59.004 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9903b46:::benchmark_data_smithi159_162467_object58035:head# 2024-04-15T18:42:59.004 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f990f4d2:::benchmark_data_smithi159_162467_object50261:head# 2024-04-15T18:42:59.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99132b3:::benchmark_data_smithi159_162467_object281:head# 2024-04-15T18:42:59.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99170c3:::benchmark_data_smithi159_162467_object59975:head# 2024-04-15T18:42:59.005 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f991b7a0:::benchmark_data_smithi159_162467_object64621:head# 2024-04-15T18:42:59.006 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f991e153:::benchmark_data_smithi159_162467_object3267:head# 2024-04-15T18:42:59.006 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f994c976:::benchmark_data_smithi159_162467_object31222:head# 2024-04-15T18:42:59.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9960c93:::benchmark_data_smithi159_162467_object47661:head# 2024-04-15T18:42:59.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9969068:::benchmark_data_smithi159_162467_object60058:head# 2024-04-15T18:42:59.007 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9970e49:::benchmark_data_smithi159_162467_object70612:head# 2024-04-15T18:42:59.008 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9994834:::benchmark_data_smithi159_162467_object19025:head# 2024-04-15T18:42:59.008 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99a0998:::benchmark_data_smithi159_162467_object51887:head# 2024-04-15T18:42:59.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99ac566:::benchmark_data_smithi159_162467_object10554:head# 2024-04-15T18:42:59.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99c740b:::benchmark_data_smithi159_162467_object46528:head# 2024-04-15T18:42:59.009 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99d0a4d:::benchmark_data_smithi159_162467_object22423:head# 2024-04-15T18:42:59.010 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99d19b9:::benchmark_data_smithi159_162467_object4599:head# 2024-04-15T18:42:59.010 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99de360:::benchmark_data_smithi159_162467_object59939:head# 2024-04-15T18:42:59.010 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f99e5899:::benchmark_data_smithi159_162467_object36984:head# 2024-04-15T18:42:59.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a0e3b3:::benchmark_data_smithi159_162467_object72397:head# 2024-04-15T18:42:59.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a22cb7:::benchmark_data_smithi159_162467_object31685:head# 2024-04-15T18:42:59.011 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a32dd3:::benchmark_data_smithi159_162467_object11024:head# 2024-04-15T18:42:59.012 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a3dac0:::benchmark_data_smithi159_162467_object24321:head# 2024-04-15T18:42:59.012 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a414ae:::benchmark_data_smithi159_162467_object62772:head# 2024-04-15T18:42:59.013 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a5e2bb:::benchmark_data_smithi159_162467_object53021:head# 2024-04-15T18:42:59.013 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a62e35:::benchmark_data_smithi159_162467_object17999:head# 2024-04-15T18:42:59.014 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a649a8:::benchmark_data_smithi159_162467_object44374:head# 2024-04-15T18:42:59.014 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9a96189:::benchmark_data_smithi159_162467_object3489:head# 2024-04-15T18:42:59.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9aa0429:::benchmark_data_smithi159_162467_object51268:head# 2024-04-15T18:42:59.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9aa82b7:::benchmark_data_smithi159_162467_object32886:head# 2024-04-15T18:42:59.015 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9aec671:::benchmark_data_smithi159_162467_object21179:head# 2024-04-15T18:42:59.016 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9af46bb:::benchmark_data_smithi159_162467_object59113:head# 2024-04-15T18:42:59.016 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9af98c7:::benchmark_data_smithi159_162467_object41590:head# 2024-04-15T18:42:59.017 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b17395:::benchmark_data_smithi159_162467_object39277:head# 2024-04-15T18:42:59.017 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b1e24b:::benchmark_data_smithi159_162467_object42520:head# 2024-04-15T18:42:59.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b49689:::benchmark_data_smithi159_162467_object51850:head# 2024-04-15T18:42:59.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b4be36:::benchmark_data_smithi159_162467_object30944:head# 2024-04-15T18:42:59.018 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b5adff:::benchmark_data_smithi159_162467_object3568:head# 2024-04-15T18:42:59.019 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b78b4d:::benchmark_data_smithi159_162467_object49855:head# 2024-04-15T18:42:59.019 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9b7fba7:::benchmark_data_smithi159_162467_object38665:head# 2024-04-15T18:42:59.020 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9bd3e58:::benchmark_data_smithi159_162467_object43452:head# 2024-04-15T18:42:59.020 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9beb936:::benchmark_data_smithi159_162467_object37091:head# 2024-04-15T18:42:59.020 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9bfb0ec:::benchmark_data_smithi159_162467_object14383:head# 2024-04-15T18:42:59.021 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9bfef7f:::benchmark_data_smithi159_162467_object30347:head# 2024-04-15T18:42:59.021 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9c1cdda:::benchmark_data_smithi159_162467_object18686:head# 2024-04-15T18:42:59.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9c4952b:::benchmark_data_smithi159_162467_object17917:head# 2024-04-15T18:42:59.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9c653d5:::benchmark_data_smithi159_162467_object29138:head# 2024-04-15T18:42:59.022 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9c9ec6a:::benchmark_data_smithi159_162467_object40707:head# 2024-04-15T18:42:59.023 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ca9eef:::benchmark_data_smithi159_162467_object60674:head# 2024-04-15T18:42:59.023 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cafe85:::benchmark_data_smithi159_162467_object46188:head# 2024-04-15T18:42:59.024 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cb56ea:::benchmark_data_smithi159_162467_object8141:head# 2024-04-15T18:42:59.024 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cc6070:::benchmark_data_smithi159_162467_object36668:head# 2024-04-15T18:42:59.024 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cced08:::benchmark_data_smithi159_162467_object31693:head# 2024-04-15T18:42:59.025 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cd08c9:::benchmark_data_smithi159_162467_object9269:head# 2024-04-15T18:42:59.025 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cd460e:::benchmark_data_smithi159_162467_object45549:head# 2024-04-15T18:42:59.026 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9cf32f5:::benchmark_data_smithi159_162467_object18811:head# 2024-04-15T18:42:59.026 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9d3e528:::benchmark_data_smithi159_162467_object54047:head# 2024-04-15T18:42:59.027 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9d4eaea:::benchmark_data_smithi159_162467_object9745:head# 2024-04-15T18:42:59.027 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9d55810:::benchmark_data_smithi159_162467_object27801:head# 2024-04-15T18:42:59.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9d5b9bd:::benchmark_data_smithi159_162467_object71564:head# 2024-04-15T18:42:59.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9d773a6:::benchmark_data_smithi159_162467_object805:head# 2024-04-15T18:42:59.028 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dade5f:::benchmark_data_smithi159_162467_object57045:head# 2024-04-15T18:42:59.029 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9db456e:::benchmark_data_smithi159_162467_object13969:head# 2024-04-15T18:42:59.029 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9db71e5:::benchmark_data_smithi159_162467_object63056:head# 2024-04-15T18:42:59.030 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dbba79:::benchmark_data_smithi159_162467_object44025:head# 2024-04-15T18:42:59.031 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dc90b0:::benchmark_data_smithi159_162467_object40378:head# 2024-04-15T18:42:59.032 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dcfc7d:::benchmark_data_smithi159_162467_object44854:head# 2024-04-15T18:42:59.033 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dd9bbc:::benchmark_data_smithi159_162467_object23989:head# 2024-04-15T18:42:59.033 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9df90f3:::benchmark_data_smithi159_162467_object71419:head# 2024-04-15T18:42:59.034 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dfd7f0:::benchmark_data_smithi159_162467_object53253:head# 2024-04-15T18:42:59.034 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9dfefee:::benchmark_data_smithi159_162467_object33291:head# 2024-04-15T18:42:59.035 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e0bbcc:::benchmark_data_smithi159_162467_object24440:head# 2024-04-15T18:42:59.035 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e0c840:::benchmark_data_smithi159_162467_object37822:head# 2024-04-15T18:42:59.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e200ed:::benchmark_data_smithi159_162467_object72261:head# 2024-04-15T18:42:59.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e21bbd:::benchmark_data_smithi159_162467_object37828:head# 2024-04-15T18:42:59.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e38189:::benchmark_data_smithi159_162467_object61667:head# 2024-04-15T18:42:59.036 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e41eba:::benchmark_data_smithi159_162467_object71819:head# 2024-04-15T18:42:59.037 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e5ca7d:::benchmark_data_smithi159_162467_object67492:head# 2024-04-15T18:42:59.037 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e5df71:::benchmark_data_smithi159_162467_object3211:head# 2024-04-15T18:42:59.038 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e7ec3d:::benchmark_data_smithi159_162467_object38713:head# 2024-04-15T18:42:59.038 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9e8b518:::benchmark_data_smithi159_162467_object42982:head# 2024-04-15T18:42:59.038 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ea2116:::benchmark_data_smithi159_162467_object24389:head# 2024-04-15T18:42:59.039 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ea625b:::benchmark_data_smithi159_162467_object31634:head# 2024-04-15T18:42:59.039 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9eacbf6:::benchmark_data_smithi159_162467_object11302:head# 2024-04-15T18:42:59.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ec20bd:::benchmark_data_smithi159_162467_object49320:head# 2024-04-15T18:42:59.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ec3c5d:::benchmark_data_smithi159_162467_object25193:head# 2024-04-15T18:42:59.040 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ec8172:::benchmark_data_smithi159_162467_object42177:head# 2024-04-15T18:42:59.041 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9ed78e5:::benchmark_data_smithi159_162467_object71751:head# 2024-04-15T18:42:59.041 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f1bfac:::benchmark_data_smithi159_162467_object11137:head# 2024-04-15T18:42:59.041 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f3b7db:::benchmark_data_smithi159_162467_object4189:head# 2024-04-15T18:42:59.042 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f3d9ee:::benchmark_data_smithi159_162467_object14703:head# 2024-04-15T18:42:59.043 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f52b84:::benchmark_data_smithi159_162467_object2173:head# 2024-04-15T18:42:59.043 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f67ca8:::benchmark_data_smithi159_162467_object9177:head# 2024-04-15T18:42:59.044 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f760b9:::benchmark_data_smithi159_162467_object50217:head# 2024-04-15T18:42:59.044 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f816a9:::benchmark_data_smithi159_162467_object13071:head# 2024-04-15T18:42:59.044 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f84bf4:::benchmark_data_smithi159_162467_object74142:head# 2024-04-15T18:42:59.045 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f87d3d:::benchmark_data_smithi159_162467_object38037:head# 2024-04-15T18:42:59.045 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f91a37:::benchmark_data_smithi159_162467_object43565:head# 2024-04-15T18:42:59.046 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9f9cb51:::benchmark_data_smithi159_162467_object52339:head# 2024-04-15T18:42:59.046 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fabd93:::benchmark_data_smithi159_162467_object68251:head# 2024-04-15T18:42:59.046 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fb2aee:::benchmark_data_smithi159_162467_object29573:head# 2024-04-15T18:42:59.049 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fbd435:::benchmark_data_smithi159_162467_object57360:head# 2024-04-15T18:42:59.049 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fbda4c:::benchmark_data_smithi159_162467_object2622:head# 2024-04-15T18:42:59.050 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fc27cc:::benchmark_data_smithi159_162467_object47598:head# 2024-04-15T18:42:59.050 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fc6a2e:::benchmark_data_smithi159_162467_object67420:head# 2024-04-15T18:42:59.050 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fc76c5:::benchmark_data_smithi159_162467_object42708:head# 2024-04-15T18:42:59.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fd7b2d:::benchmark_data_smithi159_162467_object55410:head# 2024-04-15T18:42:59.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:f9fe5e5f:::benchmark_data_smithi159_162467_object38357:head# 2024-04-15T18:42:59.051 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa014177:::benchmark_data_smithi159_162467_object7567:head# 2024-04-15T18:42:59.052 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0165cc:::benchmark_data_smithi159_162467_object9334:head# 2024-04-15T18:42:59.053 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa01a290:::benchmark_data_smithi159_162467_object8561:head# 2024-04-15T18:42:59.053 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa041694:::benchmark_data_smithi159_162467_object58083:head# 2024-04-15T18:42:59.053 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa04743a:::benchmark_data_smithi159_162467_object15654:head# 2024-04-15T18:42:59.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa04b4cc:::benchmark_data_smithi159_162467_object29349:head# 2024-04-15T18:42:59.054 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa055631:::benchmark_data_smithi159_162467_object4286:head# 2024-04-15T18:42:59.055 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0570b5:::benchmark_data_smithi159_162467_object54794:head# 2024-04-15T18:42:59.055 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa069450:::benchmark_data_smithi159_162467_object47965:head# 2024-04-15T18:42:59.055 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa077a9d:::benchmark_data_smithi159_162467_object40150:head# 2024-04-15T18:42:59.056 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa08b396:::benchmark_data_smithi159_162467_object17389:head# 2024-04-15T18:42:59.057 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0a2c2e:::benchmark_data_smithi159_162467_object41615:head# 2024-04-15T18:42:59.057 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0a455f:::benchmark_data_smithi159_162467_object34097:head# 2024-04-15T18:42:59.057 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0aa74e:::benchmark_data_smithi159_162467_object56952:head# 2024-04-15T18:42:59.058 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0ac18b:::benchmark_data_smithi159_162467_object5345:head# 2024-04-15T18:42:59.058 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0b5015:::benchmark_data_smithi159_162467_object35096:head# 2024-04-15T18:42:59.058 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0bb287:::benchmark_data_smithi159_162467_object34617:head# 2024-04-15T18:42:59.059 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0ca419:::benchmark_data_smithi159_162467_object24717:head# 2024-04-15T18:42:59.059 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0d1f93:::benchmark_data_smithi159_162467_object50949:head# 2024-04-15T18:42:59.060 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa0d2ad9:::benchmark_data_smithi159_162467_object52473:head# 2024-04-15T18:42:59.060 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa107a49:::benchmark_data_smithi159_162467_object58:head# 2024-04-15T18:42:59.061 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa10e7ee:::benchmark_data_smithi159_162467_object53882:head# 2024-04-15T18:42:59.061 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa13049c:::benchmark_data_smithi159_162467_object70283:head# 2024-04-15T18:42:59.062 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa140cc5:::benchmark_data_smithi159_162467_object13091:head# 2024-04-15T18:42:59.067 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa14d0f6:::benchmark_data_smithi159_162467_object71188:head# 2024-04-15T18:42:59.067 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa14e707:::benchmark_data_smithi159_162467_object71720:head# 2024-04-15T18:42:59.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa19b72a:::benchmark_data_smithi159_162467_object46361:head# 2024-04-15T18:42:59.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1c19ff:::benchmark_data_smithi159_162467_object8429:head# 2024-04-15T18:42:59.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1caa8d:::benchmark_data_smithi159_162467_object4256:head# 2024-04-15T18:42:59.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1d61fb:::benchmark_data_smithi159_162467_object37512:head# 2024-04-15T18:42:59.068 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1e5506:::benchmark_data_smithi159_162467_object66747:head# 2024-04-15T18:42:59.069 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1e6c97:::benchmark_data_smithi159_162467_object5608:head# 2024-04-15T18:42:59.069 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa1eeafa:::benchmark_data_smithi159_162467_object60362:head# 2024-04-15T18:42:59.069 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa206c84:::benchmark_data_smithi159_162467_object17472:head# 2024-04-15T18:42:59.069 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa21609b:::benchmark_data_smithi159_162467_object11348:head# 2024-04-15T18:42:59.070 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa21a262:::benchmark_data_smithi159_162467_object67778:head# 2024-04-15T18:42:59.071 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa21e227:::benchmark_data_smithi159_162467_object66904:head# 2024-04-15T18:42:59.072 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa24bdd7:::benchmark_data_smithi159_162467_object66249:head# 2024-04-15T18:42:59.072 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa24f371:::benchmark_data_smithi159_162467_object64432:head# 2024-04-15T18:42:59.073 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa254f63:::benchmark_data_smithi159_162467_object16666:head# 2024-04-15T18:42:59.073 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa255f23:::benchmark_data_smithi159_162467_object4331:head# 2024-04-15T18:42:59.074 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa256bc4:::benchmark_data_smithi159_162467_object56444:head# 2024-04-15T18:42:59.078 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa25c0e6:::benchmark_data_smithi159_162467_object59794:head# 2024-04-15T18:42:59.078 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa25d079:::benchmark_data_smithi159_162467_object34785:head# 2024-04-15T18:42:59.078 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa26b36f:::benchmark_data_smithi159_162467_object10108:head# 2024-04-15T18:42:59.078 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa26c0ab:::benchmark_data_smithi159_162467_object31448:head# 2024-04-15T18:42:59.078 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa272c23:::benchmark_data_smithi159_162467_object6501:head# 2024-04-15T18:42:59.079 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2a05c8:::benchmark_data_smithi159_162467_object13283:head# 2024-04-15T18:42:59.079 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2aeac2:::benchmark_data_smithi159_162467_object9866:head# 2024-04-15T18:42:59.079 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2b6516:::benchmark_data_smithi159_162467_object40749:head# 2024-04-15T18:42:59.079 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2b7028:::benchmark_data_smithi159_162467_object23167:head# 2024-04-15T18:42:59.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2c02be:::benchmark_data_smithi159_162467_object18318:head# 2024-04-15T18:42:59.080 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2d7515:::benchmark_data_smithi159_162467_object57409:head# 2024-04-15T18:42:59.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa2fac24:::benchmark_data_smithi159_162467_object22226:head# 2024-04-15T18:42:59.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa31b38b:::benchmark_data_smithi159_162467_object10966:head# 2024-04-15T18:42:59.081 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa33b0ca:::benchmark_data_smithi159_162467_object3687:head# 2024-04-15T18:42:59.082 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa343278:::benchmark_data_smithi159_162467_object37622:head# 2024-04-15T18:42:59.083 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa35ac56:::benchmark_data_smithi159_162467_object72837:head# 2024-04-15T18:42:59.083 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa360818:::benchmark_data_smithi159_162467_object26698:head# 2024-04-15T18:42:59.083 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3714de:::benchmark_data_smithi159_162467_object11902:head# 2024-04-15T18:42:59.086 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3730a1:::benchmark_data_smithi159_162467_object69316:head# 2024-04-15T18:42:59.087 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa386649:::benchmark_data_smithi159_162467_object51193:head# 2024-04-15T18:42:59.087 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa39d57d:::benchmark_data_smithi159_162467_object70568:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa39dbad:::benchmark_data_smithi159_162467_object38285:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3a5f5a:::benchmark_data_smithi159_162467_object21950:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3af852:::benchmark_data_smithi159_162467_object41000:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3bb2a4:::benchmark_data_smithi159_162467_object25998:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3c6f52:::benchmark_data_smithi159_162467_object6759:head# 2024-04-15T18:42:59.092 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3c8bc2:::benchmark_data_smithi159_162467_object20213:head# 2024-04-15T18:42:59.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3cfdeb:::benchmark_data_smithi159_162467_object49580:head# 2024-04-15T18:42:59.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa3e0c95:::benchmark_data_smithi159_162467_object39287:head# 2024-04-15T18:42:59.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa408fdc:::benchmark_data_smithi159_162467_object45668:head# 2024-04-15T18:42:59.093 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa4470fc:::benchmark_data_smithi159_162467_object41223:head# 2024-04-15T18:42:59.094 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa448c29:::benchmark_data_smithi159_162467_object35538:head# 2024-04-15T18:42:59.094 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa45ddca:::benchmark_data_smithi159_162467_object71756:head# 2024-04-15T18:42:59.095 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa45df7e:::benchmark_data_smithi159_162467_object61990:head# 2024-04-15T18:42:59.095 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa467a64:::benchmark_data_smithi159_162467_object14061:head# 2024-04-15T18:42:59.096 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa4709bf:::benchmark_data_smithi159_162467_object32364:head# 2024-04-15T18:42:59.096 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa47dc93:::benchmark_data_smithi159_162467_object63753:head# 2024-04-15T18:42:59.096 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa48016c:::benchmark_data_smithi159_162467_object56287:head# 2024-04-15T18:42:59.097 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa49888f:::benchmark_data_smithi159_162467_object44837:head# 2024-04-15T18:42:59.098 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa4b7880:::benchmark_data_smithi159_162467_object14949:head# 2024-04-15T18:42:59.098 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa4f7971:::benchmark_data_smithi159_162467_object51692:head# 2024-04-15T18:42:59.098 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa4fae0d:::benchmark_data_smithi159_162467_object56623:head# 2024-04-15T18:42:59.099 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa53b8c7:::benchmark_data_smithi159_162467_object8810:head# 2024-04-15T18:42:59.099 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa545fd4:::benchmark_data_smithi159_162467_object20830:head# 2024-04-15T18:42:59.099 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa558120:::benchmark_data_smithi159_162467_object20017:head# 2024-04-15T18:42:59.100 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa55e728:::benchmark_data_smithi159_162467_object28711:head# 2024-04-15T18:42:59.100 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa56b26c:::benchmark_data_smithi159_162467_object63065:head# 2024-04-15T18:42:59.100 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa56e54d:::benchmark_data_smithi159_162467_object54398:head# 2024-04-15T18:42:59.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa584387:::benchmark_data_smithi159_162467_object20775:head# 2024-04-15T18:42:59.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa588f9a:::benchmark_data_smithi159_162467_object66311:head# 2024-04-15T18:42:59.101 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa58d715:::benchmark_data_smithi159_162467_object27956:head# 2024-04-15T18:42:59.102 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5aec6c:::benchmark_data_smithi159_162467_object17848:head# 2024-04-15T18:42:59.104 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5c622f:::benchmark_data_smithi159_162467_object67663:head# 2024-04-15T18:42:59.104 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5cb0c6:::benchmark_data_smithi159_162467_object33767:head# 2024-04-15T18:42:59.104 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5d12da:::benchmark_data_smithi159_162467_object33322:head# 2024-04-15T18:42:59.105 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5d3149:::benchmark_data_smithi159_162467_object21837:head# 2024-04-15T18:42:59.105 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5d6b23:::benchmark_data_smithi159_162467_object15953:head# 2024-04-15T18:42:59.106 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5e9bed:::benchmark_data_smithi159_162467_object47862:head# 2024-04-15T18:42:59.106 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5f7bc6:::benchmark_data_smithi159_162467_object33528:head# 2024-04-15T18:42:59.107 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa5fb720:::benchmark_data_smithi159_162467_object36134:head# 2024-04-15T18:42:59.107 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa600f1f:::benchmark_data_smithi159_162467_object18362:head# 2024-04-15T18:42:59.108 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa60dde8:::benchmark_data_smithi159_162467_object38379:head# 2024-04-15T18:42:59.108 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa612781:::benchmark_data_smithi159_162467_object53940:head# 2024-04-15T18:42:59.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa61acef:::benchmark_data_smithi159_162467_object57270:head# 2024-04-15T18:42:59.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa623d76:::benchmark_data_smithi159_162467_object62995:head# 2024-04-15T18:42:59.109 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa65f5ec:::benchmark_data_smithi159_162467_object30021:head# 2024-04-15T18:42:59.110 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa667d69:::benchmark_data_smithi159_162467_object44786:head# 2024-04-15T18:42:59.110 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6851da:::benchmark_data_smithi159_162467_object58339:head# 2024-04-15T18:42:59.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa695584:::benchmark_data_smithi159_162467_object13977:head# 2024-04-15T18:42:59.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6968ed:::benchmark_data_smithi159_162467_object17712:head# 2024-04-15T18:42:59.111 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa699474:::benchmark_data_smithi159_162467_object32288:head# 2024-04-15T18:42:59.112 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa69ce82:::benchmark_data_smithi159_162467_object74726:head# 2024-04-15T18:42:59.112 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6a3122:::benchmark_data_smithi159_162467_object32312:head# 2024-04-15T18:42:59.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6aedc9:::benchmark_data_smithi159_162467_object20957:head# 2024-04-15T18:42:59.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6b1e42:::benchmark_data_smithi159_162467_object37641:head# 2024-04-15T18:42:59.115 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6b3172:::benchmark_data_smithi159_162467_object21074:head# 2024-04-15T18:42:59.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6b4ecd:::benchmark_data_smithi159_162467_object15003:head# 2024-04-15T18:42:59.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6b8a09:::benchmark_data_smithi159_162467_object73200:head# 2024-04-15T18:42:59.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6c50f4:::benchmark_data_smithi159_162467_object72902:head# 2024-04-15T18:42:59.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6cc904:::benchmark_data_smithi159_162467_object50745:head# 2024-04-15T18:42:59.116 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6d78c4:::benchmark_data_smithi159_162467_object58774:head# 2024-04-15T18:42:59.117 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6f0b91:::benchmark_data_smithi159_162467_object39382:head# 2024-04-15T18:42:59.117 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6f276d:::benchmark_data_smithi159_162467_object18364:head# 2024-04-15T18:42:59.117 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa6f2b47:::benchmark_data_smithi159_162467_object71347:head# 2024-04-15T18:42:59.118 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa712601:::benchmark_data_smithi159_162467_object64197:head# 2024-04-15T18:42:59.118 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7236f6:::benchmark_data_smithi159_162467_object11422:head# 2024-04-15T18:42:59.118 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa72a2e2:::benchmark_data_smithi159_162467_object51223:head# 2024-04-15T18:42:59.119 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa72a7d8:::benchmark_data_smithi159_162467_object7101:head# 2024-04-15T18:42:59.119 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa72eece:::benchmark_data_smithi159_162467_object30381:head# 2024-04-15T18:42:59.120 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa75f493:::benchmark_data_smithi159_162467_object66083:head# 2024-04-15T18:42:59.120 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa771421:::benchmark_data_smithi159_162467_object19305:head# 2024-04-15T18:42:59.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa775af8:::benchmark_data_smithi159_162467_object21751:head# 2024-04-15T18:42:59.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa77b0a0:::benchmark_data_smithi159_162467_object8630:head# 2024-04-15T18:42:59.121 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa78823b:::benchmark_data_smithi159_162467_object25458:head# 2024-04-15T18:42:59.122 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa78c4a3:::benchmark_data_smithi159_162467_object17170:head# 2024-04-15T18:42:59.122 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa795aa9:::benchmark_data_smithi159_162467_object3226:head# 2024-04-15T18:42:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7a82aa:::benchmark_data_smithi159_162467_object29184:head# 2024-04-15T18:42:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7b1596:::benchmark_data_smithi159_162467_object46166:head# 2024-04-15T18:42:59.123 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7d0b83:::benchmark_data_smithi159_162467_object17:head# 2024-04-15T18:42:59.124 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7d3eb2:::benchmark_data_smithi159_162467_object55139:head# 2024-04-15T18:42:59.124 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7e10ae:::benchmark_data_smithi159_162467_object63424:head# 2024-04-15T18:42:59.127 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7e25d9:::benchmark_data_smithi159_162467_object6184:head# 2024-04-15T18:42:59.127 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7e68b4:::benchmark_data_smithi159_162467_object45998:head# 2024-04-15T18:42:59.129 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7f00c9:::benchmark_data_smithi159_162467_object4950:head# 2024-04-15T18:42:59.129 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa7f76d8:::benchmark_data_smithi159_162467_object49570:head# 2024-04-15T18:42:59.130 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa801e7d:::benchmark_data_smithi159_162467_object40677:head# 2024-04-15T18:42:59.130 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8036f2:::benchmark_data_smithi159_162467_object54266:head# 2024-04-15T18:42:59.130 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa810fe3:::benchmark_data_smithi159_162467_object29600:head# 2024-04-15T18:42:59.130 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa815fe5:::benchmark_data_smithi159_162467_object69491:head# 2024-04-15T18:42:59.131 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa830178:::benchmark_data_smithi159_162467_object22863:head# 2024-04-15T18:42:59.131 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa835966:::benchmark_data_smithi159_162467_object45421:head# 2024-04-15T18:42:59.131 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa844dc3:::benchmark_data_smithi159_162467_object57569:head# 2024-04-15T18:42:59.132 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8675c1:::benchmark_data_smithi159_162467_object20647:head# 2024-04-15T18:42:59.132 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa875b28:::benchmark_data_smithi159_162467_object23996:head# 2024-04-15T18:42:59.133 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8850ec:::benchmark_data_smithi159_162467_object42790:head# 2024-04-15T18:42:59.133 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8906bd:::benchmark_data_smithi159_162467_object30160:head# 2024-04-15T18:42:59.133 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa893bd3:::benchmark_data_smithi159_162467_object7917:head# 2024-04-15T18:42:59.133 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8a42c6:::benchmark_data_smithi159_162467_object40606:head# 2024-04-15T18:42:59.134 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8b220e:::benchmark_data_smithi159_162467_object55040:head# 2024-04-15T18:42:59.134 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8b7227:::benchmark_data_smithi159_162467_object67013:head# 2024-04-15T18:42:59.135 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8b827a:::benchmark_data_smithi159_162467_object12008:head# 2024-04-15T18:42:59.135 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8d0675:::benchmark_data_smithi159_162467_object1047:head# 2024-04-15T18:42:59.135 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8d674f:::benchmark_data_smithi159_162467_object50470:head# 2024-04-15T18:42:59.136 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8ea5d0:::benchmark_data_smithi159_162467_object47689:head# 2024-04-15T18:42:59.136 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8edb33:::benchmark_data_smithi159_162467_object74323:head# 2024-04-15T18:42:59.137 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa8f5000:::benchmark_data_smithi159_162467_object23337:head# 2024-04-15T18:42:59.137 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa910eb2:::benchmark_data_smithi159_162467_object62260:head# 2024-04-15T18:42:59.137 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa912f8e:::benchmark_data_smithi159_162467_object31649:head# 2024-04-15T18:42:59.138 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa92c42d:::benchmark_data_smithi159_162467_object1790:head# 2024-04-15T18:42:59.138 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa95570e:::benchmark_data_smithi159_162467_object32916:head# 2024-04-15T18:42:59.138 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa956c3d:::benchmark_data_smithi159_162467_object224:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa960c35:::benchmark_data_smithi159_162467_object50930:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa979422:::benchmark_data_smithi159_162467_object48701:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa98034e:::benchmark_data_smithi159_162467_object26395:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa989fae:::benchmark_data_smithi159_162467_object58548:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa99975c:::benchmark_data_smithi159_162467_object67506:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9a2676:::benchmark_data_smithi159_162467_object31820:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9b42a2:::benchmark_data_smithi159_162467_object4417:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9b694f:::benchmark_data_smithi159_162467_object34513:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9c323e:::benchmark_data_smithi159_162467_object16962:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9c32d5:::benchmark_data_smithi159_162467_object37424:head# 2024-04-15T18:42:59.143 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9c4927:::benchmark_data_smithi159_162467_object72651:head# 2024-04-15T18:42:59.144 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9de552:::benchmark_data_smithi159_162467_object72127:head# 2024-04-15T18:42:59.144 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9f3c1e:::benchmark_data_smithi159_162467_object22954:head# 2024-04-15T18:42:59.145 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fa9fdf2b:::benchmark_data_smithi159_162467_object49323:head# 2024-04-15T18:42:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:58 smithi159 ceph-mon[45336]: pgmap v1932: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 5.7 MiB/s wr, 573 op/s; 27775/295338 objects degraded (9.404%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:42:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:42:58 smithi159 ceph-mon[45336]: osdmap e749: 12 total, 11 up, 11 in 2024-04-15T18:42:59.146 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa02a9c:::benchmark_data_smithi159_162467_object26288:head# 2024-04-15T18:42:59.146 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa1bb6f:::benchmark_data_smithi159_162467_object61141:head# 2024-04-15T18:42:59.146 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa31525:::benchmark_data_smithi159_162467_object20954:head# 2024-04-15T18:42:59.146 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa33615:::benchmark_data_smithi159_162467_object40446:head# 2024-04-15T18:42:59.146 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa4151f:::benchmark_data_smithi159_162467_object30006:head# 2024-04-15T18:42:59.147 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa4a85a:::benchmark_data_smithi159_162467_object1614:head# 2024-04-15T18:42:59.147 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa5636d:::benchmark_data_smithi159_162467_object40031:head# 2024-04-15T18:42:59.147 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa5d1ae:::benchmark_data_smithi159_162467_object26697:head# 2024-04-15T18:42:59.148 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa77890:::benchmark_data_smithi159_162467_object2613:head# 2024-04-15T18:42:59.148 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa8304e:::benchmark_data_smithi159_162467_object43473:head# 2024-04-15T18:42:59.149 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faa90b26:::benchmark_data_smithi159_162467_object74701:head# 2024-04-15T18:42:59.149 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faab8d76:::benchmark_data_smithi159_162467_object23136:head# 2024-04-15T18:42:59.149 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faabfecc:::benchmark_data_smithi159_162467_object52695:head# 2024-04-15T18:42:59.150 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faac794e:::benchmark_data_smithi159_162467_object61750:head# 2024-04-15T18:42:59.150 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faad50e9:::benchmark_data_smithi159_162467_object5403:head# 2024-04-15T18:42:59.150 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faade5b6:::benchmark_data_smithi159_162467_object74001:head# 2024-04-15T18:42:59.151 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faaff3ab:::benchmark_data_smithi159_162467_object53472:head# 2024-04-15T18:42:59.151 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab106a8:::benchmark_data_smithi159_162467_object63477:head# 2024-04-15T18:42:59.151 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab30fce:::benchmark_data_smithi159_162467_object66625:head# 2024-04-15T18:42:59.152 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab43e32:::benchmark_data_smithi159_162467_object37954:head# 2024-04-15T18:42:59.152 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab4c75c:::benchmark_data_smithi159_162467_object69802:head# 2024-04-15T18:42:59.152 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab6ef06:::benchmark_data_smithi159_162467_object36429:head# 2024-04-15T18:42:59.153 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab89afb:::benchmark_data_smithi159_162467_object3846:head# 2024-04-15T18:42:59.153 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab992c5:::benchmark_data_smithi159_162467_object2467:head# 2024-04-15T18:42:59.154 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fab9c17b:::benchmark_data_smithi159_162467_object30195:head# 2024-04-15T18:42:59.154 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fabaf665:::benchmark_data_smithi159_162467_object36143:head# 2024-04-15T18:42:59.155 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fabc110d:::benchmark_data_smithi159_162467_object48383:head# 2024-04-15T18:42:59.155 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fabd9ba5:::benchmark_data_smithi159_162467_object42208:head# 2024-04-15T18:42:59.155 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fabdec57:::benchmark_data_smithi159_162467_object32605:head# 2024-04-15T18:42:59.156 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac41a41:::benchmark_data_smithi159_162467_object23414:head# 2024-04-15T18:42:59.156 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac47db3:::benchmark_data_smithi159_162467_object73877:head# 2024-04-15T18:42:59.157 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac4dec6:::benchmark_data_smithi159_162467_object10982:head# 2024-04-15T18:42:59.157 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac55f96:::benchmark_data_smithi159_162467_object65503:head# 2024-04-15T18:42:59.157 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac5d3da:::benchmark_data_smithi159_162467_object73866:head# 2024-04-15T18:42:59.158 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac6310f:::benchmark_data_smithi159_162467_object54539:head# 2024-04-15T18:42:59.158 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac6891b:::benchmark_data_smithi159_162467_object56040:head# 2024-04-15T18:42:59.159 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac6f935:::benchmark_data_smithi159_162467_object14248:head# 2024-04-15T18:42:59.159 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac800ec:::benchmark_data_smithi159_162467_object49632:head# 2024-04-15T18:42:59.160 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac8d2c7:::benchmark_data_smithi159_162467_object5226:head# 2024-04-15T18:42:59.160 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fac8d449:::benchmark_data_smithi159_162467_object53296:head# 2024-04-15T18:42:59.160 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:facb17f2:::benchmark_data_smithi159_162467_object6462:head# 2024-04-15T18:42:59.160 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:facb60f7:::benchmark_data_smithi159_162467_object22179:head# 2024-04-15T18:42:59.161 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:facd8783:::benchmark_data_smithi159_162467_object46160:head# 2024-04-15T18:42:59.161 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faceeb7c:::benchmark_data_smithi159_162467_object18285:head# 2024-04-15T18:42:59.162 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:facf014a:::benchmark_data_smithi159_162467_object60709:head# 2024-04-15T18:42:59.162 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:facf85d7:::benchmark_data_smithi159_162467_object68784:head# 2024-04-15T18:42:59.162 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad04f7d:::benchmark_data_smithi159_162467_object46646:head# 2024-04-15T18:42:59.163 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad05417:::benchmark_data_smithi159_162467_object57192:head# 2024-04-15T18:42:59.163 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad0e782:::benchmark_data_smithi159_162467_object37163:head# 2024-04-15T18:42:59.163 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad1d254:::benchmark_data_smithi159_162467_object27718:head# 2024-04-15T18:42:59.164 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad29bfb:::benchmark_data_smithi159_162467_object1800:head# 2024-04-15T18:42:59.164 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad35d4a:::benchmark_data_smithi159_162467_object21930:head# 2024-04-15T18:42:59.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:58 smithi049 ceph-mon[40087]: pgmap v1932: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 5.7 MiB/s wr, 573 op/s; 27775/295338 objects degraded (9.404%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:42:59.165 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:42:58 smithi049 ceph-mon[40087]: osdmap e749: 12 total, 11 up, 11 in 2024-04-15T18:42:59.166 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad3f111:::benchmark_data_smithi159_162467_object66503:head# 2024-04-15T18:42:59.166 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad408fb:::benchmark_data_smithi159_162467_object43742:head# 2024-04-15T18:42:59.166 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad60338:::benchmark_data_smithi159_162467_object4566:head# 2024-04-15T18:42:59.167 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad60a17:::benchmark_data_smithi159_162467_object25495:head# 2024-04-15T18:42:59.167 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad65c79:::benchmark_data_smithi159_162467_object57921:head# 2024-04-15T18:42:59.167 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad68de8:::benchmark_data_smithi159_162467_object34049:head# 2024-04-15T18:42:59.167 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad6c403:::benchmark_data_smithi159_162467_object41255:head# 2024-04-15T18:42:59.168 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad7e318:::benchmark_data_smithi159_162467_object53069:head# 2024-04-15T18:42:59.168 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad7fed9:::benchmark_data_smithi159_162467_object3794:head# 2024-04-15T18:42:59.169 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad80554:::benchmark_data_smithi159_162467_object26817:head# 2024-04-15T18:42:59.169 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fad9b2b6:::benchmark_data_smithi159_162467_object72545:head# 2024-04-15T18:42:59.170 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadbaf9f:::benchmark_data_smithi159_162467_object61807:head# 2024-04-15T18:42:59.170 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadc07f5:::benchmark_data_smithi159_162467_object67479:head# 2024-04-15T18:42:59.170 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadc0c63:::benchmark_data_smithi159_162467_object17898:head# 2024-04-15T18:42:59.171 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadc53e5:::benchmark_data_smithi159_162467_object22154:head# 2024-04-15T18:42:59.172 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadc9bdd:::benchmark_data_smithi159_162467_object51130:head# 2024-04-15T18:42:59.172 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fade5ff0:::benchmark_data_smithi159_162467_object36606:head# 2024-04-15T18:42:59.173 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadf5a80:::benchmark_data_smithi159_162467_object46885:head# 2024-04-15T18:42:59.173 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fadfaa56:::benchmark_data_smithi159_162467_object74023:head# 2024-04-15T18:42:59.173 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae120e1:::benchmark_data_smithi159_162467_object51681:head# 2024-04-15T18:42:59.174 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae2b593:::benchmark_data_smithi159_162467_object42294:head# 2024-04-15T18:42:59.174 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae34abe:::benchmark_data_smithi159_162467_object20233:head# 2024-04-15T18:42:59.174 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae65f1b:::benchmark_data_smithi159_162467_object20024:head# 2024-04-15T18:42:59.175 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae7026f:::benchmark_data_smithi159_162467_object66057:head# 2024-04-15T18:42:59.175 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae7253e:::benchmark_data_smithi159_162467_object9323:head# 2024-04-15T18:42:59.176 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae757b4:::benchmark_data_smithi159_162467_object43405:head# 2024-04-15T18:42:59.177 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae7e21a:::benchmark_data_smithi159_162467_object49823:head# 2024-04-15T18:42:59.177 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae7e729:::benchmark_data_smithi159_162467_object49692:head# 2024-04-15T18:42:59.177 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae7e75f:::benchmark_data_smithi159_162467_object11449:head# 2024-04-15T18:42:59.178 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fae991fa:::benchmark_data_smithi159_162467_object45232:head# 2024-04-15T18:42:59.178 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faeadde6:::benchmark_data_smithi159_162467_object12792:head# 2024-04-15T18:42:59.179 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faeb5421:::benchmark_data_smithi159_162467_object67783:head# 2024-04-15T18:42:59.179 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faec47ef:::benchmark_data_smithi159_162467_object50615:head# 2024-04-15T18:42:59.179 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faeddf23:::benchmark_data_smithi159_162467_object4252:head# 2024-04-15T18:42:59.180 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faedec3b:::benchmark_data_smithi159_162467_object11365:head# 2024-04-15T18:42:59.180 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faeeed06:::benchmark_data_smithi159_162467_object30659:head# 2024-04-15T18:42:59.181 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faef5bb6:::benchmark_data_smithi159_162467_object51400:head# 2024-04-15T18:42:59.181 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf0abb2:::benchmark_data_smithi159_162467_object23704:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf0e849:::benchmark_data_smithi159_162467_object24729:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf16dd0:::benchmark_data_smithi159_162467_object17879:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf2621e:::benchmark_data_smithi159_162467_object7551:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf2ad67:::benchmark_data_smithi159_162467_object16828:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf2b7c9:::benchmark_data_smithi159_162467_object4106:head# 2024-04-15T18:42:59.184 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf34865:::benchmark_data_smithi159_162467_object18498:head# 2024-04-15T18:42:59.185 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf368bc:::benchmark_data_smithi159_162467_object9669:head# 2024-04-15T18:42:59.185 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf524c2:::benchmark_data_smithi159_162467_object9419:head# 2024-04-15T18:42:59.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:42:59.186 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf66539:::benchmark_data_smithi159_162467_object13295:head# 2024-04-15T18:42:59.186 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf8e84e:::benchmark_data_smithi159_162467_object57434:head# 2024-04-15T18:42:59.187 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf8e889:::benchmark_data_smithi159_162467_object25973:head# 2024-04-15T18:42:59.187 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faf9cbc4:::benchmark_data_smithi159_162467_object23218:head# 2024-04-15T18:42:59.188 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafa8dad:::benchmark_data_smithi159_162467_object24935:head# 2024-04-15T18:42:59.188 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafafef0:::benchmark_data_smithi159_162467_object11020:head# 2024-04-15T18:42:59.188 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafb03c2:::benchmark_data_smithi159_162467_object51737:head# 2024-04-15T18:42:59.189 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafc96e1:::benchmark_data_smithi159_162467_object1083:head# 2024-04-15T18:42:59.189 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafe0cf1:::benchmark_data_smithi159_162467_object61236:head# 2024-04-15T18:42:59.190 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafe2405:::benchmark_data_smithi159_162467_object8182:head# 2024-04-15T18:42:59.190 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fafeb876:::benchmark_data_smithi159_162467_object61462:head# 2024-04-15T18:42:59.191 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:faff04a8:::benchmark_data_smithi159_162467_object16077:head# 2024-04-15T18:42:59.191 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb015003:::benchmark_data_smithi159_162467_object37016:head# 2024-04-15T18:42:59.192 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb05a6c8:::benchmark_data_smithi159_162467_object72109:head# 2024-04-15T18:42:59.192 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb05ffcc:::benchmark_data_smithi159_162467_object41405:head# 2024-04-15T18:42:59.192 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb066ec7:::benchmark_data_smithi159_162467_object73634:head# 2024-04-15T18:42:59.193 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb06778f:::benchmark_data_smithi159_162467_object67977:head# 2024-04-15T18:42:59.193 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb08c42b:::benchmark_data_smithi159_162467_object49611:head# 2024-04-15T18:42:59.193 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb093d35:::benchmark_data_smithi159_162467_object27495:head# 2024-04-15T18:42:59.194 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb098ffb:::benchmark_data_smithi159_162467_object33438:head# 2024-04-15T18:42:59.194 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb09c15c:::benchmark_data_smithi159_162467_object18130:head# 2024-04-15T18:42:59.194 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0a1d30:::benchmark_data_smithi159_162467_object25956:head# 2024-04-15T18:42:59.195 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0b3364:::benchmark_data_smithi159_162467_object2083:head# 2024-04-15T18:42:59.195 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0c12ec:::benchmark_data_smithi159_162467_object45218:head# 2024-04-15T18:42:59.195 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0c2c55:::benchmark_data_smithi159_162467_object16799:head# 2024-04-15T18:42:59.196 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0d294b:::benchmark_data_smithi159_162467_object15259:head# 2024-04-15T18:42:59.197 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0dadbf:::benchmark_data_smithi159_162467_object16105:head# 2024-04-15T18:42:59.197 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0e55b2:::benchmark_data_smithi159_162467_object63815:head# 2024-04-15T18:42:59.198 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0e96d7:::benchmark_data_smithi159_162467_object20464:head# 2024-04-15T18:42:59.198 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0fb6d1:::benchmark_data_smithi159_162467_object50441:head# 2024-04-15T18:42:59.199 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0fe585:::benchmark_data_smithi159_162467_object6643:head# 2024-04-15T18:42:59.199 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb0ff86f:::benchmark_data_smithi159_162467_object35671:head# 2024-04-15T18:42:59.199 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1063a6:::benchmark_data_smithi159_162467_object37881:head# 2024-04-15T18:42:59.200 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb11222b:::benchmark_data_smithi159_162467_object67915:head# 2024-04-15T18:42:59.200 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb115deb:::benchmark_data_smithi159_162467_object18590:head# 2024-04-15T18:42:59.201 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb11c7fe:::benchmark_data_smithi159_162467_object16232:head# 2024-04-15T18:42:59.201 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb11c80c:::benchmark_data_smithi159_162467_object53497:head# 2024-04-15T18:42:59.201 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb11ee23:::benchmark_data_smithi159_162467_object25848:head# 2024-04-15T18:42:59.201 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb13caf0:::benchmark_data_smithi159_162467_object65584:head# 2024-04-15T18:42:59.202 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb163f1e:::benchmark_data_smithi159_162467_object54978:head# 2024-04-15T18:42:59.203 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb185fb0:::benchmark_data_smithi159_162467_object36058:head# 2024-04-15T18:42:59.203 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb19f7a3:::benchmark_data_smithi159_162467_object25255:head# 2024-04-15T18:42:59.203 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1afcab:::benchmark_data_smithi159_162467_object32111:head# 2024-04-15T18:42:59.204 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1b47bd:::benchmark_data_smithi159_162467_object68761:head# 2024-04-15T18:42:59.207 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1b5aec:::benchmark_data_smithi159_162467_object46658:head# 2024-04-15T18:42:59.207 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1bd0ff:::benchmark_data_smithi159_162467_object37937:head# 2024-04-15T18:42:59.208 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1dcf70:::benchmark_data_smithi159_162467_object64111:head# 2024-04-15T18:42:59.208 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1ddd42:::benchmark_data_smithi159_162467_object2447:head# 2024-04-15T18:42:59.208 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1ed7d3:::benchmark_data_smithi159_162467_object43110:head# 2024-04-15T18:42:59.209 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb1ef171:::benchmark_data_smithi159_162467_object54281:head# 2024-04-15T18:42:59.210 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb206db5:::benchmark_data_smithi159_162467_object36951:head# 2024-04-15T18:42:59.210 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb22671d:::benchmark_data_smithi159_162467_object71732:head# 2024-04-15T18:42:59.210 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2280f6:::benchmark_data_smithi159_162467_object60386:head# 2024-04-15T18:42:59.211 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb237b0f:::benchmark_data_smithi159_162467_object46094:head# 2024-04-15T18:42:59.211 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb267b0a:::benchmark_data_smithi159_162467_object12785:head# 2024-04-15T18:42:59.212 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2694f3:::benchmark_data_smithi159_162467_object1947:head# 2024-04-15T18:42:59.212 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb26d4b3:::benchmark_data_smithi159_162467_object27798:head# 2024-04-15T18:42:59.213 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb26d69a:::benchmark_data_smithi159_162467_object2652:head# 2024-04-15T18:42:59.213 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb28e4ed:::benchmark_data_smithi159_162467_object11315:head# 2024-04-15T18:42:59.214 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2904b7:::benchmark_data_smithi159_162467_object15542:head# 2024-04-15T18:42:59.214 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2982bf:::benchmark_data_smithi159_162467_object30141:head# 2024-04-15T18:42:59.214 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb29db14:::benchmark_data_smithi159_162467_object63329:head# 2024-04-15T18:42:59.215 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2ab101:::benchmark_data_smithi159_162467_object54548:head# 2024-04-15T18:42:59.215 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2aed08:::benchmark_data_smithi159_162467_object16226:head# 2024-04-15T18:42:59.215 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2b747e:::benchmark_data_smithi159_162467_object60412:head# 2024-04-15T18:42:59.216 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2c22be:::benchmark_data_smithi159_162467_object21970:head# 2024-04-15T18:42:59.216 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2cbc50:::benchmark_data_smithi159_162467_object25086:head# 2024-04-15T18:42:59.217 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2d4480:::benchmark_data_smithi159_162467_object68219:head# 2024-04-15T18:42:59.217 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2d4c60:::benchmark_data_smithi159_162467_object73262:head# 2024-04-15T18:42:59.220 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2e5a94:::benchmark_data_smithi159_162467_object62761:head# 2024-04-15T18:42:59.221 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2efb26:::benchmark_data_smithi159_162467_object71103:head# 2024-04-15T18:42:59.221 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2f75af:::benchmark_data_smithi159_162467_object54763:head# 2024-04-15T18:42:59.222 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb2ff4ed:::benchmark_data_smithi159_162467_object33739:head# 2024-04-15T18:42:59.223 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb30fe4d:::benchmark_data_smithi159_162467_object72806:head# 2024-04-15T18:42:59.224 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb310fdf:::benchmark_data_smithi159_162467_object47259:head# 2024-04-15T18:42:59.224 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb315ec5:::benchmark_data_smithi159_162467_object1990:head# 2024-04-15T18:42:59.225 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb31e90b:::benchmark_data_smithi159_162467_object16034:head# 2024-04-15T18:42:59.225 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb330c14:::benchmark_data_smithi159_162467_object36600:head# 2024-04-15T18:42:59.226 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb33547d:::benchmark_data_smithi159_162467_object58260:head# 2024-04-15T18:42:59.226 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb36136d:::benchmark_data_smithi159_162467_object35546:head# 2024-04-15T18:42:59.226 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb36c9fc:::benchmark_data_smithi159_162467_object40477:head# 2024-04-15T18:42:59.226 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb36cddf:::benchmark_data_smithi159_162467_object65110:head# 2024-04-15T18:42:59.229 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb38edcf:::benchmark_data_smithi159_162467_object53949:head# 2024-04-15T18:42:59.229 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb391064:::benchmark_data_smithi159_162467_object6459:head# 2024-04-15T18:42:59.230 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb39d41c:::benchmark_data_smithi159_162467_object50592:head# 2024-04-15T18:42:59.231 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb3abe2f:::benchmark_data_smithi159_162467_object31334:head# 2024-04-15T18:42:59.231 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb3d1e28:::benchmark_data_smithi159_162467_object57477:head# 2024-04-15T18:42:59.231 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb3eb825:::benchmark_data_smithi159_162467_object38748:head# 2024-04-15T18:42:59.232 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb40216f:::benchmark_data_smithi159_162467_object5827:head# 2024-04-15T18:42:59.233 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb403dca:::benchmark_data_smithi159_162467_object27637:head# 2024-04-15T18:42:59.234 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb408431:::benchmark_data_smithi159_162467_object56607:head# 2024-04-15T18:42:59.234 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb41306a:::benchmark_data_smithi159_162467_object68988:head# 2024-04-15T18:42:59.234 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb41d3ec:::benchmark_data_smithi159_162467_object73967:head# 2024-04-15T18:42:59.235 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb422489:::benchmark_data_smithi159_162467_object19807:head# 2024-04-15T18:42:59.235 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb48e07b:::benchmark_data_smithi159_162467_object13523:head# 2024-04-15T18:42:59.236 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb4b5ad5:::benchmark_data_smithi159_162467_object21126:head# 2024-04-15T18:42:59.236 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb4c78a9:::benchmark_data_smithi159_162467_object52202:head# 2024-04-15T18:42:59.236 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb4d3f8f:::benchmark_data_smithi159_162467_object49942:head# 2024-04-15T18:42:59.237 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb4d5d63:::benchmark_data_smithi159_162467_object42940:head# 2024-04-15T18:42:59.237 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb501bf5:::benchmark_data_smithi159_162467_object65601:head# 2024-04-15T18:42:59.238 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb50975e:::benchmark_data_smithi159_162467_object53244:head# 2024-04-15T18:42:59.238 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb52563b:::benchmark_data_smithi159_162467_object32179:head# 2024-04-15T18:42:59.238 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb55a814:::benchmark_data_smithi159_162467_object50155:head# 2024-04-15T18:42:59.238 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb56fa66:::benchmark_data_smithi159_162467_object56396:head# 2024-04-15T18:42:59.239 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5711cc:::benchmark_data_smithi159_162467_object23783:head# 2024-04-15T18:42:59.239 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb576f8a:::benchmark_data_smithi159_162467_object38915:head# 2024-04-15T18:42:59.239 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb577648:::benchmark_data_smithi159_162467_object44892:head# 2024-04-15T18:42:59.240 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5857d4:::benchmark_data_smithi159_162467_object18059:head# 2024-04-15T18:42:59.240 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5872a3:::benchmark_data_smithi159_162467_object56453:head# 2024-04-15T18:42:59.241 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb58a9f1:::benchmark_data_smithi159_162467_object37936:head# 2024-04-15T18:42:59.241 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5a8b6a:::benchmark_data_smithi159_162467_object24544:head# 2024-04-15T18:42:59.242 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5ac7fc:::benchmark_data_smithi159_162467_object11369:head# 2024-04-15T18:42:59.242 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5ae452:::benchmark_data_smithi159_162467_object31637:head# 2024-04-15T18:42:59.242 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5ef0a8:::benchmark_data_smithi159_162467_object72850:head# 2024-04-15T18:42:59.243 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb5f5f9c:::benchmark_data_smithi159_162467_object67984:head# 2024-04-15T18:42:59.243 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb616f85:::benchmark_data_smithi159_162467_object35465:head# 2024-04-15T18:42:59.244 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb61d44e:::benchmark_data_smithi159_162467_object67940:head# 2024-04-15T18:42:59.244 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb62230c:::benchmark_data_smithi159_162467_object30832:head# 2024-04-15T18:42:59.244 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb622774:::benchmark_data_smithi159_162467_object13712:head# 2024-04-15T18:42:59.245 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb62e097:::benchmark_data_smithi159_162467_object23630:head# 2024-04-15T18:42:59.245 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb63a41c:::benchmark_data_smithi159_162467_object28801:head# 2024-04-15T18:42:59.245 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6605cb:::benchmark_data_smithi159_162467_object67308:head# 2024-04-15T18:42:59.246 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb671ccf:::benchmark_data_smithi159_162467_object27317:head# 2024-04-15T18:42:59.246 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb673900:::benchmark_data_smithi159_162467_object12780:head# 2024-04-15T18:42:59.247 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb67bbd4:::benchmark_data_smithi159_162467_object66740:head# 2024-04-15T18:42:59.247 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb67f99a:::benchmark_data_smithi159_162467_object30233:head# 2024-04-15T18:42:59.247 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb680ff9:::benchmark_data_smithi159_162467_object72019:head# 2024-04-15T18:42:59.248 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb683496:::benchmark_data_smithi159_162467_object8995:head# 2024-04-15T18:42:59.248 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb687f4b:::benchmark_data_smithi159_162467_object8706:head# 2024-04-15T18:42:59.249 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb69eff6:::benchmark_data_smithi159_162467_object57543:head# 2024-04-15T18:42:59.249 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6a4d1d:::benchmark_data_smithi159_162467_object70082:head# 2024-04-15T18:42:59.249 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6a4f07:::benchmark_data_smithi159_162467_object41065:head# 2024-04-15T18:42:59.250 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6aff23:::benchmark_data_smithi159_162467_object38562:head# 2024-04-15T18:42:59.250 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6d26f8:::benchmark_data_smithi159_162467_object47281:head# 2024-04-15T18:42:59.251 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6e49cc:::benchmark_data_smithi159_162467_object39256:head# 2024-04-15T18:42:59.252 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb6ee442:::benchmark_data_smithi159_162467_object35368:head# 2024-04-15T18:42:59.252 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb71215a:::benchmark_data_smithi159_162467_object25163:head# 2024-04-15T18:42:59.252 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb713944:::benchmark_data_smithi159_162467_object50090:head# 2024-04-15T18:42:59.252 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb71600d:::benchmark_data_smithi159_162467_object14938:head# 2024-04-15T18:42:59.253 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb71ca2f:::benchmark_data_smithi159_162467_object55802:head# 2024-04-15T18:42:59.253 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb727cd4:::benchmark_data_smithi159_162467_object16132:head# 2024-04-15T18:42:59.254 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb746f19:::benchmark_data_smithi159_162467_object47964:head# 2024-04-15T18:42:59.254 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb76c150:::benchmark_data_smithi159_162467_object17669:head# 2024-04-15T18:42:59.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb77a396:::benchmark_data_smithi159_162467_object19309:head# 2024-04-15T18:42:59.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb793607:::benchmark_data_smithi159_162467_object38955:head# 2024-04-15T18:42:59.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7aab5c:::benchmark_data_smithi159_162467_object20273:head# 2024-04-15T18:42:59.255 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7ac3b1:::benchmark_data_smithi159_162467_object32699:head# 2024-04-15T18:42:59.256 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7b58fa:::benchmark_data_smithi159_162467_object42280:head# 2024-04-15T18:42:59.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7d35ff:::benchmark_data_smithi159_162467_object28553:head# 2024-04-15T18:42:59.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7f971c:::benchmark_data_smithi159_162467_object46795:head# 2024-04-15T18:42:59.257 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb7fff21:::benchmark_data_smithi159_162467_object44603:head# 2024-04-15T18:42:59.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8157c1:::benchmark_data_smithi159_162467_object31972:head# 2024-04-15T18:42:59.258 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb837e18:::benchmark_data_smithi159_162467_object12726:head# 2024-04-15T18:42:59.259 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb853118:::benchmark_data_smithi159_162467_object53634:head# 2024-04-15T18:42:59.259 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb85fcef:::benchmark_data_smithi159_162467_object31496:head# 2024-04-15T18:42:59.259 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb89999e:::benchmark_data_smithi159_162467_object24420:head# 2024-04-15T18:42:59.261 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8b2f30:::benchmark_data_smithi159_162467_object4540:head# 2024-04-15T18:42:59.262 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8bd6da:::benchmark_data_smithi159_162467_object2696:head# 2024-04-15T18:42:59.263 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8c59be:::benchmark_data_smithi159_162467_object61989:head# 2024-04-15T18:42:59.263 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8cdc65:::benchmark_data_smithi159_162467_object50116:head# 2024-04-15T18:42:59.263 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8d0813:::benchmark_data_smithi159_162467_object31268:head# 2024-04-15T18:42:59.264 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8e1c24:::benchmark_data_smithi159_162467_object20073:head# 2024-04-15T18:42:59.264 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8e8641:::benchmark_data_smithi159_162467_object30045:head# 2024-04-15T18:42:59.264 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8f531d:::benchmark_data_smithi159_162467_object27061:head# 2024-04-15T18:42:59.265 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb8fd5ca:::benchmark_data_smithi159_162467_object27016:head# 2024-04-15T18:42:59.266 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb903078:::benchmark_data_smithi159_162467_object48108:head# 2024-04-15T18:42:59.266 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb90a5f1:::benchmark_data_smithi159_162467_object57352:head# 2024-04-15T18:42:59.266 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb90ac56:::benchmark_data_smithi159_162467_object67481:head# 2024-04-15T18:42:59.268 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb90e2d3:::benchmark_data_smithi159_162467_object23451:head# 2024-04-15T18:42:59.268 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb917804:::benchmark_data_smithi159_162467_object2416:head# 2024-04-15T18:42:59.268 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb91da4d:::benchmark_data_smithi159_162467_object52943:head# 2024-04-15T18:42:59.268 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb924be6:::benchmark_data_smithi159_162467_object8785:head# 2024-04-15T18:42:59.268 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9330f0:::benchmark_data_smithi159_162467_object39899:head# 2024-04-15T18:42:59.269 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb94a430:::benchmark_data_smithi159_162467_object201:head# 2024-04-15T18:42:59.269 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb95472e:::benchmark_data_smithi159_162467_object45765:head# 2024-04-15T18:42:59.269 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb97b9a1:::benchmark_data_smithi159_162467_object4117:head# 2024-04-15T18:42:59.269 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb97cf7e:::benchmark_data_smithi159_162467_object3218:head# 2024-04-15T18:42:59.271 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb984f8f:::benchmark_data_smithi159_162467_object59893:head# 2024-04-15T18:42:59.271 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb99e979:::benchmark_data_smithi159_162467_object41680:head# 2024-04-15T18:42:59.271 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9c57c6:::benchmark_data_smithi159_162467_object2403:head# 2024-04-15T18:42:59.271 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9c860f:::benchmark_data_smithi159_162467_object39705:head# 2024-04-15T18:42:59.272 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9df720:::benchmark_data_smithi159_162467_object39673:head# 2024-04-15T18:42:59.272 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9e19f8:::benchmark_data_smithi159_162467_object15315:head# 2024-04-15T18:42:59.273 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fb9e2d6d:::benchmark_data_smithi159_162467_object9276:head# 2024-04-15T18:42:59.273 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba050a8:::benchmark_data_smithi159_162467_object72765:head# 2024-04-15T18:42:59.273 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba12246:::benchmark_data_smithi159_162467_object38070:head# 2024-04-15T18:42:59.273 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba5393d:::benchmark_data_smithi159_162467_object49475:head# 2024-04-15T18:42:59.275 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba5c1ee:::benchmark_data_smithi159_162467_object55368:head# 2024-04-15T18:42:59.276 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba5ddca:::benchmark_data_smithi159_162467_object27593:head# 2024-04-15T18:42:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:58 smithi082 ceph-mon[44206]: pgmap v1932: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.0 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 5.7 MiB/s wr, 573 op/s; 27775/295338 objects degraded (9.404%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:42:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:42:58 smithi082 ceph-mon[44206]: osdmap e749: 12 total, 11 up, 11 in 2024-04-15T18:42:59.277 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba664fe:::benchmark_data_smithi159_162467_object71746:head# 2024-04-15T18:42:59.277 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba70200:::benchmark_data_smithi159_162467_object26675:head# 2024-04-15T18:42:59.277 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba72047:::benchmark_data_smithi159_162467_object25874:head# 2024-04-15T18:42:59.277 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba745b1:::benchmark_data_smithi159_162467_object38943:head# 2024-04-15T18:42:59.278 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba80fae:::benchmark_data_smithi159_162467_object61006:head# 2024-04-15T18:42:59.278 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fba8aa6b:::benchmark_data_smithi159_162467_object70138:head# 2024-04-15T18:42:59.279 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbabb489:::benchmark_data_smithi159_162467_object4944:head# 2024-04-15T18:42:59.279 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbac5796:::benchmark_data_smithi159_162467_object71676:head# 2024-04-15T18:42:59.280 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbac94a0:::benchmark_data_smithi159_162467_object40881:head# 2024-04-15T18:42:59.280 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbad5a85:::benchmark_data_smithi159_162467_object32338:head# 2024-04-15T18:42:59.280 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbae117e:::benchmark_data_smithi159_162467_object34450:head# 2024-04-15T18:42:59.281 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbae1f76:::benchmark_data_smithi159_162467_object19945:head# 2024-04-15T18:42:59.281 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbae4e99:::benchmark_data_smithi159_162467_object35189:head# 2024-04-15T18:42:59.281 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb150d1:::benchmark_data_smithi159_162467_object71981:head# 2024-04-15T18:42:59.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb26d21:::benchmark_data_smithi159_162467_object33029:head# 2024-04-15T18:42:59.282 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb36c1b:::benchmark_data_smithi159_162467_object62962:head# 2024-04-15T18:42:59.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb37719:::benchmark_data_smithi159_162467_object29749:head# 2024-04-15T18:42:59.283 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb42d78:::benchmark_data_smithi159_162467_object14820:head# 2024-04-15T18:42:59.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb454d3:::benchmark_data_smithi159_162467_object16501:head# 2024-04-15T18:42:59.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb49bca:::benchmark_data_smithi159_162467_object61333:head# 2024-04-15T18:42:59.284 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb4ec67:::benchmark_data_smithi159_162467_object38443:head# 2024-04-15T18:42:59.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb609e6:::benchmark_data_smithi159_162467_object37805:head# 2024-04-15T18:42:59.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb7017d:::benchmark_data_smithi159_162467_object68705:head# 2024-04-15T18:42:59.285 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbb7568a:::benchmark_data_smithi159_162467_object32409:head# 2024-04-15T18:42:59.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbb204c:::benchmark_data_smithi159_162467_object33654:head# 2024-04-15T18:42:59.286 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbc2aaf:::benchmark_data_smithi159_162467_object65272:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbc5ff2:::benchmark_data_smithi159_162467_object64775:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbceb77:::benchmark_data_smithi159_162467_object41583:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbcfd1a:::benchmark_data_smithi159_162467_object27060:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbd31dc:::benchmark_data_smithi159_162467_object1095:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbd53b8:::benchmark_data_smithi159_162467_object68902:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbda2bf:::benchmark_data_smithi159_162467_object60153:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbe8692:::benchmark_data_smithi159_162467_object49952:head# 2024-04-15T18:42:59.290 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbef876:::benchmark_data_smithi159_162467_object50675:head# 2024-04-15T18:42:59.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbfac99:::benchmark_data_smithi159_162467_object36525:head# 2024-04-15T18:42:59.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbbfcba8:::benchmark_data_smithi159_162467_object3708:head# 2024-04-15T18:42:59.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc38194:::benchmark_data_smithi159_162467_object492:head# 2024-04-15T18:42:59.291 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc4b8ae:::benchmark_data_smithi159_162467_object1723:head# 2024-04-15T18:42:59.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc4e5c5:::benchmark_data_smithi159_162467_object4195:head# 2024-04-15T18:42:59.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc50f92:::benchmark_data_smithi159_162467_object47984:head# 2024-04-15T18:42:59.292 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc51901:::benchmark_data_smithi159_162467_object33852:head# 2024-04-15T18:42:59.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc570cc:::benchmark_data_smithi159_162467_object61530:head# 2024-04-15T18:42:59.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc6604e:::benchmark_data_smithi159_162467_object3771:head# 2024-04-15T18:42:59.293 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc6bfbf:::benchmark_data_smithi159_162467_object19642:head# 2024-04-15T18:42:59.294 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc772c7:::benchmark_data_smithi159_162467_object54986:head# 2024-04-15T18:42:59.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc7c144:::benchmark_data_smithi159_162467_object4619:head# 2024-04-15T18:42:59.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc80cbd:::benchmark_data_smithi159_162467_object23924:head# 2024-04-15T18:42:59.296 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc80ccc:::benchmark_data_smithi159_162467_object71635:head# 2024-04-15T18:42:59.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc88cff:::benchmark_data_smithi159_162467_object63471:head# 2024-04-15T18:42:59.297 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc9838e:::benchmark_data_smithi159_162467_object29271:head# 2024-04-15T18:42:59.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbc9e5ed:::benchmark_data_smithi159_162467_object8296:head# 2024-04-15T18:42:59.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbca3545:::benchmark_data_smithi159_162467_object960:head# 2024-04-15T18:42:59.298 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcbfd7c:::benchmark_data_smithi159_162467_object48109:head# 2024-04-15T18:42:59.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcc1b7e:::benchmark_data_smithi159_162467_object69405:head# 2024-04-15T18:42:59.299 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcd0daa:::benchmark_data_smithi159_162467_object44923:head# 2024-04-15T18:42:59.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcd2709:::benchmark_data_smithi159_162467_object48362:head# 2024-04-15T18:42:59.300 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcda386:::benchmark_data_smithi159_162467_object41203:head# 2024-04-15T18:42:59.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbce6140:::benchmark_data_smithi159_162467_object55634:head# 2024-04-15T18:42:59.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcefab1:::benchmark_data_smithi159_162467_object5068:head# 2024-04-15T18:42:59.301 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcf710e:::benchmark_data_smithi159_162467_object2410:head# 2024-04-15T18:42:59.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcf71b3:::benchmark_data_smithi159_162467_object12505:head# 2024-04-15T18:42:59.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbcfd5af:::benchmark_data_smithi159_162467_object31263:head# 2024-04-15T18:42:59.302 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd01266:::benchmark_data_smithi159_162467_object9541:head# 2024-04-15T18:42:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd051b6:::benchmark_data_smithi159_162467_object30531:head# 2024-04-15T18:42:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd0cd84:::benchmark_data_smithi159_162467_object10392:head# 2024-04-15T18:42:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd19290:::benchmark_data_smithi159_162467_object54215:head# 2024-04-15T18:42:59.303 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd19ea2:::benchmark_data_smithi159_162467_object2414:head# 2024-04-15T18:42:59.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd1a571:::benchmark_data_smithi159_162467_object25082:head# 2024-04-15T18:42:59.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd1d32d:::benchmark_data_smithi159_162467_object23416:head# 2024-04-15T18:42:59.304 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd205db:::benchmark_data_smithi159_162467_object47429:head# 2024-04-15T18:42:59.305 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd2ae75:::benchmark_data_smithi159_162467_object35453:head# 2024-04-15T18:42:59.306 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd2c9a5:::benchmark_data_smithi159_162467_object23519:head# 2024-04-15T18:42:59.306 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd344ff:::benchmark_data_smithi159_162467_object6165:head# 2024-04-15T18:42:59.306 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd36769:::benchmark_data_smithi159_162467_object63411:head# 2024-04-15T18:42:59.307 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd60d7a:::benchmark_data_smithi159_162467_object40059:head# 2024-04-15T18:42:59.307 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd6e0fa:::benchmark_data_smithi159_162467_object19868:head# 2024-04-15T18:42:59.307 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd80a2a:::benchmark_data_smithi159_162467_object40676:head# 2024-04-15T18:42:59.308 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd82b98:::benchmark_data_smithi159_162467_object15454:head# 2024-04-15T18:42:59.308 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd83fbe:::benchmark_data_smithi159_162467_object33735:head# 2024-04-15T18:42:59.309 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd90f1d:::benchmark_data_smithi159_162467_object7875:head# 2024-04-15T18:42:59.309 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbd93e5c:::benchmark_data_smithi159_162467_object46705:head# 2024-04-15T18:42:59.310 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbda36ea:::benchmark_data_smithi159_162467_object52016:head# 2024-04-15T18:42:59.310 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbda8908:::benchmark_data_smithi159_162467_object18264:head# 2024-04-15T18:42:59.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbdad05a:::benchmark_data_smithi159_162467_object50341:head# 2024-04-15T18:42:59.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbdb3b1a:::benchmark_data_smithi159_162467_object23805:head# 2024-04-15T18:42:59.311 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbdc5f58:::benchmark_data_smithi159_162467_object57068:head# 2024-04-15T18:42:59.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbdca3d5:::benchmark_data_smithi159_162467_object46970:head# 2024-04-15T18:42:59.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbddaab0:::benchmark_data_smithi159_162467_object11280:head# 2024-04-15T18:42:59.312 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbddd7bf:::benchmark_data_smithi159_162467_object69364:head# 2024-04-15T18:42:59.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbdf2c52:::benchmark_data_smithi159_162467_object21610:head# 2024-04-15T18:42:59.313 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe03a4b:::benchmark_data_smithi159_162467_object36223:head# 2024-04-15T18:42:59.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe10bb6:::benchmark_data_smithi159_162467_object50747:head# 2024-04-15T18:42:59.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe11192:::benchmark_data_smithi159_162467_object21224:head# 2024-04-15T18:42:59.314 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe237d6:::benchmark_data_smithi159_162467_object57954:head# 2024-04-15T18:42:59.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe315a2:::benchmark_data_smithi159_162467_object64105:head# 2024-04-15T18:42:59.315 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe3b3ea:::benchmark_data_smithi159_162467_object32246:head# 2024-04-15T18:42:59.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe3edd0:::benchmark_data_smithi159_162467_object70406:head# 2024-04-15T18:42:59.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe418c4:::benchmark_data_smithi159_162467_object33971:head# 2024-04-15T18:42:59.316 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe461fc:::benchmark_data_smithi159_162467_object26729:head# 2024-04-15T18:42:59.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe5befa:::benchmark_data_smithi159_162467_object23454:head# 2024-04-15T18:42:59.317 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe6b9f5:::benchmark_data_smithi159_162467_object60929:head# 2024-04-15T18:42:59.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbe6cc99:::benchmark_data_smithi159_162467_object8386:head# 2024-04-15T18:42:59.318 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbea7410:::benchmark_data_smithi159_162467_object24527:head# 2024-04-15T18:42:59.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbea7c80:::benchmark_data_smithi159_162467_object11978:head# 2024-04-15T18:42:59.319 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbeacdf6:::benchmark_data_smithi159_162467_object53814:head# 2024-04-15T18:42:59.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbebd217:::benchmark_data_smithi159_162467_object40119:head# 2024-04-15T18:42:59.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbebd8db:::benchmark_data_smithi159_162467_object31321:head# 2024-04-15T18:42:59.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbed1370:::benchmark_data_smithi159_162467_object56784:head# 2024-04-15T18:42:59.320 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbed7648:::benchmark_data_smithi159_162467_object12011:head# 2024-04-15T18:42:59.321 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbedd154:::benchmark_data_smithi159_162467_object35561:head# 2024-04-15T18:42:59.322 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbee68db:::benchmark_data_smithi159_162467_object11874:head# 2024-04-15T18:42:59.322 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbefacda:::benchmark_data_smithi159_162467_object72061:head# 2024-04-15T18:42:59.322 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf0a098:::benchmark_data_smithi159_162467_object5724:head# 2024-04-15T18:42:59.323 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf18d6c:::benchmark_data_smithi159_162467_object28650:head# 2024-04-15T18:42:59.323 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf1a821:::benchmark_data_smithi159_162467_object25316:head# 2024-04-15T18:42:59.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf26975:::benchmark_data_smithi159_162467_object38184:head# 2024-04-15T18:42:59.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf2c591:::benchmark_data_smithi159_162467_object69545:head# 2024-04-15T18:42:59.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf3072a:::benchmark_data_smithi159_162467_object60840:head# 2024-04-15T18:42:59.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf388b0:::benchmark_data_smithi159_162467_object24732:head# 2024-04-15T18:42:59.324 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf50490:::benchmark_data_smithi159_162467_object33147:head# 2024-04-15T18:42:59.325 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf525d4:::benchmark_data_smithi159_162467_object29388:head# 2024-04-15T18:42:59.325 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf5631a:::benchmark_data_smithi159_162467_object41793:head# 2024-04-15T18:42:59.326 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf66514:::benchmark_data_smithi159_162467_object10056:head# 2024-04-15T18:42:59.326 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf94b43:::benchmark_data_smithi159_162467_object47204:head# 2024-04-15T18:42:59.327 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf96eda:::benchmark_data_smithi159_162467_object52650:head# 2024-04-15T18:42:59.327 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf987f5:::benchmark_data_smithi159_162467_object67088:head# 2024-04-15T18:42:59.327 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbf9d2d6:::benchmark_data_smithi159_162467_object44085:head# 2024-04-15T18:42:59.328 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfa2b94:::benchmark_data_smithi159_162467_object41504:head# 2024-04-15T18:42:59.328 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfa5540:::benchmark_data_smithi159_162467_object67630:head# 2024-04-15T18:42:59.329 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfbace6:::benchmark_data_smithi159_162467_object36433:head# 2024-04-15T18:42:59.329 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfbb764:::benchmark_data_smithi159_162467_object49399:head# 2024-04-15T18:42:59.329 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfd21ec:::benchmark_data_smithi159_162467_object58510:head# 2024-04-15T18:42:59.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfe3d7d:::benchmark_data_smithi159_162467_object69110:head# 2024-04-15T18:42:59.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fbfebb7a:::benchmark_data_smithi159_162467_object5575:head# 2024-04-15T18:42:59.333 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc01283e:::benchmark_data_smithi159_162467_object25503:head# 2024-04-15T18:42:59.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc02f3d9:::benchmark_data_smithi159_162467_object53647:head# 2024-04-15T18:42:59.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc03fe6d:::benchmark_data_smithi159_162467_object12834:head# 2024-04-15T18:42:59.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0492ea:::benchmark_data_smithi159_162467_object7377:head# 2024-04-15T18:42:59.334 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc04c884:::benchmark_data_smithi159_162467_object12605:head# 2024-04-15T18:42:59.336 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc067bce:::benchmark_data_smithi159_162467_object21867:head# 2024-04-15T18:42:59.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc06fa92:::benchmark_data_smithi159_162467_object36993:head# 2024-04-15T18:42:59.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc08d1be:::benchmark_data_smithi159_162467_object17031:head# 2024-04-15T18:42:59.337 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc092179:::benchmark_data_smithi159_162467_object25949:head# 2024-04-15T18:42:59.338 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc09d3c1:::benchmark_data_smithi159_162467_object61558:head# 2024-04-15T18:42:59.338 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0acecb:::benchmark_data_smithi159_162467_object37292:head# 2024-04-15T18:42:59.339 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0c0ad3:::benchmark_data_smithi159_162467_object71538:head# 2024-04-15T18:42:59.339 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0cdd9d:::benchmark_data_smithi159_162467_object50236:head# 2024-04-15T18:42:59.340 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0cf170:::benchmark_data_smithi159_162467_object67667:head# 2024-04-15T18:42:59.340 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0d5e96:::benchmark_data_smithi159_162467_object9383:head# 2024-04-15T18:42:59.340 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0d79d6:::benchmark_data_smithi159_162467_object29870:head# 2024-04-15T18:42:59.342 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0e0bf5:::benchmark_data_smithi159_162467_object40591:head# 2024-04-15T18:42:59.342 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0e1888:::benchmark_data_smithi159_162467_object18550:head# 2024-04-15T18:42:59.343 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc0f0081:::benchmark_data_smithi159_162467_object51569:head# 2024-04-15T18:42:59.343 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc110df2:::benchmark_data_smithi159_162467_object25142:head# 2024-04-15T18:42:59.343 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc111535:::benchmark_data_smithi159_162467_object52232:head# 2024-04-15T18:42:59.343 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc12683c:::benchmark_data_smithi159_162467_object23538:head# 2024-04-15T18:42:59.344 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1291e4:::benchmark_data_smithi159_162467_object1972:head# 2024-04-15T18:42:59.344 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc12a05c:::benchmark_data_smithi159_162467_object72476:head# 2024-04-15T18:42:59.344 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc12d169:::benchmark_data_smithi159_162467_object71976:head# 2024-04-15T18:42:59.345 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc12e35a:::benchmark_data_smithi159_162467_object20494:head# 2024-04-15T18:42:59.345 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc135e59:::benchmark_data_smithi159_162467_object6727:head# 2024-04-15T18:42:59.346 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc137e31:::benchmark_data_smithi159_162467_object72358:head# 2024-04-15T18:42:59.346 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc152ac6:::benchmark_data_smithi159_162467_object38545:head# 2024-04-15T18:42:59.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc156d5d:::benchmark_data_smithi159_162467_object70831:head# 2024-04-15T18:42:59.347 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc15e9ec:::benchmark_data_smithi159_162467_object116:head# 2024-04-15T18:42:59.349 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1715ca:::benchmark_data_smithi159_162467_object57330:head# 2024-04-15T18:42:59.350 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1846f1:::benchmark_data_smithi159_162467_object16494:head# 2024-04-15T18:42:59.350 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1980df:::benchmark_data_smithi159_162467_object13480:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1a0c35:::benchmark_data_smithi159_162467_object66795:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1a610b:::benchmark_data_smithi159_162467_object9640:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1b0c94:::benchmark_data_smithi159_162467_object5481:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1b5174:::benchmark_data_smithi159_162467_object46991:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1c6b7e:::benchmark_data_smithi159_162467_object40210:head# 2024-04-15T18:42:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1cd4d1:::benchmark_data_smithi159_162467_object33838:head# 2024-04-15T18:42:59.352 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1d31e8:::benchmark_data_smithi159_162467_object63292:head# 2024-04-15T18:42:59.352 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1f109a:::benchmark_data_smithi159_162467_object67640:head# 2024-04-15T18:42:59.352 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1f1342:::benchmark_data_smithi159_162467_object50480:head# 2024-04-15T18:42:59.352 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc1fa5e9:::benchmark_data_smithi159_162467_object67073:head# 2024-04-15T18:42:59.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc201bd5:::benchmark_data_smithi159_162467_object59042:head# 2024-04-15T18:42:59.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc212ac9:::benchmark_data_smithi159_162467_object6146:head# 2024-04-15T18:42:59.353 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2186ad:::benchmark_data_smithi159_162467_object58285:head# 2024-04-15T18:42:59.354 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc219a4d:::benchmark_data_smithi159_162467_object60965:head# 2024-04-15T18:42:59.354 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc23caa5:::benchmark_data_smithi159_162467_object1330:head# 2024-04-15T18:42:59.355 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2403ed:::benchmark_data_smithi159_162467_object25390:head# 2024-04-15T18:42:59.355 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc25221f:::benchmark_data_smithi159_162467_object26752:head# 2024-04-15T18:42:59.355 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2568d3:::benchmark_data_smithi159_162467_object51454:head# 2024-04-15T18:42:59.356 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc25b372:::benchmark_data_smithi159_162467_object38221:head# 2024-04-15T18:42:59.356 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc26e6d9:::benchmark_data_smithi159_162467_object20788:head# 2024-04-15T18:42:59.356 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc273a1f:::benchmark_data_smithi159_162467_object15547:head# 2024-04-15T18:42:59.357 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc278990:::benchmark_data_smithi159_162467_object31735:head# 2024-04-15T18:42:59.357 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2821f7:::benchmark_data_smithi159_162467_object6850:head# 2024-04-15T18:42:59.358 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc286dea:::benchmark_data_smithi159_162467_object20333:head# 2024-04-15T18:42:59.358 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2a4672:::benchmark_data_smithi159_162467_object70519:head# 2024-04-15T18:42:59.358 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2a9358:::benchmark_data_smithi159_162467_object31778:head# 2024-04-15T18:42:59.359 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2b3a98:::benchmark_data_smithi159_162467_object44355:head# 2024-04-15T18:42:59.359 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2b81a2:::benchmark_data_smithi159_162467_object14745:head# 2024-04-15T18:42:59.360 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2bdd43:::benchmark_data_smithi159_162467_object22857:head# 2024-04-15T18:42:59.360 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2c8fb1:::benchmark_data_smithi159_162467_object58283:head# 2024-04-15T18:42:59.361 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2d95ad:::benchmark_data_smithi159_162467_object7790:head# 2024-04-15T18:42:59.361 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2e41b3:::benchmark_data_smithi159_162467_object59082:head# 2024-04-15T18:42:59.361 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2f77bd:::benchmark_data_smithi159_162467_object73929:head# 2024-04-15T18:42:59.362 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2fb535:::benchmark_data_smithi159_162467_object24137:head# 2024-04-15T18:42:59.362 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc2fd022:::benchmark_data_smithi159_162467_object14622:head# 2024-04-15T18:42:59.363 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc308093:::benchmark_data_smithi159_162467_object21533:head# 2024-04-15T18:42:59.363 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc30839b:::benchmark_data_smithi159_162467_object17301:head# 2024-04-15T18:42:59.363 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc31920f:::benchmark_data_smithi159_162467_object23860:head# 2024-04-15T18:42:59.363 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc31c6af:::benchmark_data_smithi159_162467_object41827:head# 2024-04-15T18:42:59.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc33527d:::benchmark_data_smithi159_162467_object54444:head# 2024-04-15T18:42:59.364 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc33da04:::benchmark_data_smithi159_162467_object39449:head# 2024-04-15T18:42:59.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc347984:::benchmark_data_smithi159_162467_object14592:head# 2024-04-15T18:42:59.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3653b3:::benchmark_data_smithi159_162467_object43935:head# 2024-04-15T18:42:59.365 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc375904:::benchmark_data_smithi159_162467_object59518:head# 2024-04-15T18:42:59.366 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc387a81:::benchmark_data_smithi159_162467_object63126:head# 2024-04-15T18:42:59.366 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3ae33c:::benchmark_data_smithi159_162467_object16824:head# 2024-04-15T18:42:59.367 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3b0ebf:::benchmark_data_smithi159_162467_object19398:head# 2024-04-15T18:42:59.367 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3bbd4a:::benchmark_data_smithi159_162467_object51221:head# 2024-04-15T18:42:59.368 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3ce787:::benchmark_data_smithi159_162467_object64699:head# 2024-04-15T18:42:59.368 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3e5e07:::benchmark_data_smithi159_162467_object3963:head# 2024-04-15T18:42:59.368 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc3f0a74:::benchmark_data_smithi159_162467_object29610:head# 2024-04-15T18:42:59.369 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc40273d:::benchmark_data_smithi159_162467_object46224:head# 2024-04-15T18:42:59.369 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc40878d:::benchmark_data_smithi159_162467_object63074:head# 2024-04-15T18:42:59.370 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc40a179:::benchmark_data_smithi159_162467_object56699:head# 2024-04-15T18:42:59.370 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4115fb:::benchmark_data_smithi159_162467_object32209:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc41a095:::benchmark_data_smithi159_162467_object11528:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc42dfdd:::benchmark_data_smithi159_162467_object32544:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc435411:::benchmark_data_smithi159_162467_object43783:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc443b77:::benchmark_data_smithi159_162467_object15291:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc446af2:::benchmark_data_smithi159_162467_object16842:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc453a2d:::benchmark_data_smithi159_162467_object39621:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc45ebf1:::benchmark_data_smithi159_162467_object17068:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc46cf4b:::benchmark_data_smithi159_162467_object25308:head# 2024-04-15T18:42:59.374 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc46d7aa:::benchmark_data_smithi159_162467_object39268:head# 2024-04-15T18:42:59.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc47513f:::benchmark_data_smithi159_162467_object3735:head# 2024-04-15T18:42:59.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc47e32c:::benchmark_data_smithi159_162467_object14217:head# 2024-04-15T18:42:59.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc47fbf9:::benchmark_data_smithi159_162467_object2531:head# 2024-04-15T18:42:59.375 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc491312:::benchmark_data_smithi159_162467_object20125:head# 2024-04-15T18:42:59.376 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc491b8d:::benchmark_data_smithi159_162467_object29325:head# 2024-04-15T18:42:59.376 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4a0841:::benchmark_data_smithi159_162467_object4362:head# 2024-04-15T18:42:59.377 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4a805a:::benchmark_data_smithi159_162467_object15929:head# 2024-04-15T18:42:59.377 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4b635b:::benchmark_data_smithi159_162467_object52516:head# 2024-04-15T18:42:59.377 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4b92f1:::benchmark_data_smithi159_162467_object49589:head# 2024-04-15T18:42:59.378 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4d2489:::benchmark_data_smithi159_162467_object19836:head# 2024-04-15T18:42:59.378 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4d674b:::benchmark_data_smithi159_162467_object13537:head# 2024-04-15T18:42:59.378 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4da1fd:::benchmark_data_smithi159_162467_object545:head# 2024-04-15T18:42:59.379 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4e1283:::benchmark_data_smithi159_162467_object47808:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4e8b56:::benchmark_data_smithi159_162467_object6802:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4ee582:::benchmark_data_smithi159_162467_object1636:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4f1aab:::benchmark_data_smithi159_162467_object29128:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc4f2a35:::benchmark_data_smithi159_162467_object33877:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc516322:::benchmark_data_smithi159_162467_object59576:head# 2024-04-15T18:42:59.382 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc528ac5:::benchmark_data_smithi159_162467_object10172:head# 2024-04-15T18:42:59.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc534be1:::benchmark_data_smithi159_162467_object62828:head# 2024-04-15T18:42:59.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc550388:::benchmark_data_smithi159_162467_object71337:head# 2024-04-15T18:42:59.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc55831c:::benchmark_data_smithi159_162467_object71952:head# 2024-04-15T18:42:59.383 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5623e9:::benchmark_data_smithi159_162467_object42561:head# 2024-04-15T18:42:59.384 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5633f6:::benchmark_data_smithi159_162467_object11177:head# 2024-04-15T18:42:59.384 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5642d3:::benchmark_data_smithi159_162467_object3714:head# 2024-04-15T18:42:59.384 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc56d17a:::benchmark_data_smithi159_162467_object26550:head# 2024-04-15T18:42:59.385 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc571dee:::benchmark_data_smithi159_162467_object16084:head# 2024-04-15T18:42:59.387 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc577c3c:::benchmark_data_smithi159_162467_object43805:head# 2024-04-15T18:42:59.387 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc58d08a:::benchmark_data_smithi159_162467_object14823:head# 2024-04-15T18:42:59.388 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5a377f:::benchmark_data_smithi159_162467_object47432:head# 2024-04-15T18:42:59.388 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5a3f3d:::benchmark_data_smithi159_162467_object38772:head# 2024-04-15T18:42:59.388 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5a4084:::benchmark_data_smithi159_162467_object72478:head# 2024-04-15T18:42:59.389 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5ae26e:::benchmark_data_smithi159_162467_object15868:head# 2024-04-15T18:42:59.389 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5b4c34:::benchmark_data_smithi159_162467_object47303:head# 2024-04-15T18:42:59.390 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5c0682:::benchmark_data_smithi159_162467_object30115:head# 2024-04-15T18:42:59.390 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5e7f32:::benchmark_data_smithi159_162467_object29403:head# 2024-04-15T18:42:59.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5e875b:::benchmark_data_smithi159_162467_object8213:head# 2024-04-15T18:42:59.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc5ef077:::benchmark_data_smithi159_162467_object49962:head# 2024-04-15T18:42:59.391 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6067b5:::benchmark_data_smithi159_162467_object29489:head# 2024-04-15T18:42:59.392 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc620aa5:::benchmark_data_smithi159_162467_object40987:head# 2024-04-15T18:42:59.393 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc62da9b:::benchmark_data_smithi159_162467_object58859:head# 2024-04-15T18:42:59.393 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc63e854:::benchmark_data_smithi159_162467_object63855:head# 2024-04-15T18:42:59.393 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6439e9:::benchmark_data_smithi159_162467_object54576:head# 2024-04-15T18:42:59.394 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc655588:::benchmark_data_smithi159_162467_object23034:head# 2024-04-15T18:42:59.394 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc656d2e:::benchmark_data_smithi159_162467_object61384:head# 2024-04-15T18:42:59.395 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc659deb:::benchmark_data_smithi159_162467_object47458:head# 2024-04-15T18:42:59.395 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc666b26:::benchmark_data_smithi159_162467_object74331:head# 2024-04-15T18:42:59.396 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc668b56:::benchmark_data_smithi159_162467_object9033:head# 2024-04-15T18:42:59.396 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc678716:::benchmark_data_smithi159_162467_object74758:head# 2024-04-15T18:42:59.396 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc67bf05:::benchmark_data_smithi159_162467_object17667:head# 2024-04-15T18:42:59.397 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc68e7b6:::benchmark_data_smithi159_162467_object2011:head# 2024-04-15T18:42:59.398 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc694386:::benchmark_data_smithi159_162467_object11883:head# 2024-04-15T18:42:59.398 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc69eb1a:::benchmark_data_smithi159_162467_object3391:head# 2024-04-15T18:42:59.399 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6add30:::benchmark_data_smithi159_162467_object32210:head# 2024-04-15T18:42:59.399 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6bdb91:::benchmark_data_smithi159_162467_object2176:head# 2024-04-15T18:42:59.399 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6c0c5b:::benchmark_data_smithi159_162467_object16830:head# 2024-04-15T18:42:59.400 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc6f994c:::benchmark_data_smithi159_162467_object65703:head# 2024-04-15T18:42:59.400 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc70f37d:::benchmark_data_smithi159_162467_object45037:head# 2024-04-15T18:42:59.401 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc711bf0:::benchmark_data_smithi159_162467_object13893:head# 2024-04-15T18:42:59.401 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc73942a:::benchmark_data_smithi159_162467_object5069:head# 2024-04-15T18:42:59.402 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc739809:::benchmark_data_smithi159_162467_object69587:head# 2024-04-15T18:42:59.402 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc74c893:::benchmark_data_smithi159_162467_object55523:head# 2024-04-15T18:42:59.402 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc755145:::benchmark_data_smithi159_162467_object42267:head# 2024-04-15T18:42:59.403 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc759f9d:::benchmark_data_smithi159_162467_object24445:head# 2024-04-15T18:42:59.403 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7613bf:::benchmark_data_smithi159_162467_object42357:head# 2024-04-15T18:42:59.404 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc76ecb8:::benchmark_data_smithi159_162467_object29040:head# 2024-04-15T18:42:59.404 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7831d5:::benchmark_data_smithi159_162467_object47461:head# 2024-04-15T18:42:59.404 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc79f84a:::benchmark_data_smithi159_162467_object55326:head# 2024-04-15T18:42:59.405 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7bf0e4:::benchmark_data_smithi159_162467_object400:head# 2024-04-15T18:42:59.405 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7c21f0:::benchmark_data_smithi159_162467_object60891:head# 2024-04-15T18:42:59.406 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7c4f04:::benchmark_data_smithi159_162467_object18747:head# 2024-04-15T18:42:59.406 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7cd119:::benchmark_data_smithi159_162467_object65485:head# 2024-04-15T18:42:59.406 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7cd9f8:::benchmark_data_smithi159_162467_object39064:head# 2024-04-15T18:42:59.407 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7d06b3:::benchmark_data_smithi159_162467_object65391:head# 2024-04-15T18:42:59.407 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7d0ae8:::benchmark_data_smithi159_162467_object62321:head# 2024-04-15T18:42:59.408 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc7ef3f7:::benchmark_data_smithi159_162467_object27082:head# 2024-04-15T18:42:59.408 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc801888:::benchmark_data_smithi159_162467_object34915:head# 2024-04-15T18:42:59.408 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc823d2f:::benchmark_data_smithi159_162467_object27308:head# 2024-04-15T18:42:59.409 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc82d8ac:::benchmark_data_smithi159_162467_object7686:head# 2024-04-15T18:42:59.409 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc82d989:::benchmark_data_smithi159_162467_object53923:head# 2024-04-15T18:42:59.409 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc834c05:::benchmark_data_smithi159_162467_object1406:head# 2024-04-15T18:42:59.410 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc84291a:::benchmark_data_smithi159_162467_object43593:head# 2024-04-15T18:42:59.410 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc84ad08:::benchmark_data_smithi159_162467_object13022:head# 2024-04-15T18:42:59.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc84d8d0:::benchmark_data_smithi159_162467_object32930:head# 2024-04-15T18:42:59.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc869075:::benchmark_data_smithi159_162467_object53943:head# 2024-04-15T18:42:59.411 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc889e24:::benchmark_data_smithi159_162467_object74199:head# 2024-04-15T18:42:59.412 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8a1897:::benchmark_data_smithi159_162467_object51786:head# 2024-04-15T18:42:59.412 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8b60c3:::benchmark_data_smithi159_162467_object34598:head# 2024-04-15T18:42:59.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8b6390:::benchmark_data_smithi159_162467_object25627:head# 2024-04-15T18:42:59.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8c8830:::benchmark_data_smithi159_162467_object7671:head# 2024-04-15T18:42:59.413 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8cfa67:::benchmark_data_smithi159_162467_object3635:head# 2024-04-15T18:42:59.414 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8d7a1d:::benchmark_data_smithi159_162467_object72840:head# 2024-04-15T18:42:59.414 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8dadcc:::benchmark_data_smithi159_162467_object71515:head# 2024-04-15T18:42:59.415 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8e5937:::benchmark_data_smithi159_162467_object45666:head# 2024-04-15T18:42:59.415 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc8f96fe:::benchmark_data_smithi159_162467_object11181:head# 2024-04-15T18:42:59.415 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc904fee:::benchmark_data_smithi159_162467_object67827:head# 2024-04-15T18:42:59.416 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9059e6:::benchmark_data_smithi159_162467_object18287:head# 2024-04-15T18:42:59.416 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9165df:::benchmark_data_smithi159_162467_object42602:head# 2024-04-15T18:42:59.416 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc924581:::benchmark_data_smithi159_162467_object17528:head# 2024-04-15T18:42:59.417 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9255d2:::benchmark_data_smithi159_162467_object53305:head# 2024-04-15T18:42:59.417 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc936071:::benchmark_data_smithi159_162467_object36788:head# 2024-04-15T18:42:59.418 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc952c3e:::benchmark_data_smithi159_162467_object59161:head# 2024-04-15T18:42:59.418 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc95cd95:::benchmark_data_smithi159_162467_object66105:head# 2024-04-15T18:42:59.418 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc963495:::benchmark_data_smithi159_162467_object20983:head# 2024-04-15T18:42:59.419 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9658ed:::benchmark_data_smithi159_162467_object16496:head# 2024-04-15T18:42:59.419 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc969b97:::benchmark_data_smithi159_162467_object20617:head# 2024-04-15T18:42:59.420 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc97d9a8:::benchmark_data_smithi159_162467_object30086:head# 2024-04-15T18:42:59.420 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc995458:::benchmark_data_smithi159_162467_object35611:head# 2024-04-15T18:42:59.420 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9a6b28:::benchmark_data_smithi159_162467_object12323:head# 2024-04-15T18:42:59.421 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9b0bc2:::benchmark_data_smithi159_162467_object32140:head# 2024-04-15T18:42:59.427 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9c3518:::benchmark_data_smithi159_162467_object57872:head# 2024-04-15T18:42:59.428 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9d5a7d:::benchmark_data_smithi159_162467_object46905:head# 2024-04-15T18:42:59.428 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9db09a:::benchmark_data_smithi159_162467_object22669:head# 2024-04-15T18:42:59.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9f6c10:::benchmark_data_smithi159_162467_object50823:head# 2024-04-15T18:42:59.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fc9f6c1e:::benchmark_data_smithi159_162467_object21567:head# 2024-04-15T18:42:59.429 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca0f5c8:::benchmark_data_smithi159_162467_object18760:head# 2024-04-15T18:42:59.430 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca1ba55:::benchmark_data_smithi159_162467_object6890:head# 2024-04-15T18:42:59.430 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca31f0d:::benchmark_data_smithi159_162467_object34475:head# 2024-04-15T18:42:59.431 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca413d8:::benchmark_data_smithi159_162467_object72448:head# 2024-04-15T18:42:59.431 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca42759:::benchmark_data_smithi159_162467_object19662:head# 2024-04-15T18:42:59.432 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca4ad8e:::benchmark_data_smithi159_162467_object23506:head# 2024-04-15T18:42:59.432 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca4de9e:::benchmark_data_smithi159_162467_object14186:head# 2024-04-15T18:42:59.432 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca52474:::benchmark_data_smithi159_162467_object59682:head# 2024-04-15T18:42:59.433 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fca77b1c:::benchmark_data_smithi159_162467_object70546:head# 2024-04-15T18:42:59.433 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcaad9df:::benchmark_data_smithi159_162467_object56847:head# 2024-04-15T18:42:59.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcab67d5:::benchmark_data_smithi159_162467_object72259:head# 2024-04-15T18:42:59.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcad0584:::benchmark_data_smithi159_162467_object43210:head# 2024-04-15T18:42:59.434 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcae0547:::benchmark_data_smithi159_162467_object37334:head# 2024-04-15T18:42:59.435 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcae6da2:::benchmark_data_smithi159_162467_object31876:head# 2024-04-15T18:42:59.435 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcaf83fa:::benchmark_data_smithi159_162467_object57692:head# 2024-04-15T18:42:59.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb04c26:::benchmark_data_smithi159_162467_object8619:head# 2024-04-15T18:42:59.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb167dd:::benchmark_data_smithi159_162467_object46673:head# 2024-04-15T18:42:59.436 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb1bda1:::benchmark_data_smithi159_162467_object47573:head# 2024-04-15T18:42:59.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb287ee:::benchmark_data_smithi159_162467_object25722:head# 2024-04-15T18:42:59.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb40041:::benchmark_data_smithi159_162467_object20766:head# 2024-04-15T18:42:59.437 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb624ab:::benchmark_data_smithi159_162467_object69515:head# 2024-04-15T18:42:59.438 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb77ed5:::benchmark_data_smithi159_162467_object73550:head# 2024-04-15T18:42:59.438 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb8ecf1:::benchmark_data_smithi159_162467_object6755:head# 2024-04-15T18:42:59.438 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcb90dc4:::benchmark_data_smithi159_162467_object48889:head# 2024-04-15T18:42:59.439 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcbbbe05:::benchmark_data_smithi159_162467_object30593:head# 2024-04-15T18:42:59.439 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcbcd820:::benchmark_data_smithi159_162467_object33750:head# 2024-04-15T18:42:59.440 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcbdfb0b:::benchmark_data_smithi159_162467_object12687:head# 2024-04-15T18:42:59.440 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcbe5739:::benchmark_data_smithi159_162467_object49980:head# 2024-04-15T18:42:59.441 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc037cf:::benchmark_data_smithi159_162467_object32506:head# 2024-04-15T18:42:59.441 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc127c8:::benchmark_data_smithi159_162467_object43187:head# 2024-04-15T18:42:59.441 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc1d974:::benchmark_data_smithi159_162467_object39104:head# 2024-04-15T18:42:59.442 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc359e6:::benchmark_data_smithi159_162467_object37497:head# 2024-04-15T18:42:59.443 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc35c59:::benchmark_data_smithi159_162467_object67859:head# 2024-04-15T18:42:59.443 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc77149:::benchmark_data_smithi159_162467_object31005:head# 2024-04-15T18:42:59.443 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcc965eb:::benchmark_data_smithi159_162467_object29164:head# 2024-04-15T18:42:59.446 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcca1d3f:::benchmark_data_smithi159_162467_object49128:head# 2024-04-15T18:42:59.446 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcca591b:::benchmark_data_smithi159_162467_object3250:head# 2024-04-15T18:42:59.447 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcca6c33:::benchmark_data_smithi159_162467_object41365:head# 2024-04-15T18:42:59.447 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcccdf44:::benchmark_data_smithi159_162467_object9646:head# 2024-04-15T18:42:59.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fccd2327:::benchmark_data_smithi159_162467_object29246:head# 2024-04-15T18:42:59.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fccd72b7:::benchmark_data_smithi159_162467_object19837:head# 2024-04-15T18:42:59.448 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fccffd40:::benchmark_data_smithi159_162467_object5421:head# 2024-04-15T18:42:59.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd2360e:::benchmark_data_smithi159_162467_object7951:head# 2024-04-15T18:42:59.449 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd2adef:::benchmark_data_smithi159_162467_object30252:head# 2024-04-15T18:42:59.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd33a8d:::benchmark_data_smithi159_162467_object38286:head# 2024-04-15T18:42:59.450 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd4ef85:::benchmark_data_smithi159_162467_object44336:head# 2024-04-15T18:42:59.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd5b79f:::benchmark_data_smithi159_162467_object46931:head# 2024-04-15T18:42:59.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd6cc62:::benchmark_data_smithi159_162467_object4477:head# 2024-04-15T18:42:59.451 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd7cd66:::benchmark_data_smithi159_162467_object71631:head# 2024-04-15T18:42:59.452 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcd8115f:::benchmark_data_smithi159_162467_object11788:head# 2024-04-15T18:42:59.452 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcdae68c:::benchmark_data_smithi159_162467_object6966:head# 2024-04-15T18:42:59.452 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcdb14a6:::benchmark_data_smithi159_162467_object61517:head# 2024-04-15T18:42:59.453 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcdb793a:::benchmark_data_smithi159_162467_object40845:head# 2024-04-15T18:42:59.453 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcdce341:::benchmark_data_smithi159_162467_object45386:head# 2024-04-15T18:42:59.454 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcdd65ac:::benchmark_data_smithi159_162467_object71105:head# 2024-04-15T18:42:59.454 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce0f5f4:::benchmark_data_smithi159_162467_object724:head# 2024-04-15T18:42:59.454 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce11dcd:::benchmark_data_smithi159_162467_object50973:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce3c387:::benchmark_data_smithi159_162467_object34288:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce4612a:::benchmark_data_smithi159_162467_object40271:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce4a8c8:::benchmark_data_smithi159_162467_object897:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce50523:::benchmark_data_smithi159_162467_object19519:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce61f7a:::benchmark_data_smithi159_162467_object19081:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce74c09:::benchmark_data_smithi159_162467_object72304:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce772fb:::benchmark_data_smithi159_162467_object71847:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce77d57:::benchmark_data_smithi159_162467_object22225:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fce7c9ca:::benchmark_data_smithi159_162467_object15896:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcea2873:::benchmark_data_smithi159_162467_object41707:head# 2024-04-15T18:42:59.459 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcea7f8c:::benchmark_data_smithi159_162467_object65163:head# 2024-04-15T18:42:59.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcea8274:::benchmark_data_smithi159_162467_object45096:head# 2024-04-15T18:42:59.460 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fceb8e85:::benchmark_data_smithi159_162467_object44680:head# 2024-04-15T18:42:59.463 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fceb9804:::benchmark_data_smithi159_162467_object16922:head# 2024-04-15T18:42:59.463 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fced38f8:::benchmark_data_smithi159_162467_object61219:head# 2024-04-15T18:42:59.464 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcee97a2:::benchmark_data_smithi159_162467_object73992:head# 2024-04-15T18:42:59.464 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fceee72f:::benchmark_data_smithi159_162467_object68675:head# 2024-04-15T18:42:59.464 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcef5962:::benchmark_data_smithi159_162467_object1899:head# 2024-04-15T18:42:59.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcefac36:::benchmark_data_smithi159_162467_object3257:head# 2024-04-15T18:42:59.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf0bc90:::benchmark_data_smithi159_162467_object30438:head# 2024-04-15T18:42:59.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf13e15:::benchmark_data_smithi159_162467_object5838:head# 2024-04-15T18:42:59.465 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf19c43:::benchmark_data_smithi159_162467_object45227:head# 2024-04-15T18:42:59.466 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf1d329:::benchmark_data_smithi159_162467_object52321:head# 2024-04-15T18:42:59.466 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf1ece8:::benchmark_data_smithi159_162467_object64627:head# 2024-04-15T18:42:59.468 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf2922a:::benchmark_data_smithi159_162467_object14482:head# 2024-04-15T18:42:59.468 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf384fa:::benchmark_data_smithi159_162467_object41967:head# 2024-04-15T18:42:59.469 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf8cb5c:::benchmark_data_smithi159_162467_object45627:head# 2024-04-15T18:42:59.469 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf8e5ce:::benchmark_data_smithi159_162467_object66882:head# 2024-04-15T18:42:59.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcf9cd68:::benchmark_data_smithi159_162467_object51860:head# 2024-04-15T18:42:59.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcfa9759:::benchmark_data_smithi159_162467_object38434:head# 2024-04-15T18:42:59.470 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcfa9ff8:::benchmark_data_smithi159_162467_object10495:head# 2024-04-15T18:42:59.471 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcfcccaf:::benchmark_data_smithi159_162467_object41964:head# 2024-04-15T18:42:59.472 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcfda1fe:::benchmark_data_smithi159_162467_object5696:head# 2024-04-15T18:42:59.472 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcfde141:::benchmark_data_smithi159_162467_object24619:head# 2024-04-15T18:42:59.472 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fcff7ec1:::benchmark_data_smithi159_162467_object20211:head# 2024-04-15T18:42:59.473 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd004552:::benchmark_data_smithi159_162467_object53426:head# 2024-04-15T18:42:59.473 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd00ea13:::benchmark_data_smithi159_162467_object57997:head# 2024-04-15T18:42:59.473 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd01d2cb:::benchmark_data_smithi159_162467_object11160:head# 2024-04-15T18:42:59.474 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd024338:::benchmark_data_smithi159_162467_object54080:head# 2024-04-15T18:42:59.474 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd024feb:::benchmark_data_smithi159_162467_object50238:head# 2024-04-15T18:42:59.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd043536:::benchmark_data_smithi159_162467_object45115:head# 2024-04-15T18:42:59.475 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0454fa:::benchmark_data_smithi159_162467_object51691:head# 2024-04-15T18:42:59.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0530bf:::benchmark_data_smithi159_162467_object14098:head# 2024-04-15T18:42:59.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd05624c:::benchmark_data_smithi159_162467_object55646:head# 2024-04-15T18:42:59.476 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd066390:::benchmark_data_smithi159_162467_object14752:head# 2024-04-15T18:42:59.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd07ab56:::benchmark_data_smithi159_162467_object49309:head# 2024-04-15T18:42:59.477 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd081a05:::benchmark_data_smithi159_162467_object66716:head# 2024-04-15T18:42:59.478 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd08dc36:::benchmark_data_smithi159_162467_object52625:head# 2024-04-15T18:42:59.478 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd097574:::benchmark_data_smithi159_162467_object48128:head# 2024-04-15T18:42:59.478 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0aa865:::benchmark_data_smithi159_162467_object70358:head# 2024-04-15T18:42:59.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0af543:::benchmark_data_smithi159_162467_object9291:head# 2024-04-15T18:42:59.479 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0bfc2f:::benchmark_data_smithi159_162467_object74201:head# 2024-04-15T18:42:59.480 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0cb433:::benchmark_data_smithi159_162467_object46098:head# 2024-04-15T18:42:59.480 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0d72dc:::benchmark_data_smithi159_162467_object69687:head# 2024-04-15T18:42:59.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0d8a61:::benchmark_data_smithi159_162467_object10372:head# 2024-04-15T18:42:59.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0edaed:::benchmark_data_smithi159_162467_object6694:head# 2024-04-15T18:42:59.481 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd0f2bc7:::benchmark_data_smithi159_162467_object10561:head# 2024-04-15T18:42:59.482 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd113511:::benchmark_data_smithi159_162467_object15797:head# 2024-04-15T18:42:59.482 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd114050:::benchmark_data_smithi159_162467_object10231:head# 2024-04-15T18:42:59.483 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd124c36:::benchmark_data_smithi159_162467_object38478:head# 2024-04-15T18:42:59.483 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd12a482:::benchmark_data_smithi159_162467_object6720:head# 2024-04-15T18:42:59.483 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd133e4b:::benchmark_data_smithi159_162467_object66276:head# 2024-04-15T18:42:59.484 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd14b322:::benchmark_data_smithi159_162467_object38780:head# 2024-04-15T18:42:59.484 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd166034:::benchmark_data_smithi159_162467_object6370:head# 2024-04-15T18:42:59.484 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd169cd6:::benchmark_data_smithi159_162467_object56974:head# 2024-04-15T18:42:59.485 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd18f144:::benchmark_data_smithi159_162467_object61161:head# 2024-04-15T18:42:59.485 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1916d2:::benchmark_data_smithi159_162467_object57065:head# 2024-04-15T18:42:59.486 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1a0bbe:::benchmark_data_smithi159_162467_object3870:head# 2024-04-15T18:42:59.486 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1acb6b:::benchmark_data_smithi159_162467_object45527:head# 2024-04-15T18:42:59.487 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1c6d4a:::benchmark_data_smithi159_162467_object30971:head# 2024-04-15T18:42:59.487 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1d861e:::benchmark_data_smithi159_162467_object57754:head# 2024-04-15T18:42:59.487 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd1d9f86:::benchmark_data_smithi159_162467_object7028:head# 2024-04-15T18:42:59.487 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd223a94:::benchmark_data_smithi159_162467_object14881:head# 2024-04-15T18:42:59.488 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd232c23:::benchmark_data_smithi159_162467_object68893:head# 2024-04-15T18:42:59.488 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2392cb:::benchmark_data_smithi159_162467_object20038:head# 2024-04-15T18:42:59.488 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2618af:::benchmark_data_smithi159_162467_object51174:head# 2024-04-15T18:42:59.490 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2685bd:::benchmark_data_smithi159_162467_object62455:head# 2024-04-15T18:42:59.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd283469:::benchmark_data_smithi159_162467_object45997:head# 2024-04-15T18:42:59.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd291d88:::benchmark_data_smithi159_162467_object74227:head# 2024-04-15T18:42:59.491 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd29b566:::benchmark_data_smithi159_162467_object68024:head# 2024-04-15T18:42:59.492 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2abae5:::benchmark_data_smithi159_162467_object7485:head# 2024-04-15T18:42:59.492 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2ac10d:::benchmark_data_smithi159_162467_object49601:head# 2024-04-15T18:42:59.494 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2ca58b:::benchmark_data_smithi159_162467_object68850:head# 2024-04-15T18:42:59.495 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2ce5d9:::benchmark_data_smithi159_162467_object37854:head# 2024-04-15T18:42:59.495 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2d2f33:::benchmark_data_smithi159_162467_object49186:head# 2024-04-15T18:42:59.496 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2d72bd:::benchmark_data_smithi159_162467_object37241:head# 2024-04-15T18:42:59.496 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd2d776d:::benchmark_data_smithi159_162467_object67818:head# 2024-04-15T18:42:59.497 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd324d19:::benchmark_data_smithi159_162467_object58077:head# 2024-04-15T18:42:59.497 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd33f6a4:::benchmark_data_smithi159_162467_object30707:head# 2024-04-15T18:42:59.498 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd34b262:::benchmark_data_smithi159_162467_object65420:head# 2024-04-15T18:42:59.498 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd34fc33:::benchmark_data_smithi159_162467_object15297:head# 2024-04-15T18:42:59.498 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3516de:::benchmark_data_smithi159_162467_object57490:head# 2024-04-15T18:42:59.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd355024:::benchmark_data_smithi159_162467_object23310:head# 2024-04-15T18:42:59.499 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd35c19e:::benchmark_data_smithi159_162467_object73570:head# 2024-04-15T18:42:59.500 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3625e1:::benchmark_data_smithi159_162467_object21701:head# 2024-04-15T18:42:59.501 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd370e9e:::benchmark_data_smithi159_162467_object49909:head# 2024-04-15T18:42:59.501 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd37b69a:::benchmark_data_smithi159_162467_object50767:head# 2024-04-15T18:42:59.501 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd38f004:::benchmark_data_smithi159_162467_object68899:head# 2024-04-15T18:42:59.502 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd38f5ea:::benchmark_data_smithi159_162467_object2886:head# 2024-04-15T18:42:59.502 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd39f648:::benchmark_data_smithi159_162467_object72369:head# 2024-04-15T18:42:59.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3b20e0:::benchmark_data_smithi159_162467_object32226:head# 2024-04-15T18:42:59.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3c62eb:::benchmark_data_smithi159_162467_object44729:head# 2024-04-15T18:42:59.503 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3c925f:::benchmark_data_smithi159_162467_object71169:head# 2024-04-15T18:42:59.504 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3ca488:::benchmark_data_smithi159_162467_object63039:head# 2024-04-15T18:42:59.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3ce994:::benchmark_data_smithi159_162467_object64669:head# 2024-04-15T18:42:59.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3ea609:::benchmark_data_smithi159_162467_object73354:head# 2024-04-15T18:42:59.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3f55c2:::benchmark_data_smithi159_162467_object6250:head# 2024-04-15T18:42:59.505 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd3fab9a:::benchmark_data_smithi159_162467_object54491:head# 2024-04-15T18:42:59.506 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd42612f:::benchmark_data_smithi159_162467_object68434:head# 2024-04-15T18:42:59.507 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd42630c:::benchmark_data_smithi159_162467_object49802:head# 2024-04-15T18:42:59.510 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd4460e5:::benchmark_data_smithi159_162467_object36260:head# 2024-04-15T18:42:59.510 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd44fd08:::benchmark_data_smithi159_162467_object12342:head# 2024-04-15T18:42:59.510 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd458259:::benchmark_data_smithi159_162467_object52997:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd498bff:::benchmark_data_smithi159_162467_object31430:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd4b2396:::benchmark_data_smithi159_162467_object64862:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd4d8460:::benchmark_data_smithi159_162467_object25943:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd4f03ea:::benchmark_data_smithi159_162467_object27726:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd50ffb2:::benchmark_data_smithi159_162467_object23564:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5169d6:::benchmark_data_smithi159_162467_object25472:head# 2024-04-15T18:42:59.511 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd52b9f7:::benchmark_data_smithi159_162467_object7155:head# 2024-04-15T18:42:59.512 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd537a8d:::benchmark_data_smithi159_162467_object69126:head# 2024-04-15T18:42:59.512 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5405c3:::benchmark_data_smithi159_162467_object66831:head# 2024-04-15T18:42:59.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd574c2c:::benchmark_data_smithi159_162467_object73913:head# 2024-04-15T18:42:59.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd57a3df:::benchmark_data_smithi159_162467_object1465:head# 2024-04-15T18:42:59.513 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd57dbd5:::benchmark_data_smithi159_162467_object58137:head# 2024-04-15T18:42:59.514 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd58ecca:::benchmark_data_smithi159_162467_object3588:head# 2024-04-15T18:42:59.516 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd593bb0:::benchmark_data_smithi159_162467_object25530:head# 2024-04-15T18:42:59.518 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd599883:::benchmark_data_smithi159_162467_object16928:head# 2024-04-15T18:42:59.519 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5aa249:::benchmark_data_smithi159_162467_object11498:head# 2024-04-15T18:42:59.519 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5b425c:::benchmark_data_smithi159_162467_object4679:head# 2024-04-15T18:42:59.519 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5b7c4d:::benchmark_data_smithi159_162467_object64708:head# 2024-04-15T18:42:59.520 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5c0efd:::benchmark_data_smithi159_162467_object50530:head# 2024-04-15T18:42:59.520 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5d989b:::benchmark_data_smithi159_162467_object31612:head# 2024-04-15T18:42:59.521 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5dd0a4:::benchmark_data_smithi159_162467_object23591:head# 2024-04-15T18:42:59.521 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5df891:::benchmark_data_smithi159_162467_object58564:head# 2024-04-15T18:42:59.522 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5e42d6:::benchmark_data_smithi159_162467_object27485:head# 2024-04-15T18:42:59.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd5f5061:::benchmark_data_smithi159_162467_object17668:head# 2024-04-15T18:42:59.523 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6051b4:::benchmark_data_smithi159_162467_object7412:head# 2024-04-15T18:42:59.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6051fb:::benchmark_data_smithi159_162467_object72080:head# 2024-04-15T18:42:59.524 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd628dff:::benchmark_data_smithi159_162467_object48284:head# 2024-04-15T18:42:59.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd636406:::benchmark_data_smithi159_162467_object41399:head# 2024-04-15T18:42:59.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd63b791:::benchmark_data_smithi159_162467_object23450:head# 2024-04-15T18:42:59.525 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd63e851:::benchmark_data_smithi159_162467_object66418:head# 2024-04-15T18:42:59.526 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd650f8a:::benchmark_data_smithi159_162467_object67132:head# 2024-04-15T18:42:59.528 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd65649e:::benchmark_data_smithi159_162467_object22512:head# 2024-04-15T18:42:59.528 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd666628:::benchmark_data_smithi159_162467_object72811:head# 2024-04-15T18:42:59.529 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd66821a:::benchmark_data_smithi159_162467_object67330:head# 2024-04-15T18:42:59.529 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6695e5:::benchmark_data_smithi159_162467_object50660:head# 2024-04-15T18:42:59.530 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd66c34b:::benchmark_data_smithi159_162467_object59654:head# 2024-04-15T18:42:59.530 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd678447:::benchmark_data_smithi159_162467_object34352:head# 2024-04-15T18:42:59.531 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd682254:::benchmark_data_smithi159_162467_object16539:head# 2024-04-15T18:42:59.532 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd687bf7:::benchmark_data_smithi159_162467_object55168:head# 2024-04-15T18:42:59.532 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6ae3e4:::benchmark_data_smithi159_162467_object65452:head# 2024-04-15T18:42:59.533 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6b27c9:::benchmark_data_smithi159_162467_object52586:head# 2024-04-15T18:42:59.533 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6b3e9a:::benchmark_data_smithi159_162467_object15931:head# 2024-04-15T18:42:59.534 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6b7c90:::benchmark_data_smithi159_162467_object62477:head# 2024-04-15T18:42:59.534 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6c189e:::benchmark_data_smithi159_162467_object19964:head# 2024-04-15T18:42:59.534 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6c2bd4:::benchmark_data_smithi159_162467_object25690:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6cb399:::benchmark_data_smithi159_162467_object55626:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6cd902:::benchmark_data_smithi159_162467_object27809:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6ce29a:::benchmark_data_smithi159_162467_object50173:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6dff87:::benchmark_data_smithi159_162467_object25533:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6e3222:::benchmark_data_smithi159_162467_object40844:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6eab2e:::benchmark_data_smithi159_162467_object39704:head# 2024-04-15T18:42:59.538 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd6eac0c:::benchmark_data_smithi159_162467_object42299:head# 2024-04-15T18:42:59.539 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7053de:::benchmark_data_smithi159_162467_object39815:head# 2024-04-15T18:42:59.539 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd708bc5:::benchmark_data_smithi159_162467_object70398:head# 2024-04-15T18:42:59.540 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7140fd:::benchmark_data_smithi159_162467_object53596:head# 2024-04-15T18:42:59.540 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd718fd3:::benchmark_data_smithi159_162467_object15150:head# 2024-04-15T18:42:59.540 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd721fcd:::benchmark_data_smithi159_162467_object67191:head# 2024-04-15T18:42:59.541 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd72871d:::benchmark_data_smithi159_162467_object66649:head# 2024-04-15T18:42:59.541 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd736571:::benchmark_data_smithi159_162467_object11077:head# 2024-04-15T18:42:59.542 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd768547:::benchmark_data_smithi159_162467_object71176:head# 2024-04-15T18:42:59.542 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd78cacb:::benchmark_data_smithi159_162467_object13494:head# 2024-04-15T18:42:59.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd78d4f0:::benchmark_data_smithi159_162467_object10656:head# 2024-04-15T18:42:59.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd78ecf8:::benchmark_data_smithi159_162467_object4336:head# 2024-04-15T18:42:59.543 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7a8006:::benchmark_data_smithi159_162467_object27598:head# 2024-04-15T18:42:59.544 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7a8485:::benchmark_data_smithi159_162467_object38220:head# 2024-04-15T18:42:59.544 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7acb06:::benchmark_data_smithi159_162467_object41323:head# 2024-04-15T18:42:59.544 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7d7d78:::benchmark_data_smithi159_162467_object11983:head# 2024-04-15T18:42:59.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd7e75c5:::benchmark_data_smithi159_162467_object69297:head# 2024-04-15T18:42:59.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd80358a:::benchmark_data_smithi159_162467_object72962:head# 2024-04-15T18:42:59.545 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd807a11:::benchmark_data_smithi159_162467_object4781:head# 2024-04-15T18:42:59.546 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd80884d:::benchmark_data_smithi159_162467_object66168:head# 2024-04-15T18:42:59.546 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd80dde2:::benchmark_data_smithi159_162467_object38894:head# 2024-04-15T18:42:59.546 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd818621:::benchmark_data_smithi159_162467_object63525:head# 2024-04-15T18:42:59.547 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd83401d:::benchmark_data_smithi159_162467_object23952:head# 2024-04-15T18:42:59.547 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd84f5a9:::benchmark_data_smithi159_162467_object47101:head# 2024-04-15T18:42:59.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd8581de:::benchmark_data_smithi159_162467_object24191:head# 2024-04-15T18:42:59.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd85d4b7:::benchmark_data_smithi159_162467_object34839:head# 2024-04-15T18:42:59.548 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd873172:::benchmark_data_smithi159_162467_object41096:head# 2024-04-15T18:42:59.549 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd87c22e:::benchmark_data_smithi159_162467_object73255:head# 2024-04-15T18:42:59.549 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd8971fb:::benchmark_data_smithi159_162467_object7144:head# 2024-04-15T18:42:59.550 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd8a4b7e:::benchmark_data_smithi159_162467_object63378:head# 2024-04-15T18:42:59.550 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd8d176b:::benchmark_data_smithi159_162467_object24808:head# 2024-04-15T18:42:59.551 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd8f80ea:::benchmark_data_smithi159_162467_object54156:head# 2024-04-15T18:42:59.553 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd901422:::benchmark_data_smithi159_162467_object28410:head# 2024-04-15T18:42:59.555 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd904ba9:::benchmark_data_smithi159_162467_object28247:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd90ad7f:::benchmark_data_smithi159_162467_object23841:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd911103:::benchmark_data_smithi159_162467_object26357:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9157c2:::benchmark_data_smithi159_162467_object39305:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd92ea1e:::benchmark_data_smithi159_162467_object24155:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9336aa:::benchmark_data_smithi159_162467_object52793:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9371c1:::benchmark_data_smithi159_162467_object1797:head# 2024-04-15T18:42:59.556 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd94cc12:::benchmark_data_smithi159_162467_object55901:head# 2024-04-15T18:42:59.558 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd950d7b:::benchmark_data_smithi159_162467_object70366:head# 2024-04-15T18:42:59.558 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd95c3ef:::benchmark_data_smithi159_162467_object43482:head# 2024-04-15T18:42:59.559 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd95ced8:::benchmark_data_smithi159_162467_object10610:head# 2024-04-15T18:42:59.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9787c2:::benchmark_data_smithi159_162467_object57549:head# 2024-04-15T18:42:59.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9796e8:::benchmark_data_smithi159_162467_object17273:head# 2024-04-15T18:42:59.560 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd987a69:::benchmark_data_smithi159_162467_object12347:head# 2024-04-15T18:42:59.561 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd99af90:::benchmark_data_smithi159_162467_object10419:head# 2024-04-15T18:42:59.561 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd99c808:::benchmark_data_smithi159_162467_object62694:head# 2024-04-15T18:42:59.562 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9a1e45:::benchmark_data_smithi159_162467_object14240:head# 2024-04-15T18:42:59.562 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9a57b5:::benchmark_data_smithi159_162467_object65488:head# 2024-04-15T18:42:59.562 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9c6ff1:::benchmark_data_smithi159_162467_object28027:head# 2024-04-15T18:42:59.563 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9d4b49:::benchmark_data_smithi159_162467_object53171:head# 2024-04-15T18:42:59.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9da3d1:::benchmark_data_smithi159_162467_object42295:head# 2024-04-15T18:42:59.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9e89a5:::benchmark_data_smithi159_162467_object57143:head# 2024-04-15T18:42:59.564 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9fa463:::benchmark_data_smithi159_162467_object6136:head# 2024-04-15T18:42:59.565 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9fa811:::benchmark_data_smithi159_162467_object14673:head# 2024-04-15T18:42:59.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9fc9d9:::benchmark_data_smithi159_162467_object43239:head# 2024-04-15T18:42:59.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fd9fdad1:::benchmark_data_smithi159_162467_object47441:head# 2024-04-15T18:42:59.566 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda046a6:::benchmark_data_smithi159_162467_object32999:head# 2024-04-15T18:42:59.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda063a5:::benchmark_data_smithi159_162467_object33531:head# 2024-04-15T18:42:59.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda0d949:::benchmark_data_smithi159_162467_object63311:head# 2024-04-15T18:42:59.567 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda2212b:::benchmark_data_smithi159_162467_object35347:head# 2024-04-15T18:42:59.568 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda33187:::benchmark_data_smithi159_162467_object8506:head# 2024-04-15T18:42:59.568 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda3402c:::benchmark_data_smithi159_162467_object62042:head# 2024-04-15T18:42:59.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda406d1:::benchmark_data_smithi159_162467_object63141:head# 2024-04-15T18:42:59.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda4c4a4:::benchmark_data_smithi159_162467_object3079:head# 2024-04-15T18:42:59.569 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda63cd6:::benchmark_data_smithi159_162467_object29105:head# 2024-04-15T18:42:59.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda68d6d:::benchmark_data_smithi159_162467_object43017:head# 2024-04-15T18:42:59.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda74d10:::benchmark_data_smithi159_162467_object11715:head# 2024-04-15T18:42:59.570 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda8408e:::benchmark_data_smithi159_162467_object36350:head# 2024-04-15T18:42:59.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda86d1c:::benchmark_data_smithi159_162467_object4123:head# 2024-04-15T18:42:59.571 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda93114:::benchmark_data_smithi159_162467_object18072:head# 2024-04-15T18:42:59.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda957f6:::benchmark_data_smithi159_162467_object2084:head# 2024-04-15T18:42:59.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fda9e514:::benchmark_data_smithi159_162467_object63740:head# 2024-04-15T18:42:59.572 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdacf6f6:::benchmark_data_smithi159_162467_object61232:head# 2024-04-15T18:42:59.573 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdad605d:::benchmark_data_smithi159_162467_object9043:head# 2024-04-15T18:42:59.573 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdadc20e:::benchmark_data_smithi159_162467_object46736:head# 2024-04-15T18:42:59.574 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdae7067:::benchmark_data_smithi159_162467_object50768:head# 2024-04-15T18:42:59.575 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdaf3ce7:::benchmark_data_smithi159_162467_object29902:head# 2024-04-15T18:42:59.575 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdaf3fa5:::benchmark_data_smithi159_162467_object3957:head# 2024-04-15T18:42:59.576 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb03631:::benchmark_data_smithi159_162467_object19904:head# 2024-04-15T18:42:59.576 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb2db25:::benchmark_data_smithi159_162467_object36257:head# 2024-04-15T18:42:59.576 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb3b9e5:::benchmark_data_smithi159_162467_object12938:head# 2024-04-15T18:42:59.577 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb68614:::benchmark_data_smithi159_162467_object61757:head# 2024-04-15T18:42:59.577 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb78635:::benchmark_data_smithi159_162467_object35361:head# 2024-04-15T18:42:59.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb8b9ca:::benchmark_data_smithi159_162467_object44548:head# 2024-04-15T18:42:59.578 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdb8cc96:::benchmark_data_smithi159_162467_object7692:head# 2024-04-15T18:42:59.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdbb3442:::benchmark_data_smithi159_162467_object62601:head# 2024-04-15T18:42:59.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc0ebe1:::benchmark_data_smithi159_162467_object39716:head# 2024-04-15T18:42:59.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc1cd7c:::benchmark_data_smithi159_162467_object22166:head# 2024-04-15T18:42:59.579 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc294a6:::benchmark_data_smithi159_162467_object52360:head# 2024-04-15T18:42:59.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc3136a:::benchmark_data_smithi159_162467_object21516:head# 2024-04-15T18:42:59.580 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc3efb9:::benchmark_data_smithi159_162467_object72495:head# 2024-04-15T18:42:59.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc69166:::benchmark_data_smithi159_162467_object32211:head# 2024-04-15T18:42:59.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc77e71:::benchmark_data_smithi159_162467_object25194:head# 2024-04-15T18:42:59.581 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc915fb:::benchmark_data_smithi159_162467_object6657:head# 2024-04-15T18:42:59.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdc92dda:::benchmark_data_smithi159_162467_object68253:head# 2024-04-15T18:42:59.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdca6cd6:::benchmark_data_smithi159_162467_object21022:head# 2024-04-15T18:42:59.582 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcad571:::benchmark_data_smithi159_162467_object67801:head# 2024-04-15T18:42:59.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcbe729:::benchmark_data_smithi159_162467_object71013:head# 2024-04-15T18:42:59.583 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcd4444:::benchmark_data_smithi159_162467_object15190:head# 2024-04-15T18:42:59.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcd6921:::benchmark_data_smithi159_162467_object31515:head# 2024-04-15T18:42:59.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcf9b89:::benchmark_data_smithi159_162467_object9543:head# 2024-04-15T18:42:59.584 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcfaa7f:::benchmark_data_smithi159_162467_object50720:head# 2024-04-15T18:42:59.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdcfe5ff:::benchmark_data_smithi159_162467_object56884:head# 2024-04-15T18:42:59.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd19978:::benchmark_data_smithi159_162467_object39368:head# 2024-04-15T18:42:59.585 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd413fc:::benchmark_data_smithi159_162467_object64019:head# 2024-04-15T18:42:59.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd44ed3:::benchmark_data_smithi159_162467_object5998:head# 2024-04-15T18:42:59.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd489e9:::benchmark_data_smithi159_162467_object25102:head# 2024-04-15T18:42:59.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd588e6:::benchmark_data_smithi159_162467_object58009:head# 2024-04-15T18:42:59.586 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd74a2a:::benchmark_data_smithi159_162467_object61050:head# 2024-04-15T18:42:59.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd783d6:::benchmark_data_smithi159_162467_object23917:head# 2024-04-15T18:42:59.587 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd784a6:::benchmark_data_smithi159_162467_object13795:head# 2024-04-15T18:42:59.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd7a168:::benchmark_data_smithi159_162467_object42626:head# 2024-04-15T18:42:59.588 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdd89d39:::benchmark_data_smithi159_162467_object33726:head# 2024-04-15T18:42:59.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fddb5998:::benchmark_data_smithi159_162467_object16175:head# 2024-04-15T18:42:59.592 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fddc0902:::benchmark_data_smithi159_162467_object57220:head# 2024-04-15T18:42:59.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fddc94e9:::benchmark_data_smithi159_162467_object60827:head# 2024-04-15T18:42:59.593 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde0a71c:::benchmark_data_smithi159_162467_object31527:head# 2024-04-15T18:42:59.594 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde3698f:::benchmark_data_smithi159_162467_object30270:head# 2024-04-15T18:42:59.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde3f3de:::benchmark_data_smithi159_162467_object47422:head# 2024-04-15T18:42:59.595 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde7b178:::benchmark_data_smithi159_162467_object13065:head# 2024-04-15T18:42:59.596 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde837f7:::benchmark_data_smithi159_162467_object34073:head# 2024-04-15T18:42:59.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde87077:::benchmark_data_smithi159_162467_object68636:head# 2024-04-15T18:42:59.597 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde912ec:::benchmark_data_smithi159_162467_object35832:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde97d11:::benchmark_data_smithi159_162467_object65800:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fde9c6e0:::benchmark_data_smithi159_162467_object17980:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdea1cea:::benchmark_data_smithi159_162467_object24189:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdeb569a:::benchmark_data_smithi159_162467_object40160:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdec0638:::benchmark_data_smithi159_162467_object36451:head# 2024-04-15T18:42:59.601 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdec50d9:::benchmark_data_smithi159_162467_object16274:head# 2024-04-15T18:42:59.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdee14db:::benchmark_data_smithi159_162467_object53828:head# 2024-04-15T18:42:59.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdee9ecc:::benchmark_data_smithi159_162467_object38863:head# 2024-04-15T18:42:59.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdef1559:::benchmark_data_smithi159_162467_object60381:head# 2024-04-15T18:42:59.602 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdef3d60:::benchmark_data_smithi159_162467_object31570:head# 2024-04-15T18:42:59.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf0a481:::benchmark_data_smithi159_162467_object16847:head# 2024-04-15T18:42:59.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf0b705:::benchmark_data_smithi159_162467_object73987:head# 2024-04-15T18:42:59.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf0c0b1:::benchmark_data_smithi159_162467_object17038:head# 2024-04-15T18:42:59.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf0faab:::benchmark_data_smithi159_162467_object31829:head# 2024-04-15T18:42:59.603 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf1a779:::benchmark_data_smithi159_162467_object4278:head# 2024-04-15T18:42:59.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf1d67b:::benchmark_data_smithi159_162467_object14467:head# 2024-04-15T18:42:59.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf22f3d:::benchmark_data_smithi159_162467_object10413:head# 2024-04-15T18:42:59.604 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf281b7:::benchmark_data_smithi159_162467_object11970:head# 2024-04-15T18:42:59.605 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf296a6:::benchmark_data_smithi159_162467_object13083:head# 2024-04-15T18:42:59.605 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf3f24e:::benchmark_data_smithi159_162467_object14184:head# 2024-04-15T18:42:59.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf5cb4d:::benchmark_data_smithi159_162467_object49487:head# 2024-04-15T18:42:59.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf63b7b:::benchmark_data_smithi159_162467_object71228:head# 2024-04-15T18:42:59.606 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf687a9:::benchmark_data_smithi159_162467_object56830:head# 2024-04-15T18:42:59.607 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf71c67:::benchmark_data_smithi159_162467_object57482:head# 2024-04-15T18:42:59.607 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf79a23:::benchmark_data_smithi159_162467_object72022:head# 2024-04-15T18:42:59.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf7ecf3:::benchmark_data_smithi159_162467_object39844:head# 2024-04-15T18:42:59.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf7f2ce:::benchmark_data_smithi159_162467_object20708:head# 2024-04-15T18:42:59.608 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf870fa:::benchmark_data_smithi159_162467_object6491:head# 2024-04-15T18:42:59.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf999ba:::benchmark_data_smithi159_162467_object49502:head# 2024-04-15T18:42:59.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdf9a9a5:::benchmark_data_smithi159_162467_object18846:head# 2024-04-15T18:42:59.609 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdfa24af:::benchmark_data_smithi159_162467_object32247:head# 2024-04-15T18:42:59.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdfc8681:::benchmark_data_smithi159_162467_object56414:head# 2024-04-15T18:42:59.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdfd7904:::benchmark_data_smithi159_162467_object58054:head# 2024-04-15T18:42:59.610 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdfe6c5b:::benchmark_data_smithi159_162467_object16097:head# 2024-04-15T18:42:59.611 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdfeb963:::benchmark_data_smithi159_162467_object64790:head# 2024-04-15T18:42:59.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fdffacfb:::benchmark_data_smithi159_162467_object68776:head# 2024-04-15T18:42:59.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0085ee:::benchmark_data_smithi159_162467_object14170:head# 2024-04-15T18:42:59.612 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe01d3a5:::benchmark_data_smithi159_162467_object2113:head# 2024-04-15T18:42:59.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe029d40:::benchmark_data_smithi159_162467_object60782:head# 2024-04-15T18:42:59.613 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe049914:::benchmark_data_smithi159_162467_object18109:head# 2024-04-15T18:42:59.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe05774a:::benchmark_data_smithi159_162467_object59954:head# 2024-04-15T18:42:59.614 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0630a1:::benchmark_data_smithi159_162467_object34686:head# 2024-04-15T18:42:59.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0955ff:::benchmark_data_smithi159_162467_object49532:head# 2024-04-15T18:42:59.615 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe09d84c:::benchmark_data_smithi159_162467_object33507:head# 2024-04-15T18:42:59.616 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0b19df:::benchmark_data_smithi159_162467_object7784:head# 2024-04-15T18:42:59.616 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0bc3e7:::benchmark_data_smithi159_162467_object72865:head# 2024-04-15T18:42:59.616 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0c4275:::benchmark_data_smithi159_162467_object68967:head# 2024-04-15T18:42:59.617 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0c6bad:::benchmark_data_smithi159_162467_object39586:head# 2024-04-15T18:42:59.618 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0cfc68:::benchmark_data_smithi159_162467_object44320:head# 2024-04-15T18:42:59.618 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0d1842:::benchmark_data_smithi159_162467_object27759:head# 2024-04-15T18:42:59.619 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0e48e8:::benchmark_data_smithi159_162467_object17454:head# 2024-04-15T18:42:59.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0eb415:::benchmark_data_smithi159_162467_object42539:head# 2024-04-15T18:42:59.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0f1501:::benchmark_data_smithi159_162467_object50500:head# 2024-04-15T18:42:59.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe0fe01f:::benchmark_data_smithi159_162467_object42807:head# 2024-04-15T18:42:59.620 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe101d4e:::benchmark_data_smithi159_162467_object21628:head# 2024-04-15T18:42:59.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe10a21c:::benchmark_data_smithi159_162467_object47098:head# 2024-04-15T18:42:59.621 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe10d465:::benchmark_data_smithi159_162467_object31290:head# 2024-04-15T18:42:59.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe124760:::benchmark_data_smithi159_162467_object24517:head# 2024-04-15T18:42:59.622 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe139dc5:::benchmark_data_smithi159_162467_object4181:head# 2024-04-15T18:42:59.623 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe14ebfa:::benchmark_data_smithi159_162467_object64176:head# 2024-04-15T18:42:59.623 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe153ce6:::benchmark_data_smithi159_162467_object33950:head# 2024-04-15T18:42:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe163194:::benchmark_data_smithi159_162467_object48157:head# 2024-04-15T18:42:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe16543e:::benchmark_data_smithi159_162467_object40301:head# 2024-04-15T18:42:59.624 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe16bc8f:::benchmark_data_smithi159_162467_object47226:head# 2024-04-15T18:42:59.625 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe17bc81:::benchmark_data_smithi159_162467_object57006:head# 2024-04-15T18:42:59.627 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1bed46:::benchmark_data_smithi159_162467_object16223:head# 2024-04-15T18:42:59.628 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1c1c6d:::benchmark_data_smithi159_162467_object73901:head# 2024-04-15T18:42:59.628 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1c45fe:::benchmark_data_smithi159_162467_object177:head# 2024-04-15T18:42:59.628 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1ceb71:::benchmark_data_smithi159_162467_object55867:head# 2024-04-15T18:42:59.629 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1d8d78:::benchmark_data_smithi159_162467_object51798:head# 2024-04-15T18:42:59.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1f2737:::benchmark_data_smithi159_162467_object13917:head# 2024-04-15T18:42:59.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1f3ec1:::benchmark_data_smithi159_162467_object15015:head# 2024-04-15T18:42:59.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe1f4c89:::benchmark_data_smithi159_162467_object71643:head# 2024-04-15T18:42:59.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2095e5:::benchmark_data_smithi159_162467_object5906:head# 2024-04-15T18:42:59.631 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe240a57:::benchmark_data_smithi159_162467_object1139:head# 2024-04-15T18:42:59.632 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe240c76:::benchmark_data_smithi159_162467_object3424:head# 2024-04-15T18:42:59.632 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2537a1:::benchmark_data_smithi159_162467_object42350:head# 2024-04-15T18:42:59.632 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe25c134:::benchmark_data_smithi159_162467_object18794:head# 2024-04-15T18:42:59.633 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe27091d:::benchmark_data_smithi159_162467_object38997:head# 2024-04-15T18:42:59.633 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe27c26d:::benchmark_data_smithi159_162467_object3844:head# 2024-04-15T18:42:59.634 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2883ad:::benchmark_data_smithi159_162467_object63794:head# 2024-04-15T18:42:59.634 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe299725:::benchmark_data_smithi159_162467_object39548:head# 2024-04-15T18:42:59.635 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2a2c94:::benchmark_data_smithi159_162467_object69769:head# 2024-04-15T18:42:59.635 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2c022e:::benchmark_data_smithi159_162467_object17082:head# 2024-04-15T18:42:59.636 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2c9000:::benchmark_data_smithi159_162467_object61302:head# 2024-04-15T18:42:59.636 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe2e4037:::benchmark_data_smithi159_162467_object35966:head# 2024-04-15T18:42:59.636 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3188a1:::benchmark_data_smithi159_162467_object67071:head# 2024-04-15T18:42:59.637 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe31f909:::benchmark_data_smithi159_162467_object62846:head# 2024-04-15T18:42:59.637 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe346f79:::benchmark_data_smithi159_162467_object72424:head# 2024-04-15T18:42:59.638 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe351cff:::benchmark_data_smithi159_162467_object68709:head# 2024-04-15T18:42:59.638 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe358628:::benchmark_data_smithi159_162467_object73853:head# 2024-04-15T18:42:59.638 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe359dc2:::benchmark_data_smithi159_162467_object29773:head# 2024-04-15T18:42:59.639 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe35c2d3:::benchmark_data_smithi159_162467_object10546:head# 2024-04-15T18:42:59.639 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3630e7:::benchmark_data_smithi159_162467_object48874:head# 2024-04-15T18:42:59.639 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe36b69d:::benchmark_data_smithi159_162467_object49183:head# 2024-04-15T18:42:59.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe382376:::benchmark_data_smithi159_162467_object38385:head# 2024-04-15T18:42:59.640 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3af978:::benchmark_data_smithi159_162467_object758:head# 2024-04-15T18:42:59.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3b27cb:::benchmark_data_smithi159_162467_object42340:head# 2024-04-15T18:42:59.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3ca233:::benchmark_data_smithi159_162467_object60863:head# 2024-04-15T18:42:59.641 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3dbbd7:::benchmark_data_smithi159_162467_object3813:head# 2024-04-15T18:42:59.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe3e185b:::benchmark_data_smithi159_162467_object72822:head# 2024-04-15T18:42:59.642 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe404f27:::benchmark_data_smithi159_162467_object15035:head# 2024-04-15T18:42:59.643 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe40ff4e:::benchmark_data_smithi159_162467_object73307:head# 2024-04-15T18:42:59.643 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe41a9f3:::benchmark_data_smithi159_162467_object38937:head# 2024-04-15T18:42:59.643 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe41d66d:::benchmark_data_smithi159_162467_object35991:head# 2024-04-15T18:42:59.644 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe41f0ed:::benchmark_data_smithi159_162467_object16490:head# 2024-04-15T18:42:59.644 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe423a96:::benchmark_data_smithi159_162467_object7351:head# 2024-04-15T18:42:59.645 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4316b9:::benchmark_data_smithi159_162467_object65596:head# 2024-04-15T18:42:59.645 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4335b1:::benchmark_data_smithi159_162467_object56046:head# 2024-04-15T18:42:59.645 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe43a264:::benchmark_data_smithi159_162467_object62025:head# 2024-04-15T18:42:59.646 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe43c3e5:::benchmark_data_smithi159_162467_object8529:head# 2024-04-15T18:42:59.646 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe43eb1a:::benchmark_data_smithi159_162467_object46962:head# 2024-04-15T18:42:59.646 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe442c47:::benchmark_data_smithi159_162467_object17891:head# 2024-04-15T18:42:59.647 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe44552a:::benchmark_data_smithi159_162467_object35022:head# 2024-04-15T18:42:59.647 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe455260:::benchmark_data_smithi159_162467_object30417:head# 2024-04-15T18:42:59.648 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4b173c:::benchmark_data_smithi159_162467_object66977:head# 2024-04-15T18:42:59.648 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4b6396:::benchmark_data_smithi159_162467_object31252:head# 2024-04-15T18:42:59.648 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4dcfe6:::benchmark_data_smithi159_162467_object39282:head# 2024-04-15T18:42:59.649 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4de8b1:::benchmark_data_smithi159_162467_object20043:head# 2024-04-15T18:42:59.649 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4e48f1:::benchmark_data_smithi159_162467_object38615:head# 2024-04-15T18:42:59.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4e82a7:::benchmark_data_smithi159_162467_object43804:head# 2024-04-15T18:42:59.650 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4eb38c:::benchmark_data_smithi159_162467_object19463:head# 2024-04-15T18:42:59.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4ec0ce:::benchmark_data_smithi159_162467_object14432:head# 2024-04-15T18:42:59.651 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4eca8a:::benchmark_data_smithi159_162467_object49135:head# 2024-04-15T18:42:59.652 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4eff37:::benchmark_data_smithi159_162467_object33088:head# 2024-04-15T18:42:59.652 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe4ff58b:::benchmark_data_smithi159_162467_object16742:head# 2024-04-15T18:42:59.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe501f07:::benchmark_data_smithi159_162467_object6619:head# 2024-04-15T18:42:59.653 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe503231:::benchmark_data_smithi159_162467_object71217:head# 2024-04-15T18:42:59.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe51f139:::benchmark_data_smithi159_162467_object27040:head# 2024-04-15T18:42:59.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe532428:::benchmark_data_smithi159_162467_object71252:head# 2024-04-15T18:42:59.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe554688:::benchmark_data_smithi159_162467_object51301:head# 2024-04-15T18:42:59.654 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe56148d:::benchmark_data_smithi159_162467_object43146:head# 2024-04-15T18:42:59.655 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe56e950:::benchmark_data_smithi159_162467_object42721:head# 2024-04-15T18:42:59.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe57acae:::benchmark_data_smithi159_162467_object72924:head# 2024-04-15T18:42:59.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe587c16:::benchmark_data_smithi159_162467_object48744:head# 2024-04-15T18:42:59.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe589ef7:::benchmark_data_smithi159_162467_object39833:head# 2024-04-15T18:42:59.656 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5b64b1:::benchmark_data_smithi159_162467_object42160:head# 2024-04-15T18:42:59.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5c398c:::benchmark_data_smithi159_162467_object62546:head# 2024-04-15T18:42:59.657 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5c6833:::benchmark_data_smithi159_162467_object41390:head# 2024-04-15T18:42:59.658 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5ca259:::benchmark_data_smithi159_162467_object42779:head# 2024-04-15T18:42:59.658 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5cb045:::benchmark_data_smithi159_162467_object64946:head# 2024-04-15T18:42:59.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5ecdc5:::benchmark_data_smithi159_162467_object4141:head# 2024-04-15T18:42:59.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe5fdab6:::benchmark_data_smithi159_162467_object18335:head# 2024-04-15T18:42:59.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe612111:::benchmark_data_smithi159_162467_object70203:head# 2024-04-15T18:42:59.659 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe636c70:::benchmark_data_smithi159_162467_object49327:head# 2024-04-15T18:42:59.660 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe658f8b:::benchmark_data_smithi159_162467_object4871:head# 2024-04-15T18:42:59.660 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe691ff8:::benchmark_data_smithi159_162467_object62602:head# 2024-04-15T18:42:59.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6aa9cf:::benchmark_data_smithi159_162467_object46685:head# 2024-04-15T18:42:59.661 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6b7b3f:::benchmark_data_smithi159_162467_object72285:head# 2024-04-15T18:42:59.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6c2858:::benchmark_data_smithi159_162467_object39398:head# 2024-04-15T18:42:59.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6c9187:::benchmark_data_smithi159_162467_object29183:head# 2024-04-15T18:42:59.662 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6ef2b1:::benchmark_data_smithi159_162467_object74239:head# 2024-04-15T18:42:59.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe6f5178:::benchmark_data_smithi159_162467_object2237:head# 2024-04-15T18:42:59.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe707dfd:::benchmark_data_smithi159_162467_object31942:head# 2024-04-15T18:42:59.663 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe713489:::benchmark_data_smithi159_162467_object58419:head# 2024-04-15T18:42:59.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7142f2:::benchmark_data_smithi159_162467_object50423:head# 2024-04-15T18:42:59.664 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe718efb:::benchmark_data_smithi159_162467_object10346:head# 2024-04-15T18:42:59.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe71b1d9:::benchmark_data_smithi159_162467_object62687:head# 2024-04-15T18:42:59.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe723778:::benchmark_data_smithi159_162467_object66828:head# 2024-04-15T18:42:59.665 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7381a9:::benchmark_data_smithi159_162467_object73845:head# 2024-04-15T18:42:59.666 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe746d18:::benchmark_data_smithi159_162467_object54304:head# 2024-04-15T18:42:59.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe747460:::benchmark_data_smithi159_162467_object69524:head# 2024-04-15T18:42:59.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe74805d:::benchmark_data_smithi159_162467_object63973:head# 2024-04-15T18:42:59.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe755ea6:::benchmark_data_smithi159_162467_object1300:head# 2024-04-15T18:42:59.667 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe79374b:::benchmark_data_smithi159_162467_object41851:head# 2024-04-15T18:42:59.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe797f34:::benchmark_data_smithi159_162467_object21839:head# 2024-04-15T18:42:59.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe79a329:::benchmark_data_smithi159_162467_object31048:head# 2024-04-15T18:42:59.668 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7a03a7:::benchmark_data_smithi159_162467_object16511:head# 2024-04-15T18:42:59.669 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7adc6f:::benchmark_data_smithi159_162467_object33290:head# 2024-04-15T18:42:59.669 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7b1928:::benchmark_data_smithi159_162467_object54863:head# 2024-04-15T18:42:59.670 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7b4b5a:::benchmark_data_smithi159_162467_object59505:head# 2024-04-15T18:42:59.670 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7b6ae4:::benchmark_data_smithi159_162467_object42906:head# 2024-04-15T18:42:59.671 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7ecfa8:::benchmark_data_smithi159_162467_object53874:head# 2024-04-15T18:42:59.671 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7ed2c5:::benchmark_data_smithi159_162467_object43298:head# 2024-04-15T18:42:59.671 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe7fc354:::benchmark_data_smithi159_162467_object51835:head# 2024-04-15T18:42:59.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe804f94:::benchmark_data_smithi159_162467_object11054:head# 2024-04-15T18:42:59.672 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe810e20:::benchmark_data_smithi159_162467_object22013:head# 2024-04-15T18:42:59.673 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8607b9:::benchmark_data_smithi159_162467_object27168:head# 2024-04-15T18:42:59.673 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe89b6df:::benchmark_data_smithi159_162467_object32285:head# 2024-04-15T18:42:59.673 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe89f56f:::benchmark_data_smithi159_162467_object32139:head# 2024-04-15T18:42:59.674 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8b6090:::benchmark_data_smithi159_162467_object56567:head# 2024-04-15T18:42:59.674 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8cb652:::benchmark_data_smithi159_162467_object66298:head# 2024-04-15T18:42:59.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8d4f95:::benchmark_data_smithi159_162467_object72149:head# 2024-04-15T18:42:59.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8ea7f6:::benchmark_data_smithi159_162467_object57469:head# 2024-04-15T18:42:59.675 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8ecc09:::benchmark_data_smithi159_162467_object45410:head# 2024-04-15T18:42:59.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8edaca:::benchmark_data_smithi159_162467_object31786:head# 2024-04-15T18:42:59.676 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe8f09b9:::benchmark_data_smithi159_162467_object72549:head# 2024-04-15T18:42:59.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe908464:::benchmark_data_smithi159_162467_object65760:head# 2024-04-15T18:42:59.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe91eb01:::benchmark_data_smithi159_162467_object46412:head# 2024-04-15T18:42:59.677 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe925ea5:::benchmark_data_smithi159_162467_object55324:head# 2024-04-15T18:42:59.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9434ac:::benchmark_data_smithi159_162467_object34849:head# 2024-04-15T18:42:59.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe94cedb:::benchmark_data_smithi159_162467_object913:head# 2024-04-15T18:42:59.678 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe962a8b:::benchmark_data_smithi159_162467_object6940:head# 2024-04-15T18:42:59.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe971e86:::benchmark_data_smithi159_162467_object25189:head# 2024-04-15T18:42:59.679 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe979c2f:::benchmark_data_smithi159_162467_object61871:head# 2024-04-15T18:42:59.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9848bc:::benchmark_data_smithi159_162467_object13910:head# 2024-04-15T18:42:59.680 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe990a31:::benchmark_data_smithi159_162467_object58716:head# 2024-04-15T18:42:59.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe99185f:::benchmark_data_smithi159_162467_object55087:head# 2024-04-15T18:42:59.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe999c47:::benchmark_data_smithi159_162467_object61813:head# 2024-04-15T18:42:59.681 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9a0260:::benchmark_data_smithi159_162467_object4533:head# 2024-04-15T18:42:59.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9ae324:::benchmark_data_smithi159_162467_object14922:head# 2024-04-15T18:42:59.682 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9bc756:::benchmark_data_smithi159_162467_object67982:head# 2024-04-15T18:42:59.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9c5410:::benchmark_data_smithi159_162467_object19286:head# 2024-04-15T18:42:59.683 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fe9f47be:::benchmark_data_smithi159_162467_object31531:head# 2024-04-15T18:42:59.684 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea0be38:::benchmark_data_smithi159_162467_object72156:head# 2024-04-15T18:42:59.684 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea1bf9c:::benchmark_data_smithi159_162467_object6817:head# 2024-04-15T18:42:59.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea258e7:::benchmark_data_smithi159_162467_object20888:head# 2024-04-15T18:42:59.685 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea2d121:::benchmark_data_smithi159_162467_object67904:head# 2024-04-15T18:42:59.686 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea337cb:::benchmark_data_smithi159_162467_object51004:head# 2024-04-15T18:42:59.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea37e12:::benchmark_data_smithi159_162467_object29744:head# 2024-04-15T18:42:59.687 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea715a5:::benchmark_data_smithi159_162467_object35075:head# 2024-04-15T18:42:59.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea71d79:::benchmark_data_smithi159_162467_object44956:head# 2024-04-15T18:42:59.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea79ca0:::benchmark_data_smithi159_162467_object72055:head# 2024-04-15T18:42:59.688 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea8107d:::benchmark_data_smithi159_162467_object57957:head# 2024-04-15T18:42:59.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea8361c:::benchmark_data_smithi159_162467_object12911:head# 2024-04-15T18:42:59.689 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea8ccbc:::benchmark_data_smithi159_162467_object41567:head# 2024-04-15T18:42:59.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea8ff76:::benchmark_data_smithi159_162467_object12493:head# 2024-04-15T18:42:59.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea92a0f:::benchmark_data_smithi159_162467_object10714:head# 2024-04-15T18:42:59.690 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fea9b193:::benchmark_data_smithi159_162467_object53349:head# 2024-04-15T18:42:59.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feaddc79:::benchmark_data_smithi159_162467_object22580:head# 2024-04-15T18:42:59.691 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feae5736:::benchmark_data_smithi159_162467_object4495:head# 2024-04-15T18:42:59.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feae5b43:::benchmark_data_smithi159_162467_object69750:head# 2024-04-15T18:42:59.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feafb104:::benchmark_data_smithi159_162467_object47897:head# 2024-04-15T18:42:59.692 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb0c962:::benchmark_data_smithi159_162467_object55414:head# 2024-04-15T18:42:59.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb1a506:::benchmark_data_smithi159_162467_object59791:head# 2024-04-15T18:42:59.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb1beeb:::benchmark_data_smithi159_162467_object35855:head# 2024-04-15T18:42:59.693 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb265a7:::benchmark_data_smithi159_162467_object64215:head# 2024-04-15T18:42:59.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb346d5:::benchmark_data_smithi159_162467_object43592:head# 2024-04-15T18:42:59.694 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb3a988:::benchmark_data_smithi159_162467_object51087:head# 2024-04-15T18:42:59.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb3c7e4:::benchmark_data_smithi159_162467_object49712:head# 2024-04-15T18:42:59.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb426a7:::benchmark_data_smithi159_162467_object25096:head# 2024-04-15T18:42:59.695 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb461f7:::benchmark_data_smithi159_162467_object29896:head# 2024-04-15T18:42:59.696 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb6bde9:::benchmark_data_smithi159_162467_object46652:head# 2024-04-15T18:42:59.697 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb6ce70:::benchmark_data_smithi159_162467_object42044:head# 2024-04-15T18:42:59.697 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb7c1dc:::benchmark_data_smithi159_162467_object1880:head# 2024-04-15T18:42:59.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb7ed20:::benchmark_data_smithi159_162467_object24662:head# 2024-04-15T18:42:59.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feb88a9f:::benchmark_data_smithi159_162467_object19140:head# 2024-04-15T18:42:59.698 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feba87a6:::benchmark_data_smithi159_162467_object56246:head# 2024-04-15T18:42:59.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febada9d:::benchmark_data_smithi159_162467_object39852:head# 2024-04-15T18:42:59.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febb6f26:::benchmark_data_smithi159_162467_object34952:head# 2024-04-15T18:42:59.699 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febb9eb1:::benchmark_data_smithi159_162467_object59962:head# 2024-04-15T18:42:59.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febcc80a:::benchmark_data_smithi159_162467_object24053:head# 2024-04-15T18:42:59.700 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febd3a4e:::benchmark_data_smithi159_162467_object12530:head# 2024-04-15T18:42:59.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:febdf3f0:::benchmark_data_smithi159_162467_object72666:head# 2024-04-15T18:42:59.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec0eaf2:::benchmark_data_smithi159_162467_object14902:head# 2024-04-15T18:42:59.701 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec0faa6:::benchmark_data_smithi159_162467_object68653:head# 2024-04-15T18:42:59.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec1839c:::benchmark_data_smithi159_162467_object24259:head# 2024-04-15T18:42:59.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec2ba19:::benchmark_data_smithi159_162467_object62050:head# 2024-04-15T18:42:59.702 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec2f4ae:::benchmark_data_smithi159_162467_object36110:head# 2024-04-15T18:42:59.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec33c1c:::benchmark_data_smithi159_162467_object71866:head# 2024-04-15T18:42:59.703 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec499b2:::benchmark_data_smithi159_162467_object7391:head# 2024-04-15T18:42:59.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec7ce50:::benchmark_data_smithi159_162467_object57717:head# 2024-04-15T18:42:59.704 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec887ae:::benchmark_data_smithi159_162467_object33662:head# 2024-04-15T18:42:59.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fec9f9f4:::benchmark_data_smithi159_162467_object59551:head# 2024-04-15T18:42:59.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecae454:::benchmark_data_smithi159_162467_object55000:head# 2024-04-15T18:42:59.705 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecbb576:::benchmark_data_smithi159_162467_object484:head# 2024-04-15T18:42:59.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feccceb4:::benchmark_data_smithi159_162467_object17622:head# 2024-04-15T18:42:59.706 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecd874a:::benchmark_data_smithi159_162467_object33784:head# 2024-04-15T18:42:59.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecf37b3:::benchmark_data_smithi159_162467_object48245:head# 2024-04-15T18:42:59.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecfa35f:::benchmark_data_smithi159_162467_object68944:head# 2024-04-15T18:42:59.707 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fecfe716:::benchmark_data_smithi159_162467_object16335:head# 2024-04-15T18:42:59.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed00a47:::benchmark_data_smithi159_162467_object28151:head# 2024-04-15T18:42:59.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed0f75b:::benchmark_data_smithi159_162467_object44031:head# 2024-04-15T18:42:59.708 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed1063a:::benchmark_data_smithi159_162467_object19854:head# 2024-04-15T18:42:59.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed1c5e3:::benchmark_data_smithi159_162467_object4670:head# 2024-04-15T18:42:59.709 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed314b2:::benchmark_data_smithi159_162467_object24463:head# 2024-04-15T18:42:59.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed40bf1:::benchmark_data_smithi159_162467_object12827:head# 2024-04-15T18:42:59.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed41fe9:::benchmark_data_smithi159_162467_object63210:head# 2024-04-15T18:42:59.710 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed466f7:::benchmark_data_smithi159_162467_object49877:head# 2024-04-15T18:42:59.711 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed5ee86:::benchmark_data_smithi159_162467_object55405:head# 2024-04-15T18:42:59.711 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed60214:::benchmark_data_smithi159_162467_object3201:head# 2024-04-15T18:42:59.712 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed6b55c:::benchmark_data_smithi159_162467_object49583:head# 2024-04-15T18:42:59.712 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fed9963f:::benchmark_data_smithi159_162467_object74226:head# 2024-04-15T18:42:59.712 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feda7789:::benchmark_data_smithi159_162467_object5700:head# 2024-04-15T18:42:59.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fedb2d04:::benchmark_data_smithi159_162467_object22078:head# 2024-04-15T18:42:59.713 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fedb45b2:::benchmark_data_smithi159_162467_object26255:head# 2024-04-15T18:42:59.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fede9505:::benchmark_data_smithi159_162467_object30095:head# 2024-04-15T18:42:59.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fedf2e31:::benchmark_data_smithi159_162467_object35519:head# 2024-04-15T18:42:59.714 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fedff0ed:::benchmark_data_smithi159_162467_object49045:head# 2024-04-15T18:42:59.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee069eb:::benchmark_data_smithi159_162467_object62445:head# 2024-04-15T18:42:59.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee088a9:::benchmark_data_smithi159_162467_object2417:head# 2024-04-15T18:42:59.715 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee1e133:::benchmark_data_smithi159_162467_object12380:head# 2024-04-15T18:42:59.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee3a78e:::benchmark_data_smithi159_162467_object14527:head# 2024-04-15T18:42:59.716 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee48834:::benchmark_data_smithi159_162467_object50947:head# 2024-04-15T18:42:59.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee5b571:::benchmark_data_smithi159_162467_object9279:head# 2024-04-15T18:42:59.717 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee612e7:::benchmark_data_smithi159_162467_object32232:head# 2024-04-15T18:42:59.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee6b553:::benchmark_data_smithi159_162467_object53361:head# 2024-04-15T18:42:59.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee74ca4:::benchmark_data_smithi159_162467_object21110:head# 2024-04-15T18:42:59.718 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee82fa4:::benchmark_data_smithi159_162467_object9021:head# 2024-04-15T18:42:59.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee8df8b:::benchmark_data_smithi159_162467_object26120:head# 2024-04-15T18:42:59.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee8ec3f:::benchmark_data_smithi159_162467_object55032:head# 2024-04-15T18:42:59.719 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee8f290:::benchmark_data_smithi159_162467_object41362:head# 2024-04-15T18:42:59.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee91508:::benchmark_data_smithi159_162467_object64634:head# 2024-04-15T18:42:59.720 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fee9c536:::benchmark_data_smithi159_162467_object1627:head# 2024-04-15T18:42:59.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feea01b7:::benchmark_data_smithi159_162467_object8338:head# 2024-04-15T18:42:59.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feea5f0a:::benchmark_data_smithi159_162467_object64992:head# 2024-04-15T18:42:59.721 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feeb2d79:::benchmark_data_smithi159_162467_object26047:head# 2024-04-15T18:42:59.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feec5096:::benchmark_data_smithi159_162467_object57936:head# 2024-04-15T18:42:59.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feed4a6e:::benchmark_data_smithi159_162467_object41688:head# 2024-04-15T18:42:59.722 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feee9cc7:::benchmark_data_smithi159_162467_object940:head# 2024-04-15T18:42:59.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feeea886:::benchmark_data_smithi159_162467_object66766:head# 2024-04-15T18:42:59.723 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feeee089:::benchmark_data_smithi159_162467_object57722:head# 2024-04-15T18:42:59.724 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feef2dbe:::benchmark_data_smithi159_162467_object70895:head# 2024-04-15T18:42:59.724 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feef415c:::benchmark_data_smithi159_162467_object11822:head# 2024-04-15T18:42:59.725 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feef9bcc:::benchmark_data_smithi159_162467_object29197:head# 2024-04-15T18:42:59.725 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef1568e:::benchmark_data_smithi159_162467_object11699:head# 2024-04-15T18:42:59.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef28e18:::benchmark_data_smithi159_162467_object64063:head# 2024-04-15T18:42:59.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef291d4:::benchmark_data_smithi159_162467_object45141:head# 2024-04-15T18:42:59.726 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef46e2c:::benchmark_data_smithi159_162467_object37174:head# 2024-04-15T18:42:59.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef4f837:::benchmark_data_smithi159_162467_object21937:head# 2024-04-15T18:42:59.727 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef559dc:::benchmark_data_smithi159_162467_object43431:head# 2024-04-15T18:42:59.728 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef58836:::benchmark_data_smithi159_162467_object52250:head# 2024-04-15T18:42:59.728 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef61c0a:::benchmark_data_smithi159_162467_object39075:head# 2024-04-15T18:42:59.729 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef623ff:::benchmark_data_smithi159_162467_object1463:head# 2024-04-15T18:42:59.729 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef6b9cd:::benchmark_data_smithi159_162467_object53800:head# 2024-04-15T18:42:59.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef71de1:::benchmark_data_smithi159_162467_object40241:head# 2024-04-15T18:42:59.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef73c2d:::benchmark_data_smithi159_162467_object44468:head# 2024-04-15T18:42:59.730 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef7efdc:::benchmark_data_smithi159_162467_object13508:head# 2024-04-15T18:42:59.731 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fef9d882:::benchmark_data_smithi159_162467_object70441:head# 2024-04-15T18:42:59.731 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefb1381:::benchmark_data_smithi159_162467_object11482:head# 2024-04-15T18:42:59.731 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefc28a9:::benchmark_data_smithi159_162467_object66048:head# 2024-04-15T18:42:59.731 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefc36f6:::benchmark_data_smithi159_162467_object22281:head# 2024-04-15T18:42:59.732 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefc3a33:::benchmark_data_smithi159_162467_object1079:head# 2024-04-15T18:42:59.732 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefdd714:::benchmark_data_smithi159_162467_object41299:head# 2024-04-15T18:42:59.733 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefe1f62:::benchmark_data_smithi159_162467_object56261:head# 2024-04-15T18:42:59.733 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefe2ff3:::benchmark_data_smithi159_162467_object69189:head# 2024-04-15T18:42:59.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefe66df:::benchmark_data_smithi159_162467_object42229:head# 2024-04-15T18:42:59.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feff2f9f:::benchmark_data_smithi159_162467_object27975:head# 2024-04-15T18:42:59.734 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feff3cdf:::benchmark_data_smithi159_162467_object22234:head# 2024-04-15T18:42:59.735 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feff8ec4:::benchmark_data_smithi159_162467_object24352:head# 2024-04-15T18:42:59.735 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feffa522:::benchmark_data_smithi159_162467_object41599:head# 2024-04-15T18:42:59.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:feffe4e4:::benchmark_data_smithi159_162467_object54313:head# 2024-04-15T18:42:59.736 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fefff879:::benchmark_data_smithi159_162467_object69481:head# 2024-04-15T18:42:59.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff00608b:::benchmark_data_smithi159_162467_object9851:head# 2024-04-15T18:42:59.737 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff015427:::benchmark_data_smithi159_162467_object62824:head# 2024-04-15T18:42:59.738 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff04667d:::benchmark_data_smithi159_162467_object21125:head# 2024-04-15T18:42:59.738 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff063723:::benchmark_data_smithi159_162467_object36927:head# 2024-04-15T18:42:59.739 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff0661ec:::benchmark_data_smithi159_162467_object15468:head# 2024-04-15T18:42:59.739 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff08b0b9:::benchmark_data_smithi159_162467_object73948:head# 2024-04-15T18:42:59.740 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff095a50:::benchmark_data_smithi159_162467_object60035:head# 2024-04-15T18:42:59.740 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff09addc:::benchmark_data_smithi159_162467_object26742:head# 2024-04-15T18:42:59.741 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff0a9c33:::benchmark_data_smithi159_162467_object17388:head# 2024-04-15T18:42:59.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff0af861:::benchmark_data_smithi159_162467_object20881:head# 2024-04-15T18:42:59.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff0b98c9:::benchmark_data_smithi159_162467_object62721:head# 2024-04-15T18:42:59.742 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff0cfecc:::benchmark_data_smithi159_162467_object56573:head# 2024-04-15T18:42:59.743 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1098d3:::benchmark_data_smithi159_162467_object35049:head# 2024-04-15T18:42:59.743 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1125f9:::benchmark_data_smithi159_162467_object74210:head# 2024-04-15T18:42:59.743 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff121888:::benchmark_data_smithi159_162467_object21109:head# 2024-04-15T18:42:59.744 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff122fd1:::benchmark_data_smithi159_162467_object18889:head# 2024-04-15T18:42:59.744 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff12a48d:::benchmark_data_smithi159_162467_object40751:head# 2024-04-15T18:42:59.744 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff12aa20:::benchmark_data_smithi159_162467_object30017:head# 2024-04-15T18:42:59.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff12fbc8:::benchmark_data_smithi159_162467_object31001:head# 2024-04-15T18:42:59.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff14d856:::benchmark_data_smithi159_162467_object57669:head# 2024-04-15T18:42:59.745 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1560bb:::benchmark_data_smithi159_162467_object51308:head# 2024-04-15T18:42:59.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff169343:::benchmark_data_smithi159_162467_object1010:head# 2024-04-15T18:42:59.746 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff16ef4f:::benchmark_data_smithi159_162467_object50315:head# 2024-04-15T18:42:59.747 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff17b64e:::benchmark_data_smithi159_162467_object2708:head# 2024-04-15T18:42:59.747 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff190bf4:::benchmark_data_smithi159_162467_object10268:head# 2024-04-15T18:42:59.748 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1a92c1:::benchmark_data_smithi159_162467_object45781:head# 2024-04-15T18:42:59.748 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1ab390:::benchmark_data_smithi159_162467_object66051:head# 2024-04-15T18:42:59.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1b6b87:::benchmark_data_smithi159_162467_object16192:head# 2024-04-15T18:42:59.749 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1bc5d8:::benchmark_data_smithi159_162467_object57500:head# 2024-04-15T18:42:59.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1bde68:::benchmark_data_smithi159_162467_object44148:head# 2024-04-15T18:42:59.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1c37c8:::benchmark_data_smithi159_162467_object37168:head# 2024-04-15T18:42:59.750 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1c6c8b:::benchmark_data_smithi159_162467_object9956:head# 2024-04-15T18:42:59.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1e3b7d:::benchmark_data_smithi159_162467_object16822:head# 2024-04-15T18:42:59.751 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1e8de8:::benchmark_data_smithi159_162467_object50103:head# 2024-04-15T18:42:59.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff1ed30c:::benchmark_data_smithi159_162467_object69299:head# 2024-04-15T18:42:59.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2163b6:::benchmark_data_smithi159_162467_object4063:head# 2024-04-15T18:42:59.752 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff21bd63:::benchmark_data_smithi159_162467_object54496:head# 2024-04-15T18:42:59.753 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff234ed9:::benchmark_data_smithi159_162467_object48350:head# 2024-04-15T18:42:59.753 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff237335:::benchmark_data_smithi159_162467_object62543:head# 2024-04-15T18:42:59.753 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff239df7:::benchmark_data_smithi159_162467_object67085:head# 2024-04-15T18:42:59.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff25547a:::benchmark_data_smithi159_162467_object71312:head# 2024-04-15T18:42:59.754 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff268992:::benchmark_data_smithi159_162467_object4121:head# 2024-04-15T18:42:59.755 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2834c3:::benchmark_data_smithi159_162467_object36809:head# 2024-04-15T18:42:59.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff28d63c:::benchmark_data_smithi159_162467_object7181:head# 2024-04-15T18:42:59.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff28f1e3:::benchmark_data_smithi159_162467_object12460:head# 2024-04-15T18:42:59.758 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff292d6d:::benchmark_data_smithi159_162467_object45935:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff29b382:::benchmark_data_smithi159_162467_object22817:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff29d624:::benchmark_data_smithi159_162467_object12044:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff29dcd0:::benchmark_data_smithi159_162467_object16:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2a422f:::benchmark_data_smithi159_162467_object20196:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2a8bd6:::benchmark_data_smithi159_162467_object21982:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2b2e56:::benchmark_data_smithi159_162467_object30180:head# 2024-04-15T18:42:59.759 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2b4413:::benchmark_data_smithi159_162467_object69632:head# 2024-04-15T18:42:59.760 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2b78a9:::benchmark_data_smithi159_162467_object27779:head# 2024-04-15T18:42:59.760 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2b8524:::benchmark_data_smithi159_162467_object3314:head# 2024-04-15T18:42:59.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2d923a:::benchmark_data_smithi159_162467_object29766:head# 2024-04-15T18:42:59.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2f1b56:::benchmark_data_smithi159_162467_object51299:head# 2024-04-15T18:42:59.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff2f417c:::benchmark_data_smithi159_162467_object74581:head# 2024-04-15T18:42:59.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff307349:::benchmark_data_smithi159_162467_object23913:head# 2024-04-15T18:42:59.761 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff310dbc:::benchmark_data_smithi159_162467_object41545:head# 2024-04-15T18:42:59.762 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff312448:::benchmark_data_smithi159_162467_object52248:head# 2024-04-15T18:42:59.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff32d895:::benchmark_data_smithi159_162467_object67956:head# 2024-04-15T18:42:59.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3346b8:::benchmark_data_smithi159_162467_object11318:head# 2024-04-15T18:42:59.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff342409:::benchmark_data_smithi159_162467_object21983:head# 2024-04-15T18:42:59.763 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff34da6c:::benchmark_data_smithi159_162467_object55045:head# 2024-04-15T18:42:59.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff34ef00:::benchmark_data_smithi159_162467_object31051:head# 2024-04-15T18:42:59.764 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff358418:::benchmark_data_smithi159_162467_object69504:head# 2024-04-15T18:42:59.765 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3589d7:::benchmark_data_smithi159_162467_object30451:head# 2024-04-15T18:42:59.765 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff358f59:::benchmark_data_smithi159_162467_object63998:head# 2024-04-15T18:42:59.765 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff37d2cb:::benchmark_data_smithi159_162467_object70864:head# 2024-04-15T18:42:59.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff39e2ba:::benchmark_data_smithi159_162467_object44382:head# 2024-04-15T18:42:59.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff39e906:::benchmark_data_smithi159_162467_object71986:head# 2024-04-15T18:42:59.766 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff39f84f:::benchmark_data_smithi159_162467_object13770:head# 2024-04-15T18:42:59.767 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3acac0:::benchmark_data_smithi159_162467_object30815:head# 2024-04-15T18:42:59.767 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3aff7e:::benchmark_data_smithi159_162467_object64135:head# 2024-04-15T18:42:59.768 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3b0748:::benchmark_data_smithi159_162467_object2972:head# 2024-04-15T18:42:59.768 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3b591f:::benchmark_data_smithi159_162467_object72831:head# 2024-04-15T18:42:59.768 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3b7916:::benchmark_data_smithi159_162467_object71:head# 2024-04-15T18:42:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3ef6d4:::benchmark_data_smithi159_162467_object1564:head# 2024-04-15T18:42:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3f1e50:::benchmark_data_smithi159_162467_object44621:head# 2024-04-15T18:42:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff3fe991:::benchmark_data_smithi159_162467_object19201:head# 2024-04-15T18:42:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff402ea4:::benchmark_data_smithi159_162467_object17687:head# 2024-04-15T18:42:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff41bd57:::benchmark_data_smithi159_162467_object52534:head# 2024-04-15T18:42:59.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff420329:::benchmark_data_smithi159_162467_object11980:head# 2024-04-15T18:42:59.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff43351c:::benchmark_data_smithi159_162467_object23993:head# 2024-04-15T18:42:59.772 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4416c3:::benchmark_data_smithi159_162467_object4191:head# 2024-04-15T18:42:59.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4546f5:::benchmark_data_smithi159_162467_object62318:head# 2024-04-15T18:42:59.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff464785:::benchmark_data_smithi159_162467_object59856:head# 2024-04-15T18:42:59.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff48a4d4:::benchmark_data_smithi159_162467_object30154:head# 2024-04-15T18:42:59.773 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff497098:::benchmark_data_smithi159_162467_object64296:head# 2024-04-15T18:42:59.774 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4a3efb:::benchmark_data_smithi159_162467_object313:head# 2024-04-15T18:42:59.774 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4c3a93:::benchmark_data_smithi159_162467_object9872:head# 2024-04-15T18:42:59.775 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4c5e6e:::benchmark_data_smithi159_162467_object28079:head# 2024-04-15T18:42:59.775 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff4d8eae:::benchmark_data_smithi159_162467_object35633:head# 2024-04-15T18:42:59.776 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff500c92:::benchmark_data_smithi159_162467_object35485:head# 2024-04-15T18:42:59.776 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff511817:::benchmark_data_smithi159_162467_object1793:head# 2024-04-15T18:42:59.777 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff521339:::benchmark_data_smithi159_162467_object50956:head# 2024-04-15T18:42:59.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff532409:::benchmark_data_smithi159_162467_object72134:head# 2024-04-15T18:42:59.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff54bd0e:::benchmark_data_smithi159_162467_object56466:head# 2024-04-15T18:42:59.778 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5503a4:::benchmark_data_smithi159_162467_object29402:head# 2024-04-15T18:42:59.779 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff585a5c:::benchmark_data_smithi159_162467_object31913:head# 2024-04-15T18:42:59.779 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5a418a:::benchmark_data_smithi159_162467_object38928:head# 2024-04-15T18:42:59.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5a53f7:::benchmark_data_smithi159_162467_object28345:head# 2024-04-15T18:42:59.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5a71d7:::benchmark_data_smithi159_162467_object61760:head# 2024-04-15T18:42:59.780 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5a7710:::benchmark_data_smithi159_162467_object52103:head# 2024-04-15T18:42:59.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5aad4a:::benchmark_data_smithi159_162467_object12466:head# 2024-04-15T18:42:59.781 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5b18cb:::benchmark_data_smithi159_162467_object29423:head# 2024-04-15T18:42:59.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5b44ed:::benchmark_data_smithi159_162467_object32983:head# 2024-04-15T18:42:59.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5ca28f:::benchmark_data_smithi159_162467_object20152:head# 2024-04-15T18:42:59.782 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5e1a22:::benchmark_data_smithi159_162467_object26103:head# 2024-04-15T18:42:59.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5e6489:::benchmark_data_smithi159_162467_object72689:head# 2024-04-15T18:42:59.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff5edf95:::benchmark_data_smithi159_162467_object45005:head# 2024-04-15T18:42:59.783 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff61a84c:::benchmark_data_smithi159_162467_object4602:head# 2024-04-15T18:42:59.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff62212a:::benchmark_data_smithi159_162467_object53501:head# 2024-04-15T18:42:59.784 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6276e1:::benchmark_data_smithi159_162467_object64984:head# 2024-04-15T18:42:59.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6332b3:::benchmark_data_smithi159_162467_object66143:head# 2024-04-15T18:42:59.785 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff640e45:::benchmark_data_smithi159_162467_object69680:head# 2024-04-15T18:42:59.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff64564b:::benchmark_data_smithi159_162467_object49062:head# 2024-04-15T18:42:59.786 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff64e52b:::benchmark_data_smithi159_162467_object49082:head# 2024-04-15T18:42:59.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff651ebc:::benchmark_data_smithi159_162467_object28958:head# 2024-04-15T18:42:59.787 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff65b818:::benchmark_data_smithi159_162467_object53132:head# 2024-04-15T18:42:59.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff65ebc1:::benchmark_data_smithi159_162467_object5245:head# 2024-04-15T18:42:59.788 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff67673d:::benchmark_data_smithi159_162467_object71445:head# 2024-04-15T18:42:59.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6767d3:::benchmark_data_smithi159_162467_object60403:head# 2024-04-15T18:42:59.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff67b2fa:::benchmark_data_smithi159_162467_object51800:head# 2024-04-15T18:42:59.789 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff697b42:::benchmark_data_smithi159_162467_object51710:head# 2024-04-15T18:42:59.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6a5193:::benchmark_data_smithi159_162467_object1311:head# 2024-04-15T18:42:59.790 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6b2c6d:::benchmark_data_smithi159_162467_object61660:head# 2024-04-15T18:42:59.791 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6ba3c8:::benchmark_data_smithi159_162467_object30619:head# 2024-04-15T18:42:59.791 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6d2888:::benchmark_data_smithi159_162467_object47379:head# 2024-04-15T18:42:59.791 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6d54b6:::benchmark_data_smithi159_162467_object66448:head# 2024-04-15T18:42:59.792 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6e6b31:::benchmark_data_smithi159_162467_object33056:head# 2024-04-15T18:42:59.792 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6ed8d7:::benchmark_data_smithi159_162467_object45502:head# 2024-04-15T18:42:59.793 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6f4d42:::benchmark_data_smithi159_162467_object48682:head# 2024-04-15T18:42:59.793 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6f5e59:::benchmark_data_smithi159_162467_object14268:head# 2024-04-15T18:42:59.793 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff6fba6d:::benchmark_data_smithi159_162467_object47418:head# 2024-04-15T18:42:59.794 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff714913:::benchmark_data_smithi159_162467_object62218:head# 2024-04-15T18:42:59.794 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff72b552:::benchmark_data_smithi159_162467_object29550:head# 2024-04-15T18:42:59.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff745c75:::benchmark_data_smithi159_162467_object68231:head# 2024-04-15T18:42:59.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7546cd:::benchmark_data_smithi159_162467_object40969:head# 2024-04-15T18:42:59.795 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff75b66e:::benchmark_data_smithi159_162467_object18230:head# 2024-04-15T18:42:59.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff763ef7:::benchmark_data_smithi159_162467_object62217:head# 2024-04-15T18:42:59.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff765bea:::benchmark_data_smithi159_162467_object43131:head# 2024-04-15T18:42:59.796 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7689e2:::benchmark_data_smithi159_162467_object33761:head# 2024-04-15T18:42:59.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff769b2f:::benchmark_data_smithi159_162467_object12541:head# 2024-04-15T18:42:59.797 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7bbd02:::benchmark_data_smithi159_162467_object15358:head# 2024-04-15T18:42:59.798 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7c7178:::benchmark_data_smithi159_162467_object41310:head# 2024-04-15T18:42:59.798 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7c89ac:::benchmark_data_smithi159_162467_object14267:head# 2024-04-15T18:42:59.798 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7cc579:::benchmark_data_smithi159_162467_object57041:head# 2024-04-15T18:42:59.799 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7db8a4:::benchmark_data_smithi159_162467_object16738:head# 2024-04-15T18:42:59.799 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7dcceb:::benchmark_data_smithi159_162467_object44356:head# 2024-04-15T18:42:59.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7df66a:::benchmark_data_smithi159_162467_object48230:head# 2024-04-15T18:42:59.800 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff7f42a2:::benchmark_data_smithi159_162467_object59436:head# 2024-04-15T18:42:59.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff802c8c:::benchmark_data_smithi159_162467_object54880:head# 2024-04-15T18:42:59.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff80af7b:::benchmark_data_smithi159_162467_object41667:head# 2024-04-15T18:42:59.801 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff82aeaa:::benchmark_data_smithi159_162467_object68743:head# 2024-04-15T18:42:59.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff837dfe:::benchmark_data_smithi159_162467_object9880:head# 2024-04-15T18:42:59.802 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff83f477:::benchmark_data_smithi159_162467_object74011:head# 2024-04-15T18:42:59.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff840ea7:::benchmark_data_smithi159_162467_object20456:head# 2024-04-15T18:42:59.803 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff84db96:::benchmark_data_smithi159_162467_object55716:head# 2024-04-15T18:42:59.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff84dee8:::benchmark_data_smithi159_162467_object21903:head# 2024-04-15T18:42:59.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff86bdb4:::benchmark_data_smithi159_162467_object3243:head# 2024-04-15T18:42:59.804 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff87928e:::benchmark_data_smithi159_162467_object58124:head# 2024-04-15T18:42:59.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff87c925:::benchmark_data_smithi159_162467_object35472:head# 2024-04-15T18:42:59.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff8bf28c:::benchmark_data_smithi159_162467_object38356:head# 2024-04-15T18:42:59.805 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff8c586f:::benchmark_data_smithi159_162467_object21371:head# 2024-04-15T18:42:59.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff8c8a64:::benchmark_data_smithi159_162467_object28798:head# 2024-04-15T18:42:59.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff8db885:::benchmark_data_smithi159_162467_object2406:head# 2024-04-15T18:42:59.806 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff8e02b5:::benchmark_data_smithi159_162467_object61435:head# 2024-04-15T18:42:59.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff93b1b7:::benchmark_data_smithi159_162467_object60850:head# 2024-04-15T18:42:59.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9410eb:::benchmark_data_smithi159_162467_object63166:head# 2024-04-15T18:42:59.807 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff942703:::benchmark_data_smithi159_162467_object6231:head# 2024-04-15T18:42:59.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff96eb1b:::benchmark_data_smithi159_162467_object35606:head# 2024-04-15T18:42:59.808 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9703d0:::benchmark_data_smithi159_162467_object30897:head# 2024-04-15T18:42:59.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff97bb53:::benchmark_data_smithi159_162467_object974:head# 2024-04-15T18:42:59.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff98c471:::benchmark_data_smithi159_162467_object6076:head# 2024-04-15T18:42:59.809 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff98d532:::benchmark_data_smithi159_162467_object14612:head# 2024-04-15T18:42:59.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff99cfca:::benchmark_data_smithi159_162467_object4736:head# 2024-04-15T18:42:59.810 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9a026c:::benchmark_data_smithi159_162467_object20256:head# 2024-04-15T18:42:59.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9a0fb1:::benchmark_data_smithi159_162467_object50365:head# 2024-04-15T18:42:59.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9a6152:::benchmark_data_smithi159_162467_object70510:head# 2024-04-15T18:42:59.811 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9af1de:::benchmark_data_smithi159_162467_object73233:head# 2024-04-15T18:42:59.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9c91d0:::benchmark_data_smithi159_162467_object47012:head# 2024-04-15T18:42:59.812 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9de957:::benchmark_data_smithi159_162467_object30207:head# 2024-04-15T18:42:59.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9dee73:::benchmark_data_smithi159_162467_object39660:head# 2024-04-15T18:42:59.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ff9df0a7:::benchmark_data_smithi159_162467_object17141:head# 2024-04-15T18:42:59.813 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa06370:::benchmark_data_smithi159_162467_object47879:head# 2024-04-15T18:42:59.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa0e5ff:::benchmark_data_smithi159_162467_object5860:head# 2024-04-15T18:42:59.814 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa1d71c:::benchmark_data_smithi159_162467_object50830:head# 2024-04-15T18:42:59.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa38ff2:::benchmark_data_smithi159_162467_object62508:head# 2024-04-15T18:42:59.815 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa3e371:::benchmark_data_smithi159_162467_object61495:head# 2024-04-15T18:42:59.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa4ec56:::benchmark_data_smithi159_162467_object69037:head# 2024-04-15T18:42:59.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa518c9:::benchmark_data_smithi159_162467_object26162:head# 2024-04-15T18:42:59.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa683da:::benchmark_data_smithi159_162467_object28642:head# 2024-04-15T18:42:59.816 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa8455d:::benchmark_data_smithi159_162467_object39666:head# 2024-04-15T18:42:59.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa913b7:::benchmark_data_smithi159_162467_object23308:head# 2024-04-15T18:42:59.817 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffa93913:::benchmark_data_smithi159_162467_object64677:head# 2024-04-15T18:42:59.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffaa7052:::benchmark_data_smithi159_162467_object24953:head# 2024-04-15T18:42:59.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffaaf560:::benchmark_data_smithi159_162467_object67145:head# 2024-04-15T18:42:59.818 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffad3012:::benchmark_data_smithi159_162467_object46362:head# 2024-04-15T18:42:59.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffad34df:::benchmark_data_smithi159_162467_object33806:head# 2024-04-15T18:42:59.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffadc207:::benchmark_data_smithi159_162467_object4006:head# 2024-04-15T18:42:59.819 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffadc72e:::benchmark_data_smithi159_162467_object63365:head# 2024-04-15T18:42:59.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffae7f2c:::benchmark_data_smithi159_162467_object26837:head# 2024-04-15T18:42:59.820 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffaeaca3:::benchmark_data_smithi159_162467_object30436:head# 2024-04-15T18:42:59.821 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffafee84:::benchmark_data_smithi159_162467_object46185:head# 2024-04-15T18:42:59.821 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb09096:::benchmark_data_smithi159_162467_object41679:head# 2024-04-15T18:42:59.821 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb2be92:::benchmark_data_smithi159_162467_object34541:head# 2024-04-15T18:42:59.822 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb2fe9e:::benchmark_data_smithi159_162467_object66443:head# 2024-04-15T18:42:59.822 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb393b7:::benchmark_data_smithi159_162467_object69444:head# 2024-04-15T18:42:59.823 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb48fb0:::benchmark_data_smithi159_162467_object32046:head# 2024-04-15T18:42:59.823 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb4d660:::benchmark_data_smithi159_162467_object59488:head# 2024-04-15T18:42:59.823 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb5006e:::benchmark_data_smithi159_162467_object61654:head# 2024-04-15T18:42:59.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb632b9:::benchmark_data_smithi159_162467_object23151:head# 2024-04-15T18:42:59.824 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb7432c:::benchmark_data_smithi159_162467_object41868:head# 2024-04-15T18:42:59.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb90e67:::benchmark_data_smithi159_162467_object74750:head# 2024-04-15T18:42:59.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffb962ca:::benchmark_data_smithi159_162467_object71395:head# 2024-04-15T18:42:59.825 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffba3345:::benchmark_data_smithi159_162467_object67958:head# 2024-04-15T18:42:59.826 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffbb1d39:::benchmark_data_smithi159_162467_object30882:head# 2024-04-15T18:42:59.826 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffbc3794:::benchmark_data_smithi159_162467_object55380:head# 2024-04-15T18:42:59.827 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffbc82c7:::benchmark_data_smithi159_162467_object57848:head# 2024-04-15T18:42:59.827 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffbe4932:::benchmark_data_smithi159_162467_object37835:head# 2024-04-15T18:42:59.828 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffbf87fa:::benchmark_data_smithi159_162467_object14765:head# 2024-04-15T18:42:59.828 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc006b7:::benchmark_data_smithi159_162467_object61550:head# 2024-04-15T18:42:59.828 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc0a2b9:::benchmark_data_smithi159_162467_object55799:head# 2024-04-15T18:42:59.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc14d43:::benchmark_data_smithi159_162467_object55464:head# 2024-04-15T18:42:59.829 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc2ada5:::benchmark_data_smithi159_162467_object15393:head# 2024-04-15T18:42:59.830 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc32e0c:::benchmark_data_smithi159_162467_object14882:head# 2024-04-15T18:42:59.830 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc362a5:::benchmark_data_smithi159_162467_object72672:head# 2024-04-15T18:42:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc3c9a3:::benchmark_data_smithi159_162467_object73191:head# 2024-04-15T18:42:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc4a5e8:::benchmark_data_smithi159_162467_object15068:head# 2024-04-15T18:42:59.831 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc50369:::benchmark_data_smithi159_162467_object38032:head# 2024-04-15T18:42:59.832 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc66fd4:::benchmark_data_smithi159_162467_object10343:head# 2024-04-15T18:42:59.833 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc70090:::benchmark_data_smithi159_162467_object10170:head# 2024-04-15T18:42:59.833 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc78ac2:::benchmark_data_smithi159_162467_object39125:head# 2024-04-15T18:42:59.834 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffc90e6e:::benchmark_data_smithi159_162467_object41672:head# 2024-04-15T18:42:59.834 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffca8916:::benchmark_data_smithi159_162467_object52687:head# 2024-04-15T18:42:59.834 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffca9181:::benchmark_data_smithi159_162467_object65356:head# 2024-04-15T18:42:59.835 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcb0026:::benchmark_data_smithi159_162467_object4984:head# 2024-04-15T18:42:59.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcc5909:::benchmark_data_smithi159_162467_object71407:head# 2024-04-15T18:42:59.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcc95e8:::benchmark_data_smithi159_162467_object26968:head# 2024-04-15T18:42:59.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcd1a85:::benchmark_data_smithi159_162467_object4890:head# 2024-04-15T18:42:59.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcd4d00:::benchmark_data_smithi159_162467_object62048:head# 2024-04-15T18:42:59.836 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffce2b2c:::benchmark_data_smithi159_162467_object3419:head# 2024-04-15T18:42:59.837 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffcf07c3:::benchmark_data_smithi159_162467_object31835:head# 2024-04-15T18:42:59.837 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd00117:::benchmark_data_smithi159_162467_object64844:head# 2024-04-15T18:42:59.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd16a29:::benchmark_data_smithi159_162467_object39453:head# 2024-04-15T18:42:59.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd19d81:::benchmark_data_smithi159_162467_object56484:head# 2024-04-15T18:42:59.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd2cb02:::benchmark_data_smithi159_162467_object58766:head# 2024-04-15T18:42:59.838 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd2d493:::benchmark_data_smithi159_162467_object22238:head# 2024-04-15T18:42:59.839 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd3f836:::benchmark_data_smithi159_162467_object45103:head# 2024-04-15T18:42:59.839 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd46e23:::benchmark_data_smithi159_162467_object65836:head# 2024-04-15T18:42:59.840 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd4bba8:::benchmark_data_smithi159_162467_object60376:head# 2024-04-15T18:42:59.840 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd506ae:::benchmark_data_smithi159_162467_object37625:head# 2024-04-15T18:42:59.840 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd56c33:::benchmark_data_smithi159_162467_object52150:head# 2024-04-15T18:42:59.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd62eb7:::benchmark_data_smithi159_162467_object72204:head# 2024-04-15T18:42:59.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd6ac97:::benchmark_data_smithi159_162467_object48155:head# 2024-04-15T18:42:59.841 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffd89115:::benchmark_data_smithi159_162467_object6121:head# 2024-04-15T18:42:59.842 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdc79b6:::benchmark_data_smithi159_162467_object34976:head# 2024-04-15T18:42:59.842 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdcb249:::benchmark_data_smithi159_162467_object26740:head# 2024-04-15T18:42:59.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdcb542:::benchmark_data_smithi159_162467_object4737:head# 2024-04-15T18:42:59.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdd39de:::benchmark_data_smithi159_162467_object63539:head# 2024-04-15T18:42:59.843 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdd4f79:::benchmark_data_smithi159_162467_object14433:head# 2024-04-15T18:42:59.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffddb95f:::benchmark_data_smithi159_162467_object93:head# 2024-04-15T18:42:59.844 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdec8ef:::benchmark_data_smithi159_162467_object27010:head# 2024-04-15T18:42:59.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdf5d35:::benchmark_data_smithi159_162467_object24828:head# 2024-04-15T18:42:59.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffdf77cd:::benchmark_data_smithi159_162467_object63709:head# 2024-04-15T18:42:59.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe14c0f:::benchmark_data_smithi159_162467_object4361:head# 2024-04-15T18:42:59.845 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe20e34:::benchmark_data_smithi159_162467_object48660:head# 2024-04-15T18:42:59.846 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe257fe:::benchmark_data_smithi159_162467_object53945:head# 2024-04-15T18:42:59.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe258bb:::benchmark_data_smithi159_162467_object26769:head# 2024-04-15T18:42:59.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe291a9:::benchmark_data_smithi159_162467_object3997:head# 2024-04-15T18:42:59.847 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe4e4d2:::benchmark_data_smithi159_162467_object70430:head# 2024-04-15T18:42:59.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe5736d:::benchmark_data_smithi159_162467_object62413:head# 2024-04-15T18:42:59.848 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe630ed:::benchmark_data_smithi159_162467_object52717:head# 2024-04-15T18:42:59.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe958ef:::benchmark_data_smithi159_162467_object65555:head# 2024-04-15T18:42:59.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffe9f707:::benchmark_data_smithi159_162467_object48593:head# 2024-04-15T18:42:59.849 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffea32b6:::benchmark_data_smithi159_162467_object39745:head# 2024-04-15T18:42:59.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffec85b0:::benchmark_data_smithi159_162467_object57037:head# 2024-04-15T18:42:59.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffedc888:::benchmark_data_smithi159_162467_object21428:head# 2024-04-15T18:42:59.850 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffef2b19:::benchmark_data_smithi159_162467_object31771:head# 2024-04-15T18:42:59.851 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:ffef5dfd:::benchmark_data_smithi159_162467_object42348:head# 2024-04-15T18:42:59.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff0c4af:::benchmark_data_smithi159_162467_object30861:head# 2024-04-15T18:42:59.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff0e60f:::benchmark_data_smithi159_162467_object62832:head# 2024-04-15T18:42:59.852 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff34702:::benchmark_data_smithi159_162467_object60166:head# 2024-04-15T18:42:59.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff43abb:::benchmark_data_smithi159_162467_object33745:head# 2024-04-15T18:42:59.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff4dd82:::benchmark_data_smithi159_162467_object58070:head# 2024-04-15T18:42:59.853 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff5e123:::benchmark_data_smithi159_162467_object58358:head# 2024-04-15T18:42:59.854 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff6b223:::benchmark_data_smithi159_162467_object20153:head# 2024-04-15T18:42:59.854 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff7f3d4:::benchmark_data_smithi159_162467_object34835:head# 2024-04-15T18:42:59.854 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff9dc52:::benchmark_data_smithi159_162467_object11373:head# 2024-04-15T18:42:59.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fff9ebc1:::benchmark_data_smithi159_162467_object32438:head# 2024-04-15T18:42:59.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fffb83d2:::benchmark_data_smithi159_162467_object70803:head# 2024-04-15T18:42:59.855 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fffcf4d7:::benchmark_data_smithi159_162467_object17464:head# 2024-04-15T18:42:59.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fffd1deb:::benchmark_data_smithi159_162467_object35389:head# 2024-04-15T18:42:59.856 INFO:teuthology.orchestra.run.smithi159.stderr:Read #8:fffeead0:::benchmark_data_smithi159_162467_object16152:head# 2024-04-15T18:42:59.858 INFO:teuthology.orchestra.run.smithi159.stderr:Export successful 2024-04-15T18:42:59.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:43:00.006 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:00.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:00.113 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:00.150 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:00.179 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:00.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:00.283 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:00.337 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:00.369 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:00.373 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:43:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:00.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:00.476 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:00.510 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:00.879 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:00 smithi159 ceph-mon[45336]: pgmap v1934: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 3.1 MiB/s wr, 701 op/s; 27582/290829 objects degraded (9.484%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:43:01.004 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.11 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal --force --op remove --pgid 8.f 2024-04-15T18:43:01.065 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:01.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:01.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:00 smithi082 ceph-mon[44206]: pgmap v1934: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 3.1 MiB/s wr, 701 op/s; 27582/290829 objects degraded (9.484%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:43:01.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:01.196 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:01.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:00 smithi049 ceph-mon[40087]: pgmap v1934: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 3.1 MiB/s wr, 701 op/s; 27582/290829 objects degraded (9.484%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:43:01.248 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.11/config 2024-04-15T18:43:01.850 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:01.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:01.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:02.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:02.541 INFO:teuthology.orchestra.run.smithi159.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:43:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:02 smithi159 ceph-mon[45336]: pgmap v1935: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.3 MiB/s wr, 585 op/s; 27385/289005 objects degraded (9.476%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:43:03.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 27385/289005 objects degraded (9.476%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:02 smithi049 ceph-mon[40087]: pgmap v1935: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.3 MiB/s wr, 585 op/s; 27385/289005 objects degraded (9.476%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:43:03.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 27385/289005 objects degraded (9.476%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:03.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:02 smithi082 ceph-mon[44206]: pgmap v1935: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.9 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 1.3 MiB/s wr, 585 op/s; 27385/289005 objects degraded (9.476%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:43:03.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 27385/289005 objects degraded (9.476%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:03.622 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:03.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:03.727 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:03.768 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:04.289 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:04.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:04.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:04.458 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:04.906 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:04 smithi082 ceph-mon[44206]: pgmap v1936: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 26750/281427 objects degraded (9.505%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:43:04.906 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:04.906 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:112512 8.d 8:bf0d31e4:::benchmark_data_smithi159_162467_object10207:head [delete] snapc 0=[] ondisk+write+known_if_redirected e750)", 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:05.203692+0000", 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.015211384, 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015233685, 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:43:05.208 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 112512 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203692+0000", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203692+0000", 2024-04-15T18:43:05.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203693+0000", 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3999999999999999e-06 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203699+0000", 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6579999999999999e-06 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:43:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203712+0000", 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.309e-05 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.203718+0000", 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4019999999999999e-06 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.213411+0000", 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0096925629999999995 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.213558+0000", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000146973 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.213695+0000", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013666899999999999 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:112512 8.d e750/743)", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:05.218267+0000", 2024-04-15T18:43:05.212 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.000635879, 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00070938399999999997, 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218267+0000", 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218267+0000", 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218268+0000", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9100000000000003e-07 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218274+0000", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1369999999999999e-06 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218305+0000", 2024-04-15T18:43:05.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1281000000000001e-05 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218313+0000", 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8809999999999996e-06 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:112512 8.d e750/743)", 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:05.218945+0000", 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 4294967295.9999585, 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1908000000000004e-05, 2024-04-15T18:43:05.215 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "queued for pg", 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218945+0000", 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218945+0000", 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218945+0000", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0699999999999997e-07 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218950+0000", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4800000000000003e-06 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218977+0000", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7138e-05 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:43:05.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:05.218984+0000", 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9970000000000001e-06 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: ], 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 3 2024-04-15T18:43:05.218 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:43:05.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:04 smithi049 ceph-mon[40087]: pgmap v1936: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 26750/281427 objects degraded (9.505%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:43:05.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:05.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:04 smithi159 ceph-mon[45336]: pgmap v1936: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 656 op/s; 26750/281427 objects degraded (9.505%); 10 MiB/s, 161 objects/s recovering 2024-04-15T18:43:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:05.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:43:06.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:05 smithi049 ceph-mon[40087]: osdmap e750: 12 total, 11 up, 11 in 2024-04-15T18:43:06.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:05 smithi082 ceph-mon[44206]: osdmap e750: 12 total, 11 up, 11 in 2024-04-15T18:43:06.310 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:43:06.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:05 smithi159 ceph-mon[45336]: osdmap e750: 12 total, 11 up, 11 in 2024-04-15T18:43:06.932 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:43:07.051 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:07.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:07.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:07.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:06 smithi049 ceph-mon[40087]: pgmap v1938: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 26750/280266 objects degraded (9.545%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:43:07.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:43:07.252 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:07.273 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:06 smithi082 ceph-mon[44206]: pgmap v1938: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 26750/280266 objects degraded (9.545%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:43:07.273 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:43:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:06 smithi159 ceph-mon[45336]: pgmap v1938: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 26750/280266 objects degraded (9.545%); 8.0 MiB/s, 128 objects/s recovering 2024-04-15T18:43:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:43:07.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:07 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:07.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:43:07.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:07 smithi082 ceph-mon[44206]: osdmap e751: 12 total, 11 up, 11 in 2024-04-15T18:43:07.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:07 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 26750/280266 objects degraded (9.545%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:08.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:07 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:08.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:43:08.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:07 smithi049 ceph-mon[40087]: osdmap e751: 12 total, 11 up, 11 in 2024-04-15T18:43:08.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:07 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 26750/280266 objects degraded (9.545%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:08.258 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:43:08.258 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:43:08.258 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:43:08.258 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:43:08.259 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:43:08.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:07 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:08.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1834621335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:43:08.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:07 smithi159 ceph-mon[45336]: osdmap e751: 12 total, 11 up, 11 in 2024-04-15T18:43:08.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:07 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 26750/280266 objects degraded (9.545%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:08.449 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:08.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:08.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:08.599 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:08.745 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:08.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:08.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:08.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:08 smithi049 ceph-mon[40087]: pgmap v1940: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 26077/276318 objects degraded (9.437%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:08.976 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:09.072 INFO:teuthology.orchestra.run.smithi159.stdout: marking collection for removal 2024-04-15T18:43:09.072 INFO:teuthology.orchestra.run.smithi159.stdout:setting '_remove' omap key 2024-04-15T18:43:09.088 INFO:teuthology.orchestra.run.smithi159.stdout:finish_remove_pgs 8.f_head removing 8.f 2024-04-15T18:43:09.205 INFO:teuthology.orchestra.run.smithi159.stdout:Remove successful 2024-04-15T18:43:09.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:08 smithi082 ceph-mon[44206]: pgmap v1940: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 26077/276318 objects degraded (9.437%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:09.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:08 smithi159 ceph-mon[45336]: pgmap v1940: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 604 op/s; 26077/276318 objects degraded (9.437%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:10.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:43:11.195 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:43:11.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:10 smithi049 ceph-mon[40087]: pgmap v1941: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 25880/272037 objects degraded (9.513%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:11.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:10 smithi082 ceph-mon[44206]: pgmap v1941: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 25880/272037 objects degraded (9.513%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:11.315 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:10 smithi159 ceph-mon[45336]: pgmap v1941: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 25880/272037 objects degraded (9.513%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:11.350 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell -n osd.11 --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-11 --journal-path /var/lib/ceph/osd/ceph-11/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.f.11 2024-04-15T18:43:11.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:43:11.553 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:43:11.597 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:43:11.638 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/osd.11/config 2024-04-15T18:43:11.648 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:43:11.687 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:43:11.734 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:43:11.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:43:11.815 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:43:11.856 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:43:11.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:43:11.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:43:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:43:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:43:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:43:11.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:43:11.945 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:43:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/866124273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:12.185 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/866124273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:12.185 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:12.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/866124273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:12.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:12.984 INFO:teuthology.orchestra.run.smithi159.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: pgmap v1942: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 25701/270204 objects degraded (9.512%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 25701/270204 objects degraded (9.512%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: pgmap v1942: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 25701/270204 objects degraded (9.512%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.249 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.250 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.250 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.250 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.250 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.250 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 25701/270204 objects degraded (9.512%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:13.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: pgmap v1942: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 25701/270204 objects degraded (9.512%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:43:13.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:13.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 25701/270204 objects degraded (9.512%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:13.363 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:43:13.363 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:43:13.568 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:13.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:13.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:13.717 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:13.789 INFO:teuthology.orchestra.run.smithi159.stdout:get_pg_num_history pg_num_history pg_num_history(e733 pg_nums {7={621=16},8={709=16}} deleted_pools 705,7) 2024-04-15T18:43:13.789 INFO:teuthology.orchestra.run.smithi159.stdout:Importing pgid 8.f 2024-04-15T18:43:13.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0007692:::benchmark_data_smithi159_162467_object9074:head# 2024-04-15T18:43:13.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f001dccb:::benchmark_data_smithi159_162467_object15227:head# 2024-04-15T18:43:13.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0027798:::benchmark_data_smithi159_162467_object35147:head# 2024-04-15T18:43:13.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0027d97:::benchmark_data_smithi159_162467_object10916:head# 2024-04-15T18:43:13.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f002cb85:::benchmark_data_smithi159_162467_object14155:head# 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0042912:::benchmark_data_smithi159_162467_object27820:head# 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0047f8b:::benchmark_data_smithi159_162467_object45337:head# 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0057d3e:::benchmark_data_smithi159_162467_object70432:head# 2024-04-15T18:43:13.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f005c5a8:::benchmark_data_smithi159_162467_object43033:head# 2024-04-15T18:43:13.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0087387:::benchmark_data_smithi159_162467_object57032:head# 2024-04-15T18:43:13.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00ae780:::benchmark_data_smithi159_162467_object11625:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00b31a4:::benchmark_data_smithi159_162467_object57881:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00d7f15:::benchmark_data_smithi159_162467_object67870:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00d9e8c:::benchmark_data_smithi159_162467_object54960:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00dec60:::benchmark_data_smithi159_162467_object16009:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00e4858:::benchmark_data_smithi159_162467_object20615:head# 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00e5e4b:::benchmark_data_smithi159_162467_object37445:head# 2024-04-15T18:43:13.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f00f66c0:::benchmark_data_smithi159_162467_object51718:head# 2024-04-15T18:43:13.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.818 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01000e3:::benchmark_data_smithi159_162467_object24788:head# 2024-04-15T18:43:13.818 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01055f4:::benchmark_data_smithi159_162467_object69981:head# 2024-04-15T18:43:13.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f010848b:::benchmark_data_smithi159_162467_object853:head# 2024-04-15T18:43:13.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01091af:::benchmark_data_smithi159_162467_object18491:head# 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f011c6f5:::benchmark_data_smithi159_162467_object35835:head# 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f013beb8:::benchmark_data_smithi159_162467_object13473:head# 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f015ee08:::benchmark_data_smithi159_162467_object21855:head# 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01942e9:::benchmark_data_smithi159_162467_object64312:head# 2024-04-15T18:43:13.821 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.821 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0196995:::benchmark_data_smithi159_162467_object17922:head# 2024-04-15T18:43:13.821 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01b3fb6:::benchmark_data_smithi159_162467_object45217:head# 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01c8eb3:::benchmark_data_smithi159_162467_object31976:head# 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01d79ca:::benchmark_data_smithi159_162467_object51773:head# 2024-04-15T18:43:13.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.826 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f01f5826:::benchmark_data_smithi159_162467_object18169:head# 2024-04-15T18:43:13.826 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.827 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02055f5:::benchmark_data_smithi159_162467_object48741:head# 2024-04-15T18:43:13.827 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.827 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f022c6ad:::benchmark_data_smithi159_162467_object23553:head# 2024-04-15T18:43:13.827 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.827 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f022d587:::benchmark_data_smithi159_162467_object20678:head# 2024-04-15T18:43:13.828 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.828 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0232719:::benchmark_data_smithi159_162467_object68123:head# 2024-04-15T18:43:13.828 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.828 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0244418:::benchmark_data_smithi159_162467_object7959:head# 2024-04-15T18:43:13.828 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f025d4b3:::benchmark_data_smithi159_162467_object49961:head# 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0270319:::benchmark_data_smithi159_162467_object21445:head# 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f028cd10:::benchmark_data_smithi159_162467_object70399:head# 2024-04-15T18:43:13.832 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02a1aa8:::benchmark_data_smithi159_162467_object6800:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02b7a1f:::benchmark_data_smithi159_162467_object5104:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02c2362:::benchmark_data_smithi159_162467_object16055:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02c2614:::benchmark_data_smithi159_162467_object10875:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02c4e85:::benchmark_data_smithi159_162467_object42070:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02d1ebb:::benchmark_data_smithi159_162467_object63181:head# 2024-04-15T18:43:13.833 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02d491d:::benchmark_data_smithi159_162467_object63617:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02d9720:::benchmark_data_smithi159_162467_object1568:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02d9cd6:::benchmark_data_smithi159_162467_object49020:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f02f4199:::benchmark_data_smithi159_162467_object73767:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f030d14a:::benchmark_data_smithi159_162467_object23976:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0323cc5:::benchmark_data_smithi159_162467_object25903:head# 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.834 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0330516:::benchmark_data_smithi159_162467_object4906:head# 2024-04-15T18:43:13.835 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0330e28:::benchmark_data_smithi159_162467_object19783:head# 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f033eb3d:::benchmark_data_smithi159_162467_object23916:head# 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f034406b:::benchmark_data_smithi159_162467_object33293:head# 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f036e1da:::benchmark_data_smithi159_162467_object40130:head# 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.840 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f037b763:::benchmark_data_smithi159_162467_object46990:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f037eda8:::benchmark_data_smithi159_162467_object73925:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0384cce:::benchmark_data_smithi159_162467_object12622:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03c2c3a:::benchmark_data_smithi159_162467_object74647:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03c4fcf:::benchmark_data_smithi159_162467_object43536:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03c59b6:::benchmark_data_smithi159_162467_object30833:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03d2284:::benchmark_data_smithi159_162467_object33618:head# 2024-04-15T18:43:13.841 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03d4b9a:::benchmark_data_smithi159_162467_object42882:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03ea7f3:::benchmark_data_smithi159_162467_object19894:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f03ee77f:::benchmark_data_smithi159_162467_object66556:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0424b88:::benchmark_data_smithi159_162467_object56655:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0429e01:::benchmark_data_smithi159_162467_object2264:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0430d2d:::benchmark_data_smithi159_162467_object65364:head# 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.842 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04662de:::benchmark_data_smithi159_162467_object25988:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04692f4:::benchmark_data_smithi159_162467_object46683:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04698e7:::benchmark_data_smithi159_162467_object34509:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04759df:::benchmark_data_smithi159_162467_object56521:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0484001:::benchmark_data_smithi159_162467_object28912:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0485e6d:::benchmark_data_smithi159_162467_object16489:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04918a1:::benchmark_data_smithi159_162467_object29182:head# 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.843 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04982a5:::benchmark_data_smithi159_162467_object19540:head# 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f049b6ed:::benchmark_data_smithi159_162467_object70866:head# 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04a15d9:::benchmark_data_smithi159_162467_object20493:head# 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04a5e7e:::benchmark_data_smithi159_162467_object42137:head# 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04b64ef:::benchmark_data_smithi159_162467_object58783:head# 2024-04-15T18:43:13.844 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.847 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04bdd72:::benchmark_data_smithi159_162467_object15721:head# 2024-04-15T18:43:13.847 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.847 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04d15a5:::benchmark_data_smithi159_162467_object29707:head# 2024-04-15T18:43:13.847 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.847 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04d193c:::benchmark_data_smithi159_162467_object31030:head# 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04d6cdf:::benchmark_data_smithi159_162467_object20441:head# 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04db0ec:::benchmark_data_smithi159_162467_object17363:head# 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.848 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f04f4b26:::benchmark_data_smithi159_162467_object43800:head# 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05119a4:::benchmark_data_smithi159_162467_object69011:head# 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f051260d:::benchmark_data_smithi159_162467_object56232:head# 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.849 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0538203:::benchmark_data_smithi159_162467_object33302:head# 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0540d4a:::benchmark_data_smithi159_162467_object38271:head# 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0549831:::benchmark_data_smithi159_162467_object61874:head# 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f055ed10:::benchmark_data_smithi159_162467_object45599:head# 2024-04-15T18:43:13.850 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f056de60:::benchmark_data_smithi159_162467_object64044:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0575bbd:::benchmark_data_smithi159_162467_object5290:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f058f8e0:::benchmark_data_smithi159_162467_object10530:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05ab18e:::benchmark_data_smithi159_162467_object45966:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05aea4e:::benchmark_data_smithi159_162467_object37871:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05c4116:::benchmark_data_smithi159_162467_object49350:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05d3782:::benchmark_data_smithi159_162467_object62447:head# 2024-04-15T18:43:13.851 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f05d46f0:::benchmark_data_smithi159_162467_object62442:head# 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f060da17:::benchmark_data_smithi159_162467_object9817:head# 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f060ed1c:::benchmark_data_smithi159_162467_object15206:head# 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0610acb:::benchmark_data_smithi159_162467_object51135:head# 2024-04-15T18:43:13.852 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.853 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06120f4:::benchmark_data_smithi159_162467_object10314:head# 2024-04-15T18:43:13.853 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.853 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0629262:::benchmark_data_smithi159_162467_object68578:head# 2024-04-15T18:43:13.853 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0633c3e:::benchmark_data_smithi159_162467_object50601:head# 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f063e2cf:::benchmark_data_smithi159_162467_object16241:head# 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f063f01e:::benchmark_data_smithi159_162467_object44282:head# 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06467a1:::benchmark_data_smithi159_162467_object61524:head# 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06686f6:::benchmark_data_smithi159_162467_object42891:head# 2024-04-15T18:43:13.855 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0674e00:::benchmark_data_smithi159_162467_object56812:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06821da:::benchmark_data_smithi159_162467_object18653:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0696ed4:::benchmark_data_smithi159_162467_object40456:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f069f940:::benchmark_data_smithi159_162467_object34827:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06a28ec:::benchmark_data_smithi159_162467_object6757:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06e2e98:::benchmark_data_smithi159_162467_object50688:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06fe90b:::benchmark_data_smithi159_162467_object27108:head# 2024-04-15T18:43:13.856 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f06fee52:::benchmark_data_smithi159_162467_object68996:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0715e97:::benchmark_data_smithi159_162467_object26372:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f073b0ed:::benchmark_data_smithi159_162467_object5023:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f073fcd8:::benchmark_data_smithi159_162467_object68195:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0746f0c:::benchmark_data_smithi159_162467_object6573:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07560e0:::benchmark_data_smithi159_162467_object40632:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f078c4f0:::benchmark_data_smithi159_162467_object22718:head# 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.857 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07a72c4:::benchmark_data_smithi159_162467_object30361:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07b4937:::benchmark_data_smithi159_162467_object69517:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07b99e0:::benchmark_data_smithi159_162467_object47996:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07d82f1:::benchmark_data_smithi159_162467_object20604:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f07e4a58:::benchmark_data_smithi159_162467_object57737:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0803b3c:::benchmark_data_smithi159_162467_object44411:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f080fe8e:::benchmark_data_smithi159_162467_object61391:head# 2024-04-15T18:43:13.858 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0819869:::benchmark_data_smithi159_162467_object52555:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f082df34:::benchmark_data_smithi159_162467_object3861:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f083ec22:::benchmark_data_smithi159_162467_object60184:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0844369:::benchmark_data_smithi159_162467_object42937:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0851cde:::benchmark_data_smithi159_162467_object396:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0859815:::benchmark_data_smithi159_162467_object49850:head# 2024-04-15T18:43:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f086090a:::benchmark_data_smithi159_162467_object25603:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0873728:::benchmark_data_smithi159_162467_object21563:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f087e281:::benchmark_data_smithi159_162467_object58270:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0894a8c:::benchmark_data_smithi159_162467_object51614:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f089d7d1:::benchmark_data_smithi159_162467_object69211:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08a21e8:::benchmark_data_smithi159_162467_object14068:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08bb5a5:::benchmark_data_smithi159_162467_object28522:head# 2024-04-15T18:43:13.863 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08d26c7:::benchmark_data_smithi159_162467_object5949:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08d64e7:::benchmark_data_smithi159_162467_object63061:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08dc0b4:::benchmark_data_smithi159_162467_object9170:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08f5481:::benchmark_data_smithi159_162467_object57099:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08fb8a2:::benchmark_data_smithi159_162467_object26269:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08fbedc:::benchmark_data_smithi159_162467_object8190:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08ffcc4:::benchmark_data_smithi159_162467_object69674:head# 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.864 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f08ffe12:::benchmark_data_smithi159_162467_object6292:head# 2024-04-15T18:43:13.865 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.865 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09117d3:::benchmark_data_smithi159_162467_object3725:head# 2024-04-15T18:43:13.865 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0912013:::benchmark_data_smithi159_162467_object56225:head# 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0917a77:::benchmark_data_smithi159_162467_object23531:head# 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f092f4a7:::benchmark_data_smithi159_162467_object47403:head# 2024-04-15T18:43:13.871 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09378e3:::benchmark_data_smithi159_162467_object7137:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09568ba:::benchmark_data_smithi159_162467_object31709:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0957137:::benchmark_data_smithi159_162467_object12612:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f095a28f:::benchmark_data_smithi159_162467_object6610:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f095f60a:::benchmark_data_smithi159_162467_object25704:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0960333:::benchmark_data_smithi159_162467_object1812:head# 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.872 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09697c9:::benchmark_data_smithi159_162467_object47853:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f097055e:::benchmark_data_smithi159_162467_object8308:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0975400:::benchmark_data_smithi159_162467_object50568:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f099d937:::benchmark_data_smithi159_162467_object4289:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f099fb87:::benchmark_data_smithi159_162467_object8450:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09a30d6:::benchmark_data_smithi159_162467_object62273:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09a7f0d:::benchmark_data_smithi159_162467_object63179:head# 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.873 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09acea7:::benchmark_data_smithi159_162467_object8436:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09c2148:::benchmark_data_smithi159_162467_object10178:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09dcb20:::benchmark_data_smithi159_162467_object61719:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09e9f6e:::benchmark_data_smithi159_162467_object59835:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f09ed140:::benchmark_data_smithi159_162467_object31652:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a0955e:::benchmark_data_smithi159_162467_object66413:head# 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.874 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a33436:::benchmark_data_smithi159_162467_object24356:head# 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a3834a:::benchmark_data_smithi159_162467_object41176:head# 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a41ea4:::benchmark_data_smithi159_162467_object33315:head# 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a46563:::benchmark_data_smithi159_162467_object33491:head# 2024-04-15T18:43:13.877 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a605c1:::benchmark_data_smithi159_162467_object38303:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a7a3f6:::benchmark_data_smithi159_162467_object41746:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a8fe82:::benchmark_data_smithi159_162467_object20469:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a927ac:::benchmark_data_smithi159_162467_object12618:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0a9b0db:::benchmark_data_smithi159_162467_object650:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0aa3735:::benchmark_data_smithi159_162467_object12300:head# 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0aaa1d7:::benchmark_data_smithi159_162467_object22963:head# 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0aadd45:::benchmark_data_smithi159_162467_object4793:head# 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ab29b7:::benchmark_data_smithi159_162467_object46267:head# 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ad2b13:::benchmark_data_smithi159_162467_object24608:head# 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ae946e:::benchmark_data_smithi159_162467_object1878:head# 2024-04-15T18:43:13.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0af6217:::benchmark_data_smithi159_162467_object68287:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0af6728:::benchmark_data_smithi159_162467_object35193:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b02023:::benchmark_data_smithi159_162467_object3144:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b16573:::benchmark_data_smithi159_162467_object36888:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b4dcc1:::benchmark_data_smithi159_162467_object42028:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b4de97:::benchmark_data_smithi159_162467_object28278:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b512d1:::benchmark_data_smithi159_162467_object19847:head# 2024-04-15T18:43:13.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b602f8:::benchmark_data_smithi159_162467_object30633:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b6e24c:::benchmark_data_smithi159_162467_object56561:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b7124c:::benchmark_data_smithi159_162467_object30799:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b7ea75:::benchmark_data_smithi159_162467_object22885:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b83154:::benchmark_data_smithi159_162467_object28287:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b9359e:::benchmark_data_smithi159_162467_object17896:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0b97ebe:::benchmark_data_smithi159_162467_object52026:head# 2024-04-15T18:43:13.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0bb368d:::benchmark_data_smithi159_162467_object39406:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0bb3ba8:::benchmark_data_smithi159_162467_object37051:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0bb417b:::benchmark_data_smithi159_162467_object70689:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0be2020:::benchmark_data_smithi159_162467_object36874:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0be213f:::benchmark_data_smithi159_162467_object44933:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0be44fc:::benchmark_data_smithi159_162467_object55777:head# 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0be5c5c:::benchmark_data_smithi159_162467_object55521:head# 2024-04-15T18:43:13.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0be9736:::benchmark_data_smithi159_162467_object13604:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c05db6:::benchmark_data_smithi159_162467_object47619:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c08d57:::benchmark_data_smithi159_162467_object51478:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c11324:::benchmark_data_smithi159_162467_object59206:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c1635f:::benchmark_data_smithi159_162467_object64199:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c2d19a:::benchmark_data_smithi159_162467_object7652:head# 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c3d755:::benchmark_data_smithi159_162467_object44766:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c41107:::benchmark_data_smithi159_162467_object69349:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c699cd:::benchmark_data_smithi159_162467_object38140:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0c772fd:::benchmark_data_smithi159_162467_object47197:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ca55a9:::benchmark_data_smithi159_162467_object48742:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cb1ff3:::benchmark_data_smithi159_162467_object42273:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cb64ca:::benchmark_data_smithi159_162467_object71274:head# 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cba3d9:::benchmark_data_smithi159_162467_object41335:head# 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cbe7f0:::benchmark_data_smithi159_162467_object3775:head# 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cc502b:::benchmark_data_smithi159_162467_object24913:head# 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ce282c:::benchmark_data_smithi159_162467_object51886:head# 2024-04-15T18:43:13.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cf039e:::benchmark_data_smithi159_162467_object34809:head# 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cf1286:::benchmark_data_smithi159_162467_object74522:head# 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0cf34d9:::benchmark_data_smithi159_162467_object11872:head# 2024-04-15T18:43:13.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d0280c:::benchmark_data_smithi159_162467_object58632:head# 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d07f6b:::benchmark_data_smithi159_162467_object47211:head# 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d1b934:::benchmark_data_smithi159_162467_object18605:head# 2024-04-15T18:43:13.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d1ce72:::benchmark_data_smithi159_162467_object51624:head# 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d27b81:::benchmark_data_smithi159_162467_object62357:head# 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d450da:::benchmark_data_smithi159_162467_object24944:head# 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d5c408:::benchmark_data_smithi159_162467_object8038:head# 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d6493e:::benchmark_data_smithi159_162467_object11477:head# 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d6eb93:::benchmark_data_smithi159_162467_object36239:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d821f7:::benchmark_data_smithi159_162467_object30669:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d8b557:::benchmark_data_smithi159_162467_object10637:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d9a15e:::benchmark_data_smithi159_162467_object50096:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0d9ea58:::benchmark_data_smithi159_162467_object5801:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0da09c5:::benchmark_data_smithi159_162467_object62651:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0dafe78:::benchmark_data_smithi159_162467_object70993:head# 2024-04-15T18:43:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0dbb8a5:::benchmark_data_smithi159_162467_object54143:head# 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0dc3b1f:::benchmark_data_smithi159_162467_object46304:head# 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0dd9562:::benchmark_data_smithi159_162467_object41088:head# 2024-04-15T18:43:13.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ddb074:::benchmark_data_smithi159_162467_object70053:head# 2024-04-15T18:43:13.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0de1091:::benchmark_data_smithi159_162467_object70342:head# 2024-04-15T18:43:13.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0de4623:::benchmark_data_smithi159_162467_object39546:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e10f3a:::benchmark_data_smithi159_162467_object1271:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e1aee4:::benchmark_data_smithi159_162467_object43235:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e26e25:::benchmark_data_smithi159_162467_object40828:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e4b7ca:::benchmark_data_smithi159_162467_object14758:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e63bd9:::benchmark_data_smithi159_162467_object27424:head# 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e86aa9:::benchmark_data_smithi159_162467_object11515:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e89f04:::benchmark_data_smithi159_162467_object60477:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e8dd80:::benchmark_data_smithi159_162467_object46294:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e976c7:::benchmark_data_smithi159_162467_object5004:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0e9addb:::benchmark_data_smithi159_162467_object64498:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ed4d78:::benchmark_data_smithi159_162467_object37471:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ef03e7:::benchmark_data_smithi159_162467_object71324:head# 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ef8bc0:::benchmark_data_smithi159_162467_object10618:head# 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0eff920:::benchmark_data_smithi159_162467_object20327:head# 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f08de8:::benchmark_data_smithi159_162467_object15418:head# 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f16bb7:::benchmark_data_smithi159_162467_object29187:head# 2024-04-15T18:43:13.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f2d535:::benchmark_data_smithi159_162467_object16510:head# 2024-04-15T18:43:13.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f4d71f:::benchmark_data_smithi159_162467_object56967:head# 2024-04-15T18:43:13.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f5dd19:::benchmark_data_smithi159_162467_object8101:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f5e1e8:::benchmark_data_smithi159_162467_object16462:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f63751:::benchmark_data_smithi159_162467_object33421:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f656ea:::benchmark_data_smithi159_162467_object52704:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f77848:::benchmark_data_smithi159_162467_object19444:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f810e4:::benchmark_data_smithi159_162467_object2649:head# 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0f93ca7:::benchmark_data_smithi159_162467_object66653:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fa27a2:::benchmark_data_smithi159_162467_object6325:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fb89a1:::benchmark_data_smithi159_162467_object63969:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fc5832:::benchmark_data_smithi159_162467_object46832:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fdc895:::benchmark_data_smithi159_162467_object28191:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fdf4dd:::benchmark_data_smithi159_162467_object5423:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0fed877:::benchmark_data_smithi159_162467_object2224:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f0ffb23b:::benchmark_data_smithi159_162467_object11896:head# 2024-04-15T18:43:13.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f100a8b4:::benchmark_data_smithi159_162467_object8565:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f100f22b:::benchmark_data_smithi159_162467_object50386:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1053937:::benchmark_data_smithi159_162467_object43369:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f105c061:::benchmark_data_smithi159_162467_object61257:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f105d816:::benchmark_data_smithi159_162467_object55489:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f105ffd5:::benchmark_data_smithi159_162467_object72255:head# 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1060b09:::benchmark_data_smithi159_162467_object59447:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f106a26d:::benchmark_data_smithi159_162467_object47526:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f106c204:::benchmark_data_smithi159_162467_object35833:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f108343c:::benchmark_data_smithi159_162467_object72572:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f108506a:::benchmark_data_smithi159_162467_object48436:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f108759d:::benchmark_data_smithi159_162467_object23476:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1098cd9:::benchmark_data_smithi159_162467_object57197:head# 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10b2cc9:::benchmark_data_smithi159_162467_object14322:head# 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10b4014:::benchmark_data_smithi159_162467_object68399:head# 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10bc48b:::benchmark_data_smithi159_162467_object57338:head# 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10bf21e:::benchmark_data_smithi159_162467_object4357:head# 2024-04-15T18:43:13.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10c97b6:::benchmark_data_smithi159_162467_object45541:head# 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10d7f6a:::benchmark_data_smithi159_162467_object69032:head# 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10de642:::benchmark_data_smithi159_162467_object54579:head# 2024-04-15T18:43:13.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f10e37e4:::benchmark_data_smithi159_162467_object27892:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f110f5c5:::benchmark_data_smithi159_162467_object9129:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1115c15:::benchmark_data_smithi159_162467_object67139:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1124338:::benchmark_data_smithi159_162467_object44826:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f115c9d0:::benchmark_data_smithi159_162467_object70962:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f115eb6e:::benchmark_data_smithi159_162467_object7900:head# 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f115edb5:::benchmark_data_smithi159_162467_object20922:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f115f837:::benchmark_data_smithi159_162467_object27768:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1173a99:::benchmark_data_smithi159_162467_object24816:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f119c5d9:::benchmark_data_smithi159_162467_object15178:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f119da25:::benchmark_data_smithi159_162467_object61485:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f11e2040:::benchmark_data_smithi159_162467_object17721:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f11eb54a:::benchmark_data_smithi159_162467_object25111:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1207b3c:::benchmark_data_smithi159_162467_object62906:head# 2024-04-15T18:43:13.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1210fa6:::benchmark_data_smithi159_162467_object8124:head# 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12141b3:::benchmark_data_smithi159_162467_object8895:head# 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f121db97:::benchmark_data_smithi159_162467_object24562:head# 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f124ca36:::benchmark_data_smithi159_162467_object35727:head# 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1251329:::benchmark_data_smithi159_162467_object51590:head# 2024-04-15T18:43:13.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1269c9c:::benchmark_data_smithi159_162467_object34087:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f128aae3:::benchmark_data_smithi159_162467_object54748:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1291883:::benchmark_data_smithi159_162467_object74134:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12994d9:::benchmark_data_smithi159_162467_object28110:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12a6d91:::benchmark_data_smithi159_162467_object28676:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12ab793:::benchmark_data_smithi159_162467_object65229:head# 2024-04-15T18:43:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12c228f:::benchmark_data_smithi159_162467_object63395:head# 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12d5860:::benchmark_data_smithi159_162467_object1784:head# 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f12ed092:::benchmark_data_smithi159_162467_object11246:head# 2024-04-15T18:43:13.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f132f249:::benchmark_data_smithi159_162467_object9975:head# 2024-04-15T18:43:13.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1333316:::benchmark_data_smithi159_162467_object36712:head# 2024-04-15T18:43:13.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1349c21:::benchmark_data_smithi159_162467_object45899:head# 2024-04-15T18:43:13.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f135e75b:::benchmark_data_smithi159_162467_object54876:head# 2024-04-15T18:43:13.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f136e305:::benchmark_data_smithi159_162467_object65191:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f136fe7c:::benchmark_data_smithi159_162467_object2512:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1377789:::benchmark_data_smithi159_162467_object20762:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f139080b:::benchmark_data_smithi159_162467_object14793:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1399cc9:::benchmark_data_smithi159_162467_object17739:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13beb02:::benchmark_data_smithi159_162467_object67967:head# 2024-04-15T18:43:13.926 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.927 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13c8898:::benchmark_data_smithi159_162467_object55325:head# 2024-04-15T18:43:13.927 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.927 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13d68c1:::benchmark_data_smithi159_162467_object30691:head# 2024-04-15T18:43:13.927 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13d8f86:::benchmark_data_smithi159_162467_object69565:head# 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13e1f4a:::benchmark_data_smithi159_162467_object33264:head# 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f13f1c3f:::benchmark_data_smithi159_162467_object44281:head# 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14017f1:::benchmark_data_smithi159_162467_object35869:head# 2024-04-15T18:43:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1414f75:::benchmark_data_smithi159_162467_object33140:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f143d5c7:::benchmark_data_smithi159_162467_object24379:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1451190:::benchmark_data_smithi159_162467_object50123:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f145a4fa:::benchmark_data_smithi159_162467_object18118:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f145a78b:::benchmark_data_smithi159_162467_object46894:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f146520f:::benchmark_data_smithi159_162467_object64959:head# 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1472ec9:::benchmark_data_smithi159_162467_object54279:head# 2024-04-15T18:43:13.931 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.931 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14759c6:::benchmark_data_smithi159_162467_object2189:head# 2024-04-15T18:43:13.931 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.931 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14b2494:::benchmark_data_smithi159_162467_object19677:head# 2024-04-15T18:43:13.931 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.933 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14c09c6:::benchmark_data_smithi159_162467_object70235:head# 2024-04-15T18:43:13.933 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.933 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14c130d:::benchmark_data_smithi159_162467_object53778:head# 2024-04-15T18:43:13.933 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14c7d4a:::benchmark_data_smithi159_162467_object48989:head# 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14c7e78:::benchmark_data_smithi159_162467_object47700:head# 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14d21fa:::benchmark_data_smithi159_162467_object56746:head# 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14e4292:::benchmark_data_smithi159_162467_object18240:head# 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14ef2be:::benchmark_data_smithi159_162467_object2042:head# 2024-04-15T18:43:13.934 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14f5efc:::benchmark_data_smithi159_162467_object53239:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f14ff2cb:::benchmark_data_smithi159_162467_object9788:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f150f8d7:::benchmark_data_smithi159_162467_object264:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1512361:::benchmark_data_smithi159_162467_object3290:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f151da88:::benchmark_data_smithi159_162467_object3251:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1527f92:::benchmark_data_smithi159_162467_object61819:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1533bfc:::benchmark_data_smithi159_162467_object41066:head# 2024-04-15T18:43:13.935 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f154c95f:::benchmark_data_smithi159_162467_object49259:head# 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1566d01:::benchmark_data_smithi159_162467_object21927:head# 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1569571:::benchmark_data_smithi159_162467_object28508:head# 2024-04-15T18:43:13.937 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f156d61d:::benchmark_data_smithi159_162467_object14172:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1571f12:::benchmark_data_smithi159_162467_object49902:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1571f96:::benchmark_data_smithi159_162467_object66947:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f158e755:::benchmark_data_smithi159_162467_object6589:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1596e4e:::benchmark_data_smithi159_162467_object22188:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15986a5:::benchmark_data_smithi159_162467_object55535:head# 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.938 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15a5190:::benchmark_data_smithi159_162467_object72132:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15a7f6f:::benchmark_data_smithi159_162467_object40563:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15b4747:::benchmark_data_smithi159_162467_object8430:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15b73f0:::benchmark_data_smithi159_162467_object11976:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15cc247:::benchmark_data_smithi159_162467_object38568:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15d0252:::benchmark_data_smithi159_162467_object12446:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15d609d:::benchmark_data_smithi159_162467_object58550:head# 2024-04-15T18:43:13.939 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15e550d:::benchmark_data_smithi159_162467_object52193:head# 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f15f739d:::benchmark_data_smithi159_162467_object6143:head# 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f16159a4:::benchmark_data_smithi159_162467_object65593:head# 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.941 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f161d733:::benchmark_data_smithi159_162467_object37301:head# 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1620b73:::benchmark_data_smithi159_162467_object37973:head# 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1640b58:::benchmark_data_smithi159_162467_object70396:head# 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1641704:::benchmark_data_smithi159_162467_object64503:head# 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f164243f:::benchmark_data_smithi159_162467_object46391:head# 2024-04-15T18:43:13.942 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f165199c:::benchmark_data_smithi159_162467_object47210:head# 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1654a28:::benchmark_data_smithi159_162467_object1096:head# 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f166baf5:::benchmark_data_smithi159_162467_object70337:head# 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f167649a:::benchmark_data_smithi159_162467_object73318:head# 2024-04-15T18:43:13.943 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.944 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1677287:::benchmark_data_smithi159_162467_object42078:head# 2024-04-15T18:43:13.944 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.944 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f16b271b:::benchmark_data_smithi159_162467_object10011:head# 2024-04-15T18:43:13.944 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f16c8eee:::benchmark_data_smithi159_162467_object32031:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f16cfbbc:::benchmark_data_smithi159_162467_object46111:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f16cfc83:::benchmark_data_smithi159_162467_object17374:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1708651:::benchmark_data_smithi159_162467_object27883:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1710361:::benchmark_data_smithi159_162467_object21415:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1714f9a:::benchmark_data_smithi159_162467_object16330:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17209d2:::benchmark_data_smithi159_162467_object12114:head# 2024-04-15T18:43:13.945 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f172d266:::benchmark_data_smithi159_162467_object9790:head# 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f173aa7b:::benchmark_data_smithi159_162467_object28743:head# 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f174e253:::benchmark_data_smithi159_162467_object19138:head# 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f176a0ad:::benchmark_data_smithi159_162467_object53104:head# 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f176b1f2:::benchmark_data_smithi159_162467_object21884:head# 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.947 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1775b95:::benchmark_data_smithi159_162467_object11819:head# 2024-04-15T18:43:13.948 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.948 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1785e28:::benchmark_data_smithi159_162467_object46110:head# 2024-04-15T18:43:13.948 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.948 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17890a7:::benchmark_data_smithi159_162467_object70116:head# 2024-04-15T18:43:13.948 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.949 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f178b592:::benchmark_data_smithi159_162467_object18298:head# 2024-04-15T18:43:13.949 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.950 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f178bd3e:::benchmark_data_smithi159_162467_object55740:head# 2024-04-15T18:43:13.950 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.951 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17931b6:::benchmark_data_smithi159_162467_object5649:head# 2024-04-15T18:43:13.951 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.951 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17abb59:::benchmark_data_smithi159_162467_object62537:head# 2024-04-15T18:43:13.951 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17d9941:::benchmark_data_smithi159_162467_object36376:head# 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f17dfd48:::benchmark_data_smithi159_162467_object45815:head# 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f180b07e:::benchmark_data_smithi159_162467_object25122:head# 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.952 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f180ffe4:::benchmark_data_smithi159_162467_object14080:head# 2024-04-15T18:43:13.953 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.954 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f182eb33:::benchmark_data_smithi159_162467_object4893:head# 2024-04-15T18:43:13.955 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.955 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f182fa28:::benchmark_data_smithi159_162467_object9162:head# 2024-04-15T18:43:13.955 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f183bae6:::benchmark_data_smithi159_162467_object21122:head# 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1854a55:::benchmark_data_smithi159_162467_object72096:head# 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1867e8e:::benchmark_data_smithi159_162467_object3785:head# 2024-04-15T18:43:13.956 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f186f1fe:::benchmark_data_smithi159_162467_object5732:head# 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1871587:::benchmark_data_smithi159_162467_object7981:head# 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f18a1a15:::benchmark_data_smithi159_162467_object19038:head# 2024-04-15T18:43:13.957 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.960 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f18a8c31:::benchmark_data_smithi159_162467_object25906:head# 2024-04-15T18:43:13.960 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f18df34e:::benchmark_data_smithi159_162467_object22437:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f18e3d8a:::benchmark_data_smithi159_162467_object56764:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f190155b:::benchmark_data_smithi159_162467_object5985:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1910822:::benchmark_data_smithi159_162467_object18757:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f191c678:::benchmark_data_smithi159_162467_object57944:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f192ebb4:::benchmark_data_smithi159_162467_object69824:head# 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.961 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19318e4:::benchmark_data_smithi159_162467_object38013:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1944090:::benchmark_data_smithi159_162467_object54907:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f194dbb9:::benchmark_data_smithi159_162467_object54034:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f194eba6:::benchmark_data_smithi159_162467_object54236:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f195d649:::benchmark_data_smithi159_162467_object53183:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1960d29:::benchmark_data_smithi159_162467_object6120:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f196545e:::benchmark_data_smithi159_162467_object58448:head# 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.962 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f199bbab:::benchmark_data_smithi159_162467_object43722:head# 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19a3142:::benchmark_data_smithi159_162467_object32655:head# 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19a9d5b:::benchmark_data_smithi159_162467_object8764:head# 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19aafbd:::benchmark_data_smithi159_162467_object65663:head# 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19ab83c:::benchmark_data_smithi159_162467_object26607:head# 2024-04-15T18:43:13.963 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.965 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19b82a8:::benchmark_data_smithi159_162467_object49612:head# 2024-04-15T18:43:13.965 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.965 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19bf61a:::benchmark_data_smithi159_162467_object71016:head# 2024-04-15T18:43:13.965 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.965 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19c9cb7:::benchmark_data_smithi159_162467_object40775:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19d7d6a:::benchmark_data_smithi159_162467_object31874:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19f229d:::benchmark_data_smithi159_162467_object71307:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f19ff6e9:::benchmark_data_smithi159_162467_object33374:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a00756:::benchmark_data_smithi159_162467_object72454:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a05012:::benchmark_data_smithi159_162467_object57411:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a14d95:::benchmark_data_smithi159_162467_object16790:head# 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.966 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a23f33:::benchmark_data_smithi159_162467_object66833:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a29ff4:::benchmark_data_smithi159_162467_object41698:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a66af3:::benchmark_data_smithi159_162467_object15969:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a68cc1:::benchmark_data_smithi159_162467_object10863:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a8afe3:::benchmark_data_smithi159_162467_object53219:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a8dd1d:::benchmark_data_smithi159_162467_object19257:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a8fadc:::benchmark_data_smithi159_162467_object42686:head# 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1a98f63:::benchmark_data_smithi159_162467_object71170:head# 2024-04-15T18:43:13.968 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.970 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ab2093:::benchmark_data_smithi159_162467_object29735:head# 2024-04-15T18:43:13.970 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.970 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ab5733:::benchmark_data_smithi159_162467_object72126:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ad1820:::benchmark_data_smithi159_162467_object48074:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ad2af9:::benchmark_data_smithi159_162467_object9730:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ad447a:::benchmark_data_smithi159_162467_object7124:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ae1c22:::benchmark_data_smithi159_162467_object43535:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1af17fd:::benchmark_data_smithi159_162467_object16278:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b0457e:::benchmark_data_smithi159_162467_object22696:head# 2024-04-15T18:43:13.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b123a4:::benchmark_data_smithi159_162467_object74801:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b21d40:::benchmark_data_smithi159_162467_object55629:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b4b3b1:::benchmark_data_smithi159_162467_object50677:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b76afb:::benchmark_data_smithi159_162467_object28143:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b7731a:::benchmark_data_smithi159_162467_object18215:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b959c5:::benchmark_data_smithi159_162467_object50353:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1b9d97b:::benchmark_data_smithi159_162467_object58613:head# 2024-04-15T18:43:13.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bb075c:::benchmark_data_smithi159_162467_object49969:head# 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bc4728:::benchmark_data_smithi159_162467_object61038:head# 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bc67ca:::benchmark_data_smithi159_162467_object45118:head# 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bda84e:::benchmark_data_smithi159_162467_object44330:head# 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bdc6ee:::benchmark_data_smithi159_162467_object55556:head# 2024-04-15T18:43:13.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1bf1977:::benchmark_data_smithi159_162467_object44947:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c03fc0:::benchmark_data_smithi159_162467_object34215:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c05213:::benchmark_data_smithi159_162467_object8971:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c0ccfa:::benchmark_data_smithi159_162467_object23092:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c14649:::benchmark_data_smithi159_162467_object45184:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c15698:::benchmark_data_smithi159_162467_object16467:head# 2024-04-15T18:43:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c1609a:::benchmark_data_smithi159_162467_object52132:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c2e399:::benchmark_data_smithi159_162467_object48050:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c3bb9b:::benchmark_data_smithi159_162467_object40531:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c4e0a0:::benchmark_data_smithi159_162467_object34588:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c5d989:::benchmark_data_smithi159_162467_object49579:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c7049a:::benchmark_data_smithi159_162467_object68545:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c76e3b:::benchmark_data_smithi159_162467_object32582:head# 2024-04-15T18:43:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1c8aa0d:::benchmark_data_smithi159_162467_object10076:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ca490a:::benchmark_data_smithi159_162467_object12085:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cba416:::benchmark_data_smithi159_162467_object55991:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cc2a93:::benchmark_data_smithi159_162467_object42011:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cd0a96:::benchmark_data_smithi159_162467_object63146:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cd16c6:::benchmark_data_smithi159_162467_object66396:head# 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cddce6:::benchmark_data_smithi159_162467_object55942:head# 2024-04-15T18:43:13.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.979 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cf4fa0:::benchmark_data_smithi159_162467_object42726:head# 2024-04-15T18:43:13.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1cf78c9:::benchmark_data_smithi159_162467_object2007:head# 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d08ca4:::benchmark_data_smithi159_162467_object65543:head# 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d1040b:::benchmark_data_smithi159_162467_object19456:head# 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d2c0ab:::benchmark_data_smithi159_162467_object2331:head# 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d350f7:::benchmark_data_smithi159_162467_object4329:head# 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d3fb93:::benchmark_data_smithi159_162467_object72051:head# 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d43adc:::benchmark_data_smithi159_162467_object21572:head# 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d48608:::benchmark_data_smithi159_162467_object5850:head# 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d8e69b:::benchmark_data_smithi159_162467_object39597:head# 2024-04-15T18:43:13.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d97add:::benchmark_data_smithi159_162467_object39272:head# 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1d983b8:::benchmark_data_smithi159_162467_object3308:head# 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1da002d:::benchmark_data_smithi159_162467_object13965:head# 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1dcd5e9:::benchmark_data_smithi159_162467_object54015:head# 2024-04-15T18:43:13.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1dcdb6a:::benchmark_data_smithi159_162467_object1131:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1dd8c97:::benchmark_data_smithi159_162467_object66013:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e0846d:::benchmark_data_smithi159_162467_object21760:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e0e4c0:::benchmark_data_smithi159_162467_object41179:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e127d9:::benchmark_data_smithi159_162467_object26225:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e155db:::benchmark_data_smithi159_162467_object63135:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e16640:::benchmark_data_smithi159_162467_object62415:head# 2024-04-15T18:43:13.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e173cd:::benchmark_data_smithi159_162467_object52373:head# 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e1890b:::benchmark_data_smithi159_162467_object43438:head# 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e3706a:::benchmark_data_smithi159_162467_object72577:head# 2024-04-15T18:43:13.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e410f8:::benchmark_data_smithi159_162467_object8580:head# 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e6cab1:::benchmark_data_smithi159_162467_object62802:head# 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e722b4:::benchmark_data_smithi159_162467_object73600:head# 2024-04-15T18:43:13.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e7cb59:::benchmark_data_smithi159_162467_object65295:head# 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e7ded0:::benchmark_data_smithi159_162467_object16031:head# 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1e9b7ac:::benchmark_data_smithi159_162467_object64551:head# 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1eaf97e:::benchmark_data_smithi159_162467_object42573:head# 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ecfc07:::benchmark_data_smithi159_162467_object2861:head# 2024-04-15T18:43:13.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ed8406:::benchmark_data_smithi159_162467_object42022:head# 2024-04-15T18:43:13.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ee63a1:::benchmark_data_smithi159_162467_object29427:head# 2024-04-15T18:43:13.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ef5b72:::benchmark_data_smithi159_162467_object51008:head# 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f0dd26:::benchmark_data_smithi159_162467_object53907:head# 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f114ab:::benchmark_data_smithi159_162467_object34145:head# 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f1bb40:::benchmark_data_smithi159_162467_object67126:head# 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f1c55b:::benchmark_data_smithi159_162467_object41758:head# 2024-04-15T18:43:13.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f289aa:::benchmark_data_smithi159_162467_object17957:head# 2024-04-15T18:43:13.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f4689c:::benchmark_data_smithi159_162467_object29478:head# 2024-04-15T18:43:13.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f4cc9a:::benchmark_data_smithi159_162467_object34978:head# 2024-04-15T18:43:13.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f51cd7:::benchmark_data_smithi159_162467_object34471:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f7835e:::benchmark_data_smithi159_162467_object12747:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f9ab10:::benchmark_data_smithi159_162467_object54234:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1f9c3c3:::benchmark_data_smithi159_162467_object25651:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1fb2a9b:::benchmark_data_smithi159_162467_object15532:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1fc5bd2:::benchmark_data_smithi159_162467_object2067:head# 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1fd0e2d:::benchmark_data_smithi159_162467_object70907:head# 2024-04-15T18:43:13.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1fe5550:::benchmark_data_smithi159_162467_object55652:head# 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ff2ed5:::benchmark_data_smithi159_162467_object49647:head# 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ff30b8:::benchmark_data_smithi159_162467_object68936:head# 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f1ffd76f:::benchmark_data_smithi159_162467_object19491:head# 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2009938:::benchmark_data_smithi159_162467_object52891:head# 2024-04-15T18:43:13.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f200b2f6:::benchmark_data_smithi159_162467_object12705:head# 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2019e03:::benchmark_data_smithi159_162467_object15611:head# 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f201be6c:::benchmark_data_smithi159_162467_object45533:head# 2024-04-15T18:43:13.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f203c047:::benchmark_data_smithi159_162467_object29292:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f203fcc7:::benchmark_data_smithi159_162467_object30023:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2045c00:::benchmark_data_smithi159_162467_object42667:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2053803:::benchmark_data_smithi159_162467_object15387:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2055e20:::benchmark_data_smithi159_162467_object26723:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2070ef0:::benchmark_data_smithi159_162467_object35298:head# 2024-04-15T18:43:13.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f207f197:::benchmark_data_smithi159_162467_object37866:head# 2024-04-15T18:43:14.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2081425:::benchmark_data_smithi159_162467_object18992:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f208c250:::benchmark_data_smithi159_162467_object14027:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2091efa:::benchmark_data_smithi159_162467_object73419:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2092277:::benchmark_data_smithi159_162467_object30564:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2095994:::benchmark_data_smithi159_162467_object8001:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f209bcaa:::benchmark_data_smithi159_162467_object38744:head# 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f209c4ac:::benchmark_data_smithi159_162467_object31859:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f209fea3:::benchmark_data_smithi159_162467_object25490:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20a84de:::benchmark_data_smithi159_162467_object33231:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20abb49:::benchmark_data_smithi159_162467_object3952:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20afded:::benchmark_data_smithi159_162467_object42195:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20bd8c3:::benchmark_data_smithi159_162467_object1647:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20c3819:::benchmark_data_smithi159_162467_object15101:head# 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f20d89be:::benchmark_data_smithi159_162467_object17076:head# 2024-04-15T18:43:14.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2113204:::benchmark_data_smithi159_162467_object26100:head# 2024-04-15T18:43:14.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f212b0c3:::benchmark_data_smithi159_162467_object42628:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2130e15:::benchmark_data_smithi159_162467_object48639:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f213cf47:::benchmark_data_smithi159_162467_object27682:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f214b74e:::benchmark_data_smithi159_162467_object48465:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f214f0ca:::benchmark_data_smithi159_162467_object68806:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2154838:::benchmark_data_smithi159_162467_object1918:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f218de75:::benchmark_data_smithi159_162467_object73244:head# 2024-04-15T18:43:14.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f219155c:::benchmark_data_smithi159_162467_object10089:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21a37a3:::benchmark_data_smithi159_162467_object70632:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21a6afc:::benchmark_data_smithi159_162467_object23268:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21acc5f:::benchmark_data_smithi159_162467_object6642:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21affc4:::benchmark_data_smithi159_162467_object30568:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21be023:::benchmark_data_smithi159_162467_object64156:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21c2fb4:::benchmark_data_smithi159_162467_object21761:head# 2024-04-15T18:43:14.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21c3a55:::benchmark_data_smithi159_162467_object23846:head# 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21cf15f:::benchmark_data_smithi159_162467_object9436:head# 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21edc7d:::benchmark_data_smithi159_162467_object67391:head# 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21f0ecd:::benchmark_data_smithi159_162467_object73696:head# 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f21f4ce8:::benchmark_data_smithi159_162467_object18389:head# 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f220152e:::benchmark_data_smithi159_162467_object20631:head# 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2203771:::benchmark_data_smithi159_162467_object73391:head# 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2207a3a:::benchmark_data_smithi159_162467_object53097:head# 2024-04-15T18:43:14.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22095dd:::benchmark_data_smithi159_162467_object63947:head# 2024-04-15T18:43:14.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2228c1d:::benchmark_data_smithi159_162467_object44038:head# 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2233e33:::benchmark_data_smithi159_162467_object62897:head# 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2236c76:::benchmark_data_smithi159_162467_object55699:head# 2024-04-15T18:43:14.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2238508:::benchmark_data_smithi159_162467_object5958:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22431a8:::benchmark_data_smithi159_162467_object59679:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f224ae3d:::benchmark_data_smithi159_162467_object1983:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22544d4:::benchmark_data_smithi159_162467_object10816:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f225593c:::benchmark_data_smithi159_162467_object16803:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22579b9:::benchmark_data_smithi159_162467_object39572:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2268834:::benchmark_data_smithi159_162467_object31650:head# 2024-04-15T18:43:14.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22724d2:::benchmark_data_smithi159_162467_object65345:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2273085:::benchmark_data_smithi159_162467_object63238:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2275760:::benchmark_data_smithi159_162467_object37476:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f228173d:::benchmark_data_smithi159_162467_object71761:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22823b0:::benchmark_data_smithi159_162467_object48909:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2286198:::benchmark_data_smithi159_162467_object31605:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f228b80d:::benchmark_data_smithi159_162467_object73933:head# 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f229cb44:::benchmark_data_smithi159_162467_object21712:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22c3aa9:::benchmark_data_smithi159_162467_object20435:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22c89a5:::benchmark_data_smithi159_162467_object60916:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22e408f:::benchmark_data_smithi159_162467_object10666:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f22efbf8:::benchmark_data_smithi159_162467_object70333:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23092cb:::benchmark_data_smithi159_162467_object68106:head# 2024-04-15T18:43:14.021 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23240f9:::benchmark_data_smithi159_162467_object48374:head# 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23401db:::benchmark_data_smithi159_162467_object14250:head# 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f234c9d0:::benchmark_data_smithi159_162467_object37361:head# 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23803e4:::benchmark_data_smithi159_162467_object17939:head# 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.022 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23877a5:::benchmark_data_smithi159_162467_object72914:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f238966b:::benchmark_data_smithi159_162467_object8831:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2397b5e:::benchmark_data_smithi159_162467_object33660:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23ac60f:::benchmark_data_smithi159_162467_object43818:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23b1e97:::benchmark_data_smithi159_162467_object34622:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23b3c8a:::benchmark_data_smithi159_162467_object67713:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23c519f:::benchmark_data_smithi159_162467_object28614:head# 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.023 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23cdde9:::benchmark_data_smithi159_162467_object30572:head# 2024-04-15T18:43:14.024 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.024 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23dafe6:::benchmark_data_smithi159_162467_object20773:head# 2024-04-15T18:43:14.024 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.024 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f23e4696:::benchmark_data_smithi159_162467_object2607:head# 2024-04-15T18:43:14.024 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.029 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2408447:::benchmark_data_smithi159_162467_object20505:head# 2024-04-15T18:43:14.029 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f242b7a0:::benchmark_data_smithi159_162467_object21024:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2442592:::benchmark_data_smithi159_162467_object10007:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f244b39a:::benchmark_data_smithi159_162467_object57568:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2471bbd:::benchmark_data_smithi159_162467_object36256:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2471fc2:::benchmark_data_smithi159_162467_object53246:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f247a96b:::benchmark_data_smithi159_162467_object29373:head# 2024-04-15T18:43:14.030 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f247c424:::benchmark_data_smithi159_162467_object7978:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2481cfb:::benchmark_data_smithi159_162467_object31068:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2485022:::benchmark_data_smithi159_162467_object37586:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24850ff:::benchmark_data_smithi159_162467_object1752:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f248e85f:::benchmark_data_smithi159_162467_object73839:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2499239:::benchmark_data_smithi159_162467_object32213:head# 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.031 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24b8816:::benchmark_data_smithi159_162467_object69108:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24c5c42:::benchmark_data_smithi159_162467_object7943:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24cc62a:::benchmark_data_smithi159_162467_object43366:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24df08c:::benchmark_data_smithi159_162467_object23634:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24e1ece:::benchmark_data_smithi159_162467_object16593:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24e3ed6:::benchmark_data_smithi159_162467_object59550:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f24f43eb:::benchmark_data_smithi159_162467_object48926:head# 2024-04-15T18:43:14.032 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2518fef:::benchmark_data_smithi159_162467_object11895:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f252b063:::benchmark_data_smithi159_162467_object19037:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f252e80c:::benchmark_data_smithi159_162467_object37606:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f254d572:::benchmark_data_smithi159_162467_object58669:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f255adae:::benchmark_data_smithi159_162467_object25628:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2575eb8:::benchmark_data_smithi159_162467_object14921:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25773ae:::benchmark_data_smithi159_162467_object16347:head# 2024-04-15T18:43:14.033 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2590cdb:::benchmark_data_smithi159_162467_object8025:head# 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25c4bf8:::benchmark_data_smithi159_162467_object52006:head# 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25d25fd:::benchmark_data_smithi159_162467_object11366:head# 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25d9876:::benchmark_data_smithi159_162467_object41202:head# 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25e33dd:::benchmark_data_smithi159_162467_object40922:head# 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.034 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f25f97c4:::benchmark_data_smithi159_162467_object68093:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2602e16:::benchmark_data_smithi159_162467_object64295:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f260a9bb:::benchmark_data_smithi159_162467_object70000:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2624383:::benchmark_data_smithi159_162467_object9402:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26292f6:::benchmark_data_smithi159_162467_object49367:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f262cc1d:::benchmark_data_smithi159_162467_object31936:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2630a6f:::benchmark_data_smithi159_162467_object40172:head# 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.035 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2636c83:::benchmark_data_smithi159_162467_object67045:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f263cbb3:::benchmark_data_smithi159_162467_object29240:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f263d400:::benchmark_data_smithi159_162467_object16370:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f264a94d:::benchmark_data_smithi159_162467_object70269:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2658f8c:::benchmark_data_smithi159_162467_object7252:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2662e81:::benchmark_data_smithi159_162467_object58698:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f266d2fa:::benchmark_data_smithi159_162467_object25042:head# 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.036 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f267259b:::benchmark_data_smithi159_162467_object26888:head# 2024-04-15T18:43:14.037 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.037 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f267db66:::benchmark_data_smithi159_162467_object41332:head# 2024-04-15T18:43:14.037 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.051 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f267fd9c:::benchmark_data_smithi159_162467_object62998:head# 2024-04-15T18:43:14.051 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2689534:::benchmark_data_smithi159_162467_object18523:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f268ae50:::benchmark_data_smithi159_162467_object3682:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26959dc:::benchmark_data_smithi159_162467_object57886:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26bc838:::benchmark_data_smithi159_162467_object30935:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26d09c8:::benchmark_data_smithi159_162467_object14273:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26e9356:::benchmark_data_smithi159_162467_object59364:head# 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.052 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f26f0013:::benchmark_data_smithi159_162467_object70753:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27018b4:::benchmark_data_smithi159_162467_object4406:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2706ea6:::benchmark_data_smithi159_162467_object44261:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27098d5:::benchmark_data_smithi159_162467_object66742:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27124bc:::benchmark_data_smithi159_162467_object8382:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f274225b:::benchmark_data_smithi159_162467_object44249:head# 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.053 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f274b4ec:::benchmark_data_smithi159_162467_object56217:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f275c5d6:::benchmark_data_smithi159_162467_object10948:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2769459:::benchmark_data_smithi159_162467_object25960:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2777431:::benchmark_data_smithi159_162467_object8829:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2778195:::benchmark_data_smithi159_162467_object51697:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f278ce51:::benchmark_data_smithi159_162467_object26801:head# 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.054 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2792311:::benchmark_data_smithi159_162467_object56902:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f279bdcd:::benchmark_data_smithi159_162467_object67910:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f279cd2d:::benchmark_data_smithi159_162467_object8615:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27b7970:::benchmark_data_smithi159_162467_object51807:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27bb612:::benchmark_data_smithi159_162467_object54531:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27ce395:::benchmark_data_smithi159_162467_object35026:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27f4911:::benchmark_data_smithi159_162467_object5160:head# 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.055 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27f7b87:::benchmark_data_smithi159_162467_object42512:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f27fcd47:::benchmark_data_smithi159_162467_object67799:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28090c4:::benchmark_data_smithi159_162467_object67558:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2812d8c:::benchmark_data_smithi159_162467_object41681:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f281cf22:::benchmark_data_smithi159_162467_object48023:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f283ae53:::benchmark_data_smithi159_162467_object71290:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2876f4a:::benchmark_data_smithi159_162467_object48929:head# 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.056 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f289f0f6:::benchmark_data_smithi159_162467_object48416:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28a3f37:::benchmark_data_smithi159_162467_object52531:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28b15b5:::benchmark_data_smithi159_162467_object36932:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28bb2d0:::benchmark_data_smithi159_162467_object72595:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28bba51:::benchmark_data_smithi159_162467_object62300:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28ee051:::benchmark_data_smithi159_162467_object19288:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f28fca23:::benchmark_data_smithi159_162467_object43180:head# 2024-04-15T18:43:14.057 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2919bac:::benchmark_data_smithi159_162467_object31766:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f293758e:::benchmark_data_smithi159_162467_object385:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2937827:::benchmark_data_smithi159_162467_object61465:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f293879f:::benchmark_data_smithi159_162467_object4062:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f293e974:::benchmark_data_smithi159_162467_object65550:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f294182b:::benchmark_data_smithi159_162467_object47151:head# 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.058 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2977c12:::benchmark_data_smithi159_162467_object52455:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2996e93:::benchmark_data_smithi159_162467_object36948:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29a6245:::benchmark_data_smithi159_162467_object7616:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29a7679:::benchmark_data_smithi159_162467_object56642:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29b7094:::benchmark_data_smithi159_162467_object52661:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29d8974:::benchmark_data_smithi159_162467_object69240:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29e8524:::benchmark_data_smithi159_162467_object58000:head# 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.059 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29ef07a:::benchmark_data_smithi159_162467_object30248:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29ef21c:::benchmark_data_smithi159_162467_object2625:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29fa01f:::benchmark_data_smithi159_162467_object43434:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f29fb707:::benchmark_data_smithi159_162467_object54074:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a0f87f:::benchmark_data_smithi159_162467_object40848:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a29c66:::benchmark_data_smithi159_162467_object55565:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a4c1e1:::benchmark_data_smithi159_162467_object54277:head# 2024-04-15T18:43:14.060 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a4cf5e:::benchmark_data_smithi159_162467_object48782:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a53560:::benchmark_data_smithi159_162467_object35382:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a56652:::benchmark_data_smithi159_162467_object52540:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a7020e:::benchmark_data_smithi159_162467_object29449:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2a7cb09:::benchmark_data_smithi159_162467_object13031:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2aae6de:::benchmark_data_smithi159_162467_object27547:head# 2024-04-15T18:43:14.061 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ab18b3:::benchmark_data_smithi159_162467_object41407:head# 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2adb979:::benchmark_data_smithi159_162467_object38967:head# 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ae2f6b:::benchmark_data_smithi159_162467_object15285:head# 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2af099c:::benchmark_data_smithi159_162467_object12652:head# 2024-04-15T18:43:14.062 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2af187e:::benchmark_data_smithi159_162467_object39887:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b06c84:::benchmark_data_smithi159_162467_object49937:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b09209:::benchmark_data_smithi159_162467_object42517:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b148b5:::benchmark_data_smithi159_162467_object1291:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b43f19:::benchmark_data_smithi159_162467_object5666:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b5b507:::benchmark_data_smithi159_162467_object32277:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b5fef8:::benchmark_data_smithi159_162467_object15908:head# 2024-04-15T18:43:14.063 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b87065:::benchmark_data_smithi159_162467_object31562:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b8a775:::benchmark_data_smithi159_162467_object40270:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b8e48d:::benchmark_data_smithi159_162467_object30281:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2b9ea07:::benchmark_data_smithi159_162467_object57305:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ba7615:::benchmark_data_smithi159_162467_object3428:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bb2de1:::benchmark_data_smithi159_162467_object6951:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bb40c2:::benchmark_data_smithi159_162467_object49984:head# 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.064 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bb9bf5:::benchmark_data_smithi159_162467_object24758:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bbf9af:::benchmark_data_smithi159_162467_object58685:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2be2749:::benchmark_data_smithi159_162467_object62222:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2be8588:::benchmark_data_smithi159_162467_object53624:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bea2aa:::benchmark_data_smithi159_162467_object58517:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2bee4b6:::benchmark_data_smithi159_162467_object19767:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c0150c:::benchmark_data_smithi159_162467_object46285:head# 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.065 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c083c4:::benchmark_data_smithi159_162467_object15502:head# 2024-04-15T18:43:14.066 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.066 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c1f1f1:::benchmark_data_smithi159_162467_object26226:head# 2024-04-15T18:43:14.066 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.066 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c25c27:::benchmark_data_smithi159_162467_object11555:head# 2024-04-15T18:43:14.066 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c27b5f:::benchmark_data_smithi159_162467_object25992:head# 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c2a718:::benchmark_data_smithi159_162467_object71661:head# 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c45f53:::benchmark_data_smithi159_162467_object17340:head# 2024-04-15T18:43:14.071 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c4dfe8:::benchmark_data_smithi159_162467_object25784:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c55ca1:::benchmark_data_smithi159_162467_object46203:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c62dbe:::benchmark_data_smithi159_162467_object14087:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c6a17a:::benchmark_data_smithi159_162467_object2379:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c8aacf:::benchmark_data_smithi159_162467_object9850:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c952c1:::benchmark_data_smithi159_162467_object26958:head# 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.072 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2c965f8:::benchmark_data_smithi159_162467_object23053:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2cc1778:::benchmark_data_smithi159_162467_object14562:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2cc49df:::benchmark_data_smithi159_162467_object11748:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2cd140a:::benchmark_data_smithi159_162467_object65123:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2cd2eb4:::benchmark_data_smithi159_162467_object30204:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2cf6e43:::benchmark_data_smithi159_162467_object53151:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d1204a:::benchmark_data_smithi159_162467_object55243:head# 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.073 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d1faec:::benchmark_data_smithi159_162467_object70477:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d24eae:::benchmark_data_smithi159_162467_object48056:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d27d28:::benchmark_data_smithi159_162467_object22814:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d49b7a:::benchmark_data_smithi159_162467_object68642:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d4d561:::benchmark_data_smithi159_162467_object9354:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d4fc7e:::benchmark_data_smithi159_162467_object32267:head# 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.074 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d5e6f1:::benchmark_data_smithi159_162467_object5289:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d6a1b5:::benchmark_data_smithi159_162467_object58161:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2d91786:::benchmark_data_smithi159_162467_object12928:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2da2ef6:::benchmark_data_smithi159_162467_object36279:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2dac05a:::benchmark_data_smithi159_162467_object23062:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2db20ce:::benchmark_data_smithi159_162467_object55632:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2db642c:::benchmark_data_smithi159_162467_object1602:head# 2024-04-15T18:43:14.075 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2db66b4:::benchmark_data_smithi159_162467_object69865:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2dbf4fe:::benchmark_data_smithi159_162467_object65671:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2dc3138:::benchmark_data_smithi159_162467_object64157:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2de8ef2:::benchmark_data_smithi159_162467_object17718:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2df32ae:::benchmark_data_smithi159_162467_object61124:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2dfd0cb:::benchmark_data_smithi159_162467_object33079:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e0031b:::benchmark_data_smithi159_162467_object68974:head# 2024-04-15T18:43:14.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e42c3c:::benchmark_data_smithi159_162467_object43687:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e43c7b:::benchmark_data_smithi159_162467_object29100:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e48a17:::benchmark_data_smithi159_162467_object4175:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e5c614:::benchmark_data_smithi159_162467_object640:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e728a9:::benchmark_data_smithi159_162467_object70058:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e7853f:::benchmark_data_smithi159_162467_object56491:head# 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e7f2b7:::benchmark_data_smithi159_162467_object19126:head# 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e88a67:::benchmark_data_smithi159_162467_object455:head# 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e8ac43:::benchmark_data_smithi159_162467_object17912:head# 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e8c946:::benchmark_data_smithi159_162467_object28919:head# 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2e9a3d9:::benchmark_data_smithi159_162467_object28783:head# 2024-04-15T18:43:14.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ea3a85:::benchmark_data_smithi159_162467_object37585:head# 2024-04-15T18:43:14.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2eb77a3:::benchmark_data_smithi159_162467_object7341:head# 2024-04-15T18:43:14.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ecb479:::benchmark_data_smithi159_162467_object21315:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2edbf51:::benchmark_data_smithi159_162467_object25078:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ef8af3:::benchmark_data_smithi159_162467_object8498:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f27617:::benchmark_data_smithi159_162467_object52366:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f38190:::benchmark_data_smithi159_162467_object57053:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f4251b:::benchmark_data_smithi159_162467_object20526:head# 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f4c395:::benchmark_data_smithi159_162467_object40108:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f57505:::benchmark_data_smithi159_162467_object11790:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f800e6:::benchmark_data_smithi159_162467_object17060:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f8400d:::benchmark_data_smithi159_162467_object39445:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2f948a3:::benchmark_data_smithi159_162467_object30679:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fab0ae:::benchmark_data_smithi159_162467_object15001:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fab2e2:::benchmark_data_smithi159_162467_object25469:head# 2024-04-15T18:43:14.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2faee4c:::benchmark_data_smithi159_162467_object68263:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fb66e4:::benchmark_data_smithi159_162467_object46776:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fbf0de:::benchmark_data_smithi159_162467_object73060:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fbf33c:::benchmark_data_smithi159_162467_object10484:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fc7f53:::benchmark_data_smithi159_162467_object6259:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fce48b:::benchmark_data_smithi159_162467_object30046:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fe12ee:::benchmark_data_smithi159_162467_object62699:head# 2024-04-15T18:43:14.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2fea33b:::benchmark_data_smithi159_162467_object62636:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2feed63:::benchmark_data_smithi159_162467_object59086:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f2ffed37:::benchmark_data_smithi159_162467_object47304:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f30082b7:::benchmark_data_smithi159_162467_object44986:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f30160df:::benchmark_data_smithi159_162467_object60742:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3026671:::benchmark_data_smithi159_162467_object32424:head# 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3027a97:::benchmark_data_smithi159_162467_object62246:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f305fad4:::benchmark_data_smithi159_162467_object7213:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f306b1cd:::benchmark_data_smithi159_162467_object54058:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3082e46:::benchmark_data_smithi159_162467_object33214:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3085290:::benchmark_data_smithi159_162467_object8370:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f308b39c:::benchmark_data_smithi159_162467_object71741:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f30b4254:::benchmark_data_smithi159_162467_object20097:head# 2024-04-15T18:43:14.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f30db67b:::benchmark_data_smithi159_162467_object71806:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3106b69:::benchmark_data_smithi159_162467_object6245:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f312c0c5:::benchmark_data_smithi159_162467_object73402:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f314cc13:::benchmark_data_smithi159_162467_object19368:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3151fcd:::benchmark_data_smithi159_162467_object38134:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f315622e:::benchmark_data_smithi159_162467_object70223:head# 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f315770e:::benchmark_data_smithi159_162467_object35652:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f315bac2:::benchmark_data_smithi159_162467_object32019:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f316a44d:::benchmark_data_smithi159_162467_object6997:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f317343b:::benchmark_data_smithi159_162467_object33932:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f317e656:::benchmark_data_smithi159_162467_object74652:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f318230e:::benchmark_data_smithi159_162467_object5007:head# 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f31840e3:::benchmark_data_smithi159_162467_object8698:head# 2024-04-15T18:43:14.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f318d78e:::benchmark_data_smithi159_162467_object29787:head# 2024-04-15T18:43:14.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3192c8f:::benchmark_data_smithi159_162467_object2943:head# 2024-04-15T18:43:14.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:43:14.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:43:14.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3193896:::benchmark_data_smithi159_162467_object8363:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3194f1a:::benchmark_data_smithi159_162467_object10876:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f31ea49b:::benchmark_data_smithi159_162467_object33980:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f31f2132:::benchmark_data_smithi159_162467_object54846:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3203984:::benchmark_data_smithi159_162467_object28965:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f320ad53:::benchmark_data_smithi159_162467_object30:head# 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f320be24:::benchmark_data_smithi159_162467_object58951:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3212609:::benchmark_data_smithi159_162467_object54832:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3236f24:::benchmark_data_smithi159_162467_object18729:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3252b0d:::benchmark_data_smithi159_162467_object64298:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f325711d:::benchmark_data_smithi159_162467_object40684:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f326a4ec:::benchmark_data_smithi159_162467_object65340:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f327c9cf:::benchmark_data_smithi159_162467_object16650:head# 2024-04-15T18:43:14.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32923e5:::benchmark_data_smithi159_162467_object2016:head# 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32977cb:::benchmark_data_smithi159_162467_object10736:head# 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32a8be8:::benchmark_data_smithi159_162467_object38122:head# 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32afb0f:::benchmark_data_smithi159_162467_object2371:head# 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32b14d0:::benchmark_data_smithi159_162467_object61034:head# 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32c04d4:::benchmark_data_smithi159_162467_object35895:head# 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f32d1705:::benchmark_data_smithi159_162467_object48867:head# 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3307629:::benchmark_data_smithi159_162467_object2223:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f330f9ed:::benchmark_data_smithi159_162467_object5000:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f331c7cf:::benchmark_data_smithi159_162467_object69396:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f331cd78:::benchmark_data_smithi159_162467_object61427:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3328a02:::benchmark_data_smithi159_162467_object49063:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f332b1f3:::benchmark_data_smithi159_162467_object26290:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f332c565:::benchmark_data_smithi159_162467_object36361:head# 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f334513b:::benchmark_data_smithi159_162467_object65268:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f33494da:::benchmark_data_smithi159_162467_object38249:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f334a39b:::benchmark_data_smithi159_162467_object9458:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f334af9e:::benchmark_data_smithi159_162467_object21593:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f335d54d:::benchmark_data_smithi159_162467_object29780:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f335f7fb:::benchmark_data_smithi159_162467_object52461:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f336c34d:::benchmark_data_smithi159_162467_object45820:head# 2024-04-15T18:43:14.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f339ed61:::benchmark_data_smithi159_162467_object39356:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f33ab726:::benchmark_data_smithi159_162467_object6008:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f33ad1ad:::benchmark_data_smithi159_162467_object17763:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f33bbf49:::benchmark_data_smithi159_162467_object68816:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f33de772:::benchmark_data_smithi159_162467_object5333:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3416cd2:::benchmark_data_smithi159_162467_object35995:head# 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3438325:::benchmark_data_smithi159_162467_object44273:head# 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3443a64:::benchmark_data_smithi159_162467_object65917:head# 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3444a4e:::benchmark_data_smithi159_162467_object71106:head# 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f344b1d2:::benchmark_data_smithi159_162467_object5781:head# 2024-04-15T18:43:14.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.130 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f344ef50:::benchmark_data_smithi159_162467_object23056:head# 2024-04-15T18:43:14.130 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.130 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3459793:::benchmark_data_smithi159_162467_object5131:head# 2024-04-15T18:43:14.130 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f345f8d2:::benchmark_data_smithi159_162467_object22872:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3464d70:::benchmark_data_smithi159_162467_object5133:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34883bf:::benchmark_data_smithi159_162467_object74115:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34a9742:::benchmark_data_smithi159_162467_object58592:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34acd5a:::benchmark_data_smithi159_162467_object66579:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34ad09b:::benchmark_data_smithi159_162467_object15770:head# 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34b37fa:::benchmark_data_smithi159_162467_object74356:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34c0553:::benchmark_data_smithi159_162467_object69019:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34d2b77:::benchmark_data_smithi159_162467_object54954:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34d2bf3:::benchmark_data_smithi159_162467_object27018:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f34e1d88:::benchmark_data_smithi159_162467_object44121:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f350192d:::benchmark_data_smithi159_162467_object27908:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f350271e:::benchmark_data_smithi159_162467_object55409:head# 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3509665:::benchmark_data_smithi159_162467_object59477:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f350aa05:::benchmark_data_smithi159_162467_object19695:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f350ccf4:::benchmark_data_smithi159_162467_object32201:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f351f3cc:::benchmark_data_smithi159_162467_object68892:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3534400:::benchmark_data_smithi159_162467_object11697:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f354eddf:::benchmark_data_smithi159_162467_object30064:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f355092a:::benchmark_data_smithi159_162467_object12952:head# 2024-04-15T18:43:14.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f355238d:::benchmark_data_smithi159_162467_object72790:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f355d5d6:::benchmark_data_smithi159_162467_object72980:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f357a72e:::benchmark_data_smithi159_162467_object6612:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f357bc06:::benchmark_data_smithi159_162467_object35038:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3586556:::benchmark_data_smithi159_162467_object36700:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f359bdab:::benchmark_data_smithi159_162467_object46117:head# 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f359e055:::benchmark_data_smithi159_162467_object56964:head# 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35a32a1:::benchmark_data_smithi159_162467_object42909:head# 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35a4b18:::benchmark_data_smithi159_162467_object44312:head# 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35abfb8:::benchmark_data_smithi159_162467_object34485:head# 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35b0c6f:::benchmark_data_smithi159_162467_object22286:head# 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35b7637:::benchmark_data_smithi159_162467_object64780:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35c49e2:::benchmark_data_smithi159_162467_object40977:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35c8336:::benchmark_data_smithi159_162467_object11334:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35d29ed:::benchmark_data_smithi159_162467_object1700:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35f03b2:::benchmark_data_smithi159_162467_object18536:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f35fade8:::benchmark_data_smithi159_162467_object30955:head# 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f36096d9:::benchmark_data_smithi159_162467_object10201:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f361146c:::benchmark_data_smithi159_162467_object41940:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3634acf:::benchmark_data_smithi159_162467_object47013:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3644429:::benchmark_data_smithi159_162467_object8686:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f364e708:::benchmark_data_smithi159_162467_object25364:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f365eb54:::benchmark_data_smithi159_162467_object9750:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f36663e4:::benchmark_data_smithi159_162467_object17158:head# 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3668ddf:::benchmark_data_smithi159_162467_object62298:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f367abd9:::benchmark_data_smithi159_162467_object20265:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f367c98b:::benchmark_data_smithi159_162467_object60878:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f36a1339:::benchmark_data_smithi159_162467_object24267:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f36aff9a:::benchmark_data_smithi159_162467_object57429:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f36fca99:::benchmark_data_smithi159_162467_object21241:head# 2024-04-15T18:43:14.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f370c19e:::benchmark_data_smithi159_162467_object71744:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37478c0:::benchmark_data_smithi159_162467_object68794:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f375cdc9:::benchmark_data_smithi159_162467_object30888:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f375ce7b:::benchmark_data_smithi159_162467_object68291:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f376005c:::benchmark_data_smithi159_162467_object44363:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3760810:::benchmark_data_smithi159_162467_object68802:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3761d17:::benchmark_data_smithi159_162467_object40955:head# 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37633c3:::benchmark_data_smithi159_162467_object40992:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f376b723:::benchmark_data_smithi159_162467_object26076:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3777091:::benchmark_data_smithi159_162467_object3013:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f377821a:::benchmark_data_smithi159_162467_object10623:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f377add1:::benchmark_data_smithi159_162467_object48813:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f377e7ea:::benchmark_data_smithi159_162467_object74636:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3786816:::benchmark_data_smithi159_162467_object20300:head# 2024-04-15T18:43:14.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3789350:::benchmark_data_smithi159_162467_object54157:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37a0075:::benchmark_data_smithi159_162467_object43432:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37ab5e5:::benchmark_data_smithi159_162467_object10616:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37c4139:::benchmark_data_smithi159_162467_object62744:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37cd0b5:::benchmark_data_smithi159_162467_object35742:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37d623d:::benchmark_data_smithi159_162467_object12942:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37d7c63:::benchmark_data_smithi159_162467_object14624:head# 2024-04-15T18:43:14.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f37fc91a:::benchmark_data_smithi159_162467_object41184:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f380b1ac:::benchmark_data_smithi159_162467_object62850:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f380b413:::benchmark_data_smithi159_162467_object8591:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3810a90:::benchmark_data_smithi159_162467_object2725:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3829918:::benchmark_data_smithi159_162467_object20619:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38359fe:::benchmark_data_smithi159_162467_object64741:head# 2024-04-15T18:43:14.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f383996b:::benchmark_data_smithi159_162467_object52795:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f383e9ae:::benchmark_data_smithi159_162467_object27421:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3841e0b:::benchmark_data_smithi159_162467_object9659:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f384cd56:::benchmark_data_smithi159_162467_object35702:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f384f7c7:::benchmark_data_smithi159_162467_object53424:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3851b45:::benchmark_data_smithi159_162467_object67911:head# 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f385a2d4:::benchmark_data_smithi159_162467_object19851:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f385db62:::benchmark_data_smithi159_162467_object60059:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f386978e:::benchmark_data_smithi159_162467_object60727:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f386cfd2:::benchmark_data_smithi159_162467_object29507:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38723e3:::benchmark_data_smithi159_162467_object31017:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38791ac:::benchmark_data_smithi159_162467_object53965:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3890ff9:::benchmark_data_smithi159_162467_object74398:head# 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3891003:::benchmark_data_smithi159_162467_object55505:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f389a351:::benchmark_data_smithi159_162467_object57474:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38b04fc:::benchmark_data_smithi159_162467_object12310:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38b19af:::benchmark_data_smithi159_162467_object48825:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38e2cac:::benchmark_data_smithi159_162467_object61961:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38e5837:::benchmark_data_smithi159_162467_object32453:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f38f574d:::benchmark_data_smithi159_162467_object30247:head# 2024-04-15T18:43:14.145 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f390ce09:::benchmark_data_smithi159_162467_object34369:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f390ce4a:::benchmark_data_smithi159_162467_object29408:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3925eaa:::benchmark_data_smithi159_162467_object46430:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39263de:::benchmark_data_smithi159_162467_object66688:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f392cf3f:::benchmark_data_smithi159_162467_object28671:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f393619b:::benchmark_data_smithi159_162467_object46952:head# 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.146 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f393a07d:::benchmark_data_smithi159_162467_object4173:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3941a14:::benchmark_data_smithi159_162467_object26295:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3944c16:::benchmark_data_smithi159_162467_object43413:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3951e5d:::benchmark_data_smithi159_162467_object34859:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39529cd:::benchmark_data_smithi159_162467_object59205:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39558f0:::benchmark_data_smithi159_162467_object69972:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f395d18b:::benchmark_data_smithi159_162467_object31717:head# 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.147 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3971031:::benchmark_data_smithi159_162467_object2498:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f397cebe:::benchmark_data_smithi159_162467_object44504:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39826b8:::benchmark_data_smithi159_162467_object45651:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39a14ae:::benchmark_data_smithi159_162467_object30159:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39a5482:::benchmark_data_smithi159_162467_object37800:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39b48a4:::benchmark_data_smithi159_162467_object29796:head# 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.148 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39be05e:::benchmark_data_smithi159_162467_object48846:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39be845:::benchmark_data_smithi159_162467_object3792:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39bea67:::benchmark_data_smithi159_162467_object59065:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39cb3c9:::benchmark_data_smithi159_162467_object63440:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39e1d43:::benchmark_data_smithi159_162467_object66894:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39f104c:::benchmark_data_smithi159_162467_object72321:head# 2024-04-15T18:43:14.149 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f39fae05:::benchmark_data_smithi159_162467_object25152:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a03815:::benchmark_data_smithi159_162467_object52945:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a0775e:::benchmark_data_smithi159_162467_object7675:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a353d2:::benchmark_data_smithi159_162467_object66582:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a3ab33:::benchmark_data_smithi159_162467_object20760:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a58dbc:::benchmark_data_smithi159_162467_object25982:head# 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.150 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a77f40:::benchmark_data_smithi159_162467_object38510:head# 2024-04-15T18:43:14.151 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a7f59d:::benchmark_data_smithi159_162467_object21119:head# 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a89593:::benchmark_data_smithi159_162467_object4457:head# 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a97dda:::benchmark_data_smithi159_162467_object22510:head# 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3a9e0d1:::benchmark_data_smithi159_162467_object51951:head# 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3aadcc3:::benchmark_data_smithi159_162467_object52242:head# 2024-04-15T18:43:14.165 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3aaee5b:::benchmark_data_smithi159_162467_object54977:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ab0e1f:::benchmark_data_smithi159_162467_object48623:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ac19fb:::benchmark_data_smithi159_162467_object17373:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ac32b5:::benchmark_data_smithi159_162467_object23021:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ad9965:::benchmark_data_smithi159_162467_object53463:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3adbab2:::benchmark_data_smithi159_162467_object69315:head# 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.166 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ae77e8:::benchmark_data_smithi159_162467_object9578:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3aea2b8:::benchmark_data_smithi159_162467_object39003:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3af8d02:::benchmark_data_smithi159_162467_object49127:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3afa726:::benchmark_data_smithi159_162467_object54530:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b03fe2:::benchmark_data_smithi159_162467_object72033:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b06b75:::benchmark_data_smithi159_162467_object31136:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b070e8:::benchmark_data_smithi159_162467_object59366:head# 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.167 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b165f3:::benchmark_data_smithi159_162467_object660:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b18168:::benchmark_data_smithi159_162467_object39433:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b259e2:::benchmark_data_smithi159_162467_object68334:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b36f15:::benchmark_data_smithi159_162467_object29929:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b3b37a:::benchmark_data_smithi159_162467_object28680:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b3cec1:::benchmark_data_smithi159_162467_object53265:head# 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.168 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b40e47:::benchmark_data_smithi159_162467_object50426:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b42f19:::benchmark_data_smithi159_162467_object31653:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b48086:::benchmark_data_smithi159_162467_object74078:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b562f9:::benchmark_data_smithi159_162467_object11901:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b5b3e0:::benchmark_data_smithi159_162467_object63075:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b5ed84:::benchmark_data_smithi159_162467_object72517:head# 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.169 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b64b00:::benchmark_data_smithi159_162467_object27952:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b6ba05:::benchmark_data_smithi159_162467_object7518:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b72ac1:::benchmark_data_smithi159_162467_object60685:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3b7d4b4:::benchmark_data_smithi159_162467_object65288:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3baaaa5:::benchmark_data_smithi159_162467_object54589:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bc382e:::benchmark_data_smithi159_162467_object8930:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bcd6d4:::benchmark_data_smithi159_162467_object62228:head# 2024-04-15T18:43:14.170 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bd6497:::benchmark_data_smithi159_162467_object69486:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bdb0c5:::benchmark_data_smithi159_162467_object74085:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bdf458:::benchmark_data_smithi159_162467_object53063:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3be0e68:::benchmark_data_smithi159_162467_object22575:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bec68f:::benchmark_data_smithi159_162467_object18195:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bf2e0f:::benchmark_data_smithi159_162467_object32456:head# 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.171 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bf75b6:::benchmark_data_smithi159_162467_object34499:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bfb6a0:::benchmark_data_smithi159_162467_object48455:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3bfdda6:::benchmark_data_smithi159_162467_object59459:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c0acc1:::benchmark_data_smithi159_162467_object39508:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c1e7f1:::benchmark_data_smithi159_162467_object51304:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c24fcb:::benchmark_data_smithi159_162467_object49878:head# 2024-04-15T18:43:14.172 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c3c991:::benchmark_data_smithi159_162467_object38965:head# 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c4c749:::benchmark_data_smithi159_162467_object8090:head# 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c54af0:::benchmark_data_smithi159_162467_object35094:head# 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c55ed3:::benchmark_data_smithi159_162467_object4:head# 2024-04-15T18:43:14.173 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c609ab:::benchmark_data_smithi159_162467_object1180:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c69164:::benchmark_data_smithi159_162467_object63022:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c71e47:::benchmark_data_smithi159_162467_object33256:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c7e452:::benchmark_data_smithi159_162467_object70933:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c80a95:::benchmark_data_smithi159_162467_object8172:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c855c6:::benchmark_data_smithi159_162467_object16385:head# 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.174 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c8d1c3:::benchmark_data_smithi159_162467_object649:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c973cf:::benchmark_data_smithi159_162467_object21194:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c99bc8:::benchmark_data_smithi159_162467_object16815:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3c9b090:::benchmark_data_smithi159_162467_object3875:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ca19d8:::benchmark_data_smithi159_162467_object72419:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ca6382:::benchmark_data_smithi159_162467_object49439:head# 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.175 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3cbdd52:::benchmark_data_smithi159_162467_object8220:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3cc2136:::benchmark_data_smithi159_162467_object74453:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3cc40fe:::benchmark_data_smithi159_162467_object22565:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3cdc193:::benchmark_data_smithi159_162467_object25756:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ce9a3c:::benchmark_data_smithi159_162467_object36514:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ce9b80:::benchmark_data_smithi159_162467_object44500:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3cef398:::benchmark_data_smithi159_162467_object58414:head# 2024-04-15T18:43:14.176 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3d2c0eb:::benchmark_data_smithi159_162467_object26201:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3d64244:::benchmark_data_smithi159_162467_object65366:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3d7e245:::benchmark_data_smithi159_162467_object21390:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3d9a5a7:::benchmark_data_smithi159_162467_object59009:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3da3bff:::benchmark_data_smithi159_162467_object3910:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3db302e:::benchmark_data_smithi159_162467_object69210:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3db9dbe:::benchmark_data_smithi159_162467_object37116:head# 2024-04-15T18:43:14.177 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3dc62f2:::benchmark_data_smithi159_162467_object65932:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3dde39d:::benchmark_data_smithi159_162467_object23907:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ddf2f6:::benchmark_data_smithi159_162467_object33804:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3de1f71:::benchmark_data_smithi159_162467_object31513:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3dfc86d:::benchmark_data_smithi159_162467_object21009:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e0a25a:::benchmark_data_smithi159_162467_object24283:head# 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.178 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e10cbb:::benchmark_data_smithi159_162467_object46807:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e1387f:::benchmark_data_smithi159_162467_object70395:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e1b8c9:::benchmark_data_smithi159_162467_object23792:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e27d33:::benchmark_data_smithi159_162467_object73684:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e3ba7d:::benchmark_data_smithi159_162467_object4101:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e50a3e:::benchmark_data_smithi159_162467_object69456:head# 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.179 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e565d3:::benchmark_data_smithi159_162467_object42108:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e8a5f4:::benchmark_data_smithi159_162467_object46915:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e95f89:::benchmark_data_smithi159_162467_object3901:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3e987a1:::benchmark_data_smithi159_162467_object54805:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3eaa719:::benchmark_data_smithi159_162467_object61052:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3eaec3a:::benchmark_data_smithi159_162467_object33297:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3eb1e14:::benchmark_data_smithi159_162467_object62568:head# 2024-04-15T18:43:14.180 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f00d9d:::benchmark_data_smithi159_162467_object45610:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f0a95f:::benchmark_data_smithi159_162467_object15616:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f1c24a:::benchmark_data_smithi159_162467_object2770:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f206d6:::benchmark_data_smithi159_162467_object43562:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f3bed8:::benchmark_data_smithi159_162467_object23930:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f3c10d:::benchmark_data_smithi159_162467_object72920:head# 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.181 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f50c7a:::benchmark_data_smithi159_162467_object35706:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f51a10:::benchmark_data_smithi159_162467_object41111:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f51d10:::benchmark_data_smithi159_162467_object9818:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f6d9f3:::benchmark_data_smithi159_162467_object58182:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f8293a:::benchmark_data_smithi159_162467_object56185:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f88cb9:::benchmark_data_smithi159_162467_object72205:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f90b4a:::benchmark_data_smithi159_162467_object24443:head# 2024-04-15T18:43:14.182 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f93a04:::benchmark_data_smithi159_162467_object43537:head# 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f9889b:::benchmark_data_smithi159_162467_object42574:head# 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3f9e194:::benchmark_data_smithi159_162467_object54964:head# 2024-04-15T18:43:14.183 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3fb7b18:::benchmark_data_smithi159_162467_object63481:head# 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3fc149e:::benchmark_data_smithi159_162467_object10332:head# 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3fcd53d:::benchmark_data_smithi159_162467_object49665:head# 2024-04-15T18:43:14.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3fd6dda:::benchmark_data_smithi159_162467_object67284:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3fd8748:::benchmark_data_smithi159_162467_object16160:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f3ffc47a:::benchmark_data_smithi159_162467_object73137:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f400868d:::benchmark_data_smithi159_162467_object64927:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f400b006:::benchmark_data_smithi159_162467_object65871:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f400e4b0:::benchmark_data_smithi159_162467_object66363:head# 2024-04-15T18:43:14.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4010aa1:::benchmark_data_smithi159_162467_object3802:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4023bb3:::benchmark_data_smithi159_162467_object57915:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f402c623:::benchmark_data_smithi159_162467_object5088:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f403587f:::benchmark_data_smithi159_162467_object40135:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f404807b:::benchmark_data_smithi159_162467_object61828:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f405417b:::benchmark_data_smithi159_162467_object33464:head# 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4074c55:::benchmark_data_smithi159_162467_object68135:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4074f92:::benchmark_data_smithi159_162467_object48224:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f407b8e6:::benchmark_data_smithi159_162467_object68567:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f408d6b0:::benchmark_data_smithi159_162467_object13555:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4093b6b:::benchmark_data_smithi159_162467_object21640:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f409df71:::benchmark_data_smithi159_162467_object45866:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40a30ad:::benchmark_data_smithi159_162467_object21127:head# 2024-04-15T18:43:14.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40ba16c:::benchmark_data_smithi159_162467_object72694:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40d797d:::benchmark_data_smithi159_162467_object39425:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40e54a2:::benchmark_data_smithi159_162467_object60126:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40eabd4:::benchmark_data_smithi159_162467_object20316:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40f0ae6:::benchmark_data_smithi159_162467_object19880:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f40fb71a:::benchmark_data_smithi159_162467_object35925:head# 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4104e45:::benchmark_data_smithi159_162467_object7217:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4105983:::benchmark_data_smithi159_162467_object7780:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4106fb7:::benchmark_data_smithi159_162467_object35488:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4118419:::benchmark_data_smithi159_162467_object6485:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41355e8:::benchmark_data_smithi159_162467_object72473:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f413de9a:::benchmark_data_smithi159_162467_object69255:head# 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4142f8f:::benchmark_data_smithi159_162467_object48953:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4143506:::benchmark_data_smithi159_162467_object32043:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f414fc99:::benchmark_data_smithi159_162467_object46189:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f415ba49:::benchmark_data_smithi159_162467_object1113:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4165eb8:::benchmark_data_smithi159_162467_object61593:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41747e3:::benchmark_data_smithi159_162467_object9258:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4199819:::benchmark_data_smithi159_162467_object24724:head# 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41a2f2c:::benchmark_data_smithi159_162467_object43126:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41aa1ac:::benchmark_data_smithi159_162467_object68166:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41b1e16:::benchmark_data_smithi159_162467_object15771:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41b2d38:::benchmark_data_smithi159_162467_object28155:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41bcc3a:::benchmark_data_smithi159_162467_object65759:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41c6260:::benchmark_data_smithi159_162467_object35370:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41eabda:::benchmark_data_smithi159_162467_object36114:head# 2024-04-15T18:43:14.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41ef5f8:::benchmark_data_smithi159_162467_object842:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41f0fde:::benchmark_data_smithi159_162467_object44346:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41f6b5f:::benchmark_data_smithi159_162467_object66677:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41f82bd:::benchmark_data_smithi159_162467_object4143:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41fc88d:::benchmark_data_smithi159_162467_object28822:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f41fcd21:::benchmark_data_smithi159_162467_object50369:head# 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4208395:::benchmark_data_smithi159_162467_object36737:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42259dd:::benchmark_data_smithi159_162467_object43259:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f422f9ff:::benchmark_data_smithi159_162467_object37107:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f423657a:::benchmark_data_smithi159_162467_object21559:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4237889:::benchmark_data_smithi159_162467_object26155:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42379ae:::benchmark_data_smithi159_162467_object25801:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f424cf31:::benchmark_data_smithi159_162467_object60901:head# 2024-04-15T18:43:14.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4254aa4:::benchmark_data_smithi159_162467_object8953:head# 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4256269:::benchmark_data_smithi159_162467_object15129:head# 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4262991:::benchmark_data_smithi159_162467_object27785:head# 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4264624:::benchmark_data_smithi159_162467_object74613:head# 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f428aad9:::benchmark_data_smithi159_162467_object57050:head# 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4299111:::benchmark_data_smithi159_162467_object45268:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42b145b:::benchmark_data_smithi159_162467_object21019:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42b3457:::benchmark_data_smithi159_162467_object8942:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42bcbd9:::benchmark_data_smithi159_162467_object25402:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42c8f77:::benchmark_data_smithi159_162467_object36843:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42e89e7:::benchmark_data_smithi159_162467_object37765:head# 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f42ec127:::benchmark_data_smithi159_162467_object66220:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4306ba4:::benchmark_data_smithi159_162467_object20311:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4306f8e:::benchmark_data_smithi159_162467_object50692:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4306fc7:::benchmark_data_smithi159_162467_object37146:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f432bc12:::benchmark_data_smithi159_162467_object66422:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f434286d:::benchmark_data_smithi159_162467_object6833:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f434d671:::benchmark_data_smithi159_162467_object74592:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43703d7:::benchmark_data_smithi159_162467_object23593:head# 2024-04-15T18:43:14.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43760c1:::benchmark_data_smithi159_162467_object54886:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4382ec7:::benchmark_data_smithi159_162467_object57579:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43a7a6d:::benchmark_data_smithi159_162467_object18842:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43a83b0:::benchmark_data_smithi159_162467_object54615:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43baf63:::benchmark_data_smithi159_162467_object42426:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43c238e:::benchmark_data_smithi159_162467_object30473:head# 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43cddb6:::benchmark_data_smithi159_162467_object26251:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43cf885:::benchmark_data_smithi159_162467_object51559:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43d1288:::benchmark_data_smithi159_162467_object52274:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43f62e1:::benchmark_data_smithi159_162467_object67146:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f43fe77a:::benchmark_data_smithi159_162467_object52446:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f440fc59:::benchmark_data_smithi159_162467_object66417:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4410c7a:::benchmark_data_smithi159_162467_object55728:head# 2024-04-15T18:43:14.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4412ea3:::benchmark_data_smithi159_162467_object73974:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f441d61a:::benchmark_data_smithi159_162467_object36023:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f443d4a5:::benchmark_data_smithi159_162467_object52735:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44477f8:::benchmark_data_smithi159_162467_object29669:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f444e495:::benchmark_data_smithi159_162467_object53879:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44688f0:::benchmark_data_smithi159_162467_object51676:head# 2024-04-15T18:43:14.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4473f43:::benchmark_data_smithi159_162467_object28272:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4484b7c:::benchmark_data_smithi159_162467_object30299:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44995f5:::benchmark_data_smithi159_162467_object35396:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f449c91a:::benchmark_data_smithi159_162467_object16078:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44ab884:::benchmark_data_smithi159_162467_object31897:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44cb027:::benchmark_data_smithi159_162467_object49525:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44defaa:::benchmark_data_smithi159_162467_object65870:head# 2024-04-15T18:43:14.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44ea7df:::benchmark_data_smithi159_162467_object6592:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f44f23cf:::benchmark_data_smithi159_162467_object72659:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4531986:::benchmark_data_smithi159_162467_object27605:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45376c6:::benchmark_data_smithi159_162467_object71461:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45455c5:::benchmark_data_smithi159_162467_object30841:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f456173f:::benchmark_data_smithi159_162467_object45691:head# 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4570db7:::benchmark_data_smithi159_162467_object39696:head# 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f458f5df:::benchmark_data_smithi159_162467_object808:head# 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f459d048:::benchmark_data_smithi159_162467_object14299:head# 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45b2c6b:::benchmark_data_smithi159_162467_object5359:head# 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45b5b46:::benchmark_data_smithi159_162467_object11091:head# 2024-04-15T18:43:14.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45bbf49:::benchmark_data_smithi159_162467_object16412:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45c3d2c:::benchmark_data_smithi159_162467_object63942:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45c6f33:::benchmark_data_smithi159_162467_object56291:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45cde3d:::benchmark_data_smithi159_162467_object63615:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f45da16a:::benchmark_data_smithi159_162467_object71692:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f460aff3:::benchmark_data_smithi159_162467_object71112:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4634264:::benchmark_data_smithi159_162467_object57814:head# 2024-04-15T18:43:14.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f465c8fc:::benchmark_data_smithi159_162467_object62876:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46664e4:::benchmark_data_smithi159_162467_object12077:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46715f2:::benchmark_data_smithi159_162467_object65497:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46780da:::benchmark_data_smithi159_162467_object64138:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46804b9:::benchmark_data_smithi159_162467_object60678:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f469a11c:::benchmark_data_smithi159_162467_object55373:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46a96fd:::benchmark_data_smithi159_162467_object27556:head# 2024-04-15T18:43:14.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46b02e7:::benchmark_data_smithi159_162467_object61371:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46bd07d:::benchmark_data_smithi159_162467_object28523:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46cb242:::benchmark_data_smithi159_162467_object52742:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46cbcf4:::benchmark_data_smithi159_162467_object10632:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46d7068:::benchmark_data_smithi159_162467_object4343:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46e3c3a:::benchmark_data_smithi159_162467_object70970:head# 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46ec5ad:::benchmark_data_smithi159_162467_object42930:head# 2024-04-15T18:43:14.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f46ff327:::benchmark_data_smithi159_162467_object10092:head# 2024-04-15T18:43:14.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47097b2:::benchmark_data_smithi159_162467_object18493:head# 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f472c7a9:::benchmark_data_smithi159_162467_object27356:head# 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4740a4a:::benchmark_data_smithi159_162467_object24024:head# 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47470ef:::benchmark_data_smithi159_162467_object51632:head# 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.245 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f476621c:::benchmark_data_smithi159_162467_object46202:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47731de:::benchmark_data_smithi159_162467_object31219:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f478f8a5:::benchmark_data_smithi159_162467_object19545:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f478fef9:::benchmark_data_smithi159_162467_object34299:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f479f4c8:::benchmark_data_smithi159_162467_object2591:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47aa3e6:::benchmark_data_smithi159_162467_object14700:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47c4cc8:::benchmark_data_smithi159_162467_object42466:head# 2024-04-15T18:43:14.246 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47c86a2:::benchmark_data_smithi159_162467_object30789:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47c96a3:::benchmark_data_smithi159_162467_object35047:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47eb6d9:::benchmark_data_smithi159_162467_object62443:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f47fafed:::benchmark_data_smithi159_162467_object72353:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48069fb:::benchmark_data_smithi159_162467_object31903:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f480e4a1:::benchmark_data_smithi159_162467_object24738:head# 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.247 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f481522f:::benchmark_data_smithi159_162467_object65668:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f481b7d1:::benchmark_data_smithi159_162467_object61627:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f482c814:::benchmark_data_smithi159_162467_object53526:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f483007f:::benchmark_data_smithi159_162467_object51898:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f483060b:::benchmark_data_smithi159_162467_object33426:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f483f1a2:::benchmark_data_smithi159_162467_object1521:head# 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f483ffee:::benchmark_data_smithi159_162467_object17489:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48431ac:::benchmark_data_smithi159_162467_object10915:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48604b3:::benchmark_data_smithi159_162467_object40584:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4861bbc:::benchmark_data_smithi159_162467_object34473:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4863e34:::benchmark_data_smithi159_162467_object53176:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f488ac35:::benchmark_data_smithi159_162467_object3005:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48b00a3:::benchmark_data_smithi159_162467_object55232:head# 2024-04-15T18:43:14.249 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48cda7b:::benchmark_data_smithi159_162467_object73412:head# 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f48f657f:::benchmark_data_smithi159_162467_object4927:head# 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f491b109:::benchmark_data_smithi159_162467_object69332:head# 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f493f7ea:::benchmark_data_smithi159_162467_object46933:head# 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4980352:::benchmark_data_smithi159_162467_object17548:head# 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.250 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f498fafd:::benchmark_data_smithi159_162467_object6983:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4997281:::benchmark_data_smithi159_162467_object21756:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f499a399:::benchmark_data_smithi159_162467_object59980:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f499b8b3:::benchmark_data_smithi159_162467_object6876:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49a51e7:::benchmark_data_smithi159_162467_object53162:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49ad463:::benchmark_data_smithi159_162467_object49094:head# 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.251 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49c3eec:::benchmark_data_smithi159_162467_object50989:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49cb340:::benchmark_data_smithi159_162467_object9:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49d3723:::benchmark_data_smithi159_162467_object46621:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f49da034:::benchmark_data_smithi159_162467_object26397:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a04f97:::benchmark_data_smithi159_162467_object54191:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a11488:::benchmark_data_smithi159_162467_object19626:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a153dc:::benchmark_data_smithi159_162467_object11356:head# 2024-04-15T18:43:14.252 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a26a4c:::benchmark_data_smithi159_162467_object17861:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a27a98:::benchmark_data_smithi159_162467_object65227:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a289a3:::benchmark_data_smithi159_162467_object5614:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a298e5:::benchmark_data_smithi159_162467_object67717:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a2ea0a:::benchmark_data_smithi159_162467_object25116:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a43f8e:::benchmark_data_smithi159_162467_object56901:head# 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a51bfa:::benchmark_data_smithi159_162467_object62539:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a5da51:::benchmark_data_smithi159_162467_object18931:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a62551:::benchmark_data_smithi159_162467_object13231:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a8129f:::benchmark_data_smithi159_162467_object63485:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a943c4:::benchmark_data_smithi159_162467_object24810:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a9c3f1:::benchmark_data_smithi159_162467_object52112:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4a9c7bb:::benchmark_data_smithi159_162467_object67282:head# 2024-04-15T18:43:14.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4aa9cb1:::benchmark_data_smithi159_162467_object69619:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ab24c2:::benchmark_data_smithi159_162467_object33343:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4aedb69:::benchmark_data_smithi159_162467_object36608:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b0ed75:::benchmark_data_smithi159_162467_object64350:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b1b74a:::benchmark_data_smithi159_162467_object4974:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b2532a:::benchmark_data_smithi159_162467_object32913:head# 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b3c2a0:::benchmark_data_smithi159_162467_object59111:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b466a5:::benchmark_data_smithi159_162467_object67475:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b4d298:::benchmark_data_smithi159_162467_object67834:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b5399d:::benchmark_data_smithi159_162467_object41048:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b54861:::benchmark_data_smithi159_162467_object62100:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b67fc7:::benchmark_data_smithi159_162467_object39283:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b68a36:::benchmark_data_smithi159_162467_object3364:head# 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b8ca7f:::benchmark_data_smithi159_162467_object7716:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b8cc5f:::benchmark_data_smithi159_162467_object71032:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4b90748:::benchmark_data_smithi159_162467_object57558:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ba470e:::benchmark_data_smithi159_162467_object33600:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4bf372b:::benchmark_data_smithi159_162467_object20047:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c0f0af:::benchmark_data_smithi159_162467_object2117:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c16dd6:::benchmark_data_smithi159_162467_object6451:head# 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c1864d:::benchmark_data_smithi159_162467_object67247:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c29b60:::benchmark_data_smithi159_162467_object25249:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c3603d:::benchmark_data_smithi159_162467_object3454:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c428cb:::benchmark_data_smithi159_162467_object73292:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c42a1a:::benchmark_data_smithi159_162467_object34395:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c60c0e:::benchmark_data_smithi159_162467_object3488:head# 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c6a712:::benchmark_data_smithi159_162467_object7393:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c710b5:::benchmark_data_smithi159_162467_object42025:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c76255:::benchmark_data_smithi159_162467_object21762:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c7ae8c:::benchmark_data_smithi159_162467_object68277:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c81bfc:::benchmark_data_smithi159_162467_object70467:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c850d2:::benchmark_data_smithi159_162467_object47409:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c8e13d:::benchmark_data_smithi159_162467_object53352:head# 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c8f84d:::benchmark_data_smithi159_162467_object48395:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c97a8e:::benchmark_data_smithi159_162467_object871:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4c9fa60:::benchmark_data_smithi159_162467_object20575:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4cad362:::benchmark_data_smithi159_162467_object35076:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4cc77c0:::benchmark_data_smithi159_162467_object50455:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ce812c:::benchmark_data_smithi159_162467_object13495:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4cedbcf:::benchmark_data_smithi159_162467_object38304:head# 2024-04-15T18:43:14.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4cf7cd0:::benchmark_data_smithi159_162467_object7591:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d081e0:::benchmark_data_smithi159_162467_object13655:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d09f36:::benchmark_data_smithi159_162467_object15603:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d15576:::benchmark_data_smithi159_162467_object54699:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d1a772:::benchmark_data_smithi159_162467_object61738:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d24ff7:::benchmark_data_smithi159_162467_object40580:head# 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d3fb18:::benchmark_data_smithi159_162467_object14652:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d40b26:::benchmark_data_smithi159_162467_object33220:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d90bda:::benchmark_data_smithi159_162467_object34544:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d981a2:::benchmark_data_smithi159_162467_object24696:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4d9cf04:::benchmark_data_smithi159_162467_object52292:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dadcb8:::benchmark_data_smithi159_162467_object47301:head# 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dbee66:::benchmark_data_smithi159_162467_object16517:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dc0cf3:::benchmark_data_smithi159_162467_object51904:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4de7737:::benchmark_data_smithi159_162467_object35552:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dec761:::benchmark_data_smithi159_162467_object19149:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dee69e:::benchmark_data_smithi159_162467_object625:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4dfdc1f:::benchmark_data_smithi159_162467_object20009:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e2ba17:::benchmark_data_smithi159_162467_object68976:head# 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e41b8d:::benchmark_data_smithi159_162467_object39668:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e438cb:::benchmark_data_smithi159_162467_object47085:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e4a7df:::benchmark_data_smithi159_162467_object30639:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e4c2f3:::benchmark_data_smithi159_162467_object57144:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e54742:::benchmark_data_smithi159_162467_object34003:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e56f4e:::benchmark_data_smithi159_162467_object34345:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e67025:::benchmark_data_smithi159_162467_object8174:head# 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e75b7a:::benchmark_data_smithi159_162467_object24307:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e8e203:::benchmark_data_smithi159_162467_object13827:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4e99be3:::benchmark_data_smithi159_162467_object53640:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ea3397:::benchmark_data_smithi159_162467_object20496:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ea9a22:::benchmark_data_smithi159_162467_object4867:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ec6307:::benchmark_data_smithi159_162467_object12439:head# 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.265 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ed9235:::benchmark_data_smithi159_162467_object8929:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ed9f2b:::benchmark_data_smithi159_162467_object45561:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4edcc0b:::benchmark_data_smithi159_162467_object38537:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4eddc35:::benchmark_data_smithi159_162467_object30982:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4edf28d:::benchmark_data_smithi159_162467_object36206:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ee0125:::benchmark_data_smithi159_162467_object42153:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ee0943:::benchmark_data_smithi159_162467_object29093:head# 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.266 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ee69f0:::benchmark_data_smithi159_162467_object31359:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4ef441c:::benchmark_data_smithi159_162467_object15963:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4efe966:::benchmark_data_smithi159_162467_object64407:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f213f8:::benchmark_data_smithi159_162467_object14909:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f28ff6:::benchmark_data_smithi159_162467_object64272:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f341df:::benchmark_data_smithi159_162467_object67825:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f37744:::benchmark_data_smithi159_162467_object31352:head# 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.267 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f3cf94:::benchmark_data_smithi159_162467_object55036:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f53f4c:::benchmark_data_smithi159_162467_object57441:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f57aa1:::benchmark_data_smithi159_162467_object61136:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f5afd0:::benchmark_data_smithi159_162467_object27093:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f65817:::benchmark_data_smithi159_162467_object67097:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f816c8:::benchmark_data_smithi159_162467_object51003:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f8238d:::benchmark_data_smithi159_162467_object47469:head# 2024-04-15T18:43:14.268 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.269 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4f99ed4:::benchmark_data_smithi159_162467_object50234:head# 2024-04-15T18:43:14.269 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.269 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fa8fa0:::benchmark_data_smithi159_162467_object63046:head# 2024-04-15T18:43:14.269 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.297 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fa9306:::benchmark_data_smithi159_162467_object52434:head# 2024-04-15T18:43:14.297 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4faf3e6:::benchmark_data_smithi159_162467_object15313:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fb43e2:::benchmark_data_smithi159_162467_object3099:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fbdba9:::benchmark_data_smithi159_162467_object38268:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fbfe27:::benchmark_data_smithi159_162467_object23965:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fcd07c:::benchmark_data_smithi159_162467_object63099:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fcd4f9:::benchmark_data_smithi159_162467_object24719:head# 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.298 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fd6d99:::benchmark_data_smithi159_162467_object64966:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fe6207:::benchmark_data_smithi159_162467_object35986:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f4fee1c0:::benchmark_data_smithi159_162467_object17295:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50046ec:::benchmark_data_smithi159_162467_object28602:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5005df7:::benchmark_data_smithi159_162467_object71276:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5005dfd:::benchmark_data_smithi159_162467_object11429:head# 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.299 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5015527:::benchmark_data_smithi159_162467_object45692:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5018371:::benchmark_data_smithi159_162467_object59751:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5022b06:::benchmark_data_smithi159_162467_object43453:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5042e3d:::benchmark_data_smithi159_162467_object12037:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5051b5c:::benchmark_data_smithi159_162467_object65466:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f506f86a:::benchmark_data_smithi159_162467_object63742:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f507ff9d:::benchmark_data_smithi159_162467_object18259:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5081bd3:::benchmark_data_smithi159_162467_object57611:head# 2024-04-15T18:43:14.300 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50823e7:::benchmark_data_smithi159_162467_object64233:head# 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f509b5e1:::benchmark_data_smithi159_162467_object24573:head# 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50a8dcd:::benchmark_data_smithi159_162467_object15674:head# 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50b9631:::benchmark_data_smithi159_162467_object38703:head# 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.301 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50c7a94:::benchmark_data_smithi159_162467_object9089:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50d5b50:::benchmark_data_smithi159_162467_object47579:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50e412d:::benchmark_data_smithi159_162467_object39700:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50ebfd0:::benchmark_data_smithi159_162467_object32912:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50f10d4:::benchmark_data_smithi159_162467_object39978:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50f23c8:::benchmark_data_smithi159_162467_object9199:head# 2024-04-15T18:43:14.302 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50f68e7:::benchmark_data_smithi159_162467_object29085:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f50f751d:::benchmark_data_smithi159_162467_object20599:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51240a4:::benchmark_data_smithi159_162467_object53298:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5125942:::benchmark_data_smithi159_162467_object571:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51288df:::benchmark_data_smithi159_162467_object72421:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f512cfdb:::benchmark_data_smithi159_162467_object13482:head# 2024-04-15T18:43:14.303 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51463c1:::benchmark_data_smithi159_162467_object49634:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51587e4:::benchmark_data_smithi159_162467_object1307:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5167c3b:::benchmark_data_smithi159_162467_object32662:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f516e611:::benchmark_data_smithi159_162467_object31670:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5179827:::benchmark_data_smithi159_162467_object21334:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51a52a1:::benchmark_data_smithi159_162467_object73707:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51b68d5:::benchmark_data_smithi159_162467_object60633:head# 2024-04-15T18:43:14.304 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51c4a1c:::benchmark_data_smithi159_162467_object7743:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f51d1af7:::benchmark_data_smithi159_162467_object46181:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5208667:::benchmark_data_smithi159_162467_object12205:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f520fec6:::benchmark_data_smithi159_162467_object55599:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f521725b:::benchmark_data_smithi159_162467_object31729:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5219ab2:::benchmark_data_smithi159_162467_object66082:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f523dac5:::benchmark_data_smithi159_162467_object3858:head# 2024-04-15T18:43:14.305 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52411a0:::benchmark_data_smithi159_162467_object1213:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f524e7e3:::benchmark_data_smithi159_162467_object10753:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52671fd:::benchmark_data_smithi159_162467_object54755:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f526acdd:::benchmark_data_smithi159_162467_object64486:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f526e030:::benchmark_data_smithi159_162467_object10393:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f528068d:::benchmark_data_smithi159_162467_object20641:head# 2024-04-15T18:43:14.306 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52cee38:::benchmark_data_smithi159_162467_object22851:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52d2c3d:::benchmark_data_smithi159_162467_object59761:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52d82ab:::benchmark_data_smithi159_162467_object42799:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f52f35c0:::benchmark_data_smithi159_162467_object7776:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f530119b:::benchmark_data_smithi159_162467_object18157:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5318cfc:::benchmark_data_smithi159_162467_object5304:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f532a736:::benchmark_data_smithi159_162467_object21097:head# 2024-04-15T18:43:14.307 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f532f681:::benchmark_data_smithi159_162467_object10712:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5346198:::benchmark_data_smithi159_162467_object14694:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f534cb97:::benchmark_data_smithi159_162467_object71546:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f534fd1d:::benchmark_data_smithi159_162467_object68927:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5356cb2:::benchmark_data_smithi159_162467_object57401:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f535dd93:::benchmark_data_smithi159_162467_object21121:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5373bf3:::benchmark_data_smithi159_162467_object45529:head# 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.308 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53779cb:::benchmark_data_smithi159_162467_object38270:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f537d1a7:::benchmark_data_smithi159_162467_object42703:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53826e1:::benchmark_data_smithi159_162467_object17422:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5384ae7:::benchmark_data_smithi159_162467_object25375:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53863b4:::benchmark_data_smithi159_162467_object21472:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f538ce28:::benchmark_data_smithi159_162467_object17467:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f538e566:::benchmark_data_smithi159_162467_object72278:head# 2024-04-15T18:43:14.309 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53b04b1:::benchmark_data_smithi159_162467_object16352:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53b44e3:::benchmark_data_smithi159_162467_object24600:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53bce43:::benchmark_data_smithi159_162467_object67851:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53d586a:::benchmark_data_smithi159_162467_object55751:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53df78a:::benchmark_data_smithi159_162467_object27996:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53e90c2:::benchmark_data_smithi159_162467_object54450:head# 2024-04-15T18:43:14.310 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53f5dbc:::benchmark_data_smithi159_162467_object17050:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53fbde6:::benchmark_data_smithi159_162467_object72375:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f53fd481:::benchmark_data_smithi159_162467_object44271:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5401756:::benchmark_data_smithi159_162467_object13914:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f540c9fe:::benchmark_data_smithi159_162467_object19507:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5441d6e:::benchmark_data_smithi159_162467_object1524:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f544ab65:::benchmark_data_smithi159_162467_object65581:head# 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.311 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f546579a:::benchmark_data_smithi159_162467_object39048:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5479294:::benchmark_data_smithi159_162467_object6586:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f549997e:::benchmark_data_smithi159_162467_object74129:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54af2b0:::benchmark_data_smithi159_162467_object67240:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54b63c2:::benchmark_data_smithi159_162467_object29247:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54bd7f5:::benchmark_data_smithi159_162467_object18705:head# 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.312 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54ca04e:::benchmark_data_smithi159_162467_object32116:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54d2be4:::benchmark_data_smithi159_162467_object40518:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54da7e2:::benchmark_data_smithi159_162467_object37274:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54e3893:::benchmark_data_smithi159_162467_object33571:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f54f4d04:::benchmark_data_smithi159_162467_object45560:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f552a907:::benchmark_data_smithi159_162467_object25317:head# 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.313 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f552f17a:::benchmark_data_smithi159_162467_object44299:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5546e6e:::benchmark_data_smithi159_162467_object16505:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f554854d:::benchmark_data_smithi159_162467_object62705:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5555ec6:::benchmark_data_smithi159_162467_object43602:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5572e8f:::benchmark_data_smithi159_162467_object1565:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f557d318:::benchmark_data_smithi159_162467_object30022:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5587cbe:::benchmark_data_smithi159_162467_object59021:head# 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55a4e30:::benchmark_data_smithi159_162467_object61962:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55b241f:::benchmark_data_smithi159_162467_object17061:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55c859d:::benchmark_data_smithi159_162467_object20785:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55c9506:::benchmark_data_smithi159_162467_object2177:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55d9697:::benchmark_data_smithi159_162467_object47899:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f55ea8e0:::benchmark_data_smithi159_162467_object42412:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5600ec3:::benchmark_data_smithi159_162467_object41444:head# 2024-04-15T18:43:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5613027:::benchmark_data_smithi159_162467_object7421:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56142e2:::benchmark_data_smithi159_162467_object60168:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5619ab5:::benchmark_data_smithi159_162467_object4658:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5621427:::benchmark_data_smithi159_162467_object21845:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f562fe2f:::benchmark_data_smithi159_162467_object29419:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5653d28:::benchmark_data_smithi159_162467_object14631:head# 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.316 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f566bc2b:::benchmark_data_smithi159_162467_object13284:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5698eb5:::benchmark_data_smithi159_162467_object4590:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56a093f:::benchmark_data_smithi159_162467_object36364:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56b73b8:::benchmark_data_smithi159_162467_object18950:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56b8705:::benchmark_data_smithi159_162467_object10073:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56cf793:::benchmark_data_smithi159_162467_object74337:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56dbf6d:::benchmark_data_smithi159_162467_object12860:head# 2024-04-15T18:43:14.317 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56dfda2:::benchmark_data_smithi159_162467_object71182:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f56f9682:::benchmark_data_smithi159_162467_object10420:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f570cdfa:::benchmark_data_smithi159_162467_object21532:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f570d4b9:::benchmark_data_smithi159_162467_object26460:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f571ef39:::benchmark_data_smithi159_162467_object25590:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f571ff37:::benchmark_data_smithi159_162467_object29615:head# 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.318 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5721b8f:::benchmark_data_smithi159_162467_object20825:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f572a547:::benchmark_data_smithi159_162467_object58813:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f574ba1b:::benchmark_data_smithi159_162467_object52602:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5754806:::benchmark_data_smithi159_162467_object37075:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57575ba:::benchmark_data_smithi159_162467_object18345:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f575ce93:::benchmark_data_smithi159_162467_object14464:head# 2024-04-15T18:43:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57698b4:::benchmark_data_smithi159_162467_object7071:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5776c5d:::benchmark_data_smithi159_162467_object40382:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f579cbc4:::benchmark_data_smithi159_162467_object23399:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57a1a5e:::benchmark_data_smithi159_162467_object25315:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57c4c02:::benchmark_data_smithi159_162467_object12162:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57c5f62:::benchmark_data_smithi159_162467_object72035:head# 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.320 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57d0fd4:::benchmark_data_smithi159_162467_object66359:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57e8ec9:::benchmark_data_smithi159_162467_object39836:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57eca7d:::benchmark_data_smithi159_162467_object7061:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57fa332:::benchmark_data_smithi159_162467_object61246:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f57fcc5b:::benchmark_data_smithi159_162467_object28819:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f580af9c:::benchmark_data_smithi159_162467_object23191:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f580ca52:::benchmark_data_smithi159_162467_object21279:head# 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f580f3aa:::benchmark_data_smithi159_162467_object25733:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58158f0:::benchmark_data_smithi159_162467_object32345:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5825d9b:::benchmark_data_smithi159_162467_object41677:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58460cc:::benchmark_data_smithi159_162467_object56077:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5872bbc:::benchmark_data_smithi159_162467_object41673:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f587c23a:::benchmark_data_smithi159_162467_object748:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f588d925:::benchmark_data_smithi159_162467_object29522:head# 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58bfea5:::benchmark_data_smithi159_162467_object51435:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58d81f9:::benchmark_data_smithi159_162467_object10931:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58dcf6d:::benchmark_data_smithi159_162467_object71427:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58dfebe:::benchmark_data_smithi159_162467_object23421:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58e699b:::benchmark_data_smithi159_162467_object66782:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f58ea23d:::benchmark_data_smithi159_162467_object65574:head# 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59186a9:::benchmark_data_smithi159_162467_object28439:head# 2024-04-15T18:43:14.324 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.324 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f592433f:::benchmark_data_smithi159_162467_object30040:head# 2024-04-15T18:43:14.324 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5926bc8:::benchmark_data_smithi159_162467_object48624:head# 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59479f3:::benchmark_data_smithi159_162467_object56116:head# 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59652e8:::benchmark_data_smithi159_162467_object53603:head# 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5970e43:::benchmark_data_smithi159_162467_object24272:head# 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5983658:::benchmark_data_smithi159_162467_object68377:head# 2024-04-15T18:43:14.332 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5984eae:::benchmark_data_smithi159_162467_object45015:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f598c5f7:::benchmark_data_smithi159_162467_object37785:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59a0a65:::benchmark_data_smithi159_162467_object39870:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59a696d:::benchmark_data_smithi159_162467_object11590:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59b5a40:::benchmark_data_smithi159_162467_object42816:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59b9273:::benchmark_data_smithi159_162467_object63133:head# 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.333 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59df799:::benchmark_data_smithi159_162467_object61831:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59e2681:::benchmark_data_smithi159_162467_object6169:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f59f1072:::benchmark_data_smithi159_162467_object6114:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a0f33f:::benchmark_data_smithi159_162467_object16310:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a28e63:::benchmark_data_smithi159_162467_object48459:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a2b789:::benchmark_data_smithi159_162467_object65036:head# 2024-04-15T18:43:14.334 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a35dd7:::benchmark_data_smithi159_162467_object56924:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a3c3fe:::benchmark_data_smithi159_162467_object51365:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a3ebaf:::benchmark_data_smithi159_162467_object16059:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a4e3dd:::benchmark_data_smithi159_162467_object13396:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a67c3d:::benchmark_data_smithi159_162467_object1811:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a711d4:::benchmark_data_smithi159_162467_object36666:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a7132c:::benchmark_data_smithi159_162467_object33844:head# 2024-04-15T18:43:14.335 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a8040d:::benchmark_data_smithi159_162467_object31956:head# 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a84d8a:::benchmark_data_smithi159_162467_object28416:head# 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5a94fcd:::benchmark_data_smithi159_162467_object54492:head# 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5abf04c:::benchmark_data_smithi159_162467_object65995:head# 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ac54fc:::benchmark_data_smithi159_162467_object41403:head# 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.336 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ac69cf:::benchmark_data_smithi159_162467_object71725:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ac6e7c:::benchmark_data_smithi159_162467_object6111:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ac817f:::benchmark_data_smithi159_162467_object26066:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5acf62f:::benchmark_data_smithi159_162467_object31247:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ae33fb:::benchmark_data_smithi159_162467_object35936:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ae921f:::benchmark_data_smithi159_162467_object42051:head# 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.337 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5aff36b:::benchmark_data_smithi159_162467_object72266:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b14f14:::benchmark_data_smithi159_162467_object47273:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b15042:::benchmark_data_smithi159_162467_object60393:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b2ffa1:::benchmark_data_smithi159_162467_object8602:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b323dc:::benchmark_data_smithi159_162467_object68703:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b34e53:::benchmark_data_smithi159_162467_object36091:head# 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b4072a:::benchmark_data_smithi159_162467_object14978:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b507a4:::benchmark_data_smithi159_162467_object54361:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b5d3ee:::benchmark_data_smithi159_162467_object9273:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b678d1:::benchmark_data_smithi159_162467_object61766:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b7dd1a:::benchmark_data_smithi159_162467_object52776:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b8de08:::benchmark_data_smithi159_162467_object53830:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5b8e9bd:::benchmark_data_smithi159_162467_object15792:head# 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ba01ec:::benchmark_data_smithi159_162467_object44768:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ba0342:::benchmark_data_smithi159_162467_object54389:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ba0e1e:::benchmark_data_smithi159_162467_object38658:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ba54b0:::benchmark_data_smithi159_162467_object12250:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5bac263:::benchmark_data_smithi159_162467_object23701:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5bb797c:::benchmark_data_smithi159_162467_object59985:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5bbbab6:::benchmark_data_smithi159_162467_object53982:head# 2024-04-15T18:43:14.340 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5bc4d0c:::benchmark_data_smithi159_162467_object18783:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5bd1a01:::benchmark_data_smithi159_162467_object43774:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5be0f43:::benchmark_data_smithi159_162467_object22440:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5be7d10:::benchmark_data_smithi159_162467_object19266:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c03974:::benchmark_data_smithi159_162467_object12913:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c078aa:::benchmark_data_smithi159_162467_object66826:head# 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.341 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c16ad8:::benchmark_data_smithi159_162467_object69552:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c235ce:::benchmark_data_smithi159_162467_object56418:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c3ca0b:::benchmark_data_smithi159_162467_object27948:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c3d112:::benchmark_data_smithi159_162467_object646:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c5512b:::benchmark_data_smithi159_162467_object50891:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c5bb4e:::benchmark_data_smithi159_162467_object1758:head# 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.342 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c692c8:::benchmark_data_smithi159_162467_object37688:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c7d7c6:::benchmark_data_smithi159_162467_object69213:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c7f44d:::benchmark_data_smithi159_162467_object37803:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c800f5:::benchmark_data_smithi159_162467_object57038:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c8ad34:::benchmark_data_smithi159_162467_object64473:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5c9e90f:::benchmark_data_smithi159_162467_object29259:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5caf814:::benchmark_data_smithi159_162467_object37867:head# 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.343 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5cc3f6a:::benchmark_data_smithi159_162467_object58528:head# 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ccd464:::benchmark_data_smithi159_162467_object3422:head# 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ccf912:::benchmark_data_smithi159_162467_object47686:head# 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5cf2089:::benchmark_data_smithi159_162467_object58303:head# 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5cf993b:::benchmark_data_smithi159_162467_object69850:head# 2024-04-15T18:43:14.344 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5cfdeee:::benchmark_data_smithi159_162467_object40830:head# 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d00bd6:::benchmark_data_smithi159_162467_object60340:head# 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d0d1b8:::benchmark_data_smithi159_162467_object58247:head# 2024-04-15T18:43:14.345 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.353 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d33c56:::benchmark_data_smithi159_162467_object72106:head# 2024-04-15T18:43:14.353 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.353 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d35cdd:::benchmark_data_smithi159_162467_object17024:head# 2024-04-15T18:43:14.353 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.353 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d39fea:::benchmark_data_smithi159_162467_object2348:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d3b50d:::benchmark_data_smithi159_162467_object10496:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d3faf1:::benchmark_data_smithi159_162467_object74559:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d644dc:::benchmark_data_smithi159_162467_object32264:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d6cc3e:::benchmark_data_smithi159_162467_object29474:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d8ca03:::benchmark_data_smithi159_162467_object52751:head# 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.354 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5d9636a:::benchmark_data_smithi159_162467_object46299:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dbc50e:::benchmark_data_smithi159_162467_object66574:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dbc907:::benchmark_data_smithi159_162467_object71233:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dbf6ec:::benchmark_data_smithi159_162467_object32978:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dd96b7:::benchmark_data_smithi159_162467_object37027:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dfe3b7:::benchmark_data_smithi159_162467_object22103:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5dff6cb:::benchmark_data_smithi159_162467_object66026:head# 2024-04-15T18:43:14.355 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e06e0e:::benchmark_data_smithi159_162467_object57999:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e0cfa1:::benchmark_data_smithi159_162467_object53855:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e0e9db:::benchmark_data_smithi159_162467_object5145:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e121a0:::benchmark_data_smithi159_162467_object40539:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e14c41:::benchmark_data_smithi159_162467_object15105:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e22bd8:::benchmark_data_smithi159_162467_object41752:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e2ed78:::benchmark_data_smithi159_162467_object67491:head# 2024-04-15T18:43:14.356 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e2f950:::benchmark_data_smithi159_162467_object45745:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e356ac:::benchmark_data_smithi159_162467_object30416:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e5cb1d:::benchmark_data_smithi159_162467_object52543:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e66df2:::benchmark_data_smithi159_162467_object48492:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e680bd:::benchmark_data_smithi159_162467_object50338:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e6a72b:::benchmark_data_smithi159_162467_object9706:head# 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.357 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e80225:::benchmark_data_smithi159_162467_object15806:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e952b0:::benchmark_data_smithi159_162467_object8284:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e99231:::benchmark_data_smithi159_162467_object15936:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e9c561:::benchmark_data_smithi159_162467_object50348:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5e9e68f:::benchmark_data_smithi159_162467_object11102:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ea67e8:::benchmark_data_smithi159_162467_object33652:head# 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.358 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ec083f:::benchmark_data_smithi159_162467_object57237:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ec122f:::benchmark_data_smithi159_162467_object64559:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5eec1f6:::benchmark_data_smithi159_162467_object25643:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f0c134:::benchmark_data_smithi159_162467_object18117:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f1420d:::benchmark_data_smithi159_162467_object21906:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f1b343:::benchmark_data_smithi159_162467_object26065:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f235f3:::benchmark_data_smithi159_162467_object28207:head# 2024-04-15T18:43:14.359 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f3d210:::benchmark_data_smithi159_162467_object63693:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f47816:::benchmark_data_smithi159_162467_object51953:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f5b73b:::benchmark_data_smithi159_162467_object47062:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f60001:::benchmark_data_smithi159_162467_object67755:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f8ae03:::benchmark_data_smithi159_162467_object3439:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f91cac:::benchmark_data_smithi159_162467_object74193:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5f94d8b:::benchmark_data_smithi159_162467_object1327:head# 2024-04-15T18:43:14.360 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fa86b0:::benchmark_data_smithi159_162467_object31940:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fa96be:::benchmark_data_smithi159_162467_object28713:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5facc3a:::benchmark_data_smithi159_162467_object74188:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5faf206:::benchmark_data_smithi159_162467_object19363:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fc98b8:::benchmark_data_smithi159_162467_object35688:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fd2421:::benchmark_data_smithi159_162467_object16259:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fd9f02:::benchmark_data_smithi159_162467_object6394:head# 2024-04-15T18:43:14.361 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fe0fcf:::benchmark_data_smithi159_162467_object60977:head# 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fe307b:::benchmark_data_smithi159_162467_object65844:head# 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5fe532a:::benchmark_data_smithi159_162467_object6680:head# 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ff4df1:::benchmark_data_smithi159_162467_object759:head# 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f5ff7214:::benchmark_data_smithi159_162467_object11856:head# 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6007742:::benchmark_data_smithi159_162467_object35718:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f601670f:::benchmark_data_smithi159_162467_object14819:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6029988:::benchmark_data_smithi159_162467_object47305:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f602cfe4:::benchmark_data_smithi159_162467_object66891:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f603688e:::benchmark_data_smithi159_162467_object63252:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6037a18:::benchmark_data_smithi159_162467_object52167:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f603e644:::benchmark_data_smithi159_162467_object405:head# 2024-04-15T18:43:14.363 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f605e199:::benchmark_data_smithi159_162467_object47193:head# 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60632ec:::benchmark_data_smithi159_162467_object68301:head# 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f606869d:::benchmark_data_smithi159_162467_object17608:head# 2024-04-15T18:43:14.372 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6069be4:::benchmark_data_smithi159_162467_object69931:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f606adf8:::benchmark_data_smithi159_162467_object48093:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f608c35c:::benchmark_data_smithi159_162467_object69207:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f609bc74:::benchmark_data_smithi159_162467_object7775:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f609ce6f:::benchmark_data_smithi159_162467_object11663:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60a32fa:::benchmark_data_smithi159_162467_object15255:head# 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.373 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60ad938:::benchmark_data_smithi159_162467_object55378:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60e4c49:::benchmark_data_smithi159_162467_object10097:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60e9758:::benchmark_data_smithi159_162467_object35136:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f60f3cd3:::benchmark_data_smithi159_162467_object27305:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f610c24b:::benchmark_data_smithi159_162467_object26621:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f610d8e2:::benchmark_data_smithi159_162467_object27918:head# 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.374 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f611132b:::benchmark_data_smithi159_162467_object20889:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6112579:::benchmark_data_smithi159_162467_object10222:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f611333a:::benchmark_data_smithi159_162467_object67128:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f612a827:::benchmark_data_smithi159_162467_object64661:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6131b6c:::benchmark_data_smithi159_162467_object53939:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f616640b:::benchmark_data_smithi159_162467_object39654:head# 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.375 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f616a10f:::benchmark_data_smithi159_162467_object61018:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f616b2fb:::benchmark_data_smithi159_162467_object9744:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f616b5b8:::benchmark_data_smithi159_162467_object63819:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6173690:::benchmark_data_smithi159_162467_object13582:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6185fd5:::benchmark_data_smithi159_162467_object55986:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6189ba1:::benchmark_data_smithi159_162467_object21300:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f618d623:::benchmark_data_smithi159_162467_object50546:head# 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.376 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f618f4de:::benchmark_data_smithi159_162467_object26725:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f619c877:::benchmark_data_smithi159_162467_object5244:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f619ca98:::benchmark_data_smithi159_162467_object11174:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61a2c64:::benchmark_data_smithi159_162467_object6168:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61ac4e4:::benchmark_data_smithi159_162467_object66023:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61c3215:::benchmark_data_smithi159_162467_object21735:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61d116b:::benchmark_data_smithi159_162467_object63766:head# 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.377 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61d48ea:::benchmark_data_smithi159_162467_object12233:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61e6b1b:::benchmark_data_smithi159_162467_object19661:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61ec8cf:::benchmark_data_smithi159_162467_object66201:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f61fcd44:::benchmark_data_smithi159_162467_object31467:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62140d2:::benchmark_data_smithi159_162467_object22735:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f621b731:::benchmark_data_smithi159_162467_object46695:head# 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.378 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6224d3e:::benchmark_data_smithi159_162467_object24837:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62299b0:::benchmark_data_smithi159_162467_object24414:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6232372:::benchmark_data_smithi159_162467_object39667:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6237912:::benchmark_data_smithi159_162467_object26535:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f624cede:::benchmark_data_smithi159_162467_object26436:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6268f7f:::benchmark_data_smithi159_162467_object47486:head# 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.379 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f626e051:::benchmark_data_smithi159_162467_object53592:head# 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6270cb7:::benchmark_data_smithi159_162467_object40937:head# 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62794ce:::benchmark_data_smithi159_162467_object62957:head# 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.380 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f628a4e8:::benchmark_data_smithi159_162467_object5416:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f629bfde:::benchmark_data_smithi159_162467_object45844:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62b554a:::benchmark_data_smithi159_162467_object26622:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62d3ccc:::benchmark_data_smithi159_162467_object49707:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62e5d8e:::benchmark_data_smithi159_162467_object32682:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f62fd572:::benchmark_data_smithi159_162467_object16314:head# 2024-04-15T18:43:14.381 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6303e2f:::benchmark_data_smithi159_162467_object63698:head# 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f630a612:::benchmark_data_smithi159_162467_object26939:head# 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6327559:::benchmark_data_smithi159_162467_object12949:head# 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f632ca26:::benchmark_data_smithi159_162467_object53817:head# 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f633162a:::benchmark_data_smithi159_162467_object60658:head# 2024-04-15T18:43:14.382 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.399 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6337e0d:::benchmark_data_smithi159_162467_object63611:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6351af2:::benchmark_data_smithi159_162467_object66599:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6354520:::benchmark_data_smithi159_162467_object39713:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63652a2:::benchmark_data_smithi159_162467_object25338:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63685a0:::benchmark_data_smithi159_162467_object23340:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f639c6d9:::benchmark_data_smithi159_162467_object18621:head# 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.400 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63b97c2:::benchmark_data_smithi159_162467_object6615:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63be20e:::benchmark_data_smithi159_162467_object27241:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63d4d15:::benchmark_data_smithi159_162467_object47078:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63d9185:::benchmark_data_smithi159_162467_object61:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63e7fe6:::benchmark_data_smithi159_162467_object54052:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63f1ec3:::benchmark_data_smithi159_162467_object5312:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f63f808c:::benchmark_data_smithi159_162467_object23885:head# 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.401 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64069d6:::benchmark_data_smithi159_162467_object351:head# 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f640e920:::benchmark_data_smithi159_162467_object47685:head# 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f641e61e:::benchmark_data_smithi159_162467_object16949:head# 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6427e53:::benchmark_data_smithi159_162467_object23879:head# 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f642bd57:::benchmark_data_smithi159_162467_object6831:head# 2024-04-15T18:43:14.402 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f646ac0f:::benchmark_data_smithi159_162467_object7648:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f646e53a:::benchmark_data_smithi159_162467_object60100:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6470806:::benchmark_data_smithi159_162467_object56829:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6477025:::benchmark_data_smithi159_162467_object15098:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f647947e:::benchmark_data_smithi159_162467_object24309:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f647b167:::benchmark_data_smithi159_162467_object67386:head# 2024-04-15T18:43:14.403 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f649765b:::benchmark_data_smithi159_162467_object72450:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6497dac:::benchmark_data_smithi159_162467_object21603:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64a8fd1:::benchmark_data_smithi159_162467_object40169:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64ab3f0:::benchmark_data_smithi159_162467_object44509:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64b62f9:::benchmark_data_smithi159_162467_object63592:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64bef0b:::benchmark_data_smithi159_162467_object73249:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64ce5ab:::benchmark_data_smithi159_162467_object73217:head# 2024-04-15T18:43:14.404 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64e6dca:::benchmark_data_smithi159_162467_object12470:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f64f7183:::benchmark_data_smithi159_162467_object65637:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6505646:::benchmark_data_smithi159_162467_object8666:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f651b6d9:::benchmark_data_smithi159_162467_object59671:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f651dac0:::benchmark_data_smithi159_162467_object23856:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f653c3cb:::benchmark_data_smithi159_162467_object16898:head# 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.405 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f654e7d5:::benchmark_data_smithi159_162467_object45546:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6553378:::benchmark_data_smithi159_162467_object30759:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f655a9ec:::benchmark_data_smithi159_162467_object900:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f656eaaa:::benchmark_data_smithi159_162467_object27522:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65797a5:::benchmark_data_smithi159_162467_object11885:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65806bd:::benchmark_data_smithi159_162467_object44613:head# 2024-04-15T18:43:14.406 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f658cd1c:::benchmark_data_smithi159_162467_object70505:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6594f8a:::benchmark_data_smithi159_162467_object34767:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6598efa:::benchmark_data_smithi159_162467_object48725:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65e3c03:::benchmark_data_smithi159_162467_object53325:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65ed08a:::benchmark_data_smithi159_162467_object46284:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65f78e8:::benchmark_data_smithi159_162467_object9751:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f65fc9d8:::benchmark_data_smithi159_162467_object17362:head# 2024-04-15T18:43:14.407 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f660cda2:::benchmark_data_smithi159_162467_object58763:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6614fd6:::benchmark_data_smithi159_162467_object23103:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6615a50:::benchmark_data_smithi159_162467_object18663:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6618a5e:::benchmark_data_smithi159_162467_object69233:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6635208:::benchmark_data_smithi159_162467_object62757:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f664bee5:::benchmark_data_smithi159_162467_object11944:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f665ff3e:::benchmark_data_smithi159_162467_object46829:head# 2024-04-15T18:43:14.408 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6664146:::benchmark_data_smithi159_162467_object68948:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f666f629:::benchmark_data_smithi159_162467_object38137:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f66918e7:::benchmark_data_smithi159_162467_object68726:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6693ec7:::benchmark_data_smithi159_162467_object33362:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f669b62b:::benchmark_data_smithi159_162467_object67180:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f669c35b:::benchmark_data_smithi159_162467_object73577:head# 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.409 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f66cbb60:::benchmark_data_smithi159_162467_object19554:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f66daed3:::benchmark_data_smithi159_162467_object29070:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f66e2da7:::benchmark_data_smithi159_162467_object14963:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6718f66:::benchmark_data_smithi159_162467_object39387:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f672771f:::benchmark_data_smithi159_162467_object23003:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67342ac:::benchmark_data_smithi159_162467_object3205:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67404c0:::benchmark_data_smithi159_162467_object64375:head# 2024-04-15T18:43:14.410 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67411d4:::benchmark_data_smithi159_162467_object65495:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6747f78:::benchmark_data_smithi159_162467_object24027:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6752c10:::benchmark_data_smithi159_162467_object4853:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f675fb55:::benchmark_data_smithi159_162467_object31206:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6764664:::benchmark_data_smithi159_162467_object39901:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f676c57c:::benchmark_data_smithi159_162467_object38115:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6770898:::benchmark_data_smithi159_162467_object65178:head# 2024-04-15T18:43:14.411 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6777d88:::benchmark_data_smithi159_162467_object6611:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f678c3f3:::benchmark_data_smithi159_162467_object51593:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6791bac:::benchmark_data_smithi159_162467_object19032:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67a1476:::benchmark_data_smithi159_162467_object29335:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67a2126:::benchmark_data_smithi159_162467_object71412:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67a83af:::benchmark_data_smithi159_162467_object45621:head# 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.412 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67ca428:::benchmark_data_smithi159_162467_object55372:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67cb6a8:::benchmark_data_smithi159_162467_object19315:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67cbec2:::benchmark_data_smithi159_162467_object48580:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67ce64d:::benchmark_data_smithi159_162467_object40282:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67ceb1b:::benchmark_data_smithi159_162467_object57507:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67d01d3:::benchmark_data_smithi159_162467_object34616:head# 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.413 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67dd41f:::benchmark_data_smithi159_162467_object50149:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67e21a2:::benchmark_data_smithi159_162467_object17311:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67e60ba:::benchmark_data_smithi159_162467_object27179:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67f0fa8:::benchmark_data_smithi159_162467_object61544:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f67f9e4f:::benchmark_data_smithi159_162467_object54768:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6800a35:::benchmark_data_smithi159_162467_object66799:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f680acfa:::benchmark_data_smithi159_162467_object66636:head# 2024-04-15T18:43:14.414 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f680b224:::benchmark_data_smithi159_162467_object29490:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68140af:::benchmark_data_smithi159_162467_object54071:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6818739:::benchmark_data_smithi159_162467_object40397:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6822ddf:::benchmark_data_smithi159_162467_object28402:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f682a366:::benchmark_data_smithi159_162467_object74279:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6854d67:::benchmark_data_smithi159_162467_object56953:head# 2024-04-15T18:43:14.415 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6863ac5:::benchmark_data_smithi159_162467_object70595:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6871fb4:::benchmark_data_smithi159_162467_object74602:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68721e5:::benchmark_data_smithi159_162467_object71709:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f687555c:::benchmark_data_smithi159_162467_object2986:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68783fa:::benchmark_data_smithi159_162467_object36686:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f687f539:::benchmark_data_smithi159_162467_object44159:head# 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.416 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f688eef7:::benchmark_data_smithi159_162467_object45847:head# 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f689a143:::benchmark_data_smithi159_162467_object13271:head# 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68a0133:::benchmark_data_smithi159_162467_object53559:head# 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68a1dcf:::benchmark_data_smithi159_162467_object23525:head# 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68cc2f3:::benchmark_data_smithi159_162467_object3925:head# 2024-04-15T18:43:14.417 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68d545a:::benchmark_data_smithi159_162467_object521:head# 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68ed53d:::benchmark_data_smithi159_162467_object9309:head# 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68f1536:::benchmark_data_smithi159_162467_object74066:head# 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68ff627:::benchmark_data_smithi159_162467_object9924:head# 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.437 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f68ffed9:::benchmark_data_smithi159_162467_object3012:head# 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69222fe:::benchmark_data_smithi159_162467_object46072:head# 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69299d2:::benchmark_data_smithi159_162467_object28279:head# 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f692e0d8:::benchmark_data_smithi159_162467_object39217:head# 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6938252:::benchmark_data_smithi159_162467_object49461:head# 2024-04-15T18:43:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6954c50:::benchmark_data_smithi159_162467_object3872:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69561de:::benchmark_data_smithi159_162467_object56220:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f695cc3f:::benchmark_data_smithi159_162467_object27706:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6982ff2:::benchmark_data_smithi159_162467_object46022:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f698f2e8:::benchmark_data_smithi159_162467_object17553:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f699f4cd:::benchmark_data_smithi159_162467_object71730:head# 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.439 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69a2f73:::benchmark_data_smithi159_162467_object47614:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69abc42:::benchmark_data_smithi159_162467_object32989:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69ae747:::benchmark_data_smithi159_162467_object50562:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69be367:::benchmark_data_smithi159_162467_object41965:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69c6fd6:::benchmark_data_smithi159_162467_object2132:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69e79c1:::benchmark_data_smithi159_162467_object73425:head# 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.440 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f69fc67f:::benchmark_data_smithi159_162467_object17104:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a0e08b:::benchmark_data_smithi159_162467_object44746:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a20b2e:::benchmark_data_smithi159_162467_object42692:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a2c83f:::benchmark_data_smithi159_162467_object33690:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a44d1b:::benchmark_data_smithi159_162467_object53188:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a62fed:::benchmark_data_smithi159_162467_object68724:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a69b60:::benchmark_data_smithi159_162467_object28915:head# 2024-04-15T18:43:14.441 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a6bb41:::benchmark_data_smithi159_162467_object45756:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a74896:::benchmark_data_smithi159_162467_object12733:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a7ed89:::benchmark_data_smithi159_162467_object48827:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a98bbe:::benchmark_data_smithi159_162467_object27651:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6a9f3c0:::benchmark_data_smithi159_162467_object14073:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ab721d:::benchmark_data_smithi159_162467_object22310:head# 2024-04-15T18:43:14.442 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ac8b24:::benchmark_data_smithi159_162467_object50147:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ad28c9:::benchmark_data_smithi159_162467_object58742:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ad5529:::benchmark_data_smithi159_162467_object16461:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6adaf6a:::benchmark_data_smithi159_162467_object63701:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6adc81a:::benchmark_data_smithi159_162467_object39204:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6add549:::benchmark_data_smithi159_162467_object41732:head# 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.443 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ae16db:::benchmark_data_smithi159_162467_object30246:head# 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ae4e1e:::benchmark_data_smithi159_162467_object14714:head# 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6af9b43:::benchmark_data_smithi159_162467_object69581:head# 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b06c6b:::benchmark_data_smithi159_162467_object7044:head# 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.444 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b0c902:::benchmark_data_smithi159_162467_object5854:head# 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b285ae:::benchmark_data_smithi159_162467_object20149:head# 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b37d44:::benchmark_data_smithi159_162467_object17493:head# 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b4172c:::benchmark_data_smithi159_162467_object61085:head# 2024-04-15T18:43:14.445 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b6a845:::benchmark_data_smithi159_162467_object55347:head# 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b8309e:::benchmark_data_smithi159_162467_object44205:head# 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b8e2d6:::benchmark_data_smithi159_162467_object64202:head# 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6b940fc:::benchmark_data_smithi159_162467_object8154:head# 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bb6918:::benchmark_data_smithi159_162467_object3416:head# 2024-04-15T18:43:14.446 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bc31db:::benchmark_data_smithi159_162467_object55864:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bc759c:::benchmark_data_smithi159_162467_object54434:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bcba9f:::benchmark_data_smithi159_162467_object60369:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bd4f70:::benchmark_data_smithi159_162467_object11760:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6bddeb7:::benchmark_data_smithi159_162467_object56254:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c0d514:::benchmark_data_smithi159_162467_object28580:head# 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.447 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c32efc:::benchmark_data_smithi159_162467_object69740:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c4e65d:::benchmark_data_smithi159_162467_object49330:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c5b97a:::benchmark_data_smithi159_162467_object57714:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c6a392:::benchmark_data_smithi159_162467_object71687:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c73d70:::benchmark_data_smithi159_162467_object31216:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c8886e:::benchmark_data_smithi159_162467_object12490:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c892ce:::benchmark_data_smithi159_162467_object2201:head# 2024-04-15T18:43:14.448 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c8e0f5:::benchmark_data_smithi159_162467_object54892:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c90454:::benchmark_data_smithi159_162467_object33818:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6c925c0:::benchmark_data_smithi159_162467_object464:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6cbf03a:::benchmark_data_smithi159_162467_object69920:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6cd8404:::benchmark_data_smithi159_162467_object29754:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6cdd3e1:::benchmark_data_smithi159_162467_object72117:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ce1191:::benchmark_data_smithi159_162467_object8900:head# 2024-04-15T18:43:14.449 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ce65cd:::benchmark_data_smithi159_162467_object64904:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6cfcb7d:::benchmark_data_smithi159_162467_object35740:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d04383:::benchmark_data_smithi159_162467_object74744:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d0ef25:::benchmark_data_smithi159_162467_object40917:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d15394:::benchmark_data_smithi159_162467_object1353:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d184f3:::benchmark_data_smithi159_162467_object74238:head# 2024-04-15T18:43:14.450 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d25607:::benchmark_data_smithi159_162467_object44481:head# 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d257e4:::benchmark_data_smithi159_162467_object4363:head# 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d27f56:::benchmark_data_smithi159_162467_object48622:head# 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d29367:::benchmark_data_smithi159_162467_object11149:head# 2024-04-15T18:43:14.451 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d3897e:::benchmark_data_smithi159_162467_object42172:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d45052:::benchmark_data_smithi159_162467_object32136:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d5d481:::benchmark_data_smithi159_162467_object67751:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d6d039:::benchmark_data_smithi159_162467_object23825:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d78b40:::benchmark_data_smithi159_162467_object37591:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d79224:::benchmark_data_smithi159_162467_object5553:head# 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.452 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d8173d:::benchmark_data_smithi159_162467_object71872:head# 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d8b2b0:::benchmark_data_smithi159_162467_object50618:head# 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d95ce5:::benchmark_data_smithi159_162467_object68733:head# 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.453 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6d9a8dc:::benchmark_data_smithi159_162467_object62015:head# 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6da0b0d:::benchmark_data_smithi159_162467_object3474:head# 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6db668c:::benchmark_data_smithi159_162467_object22337:head# 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6db992d:::benchmark_data_smithi159_162467_object28736:head# 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6dbca0c:::benchmark_data_smithi159_162467_object35512:head# 2024-04-15T18:43:14.454 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6dc7824:::benchmark_data_smithi159_162467_object21840:head# 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6dd7f7f:::benchmark_data_smithi159_162467_object72901:head# 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ddbe39:::benchmark_data_smithi159_162467_object67691:head# 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.455 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6de8927:::benchmark_data_smithi159_162467_object33442:head# 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6e30a79:::benchmark_data_smithi159_162467_object14430:head# 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6e34918:::benchmark_data_smithi159_162467_object60848:head# 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6e6c7a0:::benchmark_data_smithi159_162467_object18554:head# 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.456 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6e8773c:::benchmark_data_smithi159_162467_object69347:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6e89137:::benchmark_data_smithi159_162467_object66332:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ea1984:::benchmark_data_smithi159_162467_object28968:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ece448:::benchmark_data_smithi159_162467_object19860:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f28ef3:::benchmark_data_smithi159_162467_object22655:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f41838:::benchmark_data_smithi159_162467_object51005:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f542ea:::benchmark_data_smithi159_162467_object17113:head# 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.457 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f5c438:::benchmark_data_smithi159_162467_object1677:head# 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f5d5b6:::benchmark_data_smithi159_162467_object4910:head# 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f6726c:::benchmark_data_smithi159_162467_object61960:head# 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f70dfc:::benchmark_data_smithi159_162467_object12369:head# 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f7fa74:::benchmark_data_smithi159_162467_object8357:head# 2024-04-15T18:43:14.458 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f94ba7:::benchmark_data_smithi159_162467_object31308:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6f96ffa:::benchmark_data_smithi159_162467_object66779:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fb5e36:::benchmark_data_smithi159_162467_object50219:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fb694d:::benchmark_data_smithi159_162467_object17448:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fb9dde:::benchmark_data_smithi159_162467_object72832:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fca01e:::benchmark_data_smithi159_162467_object31725:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fd037b:::benchmark_data_smithi159_162467_object28262:head# 2024-04-15T18:43:14.459 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fe32ea:::benchmark_data_smithi159_162467_object40127:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fe477e:::benchmark_data_smithi159_162467_object20440:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6fef370:::benchmark_data_smithi159_162467_object45865:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f6ff071f:::benchmark_data_smithi159_162467_object68388:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7004f70:::benchmark_data_smithi159_162467_object2653:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f700c0b9:::benchmark_data_smithi159_162467_object54631:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f700d268:::benchmark_data_smithi159_162467_object20976:head# 2024-04-15T18:43:14.460 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7036d06:::benchmark_data_smithi159_162467_object57236:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7040744:::benchmark_data_smithi159_162467_object8651:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7057f0a:::benchmark_data_smithi159_162467_object50177:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f706227a:::benchmark_data_smithi159_162467_object27946:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f706b8e8:::benchmark_data_smithi159_162467_object27201:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7074187:::benchmark_data_smithi159_162467_object1042:head# 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.461 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f70752a4:::benchmark_data_smithi159_162467_object68575:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7077d6c:::benchmark_data_smithi159_162467_object24154:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f708597d:::benchmark_data_smithi159_162467_object28609:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f708e9d5:::benchmark_data_smithi159_162467_object65025:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f709e217:::benchmark_data_smithi159_162467_object50345:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f70ddf16:::benchmark_data_smithi159_162467_object22543:head# 2024-04-15T18:43:14.462 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f70e5f2e:::benchmark_data_smithi159_162467_object42312:head# 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f70f2f72:::benchmark_data_smithi159_162467_object73360:head# 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f70fca66:::benchmark_data_smithi159_162467_object47615:head# 2024-04-15T18:43:14.463 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.487 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f710eb73:::benchmark_data_smithi159_162467_object13324:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7125204:::benchmark_data_smithi159_162467_object8397:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f712593f:::benchmark_data_smithi159_162467_object17889:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71331a1:::benchmark_data_smithi159_162467_object8385:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f713ecb2:::benchmark_data_smithi159_162467_object7665:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7163c4e:::benchmark_data_smithi159_162467_object61227:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f716f775:::benchmark_data_smithi159_162467_object47399:head# 2024-04-15T18:43:14.488 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7175c4e:::benchmark_data_smithi159_162467_object45662:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7190be7:::benchmark_data_smithi159_162467_object69005:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7191e5b:::benchmark_data_smithi159_162467_object19482:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f719668a:::benchmark_data_smithi159_162467_object35992:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71b1ebb:::benchmark_data_smithi159_162467_object59731:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71d621d:::benchmark_data_smithi159_162467_object22029:head# 2024-04-15T18:43:14.489 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71d7b56:::benchmark_data_smithi159_162467_object59564:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71dbb23:::benchmark_data_smithi159_162467_object15497:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71ddfe8:::benchmark_data_smithi159_162467_object63681:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71edfa1:::benchmark_data_smithi159_162467_object33862:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71f085b:::benchmark_data_smithi159_162467_object7172:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71f534d:::benchmark_data_smithi159_162467_object58063:head# 2024-04-15T18:43:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71fa6b1:::benchmark_data_smithi159_162467_object40298:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f71fce7b:::benchmark_data_smithi159_162467_object1525:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7211cfe:::benchmark_data_smithi159_162467_object45414:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7213c24:::benchmark_data_smithi159_162467_object18690:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7214563:::benchmark_data_smithi159_162467_object36590:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f721979a:::benchmark_data_smithi159_162467_object69764:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7228253:::benchmark_data_smithi159_162467_object18353:head# 2024-04-15T18:43:14.491 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.492 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f722af15:::benchmark_data_smithi159_162467_object61366:head# 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f722eb0d:::benchmark_data_smithi159_162467_object25584:head# 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7231b7b:::benchmark_data_smithi159_162467_object16983:head# 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7234e6b:::benchmark_data_smithi159_162467_object66172:head# 2024-04-15T18:43:14.493 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f723ce06:::benchmark_data_smithi159_162467_object42491:head# 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f724d3e9:::benchmark_data_smithi159_162467_object55005:head# 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f727341e:::benchmark_data_smithi159_162467_object71623:head# 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f728828d:::benchmark_data_smithi159_162467_object16767:head# 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f728b3c7:::benchmark_data_smithi159_162467_object29219:head# 2024-04-15T18:43:14.494 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72b4c8b:::benchmark_data_smithi159_162467_object36725:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72c1aff:::benchmark_data_smithi159_162467_object2197:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72c31a0:::benchmark_data_smithi159_162467_object17220:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72e2811:::benchmark_data_smithi159_162467_object19757:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72e49c0:::benchmark_data_smithi159_162467_object14192:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72e66fb:::benchmark_data_smithi159_162467_object31565:head# 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.495 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f72f4d94:::benchmark_data_smithi159_162467_object58946:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7302337:::benchmark_data_smithi159_162467_object57117:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f730e2ae:::benchmark_data_smithi159_162467_object50510:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7346796:::benchmark_data_smithi159_162467_object12295:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f734b245:::benchmark_data_smithi159_162467_object39633:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73527f4:::benchmark_data_smithi159_162467_object7392:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f735ea82:::benchmark_data_smithi159_162467_object69455:head# 2024-04-15T18:43:14.496 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7378151:::benchmark_data_smithi159_162467_object30967:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7394440:::benchmark_data_smithi159_162467_object46951:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7397319:::benchmark_data_smithi159_162467_object62253:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73dc20e:::benchmark_data_smithi159_162467_object31033:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73decc4:::benchmark_data_smithi159_162467_object30641:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73e4145:::benchmark_data_smithi159_162467_object38353:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73eb084:::benchmark_data_smithi159_162467_object57824:head# 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.497 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73ef3b7:::benchmark_data_smithi159_162467_object35402:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f73fb50c:::benchmark_data_smithi159_162467_object23894:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74027af:::benchmark_data_smithi159_162467_object69721:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74115c8:::benchmark_data_smithi159_162467_object65459:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7411641:::benchmark_data_smithi159_162467_object41784:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f741967a:::benchmark_data_smithi159_162467_object39153:head# 2024-04-15T18:43:14.498 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f741e07e:::benchmark_data_smithi159_162467_object48731:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7434145:::benchmark_data_smithi159_162467_object26424:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f743420f:::benchmark_data_smithi159_162467_object63144:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7435527:::benchmark_data_smithi159_162467_object52765:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f743697e:::benchmark_data_smithi159_162467_object6172:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74448d2:::benchmark_data_smithi159_162467_object52309:head# 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.499 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74453a7:::benchmark_data_smithi159_162467_object13033:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f744a6ab:::benchmark_data_smithi159_162467_object9125:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f744b97b:::benchmark_data_smithi159_162467_object29817:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7451446:::benchmark_data_smithi159_162467_object830:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7451c30:::benchmark_data_smithi159_162467_object21987:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f747a739:::benchmark_data_smithi159_162467_object41196:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f747aa1e:::benchmark_data_smithi159_162467_object1850:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7487d9f:::benchmark_data_smithi159_162467_object29092:head# 2024-04-15T18:43:14.500 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74a57fe:::benchmark_data_smithi159_162467_object42965:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74b33a8:::benchmark_data_smithi159_162467_object14853:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74bab51:::benchmark_data_smithi159_162467_object71677:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74cc811:::benchmark_data_smithi159_162467_object48376:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74d9cb5:::benchmark_data_smithi159_162467_object32410:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74da7cc:::benchmark_data_smithi159_162467_object26253:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74dbde1:::benchmark_data_smithi159_162467_object52303:head# 2024-04-15T18:43:14.501 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74e4b0d:::benchmark_data_smithi159_162467_object9478:head# 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f74f8f28:::benchmark_data_smithi159_162467_object65329:head# 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7500736:::benchmark_data_smithi159_162467_object3874:head# 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7505699:::benchmark_data_smithi159_162467_object38884:head# 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7506d21:::benchmark_data_smithi159_162467_object12770:head# 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.502 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f750a90b:::benchmark_data_smithi159_162467_object45978:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f751cc19:::benchmark_data_smithi159_162467_object21002:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7522a9a:::benchmark_data_smithi159_162467_object36180:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f752e33e:::benchmark_data_smithi159_162467_object1619:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7556ca1:::benchmark_data_smithi159_162467_object4268:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f755ec0f:::benchmark_data_smithi159_162467_object29024:head# 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.503 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7568856:::benchmark_data_smithi159_162467_object26019:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7584ea3:::benchmark_data_smithi159_162467_object35379:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f758cfa5:::benchmark_data_smithi159_162467_object29526:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f758fbbd:::benchmark_data_smithi159_162467_object15096:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75957a6:::benchmark_data_smithi159_162467_object52864:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75af21d:::benchmark_data_smithi159_162467_object69707:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75ba727:::benchmark_data_smithi159_162467_object60128:head# 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.504 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75c78a0:::benchmark_data_smithi159_162467_object27607:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75cb8e9:::benchmark_data_smithi159_162467_object23146:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f75dec8d:::benchmark_data_smithi159_162467_object23351:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7613b0f:::benchmark_data_smithi159_162467_object39932:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f761e626:::benchmark_data_smithi159_162467_object72942:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f762b43b:::benchmark_data_smithi159_162467_object24694:head# 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.505 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f765a7be:::benchmark_data_smithi159_162467_object67647:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f765fec7:::benchmark_data_smithi159_162467_object18738:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7666f50:::benchmark_data_smithi159_162467_object61972:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f766ae00:::benchmark_data_smithi159_162467_object13125:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f766ed31:::benchmark_data_smithi159_162467_object3512:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f767af41:::benchmark_data_smithi159_162467_object40257:head# 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.506 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f768c890:::benchmark_data_smithi159_162467_object41506:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7695730:::benchmark_data_smithi159_162467_object63843:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76a8739:::benchmark_data_smithi159_162467_object51249:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76afbea:::benchmark_data_smithi159_162467_object23738:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76b00df:::benchmark_data_smithi159_162467_object1231:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76c31d2:::benchmark_data_smithi159_162467_object7809:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76c8ca7:::benchmark_data_smithi159_162467_object1494:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76deb54:::benchmark_data_smithi159_162467_object70685:head# 2024-04-15T18:43:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76e128b:::benchmark_data_smithi159_162467_object74145:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76f337f:::benchmark_data_smithi159_162467_object40960:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f76fa617:::benchmark_data_smithi159_162467_object33089:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77000c5:::benchmark_data_smithi159_162467_object38909:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f770e5af:::benchmark_data_smithi159_162467_object57928:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7716886:::benchmark_data_smithi159_162467_object31185:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f771efd0:::benchmark_data_smithi159_162467_object65073:head# 2024-04-15T18:43:14.508 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f771f62a:::benchmark_data_smithi159_162467_object25118:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f772cf5d:::benchmark_data_smithi159_162467_object62799:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7737888:::benchmark_data_smithi159_162467_object5705:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7751367:::benchmark_data_smithi159_162467_object21951:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77515b3:::benchmark_data_smithi159_162467_object61017:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f775a163:::benchmark_data_smithi159_162467_object60155:head# 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.509 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f776955e:::benchmark_data_smithi159_162467_object22526:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f778abf4:::benchmark_data_smithi159_162467_object59368:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77aa13c:::benchmark_data_smithi159_162467_object50183:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77b7dcd:::benchmark_data_smithi159_162467_object30406:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77c43f0:::benchmark_data_smithi159_162467_object22668:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77d251a:::benchmark_data_smithi159_162467_object36915:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f77f85bb:::benchmark_data_smithi159_162467_object13259:head# 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.510 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f780de76:::benchmark_data_smithi159_162467_object11977:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7816044:::benchmark_data_smithi159_162467_object60245:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f781e015:::benchmark_data_smithi159_162467_object25513:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f781e8d8:::benchmark_data_smithi159_162467_object548:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7820e51:::benchmark_data_smithi159_162467_object32083:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f782ed9a:::benchmark_data_smithi159_162467_object39331:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7831d8b:::benchmark_data_smithi159_162467_object22458:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7833944:::benchmark_data_smithi159_162467_object2095:head# 2024-04-15T18:43:14.511 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f783bff9:::benchmark_data_smithi159_162467_object5578:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f78423ff:::benchmark_data_smithi159_162467_object67415:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f78438ed:::benchmark_data_smithi159_162467_object53809:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7869127:::benchmark_data_smithi159_162467_object69258:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7871cec:::benchmark_data_smithi159_162467_object341:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f787de91:::benchmark_data_smithi159_162467_object32524:head# 2024-04-15T18:43:14.512 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7880d02:::benchmark_data_smithi159_162467_object48159:head# 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f788c724:::benchmark_data_smithi159_162467_object73935:head# 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7891082:::benchmark_data_smithi159_162467_object28462:head# 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f78bad19:::benchmark_data_smithi159_162467_object26023:head# 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f78d82af:::benchmark_data_smithi159_162467_object21393:head# 2024-04-15T18:43:14.513 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f78eb5f6:::benchmark_data_smithi159_162467_object57561:head# 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7944087:::benchmark_data_smithi159_162467_object52209:head# 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79505a8:::benchmark_data_smithi159_162467_object14372:head# 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7958aab:::benchmark_data_smithi159_162467_object10281:head# 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f795da8d:::benchmark_data_smithi159_162467_object2453:head# 2024-04-15T18:43:14.543 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f795f356:::benchmark_data_smithi159_162467_object5997:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7960ef5:::benchmark_data_smithi159_162467_object52725:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79629b9:::benchmark_data_smithi159_162467_object23515:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f796d936:::benchmark_data_smithi159_162467_object38480:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79790b9:::benchmark_data_smithi159_162467_object70529:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79817e8:::benchmark_data_smithi159_162467_object18978:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7995947:::benchmark_data_smithi159_162467_object5001:head# 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.544 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79a7fc8:::benchmark_data_smithi159_162467_object45503:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79b1894:::benchmark_data_smithi159_162467_object62155:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79b365f:::benchmark_data_smithi159_162467_object48140:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79ca467:::benchmark_data_smithi159_162467_object54609:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79ccf53:::benchmark_data_smithi159_162467_object18804:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79cede3:::benchmark_data_smithi159_162467_object62148:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79d7835:::benchmark_data_smithi159_162467_object31288:head# 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.545 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79ea405:::benchmark_data_smithi159_162467_object20252:head# 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f79fd821:::benchmark_data_smithi159_162467_object30120:head# 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a0ad30:::benchmark_data_smithi159_162467_object9989:head# 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a0de99:::benchmark_data_smithi159_162467_object27697:head# 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a0fd04:::benchmark_data_smithi159_162467_object11119:head# 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.546 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a15ece:::benchmark_data_smithi159_162467_object21591:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a28e0f:::benchmark_data_smithi159_162467_object64925:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a3e04c:::benchmark_data_smithi159_162467_object11322:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a5005f:::benchmark_data_smithi159_162467_object60730:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a56e1b:::benchmark_data_smithi159_162467_object27406:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a5cb8c:::benchmark_data_smithi159_162467_object33542:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a6226e:::benchmark_data_smithi159_162467_object62170:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a6e7ac:::benchmark_data_smithi159_162467_object62541:head# 2024-04-15T18:43:14.547 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a784d6:::benchmark_data_smithi159_162467_object57410:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a7a0cb:::benchmark_data_smithi159_162467_object43383:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a7fe7f:::benchmark_data_smithi159_162467_object33899:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7a8afb2:::benchmark_data_smithi159_162467_object26587:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7aa2aca:::benchmark_data_smithi159_162467_object55249:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7aa907e:::benchmark_data_smithi159_162467_object68673:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7abfa4d:::benchmark_data_smithi159_162467_object22082:head# 2024-04-15T18:43:14.548 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b06216:::benchmark_data_smithi159_162467_object1941:head# 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b0757c:::benchmark_data_smithi159_162467_object34041:head# 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b0ae63:::benchmark_data_smithi159_162467_object39167:head# 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b0efb8:::benchmark_data_smithi159_162467_object45142:head# 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.549 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b1c98b:::benchmark_data_smithi159_162467_object44415:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b29f74:::benchmark_data_smithi159_162467_object47657:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b2a5ad:::benchmark_data_smithi159_162467_object32269:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b354ba:::benchmark_data_smithi159_162467_object43256:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b57062:::benchmark_data_smithi159_162467_object11179:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b72e84:::benchmark_data_smithi159_162467_object547:head# 2024-04-15T18:43:14.550 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b73c22:::benchmark_data_smithi159_162467_object15205:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b74c78:::benchmark_data_smithi159_162467_object64223:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b754c5:::benchmark_data_smithi159_162467_object9481:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7b7ef5a:::benchmark_data_smithi159_162467_object61554:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bb878f:::benchmark_data_smithi159_162467_object8153:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bb8bba:::benchmark_data_smithi159_162467_object49207:head# 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.551 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bbc9cf:::benchmark_data_smithi159_162467_object10685:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bc7aa2:::benchmark_data_smithi159_162467_object60604:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bd998c:::benchmark_data_smithi159_162467_object18892:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7be409e:::benchmark_data_smithi159_162467_object31129:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bf1ff4:::benchmark_data_smithi159_162467_object1516:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bf4df1:::benchmark_data_smithi159_162467_object46668:head# 2024-04-15T18:43:14.552 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bfa512:::benchmark_data_smithi159_162467_object2100:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7bffe88:::benchmark_data_smithi159_162467_object50327:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c2077a:::benchmark_data_smithi159_162467_object65034:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c231ac:::benchmark_data_smithi159_162467_object4364:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c3bfd2:::benchmark_data_smithi159_162467_object21828:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c3e458:::benchmark_data_smithi159_162467_object33945:head# 2024-04-15T18:43:14.553 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c40f70:::benchmark_data_smithi159_162467_object50318:head# 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c5f4da:::benchmark_data_smithi159_162467_object64916:head# 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c6af89:::benchmark_data_smithi159_162467_object40856:head# 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c82e50:::benchmark_data_smithi159_162467_object62502:head# 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.554 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c8ea5e:::benchmark_data_smithi159_162467_object29395:head# 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7c959f0:::benchmark_data_smithi159_162467_object57771:head# 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ca7af8:::benchmark_data_smithi159_162467_object39995:head# 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7cd72e0:::benchmark_data_smithi159_162467_object28837:head# 2024-04-15T18:43:14.555 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ce1274:::benchmark_data_smithi159_162467_object21557:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7cf340e:::benchmark_data_smithi159_162467_object53076:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7cfe61c:::benchmark_data_smithi159_162467_object43300:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d0119d:::benchmark_data_smithi159_162467_object69748:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d1ebf5:::benchmark_data_smithi159_162467_object30339:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d22118:::benchmark_data_smithi159_162467_object34256:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d554e1:::benchmark_data_smithi159_162467_object50191:head# 2024-04-15T18:43:14.556 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d83f20:::benchmark_data_smithi159_162467_object51169:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d848d3:::benchmark_data_smithi159_162467_object26679:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7d84fc9:::benchmark_data_smithi159_162467_object20466:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7daa57b:::benchmark_data_smithi159_162467_object18944:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7dac0c1:::benchmark_data_smithi159_162467_object60622:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7dc5475:::benchmark_data_smithi159_162467_object28540:head# 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.557 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7dcbb0c:::benchmark_data_smithi159_162467_object20457:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7dcea0b:::benchmark_data_smithi159_162467_object20868:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7dd7bd3:::benchmark_data_smithi159_162467_object72011:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7de0347:::benchmark_data_smithi159_162467_object14124:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7de6824:::benchmark_data_smithi159_162467_object42713:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e0906b:::benchmark_data_smithi159_162467_object30410:head# 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.558 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e0d4bf:::benchmark_data_smithi159_162467_object23881:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e139b9:::benchmark_data_smithi159_162467_object1848:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e2b8f1:::benchmark_data_smithi159_162467_object62011:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e357c3:::benchmark_data_smithi159_162467_object45871:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e3c48d:::benchmark_data_smithi159_162467_object24186:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e54896:::benchmark_data_smithi159_162467_object10639:head# 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.559 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e5834b:::benchmark_data_smithi159_162467_object6455:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e5dee6:::benchmark_data_smithi159_162467_object61121:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e600bc:::benchmark_data_smithi159_162467_object587:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e6384a:::benchmark_data_smithi159_162467_object61586:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e75c34:::benchmark_data_smithi159_162467_object26011:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7e7ce20:::benchmark_data_smithi159_162467_object25789:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ea2a30:::benchmark_data_smithi159_162467_object11693:head# 2024-04-15T18:43:14.560 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ed1306:::benchmark_data_smithi159_162467_object34800:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7edd8d2:::benchmark_data_smithi159_162467_object37269:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ede902:::benchmark_data_smithi159_162467_object6262:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ee740f:::benchmark_data_smithi159_162467_object18325:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7eefa09:::benchmark_data_smithi159_162467_object72281:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7efa781:::benchmark_data_smithi159_162467_object14841:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f00e01:::benchmark_data_smithi159_162467_object44143:head# 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.561 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f02c57:::benchmark_data_smithi159_162467_object27154:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f39304:::benchmark_data_smithi159_162467_object62209:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f3b9ac:::benchmark_data_smithi159_162467_object48113:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f4543c:::benchmark_data_smithi159_162467_object52907:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f4a29c:::benchmark_data_smithi159_162467_object24194:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f65164:::benchmark_data_smithi159_162467_object24480:head# 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.562 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f6a6af:::benchmark_data_smithi159_162467_object51372:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f8270c:::benchmark_data_smithi159_162467_object69336:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f82aab:::benchmark_data_smithi159_162467_object63801:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f83cd2:::benchmark_data_smithi159_162467_object35213:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f94ed1:::benchmark_data_smithi159_162467_object19446:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7f9be70:::benchmark_data_smithi159_162467_object42967:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7fafcce:::benchmark_data_smithi159_162467_object39127:head# 2024-04-15T18:43:14.563 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7fbafaf:::benchmark_data_smithi159_162467_object20081:head# 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7fc7fbf:::benchmark_data_smithi159_162467_object33118:head# 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7fd3b04:::benchmark_data_smithi159_162467_object44944:head# 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7fe9a05:::benchmark_data_smithi159_162467_object20956:head# 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f7ff1440:::benchmark_data_smithi159_162467_object56266:head# 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.564 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8016939:::benchmark_data_smithi159_162467_object344:head# 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8023e38:::benchmark_data_smithi159_162467_object72042:head# 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f803c79d:::benchmark_data_smithi159_162467_object21236:head# 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8046ff2:::benchmark_data_smithi159_162467_object4616:head# 2024-04-15T18:43:14.565 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f805e4ac:::benchmark_data_smithi159_162467_object60087:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8062515:::benchmark_data_smithi159_162467_object31534:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8066d16:::benchmark_data_smithi159_162467_object67837:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8071ada:::benchmark_data_smithi159_162467_object47316:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8078152:::benchmark_data_smithi159_162467_object30486:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8079bd2:::benchmark_data_smithi159_162467_object4977:head# 2024-04-15T18:43:14.566 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f807a1ef:::benchmark_data_smithi159_162467_object38391:head# 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f807ba43:::benchmark_data_smithi159_162467_object25303:head# 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8086f65:::benchmark_data_smithi159_162467_object64659:head# 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80cc6be:::benchmark_data_smithi159_162467_object40386:head# 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80cc917:::benchmark_data_smithi159_162467_object43769:head# 2024-04-15T18:43:14.567 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80def32:::benchmark_data_smithi159_162467_object30775:head# 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80e48df:::benchmark_data_smithi159_162467_object65050:head# 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80f9948:::benchmark_data_smithi159_162467_object13692:head# 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f80fc848:::benchmark_data_smithi159_162467_object31917:head# 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.568 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8110f35:::benchmark_data_smithi159_162467_object43645:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8112701:::benchmark_data_smithi159_162467_object28541:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81146eb:::benchmark_data_smithi159_162467_object2917:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f811b9bf:::benchmark_data_smithi159_162467_object60491:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81282b1:::benchmark_data_smithi159_162467_object50775:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f814011f:::benchmark_data_smithi159_162467_object125:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8170f6a:::benchmark_data_smithi159_162467_object60723:head# 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.569 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8181dd5:::benchmark_data_smithi159_162467_object38167:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8186513:::benchmark_data_smithi159_162467_object51093:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f818c593:::benchmark_data_smithi159_162467_object59179:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f819c838:::benchmark_data_smithi159_162467_object58387:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81a0632:::benchmark_data_smithi159_162467_object14449:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81ae78e:::benchmark_data_smithi159_162467_object37963:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81b0ae7:::benchmark_data_smithi159_162467_object39924:head# 2024-04-15T18:43:14.570 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81b261f:::benchmark_data_smithi159_162467_object15135:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81ba34a:::benchmark_data_smithi159_162467_object63037:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81bc854:::benchmark_data_smithi159_162467_object18594:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81c4fb5:::benchmark_data_smithi159_162467_object64770:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81cb0d7:::benchmark_data_smithi159_162467_object12650:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81ecc59:::benchmark_data_smithi159_162467_object46042:head# 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.571 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81f79bb:::benchmark_data_smithi159_162467_object65119:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f81f7bde:::benchmark_data_smithi159_162467_object42317:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8204e93:::benchmark_data_smithi159_162467_object64679:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82109e3:::benchmark_data_smithi159_162467_object62634:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8217e9d:::benchmark_data_smithi159_162467_object64933:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8224433:::benchmark_data_smithi159_162467_object45314:head# 2024-04-15T18:43:14.572 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82271fa:::benchmark_data_smithi159_162467_object55355:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f822a100:::benchmark_data_smithi159_162467_object14031:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f822a8e8:::benchmark_data_smithi159_162467_object72581:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f823d3f1:::benchmark_data_smithi159_162467_object63096:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8246ca0:::benchmark_data_smithi159_162467_object5297:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8248418:::benchmark_data_smithi159_162467_object42123:head# 2024-04-15T18:43:14.573 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82676f4:::benchmark_data_smithi159_162467_object26682:head# 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8267b73:::benchmark_data_smithi159_162467_object64333:head# 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82686b4:::benchmark_data_smithi159_162467_object30960:head# 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8273a3a:::benchmark_data_smithi159_162467_object41204:head# 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f827f78a:::benchmark_data_smithi159_162467_object30689:head# 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.574 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8284d92:::benchmark_data_smithi159_162467_object27880:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82864b2:::benchmark_data_smithi159_162467_object57026:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f829553b:::benchmark_data_smithi159_162467_object12555:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8297a64:::benchmark_data_smithi159_162467_object23799:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8297ec3:::benchmark_data_smithi159_162467_object49091:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8298804:::benchmark_data_smithi159_162467_object13542:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f829d214:::benchmark_data_smithi159_162467_object42239:head# 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.575 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82ae1ef:::benchmark_data_smithi159_162467_object4966:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82d11b6:::benchmark_data_smithi159_162467_object64225:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82e717c:::benchmark_data_smithi159_162467_object41276:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82eedae:::benchmark_data_smithi159_162467_object55253:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82f248a:::benchmark_data_smithi159_162467_object7945:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f82f7e09:::benchmark_data_smithi159_162467_object23282:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8317770:::benchmark_data_smithi159_162467_object57153:head# 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.576 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f831d7a2:::benchmark_data_smithi159_162467_object42672:head# 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83281ed:::benchmark_data_smithi159_162467_object18966:head# 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83316ab:::benchmark_data_smithi159_162467_object10882:head# 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8336599:::benchmark_data_smithi159_162467_object28639:head# 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f834c59c:::benchmark_data_smithi159_162467_object6974:head# 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.577 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f834e286:::benchmark_data_smithi159_162467_object19434:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8352785:::benchmark_data_smithi159_162467_object50404:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f836aa42:::benchmark_data_smithi159_162467_object45070:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f836e825:::benchmark_data_smithi159_162467_object59131:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f838ca56:::benchmark_data_smithi159_162467_object52124:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f839ff4c:::benchmark_data_smithi159_162467_object74450:head# 2024-04-15T18:43:14.578 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83b2f6a:::benchmark_data_smithi159_162467_object6823:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83d09db:::benchmark_data_smithi159_162467_object18196:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83e47f6:::benchmark_data_smithi159_162467_object12649:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83e58c7:::benchmark_data_smithi159_162467_object22898:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83edaaa:::benchmark_data_smithi159_162467_object45669:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f83f47cf:::benchmark_data_smithi159_162467_object17886:head# 2024-04-15T18:43:14.579 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.616 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84097f4:::benchmark_data_smithi159_162467_object24031:head# 2024-04-15T18:43:14.616 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.616 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f841a0e1:::benchmark_data_smithi159_162467_object38809:head# 2024-04-15T18:43:14.616 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f841d1ff:::benchmark_data_smithi159_162467_object36253:head# 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f842053f:::benchmark_data_smithi159_162467_object55685:head# 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8429a5a:::benchmark_data_smithi159_162467_object64682:head# 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8445bc1:::benchmark_data_smithi159_162467_object45273:head# 2024-04-15T18:43:14.617 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f844a357:::benchmark_data_smithi159_162467_object20445:head# 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f845af51:::benchmark_data_smithi159_162467_object56430:head# 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8461a10:::benchmark_data_smithi159_162467_object43469:head# 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8468302:::benchmark_data_smithi159_162467_object27203:head# 2024-04-15T18:43:14.618 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8469842:::benchmark_data_smithi159_162467_object62325:head# 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84703f2:::benchmark_data_smithi159_162467_object29267:head# 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84be26b:::benchmark_data_smithi159_162467_object42144:head# 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84c1679:::benchmark_data_smithi159_162467_object39526:head# 2024-04-15T18:43:14.619 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84c4618:::benchmark_data_smithi159_162467_object64145:head# 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84ddfb2:::benchmark_data_smithi159_162467_object2477:head# 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f84fdf63:::benchmark_data_smithi159_162467_object28700:head# 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8519833:::benchmark_data_smithi159_162467_object37281:head# 2024-04-15T18:43:14.620 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f851d71e:::benchmark_data_smithi159_162467_object28941:head# 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8522514:::benchmark_data_smithi159_162467_object56636:head# 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85287f2:::benchmark_data_smithi159_162467_object17944:head# 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f853a625:::benchmark_data_smithi159_162467_object68589:head# 2024-04-15T18:43:14.621 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8547759:::benchmark_data_smithi159_162467_object57580:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85545f9:::benchmark_data_smithi159_162467_object35001:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8555ea4:::benchmark_data_smithi159_162467_object64090:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8558eee:::benchmark_data_smithi159_162467_object10602:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f856805c:::benchmark_data_smithi159_162467_object28699:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8575bf9:::benchmark_data_smithi159_162467_object999:head# 2024-04-15T18:43:14.622 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f857bf24:::benchmark_data_smithi159_162467_object38747:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f859513c:::benchmark_data_smithi159_162467_object58939:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f859c5ab:::benchmark_data_smithi159_162467_object73645:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85aa0b3:::benchmark_data_smithi159_162467_object51142:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85ac684:::benchmark_data_smithi159_162467_object10839:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85cca7e:::benchmark_data_smithi159_162467_object10728:head# 2024-04-15T18:43:14.623 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85d6906:::benchmark_data_smithi159_162467_object50599:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85dfe7f:::benchmark_data_smithi159_162467_object55980:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85e5031:::benchmark_data_smithi159_162467_object5941:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85e57ba:::benchmark_data_smithi159_162467_object55830:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f85e6a81:::benchmark_data_smithi159_162467_object10836:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f860a603:::benchmark_data_smithi159_162467_object27559:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f86165b9:::benchmark_data_smithi159_162467_object43553:head# 2024-04-15T18:43:14.624 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f862c68f:::benchmark_data_smithi159_162467_object33857:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8646723:::benchmark_data_smithi159_162467_object47531:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f864784b:::benchmark_data_smithi159_162467_object33769:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f865af2f:::benchmark_data_smithi159_162467_object33188:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f866e9ea:::benchmark_data_smithi159_162467_object15364:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8681135:::benchmark_data_smithi159_162467_object10525:head# 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.625 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f86888cd:::benchmark_data_smithi159_162467_object2706:head# 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f86ccefd:::benchmark_data_smithi159_162467_object12478:head# 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8709748:::benchmark_data_smithi159_162467_object24408:head# 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f870a740:::benchmark_data_smithi159_162467_object2051:head# 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f870c99c:::benchmark_data_smithi159_162467_object37105:head# 2024-04-15T18:43:14.626 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8713a2c:::benchmark_data_smithi159_162467_object9294:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87147e0:::benchmark_data_smithi159_162467_object69708:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8719ecc:::benchmark_data_smithi159_162467_object34864:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8732915:::benchmark_data_smithi159_162467_object16172:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8741c06:::benchmark_data_smithi159_162467_object44323:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8741c43:::benchmark_data_smithi159_162467_object39492:head# 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.627 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8742efd:::benchmark_data_smithi159_162467_object41740:head# 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f874dd9c:::benchmark_data_smithi159_162467_object46841:head# 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f876a025:::benchmark_data_smithi159_162467_object38533:head# 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f879d61b:::benchmark_data_smithi159_162467_object8836:head# 2024-04-15T18:43:14.628 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87a76b4:::benchmark_data_smithi159_162467_object55859:head# 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87b7272:::benchmark_data_smithi159_162467_object46421:head# 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87baead:::benchmark_data_smithi159_162467_object35039:head# 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.629 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87bb355:::benchmark_data_smithi159_162467_object56733:head# 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87c4375:::benchmark_data_smithi159_162467_object46744:head# 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87d026d:::benchmark_data_smithi159_162467_object39276:head# 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87db4c5:::benchmark_data_smithi159_162467_object44113:head# 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87e2411:::benchmark_data_smithi159_162467_object45575:head# 2024-04-15T18:43:14.630 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f87e55cc:::benchmark_data_smithi159_162467_object69845:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8805f96:::benchmark_data_smithi159_162467_object58055:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8810e84:::benchmark_data_smithi159_162467_object16057:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88147ed:::benchmark_data_smithi159_162467_object37235:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8822272:::benchmark_data_smithi159_162467_object69377:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8841cb3:::benchmark_data_smithi159_162467_object12401:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88588fb:::benchmark_data_smithi159_162467_object51011:head# 2024-04-15T18:43:14.631 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f886d350:::benchmark_data_smithi159_162467_object814:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88a455a:::benchmark_data_smithi159_162467_object35111:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88a4fe0:::benchmark_data_smithi159_162467_object62405:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88bb044:::benchmark_data_smithi159_162467_object19120:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88ccebc:::benchmark_data_smithi159_162467_object29400:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88d55e7:::benchmark_data_smithi159_162467_object67383:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88dd406:::benchmark_data_smithi159_162467_object9417:head# 2024-04-15T18:43:14.632 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88e2142:::benchmark_data_smithi159_162467_object48215:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88ecb30:::benchmark_data_smithi159_162467_object72468:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88f7b3a:::benchmark_data_smithi159_162467_object38945:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f88f8f6a:::benchmark_data_smithi159_162467_object26415:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8903de7:::benchmark_data_smithi159_162467_object73772:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f890da4f:::benchmark_data_smithi159_162467_object50850:head# 2024-04-15T18:43:14.633 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f891c98e:::benchmark_data_smithi159_162467_object33986:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8923d11:::benchmark_data_smithi159_162467_object52018:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f892e713:::benchmark_data_smithi159_162467_object65520:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f892fb93:::benchmark_data_smithi159_162467_object10976:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8932807:::benchmark_data_smithi159_162467_object30910:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89583a7:::benchmark_data_smithi159_162467_object62204:head# 2024-04-15T18:43:14.634 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8958b3b:::benchmark_data_smithi159_162467_object59371:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f895eb66:::benchmark_data_smithi159_162467_object26512:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f896627d:::benchmark_data_smithi159_162467_object37222:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f896e03e:::benchmark_data_smithi159_162467_object72677:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89740e9:::benchmark_data_smithi159_162467_object22925:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f898cabc:::benchmark_data_smithi159_162467_object52592:head# 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f899ad33:::benchmark_data_smithi159_162467_object64754:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f899e34e:::benchmark_data_smithi159_162467_object11406:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89aa753:::benchmark_data_smithi159_162467_object58467:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89aca39:::benchmark_data_smithi159_162467_object60355:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89b7cf4:::benchmark_data_smithi159_162467_object48319:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89bd98f:::benchmark_data_smithi159_162467_object33631:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89c9ef2:::benchmark_data_smithi159_162467_object5210:head# 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89e0911:::benchmark_data_smithi159_162467_object17055:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89ebd4f:::benchmark_data_smithi159_162467_object68777:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89ec5ef:::benchmark_data_smithi159_162467_object65182:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f89fadd9:::benchmark_data_smithi159_162467_object133:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a21883:::benchmark_data_smithi159_162467_object10263:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a230e4:::benchmark_data_smithi159_162467_object47011:head# 2024-04-15T18:43:14.637 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a2484f:::benchmark_data_smithi159_162467_object4265:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a27f8c:::benchmark_data_smithi159_162467_object62999:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a506e5:::benchmark_data_smithi159_162467_object37329:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a51c41:::benchmark_data_smithi159_162467_object10819:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a6ed77:::benchmark_data_smithi159_162467_object7643:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a7b81b:::benchmark_data_smithi159_162467_object39709:head# 2024-04-15T18:43:14.638 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a7d8b3:::benchmark_data_smithi159_162467_object62303:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a86769:::benchmark_data_smithi159_162467_object52406:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a88372:::benchmark_data_smithi159_162467_object13311:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a88455:::benchmark_data_smithi159_162467_object71716:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a8a17e:::benchmark_data_smithi159_162467_object8730:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8a9ac87:::benchmark_data_smithi159_162467_object72509:head# 2024-04-15T18:43:14.639 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8aa8c41:::benchmark_data_smithi159_162467_object53788:head# 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ab32b7:::benchmark_data_smithi159_162467_object47763:head# 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8aba79a:::benchmark_data_smithi159_162467_object46597:head# 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ad260f:::benchmark_data_smithi159_162467_object27687:head# 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8adfe8d:::benchmark_data_smithi159_162467_object33635:head# 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.640 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8aea74c:::benchmark_data_smithi159_162467_object15697:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b04dfa:::benchmark_data_smithi159_162467_object10766:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b0578d:::benchmark_data_smithi159_162467_object3574:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b107db:::benchmark_data_smithi159_162467_object1375:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b33dfb:::benchmark_data_smithi159_162467_object67026:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b35681:::benchmark_data_smithi159_162467_object23097:head# 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b4641b:::benchmark_data_smithi159_162467_object6579:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b47c60:::benchmark_data_smithi159_162467_object57253:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b525e7:::benchmark_data_smithi159_162467_object47178:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b62a36:::benchmark_data_smithi159_162467_object37376:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b6599b:::benchmark_data_smithi159_162467_object21048:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b72dc2:::benchmark_data_smithi159_162467_object70794:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b79739:::benchmark_data_smithi159_162467_object42893:head# 2024-04-15T18:43:14.642 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b7bb26:::benchmark_data_smithi159_162467_object63487:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b8006f:::benchmark_data_smithi159_162467_object38511:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b86745:::benchmark_data_smithi159_162467_object63067:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8b8d297:::benchmark_data_smithi159_162467_object51845:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8bb74b1:::benchmark_data_smithi159_162467_object28254:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8bb7580:::benchmark_data_smithi159_162467_object7326:head# 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.643 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8bc7f7f:::benchmark_data_smithi159_162467_object20877:head# 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8bcd4fe:::benchmark_data_smithi159_162467_object16418:head# 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8bf24b2:::benchmark_data_smithi159_162467_object8164:head# 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c2743f:::benchmark_data_smithi159_162467_object35434:head# 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.644 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c2f4ee:::benchmark_data_smithi159_162467_object7065:head# 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c37df5:::benchmark_data_smithi159_162467_object46387:head# 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c38c0b:::benchmark_data_smithi159_162467_object15384:head# 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c48238:::benchmark_data_smithi159_162467_object63654:head# 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.645 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c4d306:::benchmark_data_smithi159_162467_object45543:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c70f55:::benchmark_data_smithi159_162467_object24243:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c736fe:::benchmark_data_smithi159_162467_object27191:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c82aab:::benchmark_data_smithi159_162467_object52622:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c88048:::benchmark_data_smithi159_162467_object36276:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8c9780c:::benchmark_data_smithi159_162467_object22494:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8cc13d0:::benchmark_data_smithi159_162467_object73133:head# 2024-04-15T18:43:14.646 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8cc1ce4:::benchmark_data_smithi159_162467_object37749:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8cca656:::benchmark_data_smithi159_162467_object66616:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ce4b15:::benchmark_data_smithi159_162467_object41019:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ce6397:::benchmark_data_smithi159_162467_object15459:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8cf8c04:::benchmark_data_smithi159_162467_object69033:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d27092:::benchmark_data_smithi159_162467_object14329:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d275b8:::benchmark_data_smithi159_162467_object55033:head# 2024-04-15T18:43:14.647 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d3380c:::benchmark_data_smithi159_162467_object51714:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d3d7e4:::benchmark_data_smithi159_162467_object55872:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d4cc44:::benchmark_data_smithi159_162467_object70071:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d54ba4:::benchmark_data_smithi159_162467_object4959:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d5f237:::benchmark_data_smithi159_162467_object26562:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d779eb:::benchmark_data_smithi159_162467_object60623:head# 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.648 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d78f7d:::benchmark_data_smithi159_162467_object8715:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d79814:::benchmark_data_smithi159_162467_object54013:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8d8d31b:::benchmark_data_smithi159_162467_object35568:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8da9aca:::benchmark_data_smithi159_162467_object3921:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dad246:::benchmark_data_smithi159_162467_object69360:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8db1e61:::benchmark_data_smithi159_162467_object73135:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dc0154:::benchmark_data_smithi159_162467_object34791:head# 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.649 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dc0e51:::benchmark_data_smithi159_162467_object74122:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dc2605:::benchmark_data_smithi159_162467_object62462:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dd6082:::benchmark_data_smithi159_162467_object58701:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dda072:::benchmark_data_smithi159_162467_object49874:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dea922:::benchmark_data_smithi159_162467_object32545:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dedc88:::benchmark_data_smithi159_162467_object58090:head# 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.650 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8df1396:::benchmark_data_smithi159_162467_object57508:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8dffb0e:::benchmark_data_smithi159_162467_object57751:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e0ba58:::benchmark_data_smithi159_162467_object7810:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e1689d:::benchmark_data_smithi159_162467_object45445:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e2863c:::benchmark_data_smithi159_162467_object38680:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e3d055:::benchmark_data_smithi159_162467_object22858:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e3d664:::benchmark_data_smithi159_162467_object2839:head# 2024-04-15T18:43:14.651 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e5764b:::benchmark_data_smithi159_162467_object6454:head# 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e59da6:::benchmark_data_smithi159_162467_object34992:head# 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e7d004:::benchmark_data_smithi159_162467_object31458:head# 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e861b7:::benchmark_data_smithi159_162467_object5509:head# 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e89351:::benchmark_data_smithi159_162467_object19063:head# 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.652 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8e9eb8a:::benchmark_data_smithi159_162467_object39573:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ead14c:::benchmark_data_smithi159_162467_object54870:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8eb17e3:::benchmark_data_smithi159_162467_object37887:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ebcac2:::benchmark_data_smithi159_162467_object47124:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ec4917:::benchmark_data_smithi159_162467_object21746:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8edca8a:::benchmark_data_smithi159_162467_object1028:head# 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.653 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8ee0d88:::benchmark_data_smithi159_162467_object23070:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f39ebe:::benchmark_data_smithi159_162467_object40448:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f3bff6:::benchmark_data_smithi159_162467_object54682:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f4311b:::benchmark_data_smithi159_162467_object68368:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f4cc4b:::benchmark_data_smithi159_162467_object927:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f4fd4d:::benchmark_data_smithi159_162467_object67761:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f670d7:::benchmark_data_smithi159_162467_object4493:head# 2024-04-15T18:43:14.654 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f693f3:::benchmark_data_smithi159_162467_object43297:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f7646a:::benchmark_data_smithi159_162467_object34464:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f79e9f:::benchmark_data_smithi159_162467_object62248:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f83cfc:::benchmark_data_smithi159_162467_object55855:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8f9d768:::benchmark_data_smithi159_162467_object511:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8fb61a7:::benchmark_data_smithi159_162467_object24005:head# 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.655 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8fc9c20:::benchmark_data_smithi159_162467_object47596:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8fd3c9f:::benchmark_data_smithi159_162467_object58064:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8fe42cd:::benchmark_data_smithi159_162467_object3683:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f8feadc7:::benchmark_data_smithi159_162467_object58216:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f903af4a:::benchmark_data_smithi159_162467_object66296:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f903e395:::benchmark_data_smithi159_162467_object593:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f905b938:::benchmark_data_smithi159_162467_object17438:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9082a18:::benchmark_data_smithi159_162467_object20838:head# 2024-04-15T18:43:14.656 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f908679b:::benchmark_data_smithi159_162467_object13064:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f909c17b:::benchmark_data_smithi159_162467_object58876:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90b479f:::benchmark_data_smithi159_162467_object39096:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90bf4ec:::benchmark_data_smithi159_162467_object61183:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90bf9d3:::benchmark_data_smithi159_162467_object8288:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90c8289:::benchmark_data_smithi159_162467_object5821:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90ca609:::benchmark_data_smithi159_162467_object39813:head# 2024-04-15T18:43:14.657 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90d2020:::benchmark_data_smithi159_162467_object17268:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90d7d9b:::benchmark_data_smithi159_162467_object34555:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90dc452:::benchmark_data_smithi159_162467_object71743:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f90ddefa:::benchmark_data_smithi159_162467_object32159:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9108385:::benchmark_data_smithi159_162467_object61792:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9114cea:::benchmark_data_smithi159_162467_object7196:head# 2024-04-15T18:43:14.658 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9128f12:::benchmark_data_smithi159_162467_object3023:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f912970f:::benchmark_data_smithi159_162467_object22344:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f912ffd9:::benchmark_data_smithi159_162467_object20031:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9134171:::benchmark_data_smithi159_162467_object43060:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91383eb:::benchmark_data_smithi159_162467_object52380:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f913b97e:::benchmark_data_smithi159_162467_object67666:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9153b7e:::benchmark_data_smithi159_162467_object52716:head# 2024-04-15T18:43:14.659 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f915933c:::benchmark_data_smithi159_162467_object52601:head# 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9167164:::benchmark_data_smithi159_162467_object17539:head# 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f917ba71:::benchmark_data_smithi159_162467_object63018:head# 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9181b12:::benchmark_data_smithi159_162467_object11190:head# 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91bc428:::benchmark_data_smithi159_162467_object59098:head# 2024-04-15T18:43:14.660 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91c3188:::benchmark_data_smithi159_162467_object8931:head# 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91dd4cc:::benchmark_data_smithi159_162467_object24421:head# 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91e14ef:::benchmark_data_smithi159_162467_object28129:head# 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.692 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91eef24:::benchmark_data_smithi159_162467_object15606:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91f1a7f:::benchmark_data_smithi159_162467_object21289:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f91f3132:::benchmark_data_smithi159_162467_object29660:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f920df60:::benchmark_data_smithi159_162467_object43183:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f923b903:::benchmark_data_smithi159_162467_object42170:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92426ef:::benchmark_data_smithi159_162467_object28034:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f925656c:::benchmark_data_smithi159_162467_object59487:head# 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.693 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92585da:::benchmark_data_smithi159_162467_object6256:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f925c060:::benchmark_data_smithi159_162467_object22837:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9268c5d:::benchmark_data_smithi159_162467_object68683:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f927967e:::benchmark_data_smithi159_162467_object512:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f927ef11:::benchmark_data_smithi159_162467_object29315:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9285325:::benchmark_data_smithi159_162467_object10428:head# 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.694 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9297985:::benchmark_data_smithi159_162467_object37821:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92a0cfe:::benchmark_data_smithi159_162467_object12138:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92a2c9a:::benchmark_data_smithi159_162467_object33880:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92a3465:::benchmark_data_smithi159_162467_object70875:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92af738:::benchmark_data_smithi159_162467_object20409:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92afc15:::benchmark_data_smithi159_162467_object71818:head# 2024-04-15T18:43:14.695 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f92bb55a:::benchmark_data_smithi159_162467_object20417:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93208d5:::benchmark_data_smithi159_162467_object53629:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f933c0b0:::benchmark_data_smithi159_162467_object53609:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9340395:::benchmark_data_smithi159_162467_object15884:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f934e789:::benchmark_data_smithi159_162467_object13076:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9351328:::benchmark_data_smithi159_162467_object41453:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9351421:::benchmark_data_smithi159_162467_object72625:head# 2024-04-15T18:43:14.696 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f935b7e6:::benchmark_data_smithi159_162467_object55612:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9360f72:::benchmark_data_smithi159_162467_object50196:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9364092:::benchmark_data_smithi159_162467_object3942:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f937005b:::benchmark_data_smithi159_162467_object22534:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93865e0:::benchmark_data_smithi159_162467_object55272:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93899fe:::benchmark_data_smithi159_162467_object50904:head# 2024-04-15T18:43:14.697 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f938ec97:::benchmark_data_smithi159_162467_object55406:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9391f20:::benchmark_data_smithi159_162467_object15725:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f939e5e9:::benchmark_data_smithi159_162467_object34707:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93a150e:::benchmark_data_smithi159_162467_object36575:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93ae237:::benchmark_data_smithi159_162467_object34328:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93c286f:::benchmark_data_smithi159_162467_object71451:head# 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.698 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93c4247:::benchmark_data_smithi159_162467_object67731:head# 2024-04-15T18:43:14.699 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.699 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f93d1db0:::benchmark_data_smithi159_162467_object2166:head# 2024-04-15T18:43:14.699 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f940ad09:::benchmark_data_smithi159_162467_object55679:head# 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f942bbf3:::benchmark_data_smithi159_162467_object5908:head# 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f942dbb0:::benchmark_data_smithi159_162467_object12581:head# 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.701 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9438fd6:::benchmark_data_smithi159_162467_object12968:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f943bce1:::benchmark_data_smithi159_162467_object71202:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9451d46:::benchmark_data_smithi159_162467_object60648:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f945f5f3:::benchmark_data_smithi159_162467_object69254:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f946f506:::benchmark_data_smithi159_162467_object65198:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94716bc:::benchmark_data_smithi159_162467_object9835:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94829dc:::benchmark_data_smithi159_162467_object53627:head# 2024-04-15T18:43:14.702 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f949a2dc:::benchmark_data_smithi159_162467_object5796:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94a0e85:::benchmark_data_smithi159_162467_object2800:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94a958d:::benchmark_data_smithi159_162467_object59834:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94b54e6:::benchmark_data_smithi159_162467_object16997:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94cf127:::benchmark_data_smithi159_162467_object28665:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94d0c11:::benchmark_data_smithi159_162467_object62047:head# 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.703 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94f5a62:::benchmark_data_smithi159_162467_object7357:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f94f74ac:::benchmark_data_smithi159_162467_object4349:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95001a1:::benchmark_data_smithi159_162467_object69735:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9505282:::benchmark_data_smithi159_162467_object41960:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9514fbb:::benchmark_data_smithi159_162467_object49347:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f954eb82:::benchmark_data_smithi159_162467_object33239:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f955df38:::benchmark_data_smithi159_162467_object55541:head# 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.704 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f957dc26:::benchmark_data_smithi159_162467_object27719:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f958c495:::benchmark_data_smithi159_162467_object45923:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9593ad6:::benchmark_data_smithi159_162467_object69334:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9599d40:::benchmark_data_smithi159_162467_object9655:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95aed76:::benchmark_data_smithi159_162467_object17100:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95bc1ac:::benchmark_data_smithi159_162467_object71745:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95c81bf:::benchmark_data_smithi159_162467_object26053:head# 2024-04-15T18:43:14.705 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95c93f5:::benchmark_data_smithi159_162467_object59876:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95eb223:::benchmark_data_smithi159_162467_object45711:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95f3dcc:::benchmark_data_smithi159_162467_object53783:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f95f8f19:::benchmark_data_smithi159_162467_object29561:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9619edc:::benchmark_data_smithi159_162467_object71293:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f961d6d1:::benchmark_data_smithi159_162467_object59096:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96223e8:::benchmark_data_smithi159_162467_object24310:head# 2024-04-15T18:43:14.706 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9624055:::benchmark_data_smithi159_162467_object73378:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f962d442:::benchmark_data_smithi159_162467_object49691:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96337ef:::benchmark_data_smithi159_162467_object9933:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f964930b:::benchmark_data_smithi159_162467_object62581:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9657e79:::benchmark_data_smithi159_162467_object62670:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f965d5b9:::benchmark_data_smithi159_162467_object62672:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f965f10e:::benchmark_data_smithi159_162467_object7651:head# 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.707 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9667c4f:::benchmark_data_smithi159_162467_object28842:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f968ce1a:::benchmark_data_smithi159_162467_object27329:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f969f4cf:::benchmark_data_smithi159_162467_object6969:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96a32f0:::benchmark_data_smithi159_162467_object47189:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96ac612:::benchmark_data_smithi159_162467_object41833:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96aec13:::benchmark_data_smithi159_162467_object30755:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96c5e2e:::benchmark_data_smithi159_162467_object52046:head# 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.708 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96da3ee:::benchmark_data_smithi159_162467_object4304:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96e1a3c:::benchmark_data_smithi159_162467_object48393:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f96fa331:::benchmark_data_smithi159_162467_object908:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9703453:::benchmark_data_smithi159_162467_object31147:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f970afc6:::benchmark_data_smithi159_162467_object66957:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f970f958:::benchmark_data_smithi159_162467_object71100:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9717e67:::benchmark_data_smithi159_162467_object66018:head# 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.709 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9719e5e:::benchmark_data_smithi159_162467_object20400:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f971e589:::benchmark_data_smithi159_162467_object50903:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9727c4d:::benchmark_data_smithi159_162467_object58832:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f972bddb:::benchmark_data_smithi159_162467_object70600:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f972dfa9:::benchmark_data_smithi159_162467_object16804:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f972f6be:::benchmark_data_smithi159_162467_object39009:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f973344e:::benchmark_data_smithi159_162467_object47890:head# 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.710 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9734c7f:::benchmark_data_smithi159_162467_object5386:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9735ca7:::benchmark_data_smithi159_162467_object4967:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f973c9e1:::benchmark_data_smithi159_162467_object38970:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f973ef56:::benchmark_data_smithi159_162467_object12482:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f974d5d7:::benchmark_data_smithi159_162467_object9338:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f974f01d:::benchmark_data_smithi159_162467_object38368:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9760e32:::benchmark_data_smithi159_162467_object67572:head# 2024-04-15T18:43:14.711 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97610b3:::benchmark_data_smithi159_162467_object39592:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9769098:::benchmark_data_smithi159_162467_object108:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f976a691:::benchmark_data_smithi159_162467_object16739:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f978ae74:::benchmark_data_smithi159_162467_object30476:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9792080:::benchmark_data_smithi159_162467_object21416:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f979922e:::benchmark_data_smithi159_162467_object3816:head# 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.712 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97abc1a:::benchmark_data_smithi159_162467_object42462:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97bfb67:::benchmark_data_smithi159_162467_object39107:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97ce137:::benchmark_data_smithi159_162467_object71820:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97d8c81:::benchmark_data_smithi159_162467_object31659:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f97ea7c2:::benchmark_data_smithi159_162467_object60136:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98068a1:::benchmark_data_smithi159_162467_object28486:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9829923:::benchmark_data_smithi159_162467_object26199:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9839b2a:::benchmark_data_smithi159_162467_object23690:head# 2024-04-15T18:43:14.713 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98495bd:::benchmark_data_smithi159_162467_object53946:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98597d8:::benchmark_data_smithi159_162467_object72556:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9865d97:::benchmark_data_smithi159_162467_object19232:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98970d3:::benchmark_data_smithi159_162467_object74794:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f989add8:::benchmark_data_smithi159_162467_object48051:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98d0751:::benchmark_data_smithi159_162467_object7342:head# 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.714 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98d35af:::benchmark_data_smithi159_162467_object11157:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98d88fd:::benchmark_data_smithi159_162467_object31214:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98da52f:::benchmark_data_smithi159_162467_object1370:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f98e6c4c:::benchmark_data_smithi159_162467_object25827:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9903b46:::benchmark_data_smithi159_162467_object58035:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f990f4d2:::benchmark_data_smithi159_162467_object50261:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99132b3:::benchmark_data_smithi159_162467_object281:head# 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.715 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99170c3:::benchmark_data_smithi159_162467_object59975:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f991b7a0:::benchmark_data_smithi159_162467_object64621:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f991e153:::benchmark_data_smithi159_162467_object3267:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f994c976:::benchmark_data_smithi159_162467_object31222:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9960c93:::benchmark_data_smithi159_162467_object47661:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9969068:::benchmark_data_smithi159_162467_object60058:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9970e49:::benchmark_data_smithi159_162467_object70612:head# 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.716 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9994834:::benchmark_data_smithi159_162467_object19025:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99a0998:::benchmark_data_smithi159_162467_object51887:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99ac566:::benchmark_data_smithi159_162467_object10554:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99c740b:::benchmark_data_smithi159_162467_object46528:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99d0a4d:::benchmark_data_smithi159_162467_object22423:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99d19b9:::benchmark_data_smithi159_162467_object4599:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99de360:::benchmark_data_smithi159_162467_object59939:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f99e5899:::benchmark_data_smithi159_162467_object36984:head# 2024-04-15T18:43:14.717 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a0e3b3:::benchmark_data_smithi159_162467_object72397:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a22cb7:::benchmark_data_smithi159_162467_object31685:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a32dd3:::benchmark_data_smithi159_162467_object11024:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a3dac0:::benchmark_data_smithi159_162467_object24321:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a414ae:::benchmark_data_smithi159_162467_object62772:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a5e2bb:::benchmark_data_smithi159_162467_object53021:head# 2024-04-15T18:43:14.718 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a62e35:::benchmark_data_smithi159_162467_object17999:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a649a8:::benchmark_data_smithi159_162467_object44374:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9a96189:::benchmark_data_smithi159_162467_object3489:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9aa0429:::benchmark_data_smithi159_162467_object51268:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9aa82b7:::benchmark_data_smithi159_162467_object32886:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9aec671:::benchmark_data_smithi159_162467_object21179:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9af46bb:::benchmark_data_smithi159_162467_object59113:head# 2024-04-15T18:43:14.719 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9af98c7:::benchmark_data_smithi159_162467_object41590:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b17395:::benchmark_data_smithi159_162467_object39277:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b1e24b:::benchmark_data_smithi159_162467_object42520:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b49689:::benchmark_data_smithi159_162467_object51850:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b4be36:::benchmark_data_smithi159_162467_object30944:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b5adff:::benchmark_data_smithi159_162467_object3568:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b78b4d:::benchmark_data_smithi159_162467_object49855:head# 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.720 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9b7fba7:::benchmark_data_smithi159_162467_object38665:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9bd3e58:::benchmark_data_smithi159_162467_object43452:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9beb936:::benchmark_data_smithi159_162467_object37091:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9bfb0ec:::benchmark_data_smithi159_162467_object14383:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9bfef7f:::benchmark_data_smithi159_162467_object30347:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9c1cdda:::benchmark_data_smithi159_162467_object18686:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9c4952b:::benchmark_data_smithi159_162467_object17917:head# 2024-04-15T18:43:14.721 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9c653d5:::benchmark_data_smithi159_162467_object29138:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9c9ec6a:::benchmark_data_smithi159_162467_object40707:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ca9eef:::benchmark_data_smithi159_162467_object60674:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cafe85:::benchmark_data_smithi159_162467_object46188:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cb56ea:::benchmark_data_smithi159_162467_object8141:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cc6070:::benchmark_data_smithi159_162467_object36668:head# 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.722 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cced08:::benchmark_data_smithi159_162467_object31693:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cd08c9:::benchmark_data_smithi159_162467_object9269:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cd460e:::benchmark_data_smithi159_162467_object45549:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9cf32f5:::benchmark_data_smithi159_162467_object18811:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9d3e528:::benchmark_data_smithi159_162467_object54047:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9d4eaea:::benchmark_data_smithi159_162467_object9745:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9d55810:::benchmark_data_smithi159_162467_object27801:head# 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.723 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9d5b9bd:::benchmark_data_smithi159_162467_object71564:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9d773a6:::benchmark_data_smithi159_162467_object805:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dade5f:::benchmark_data_smithi159_162467_object57045:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9db456e:::benchmark_data_smithi159_162467_object13969:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9db71e5:::benchmark_data_smithi159_162467_object63056:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dbba79:::benchmark_data_smithi159_162467_object44025:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dc90b0:::benchmark_data_smithi159_162467_object40378:head# 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.724 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dcfc7d:::benchmark_data_smithi159_162467_object44854:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dd9bbc:::benchmark_data_smithi159_162467_object23989:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9df90f3:::benchmark_data_smithi159_162467_object71419:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dfd7f0:::benchmark_data_smithi159_162467_object53253:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9dfefee:::benchmark_data_smithi159_162467_object33291:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e0bbcc:::benchmark_data_smithi159_162467_object24440:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e0c840:::benchmark_data_smithi159_162467_object37822:head# 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.725 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e200ed:::benchmark_data_smithi159_162467_object72261:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e21bbd:::benchmark_data_smithi159_162467_object37828:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e38189:::benchmark_data_smithi159_162467_object61667:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e41eba:::benchmark_data_smithi159_162467_object71819:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e5ca7d:::benchmark_data_smithi159_162467_object67492:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e5df71:::benchmark_data_smithi159_162467_object3211:head# 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.726 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e7ec3d:::benchmark_data_smithi159_162467_object38713:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9e8b518:::benchmark_data_smithi159_162467_object42982:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ea2116:::benchmark_data_smithi159_162467_object24389:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ea625b:::benchmark_data_smithi159_162467_object31634:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9eacbf6:::benchmark_data_smithi159_162467_object11302:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ec20bd:::benchmark_data_smithi159_162467_object49320:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ec3c5d:::benchmark_data_smithi159_162467_object25193:head# 2024-04-15T18:43:14.727 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ec8172:::benchmark_data_smithi159_162467_object42177:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9ed78e5:::benchmark_data_smithi159_162467_object71751:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f1bfac:::benchmark_data_smithi159_162467_object11137:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f3b7db:::benchmark_data_smithi159_162467_object4189:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f3d9ee:::benchmark_data_smithi159_162467_object14703:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f52b84:::benchmark_data_smithi159_162467_object2173:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f67ca8:::benchmark_data_smithi159_162467_object9177:head# 2024-04-15T18:43:14.728 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f760b9:::benchmark_data_smithi159_162467_object50217:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f816a9:::benchmark_data_smithi159_162467_object13071:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f84bf4:::benchmark_data_smithi159_162467_object74142:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f87d3d:::benchmark_data_smithi159_162467_object38037:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f91a37:::benchmark_data_smithi159_162467_object43565:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9f9cb51:::benchmark_data_smithi159_162467_object52339:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fabd93:::benchmark_data_smithi159_162467_object68251:head# 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.729 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fb2aee:::benchmark_data_smithi159_162467_object29573:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fbd435:::benchmark_data_smithi159_162467_object57360:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fbda4c:::benchmark_data_smithi159_162467_object2622:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fc27cc:::benchmark_data_smithi159_162467_object47598:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fc6a2e:::benchmark_data_smithi159_162467_object67420:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fc76c5:::benchmark_data_smithi159_162467_object42708:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fd7b2d:::benchmark_data_smithi159_162467_object55410:head# 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.730 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:f9fe5e5f:::benchmark_data_smithi159_162467_object38357:head# 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa014177:::benchmark_data_smithi159_162467_object7567:head# 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0165cc:::benchmark_data_smithi159_162467_object9334:head# 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa01a290:::benchmark_data_smithi159_162467_object8561:head# 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa041694:::benchmark_data_smithi159_162467_object58083:head# 2024-04-15T18:43:14.731 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa04743a:::benchmark_data_smithi159_162467_object15654:head# 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa04b4cc:::benchmark_data_smithi159_162467_object29349:head# 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa055631:::benchmark_data_smithi159_162467_object4286:head# 2024-04-15T18:43:14.761 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0570b5:::benchmark_data_smithi159_162467_object54794:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa069450:::benchmark_data_smithi159_162467_object47965:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa077a9d:::benchmark_data_smithi159_162467_object40150:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa08b396:::benchmark_data_smithi159_162467_object17389:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0a2c2e:::benchmark_data_smithi159_162467_object41615:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0a455f:::benchmark_data_smithi159_162467_object34097:head# 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.762 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0aa74e:::benchmark_data_smithi159_162467_object56952:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0ac18b:::benchmark_data_smithi159_162467_object5345:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0b5015:::benchmark_data_smithi159_162467_object35096:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0bb287:::benchmark_data_smithi159_162467_object34617:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0ca419:::benchmark_data_smithi159_162467_object24717:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0d1f93:::benchmark_data_smithi159_162467_object50949:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa0d2ad9:::benchmark_data_smithi159_162467_object52473:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa107a49:::benchmark_data_smithi159_162467_object58:head# 2024-04-15T18:43:14.763 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa10e7ee:::benchmark_data_smithi159_162467_object53882:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa13049c:::benchmark_data_smithi159_162467_object70283:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa140cc5:::benchmark_data_smithi159_162467_object13091:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa14d0f6:::benchmark_data_smithi159_162467_object71188:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa14e707:::benchmark_data_smithi159_162467_object71720:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa19b72a:::benchmark_data_smithi159_162467_object46361:head# 2024-04-15T18:43:14.764 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1c19ff:::benchmark_data_smithi159_162467_object8429:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1caa8d:::benchmark_data_smithi159_162467_object4256:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1d61fb:::benchmark_data_smithi159_162467_object37512:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1e5506:::benchmark_data_smithi159_162467_object66747:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1e6c97:::benchmark_data_smithi159_162467_object5608:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa1eeafa:::benchmark_data_smithi159_162467_object60362:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa206c84:::benchmark_data_smithi159_162467_object17472:head# 2024-04-15T18:43:14.765 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa21609b:::benchmark_data_smithi159_162467_object11348:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa21a262:::benchmark_data_smithi159_162467_object67778:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa21e227:::benchmark_data_smithi159_162467_object66904:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa24bdd7:::benchmark_data_smithi159_162467_object66249:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa24f371:::benchmark_data_smithi159_162467_object64432:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa254f63:::benchmark_data_smithi159_162467_object16666:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa255f23:::benchmark_data_smithi159_162467_object4331:head# 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa256bc4:::benchmark_data_smithi159_162467_object56444:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa25c0e6:::benchmark_data_smithi159_162467_object59794:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa25d079:::benchmark_data_smithi159_162467_object34785:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa26b36f:::benchmark_data_smithi159_162467_object10108:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa26c0ab:::benchmark_data_smithi159_162467_object31448:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa272c23:::benchmark_data_smithi159_162467_object6501:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2a05c8:::benchmark_data_smithi159_162467_object13283:head# 2024-04-15T18:43:14.767 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2aeac2:::benchmark_data_smithi159_162467_object9866:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2b6516:::benchmark_data_smithi159_162467_object40749:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2b7028:::benchmark_data_smithi159_162467_object23167:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2c02be:::benchmark_data_smithi159_162467_object18318:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2d7515:::benchmark_data_smithi159_162467_object57409:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa2fac24:::benchmark_data_smithi159_162467_object22226:head# 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.768 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa31b38b:::benchmark_data_smithi159_162467_object10966:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa33b0ca:::benchmark_data_smithi159_162467_object3687:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa343278:::benchmark_data_smithi159_162467_object37622:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa35ac56:::benchmark_data_smithi159_162467_object72837:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa360818:::benchmark_data_smithi159_162467_object26698:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3714de:::benchmark_data_smithi159_162467_object11902:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3730a1:::benchmark_data_smithi159_162467_object69316:head# 2024-04-15T18:43:14.769 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa386649:::benchmark_data_smithi159_162467_object51193:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa39d57d:::benchmark_data_smithi159_162467_object70568:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa39dbad:::benchmark_data_smithi159_162467_object38285:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3a5f5a:::benchmark_data_smithi159_162467_object21950:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3af852:::benchmark_data_smithi159_162467_object41000:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3bb2a4:::benchmark_data_smithi159_162467_object25998:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3c6f52:::benchmark_data_smithi159_162467_object6759:head# 2024-04-15T18:43:14.770 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3c8bc2:::benchmark_data_smithi159_162467_object20213:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3cfdeb:::benchmark_data_smithi159_162467_object49580:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa3e0c95:::benchmark_data_smithi159_162467_object39287:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa408fdc:::benchmark_data_smithi159_162467_object45668:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa4470fc:::benchmark_data_smithi159_162467_object41223:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa448c29:::benchmark_data_smithi159_162467_object35538:head# 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.771 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa45ddca:::benchmark_data_smithi159_162467_object71756:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa45df7e:::benchmark_data_smithi159_162467_object61990:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa467a64:::benchmark_data_smithi159_162467_object14061:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa4709bf:::benchmark_data_smithi159_162467_object32364:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa47dc93:::benchmark_data_smithi159_162467_object63753:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa48016c:::benchmark_data_smithi159_162467_object56287:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa49888f:::benchmark_data_smithi159_162467_object44837:head# 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.772 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa4b7880:::benchmark_data_smithi159_162467_object14949:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa4f7971:::benchmark_data_smithi159_162467_object51692:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa4fae0d:::benchmark_data_smithi159_162467_object56623:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa53b8c7:::benchmark_data_smithi159_162467_object8810:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa545fd4:::benchmark_data_smithi159_162467_object20830:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa558120:::benchmark_data_smithi159_162467_object20017:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa55e728:::benchmark_data_smithi159_162467_object28711:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa56b26c:::benchmark_data_smithi159_162467_object63065:head# 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.773 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa56e54d:::benchmark_data_smithi159_162467_object54398:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa584387:::benchmark_data_smithi159_162467_object20775:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa588f9a:::benchmark_data_smithi159_162467_object66311:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa58d715:::benchmark_data_smithi159_162467_object27956:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5aec6c:::benchmark_data_smithi159_162467_object17848:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5c622f:::benchmark_data_smithi159_162467_object67663:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5cb0c6:::benchmark_data_smithi159_162467_object33767:head# 2024-04-15T18:43:14.774 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5d12da:::benchmark_data_smithi159_162467_object33322:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5d3149:::benchmark_data_smithi159_162467_object21837:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5d6b23:::benchmark_data_smithi159_162467_object15953:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5e9bed:::benchmark_data_smithi159_162467_object47862:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5f7bc6:::benchmark_data_smithi159_162467_object33528:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa5fb720:::benchmark_data_smithi159_162467_object36134:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa600f1f:::benchmark_data_smithi159_162467_object18362:head# 2024-04-15T18:43:14.775 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa60dde8:::benchmark_data_smithi159_162467_object38379:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa612781:::benchmark_data_smithi159_162467_object53940:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa61acef:::benchmark_data_smithi159_162467_object57270:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa623d76:::benchmark_data_smithi159_162467_object62995:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa65f5ec:::benchmark_data_smithi159_162467_object30021:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa667d69:::benchmark_data_smithi159_162467_object44786:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6851da:::benchmark_data_smithi159_162467_object58339:head# 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.776 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa695584:::benchmark_data_smithi159_162467_object13977:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6968ed:::benchmark_data_smithi159_162467_object17712:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa699474:::benchmark_data_smithi159_162467_object32288:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa69ce82:::benchmark_data_smithi159_162467_object74726:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6a3122:::benchmark_data_smithi159_162467_object32312:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6aedc9:::benchmark_data_smithi159_162467_object20957:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6b1e42:::benchmark_data_smithi159_162467_object37641:head# 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.777 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6b3172:::benchmark_data_smithi159_162467_object21074:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6b4ecd:::benchmark_data_smithi159_162467_object15003:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6b8a09:::benchmark_data_smithi159_162467_object73200:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6c50f4:::benchmark_data_smithi159_162467_object72902:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6cc904:::benchmark_data_smithi159_162467_object50745:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6d78c4:::benchmark_data_smithi159_162467_object58774:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6f0b91:::benchmark_data_smithi159_162467_object39382:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6f276d:::benchmark_data_smithi159_162467_object18364:head# 2024-04-15T18:43:14.778 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa6f2b47:::benchmark_data_smithi159_162467_object71347:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa712601:::benchmark_data_smithi159_162467_object64197:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7236f6:::benchmark_data_smithi159_162467_object11422:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa72a2e2:::benchmark_data_smithi159_162467_object51223:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa72a7d8:::benchmark_data_smithi159_162467_object7101:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa72eece:::benchmark_data_smithi159_162467_object30381:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa75f493:::benchmark_data_smithi159_162467_object66083:head# 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.779 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa771421:::benchmark_data_smithi159_162467_object19305:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa775af8:::benchmark_data_smithi159_162467_object21751:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa77b0a0:::benchmark_data_smithi159_162467_object8630:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa78823b:::benchmark_data_smithi159_162467_object25458:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa78c4a3:::benchmark_data_smithi159_162467_object17170:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa795aa9:::benchmark_data_smithi159_162467_object3226:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7a82aa:::benchmark_data_smithi159_162467_object29184:head# 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.780 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7b1596:::benchmark_data_smithi159_162467_object46166:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7d0b83:::benchmark_data_smithi159_162467_object17:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7d3eb2:::benchmark_data_smithi159_162467_object55139:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7e10ae:::benchmark_data_smithi159_162467_object63424:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7e25d9:::benchmark_data_smithi159_162467_object6184:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7e68b4:::benchmark_data_smithi159_162467_object45998:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7f00c9:::benchmark_data_smithi159_162467_object4950:head# 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.781 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa7f76d8:::benchmark_data_smithi159_162467_object49570:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa801e7d:::benchmark_data_smithi159_162467_object40677:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8036f2:::benchmark_data_smithi159_162467_object54266:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa810fe3:::benchmark_data_smithi159_162467_object29600:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa815fe5:::benchmark_data_smithi159_162467_object69491:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa830178:::benchmark_data_smithi159_162467_object22863:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa835966:::benchmark_data_smithi159_162467_object45421:head# 2024-04-15T18:43:14.782 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa844dc3:::benchmark_data_smithi159_162467_object57569:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8675c1:::benchmark_data_smithi159_162467_object20647:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa875b28:::benchmark_data_smithi159_162467_object23996:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8850ec:::benchmark_data_smithi159_162467_object42790:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8906bd:::benchmark_data_smithi159_162467_object30160:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa893bd3:::benchmark_data_smithi159_162467_object7917:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8a42c6:::benchmark_data_smithi159_162467_object40606:head# 2024-04-15T18:43:14.783 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8b220e:::benchmark_data_smithi159_162467_object55040:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8b7227:::benchmark_data_smithi159_162467_object67013:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8b827a:::benchmark_data_smithi159_162467_object12008:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8d0675:::benchmark_data_smithi159_162467_object1047:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8d674f:::benchmark_data_smithi159_162467_object50470:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8ea5d0:::benchmark_data_smithi159_162467_object47689:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8edb33:::benchmark_data_smithi159_162467_object74323:head# 2024-04-15T18:43:14.784 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa8f5000:::benchmark_data_smithi159_162467_object23337:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa910eb2:::benchmark_data_smithi159_162467_object62260:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa912f8e:::benchmark_data_smithi159_162467_object31649:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa92c42d:::benchmark_data_smithi159_162467_object1790:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa95570e:::benchmark_data_smithi159_162467_object32916:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa956c3d:::benchmark_data_smithi159_162467_object224:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa960c35:::benchmark_data_smithi159_162467_object50930:head# 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.785 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa979422:::benchmark_data_smithi159_162467_object48701:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa98034e:::benchmark_data_smithi159_162467_object26395:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa989fae:::benchmark_data_smithi159_162467_object58548:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa99975c:::benchmark_data_smithi159_162467_object67506:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9a2676:::benchmark_data_smithi159_162467_object31820:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9b42a2:::benchmark_data_smithi159_162467_object4417:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9b694f:::benchmark_data_smithi159_162467_object34513:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9c323e:::benchmark_data_smithi159_162467_object16962:head# 2024-04-15T18:43:14.786 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9c32d5:::benchmark_data_smithi159_162467_object37424:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9c4927:::benchmark_data_smithi159_162467_object72651:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9de552:::benchmark_data_smithi159_162467_object72127:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9f3c1e:::benchmark_data_smithi159_162467_object22954:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fa9fdf2b:::benchmark_data_smithi159_162467_object49323:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa02a9c:::benchmark_data_smithi159_162467_object26288:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa1bb6f:::benchmark_data_smithi159_162467_object61141:head# 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.787 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa31525:::benchmark_data_smithi159_162467_object20954:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa33615:::benchmark_data_smithi159_162467_object40446:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa4151f:::benchmark_data_smithi159_162467_object30006:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa4a85a:::benchmark_data_smithi159_162467_object1614:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa5636d:::benchmark_data_smithi159_162467_object40031:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa5d1ae:::benchmark_data_smithi159_162467_object26697:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa77890:::benchmark_data_smithi159_162467_object2613:head# 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.788 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa8304e:::benchmark_data_smithi159_162467_object43473:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faa90b26:::benchmark_data_smithi159_162467_object74701:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faab8d76:::benchmark_data_smithi159_162467_object23136:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faabfecc:::benchmark_data_smithi159_162467_object52695:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faac794e:::benchmark_data_smithi159_162467_object61750:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faad50e9:::benchmark_data_smithi159_162467_object5403:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faade5b6:::benchmark_data_smithi159_162467_object74001:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faaff3ab:::benchmark_data_smithi159_162467_object53472:head# 2024-04-15T18:43:14.789 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab106a8:::benchmark_data_smithi159_162467_object63477:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab30fce:::benchmark_data_smithi159_162467_object66625:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab43e32:::benchmark_data_smithi159_162467_object37954:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab4c75c:::benchmark_data_smithi159_162467_object69802:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab6ef06:::benchmark_data_smithi159_162467_object36429:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab89afb:::benchmark_data_smithi159_162467_object3846:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab992c5:::benchmark_data_smithi159_162467_object2467:head# 2024-04-15T18:43:14.790 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fab9c17b:::benchmark_data_smithi159_162467_object30195:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fabaf665:::benchmark_data_smithi159_162467_object36143:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fabc110d:::benchmark_data_smithi159_162467_object48383:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fabd9ba5:::benchmark_data_smithi159_162467_object42208:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fabdec57:::benchmark_data_smithi159_162467_object32605:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac41a41:::benchmark_data_smithi159_162467_object23414:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac47db3:::benchmark_data_smithi159_162467_object73877:head# 2024-04-15T18:43:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac4dec6:::benchmark_data_smithi159_162467_object10982:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac55f96:::benchmark_data_smithi159_162467_object65503:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac5d3da:::benchmark_data_smithi159_162467_object73866:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac6310f:::benchmark_data_smithi159_162467_object54539:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac6891b:::benchmark_data_smithi159_162467_object56040:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac6f935:::benchmark_data_smithi159_162467_object14248:head# 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.792 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac800ec:::benchmark_data_smithi159_162467_object49632:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac8d2c7:::benchmark_data_smithi159_162467_object5226:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fac8d449:::benchmark_data_smithi159_162467_object53296:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:facb17f2:::benchmark_data_smithi159_162467_object6462:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:facb60f7:::benchmark_data_smithi159_162467_object22179:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:facd8783:::benchmark_data_smithi159_162467_object46160:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faceeb7c:::benchmark_data_smithi159_162467_object18285:head# 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.793 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:facf014a:::benchmark_data_smithi159_162467_object60709:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:facf85d7:::benchmark_data_smithi159_162467_object68784:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad04f7d:::benchmark_data_smithi159_162467_object46646:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad05417:::benchmark_data_smithi159_162467_object57192:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad0e782:::benchmark_data_smithi159_162467_object37163:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad1d254:::benchmark_data_smithi159_162467_object27718:head# 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.794 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad29bfb:::benchmark_data_smithi159_162467_object1800:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad35d4a:::benchmark_data_smithi159_162467_object21930:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad3f111:::benchmark_data_smithi159_162467_object66503:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad408fb:::benchmark_data_smithi159_162467_object43742:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad60338:::benchmark_data_smithi159_162467_object4566:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad60a17:::benchmark_data_smithi159_162467_object25495:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad65c79:::benchmark_data_smithi159_162467_object57921:head# 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.795 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad68de8:::benchmark_data_smithi159_162467_object34049:head# 2024-04-15T18:43:14.796 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.796 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad6c403:::benchmark_data_smithi159_162467_object41255:head# 2024-04-15T18:43:14.796 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.796 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad7e318:::benchmark_data_smithi159_162467_object53069:head# 2024-04-15T18:43:14.796 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.805 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad7fed9:::benchmark_data_smithi159_162467_object3794:head# 2024-04-15T18:43:14.805 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.805 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad80554:::benchmark_data_smithi159_162467_object26817:head# 2024-04-15T18:43:14.805 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fad9b2b6:::benchmark_data_smithi159_162467_object72545:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadbaf9f:::benchmark_data_smithi159_162467_object61807:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadc07f5:::benchmark_data_smithi159_162467_object67479:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadc0c63:::benchmark_data_smithi159_162467_object17898:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadc53e5:::benchmark_data_smithi159_162467_object22154:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadc9bdd:::benchmark_data_smithi159_162467_object51130:head# 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.806 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fade5ff0:::benchmark_data_smithi159_162467_object36606:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadf5a80:::benchmark_data_smithi159_162467_object46885:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fadfaa56:::benchmark_data_smithi159_162467_object74023:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae120e1:::benchmark_data_smithi159_162467_object51681:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae2b593:::benchmark_data_smithi159_162467_object42294:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae34abe:::benchmark_data_smithi159_162467_object20233:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae65f1b:::benchmark_data_smithi159_162467_object20024:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae7026f:::benchmark_data_smithi159_162467_object66057:head# 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.807 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae7253e:::benchmark_data_smithi159_162467_object9323:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae757b4:::benchmark_data_smithi159_162467_object43405:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae7e21a:::benchmark_data_smithi159_162467_object49823:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae7e729:::benchmark_data_smithi159_162467_object49692:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae7e75f:::benchmark_data_smithi159_162467_object11449:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fae991fa:::benchmark_data_smithi159_162467_object45232:head# 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.808 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faeadde6:::benchmark_data_smithi159_162467_object12792:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faeb5421:::benchmark_data_smithi159_162467_object67783:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faec47ef:::benchmark_data_smithi159_162467_object50615:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faeddf23:::benchmark_data_smithi159_162467_object4252:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faedec3b:::benchmark_data_smithi159_162467_object11365:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faeeed06:::benchmark_data_smithi159_162467_object30659:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faef5bb6:::benchmark_data_smithi159_162467_object51400:head# 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.809 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf0abb2:::benchmark_data_smithi159_162467_object23704:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf0e849:::benchmark_data_smithi159_162467_object24729:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf16dd0:::benchmark_data_smithi159_162467_object17879:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf2621e:::benchmark_data_smithi159_162467_object7551:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf2ad67:::benchmark_data_smithi159_162467_object16828:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf2b7c9:::benchmark_data_smithi159_162467_object4106:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf34865:::benchmark_data_smithi159_162467_object18498:head# 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.810 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf368bc:::benchmark_data_smithi159_162467_object9669:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf524c2:::benchmark_data_smithi159_162467_object9419:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf66539:::benchmark_data_smithi159_162467_object13295:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf8e84e:::benchmark_data_smithi159_162467_object57434:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf8e889:::benchmark_data_smithi159_162467_object25973:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faf9cbc4:::benchmark_data_smithi159_162467_object23218:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafa8dad:::benchmark_data_smithi159_162467_object24935:head# 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.811 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafafef0:::benchmark_data_smithi159_162467_object11020:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafb03c2:::benchmark_data_smithi159_162467_object51737:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafc96e1:::benchmark_data_smithi159_162467_object1083:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafe0cf1:::benchmark_data_smithi159_162467_object61236:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafe2405:::benchmark_data_smithi159_162467_object8182:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fafeb876:::benchmark_data_smithi159_162467_object61462:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:faff04a8:::benchmark_data_smithi159_162467_object16077:head# 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.812 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb015003:::benchmark_data_smithi159_162467_object37016:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb05a6c8:::benchmark_data_smithi159_162467_object72109:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb05ffcc:::benchmark_data_smithi159_162467_object41405:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb066ec7:::benchmark_data_smithi159_162467_object73634:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb06778f:::benchmark_data_smithi159_162467_object67977:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb08c42b:::benchmark_data_smithi159_162467_object49611:head# 2024-04-15T18:43:14.813 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb093d35:::benchmark_data_smithi159_162467_object27495:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb098ffb:::benchmark_data_smithi159_162467_object33438:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb09c15c:::benchmark_data_smithi159_162467_object18130:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0a1d30:::benchmark_data_smithi159_162467_object25956:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0b3364:::benchmark_data_smithi159_162467_object2083:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0c12ec:::benchmark_data_smithi159_162467_object45218:head# 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.814 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0c2c55:::benchmark_data_smithi159_162467_object16799:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0d294b:::benchmark_data_smithi159_162467_object15259:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0dadbf:::benchmark_data_smithi159_162467_object16105:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0e55b2:::benchmark_data_smithi159_162467_object63815:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0e96d7:::benchmark_data_smithi159_162467_object20464:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0fb6d1:::benchmark_data_smithi159_162467_object50441:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0fe585:::benchmark_data_smithi159_162467_object6643:head# 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.815 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb0ff86f:::benchmark_data_smithi159_162467_object35671:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1063a6:::benchmark_data_smithi159_162467_object37881:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb11222b:::benchmark_data_smithi159_162467_object67915:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb115deb:::benchmark_data_smithi159_162467_object18590:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb11c7fe:::benchmark_data_smithi159_162467_object16232:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb11c80c:::benchmark_data_smithi159_162467_object53497:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb11ee23:::benchmark_data_smithi159_162467_object25848:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb13caf0:::benchmark_data_smithi159_162467_object65584:head# 2024-04-15T18:43:14.816 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb163f1e:::benchmark_data_smithi159_162467_object54978:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb185fb0:::benchmark_data_smithi159_162467_object36058:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb19f7a3:::benchmark_data_smithi159_162467_object25255:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1afcab:::benchmark_data_smithi159_162467_object32111:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1b47bd:::benchmark_data_smithi159_162467_object68761:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1b5aec:::benchmark_data_smithi159_162467_object46658:head# 2024-04-15T18:43:14.817 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1bd0ff:::benchmark_data_smithi159_162467_object37937:head# 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1dcf70:::benchmark_data_smithi159_162467_object64111:head# 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1ddd42:::benchmark_data_smithi159_162467_object2447:head# 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.818 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1ed7d3:::benchmark_data_smithi159_162467_object43110:head# 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb1ef171:::benchmark_data_smithi159_162467_object54281:head# 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb206db5:::benchmark_data_smithi159_162467_object36951:head# 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb22671d:::benchmark_data_smithi159_162467_object71732:head# 2024-04-15T18:43:14.819 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2280f6:::benchmark_data_smithi159_162467_object60386:head# 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb237b0f:::benchmark_data_smithi159_162467_object46094:head# 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb267b0a:::benchmark_data_smithi159_162467_object12785:head# 2024-04-15T18:43:14.820 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2694f3:::benchmark_data_smithi159_162467_object1947:head# 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb26d4b3:::benchmark_data_smithi159_162467_object27798:head# 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb26d69a:::benchmark_data_smithi159_162467_object2652:head# 2024-04-15T18:43:14.821 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb28e4ed:::benchmark_data_smithi159_162467_object11315:head# 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2904b7:::benchmark_data_smithi159_162467_object15542:head# 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2982bf:::benchmark_data_smithi159_162467_object30141:head# 2024-04-15T18:43:14.822 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb29db14:::benchmark_data_smithi159_162467_object63329:head# 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2ab101:::benchmark_data_smithi159_162467_object54548:head# 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2aed08:::benchmark_data_smithi159_162467_object16226:head# 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.823 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2b747e:::benchmark_data_smithi159_162467_object60412:head# 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2c22be:::benchmark_data_smithi159_162467_object21970:head# 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2cbc50:::benchmark_data_smithi159_162467_object25086:head# 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2d4480:::benchmark_data_smithi159_162467_object68219:head# 2024-04-15T18:43:14.824 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2d4c60:::benchmark_data_smithi159_162467_object73262:head# 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2e5a94:::benchmark_data_smithi159_162467_object62761:head# 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2efb26:::benchmark_data_smithi159_162467_object71103:head# 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.825 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2f75af:::benchmark_data_smithi159_162467_object54763:head# 2024-04-15T18:43:14.826 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.826 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb2ff4ed:::benchmark_data_smithi159_162467_object33739:head# 2024-04-15T18:43:14.826 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb30fe4d:::benchmark_data_smithi159_162467_object72806:head# 2024-04-15T18:43:14.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.878 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb310fdf:::benchmark_data_smithi159_162467_object47259:head# 2024-04-15T18:43:14.878 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb315ec5:::benchmark_data_smithi159_162467_object1990:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb31e90b:::benchmark_data_smithi159_162467_object16034:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb330c14:::benchmark_data_smithi159_162467_object36600:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb33547d:::benchmark_data_smithi159_162467_object58260:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb36136d:::benchmark_data_smithi159_162467_object35546:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb36c9fc:::benchmark_data_smithi159_162467_object40477:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb36cddf:::benchmark_data_smithi159_162467_object65110:head# 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.879 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb38edcf:::benchmark_data_smithi159_162467_object53949:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb391064:::benchmark_data_smithi159_162467_object6459:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb39d41c:::benchmark_data_smithi159_162467_object50592:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb3abe2f:::benchmark_data_smithi159_162467_object31334:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb3d1e28:::benchmark_data_smithi159_162467_object57477:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb3eb825:::benchmark_data_smithi159_162467_object38748:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb40216f:::benchmark_data_smithi159_162467_object5827:head# 2024-04-15T18:43:14.880 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb403dca:::benchmark_data_smithi159_162467_object27637:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb408431:::benchmark_data_smithi159_162467_object56607:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb41306a:::benchmark_data_smithi159_162467_object68988:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb41d3ec:::benchmark_data_smithi159_162467_object73967:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb422489:::benchmark_data_smithi159_162467_object19807:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb48e07b:::benchmark_data_smithi159_162467_object13523:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb4b5ad5:::benchmark_data_smithi159_162467_object21126:head# 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.881 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb4c78a9:::benchmark_data_smithi159_162467_object52202:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb4d3f8f:::benchmark_data_smithi159_162467_object49942:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb4d5d63:::benchmark_data_smithi159_162467_object42940:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb501bf5:::benchmark_data_smithi159_162467_object65601:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb50975e:::benchmark_data_smithi159_162467_object53244:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb52563b:::benchmark_data_smithi159_162467_object32179:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb55a814:::benchmark_data_smithi159_162467_object50155:head# 2024-04-15T18:43:14.882 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb56fa66:::benchmark_data_smithi159_162467_object56396:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5711cc:::benchmark_data_smithi159_162467_object23783:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb576f8a:::benchmark_data_smithi159_162467_object38915:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb577648:::benchmark_data_smithi159_162467_object44892:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5857d4:::benchmark_data_smithi159_162467_object18059:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5872a3:::benchmark_data_smithi159_162467_object56453:head# 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.883 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb58a9f1:::benchmark_data_smithi159_162467_object37936:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5a8b6a:::benchmark_data_smithi159_162467_object24544:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5ac7fc:::benchmark_data_smithi159_162467_object11369:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5ae452:::benchmark_data_smithi159_162467_object31637:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5ef0a8:::benchmark_data_smithi159_162467_object72850:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb5f5f9c:::benchmark_data_smithi159_162467_object67984:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb616f85:::benchmark_data_smithi159_162467_object35465:head# 2024-04-15T18:43:14.884 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb61d44e:::benchmark_data_smithi159_162467_object67940:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb62230c:::benchmark_data_smithi159_162467_object30832:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb622774:::benchmark_data_smithi159_162467_object13712:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb62e097:::benchmark_data_smithi159_162467_object23630:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb63a41c:::benchmark_data_smithi159_162467_object28801:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6605cb:::benchmark_data_smithi159_162467_object67308:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb671ccf:::benchmark_data_smithi159_162467_object27317:head# 2024-04-15T18:43:14.885 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb673900:::benchmark_data_smithi159_162467_object12780:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb67bbd4:::benchmark_data_smithi159_162467_object66740:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb67f99a:::benchmark_data_smithi159_162467_object30233:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb680ff9:::benchmark_data_smithi159_162467_object72019:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb683496:::benchmark_data_smithi159_162467_object8995:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb687f4b:::benchmark_data_smithi159_162467_object8706:head# 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.886 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb69eff6:::benchmark_data_smithi159_162467_object57543:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6a4d1d:::benchmark_data_smithi159_162467_object70082:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6a4f07:::benchmark_data_smithi159_162467_object41065:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6aff23:::benchmark_data_smithi159_162467_object38562:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6d26f8:::benchmark_data_smithi159_162467_object47281:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6e49cc:::benchmark_data_smithi159_162467_object39256:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb6ee442:::benchmark_data_smithi159_162467_object35368:head# 2024-04-15T18:43:14.887 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb71215a:::benchmark_data_smithi159_162467_object25163:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb713944:::benchmark_data_smithi159_162467_object50090:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb71600d:::benchmark_data_smithi159_162467_object14938:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb71ca2f:::benchmark_data_smithi159_162467_object55802:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb727cd4:::benchmark_data_smithi159_162467_object16132:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb746f19:::benchmark_data_smithi159_162467_object47964:head# 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.888 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb76c150:::benchmark_data_smithi159_162467_object17669:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb77a396:::benchmark_data_smithi159_162467_object19309:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb793607:::benchmark_data_smithi159_162467_object38955:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7aab5c:::benchmark_data_smithi159_162467_object20273:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7ac3b1:::benchmark_data_smithi159_162467_object32699:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7b58fa:::benchmark_data_smithi159_162467_object42280:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7d35ff:::benchmark_data_smithi159_162467_object28553:head# 2024-04-15T18:43:14.889 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7f971c:::benchmark_data_smithi159_162467_object46795:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb7fff21:::benchmark_data_smithi159_162467_object44603:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8157c1:::benchmark_data_smithi159_162467_object31972:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb837e18:::benchmark_data_smithi159_162467_object12726:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb853118:::benchmark_data_smithi159_162467_object53634:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb85fcef:::benchmark_data_smithi159_162467_object31496:head# 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.890 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb89999e:::benchmark_data_smithi159_162467_object24420:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8b2f30:::benchmark_data_smithi159_162467_object4540:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8bd6da:::benchmark_data_smithi159_162467_object2696:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8c59be:::benchmark_data_smithi159_162467_object61989:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8cdc65:::benchmark_data_smithi159_162467_object50116:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8d0813:::benchmark_data_smithi159_162467_object31268:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8e1c24:::benchmark_data_smithi159_162467_object20073:head# 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.891 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8e8641:::benchmark_data_smithi159_162467_object30045:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8f531d:::benchmark_data_smithi159_162467_object27061:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb8fd5ca:::benchmark_data_smithi159_162467_object27016:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb903078:::benchmark_data_smithi159_162467_object48108:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb90a5f1:::benchmark_data_smithi159_162467_object57352:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb90ac56:::benchmark_data_smithi159_162467_object67481:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb90e2d3:::benchmark_data_smithi159_162467_object23451:head# 2024-04-15T18:43:14.892 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb917804:::benchmark_data_smithi159_162467_object2416:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb91da4d:::benchmark_data_smithi159_162467_object52943:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb924be6:::benchmark_data_smithi159_162467_object8785:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9330f0:::benchmark_data_smithi159_162467_object39899:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb94a430:::benchmark_data_smithi159_162467_object201:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb95472e:::benchmark_data_smithi159_162467_object45765:head# 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.893 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb97b9a1:::benchmark_data_smithi159_162467_object4117:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb97cf7e:::benchmark_data_smithi159_162467_object3218:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb984f8f:::benchmark_data_smithi159_162467_object59893:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb99e979:::benchmark_data_smithi159_162467_object41680:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9c57c6:::benchmark_data_smithi159_162467_object2403:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9c860f:::benchmark_data_smithi159_162467_object39705:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9df720:::benchmark_data_smithi159_162467_object39673:head# 2024-04-15T18:43:14.894 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9e19f8:::benchmark_data_smithi159_162467_object15315:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fb9e2d6d:::benchmark_data_smithi159_162467_object9276:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba050a8:::benchmark_data_smithi159_162467_object72765:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba12246:::benchmark_data_smithi159_162467_object38070:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba5393d:::benchmark_data_smithi159_162467_object49475:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba5c1ee:::benchmark_data_smithi159_162467_object55368:head# 2024-04-15T18:43:14.895 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba5ddca:::benchmark_data_smithi159_162467_object27593:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba664fe:::benchmark_data_smithi159_162467_object71746:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba70200:::benchmark_data_smithi159_162467_object26675:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba72047:::benchmark_data_smithi159_162467_object25874:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba745b1:::benchmark_data_smithi159_162467_object38943:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba80fae:::benchmark_data_smithi159_162467_object61006:head# 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.896 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fba8aa6b:::benchmark_data_smithi159_162467_object70138:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbabb489:::benchmark_data_smithi159_162467_object4944:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbac5796:::benchmark_data_smithi159_162467_object71676:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbac94a0:::benchmark_data_smithi159_162467_object40881:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbad5a85:::benchmark_data_smithi159_162467_object32338:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbae117e:::benchmark_data_smithi159_162467_object34450:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbae1f76:::benchmark_data_smithi159_162467_object19945:head# 2024-04-15T18:43:14.897 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbae4e99:::benchmark_data_smithi159_162467_object35189:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb150d1:::benchmark_data_smithi159_162467_object71981:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb26d21:::benchmark_data_smithi159_162467_object33029:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb36c1b:::benchmark_data_smithi159_162467_object62962:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb37719:::benchmark_data_smithi159_162467_object29749:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb42d78:::benchmark_data_smithi159_162467_object14820:head# 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.898 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb454d3:::benchmark_data_smithi159_162467_object16501:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb49bca:::benchmark_data_smithi159_162467_object61333:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb4ec67:::benchmark_data_smithi159_162467_object38443:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb609e6:::benchmark_data_smithi159_162467_object37805:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb7017d:::benchmark_data_smithi159_162467_object68705:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbb7568a:::benchmark_data_smithi159_162467_object32409:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbb204c:::benchmark_data_smithi159_162467_object33654:head# 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.899 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbc2aaf:::benchmark_data_smithi159_162467_object65272:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbc5ff2:::benchmark_data_smithi159_162467_object64775:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbceb77:::benchmark_data_smithi159_162467_object41583:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbcfd1a:::benchmark_data_smithi159_162467_object27060:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbd31dc:::benchmark_data_smithi159_162467_object1095:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbd53b8:::benchmark_data_smithi159_162467_object68902:head# 2024-04-15T18:43:14.900 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbda2bf:::benchmark_data_smithi159_162467_object60153:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbe8692:::benchmark_data_smithi159_162467_object49952:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbef876:::benchmark_data_smithi159_162467_object50675:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbfac99:::benchmark_data_smithi159_162467_object36525:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbbfcba8:::benchmark_data_smithi159_162467_object3708:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc38194:::benchmark_data_smithi159_162467_object492:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc4b8ae:::benchmark_data_smithi159_162467_object1723:head# 2024-04-15T18:43:14.901 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc4e5c5:::benchmark_data_smithi159_162467_object4195:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc50f92:::benchmark_data_smithi159_162467_object47984:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc51901:::benchmark_data_smithi159_162467_object33852:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc570cc:::benchmark_data_smithi159_162467_object61530:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc6604e:::benchmark_data_smithi159_162467_object3771:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc6bfbf:::benchmark_data_smithi159_162467_object19642:head# 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.902 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc772c7:::benchmark_data_smithi159_162467_object54986:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc7c144:::benchmark_data_smithi159_162467_object4619:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc80cbd:::benchmark_data_smithi159_162467_object23924:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc80ccc:::benchmark_data_smithi159_162467_object71635:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc88cff:::benchmark_data_smithi159_162467_object63471:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc9838e:::benchmark_data_smithi159_162467_object29271:head# 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.903 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbc9e5ed:::benchmark_data_smithi159_162467_object8296:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbca3545:::benchmark_data_smithi159_162467_object960:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcbfd7c:::benchmark_data_smithi159_162467_object48109:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcc1b7e:::benchmark_data_smithi159_162467_object69405:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcd0daa:::benchmark_data_smithi159_162467_object44923:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcd2709:::benchmark_data_smithi159_162467_object48362:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcda386:::benchmark_data_smithi159_162467_object41203:head# 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.904 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbce6140:::benchmark_data_smithi159_162467_object55634:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcefab1:::benchmark_data_smithi159_162467_object5068:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcf710e:::benchmark_data_smithi159_162467_object2410:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcf71b3:::benchmark_data_smithi159_162467_object12505:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbcfd5af:::benchmark_data_smithi159_162467_object31263:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd01266:::benchmark_data_smithi159_162467_object9541:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd051b6:::benchmark_data_smithi159_162467_object30531:head# 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.905 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd0cd84:::benchmark_data_smithi159_162467_object10392:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd19290:::benchmark_data_smithi159_162467_object54215:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd19ea2:::benchmark_data_smithi159_162467_object2414:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd1a571:::benchmark_data_smithi159_162467_object25082:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd1d32d:::benchmark_data_smithi159_162467_object23416:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd205db:::benchmark_data_smithi159_162467_object47429:head# 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.906 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd2ae75:::benchmark_data_smithi159_162467_object35453:head# 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd2c9a5:::benchmark_data_smithi159_162467_object23519:head# 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd344ff:::benchmark_data_smithi159_162467_object6165:head# 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd36769:::benchmark_data_smithi159_162467_object63411:head# 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd60d7a:::benchmark_data_smithi159_162467_object40059:head# 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.907 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd6e0fa:::benchmark_data_smithi159_162467_object19868:head# 2024-04-15T18:43:14.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.908 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd80a2a:::benchmark_data_smithi159_162467_object40676:head# 2024-04-15T18:43:14.908 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd82b98:::benchmark_data_smithi159_162467_object15454:head# 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd83fbe:::benchmark_data_smithi159_162467_object33735:head# 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd90f1d:::benchmark_data_smithi159_162467_object7875:head# 2024-04-15T18:43:14.909 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbd93e5c:::benchmark_data_smithi159_162467_object46705:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbda36ea:::benchmark_data_smithi159_162467_object52016:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbda8908:::benchmark_data_smithi159_162467_object18264:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbdad05a:::benchmark_data_smithi159_162467_object50341:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbdb3b1a:::benchmark_data_smithi159_162467_object23805:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbdc5f58:::benchmark_data_smithi159_162467_object57068:head# 2024-04-15T18:43:14.910 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbdca3d5:::benchmark_data_smithi159_162467_object46970:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbddaab0:::benchmark_data_smithi159_162467_object11280:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbddd7bf:::benchmark_data_smithi159_162467_object69364:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbdf2c52:::benchmark_data_smithi159_162467_object21610:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe03a4b:::benchmark_data_smithi159_162467_object36223:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe10bb6:::benchmark_data_smithi159_162467_object50747:head# 2024-04-15T18:43:14.911 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe11192:::benchmark_data_smithi159_162467_object21224:head# 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe237d6:::benchmark_data_smithi159_162467_object57954:head# 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe315a2:::benchmark_data_smithi159_162467_object64105:head# 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe3b3ea:::benchmark_data_smithi159_162467_object32246:head# 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe3edd0:::benchmark_data_smithi159_162467_object70406:head# 2024-04-15T18:43:14.912 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe418c4:::benchmark_data_smithi159_162467_object33971:head# 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe461fc:::benchmark_data_smithi159_162467_object26729:head# 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe5befa:::benchmark_data_smithi159_162467_object23454:head# 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe6b9f5:::benchmark_data_smithi159_162467_object60929:head# 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbe6cc99:::benchmark_data_smithi159_162467_object8386:head# 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.913 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbea7410:::benchmark_data_smithi159_162467_object24527:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbea7c80:::benchmark_data_smithi159_162467_object11978:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbeacdf6:::benchmark_data_smithi159_162467_object53814:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbebd217:::benchmark_data_smithi159_162467_object40119:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbebd8db:::benchmark_data_smithi159_162467_object31321:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbed1370:::benchmark_data_smithi159_162467_object56784:head# 2024-04-15T18:43:14.914 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbed7648:::benchmark_data_smithi159_162467_object12011:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbedd154:::benchmark_data_smithi159_162467_object35561:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbee68db:::benchmark_data_smithi159_162467_object11874:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbefacda:::benchmark_data_smithi159_162467_object72061:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf0a098:::benchmark_data_smithi159_162467_object5724:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf18d6c:::benchmark_data_smithi159_162467_object28650:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf1a821:::benchmark_data_smithi159_162467_object25316:head# 2024-04-15T18:43:14.915 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf26975:::benchmark_data_smithi159_162467_object38184:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf2c591:::benchmark_data_smithi159_162467_object69545:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf3072a:::benchmark_data_smithi159_162467_object60840:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf388b0:::benchmark_data_smithi159_162467_object24732:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf50490:::benchmark_data_smithi159_162467_object33147:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf525d4:::benchmark_data_smithi159_162467_object29388:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf5631a:::benchmark_data_smithi159_162467_object41793:head# 2024-04-15T18:43:14.916 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf66514:::benchmark_data_smithi159_162467_object10056:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf94b43:::benchmark_data_smithi159_162467_object47204:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf96eda:::benchmark_data_smithi159_162467_object52650:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf987f5:::benchmark_data_smithi159_162467_object67088:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbf9d2d6:::benchmark_data_smithi159_162467_object44085:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfa2b94:::benchmark_data_smithi159_162467_object41504:head# 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.917 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfa5540:::benchmark_data_smithi159_162467_object67630:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfbace6:::benchmark_data_smithi159_162467_object36433:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfbb764:::benchmark_data_smithi159_162467_object49399:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfd21ec:::benchmark_data_smithi159_162467_object58510:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfe3d7d:::benchmark_data_smithi159_162467_object69110:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fbfebb7a:::benchmark_data_smithi159_162467_object5575:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc01283e:::benchmark_data_smithi159_162467_object25503:head# 2024-04-15T18:43:14.918 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc02f3d9:::benchmark_data_smithi159_162467_object53647:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc03fe6d:::benchmark_data_smithi159_162467_object12834:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0492ea:::benchmark_data_smithi159_162467_object7377:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc04c884:::benchmark_data_smithi159_162467_object12605:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc067bce:::benchmark_data_smithi159_162467_object21867:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc06fa92:::benchmark_data_smithi159_162467_object36993:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc08d1be:::benchmark_data_smithi159_162467_object17031:head# 2024-04-15T18:43:14.919 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc092179:::benchmark_data_smithi159_162467_object25949:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc09d3c1:::benchmark_data_smithi159_162467_object61558:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0acecb:::benchmark_data_smithi159_162467_object37292:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0c0ad3:::benchmark_data_smithi159_162467_object71538:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0cdd9d:::benchmark_data_smithi159_162467_object50236:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0cf170:::benchmark_data_smithi159_162467_object67667:head# 2024-04-15T18:43:14.920 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0d5e96:::benchmark_data_smithi159_162467_object9383:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0d79d6:::benchmark_data_smithi159_162467_object29870:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0e0bf5:::benchmark_data_smithi159_162467_object40591:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0e1888:::benchmark_data_smithi159_162467_object18550:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc0f0081:::benchmark_data_smithi159_162467_object51569:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc110df2:::benchmark_data_smithi159_162467_object25142:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc111535:::benchmark_data_smithi159_162467_object52232:head# 2024-04-15T18:43:14.921 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc12683c:::benchmark_data_smithi159_162467_object23538:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1291e4:::benchmark_data_smithi159_162467_object1972:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc12a05c:::benchmark_data_smithi159_162467_object72476:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc12d169:::benchmark_data_smithi159_162467_object71976:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc12e35a:::benchmark_data_smithi159_162467_object20494:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc135e59:::benchmark_data_smithi159_162467_object6727:head# 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.922 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc137e31:::benchmark_data_smithi159_162467_object72358:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc152ac6:::benchmark_data_smithi159_162467_object38545:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc156d5d:::benchmark_data_smithi159_162467_object70831:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc15e9ec:::benchmark_data_smithi159_162467_object116:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1715ca:::benchmark_data_smithi159_162467_object57330:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1846f1:::benchmark_data_smithi159_162467_object16494:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1980df:::benchmark_data_smithi159_162467_object13480:head# 2024-04-15T18:43:14.923 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1a0c35:::benchmark_data_smithi159_162467_object66795:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1a610b:::benchmark_data_smithi159_162467_object9640:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1b0c94:::benchmark_data_smithi159_162467_object5481:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1b5174:::benchmark_data_smithi159_162467_object46991:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1c6b7e:::benchmark_data_smithi159_162467_object40210:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1cd4d1:::benchmark_data_smithi159_162467_object33838:head# 2024-04-15T18:43:14.924 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1d31e8:::benchmark_data_smithi159_162467_object63292:head# 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1f109a:::benchmark_data_smithi159_162467_object67640:head# 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1f1342:::benchmark_data_smithi159_162467_object50480:head# 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc1fa5e9:::benchmark_data_smithi159_162467_object67073:head# 2024-04-15T18:43:14.925 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc201bd5:::benchmark_data_smithi159_162467_object59042:head# 2024-04-15T18:43:14.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.967 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc212ac9:::benchmark_data_smithi159_162467_object6146:head# 2024-04-15T18:43:14.967 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2186ad:::benchmark_data_smithi159_162467_object58285:head# 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc219a4d:::benchmark_data_smithi159_162467_object60965:head# 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc23caa5:::benchmark_data_smithi159_162467_object1330:head# 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2403ed:::benchmark_data_smithi159_162467_object25390:head# 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.968 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc25221f:::benchmark_data_smithi159_162467_object26752:head# 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2568d3:::benchmark_data_smithi159_162467_object51454:head# 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc25b372:::benchmark_data_smithi159_162467_object38221:head# 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc26e6d9:::benchmark_data_smithi159_162467_object20788:head# 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.969 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc273a1f:::benchmark_data_smithi159_162467_object15547:head# 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc278990:::benchmark_data_smithi159_162467_object31735:head# 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2821f7:::benchmark_data_smithi159_162467_object6850:head# 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc286dea:::benchmark_data_smithi159_162467_object20333:head# 2024-04-15T18:43:14.970 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2a4672:::benchmark_data_smithi159_162467_object70519:head# 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2a9358:::benchmark_data_smithi159_162467_object31778:head# 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2b3a98:::benchmark_data_smithi159_162467_object44355:head# 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2b81a2:::benchmark_data_smithi159_162467_object14745:head# 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.971 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2bdd43:::benchmark_data_smithi159_162467_object22857:head# 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2c8fb1:::benchmark_data_smithi159_162467_object58283:head# 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2d95ad:::benchmark_data_smithi159_162467_object7790:head# 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2e41b3:::benchmark_data_smithi159_162467_object59082:head# 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.972 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2f77bd:::benchmark_data_smithi159_162467_object73929:head# 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2fb535:::benchmark_data_smithi159_162467_object24137:head# 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc2fd022:::benchmark_data_smithi159_162467_object14622:head# 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc308093:::benchmark_data_smithi159_162467_object21533:head# 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.973 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc30839b:::benchmark_data_smithi159_162467_object17301:head# 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc31920f:::benchmark_data_smithi159_162467_object23860:head# 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc31c6af:::benchmark_data_smithi159_162467_object41827:head# 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc33527d:::benchmark_data_smithi159_162467_object54444:head# 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.974 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc33da04:::benchmark_data_smithi159_162467_object39449:head# 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc347984:::benchmark_data_smithi159_162467_object14592:head# 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3653b3:::benchmark_data_smithi159_162467_object43935:head# 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc375904:::benchmark_data_smithi159_162467_object59518:head# 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.975 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc387a81:::benchmark_data_smithi159_162467_object63126:head# 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3ae33c:::benchmark_data_smithi159_162467_object16824:head# 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3b0ebf:::benchmark_data_smithi159_162467_object19398:head# 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3bbd4a:::benchmark_data_smithi159_162467_object51221:head# 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.976 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3ce787:::benchmark_data_smithi159_162467_object64699:head# 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3e5e07:::benchmark_data_smithi159_162467_object3963:head# 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc3f0a74:::benchmark_data_smithi159_162467_object29610:head# 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc40273d:::benchmark_data_smithi159_162467_object46224:head# 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.977 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc40878d:::benchmark_data_smithi159_162467_object63074:head# 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc40a179:::benchmark_data_smithi159_162467_object56699:head# 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4115fb:::benchmark_data_smithi159_162467_object32209:head# 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc41a095:::benchmark_data_smithi159_162467_object11528:head# 2024-04-15T18:43:14.978 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc42dfdd:::benchmark_data_smithi159_162467_object32544:head# 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc435411:::benchmark_data_smithi159_162467_object43783:head# 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc443b77:::benchmark_data_smithi159_162467_object15291:head# 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc446af2:::benchmark_data_smithi159_162467_object16842:head# 2024-04-15T18:43:14.979 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc453a2d:::benchmark_data_smithi159_162467_object39621:head# 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc45ebf1:::benchmark_data_smithi159_162467_object17068:head# 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc46cf4b:::benchmark_data_smithi159_162467_object25308:head# 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc46d7aa:::benchmark_data_smithi159_162467_object39268:head# 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.980 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc47513f:::benchmark_data_smithi159_162467_object3735:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc47e32c:::benchmark_data_smithi159_162467_object14217:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc47fbf9:::benchmark_data_smithi159_162467_object2531:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc491312:::benchmark_data_smithi159_162467_object20125:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc491b8d:::benchmark_data_smithi159_162467_object29325:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4a0841:::benchmark_data_smithi159_162467_object4362:head# 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.981 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4a805a:::benchmark_data_smithi159_162467_object15929:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4b635b:::benchmark_data_smithi159_162467_object52516:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4b92f1:::benchmark_data_smithi159_162467_object49589:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4d2489:::benchmark_data_smithi159_162467_object19836:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4d674b:::benchmark_data_smithi159_162467_object13537:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4da1fd:::benchmark_data_smithi159_162467_object545:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4e1283:::benchmark_data_smithi159_162467_object47808:head# 2024-04-15T18:43:14.982 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4e8b56:::benchmark_data_smithi159_162467_object6802:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4ee582:::benchmark_data_smithi159_162467_object1636:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4f1aab:::benchmark_data_smithi159_162467_object29128:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc4f2a35:::benchmark_data_smithi159_162467_object33877:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc516322:::benchmark_data_smithi159_162467_object59576:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc528ac5:::benchmark_data_smithi159_162467_object10172:head# 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.983 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc534be1:::benchmark_data_smithi159_162467_object62828:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc550388:::benchmark_data_smithi159_162467_object71337:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc55831c:::benchmark_data_smithi159_162467_object71952:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5623e9:::benchmark_data_smithi159_162467_object42561:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5633f6:::benchmark_data_smithi159_162467_object11177:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5642d3:::benchmark_data_smithi159_162467_object3714:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc56d17a:::benchmark_data_smithi159_162467_object26550:head# 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.984 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc571dee:::benchmark_data_smithi159_162467_object16084:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc577c3c:::benchmark_data_smithi159_162467_object43805:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc58d08a:::benchmark_data_smithi159_162467_object14823:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5a377f:::benchmark_data_smithi159_162467_object47432:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5a3f3d:::benchmark_data_smithi159_162467_object38772:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5a4084:::benchmark_data_smithi159_162467_object72478:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5ae26e:::benchmark_data_smithi159_162467_object15868:head# 2024-04-15T18:43:14.985 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5b4c34:::benchmark_data_smithi159_162467_object47303:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5c0682:::benchmark_data_smithi159_162467_object30115:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5e7f32:::benchmark_data_smithi159_162467_object29403:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5e875b:::benchmark_data_smithi159_162467_object8213:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc5ef077:::benchmark_data_smithi159_162467_object49962:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6067b5:::benchmark_data_smithi159_162467_object29489:head# 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.986 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc620aa5:::benchmark_data_smithi159_162467_object40987:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc62da9b:::benchmark_data_smithi159_162467_object58859:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc63e854:::benchmark_data_smithi159_162467_object63855:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6439e9:::benchmark_data_smithi159_162467_object54576:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc655588:::benchmark_data_smithi159_162467_object23034:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc656d2e:::benchmark_data_smithi159_162467_object61384:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc659deb:::benchmark_data_smithi159_162467_object47458:head# 2024-04-15T18:43:14.987 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc666b26:::benchmark_data_smithi159_162467_object74331:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc668b56:::benchmark_data_smithi159_162467_object9033:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc678716:::benchmark_data_smithi159_162467_object74758:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc67bf05:::benchmark_data_smithi159_162467_object17667:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc68e7b6:::benchmark_data_smithi159_162467_object2011:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc694386:::benchmark_data_smithi159_162467_object11883:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc69eb1a:::benchmark_data_smithi159_162467_object3391:head# 2024-04-15T18:43:14.988 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6add30:::benchmark_data_smithi159_162467_object32210:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6bdb91:::benchmark_data_smithi159_162467_object2176:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6c0c5b:::benchmark_data_smithi159_162467_object16830:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc6f994c:::benchmark_data_smithi159_162467_object65703:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc70f37d:::benchmark_data_smithi159_162467_object45037:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc711bf0:::benchmark_data_smithi159_162467_object13893:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc73942a:::benchmark_data_smithi159_162467_object5069:head# 2024-04-15T18:43:14.989 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc739809:::benchmark_data_smithi159_162467_object69587:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc74c893:::benchmark_data_smithi159_162467_object55523:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc755145:::benchmark_data_smithi159_162467_object42267:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc759f9d:::benchmark_data_smithi159_162467_object24445:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7613bf:::benchmark_data_smithi159_162467_object42357:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc76ecb8:::benchmark_data_smithi159_162467_object29040:head# 2024-04-15T18:43:14.990 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7831d5:::benchmark_data_smithi159_162467_object47461:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc79f84a:::benchmark_data_smithi159_162467_object55326:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7bf0e4:::benchmark_data_smithi159_162467_object400:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7c21f0:::benchmark_data_smithi159_162467_object60891:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7c4f04:::benchmark_data_smithi159_162467_object18747:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7cd119:::benchmark_data_smithi159_162467_object65485:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7cd9f8:::benchmark_data_smithi159_162467_object39064:head# 2024-04-15T18:43:14.991 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7d06b3:::benchmark_data_smithi159_162467_object65391:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7d0ae8:::benchmark_data_smithi159_162467_object62321:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc7ef3f7:::benchmark_data_smithi159_162467_object27082:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc801888:::benchmark_data_smithi159_162467_object34915:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc823d2f:::benchmark_data_smithi159_162467_object27308:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc82d8ac:::benchmark_data_smithi159_162467_object7686:head# 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc82d989:::benchmark_data_smithi159_162467_object53923:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc834c05:::benchmark_data_smithi159_162467_object1406:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc84291a:::benchmark_data_smithi159_162467_object43593:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc84ad08:::benchmark_data_smithi159_162467_object13022:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc84d8d0:::benchmark_data_smithi159_162467_object32930:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc869075:::benchmark_data_smithi159_162467_object53943:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc889e24:::benchmark_data_smithi159_162467_object74199:head# 2024-04-15T18:43:14.993 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8a1897:::benchmark_data_smithi159_162467_object51786:head# 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8b60c3:::benchmark_data_smithi159_162467_object34598:head# 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8b6390:::benchmark_data_smithi159_162467_object25627:head# 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8c8830:::benchmark_data_smithi159_162467_object7671:head# 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8cfa67:::benchmark_data_smithi159_162467_object3635:head# 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.994 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8d7a1d:::benchmark_data_smithi159_162467_object72840:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8dadcc:::benchmark_data_smithi159_162467_object71515:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8e5937:::benchmark_data_smithi159_162467_object45666:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc8f96fe:::benchmark_data_smithi159_162467_object11181:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc904fee:::benchmark_data_smithi159_162467_object67827:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9059e6:::benchmark_data_smithi159_162467_object18287:head# 2024-04-15T18:43:14.995 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9165df:::benchmark_data_smithi159_162467_object42602:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc924581:::benchmark_data_smithi159_162467_object17528:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9255d2:::benchmark_data_smithi159_162467_object53305:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc936071:::benchmark_data_smithi159_162467_object36788:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc952c3e:::benchmark_data_smithi159_162467_object59161:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc95cd95:::benchmark_data_smithi159_162467_object66105:head# 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc963495:::benchmark_data_smithi159_162467_object20983:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9658ed:::benchmark_data_smithi159_162467_object16496:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc969b97:::benchmark_data_smithi159_162467_object20617:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc97d9a8:::benchmark_data_smithi159_162467_object30086:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc995458:::benchmark_data_smithi159_162467_object35611:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9a6b28:::benchmark_data_smithi159_162467_object12323:head# 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.997 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9b0bc2:::benchmark_data_smithi159_162467_object32140:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9c3518:::benchmark_data_smithi159_162467_object57872:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9d5a7d:::benchmark_data_smithi159_162467_object46905:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9db09a:::benchmark_data_smithi159_162467_object22669:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9f6c10:::benchmark_data_smithi159_162467_object50823:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fc9f6c1e:::benchmark_data_smithi159_162467_object21567:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca0f5c8:::benchmark_data_smithi159_162467_object18760:head# 2024-04-15T18:43:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca1ba55:::benchmark_data_smithi159_162467_object6890:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca31f0d:::benchmark_data_smithi159_162467_object34475:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca413d8:::benchmark_data_smithi159_162467_object72448:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca42759:::benchmark_data_smithi159_162467_object19662:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca4ad8e:::benchmark_data_smithi159_162467_object23506:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca4de9e:::benchmark_data_smithi159_162467_object14186:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca52474:::benchmark_data_smithi159_162467_object59682:head# 2024-04-15T18:43:14.999 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fca77b1c:::benchmark_data_smithi159_162467_object70546:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcaad9df:::benchmark_data_smithi159_162467_object56847:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcab67d5:::benchmark_data_smithi159_162467_object72259:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcad0584:::benchmark_data_smithi159_162467_object43210:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcae0547:::benchmark_data_smithi159_162467_object37334:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcae6da2:::benchmark_data_smithi159_162467_object31876:head# 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.000 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcaf83fa:::benchmark_data_smithi159_162467_object57692:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb04c26:::benchmark_data_smithi159_162467_object8619:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb167dd:::benchmark_data_smithi159_162467_object46673:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb1bda1:::benchmark_data_smithi159_162467_object47573:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb287ee:::benchmark_data_smithi159_162467_object25722:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb40041:::benchmark_data_smithi159_162467_object20766:head# 2024-04-15T18:43:15.001 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb624ab:::benchmark_data_smithi159_162467_object69515:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb77ed5:::benchmark_data_smithi159_162467_object73550:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb8ecf1:::benchmark_data_smithi159_162467_object6755:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcb90dc4:::benchmark_data_smithi159_162467_object48889:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcbbbe05:::benchmark_data_smithi159_162467_object30593:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcbcd820:::benchmark_data_smithi159_162467_object33750:head# 2024-04-15T18:43:15.002 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcbdfb0b:::benchmark_data_smithi159_162467_object12687:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcbe5739:::benchmark_data_smithi159_162467_object49980:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc037cf:::benchmark_data_smithi159_162467_object32506:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc127c8:::benchmark_data_smithi159_162467_object43187:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc1d974:::benchmark_data_smithi159_162467_object39104:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc359e6:::benchmark_data_smithi159_162467_object37497:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc35c59:::benchmark_data_smithi159_162467_object67859:head# 2024-04-15T18:43:15.003 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc77149:::benchmark_data_smithi159_162467_object31005:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcc965eb:::benchmark_data_smithi159_162467_object29164:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcca1d3f:::benchmark_data_smithi159_162467_object49128:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcca591b:::benchmark_data_smithi159_162467_object3250:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcca6c33:::benchmark_data_smithi159_162467_object41365:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcccdf44:::benchmark_data_smithi159_162467_object9646:head# 2024-04-15T18:43:15.004 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fccd2327:::benchmark_data_smithi159_162467_object29246:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fccd72b7:::benchmark_data_smithi159_162467_object19837:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fccffd40:::benchmark_data_smithi159_162467_object5421:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd2360e:::benchmark_data_smithi159_162467_object7951:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd2adef:::benchmark_data_smithi159_162467_object30252:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd33a8d:::benchmark_data_smithi159_162467_object38286:head# 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.005 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd4ef85:::benchmark_data_smithi159_162467_object44336:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd5b79f:::benchmark_data_smithi159_162467_object46931:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd6cc62:::benchmark_data_smithi159_162467_object4477:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd7cd66:::benchmark_data_smithi159_162467_object71631:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcd8115f:::benchmark_data_smithi159_162467_object11788:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcdae68c:::benchmark_data_smithi159_162467_object6966:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcdb14a6:::benchmark_data_smithi159_162467_object61517:head# 2024-04-15T18:43:15.006 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcdb793a:::benchmark_data_smithi159_162467_object40845:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcdce341:::benchmark_data_smithi159_162467_object45386:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcdd65ac:::benchmark_data_smithi159_162467_object71105:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce0f5f4:::benchmark_data_smithi159_162467_object724:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce11dcd:::benchmark_data_smithi159_162467_object50973:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce3c387:::benchmark_data_smithi159_162467_object34288:head# 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.007 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce4612a:::benchmark_data_smithi159_162467_object40271:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce4a8c8:::benchmark_data_smithi159_162467_object897:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce50523:::benchmark_data_smithi159_162467_object19519:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce61f7a:::benchmark_data_smithi159_162467_object19081:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce74c09:::benchmark_data_smithi159_162467_object72304:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce772fb:::benchmark_data_smithi159_162467_object71847:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce77d57:::benchmark_data_smithi159_162467_object22225:head# 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.008 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fce7c9ca:::benchmark_data_smithi159_162467_object15896:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcea2873:::benchmark_data_smithi159_162467_object41707:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcea7f8c:::benchmark_data_smithi159_162467_object65163:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcea8274:::benchmark_data_smithi159_162467_object45096:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fceb8e85:::benchmark_data_smithi159_162467_object44680:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fceb9804:::benchmark_data_smithi159_162467_object16922:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fced38f8:::benchmark_data_smithi159_162467_object61219:head# 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.009 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcee97a2:::benchmark_data_smithi159_162467_object73992:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fceee72f:::benchmark_data_smithi159_162467_object68675:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcef5962:::benchmark_data_smithi159_162467_object1899:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcefac36:::benchmark_data_smithi159_162467_object3257:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf0bc90:::benchmark_data_smithi159_162467_object30438:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf13e15:::benchmark_data_smithi159_162467_object5838:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf19c43:::benchmark_data_smithi159_162467_object45227:head# 2024-04-15T18:43:15.010 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf1d329:::benchmark_data_smithi159_162467_object52321:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf1ece8:::benchmark_data_smithi159_162467_object64627:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf2922a:::benchmark_data_smithi159_162467_object14482:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf384fa:::benchmark_data_smithi159_162467_object41967:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf8cb5c:::benchmark_data_smithi159_162467_object45627:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf8e5ce:::benchmark_data_smithi159_162467_object66882:head# 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.011 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcf9cd68:::benchmark_data_smithi159_162467_object51860:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcfa9759:::benchmark_data_smithi159_162467_object38434:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcfa9ff8:::benchmark_data_smithi159_162467_object10495:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcfcccaf:::benchmark_data_smithi159_162467_object41964:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcfda1fe:::benchmark_data_smithi159_162467_object5696:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcfde141:::benchmark_data_smithi159_162467_object24619:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fcff7ec1:::benchmark_data_smithi159_162467_object20211:head# 2024-04-15T18:43:15.012 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd004552:::benchmark_data_smithi159_162467_object53426:head# 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd00ea13:::benchmark_data_smithi159_162467_object57997:head# 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd01d2cb:::benchmark_data_smithi159_162467_object11160:head# 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd024338:::benchmark_data_smithi159_162467_object54080:head# 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd024feb:::benchmark_data_smithi159_162467_object50238:head# 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.013 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd043536:::benchmark_data_smithi159_162467_object45115:head# 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0454fa:::benchmark_data_smithi159_162467_object51691:head# 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0530bf:::benchmark_data_smithi159_162467_object14098:head# 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd05624c:::benchmark_data_smithi159_162467_object55646:head# 2024-04-15T18:43:15.014 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd066390:::benchmark_data_smithi159_162467_object14752:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd07ab56:::benchmark_data_smithi159_162467_object49309:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd081a05:::benchmark_data_smithi159_162467_object66716:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd08dc36:::benchmark_data_smithi159_162467_object52625:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd097574:::benchmark_data_smithi159_162467_object48128:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0aa865:::benchmark_data_smithi159_162467_object70358:head# 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.015 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0af543:::benchmark_data_smithi159_162467_object9291:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0bfc2f:::benchmark_data_smithi159_162467_object74201:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0cb433:::benchmark_data_smithi159_162467_object46098:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0d72dc:::benchmark_data_smithi159_162467_object69687:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0d8a61:::benchmark_data_smithi159_162467_object10372:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0edaed:::benchmark_data_smithi159_162467_object6694:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd0f2bc7:::benchmark_data_smithi159_162467_object10561:head# 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.016 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd113511:::benchmark_data_smithi159_162467_object15797:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd114050:::benchmark_data_smithi159_162467_object10231:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd124c36:::benchmark_data_smithi159_162467_object38478:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd12a482:::benchmark_data_smithi159_162467_object6720:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd133e4b:::benchmark_data_smithi159_162467_object66276:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd14b322:::benchmark_data_smithi159_162467_object38780:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd166034:::benchmark_data_smithi159_162467_object6370:head# 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.017 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd169cd6:::benchmark_data_smithi159_162467_object56974:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd18f144:::benchmark_data_smithi159_162467_object61161:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1916d2:::benchmark_data_smithi159_162467_object57065:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1a0bbe:::benchmark_data_smithi159_162467_object3870:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1acb6b:::benchmark_data_smithi159_162467_object45527:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1c6d4a:::benchmark_data_smithi159_162467_object30971:head# 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.018 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1d861e:::benchmark_data_smithi159_162467_object57754:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd1d9f86:::benchmark_data_smithi159_162467_object7028:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd223a94:::benchmark_data_smithi159_162467_object14881:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd232c23:::benchmark_data_smithi159_162467_object68893:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2392cb:::benchmark_data_smithi159_162467_object20038:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2618af:::benchmark_data_smithi159_162467_object51174:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2685bd:::benchmark_data_smithi159_162467_object62455:head# 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.019 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd283469:::benchmark_data_smithi159_162467_object45997:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd291d88:::benchmark_data_smithi159_162467_object74227:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd29b566:::benchmark_data_smithi159_162467_object68024:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2abae5:::benchmark_data_smithi159_162467_object7485:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2ac10d:::benchmark_data_smithi159_162467_object49601:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2ca58b:::benchmark_data_smithi159_162467_object68850:head# 2024-04-15T18:43:15.020 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.075 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2ce5d9:::benchmark_data_smithi159_162467_object37854:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2d2f33:::benchmark_data_smithi159_162467_object49186:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2d72bd:::benchmark_data_smithi159_162467_object37241:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd2d776d:::benchmark_data_smithi159_162467_object67818:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd324d19:::benchmark_data_smithi159_162467_object58077:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd33f6a4:::benchmark_data_smithi159_162467_object30707:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd34b262:::benchmark_data_smithi159_162467_object65420:head# 2024-04-15T18:43:15.076 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd34fc33:::benchmark_data_smithi159_162467_object15297:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3516de:::benchmark_data_smithi159_162467_object57490:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd355024:::benchmark_data_smithi159_162467_object23310:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd35c19e:::benchmark_data_smithi159_162467_object73570:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3625e1:::benchmark_data_smithi159_162467_object21701:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd370e9e:::benchmark_data_smithi159_162467_object49909:head# 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.077 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd37b69a:::benchmark_data_smithi159_162467_object50767:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd38f004:::benchmark_data_smithi159_162467_object68899:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd38f5ea:::benchmark_data_smithi159_162467_object2886:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd39f648:::benchmark_data_smithi159_162467_object72369:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3b20e0:::benchmark_data_smithi159_162467_object32226:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3c62eb:::benchmark_data_smithi159_162467_object44729:head# 2024-04-15T18:43:15.078 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3c925f:::benchmark_data_smithi159_162467_object71169:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3ca488:::benchmark_data_smithi159_162467_object63039:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3ce994:::benchmark_data_smithi159_162467_object64669:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3ea609:::benchmark_data_smithi159_162467_object73354:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3f55c2:::benchmark_data_smithi159_162467_object6250:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd3fab9a:::benchmark_data_smithi159_162467_object54491:head# 2024-04-15T18:43:15.079 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd42612f:::benchmark_data_smithi159_162467_object68434:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd42630c:::benchmark_data_smithi159_162467_object49802:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd4460e5:::benchmark_data_smithi159_162467_object36260:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd44fd08:::benchmark_data_smithi159_162467_object12342:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd458259:::benchmark_data_smithi159_162467_object52997:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd498bff:::benchmark_data_smithi159_162467_object31430:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd4b2396:::benchmark_data_smithi159_162467_object64862:head# 2024-04-15T18:43:15.080 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd4d8460:::benchmark_data_smithi159_162467_object25943:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd4f03ea:::benchmark_data_smithi159_162467_object27726:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd50ffb2:::benchmark_data_smithi159_162467_object23564:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5169d6:::benchmark_data_smithi159_162467_object25472:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd52b9f7:::benchmark_data_smithi159_162467_object7155:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd537a8d:::benchmark_data_smithi159_162467_object69126:head# 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.081 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5405c3:::benchmark_data_smithi159_162467_object66831:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd574c2c:::benchmark_data_smithi159_162467_object73913:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd57a3df:::benchmark_data_smithi159_162467_object1465:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd57dbd5:::benchmark_data_smithi159_162467_object58137:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd58ecca:::benchmark_data_smithi159_162467_object3588:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd593bb0:::benchmark_data_smithi159_162467_object25530:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd599883:::benchmark_data_smithi159_162467_object16928:head# 2024-04-15T18:43:15.082 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5aa249:::benchmark_data_smithi159_162467_object11498:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5b425c:::benchmark_data_smithi159_162467_object4679:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5b7c4d:::benchmark_data_smithi159_162467_object64708:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5c0efd:::benchmark_data_smithi159_162467_object50530:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5d989b:::benchmark_data_smithi159_162467_object31612:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5dd0a4:::benchmark_data_smithi159_162467_object23591:head# 2024-04-15T18:43:15.083 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5df891:::benchmark_data_smithi159_162467_object58564:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5e42d6:::benchmark_data_smithi159_162467_object27485:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd5f5061:::benchmark_data_smithi159_162467_object17668:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6051b4:::benchmark_data_smithi159_162467_object7412:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6051fb:::benchmark_data_smithi159_162467_object72080:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd628dff:::benchmark_data_smithi159_162467_object48284:head# 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.084 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd636406:::benchmark_data_smithi159_162467_object41399:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd63b791:::benchmark_data_smithi159_162467_object23450:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd63e851:::benchmark_data_smithi159_162467_object66418:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd650f8a:::benchmark_data_smithi159_162467_object67132:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd65649e:::benchmark_data_smithi159_162467_object22512:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd666628:::benchmark_data_smithi159_162467_object72811:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd66821a:::benchmark_data_smithi159_162467_object67330:head# 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.085 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6695e5:::benchmark_data_smithi159_162467_object50660:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd66c34b:::benchmark_data_smithi159_162467_object59654:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd678447:::benchmark_data_smithi159_162467_object34352:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd682254:::benchmark_data_smithi159_162467_object16539:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd687bf7:::benchmark_data_smithi159_162467_object55168:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6ae3e4:::benchmark_data_smithi159_162467_object65452:head# 2024-04-15T18:43:15.086 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6b27c9:::benchmark_data_smithi159_162467_object52586:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6b3e9a:::benchmark_data_smithi159_162467_object15931:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6b7c90:::benchmark_data_smithi159_162467_object62477:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6c189e:::benchmark_data_smithi159_162467_object19964:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6c2bd4:::benchmark_data_smithi159_162467_object25690:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6cb399:::benchmark_data_smithi159_162467_object55626:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6cd902:::benchmark_data_smithi159_162467_object27809:head# 2024-04-15T18:43:15.087 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6ce29a:::benchmark_data_smithi159_162467_object50173:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6dff87:::benchmark_data_smithi159_162467_object25533:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6e3222:::benchmark_data_smithi159_162467_object40844:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6eab2e:::benchmark_data_smithi159_162467_object39704:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd6eac0c:::benchmark_data_smithi159_162467_object42299:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7053de:::benchmark_data_smithi159_162467_object39815:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd708bc5:::benchmark_data_smithi159_162467_object70398:head# 2024-04-15T18:43:15.088 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7140fd:::benchmark_data_smithi159_162467_object53596:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd718fd3:::benchmark_data_smithi159_162467_object15150:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd721fcd:::benchmark_data_smithi159_162467_object67191:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd72871d:::benchmark_data_smithi159_162467_object66649:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd736571:::benchmark_data_smithi159_162467_object11077:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd768547:::benchmark_data_smithi159_162467_object71176:head# 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.089 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd78cacb:::benchmark_data_smithi159_162467_object13494:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd78d4f0:::benchmark_data_smithi159_162467_object10656:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd78ecf8:::benchmark_data_smithi159_162467_object4336:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7a8006:::benchmark_data_smithi159_162467_object27598:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7a8485:::benchmark_data_smithi159_162467_object38220:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7acb06:::benchmark_data_smithi159_162467_object41323:head# 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.090 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7d7d78:::benchmark_data_smithi159_162467_object11983:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd7e75c5:::benchmark_data_smithi159_162467_object69297:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd80358a:::benchmark_data_smithi159_162467_object72962:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd807a11:::benchmark_data_smithi159_162467_object4781:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd80884d:::benchmark_data_smithi159_162467_object66168:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd80dde2:::benchmark_data_smithi159_162467_object38894:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd818621:::benchmark_data_smithi159_162467_object63525:head# 2024-04-15T18:43:15.091 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd83401d:::benchmark_data_smithi159_162467_object23952:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd84f5a9:::benchmark_data_smithi159_162467_object47101:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd8581de:::benchmark_data_smithi159_162467_object24191:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd85d4b7:::benchmark_data_smithi159_162467_object34839:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd873172:::benchmark_data_smithi159_162467_object41096:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd87c22e:::benchmark_data_smithi159_162467_object73255:head# 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.092 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd8971fb:::benchmark_data_smithi159_162467_object7144:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd8a4b7e:::benchmark_data_smithi159_162467_object63378:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd8d176b:::benchmark_data_smithi159_162467_object24808:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd8f80ea:::benchmark_data_smithi159_162467_object54156:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd901422:::benchmark_data_smithi159_162467_object28410:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd904ba9:::benchmark_data_smithi159_162467_object28247:head# 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.093 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd90ad7f:::benchmark_data_smithi159_162467_object23841:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd911103:::benchmark_data_smithi159_162467_object26357:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9157c2:::benchmark_data_smithi159_162467_object39305:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd92ea1e:::benchmark_data_smithi159_162467_object24155:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9336aa:::benchmark_data_smithi159_162467_object52793:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9371c1:::benchmark_data_smithi159_162467_object1797:head# 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.094 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd94cc12:::benchmark_data_smithi159_162467_object55901:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd950d7b:::benchmark_data_smithi159_162467_object70366:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd95c3ef:::benchmark_data_smithi159_162467_object43482:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd95ced8:::benchmark_data_smithi159_162467_object10610:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9787c2:::benchmark_data_smithi159_162467_object57549:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9796e8:::benchmark_data_smithi159_162467_object17273:head# 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.095 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd987a69:::benchmark_data_smithi159_162467_object12347:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd99af90:::benchmark_data_smithi159_162467_object10419:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd99c808:::benchmark_data_smithi159_162467_object62694:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9a1e45:::benchmark_data_smithi159_162467_object14240:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9a57b5:::benchmark_data_smithi159_162467_object65488:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9c6ff1:::benchmark_data_smithi159_162467_object28027:head# 2024-04-15T18:43:15.096 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9d4b49:::benchmark_data_smithi159_162467_object53171:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9da3d1:::benchmark_data_smithi159_162467_object42295:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9e89a5:::benchmark_data_smithi159_162467_object57143:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9fa463:::benchmark_data_smithi159_162467_object6136:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9fa811:::benchmark_data_smithi159_162467_object14673:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9fc9d9:::benchmark_data_smithi159_162467_object43239:head# 2024-04-15T18:43:15.097 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fd9fdad1:::benchmark_data_smithi159_162467_object47441:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda046a6:::benchmark_data_smithi159_162467_object32999:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda063a5:::benchmark_data_smithi159_162467_object33531:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda0d949:::benchmark_data_smithi159_162467_object63311:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda2212b:::benchmark_data_smithi159_162467_object35347:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda33187:::benchmark_data_smithi159_162467_object8506:head# 2024-04-15T18:43:15.098 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda3402c:::benchmark_data_smithi159_162467_object62042:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda406d1:::benchmark_data_smithi159_162467_object63141:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda4c4a4:::benchmark_data_smithi159_162467_object3079:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda63cd6:::benchmark_data_smithi159_162467_object29105:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda68d6d:::benchmark_data_smithi159_162467_object43017:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda74d10:::benchmark_data_smithi159_162467_object11715:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda8408e:::benchmark_data_smithi159_162467_object36350:head# 2024-04-15T18:43:15.099 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda86d1c:::benchmark_data_smithi159_162467_object4123:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda93114:::benchmark_data_smithi159_162467_object18072:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda957f6:::benchmark_data_smithi159_162467_object2084:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fda9e514:::benchmark_data_smithi159_162467_object63740:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdacf6f6:::benchmark_data_smithi159_162467_object61232:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdad605d:::benchmark_data_smithi159_162467_object9043:head# 2024-04-15T18:43:15.100 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdadc20e:::benchmark_data_smithi159_162467_object46736:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdae7067:::benchmark_data_smithi159_162467_object50768:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdaf3ce7:::benchmark_data_smithi159_162467_object29902:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdaf3fa5:::benchmark_data_smithi159_162467_object3957:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb03631:::benchmark_data_smithi159_162467_object19904:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb2db25:::benchmark_data_smithi159_162467_object36257:head# 2024-04-15T18:43:15.101 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb3b9e5:::benchmark_data_smithi159_162467_object12938:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb68614:::benchmark_data_smithi159_162467_object61757:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb78635:::benchmark_data_smithi159_162467_object35361:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb8b9ca:::benchmark_data_smithi159_162467_object44548:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdb8cc96:::benchmark_data_smithi159_162467_object7692:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdbb3442:::benchmark_data_smithi159_162467_object62601:head# 2024-04-15T18:43:15.102 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc0ebe1:::benchmark_data_smithi159_162467_object39716:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc1cd7c:::benchmark_data_smithi159_162467_object22166:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc294a6:::benchmark_data_smithi159_162467_object52360:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc3136a:::benchmark_data_smithi159_162467_object21516:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc3efb9:::benchmark_data_smithi159_162467_object72495:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc69166:::benchmark_data_smithi159_162467_object32211:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc77e71:::benchmark_data_smithi159_162467_object25194:head# 2024-04-15T18:43:15.103 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc915fb:::benchmark_data_smithi159_162467_object6657:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdc92dda:::benchmark_data_smithi159_162467_object68253:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdca6cd6:::benchmark_data_smithi159_162467_object21022:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcad571:::benchmark_data_smithi159_162467_object67801:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcbe729:::benchmark_data_smithi159_162467_object71013:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcd4444:::benchmark_data_smithi159_162467_object15190:head# 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.104 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcd6921:::benchmark_data_smithi159_162467_object31515:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcf9b89:::benchmark_data_smithi159_162467_object9543:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcfaa7f:::benchmark_data_smithi159_162467_object50720:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdcfe5ff:::benchmark_data_smithi159_162467_object56884:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd19978:::benchmark_data_smithi159_162467_object39368:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd413fc:::benchmark_data_smithi159_162467_object64019:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd44ed3:::benchmark_data_smithi159_162467_object5998:head# 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.105 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd489e9:::benchmark_data_smithi159_162467_object25102:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd588e6:::benchmark_data_smithi159_162467_object58009:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd74a2a:::benchmark_data_smithi159_162467_object61050:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd783d6:::benchmark_data_smithi159_162467_object23917:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd784a6:::benchmark_data_smithi159_162467_object13795:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd7a168:::benchmark_data_smithi159_162467_object42626:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdd89d39:::benchmark_data_smithi159_162467_object33726:head# 2024-04-15T18:43:15.106 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fddb5998:::benchmark_data_smithi159_162467_object16175:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fddc0902:::benchmark_data_smithi159_162467_object57220:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fddc94e9:::benchmark_data_smithi159_162467_object60827:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde0a71c:::benchmark_data_smithi159_162467_object31527:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde3698f:::benchmark_data_smithi159_162467_object30270:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde3f3de:::benchmark_data_smithi159_162467_object47422:head# 2024-04-15T18:43:15.107 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde7b178:::benchmark_data_smithi159_162467_object13065:head# 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde837f7:::benchmark_data_smithi159_162467_object34073:head# 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde87077:::benchmark_data_smithi159_162467_object68636:head# 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde912ec:::benchmark_data_smithi159_162467_object35832:head# 2024-04-15T18:43:15.109 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde97d11:::benchmark_data_smithi159_162467_object65800:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fde9c6e0:::benchmark_data_smithi159_162467_object17980:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdea1cea:::benchmark_data_smithi159_162467_object24189:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdeb569a:::benchmark_data_smithi159_162467_object40160:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdec0638:::benchmark_data_smithi159_162467_object36451:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdec50d9:::benchmark_data_smithi159_162467_object16274:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdee14db:::benchmark_data_smithi159_162467_object53828:head# 2024-04-15T18:43:15.110 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdee9ecc:::benchmark_data_smithi159_162467_object38863:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdef1559:::benchmark_data_smithi159_162467_object60381:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdef3d60:::benchmark_data_smithi159_162467_object31570:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf0a481:::benchmark_data_smithi159_162467_object16847:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf0b705:::benchmark_data_smithi159_162467_object73987:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf0c0b1:::benchmark_data_smithi159_162467_object17038:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf0faab:::benchmark_data_smithi159_162467_object31829:head# 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.111 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf1a779:::benchmark_data_smithi159_162467_object4278:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf1d67b:::benchmark_data_smithi159_162467_object14467:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf22f3d:::benchmark_data_smithi159_162467_object10413:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf281b7:::benchmark_data_smithi159_162467_object11970:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf296a6:::benchmark_data_smithi159_162467_object13083:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf3f24e:::benchmark_data_smithi159_162467_object14184:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf5cb4d:::benchmark_data_smithi159_162467_object49487:head# 2024-04-15T18:43:15.112 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf63b7b:::benchmark_data_smithi159_162467_object71228:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf687a9:::benchmark_data_smithi159_162467_object56830:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf71c67:::benchmark_data_smithi159_162467_object57482:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf79a23:::benchmark_data_smithi159_162467_object72022:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf7ecf3:::benchmark_data_smithi159_162467_object39844:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf7f2ce:::benchmark_data_smithi159_162467_object20708:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf870fa:::benchmark_data_smithi159_162467_object6491:head# 2024-04-15T18:43:15.113 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf999ba:::benchmark_data_smithi159_162467_object49502:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdf9a9a5:::benchmark_data_smithi159_162467_object18846:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdfa24af:::benchmark_data_smithi159_162467_object32247:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdfc8681:::benchmark_data_smithi159_162467_object56414:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdfd7904:::benchmark_data_smithi159_162467_object58054:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdfe6c5b:::benchmark_data_smithi159_162467_object16097:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdfeb963:::benchmark_data_smithi159_162467_object64790:head# 2024-04-15T18:43:15.114 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fdffacfb:::benchmark_data_smithi159_162467_object68776:head# 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0085ee:::benchmark_data_smithi159_162467_object14170:head# 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe01d3a5:::benchmark_data_smithi159_162467_object2113:head# 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe029d40:::benchmark_data_smithi159_162467_object60782:head# 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe049914:::benchmark_data_smithi159_162467_object18109:head# 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.115 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe05774a:::benchmark_data_smithi159_162467_object59954:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0630a1:::benchmark_data_smithi159_162467_object34686:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0955ff:::benchmark_data_smithi159_162467_object49532:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe09d84c:::benchmark_data_smithi159_162467_object33507:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0b19df:::benchmark_data_smithi159_162467_object7784:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0bc3e7:::benchmark_data_smithi159_162467_object72865:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0c4275:::benchmark_data_smithi159_162467_object68967:head# 2024-04-15T18:43:15.116 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0c6bad:::benchmark_data_smithi159_162467_object39586:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0cfc68:::benchmark_data_smithi159_162467_object44320:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0d1842:::benchmark_data_smithi159_162467_object27759:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0e48e8:::benchmark_data_smithi159_162467_object17454:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0eb415:::benchmark_data_smithi159_162467_object42539:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0f1501:::benchmark_data_smithi159_162467_object50500:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe0fe01f:::benchmark_data_smithi159_162467_object42807:head# 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.117 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe101d4e:::benchmark_data_smithi159_162467_object21628:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe10a21c:::benchmark_data_smithi159_162467_object47098:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe10d465:::benchmark_data_smithi159_162467_object31290:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe124760:::benchmark_data_smithi159_162467_object24517:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe139dc5:::benchmark_data_smithi159_162467_object4181:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe14ebfa:::benchmark_data_smithi159_162467_object64176:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe153ce6:::benchmark_data_smithi159_162467_object33950:head# 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.118 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe163194:::benchmark_data_smithi159_162467_object48157:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe16543e:::benchmark_data_smithi159_162467_object40301:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe16bc8f:::benchmark_data_smithi159_162467_object47226:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe17bc81:::benchmark_data_smithi159_162467_object57006:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1bed46:::benchmark_data_smithi159_162467_object16223:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1c1c6d:::benchmark_data_smithi159_162467_object73901:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1c45fe:::benchmark_data_smithi159_162467_object177:head# 2024-04-15T18:43:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1ceb71:::benchmark_data_smithi159_162467_object55867:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1d8d78:::benchmark_data_smithi159_162467_object51798:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1f2737:::benchmark_data_smithi159_162467_object13917:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1f3ec1:::benchmark_data_smithi159_162467_object15015:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe1f4c89:::benchmark_data_smithi159_162467_object71643:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2095e5:::benchmark_data_smithi159_162467_object5906:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe240a57:::benchmark_data_smithi159_162467_object1139:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe240c76:::benchmark_data_smithi159_162467_object3424:head# 2024-04-15T18:43:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2537a1:::benchmark_data_smithi159_162467_object42350:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe25c134:::benchmark_data_smithi159_162467_object18794:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe27091d:::benchmark_data_smithi159_162467_object38997:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe27c26d:::benchmark_data_smithi159_162467_object3844:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2883ad:::benchmark_data_smithi159_162467_object63794:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe299725:::benchmark_data_smithi159_162467_object39548:head# 2024-04-15T18:43:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2a2c94:::benchmark_data_smithi159_162467_object69769:head# 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2c022e:::benchmark_data_smithi159_162467_object17082:head# 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2c9000:::benchmark_data_smithi159_162467_object61302:head# 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.122 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe2e4037:::benchmark_data_smithi159_162467_object35966:head# 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3188a1:::benchmark_data_smithi159_162467_object67071:head# 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe31f909:::benchmark_data_smithi159_162467_object62846:head# 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe346f79:::benchmark_data_smithi159_162467_object72424:head# 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe351cff:::benchmark_data_smithi159_162467_object68709:head# 2024-04-15T18:43:15.123 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe358628:::benchmark_data_smithi159_162467_object73853:head# 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe359dc2:::benchmark_data_smithi159_162467_object29773:head# 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe35c2d3:::benchmark_data_smithi159_162467_object10546:head# 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3630e7:::benchmark_data_smithi159_162467_object48874:head# 2024-04-15T18:43:15.124 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe36b69d:::benchmark_data_smithi159_162467_object49183:head# 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe382376:::benchmark_data_smithi159_162467_object38385:head# 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3af978:::benchmark_data_smithi159_162467_object758:head# 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3b27cb:::benchmark_data_smithi159_162467_object42340:head# 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.125 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3ca233:::benchmark_data_smithi159_162467_object60863:head# 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3dbbd7:::benchmark_data_smithi159_162467_object3813:head# 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe3e185b:::benchmark_data_smithi159_162467_object72822:head# 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.126 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe404f27:::benchmark_data_smithi159_162467_object15035:head# 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe40ff4e:::benchmark_data_smithi159_162467_object73307:head# 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe41a9f3:::benchmark_data_smithi159_162467_object38937:head# 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe41d66d:::benchmark_data_smithi159_162467_object35991:head# 2024-04-15T18:43:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe41f0ed:::benchmark_data_smithi159_162467_object16490:head# 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe423a96:::benchmark_data_smithi159_162467_object7351:head# 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4316b9:::benchmark_data_smithi159_162467_object65596:head# 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.128 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4335b1:::benchmark_data_smithi159_162467_object56046:head# 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe43a264:::benchmark_data_smithi159_162467_object62025:head# 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe43c3e5:::benchmark_data_smithi159_162467_object8529:head# 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe43eb1a:::benchmark_data_smithi159_162467_object46962:head# 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe442c47:::benchmark_data_smithi159_162467_object17891:head# 2024-04-15T18:43:15.129 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe44552a:::benchmark_data_smithi159_162467_object35022:head# 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe455260:::benchmark_data_smithi159_162467_object30417:head# 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4b173c:::benchmark_data_smithi159_162467_object66977:head# 2024-04-15T18:43:15.130 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4b6396:::benchmark_data_smithi159_162467_object31252:head# 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4dcfe6:::benchmark_data_smithi159_162467_object39282:head# 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4de8b1:::benchmark_data_smithi159_162467_object20043:head# 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.131 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4e48f1:::benchmark_data_smithi159_162467_object38615:head# 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4e82a7:::benchmark_data_smithi159_162467_object43804:head# 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4eb38c:::benchmark_data_smithi159_162467_object19463:head# 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.132 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4ec0ce:::benchmark_data_smithi159_162467_object14432:head# 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4eca8a:::benchmark_data_smithi159_162467_object49135:head# 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4eff37:::benchmark_data_smithi159_162467_object33088:head# 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe4ff58b:::benchmark_data_smithi159_162467_object16742:head# 2024-04-15T18:43:15.133 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe501f07:::benchmark_data_smithi159_162467_object6619:head# 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe503231:::benchmark_data_smithi159_162467_object71217:head# 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe51f139:::benchmark_data_smithi159_162467_object27040:head# 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe532428:::benchmark_data_smithi159_162467_object71252:head# 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.134 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe554688:::benchmark_data_smithi159_162467_object51301:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe56148d:::benchmark_data_smithi159_162467_object43146:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe56e950:::benchmark_data_smithi159_162467_object42721:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe57acae:::benchmark_data_smithi159_162467_object72924:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe587c16:::benchmark_data_smithi159_162467_object48744:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe589ef7:::benchmark_data_smithi159_162467_object39833:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5b64b1:::benchmark_data_smithi159_162467_object42160:head# 2024-04-15T18:43:15.135 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5c398c:::benchmark_data_smithi159_162467_object62546:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5c6833:::benchmark_data_smithi159_162467_object41390:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5ca259:::benchmark_data_smithi159_162467_object42779:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5cb045:::benchmark_data_smithi159_162467_object64946:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5ecdc5:::benchmark_data_smithi159_162467_object4141:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe5fdab6:::benchmark_data_smithi159_162467_object18335:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe612111:::benchmark_data_smithi159_162467_object70203:head# 2024-04-15T18:43:15.136 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe636c70:::benchmark_data_smithi159_162467_object49327:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe658f8b:::benchmark_data_smithi159_162467_object4871:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe691ff8:::benchmark_data_smithi159_162467_object62602:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6aa9cf:::benchmark_data_smithi159_162467_object46685:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6b7b3f:::benchmark_data_smithi159_162467_object72285:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6c2858:::benchmark_data_smithi159_162467_object39398:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6c9187:::benchmark_data_smithi159_162467_object29183:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6ef2b1:::benchmark_data_smithi159_162467_object74239:head# 2024-04-15T18:43:15.137 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe6f5178:::benchmark_data_smithi159_162467_object2237:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe707dfd:::benchmark_data_smithi159_162467_object31942:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe713489:::benchmark_data_smithi159_162467_object58419:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7142f2:::benchmark_data_smithi159_162467_object50423:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe718efb:::benchmark_data_smithi159_162467_object10346:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe71b1d9:::benchmark_data_smithi159_162467_object62687:head# 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.138 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe723778:::benchmark_data_smithi159_162467_object66828:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7381a9:::benchmark_data_smithi159_162467_object73845:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe746d18:::benchmark_data_smithi159_162467_object54304:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe747460:::benchmark_data_smithi159_162467_object69524:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe74805d:::benchmark_data_smithi159_162467_object63973:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe755ea6:::benchmark_data_smithi159_162467_object1300:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe79374b:::benchmark_data_smithi159_162467_object41851:head# 2024-04-15T18:43:15.139 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe797f34:::benchmark_data_smithi159_162467_object21839:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe79a329:::benchmark_data_smithi159_162467_object31048:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7a03a7:::benchmark_data_smithi159_162467_object16511:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7adc6f:::benchmark_data_smithi159_162467_object33290:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7b1928:::benchmark_data_smithi159_162467_object54863:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7b4b5a:::benchmark_data_smithi159_162467_object59505:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7b6ae4:::benchmark_data_smithi159_162467_object42906:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7ecfa8:::benchmark_data_smithi159_162467_object53874:head# 2024-04-15T18:43:15.140 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7ed2c5:::benchmark_data_smithi159_162467_object43298:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe7fc354:::benchmark_data_smithi159_162467_object51835:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe804f94:::benchmark_data_smithi159_162467_object11054:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe810e20:::benchmark_data_smithi159_162467_object22013:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8607b9:::benchmark_data_smithi159_162467_object27168:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe89b6df:::benchmark_data_smithi159_162467_object32285:head# 2024-04-15T18:43:15.141 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe89f56f:::benchmark_data_smithi159_162467_object32139:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8b6090:::benchmark_data_smithi159_162467_object56567:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8cb652:::benchmark_data_smithi159_162467_object66298:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8d4f95:::benchmark_data_smithi159_162467_object72149:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8ea7f6:::benchmark_data_smithi159_162467_object57469:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8ecc09:::benchmark_data_smithi159_162467_object45410:head# 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.142 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8edaca:::benchmark_data_smithi159_162467_object31786:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe8f09b9:::benchmark_data_smithi159_162467_object72549:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe908464:::benchmark_data_smithi159_162467_object65760:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe91eb01:::benchmark_data_smithi159_162467_object46412:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe925ea5:::benchmark_data_smithi159_162467_object55324:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9434ac:::benchmark_data_smithi159_162467_object34849:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe94cedb:::benchmark_data_smithi159_162467_object913:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe962a8b:::benchmark_data_smithi159_162467_object6940:head# 2024-04-15T18:43:15.143 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.144 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe971e86:::benchmark_data_smithi159_162467_object25189:head# 2024-04-15T18:43:15.144 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.192 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe979c2f:::benchmark_data_smithi159_162467_object61871:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9848bc:::benchmark_data_smithi159_162467_object13910:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe990a31:::benchmark_data_smithi159_162467_object58716:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe99185f:::benchmark_data_smithi159_162467_object55087:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe999c47:::benchmark_data_smithi159_162467_object61813:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9a0260:::benchmark_data_smithi159_162467_object4533:head# 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.193 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9ae324:::benchmark_data_smithi159_162467_object14922:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9bc756:::benchmark_data_smithi159_162467_object67982:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9c5410:::benchmark_data_smithi159_162467_object19286:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fe9f47be:::benchmark_data_smithi159_162467_object31531:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea0be38:::benchmark_data_smithi159_162467_object72156:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea1bf9c:::benchmark_data_smithi159_162467_object6817:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea258e7:::benchmark_data_smithi159_162467_object20888:head# 2024-04-15T18:43:15.194 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea2d121:::benchmark_data_smithi159_162467_object67904:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea337cb:::benchmark_data_smithi159_162467_object51004:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea37e12:::benchmark_data_smithi159_162467_object29744:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea715a5:::benchmark_data_smithi159_162467_object35075:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea71d79:::benchmark_data_smithi159_162467_object44956:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea79ca0:::benchmark_data_smithi159_162467_object72055:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea8107d:::benchmark_data_smithi159_162467_object57957:head# 2024-04-15T18:43:15.195 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea8361c:::benchmark_data_smithi159_162467_object12911:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea8ccbc:::benchmark_data_smithi159_162467_object41567:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea8ff76:::benchmark_data_smithi159_162467_object12493:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea92a0f:::benchmark_data_smithi159_162467_object10714:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fea9b193:::benchmark_data_smithi159_162467_object53349:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feaddc79:::benchmark_data_smithi159_162467_object22580:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feae5736:::benchmark_data_smithi159_162467_object4495:head# 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.196 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feae5b43:::benchmark_data_smithi159_162467_object69750:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feafb104:::benchmark_data_smithi159_162467_object47897:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb0c962:::benchmark_data_smithi159_162467_object55414:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb1a506:::benchmark_data_smithi159_162467_object59791:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb1beeb:::benchmark_data_smithi159_162467_object35855:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb265a7:::benchmark_data_smithi159_162467_object64215:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb346d5:::benchmark_data_smithi159_162467_object43592:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb3a988:::benchmark_data_smithi159_162467_object51087:head# 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.197 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb3c7e4:::benchmark_data_smithi159_162467_object49712:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb426a7:::benchmark_data_smithi159_162467_object25096:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb461f7:::benchmark_data_smithi159_162467_object29896:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb6bde9:::benchmark_data_smithi159_162467_object46652:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb6ce70:::benchmark_data_smithi159_162467_object42044:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb7c1dc:::benchmark_data_smithi159_162467_object1880:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb7ed20:::benchmark_data_smithi159_162467_object24662:head# 2024-04-15T18:43:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feb88a9f:::benchmark_data_smithi159_162467_object19140:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feba87a6:::benchmark_data_smithi159_162467_object56246:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febada9d:::benchmark_data_smithi159_162467_object39852:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febb6f26:::benchmark_data_smithi159_162467_object34952:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febb9eb1:::benchmark_data_smithi159_162467_object59962:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febcc80a:::benchmark_data_smithi159_162467_object24053:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febd3a4e:::benchmark_data_smithi159_162467_object12530:head# 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:febdf3f0:::benchmark_data_smithi159_162467_object72666:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec0eaf2:::benchmark_data_smithi159_162467_object14902:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec0faa6:::benchmark_data_smithi159_162467_object68653:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec1839c:::benchmark_data_smithi159_162467_object24259:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec2ba19:::benchmark_data_smithi159_162467_object62050:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec2f4ae:::benchmark_data_smithi159_162467_object36110:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec33c1c:::benchmark_data_smithi159_162467_object71866:head# 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.200 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec499b2:::benchmark_data_smithi159_162467_object7391:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec7ce50:::benchmark_data_smithi159_162467_object57717:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec887ae:::benchmark_data_smithi159_162467_object33662:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fec9f9f4:::benchmark_data_smithi159_162467_object59551:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecae454:::benchmark_data_smithi159_162467_object55000:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecbb576:::benchmark_data_smithi159_162467_object484:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feccceb4:::benchmark_data_smithi159_162467_object17622:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecd874a:::benchmark_data_smithi159_162467_object33784:head# 2024-04-15T18:43:15.201 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecf37b3:::benchmark_data_smithi159_162467_object48245:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecfa35f:::benchmark_data_smithi159_162467_object68944:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fecfe716:::benchmark_data_smithi159_162467_object16335:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed00a47:::benchmark_data_smithi159_162467_object28151:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed0f75b:::benchmark_data_smithi159_162467_object44031:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed1063a:::benchmark_data_smithi159_162467_object19854:head# 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.202 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed1c5e3:::benchmark_data_smithi159_162467_object4670:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed314b2:::benchmark_data_smithi159_162467_object24463:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed40bf1:::benchmark_data_smithi159_162467_object12827:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed41fe9:::benchmark_data_smithi159_162467_object63210:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed466f7:::benchmark_data_smithi159_162467_object49877:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed5ee86:::benchmark_data_smithi159_162467_object55405:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed60214:::benchmark_data_smithi159_162467_object3201:head# 2024-04-15T18:43:15.203 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed6b55c:::benchmark_data_smithi159_162467_object49583:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fed9963f:::benchmark_data_smithi159_162467_object74226:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feda7789:::benchmark_data_smithi159_162467_object5700:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fedb2d04:::benchmark_data_smithi159_162467_object22078:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fedb45b2:::benchmark_data_smithi159_162467_object26255:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fede9505:::benchmark_data_smithi159_162467_object30095:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fedf2e31:::benchmark_data_smithi159_162467_object35519:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fedff0ed:::benchmark_data_smithi159_162467_object49045:head# 2024-04-15T18:43:15.204 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee069eb:::benchmark_data_smithi159_162467_object62445:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee088a9:::benchmark_data_smithi159_162467_object2417:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee1e133:::benchmark_data_smithi159_162467_object12380:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee3a78e:::benchmark_data_smithi159_162467_object14527:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee48834:::benchmark_data_smithi159_162467_object50947:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee5b571:::benchmark_data_smithi159_162467_object9279:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee612e7:::benchmark_data_smithi159_162467_object32232:head# 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.205 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee6b553:::benchmark_data_smithi159_162467_object53361:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee74ca4:::benchmark_data_smithi159_162467_object21110:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee82fa4:::benchmark_data_smithi159_162467_object9021:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee8df8b:::benchmark_data_smithi159_162467_object26120:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee8ec3f:::benchmark_data_smithi159_162467_object55032:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee8f290:::benchmark_data_smithi159_162467_object41362:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee91508:::benchmark_data_smithi159_162467_object64634:head# 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fee9c536:::benchmark_data_smithi159_162467_object1627:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feea01b7:::benchmark_data_smithi159_162467_object8338:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feea5f0a:::benchmark_data_smithi159_162467_object64992:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feeb2d79:::benchmark_data_smithi159_162467_object26047:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feec5096:::benchmark_data_smithi159_162467_object57936:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feed4a6e:::benchmark_data_smithi159_162467_object41688:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feee9cc7:::benchmark_data_smithi159_162467_object940:head# 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.207 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feeea886:::benchmark_data_smithi159_162467_object66766:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feeee089:::benchmark_data_smithi159_162467_object57722:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feef2dbe:::benchmark_data_smithi159_162467_object70895:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feef415c:::benchmark_data_smithi159_162467_object11822:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feef9bcc:::benchmark_data_smithi159_162467_object29197:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef1568e:::benchmark_data_smithi159_162467_object11699:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef28e18:::benchmark_data_smithi159_162467_object64063:head# 2024-04-15T18:43:15.208 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef291d4:::benchmark_data_smithi159_162467_object45141:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef46e2c:::benchmark_data_smithi159_162467_object37174:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef4f837:::benchmark_data_smithi159_162467_object21937:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef559dc:::benchmark_data_smithi159_162467_object43431:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef58836:::benchmark_data_smithi159_162467_object52250:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef61c0a:::benchmark_data_smithi159_162467_object39075:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef623ff:::benchmark_data_smithi159_162467_object1463:head# 2024-04-15T18:43:15.209 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef6b9cd:::benchmark_data_smithi159_162467_object53800:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef71de1:::benchmark_data_smithi159_162467_object40241:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef73c2d:::benchmark_data_smithi159_162467_object44468:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef7efdc:::benchmark_data_smithi159_162467_object13508:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fef9d882:::benchmark_data_smithi159_162467_object70441:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefb1381:::benchmark_data_smithi159_162467_object11482:head# 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.210 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefc28a9:::benchmark_data_smithi159_162467_object66048:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefc36f6:::benchmark_data_smithi159_162467_object22281:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefc3a33:::benchmark_data_smithi159_162467_object1079:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefdd714:::benchmark_data_smithi159_162467_object41299:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefe1f62:::benchmark_data_smithi159_162467_object56261:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefe2ff3:::benchmark_data_smithi159_162467_object69189:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefe66df:::benchmark_data_smithi159_162467_object42229:head# 2024-04-15T18:43:15.211 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feff2f9f:::benchmark_data_smithi159_162467_object27975:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feff3cdf:::benchmark_data_smithi159_162467_object22234:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feff8ec4:::benchmark_data_smithi159_162467_object24352:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feffa522:::benchmark_data_smithi159_162467_object41599:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:feffe4e4:::benchmark_data_smithi159_162467_object54313:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fefff879:::benchmark_data_smithi159_162467_object69481:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff00608b:::benchmark_data_smithi159_162467_object9851:head# 2024-04-15T18:43:15.212 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff015427:::benchmark_data_smithi159_162467_object62824:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff04667d:::benchmark_data_smithi159_162467_object21125:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff063723:::benchmark_data_smithi159_162467_object36927:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff0661ec:::benchmark_data_smithi159_162467_object15468:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff08b0b9:::benchmark_data_smithi159_162467_object73948:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff095a50:::benchmark_data_smithi159_162467_object60035:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff09addc:::benchmark_data_smithi159_162467_object26742:head# 2024-04-15T18:43:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff0a9c33:::benchmark_data_smithi159_162467_object17388:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff0af861:::benchmark_data_smithi159_162467_object20881:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff0b98c9:::benchmark_data_smithi159_162467_object62721:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff0cfecc:::benchmark_data_smithi159_162467_object56573:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1098d3:::benchmark_data_smithi159_162467_object35049:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1125f9:::benchmark_data_smithi159_162467_object74210:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff121888:::benchmark_data_smithi159_162467_object21109:head# 2024-04-15T18:43:15.214 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff122fd1:::benchmark_data_smithi159_162467_object18889:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff12a48d:::benchmark_data_smithi159_162467_object40751:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff12aa20:::benchmark_data_smithi159_162467_object30017:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff12fbc8:::benchmark_data_smithi159_162467_object31001:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff14d856:::benchmark_data_smithi159_162467_object57669:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1560bb:::benchmark_data_smithi159_162467_object51308:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff169343:::benchmark_data_smithi159_162467_object1010:head# 2024-04-15T18:43:15.215 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff16ef4f:::benchmark_data_smithi159_162467_object50315:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff17b64e:::benchmark_data_smithi159_162467_object2708:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff190bf4:::benchmark_data_smithi159_162467_object10268:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1a92c1:::benchmark_data_smithi159_162467_object45781:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1ab390:::benchmark_data_smithi159_162467_object66051:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1b6b87:::benchmark_data_smithi159_162467_object16192:head# 2024-04-15T18:43:15.216 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1bc5d8:::benchmark_data_smithi159_162467_object57500:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1bde68:::benchmark_data_smithi159_162467_object44148:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1c37c8:::benchmark_data_smithi159_162467_object37168:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1c6c8b:::benchmark_data_smithi159_162467_object9956:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1e3b7d:::benchmark_data_smithi159_162467_object16822:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1e8de8:::benchmark_data_smithi159_162467_object50103:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff1ed30c:::benchmark_data_smithi159_162467_object69299:head# 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.217 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2163b6:::benchmark_data_smithi159_162467_object4063:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff21bd63:::benchmark_data_smithi159_162467_object54496:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff234ed9:::benchmark_data_smithi159_162467_object48350:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff237335:::benchmark_data_smithi159_162467_object62543:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff239df7:::benchmark_data_smithi159_162467_object67085:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff25547a:::benchmark_data_smithi159_162467_object71312:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff268992:::benchmark_data_smithi159_162467_object4121:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2834c3:::benchmark_data_smithi159_162467_object36809:head# 2024-04-15T18:43:15.218 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff28d63c:::benchmark_data_smithi159_162467_object7181:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff28f1e3:::benchmark_data_smithi159_162467_object12460:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff292d6d:::benchmark_data_smithi159_162467_object45935:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff29b382:::benchmark_data_smithi159_162467_object22817:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff29d624:::benchmark_data_smithi159_162467_object12044:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff29dcd0:::benchmark_data_smithi159_162467_object16:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2a422f:::benchmark_data_smithi159_162467_object20196:head# 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.219 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2a8bd6:::benchmark_data_smithi159_162467_object21982:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2b2e56:::benchmark_data_smithi159_162467_object30180:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2b4413:::benchmark_data_smithi159_162467_object69632:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2b78a9:::benchmark_data_smithi159_162467_object27779:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2b8524:::benchmark_data_smithi159_162467_object3314:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2d923a:::benchmark_data_smithi159_162467_object29766:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2f1b56:::benchmark_data_smithi159_162467_object51299:head# 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.220 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff2f417c:::benchmark_data_smithi159_162467_object74581:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff307349:::benchmark_data_smithi159_162467_object23913:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff310dbc:::benchmark_data_smithi159_162467_object41545:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff312448:::benchmark_data_smithi159_162467_object52248:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff32d895:::benchmark_data_smithi159_162467_object67956:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3346b8:::benchmark_data_smithi159_162467_object11318:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff342409:::benchmark_data_smithi159_162467_object21983:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff34da6c:::benchmark_data_smithi159_162467_object55045:head# 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.221 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff34ef00:::benchmark_data_smithi159_162467_object31051:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff358418:::benchmark_data_smithi159_162467_object69504:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3589d7:::benchmark_data_smithi159_162467_object30451:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff358f59:::benchmark_data_smithi159_162467_object63998:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff37d2cb:::benchmark_data_smithi159_162467_object70864:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff39e2ba:::benchmark_data_smithi159_162467_object44382:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff39e906:::benchmark_data_smithi159_162467_object71986:head# 2024-04-15T18:43:15.222 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff39f84f:::benchmark_data_smithi159_162467_object13770:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3acac0:::benchmark_data_smithi159_162467_object30815:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3aff7e:::benchmark_data_smithi159_162467_object64135:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3b0748:::benchmark_data_smithi159_162467_object2972:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3b591f:::benchmark_data_smithi159_162467_object72831:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3b7916:::benchmark_data_smithi159_162467_object71:head# 2024-04-15T18:43:15.223 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3ef6d4:::benchmark_data_smithi159_162467_object1564:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3f1e50:::benchmark_data_smithi159_162467_object44621:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff3fe991:::benchmark_data_smithi159_162467_object19201:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff402ea4:::benchmark_data_smithi159_162467_object17687:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff41bd57:::benchmark_data_smithi159_162467_object52534:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff420329:::benchmark_data_smithi159_162467_object11980:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff43351c:::benchmark_data_smithi159_162467_object23993:head# 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.224 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4416c3:::benchmark_data_smithi159_162467_object4191:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4546f5:::benchmark_data_smithi159_162467_object62318:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff464785:::benchmark_data_smithi159_162467_object59856:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff48a4d4:::benchmark_data_smithi159_162467_object30154:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff497098:::benchmark_data_smithi159_162467_object64296:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4a3efb:::benchmark_data_smithi159_162467_object313:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4c3a93:::benchmark_data_smithi159_162467_object9872:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4c5e6e:::benchmark_data_smithi159_162467_object28079:head# 2024-04-15T18:43:15.225 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff4d8eae:::benchmark_data_smithi159_162467_object35633:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff500c92:::benchmark_data_smithi159_162467_object35485:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff511817:::benchmark_data_smithi159_162467_object1793:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff521339:::benchmark_data_smithi159_162467_object50956:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff532409:::benchmark_data_smithi159_162467_object72134:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff54bd0e:::benchmark_data_smithi159_162467_object56466:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5503a4:::benchmark_data_smithi159_162467_object29402:head# 2024-04-15T18:43:15.226 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff585a5c:::benchmark_data_smithi159_162467_object31913:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5a418a:::benchmark_data_smithi159_162467_object38928:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5a53f7:::benchmark_data_smithi159_162467_object28345:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5a71d7:::benchmark_data_smithi159_162467_object61760:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5a7710:::benchmark_data_smithi159_162467_object52103:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5aad4a:::benchmark_data_smithi159_162467_object12466:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5b18cb:::benchmark_data_smithi159_162467_object29423:head# 2024-04-15T18:43:15.227 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5b44ed:::benchmark_data_smithi159_162467_object32983:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5ca28f:::benchmark_data_smithi159_162467_object20152:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5e1a22:::benchmark_data_smithi159_162467_object26103:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5e6489:::benchmark_data_smithi159_162467_object72689:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff5edf95:::benchmark_data_smithi159_162467_object45005:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff61a84c:::benchmark_data_smithi159_162467_object4602:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff62212a:::benchmark_data_smithi159_162467_object53501:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6276e1:::benchmark_data_smithi159_162467_object64984:head# 2024-04-15T18:43:15.228 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6332b3:::benchmark_data_smithi159_162467_object66143:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff640e45:::benchmark_data_smithi159_162467_object69680:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff64564b:::benchmark_data_smithi159_162467_object49062:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff64e52b:::benchmark_data_smithi159_162467_object49082:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff651ebc:::benchmark_data_smithi159_162467_object28958:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff65b818:::benchmark_data_smithi159_162467_object53132:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff65ebc1:::benchmark_data_smithi159_162467_object5245:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff67673d:::benchmark_data_smithi159_162467_object71445:head# 2024-04-15T18:43:15.229 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6767d3:::benchmark_data_smithi159_162467_object60403:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff67b2fa:::benchmark_data_smithi159_162467_object51800:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff697b42:::benchmark_data_smithi159_162467_object51710:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6a5193:::benchmark_data_smithi159_162467_object1311:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6b2c6d:::benchmark_data_smithi159_162467_object61660:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6ba3c8:::benchmark_data_smithi159_162467_object30619:head# 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.230 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6d2888:::benchmark_data_smithi159_162467_object47379:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6d54b6:::benchmark_data_smithi159_162467_object66448:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6e6b31:::benchmark_data_smithi159_162467_object33056:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6ed8d7:::benchmark_data_smithi159_162467_object45502:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6f4d42:::benchmark_data_smithi159_162467_object48682:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6f5e59:::benchmark_data_smithi159_162467_object14268:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff6fba6d:::benchmark_data_smithi159_162467_object47418:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff714913:::benchmark_data_smithi159_162467_object62218:head# 2024-04-15T18:43:15.231 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff72b552:::benchmark_data_smithi159_162467_object29550:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff745c75:::benchmark_data_smithi159_162467_object68231:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7546cd:::benchmark_data_smithi159_162467_object40969:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff75b66e:::benchmark_data_smithi159_162467_object18230:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff763ef7:::benchmark_data_smithi159_162467_object62217:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff765bea:::benchmark_data_smithi159_162467_object43131:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7689e2:::benchmark_data_smithi159_162467_object33761:head# 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.232 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff769b2f:::benchmark_data_smithi159_162467_object12541:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7bbd02:::benchmark_data_smithi159_162467_object15358:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7c7178:::benchmark_data_smithi159_162467_object41310:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7c89ac:::benchmark_data_smithi159_162467_object14267:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7cc579:::benchmark_data_smithi159_162467_object57041:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7db8a4:::benchmark_data_smithi159_162467_object16738:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7dcceb:::benchmark_data_smithi159_162467_object44356:head# 2024-04-15T18:43:15.233 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7df66a:::benchmark_data_smithi159_162467_object48230:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff7f42a2:::benchmark_data_smithi159_162467_object59436:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff802c8c:::benchmark_data_smithi159_162467_object54880:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff80af7b:::benchmark_data_smithi159_162467_object41667:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff82aeaa:::benchmark_data_smithi159_162467_object68743:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff837dfe:::benchmark_data_smithi159_162467_object9880:head# 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.234 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff83f477:::benchmark_data_smithi159_162467_object74011:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff840ea7:::benchmark_data_smithi159_162467_object20456:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff84db96:::benchmark_data_smithi159_162467_object55716:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff84dee8:::benchmark_data_smithi159_162467_object21903:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff86bdb4:::benchmark_data_smithi159_162467_object3243:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff87928e:::benchmark_data_smithi159_162467_object58124:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff87c925:::benchmark_data_smithi159_162467_object35472:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff8bf28c:::benchmark_data_smithi159_162467_object38356:head# 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.235 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff8c586f:::benchmark_data_smithi159_162467_object21371:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff8c8a64:::benchmark_data_smithi159_162467_object28798:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff8db885:::benchmark_data_smithi159_162467_object2406:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff8e02b5:::benchmark_data_smithi159_162467_object61435:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff93b1b7:::benchmark_data_smithi159_162467_object60850:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9410eb:::benchmark_data_smithi159_162467_object63166:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff942703:::benchmark_data_smithi159_162467_object6231:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff96eb1b:::benchmark_data_smithi159_162467_object35606:head# 2024-04-15T18:43:15.236 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9703d0:::benchmark_data_smithi159_162467_object30897:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff97bb53:::benchmark_data_smithi159_162467_object974:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff98c471:::benchmark_data_smithi159_162467_object6076:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff98d532:::benchmark_data_smithi159_162467_object14612:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff99cfca:::benchmark_data_smithi159_162467_object4736:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9a026c:::benchmark_data_smithi159_162467_object20256:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9a0fb1:::benchmark_data_smithi159_162467_object50365:head# 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.237 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9a6152:::benchmark_data_smithi159_162467_object70510:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9af1de:::benchmark_data_smithi159_162467_object73233:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9c91d0:::benchmark_data_smithi159_162467_object47012:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9de957:::benchmark_data_smithi159_162467_object30207:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9dee73:::benchmark_data_smithi159_162467_object39660:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ff9df0a7:::benchmark_data_smithi159_162467_object17141:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa06370:::benchmark_data_smithi159_162467_object47879:head# 2024-04-15T18:43:15.238 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa0e5ff:::benchmark_data_smithi159_162467_object5860:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa1d71c:::benchmark_data_smithi159_162467_object50830:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa38ff2:::benchmark_data_smithi159_162467_object62508:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa3e371:::benchmark_data_smithi159_162467_object61495:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa4ec56:::benchmark_data_smithi159_162467_object69037:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa518c9:::benchmark_data_smithi159_162467_object26162:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa683da:::benchmark_data_smithi159_162467_object28642:head# 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.239 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa8455d:::benchmark_data_smithi159_162467_object39666:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa913b7:::benchmark_data_smithi159_162467_object23308:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffa93913:::benchmark_data_smithi159_162467_object64677:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffaa7052:::benchmark_data_smithi159_162467_object24953:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffaaf560:::benchmark_data_smithi159_162467_object67145:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffad3012:::benchmark_data_smithi159_162467_object46362:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffad34df:::benchmark_data_smithi159_162467_object33806:head# 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.240 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffadc207:::benchmark_data_smithi159_162467_object4006:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffadc72e:::benchmark_data_smithi159_162467_object63365:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffae7f2c:::benchmark_data_smithi159_162467_object26837:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffaeaca3:::benchmark_data_smithi159_162467_object30436:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffafee84:::benchmark_data_smithi159_162467_object46185:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb09096:::benchmark_data_smithi159_162467_object41679:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb2be92:::benchmark_data_smithi159_162467_object34541:head# 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.241 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb2fe9e:::benchmark_data_smithi159_162467_object66443:head# 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb393b7:::benchmark_data_smithi159_162467_object69444:head# 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb48fb0:::benchmark_data_smithi159_162467_object32046:head# 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb4d660:::benchmark_data_smithi159_162467_object59488:head# 2024-04-15T18:43:15.242 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:14 smithi049 ceph-mon[40087]: pgmap v1943: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 25042/262440 objects degraded (9.542%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:43:15.252 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb5006e:::benchmark_data_smithi159_162467_object61654:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb632b9:::benchmark_data_smithi159_162467_object23151:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb7432c:::benchmark_data_smithi159_162467_object41868:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb90e67:::benchmark_data_smithi159_162467_object74750:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffb962ca:::benchmark_data_smithi159_162467_object71395:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffba3345:::benchmark_data_smithi159_162467_object67958:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffbb1d39:::benchmark_data_smithi159_162467_object30882:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffbc3794:::benchmark_data_smithi159_162467_object55380:head# 2024-04-15T18:43:15.253 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffbc82c7:::benchmark_data_smithi159_162467_object57848:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffbe4932:::benchmark_data_smithi159_162467_object37835:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffbf87fa:::benchmark_data_smithi159_162467_object14765:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc006b7:::benchmark_data_smithi159_162467_object61550:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc0a2b9:::benchmark_data_smithi159_162467_object55799:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc14d43:::benchmark_data_smithi159_162467_object55464:head# 2024-04-15T18:43:15.254 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc2ada5:::benchmark_data_smithi159_162467_object15393:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc32e0c:::benchmark_data_smithi159_162467_object14882:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc362a5:::benchmark_data_smithi159_162467_object72672:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc3c9a3:::benchmark_data_smithi159_162467_object73191:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc4a5e8:::benchmark_data_smithi159_162467_object15068:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc50369:::benchmark_data_smithi159_162467_object38032:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc66fd4:::benchmark_data_smithi159_162467_object10343:head# 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.255 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc70090:::benchmark_data_smithi159_162467_object10170:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc78ac2:::benchmark_data_smithi159_162467_object39125:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffc90e6e:::benchmark_data_smithi159_162467_object41672:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffca8916:::benchmark_data_smithi159_162467_object52687:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffca9181:::benchmark_data_smithi159_162467_object65356:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcb0026:::benchmark_data_smithi159_162467_object4984:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcc5909:::benchmark_data_smithi159_162467_object71407:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcc95e8:::benchmark_data_smithi159_162467_object26968:head# 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.256 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcd1a85:::benchmark_data_smithi159_162467_object4890:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcd4d00:::benchmark_data_smithi159_162467_object62048:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffce2b2c:::benchmark_data_smithi159_162467_object3419:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffcf07c3:::benchmark_data_smithi159_162467_object31835:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd00117:::benchmark_data_smithi159_162467_object64844:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd16a29:::benchmark_data_smithi159_162467_object39453:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd19d81:::benchmark_data_smithi159_162467_object56484:head# 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.257 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd2cb02:::benchmark_data_smithi159_162467_object58766:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd2d493:::benchmark_data_smithi159_162467_object22238:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd3f836:::benchmark_data_smithi159_162467_object45103:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd46e23:::benchmark_data_smithi159_162467_object65836:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd4bba8:::benchmark_data_smithi159_162467_object60376:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd506ae:::benchmark_data_smithi159_162467_object37625:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd56c33:::benchmark_data_smithi159_162467_object52150:head# 2024-04-15T18:43:15.258 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd62eb7:::benchmark_data_smithi159_162467_object72204:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd6ac97:::benchmark_data_smithi159_162467_object48155:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffd89115:::benchmark_data_smithi159_162467_object6121:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdc79b6:::benchmark_data_smithi159_162467_object34976:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdcb249:::benchmark_data_smithi159_162467_object26740:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdcb542:::benchmark_data_smithi159_162467_object4737:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdd39de:::benchmark_data_smithi159_162467_object63539:head# 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.259 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdd4f79:::benchmark_data_smithi159_162467_object14433:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffddb95f:::benchmark_data_smithi159_162467_object93:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdec8ef:::benchmark_data_smithi159_162467_object27010:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdf5d35:::benchmark_data_smithi159_162467_object24828:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffdf77cd:::benchmark_data_smithi159_162467_object63709:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe14c0f:::benchmark_data_smithi159_162467_object4361:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe20e34:::benchmark_data_smithi159_162467_object48660:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe257fe:::benchmark_data_smithi159_162467_object53945:head# 2024-04-15T18:43:15.260 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe258bb:::benchmark_data_smithi159_162467_object26769:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe291a9:::benchmark_data_smithi159_162467_object3997:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe4e4d2:::benchmark_data_smithi159_162467_object70430:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe5736d:::benchmark_data_smithi159_162467_object62413:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe630ed:::benchmark_data_smithi159_162467_object52717:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe958ef:::benchmark_data_smithi159_162467_object65555:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffe9f707:::benchmark_data_smithi159_162467_object48593:head# 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.261 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffea32b6:::benchmark_data_smithi159_162467_object39745:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffec85b0:::benchmark_data_smithi159_162467_object57037:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffedc888:::benchmark_data_smithi159_162467_object21428:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffef2b19:::benchmark_data_smithi159_162467_object31771:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:ffef5dfd:::benchmark_data_smithi159_162467_object42348:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff0c4af:::benchmark_data_smithi159_162467_object30861:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff0e60f:::benchmark_data_smithi159_162467_object62832:head# 2024-04-15T18:43:15.262 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff34702:::benchmark_data_smithi159_162467_object60166:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff43abb:::benchmark_data_smithi159_162467_object33745:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff4dd82:::benchmark_data_smithi159_162467_object58070:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff5e123:::benchmark_data_smithi159_162467_object58358:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff6b223:::benchmark_data_smithi159_162467_object20153:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff7f3d4:::benchmark_data_smithi159_162467_object34835:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff9dc52:::benchmark_data_smithi159_162467_object11373:head# 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.263 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fff9ebc1:::benchmark_data_smithi159_162467_object32438:head# 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fffb83d2:::benchmark_data_smithi159_162467_object70803:head# 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fffcf4d7:::benchmark_data_smithi159_162467_object17464:head# 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fffd1deb:::benchmark_data_smithi159_162467_object35389:head# 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:Write #8:fffeead0:::benchmark_data_smithi159_162467_object16152:head# 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:snapset 0=[]:{} 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:write_pg epoch 733 info 8.f( v 733'4786 (0'0,733'4786] local-lis/les=709/710 n=4786 ec=709/709 lis/c=709/709 les/c/f=710/710/0 sis=709) 2024-04-15T18:43:15.264 INFO:teuthology.orchestra.run.smithi159.stdout:Import successful 2024-04-15T18:43:15.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:14 smithi082 ceph-mon[44206]: pgmap v1943: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 25042/262440 objects degraded (9.542%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:43:15.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:14 smithi159 ceph-mon[45336]: pgmap v1943: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 25042/262440 objects degraded (9.542%); 12 MiB/s, 196 objects/s recovering 2024-04-15T18:43:15.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:43:15.572 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:15.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:15.600 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:43:15.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:43:15.718 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:15.743 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:43:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:15.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:43:15.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:15.884 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:15.913 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:15.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:16.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:43:16.041 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:16.076 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:16.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:43:16.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:15 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:16.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:15 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:16.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:15 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:16.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:15 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:15 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:16.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:15 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:16.473 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/exp.8.f.11 2024-04-15T18:43:16.556 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:16.557 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-04-15T18:43:16.557 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T18:43:16.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:43:17.105 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:16 smithi159 ceph-mon[45336]: pgmap v1944: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 25042/261186 objects degraded (9.588%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:43:17.105 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:16 smithi159 ceph-mon[45336]: osdmap e752: 12 total, 11 up, 11 in 2024-04-15T18:43:17.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:16 smithi082 ceph-mon[44206]: pgmap v1944: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 25042/261186 objects degraded (9.588%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:43:17.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:16 smithi082 ceph-mon[44206]: osdmap e752: 12 total, 11 up, 11 in 2024-04-15T18:43:17.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:16 smithi049 ceph-mon[40087]: pgmap v1944: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 635 op/s; 25042/261186 objects degraded (9.588%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:43:17.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:16 smithi049 ceph-mon[40087]: osdmap e752: 12 total, 11 up, 11 in 2024-04-15T18:43:18.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 25042/261186 objects degraded (9.588%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:18.197 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:43:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:18.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:43:18.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 25042/261186 objects degraded (9.588%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:18.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:18.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 25042/261186 objects degraded (9.588%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:18.336 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:18.358 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:18.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:18.447 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:43:18.447 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:43:18.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:18.496 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:18.703 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:18.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:18.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:18.850 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:18.923 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:18.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:19.003 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:19 smithi049 ceph-mon[40087]: pgmap v1946: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 24425/256773 objects degraded (9.512%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:19.027 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:19.063 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:19.120 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:19 smithi159 ceph-mon[45336]: pgmap v1946: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 24425/256773 objects degraded (9.512%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:19.124 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:19.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:19.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:43:19.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:19.261 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:19 smithi082 ceph-mon[44206]: pgmap v1946: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 24425/256773 objects degraded (9.512%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:19.263 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:19.288 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:19.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:19.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:19.400 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:19.505 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:19.528 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:19.574 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:19.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:19.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:20.029 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:20.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:20.138 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:20.167 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:20.366 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:20.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:20.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:20.560 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:20.606 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:20.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:20.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:20.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:20.777 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:20.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:20.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:20.910 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:20.933 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:20.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:21.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:21 smithi082 ceph-mon[44206]: pgmap v1947: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 24222/252705 objects degraded (9.585%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:21.005 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:43:21.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:21.051 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:43:21.072 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:21.098 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:21.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:21 smithi159 ceph-mon[45336]: pgmap v1947: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 24222/252705 objects degraded (9.585%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:43:21.204 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:43:21.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:21 smithi049 ceph-mon[40087]: pgmap v1947: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 24222/252705 objects degraded (9.585%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:43:21.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:43:21.243 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:21.344 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:43:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:21.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:43:21.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:21.484 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:21.503 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:43:21.503 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:43:21.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:21.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:21.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:22.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:22 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:22.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:43:22.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:22 smithi159 ceph-mon[45336]: osdmap e753: 12 total, 11 up, 11 in 2024-04-15T18:43:22.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:43:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:22 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:43:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:22 smithi082 ceph-mon[44206]: osdmap e753: 12 total, 11 up, 11 in 2024-04-15T18:43:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:22 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1637275954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:43:22.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:22 smithi049 ceph-mon[40087]: osdmap e753: 12 total, 11 up, 11 in 2024-04-15T18:43:22.557 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:22.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:22.573 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:22.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:43:22.731 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:22.812 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:43:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:22.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:43:22.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:22.952 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:22.987 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:22.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:23.096 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:23.103 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:23 smithi159 ceph-mon[45336]: pgmap v1949: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 24003/251040 objects degraded (9.561%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:43:23.103 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 24003/251040 objects degraded (9.561%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:23.127 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:23.149 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:23.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:23.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:23.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:23 smithi082 ceph-mon[44206]: pgmap v1949: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 24003/251040 objects degraded (9.561%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:43:23.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 24003/251040 objects degraded (9.561%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:23.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:23 smithi049 ceph-mon[40087]: pgmap v1949: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 24003/251040 objects degraded (9.561%); 8.1 MiB/s, 129 objects/s recovering 2024-04-15T18:43:23.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 24003/251040 objects degraded (9.561%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:23.336 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:23.394 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:23.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:23.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:23.537 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:23.565 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:23.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:23.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:23.735 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:24.016 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:24.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:24.124 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:24.155 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:24.187 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:24.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:24.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:24.317 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:24.318 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:24.322 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:24.342 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:24.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:24.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:43:24.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:24.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:24.499 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:43:24.499 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:43:24.506 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:24.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:43:25.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:43:25.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:43:25.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:43:25.009 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: pgmap v1950: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 23351/243729 objects degraded (9.581%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: osdmap e754: 12 total, 11 up, 11 in 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2977554679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:25 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.222 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: pgmap v1950: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 23351/243729 objects degraded (9.581%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:43:25.222 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: osdmap e754: 12 total, 11 up, 11 in 2024-04-15T18:43:25.223 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2977554679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:25.223 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.223 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.223 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:25 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:43:25.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: pgmap v1950: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 23351/243729 objects degraded (9.581%); 13 MiB/s, 211 objects/s recovering 2024-04-15T18:43:25.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: osdmap e754: 12 total, 11 up, 11 in 2024-04-15T18:43:25.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2977554679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:25 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:25.634 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:26.133 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: from='client.26926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.134 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:26 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.167 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:43:26.167 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: from='client.26926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:26 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: from='client.26926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:26 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:27.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:27 smithi159 ceph-mon[45336]: pgmap v1952: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 590 op/s; 23351/242595 objects degraded (9.626%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:43:27.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:27 smithi049 ceph-mon[40087]: pgmap v1952: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 590 op/s; 23351/242595 objects degraded (9.626%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:43:27.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:27 smithi082 ceph-mon[44206]: pgmap v1952: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 590 op/s; 23351/242595 objects degraded (9.626%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:43:28.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 23351/242595 objects degraded (9.626%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:28.403 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:43:28.406 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 23351/242595 objects degraded (9.626%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:28.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:43:28.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:28.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 23351/242595 objects degraded (9.626%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:28.539 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:28.564 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:28.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:28.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:28.785 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:29.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:29 smithi049 ceph-mon[40087]: pgmap v1953: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 22663/238497 objects degraded (9.502%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:43:29.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:29 smithi159 ceph-mon[45336]: pgmap v1953: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 22663/238497 objects degraded (9.502%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:43:29.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:29 smithi082 ceph-mon[44206]: pgmap v1953: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 22663/238497 objects degraded (9.502%); 12 MiB/s, 194 objects/s recovering 2024-04-15T18:43:30.258 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:30.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:30.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:30.401 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:30.428 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:30.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:30.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:30.582 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:31.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:31 smithi159 ceph-mon[45336]: pgmap v1954: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 22471/234420 objects degraded (9.586%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:43:31.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:31 smithi049 ceph-mon[40087]: pgmap v1954: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 22471/234420 objects degraded (9.586%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:43:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:31 smithi082 ceph-mon[44206]: pgmap v1954: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 22471/234420 objects degraded (9.586%); 13 MiB/s, 203 objects/s recovering 2024-04-15T18:43:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:33 smithi159 ceph-mon[45336]: pgmap v1955: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 22166/232554 objects degraded (9.532%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:43:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:33 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 22166/232554 objects degraded (9.532%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:33.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:33 smithi049 ceph-mon[40087]: pgmap v1955: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 22166/232554 objects degraded (9.532%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:43:33.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:33 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 22166/232554 objects degraded (9.532%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:33.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:33 smithi082 ceph-mon[44206]: pgmap v1955: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 22166/232554 objects degraded (9.532%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:43:33.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:33 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 22166/232554 objects degraded (9.532%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:34.080 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:43:34.080 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:43:34.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26791.0:130437 8.1 e754/709 8:860f69dc:::benchmark_data_smithi159_162467_object28132:head v 754'8055, mlcod=754'8053)", 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:33.673130+0000", 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0035106149999999999, 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035336130000000001, 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673130+0000", 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:34.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673130+0000", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673131+0000", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6499999999999998e-07 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673153+0000", 2024-04-15T18:43:34.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2339000000000001e-05 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673185+0000", 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1618000000000001e-05 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673199+0000", 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4250000000000001e-05 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:34.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673265+0000", 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6239000000000001e-05 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.673293+0000", 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7654e-05 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26791.0:130443 8.0 e754/709 8:0ca709fd:::benchmark_data_smithi159_162467_object28138:head v 754'8164, mlcod=754'8162)", 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:33.675641+0000", 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0010001330000000001, 2024-04-15T18:43:34.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001063558, 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675641+0000", 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675641+0000", 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675641+0000", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2900000000000002e-07 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675649+0000", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1760000000000004e-06 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675675+0000", 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6222999999999999e-05 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675680+0000", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3720000000000004e-06 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675731+0000", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0290000000000001e-05 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:33.675751+0000", 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0063000000000001e-05 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:34.087 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:34.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:34.088 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:34.088 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:43:34.088 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 2 2024-04-15T18:43:34.088 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:43:34.605 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:34.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:43:34.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:43:34.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:34.767 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:34.922 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:34.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:34.979 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1951,"stamp":"2024-04-15T18:43:24.065702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5324735032,"num_objects":81243,"num_object_clones":0,"num_object_copies":243729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81243,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":123546,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7021,"num_bytes_recovered":462983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123504,"ondisk_log_size":123504,"up":51,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19856728,"kb_used_data":14230896,"kb_used_omap":285,"kb_used_meta":5625250,"kb_avail":1011339944,"statfs":{"total":1055945392128,"available":1035612102656,"internally_reserved":0,"allocated":14572437504,"data_stored":14569238999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":5760256213},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-381353984,"num_objects":-5819,"num_object_clones":0,"num_object_copies":-17457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5819,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1691,"num_bytes_recovered":110559232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.076759"},"pg_stats":[{"pgid":"8.f","version":"753'7899","reported_seq":10408,"reported_epoch":753,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:22.360158+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:43:22.360158+0000","last_peered":"2024-04-15T18:43:22.360158+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:43:22.360158+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":7899,"log_dups_size":0,"ondisk_log_size":7899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339804160,"num_objects":5185,"num_object_clones":0,"num_object_copies":15555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7899,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1203,"num_bytes_recovered":78643200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":230}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"753'7789","reported_seq":7952,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:22.354883+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:43:22.354883+0000","last_peered":"2024-04-15T18:43:22.354883+0000","last_clean":"2024-04-15T18:43:22.354883+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:43:22.354883+0000","last_undegraded":"2024-04-15T18:43:22.354883+0000","last_fullsized":"2024-04-15T18:43:22.354883+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:33.313911+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335872000,"num_objects":5125,"num_object_clones":0,"num_object_copies":15375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7789,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"753'7747","reported_seq":11182,"reported_epoch":753,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:21.451400+0000","last_change":"2024-04-15T18:42:40.129165+0000","last_active":"2024-04-15T18:43:21.451400+0000","last_peered":"2024-04-15T18:43:21.451400+0000","last_clean":"2024-04-15T18:42:15.046200+0000","last_became_active":"2024-04-15T18:42:40.077364+0000","last_became_peered":"2024-04-15T18:42:40.077364+0000","last_unstale":"2024-04-15T18:43:21.451400+0000","last_undegraded":"2024-04-15T18:42:39.778417+0000","last_fullsized":"2024-04-15T18:42:39.749589+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340656152,"num_objects":5199,"num_object_clones":0,"num_object_copies":15597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7747,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1665,"num_bytes_recovered":109117440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":733}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"753'7878","reported_seq":8042,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:22.587644+0000","last_change":"2024-04-15T18:42:35.114708+0000","last_active":"2024-04-15T18:43:22.587644+0000","last_peered":"2024-04-15T18:43:22.587644+0000","last_clean":"2024-04-15T18:43:22.587644+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:22.587644+0000","last_undegraded":"2024-04-15T18:43:22.587644+0000","last_fullsized":"2024-04-15T18:43:22.587644+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5959","last_scrub_stamp":"2024-04-15T18:42:35.114530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.114530+0000","objects_scrubbed":5915,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:58.842544+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336723968,"num_objects":5138,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"752'7631","reported_seq":7792,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-15T18:43:20.115128+0000","last_change":"2024-04-15T18:42:32.425776+0000","last_active":"2024-04-15T18:43:20.115128+0000","last_peered":"2024-04-15T18:43:20.115128+0000","last_clean":"2024-04-15T18:43:20.115128+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:43:20.115128+0000","last_undegraded":"2024-04-15T18:43:20.115128+0000","last_fullsized":"2024-04-15T18:43:20.115128+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5725","last_scrub_stamp":"2024-04-15T18:42:32.425642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.425642+0000","objects_scrubbed":5700,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:48.489523+0000","scrub_duration":821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339804160,"num_objects":5185,"num_object_clones":0,"num_object_copies":15555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"752'7581","reported_seq":7743,"reported_epoch":752,"state":"active+clean","last_fresh":"2024-04-15T18:43:20.149534+0000","last_change":"2024-04-15T18:42:32.253080+0000","last_active":"2024-04-15T18:43:20.149534+0000","last_peered":"2024-04-15T18:43:20.149534+0000","last_clean":"2024-04-15T18:43:20.149534+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:43:20.149534+0000","last_undegraded":"2024-04-15T18:43:20.149534+0000","last_fullsized":"2024-04-15T18:43:20.149534+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5682","last_scrub_stamp":"2024-04-15T18:42:32.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.252916+0000","objects_scrubbed":5635,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:32.537760+0000","scrub_duration":1663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337575936,"num_objects":5151,"num_object_clones":0,"num_object_copies":15453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7581,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"753'7723","reported_seq":7890,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:22.915194+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:43:22.915194+0000","last_peered":"2024-04-15T18:43:22.915194+0000","last_clean":"2024-04-15T18:43:22.915194+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:43:22.915194+0000","last_undegraded":"2024-04-15T18:43:22.915194+0000","last_fullsized":"2024-04-15T18:43:22.915194+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:57.447580+0000","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7723,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"753'7745","reported_seq":9936,"reported_epoch":753,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:21.842564+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:43:21.842564+0000","last_peered":"2024-04-15T18:43:21.842564+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:43:21.842564+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326434816,"num_objects":4981,"num_object_clones":0,"num_object_copies":14943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7745,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1043,"num_bytes_recovered":68222976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-16}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"753'7755","reported_seq":11163,"reported_epoch":753,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:23.477423+0000","last_change":"2024-04-15T18:42:39.959765+0000","last_active":"2024-04-15T18:43:23.477423+0000","last_peered":"2024-04-15T18:43:23.477423+0000","last_clean":"2024-04-15T18:42:15.002893+0000","last_became_active":"2024-04-15T18:42:39.939882+0000","last_became_peered":"2024-04-15T18:42:39.939882+0000","last_unstale":"2024-04-15T18:43:23.477423+0000","last_undegraded":"2024-04-15T18:42:39.770656+0000","last_fullsized":"2024-04-15T18:42:39.750615+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325124096,"num_objects":4961,"num_object_clones":0,"num_object_copies":14883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":108331008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":626}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"753'7597","reported_seq":7760,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:23.455064+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:43:23.455064+0000","last_peered":"2024-04-15T18:43:23.455064+0000","last_clean":"2024-04-15T18:43:23.455064+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:43:23.455064+0000","last_undegraded":"2024-04-15T18:43:23.455064+0000","last_fullsized":"2024-04-15T18:43:23.455064+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:56.743295+0000","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329842688,"num_objects":5033,"num_object_clones":0,"num_object_copies":15099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7597,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"753'7506","reported_seq":7669,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:22.904769+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:43:22.904769+0000","last_peered":"2024-04-15T18:43:22.904769+0000","last_clean":"2024-04-15T18:43:22.904769+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:43:22.904769+0000","last_undegraded":"2024-04-15T18:43:22.904769+0000","last_fullsized":"2024-04-15T18:43:22.904769+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":7506,"log_dups_size":0,"ondisk_log_size":7506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:38.275571+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7506,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"753'7692","reported_seq":10677,"reported_epoch":753,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:21.832607+0000","last_change":"2024-04-15T18:42:39.993114+0000","last_active":"2024-04-15T18:43:21.832607+0000","last_peered":"2024-04-15T18:43:21.832607+0000","last_clean":"2024-04-15T18:42:15.025871+0000","last_became_active":"2024-04-15T18:42:39.954118+0000","last_became_peered":"2024-04-15T18:42:39.954118+0000","last_unstale":"2024-04-15T18:43:21.832607+0000","last_undegraded":"2024-04-15T18:42:39.885263+0000","last_fullsized":"2024-04-15T18:42:39.747827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336986112,"num_objects":5142,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":94437376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"753'7826","reported_seq":7990,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:23.477919+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:43:23.477919+0000","last_peered":"2024-04-15T18:43:23.477919+0000","last_clean":"2024-04-15T18:43:23.477919+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:43:23.477919+0000","last_undegraded":"2024-04-15T18:43:23.477919+0000","last_fullsized":"2024-04-15T18:43:23.477919+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:44:03.860738+0000","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"753'7571","reported_seq":7733,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:21.757395+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:43:21.757395+0000","last_peered":"2024-04-15T18:43:21.757395+0000","last_clean":"2024-04-15T18:43:21.757395+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:43:21.757395+0000","last_undegraded":"2024-04-15T18:43:21.757395+0000","last_fullsized":"2024-04-15T18:43:21.757395+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":7571,"log_dups_size":0,"ondisk_log_size":7571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:34.963136+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119488,"num_objects":5083,"num_object_clones":0,"num_object_copies":15249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"753'7750","reported_seq":7913,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:22.888477+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:43:22.888477+0000","last_peered":"2024-04-15T18:43:22.888477+0000","last_clean":"2024-04-15T18:43:22.888477+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:22.888477+0000","last_undegraded":"2024-04-15T18:43:22.888477+0000","last_fullsized":"2024-04-15T18:43:22.888477+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:41.360332+0000","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323485696,"num_objects":4936,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7750,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"753'7675","reported_seq":7839,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:23.429921+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:43:23.429921+0000","last_peered":"2024-04-15T18:43:23.429921+0000","last_clean":"2024-04-15T18:43:23.429921+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:43:23.429921+0000","last_undegraded":"2024-04-15T18:43:23.429921+0000","last_fullsized":"2024-04-15T18:43:23.429921+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:51.069715+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332201984,"num_objects":5069,"num_object_clones":0,"num_object_copies":15207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1869,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-04-15T18:43:21.013368+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:43:21.013368+0000","last_peered":"2024-04-15T18:43:21.013368+0000","last_clean":"2024-04-15T18:43:21.013368+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:43:21.013368+0000","last_undegraded":"2024-04-15T18:43:21.013368+0000","last_fullsized":"2024-04-15T18:43:21.013368+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5324144664,"num_objects":81241,"num_object_clones":0,"num_object_copies":243723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81241,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123365,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7005,"num_bytes_recovered":458752000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16036081664,"data_stored":16036069448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123365,"ondisk_log_size":123365,"up":48,"acting":43,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2462240,"kb_used_data":1925720,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91282912,"statfs":{"total":95995035648,"available":93473701888,"internally_reserved":0,"allocated":1971937280,"data_stored":1971646584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514919,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2597420,"kb_used_data":1948384,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91147732,"statfs":{"total":95995035648,"available":93335277568,"internally_reserved":0,"allocated":1995145216,"data_stored":1994850968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874664,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391380,"kb_used_data":999244,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92353772,"statfs":{"total":95995035648,"available":94570262528,"internally_reserved":0,"allocated":1023225856,"data_stored":1022939563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169005,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1131616,"kb_used_data":663128,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92613536,"statfs":{"total":95995035648,"available":94836260864,"internally_reserved":0,"allocated":679043072,"data_stored":678749155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207324,"kb_used_data":36052,"kb_used_omap":111,"kb_used_meta":171152,"kb_avail":93537828,"statfs":{"total":95995035648,"available":95782735872,"internally_reserved":0,"allocated":36917248,"data_stored":36627411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":175260315},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575312,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2602580,"kb_used_data":1975756,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91142572,"statfs":{"total":95995035648,"available":93329993728,"internally_reserved":0,"allocated":2023174144,"data_stored":2022887851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1644628,"kb_used_data":995660,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92100524,"statfs":{"total":95995035648,"available":94310936576,"internally_reserved":0,"allocated":1019555840,"data_stored":1019269547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194084,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2146856,"kb_used_data":1709404,"kb_used_omap":114,"kb_used_meta":437261,"kb_avail":91598296,"statfs":{"total":95995035648,"available":93796655104,"internally_reserved":0,"allocated":1750429696,"data_stored":1750134928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":447755505},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609214,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1674532,"kb_used_data":973592,"kb_used_omap":44,"kb_used_meta":700883,"kb_avail":92070620,"statfs":{"total":95995035648,"available":94280314880,"internally_reserved":0,"allocated":996958208,"data_stored":996667512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":717704746},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986621,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932388,"kb_used_data":1333400,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91812764,"statfs":{"total":95995035648,"available":94016270336,"internally_reserved":0,"allocated":1365401600,"data_stored":1365110904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703515,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2065764,"kb_used_data":1670556,"kb_used_omap":4,"kb_used_meta":395195,"kb_avail":91679388,"statfs":{"total":95995035648,"available":93879693312,"internally_reserved":0,"allocated":1710649344,"data_stored":1710354576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":404680385},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1017118720,"data_stored":1017118720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2007433216,"data_stored":2007433216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1739722752,"data_stored":1739718680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":991232000,"data_stored":991232000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1355874304,"data_stored":1355874304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1697124352,"data_stored":1697120280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36372480,"data_stored":36372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":671944704,"data_stored":671940632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1020264448,"data_stored":1020264448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1985478656,"data_stored":1985478656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1963393024,"data_stored":1963393024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:43:34.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:43:35.008 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:35.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:35.068 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:35.092 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:35.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:35.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:35 smithi049 ceph-mon[40087]: pgmap v1956: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 21460/224892 objects degraded (9.542%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:35.225 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:43:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:35.334 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:35.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:35 smithi159 ceph-mon[45336]: pgmap v1956: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 21460/224892 objects degraded (9.542%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:35.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:35.482 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:35.509 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:35.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:35.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:35 smithi082 ceph-mon[44206]: pgmap v1956: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 21460/224892 objects degraded (9.542%); 12 MiB/s, 188 objects/s recovering 2024-04-15T18:43:35.564 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:35.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:35.691 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:36.167 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:36.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:36.178 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:36.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:36.442 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:36.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:43:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:37 smithi159 ceph-mon[45336]: pgmap v1957: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 21460/223752 objects degraded (9.591%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:43:37.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:37 smithi049 ceph-mon[40087]: pgmap v1957: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 21460/223752 objects degraded (9.591%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:43:37.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:37 smithi082 ceph-mon[44206]: pgmap v1957: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 21460/223752 objects degraded (9.591%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:43:38.302 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:43:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:43:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:43:38.303 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:43:38.303 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:43:38.330 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:38 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 21460/223752 objects degraded (9.591%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:38.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:38 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 21460/223752 objects degraded (9.591%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:38.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:38 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 21460/223752 objects degraded (9.591%), 5 pgs degraded (PG_DEGRADED) 2024-04-15T18:43:38.942 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:38.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:39.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:39.088 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:39.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:39 smithi159 ceph-mon[45336]: pgmap v1958: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 20780/219660 objects degraded (9.460%); 13 MiB/s, 213 objects/s recovering 2024-04-15T18:43:39.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:39 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:39.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:39 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:39.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:39 smithi159 ceph-mon[45336]: osdmap e755: 12 total, 11 up, 11 in 2024-04-15T18:43:39.376 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:39.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:39.408 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:43:39.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:39 smithi049 ceph-mon[40087]: pgmap v1958: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 20780/219660 objects degraded (9.460%); 13 MiB/s, 213 objects/s recovering 2024-04-15T18:43:39.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:39 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:39.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:39 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:39.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:39 smithi049 ceph-mon[40087]: osdmap e755: 12 total, 11 up, 11 in 2024-04-15T18:43:39.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:39.560 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:39 smithi082 ceph-mon[44206]: pgmap v1958: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 20780/219660 objects degraded (9.460%); 13 MiB/s, 213 objects/s recovering 2024-04-15T18:43:39.560 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:39 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:39.560 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:39 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:39.560 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:39 smithi082 ceph-mon[44206]: osdmap e755: 12 total, 11 up, 11 in 2024-04-15T18:43:39.596 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:39.626 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:39.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:39.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:39.776 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:39.786 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:40.257 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:40.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:40.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:40.460 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:40.726 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:40.728 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:40.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:40.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:40.876 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:41.135 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:43:41.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:41 smithi049 ceph-mon[40087]: pgmap v1960: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 622 op/s; 20608/215706 objects degraded (9.554%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:43:41.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:43:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:41 smithi159 ceph-mon[45336]: pgmap v1960: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 622 op/s; 20608/215706 objects degraded (9.554%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:43:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:43:41.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:41 smithi082 ceph-mon[44206]: pgmap v1960: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 622 op/s; 20608/215706 objects degraded (9.554%); 12 MiB/s, 185 objects/s recovering 2024-04-15T18:43:41.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:43:41.935 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:41.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:42.041 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:42.079 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:42.260 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:42 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:42.260 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:43:42.260 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:42 smithi159 ceph-mon[45336]: osdmap e756: 12 total, 11 up, 11 in 2024-04-15T18:43:42.260 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:42 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:42.281 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:42.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:42.296 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:43:42.321 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:43:42.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:43:42.419 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:42.448 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:43:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:42.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:43:42.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:42 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:42.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:43:42.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:42 smithi049 ceph-mon[40087]: osdmap e756: 12 total, 11 up, 11 in 2024-04-15T18:43:42.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:42 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:42.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:42 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:43:42.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/177170143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:43:42.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:42 smithi082 ceph-mon[44206]: osdmap e756: 12 total, 11 up, 11 in 2024-04-15T18:43:42.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:42 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:42.553 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:42.595 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:42.623 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:42.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:89863 6.9 6:91fd2ab3:::benchmark_data_smithi159_139342_object89862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e578)", 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 573.69459585899995, 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.37690326899999999, 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.636 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 89863 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945056+0000", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945058+0000", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3689999999999999e-06 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945076+0000", 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8085999999999999e-05 2024-04-15T18:43:42.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945086+0000", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7739999999999999e-06 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945091+0000", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1290000000000004e-06 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945132+0000", 2024-04-15T18:43:42.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1081000000000002e-05 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945234+0000", 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000102428 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.945337+0000", 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010299899999999999 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.950097+0000", 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047599950000000004 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:08.950294+0000", 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000196813 2024-04-15T18:43:42.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321843+0000", 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.371548516 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321905+0000", 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2470000000000003e-05 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:34:09.321960+0000", 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4608999999999998e-05 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26224.0:155611 6.a e598/575 6:5c79a8ec:::benchmark_data_smithi159_139342_object65732:head v 598'9678, mlcod=567'4930)", 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 479.520854929, 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.232901525, 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.643 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118797+0000", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118798+0000", 2024-04-15T18:43:42.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4799999999999997e-07 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.118803+0000", 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0830000000000003e-06 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348129+0000", 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229325801 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348135+0000", 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4189999999999999e-06 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348195+0000", 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9808000000000001e-05 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.348223+0000", 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7732000000000001e-05 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.351657+0000", 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034345170000000002 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:43.351699+0000", 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1417e-05 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:158190 6.9 6:9d4dbcde:::benchmark_data_smithi159_139342_object68311:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:43:42.647 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 475.50697591599999, 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24825063999999999, 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158190 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132676+0000", 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132677+0000", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132683+0000", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5289999999999999e-06 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132694+0000", 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1341e-05 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132700+0000", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5350000000000004e-06 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132742+0000", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2536000000000001e-05 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132839+0000", 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6197000000000001e-05 2024-04-15T18:43:42.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.132915+0000", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6719999999999997e-05 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.137560+0000", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004644385 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.137882+0000", 2024-04-15T18:43:42.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032201499999999999 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380850+0000", 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24296841699999999 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380890+0000", 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0098999999999997e-05 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.380927+0000", 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6789000000000003e-05 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:158202 6.9 6:937e2fa2:::benchmark_data_smithi159_139342_object68323:head [delete] snapc 0=[] ondisk+write+known_if_redirected e600)", 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 475.46974702799997, 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.231209677, 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.654 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 158202 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169905+0000", 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169906+0000", 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.187e-06 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169911+0000", 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0329999999999999e-06 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169923+0000", 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1868000000000001e-05 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169929+0000", 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7610000000000004e-06 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.169984+0000", 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4746000000000001e-05 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.170084+0000", 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010006200000000001 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.170160+0000", 2024-04-15T18:43:42.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6305999999999998e-05 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.173978+0000", 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003817745 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.174149+0000", 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017113999999999999 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401048+0000", 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22689936099999999 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401084+0000", 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5321000000000003e-05 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:35:47.401115+0000", 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1146999999999998e-05 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.660 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:177249 6.9 6:9e4639b2:::benchmark_data_smithi159_139342_object87370:head [delete] snapc 0=[] ondisk+write+known_if_redirected e609)", 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:18.051179+0000", 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 444.588473394, 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26025119000000002, 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 177249 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.661 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051179+0000", 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051179+0000", 2024-04-15T18:43:42.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051180+0000", 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.268e-06 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051185+0000", 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8949999999999997e-06 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051196+0000", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0910000000000001e-05 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051202+0000", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2519999999999996e-06 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051248+0000", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.545e-05 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051344+0000", 2024-04-15T18:43:42.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6339e-05 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.051439+0000", 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5169999999999999e-05 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.055349+0000", 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003910101 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261595+0000", 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.206245971 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.311327+0000", 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.049731895999999998 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.311389+0000", 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2209e-05 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.311430+0000", 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0729000000000003e-05 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26224.0:177249 6.9 e609/541 ondisk, result = 0)", 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:18.055127+0000", 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 444.584524961, 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20647380000000001, 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:43:42.667 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.055127+0000", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.055127+0000", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.055128+0000", 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7199999999999998e-07 2024-04-15T18:43:42.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.055136+0000", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6650000000000003e-06 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261393+0000", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20625742499999999 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261406+0000", 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3295999999999999e-05 2024-04-15T18:43:42.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261557+0000", 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000150991 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261574+0000", 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6969999999999998e-05 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:18.261601+0000", 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6681000000000001e-05 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26224.0:179741 6.9 6:91fd2ab3:::benchmark_data_smithi159_139342_object89862:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:36:21.358709+0000", 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 441.28094328499998, 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.91310919700000004, 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.671 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26224", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 179741 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358709+0000", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358709+0000", 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358710+0000", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.454e-06 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358718+0000", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8990000000000001e-06 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358735+0000", 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6222000000000002e-05 2024-04-15T18:43:42.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358744+0000", 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9040000000000004e-06 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358805+0000", 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0535000000000002e-05 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.358949+0000", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014381999999999999 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,8", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.359083+0000", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013390799999999999 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.364765+0000", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056824279999999998 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:21.367534+0000", 2024-04-15T18:43:42.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027690589999999999 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.271743+0000", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.90420847100000001 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.271782+0000", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9597000000000001e-05 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:36:22.271818+0000", 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5899999999999998e-05 2024-04-15T18:43:42.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:17276 7.f e627/621 7:f9e2ce40:::benchmark_data_smithi159_149998_object17275:head v 627'1044, mlcod=627'1041)", 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:00.706132+0000", 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 401.93352054899998, 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.230578229, 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706132+0000", 2024-04-15T18:43:42.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706132+0000", 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706132+0000", 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7899999999999997e-07 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.706154+0000", 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1943999999999999e-05 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.935915+0000", 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229760672 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.935921+0000", 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2940000000000002e-06 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.935971+0000", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9254e-05 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.935992+0000", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.179e-05 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.936678+0000", 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068585100000000002 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:00.936710+0000", 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1644999999999999e-05 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:31794 7.f e631/621 7:f52c522c:::benchmark_data_smithi159_149998_object31793:head v 631'1925, mlcod=631'1922)", 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.049721+0000", 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 391.58993130200002, 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.29013999400000001, 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049721+0000", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049721+0000", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049722+0000", 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.082e-06 2024-04-15T18:43:42.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.049745+0000", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3382000000000001e-05 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.338678+0000", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.288932419 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.338683+0000", 2024-04-15T18:43:42.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0250000000000002e-06 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.338763+0000", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9795000000000004e-05 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.338778+0000", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5009e-05 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.339826+0000", 2024-04-15T18:43:42.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010481830000000001 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.339861+0000", 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5098999999999997e-05 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:31805 7.a 7:5db3329e:::benchmark_data_smithi159_149998_object31804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:37:11.116604+0000", 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 391.52304815899998, 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.229675816, 2024-04-15T18:43:42.685 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 31805 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116604+0000", 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116604+0000", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116605+0000", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2389999999999999e-06 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116627+0000", 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1846999999999999e-05 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116640+0000", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2702e-05 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116647+0000", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1400000000000002e-06 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116694+0000", 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7284000000000003e-05 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116793+0000", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8750000000000002e-05 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,11", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.116883+0000", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0322999999999996e-05 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.118190+0000", 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001306754 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.118435+0000", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024489500000000002 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.346214+0000", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22777892499999999 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.346249+0000", 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5534000000000002e-05 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:37:11.346280+0000", 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0423e-05 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26545.0:110637 7.f e658/621 7:f23ca5e6:::benchmark_data_smithi159_149998_object14637:head v 658'6993, mlcod=658'6990)", 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:38:40.606307+0000", 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 302.03334483899999, 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.204947183, 2024-04-15T18:43:42.691 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606307+0000", 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606307+0000", 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606308+0000", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3699999999999999e-07 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.606315+0000", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3350000000000003e-06 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.808407+0000", 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20209223300000001 2024-04-15T18:43:42.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.808415+0000", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1319999999999994e-06 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.808476+0000", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1378999999999996e-05 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.808499+0000", 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2758000000000001e-05 2024-04-15T18:43:42.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.811217+0000", 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027179859999999999 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:38:40.811255+0000", 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7523e-05 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:2788 8.c 8:3f38cce4:::benchmark_data_smithi159_162467_object2787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e718)", 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:41:19.712706+0000", 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.92694650799999, 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.220541609, 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2788 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712706+0000", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712706+0000", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712707+0000", 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5e-06 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712726+0000", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8734999999999999e-05 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712736+0000", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0193999999999999e-05 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712742+0000", 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5040000000000002e-06 2024-04-15T18:43:42.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712781+0000", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9830999999999999e-05 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712865+0000", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3219000000000004e-05 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.712953+0000", 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8376999999999998e-05 2024-04-15T18:43:42.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.713824+0000", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00087115599999999999 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.714175+0000", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035062500000000003 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933136+0000", 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.218960917 2024-04-15T18:43:42.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933191+0000", 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5549e-05 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933247+0000", 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6001999999999997e-05 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26791.0:2788 8.c e718/709 ondisk, result = 0)", 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:41:19.713765+0000", 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.92588771199999, 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21949576400000001, 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.713765+0000", 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.713765+0000", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.713765+0000", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.713773+0000", 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4789999999999999e-06 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.932520+0000", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21874732399999999 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.932526+0000", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6840000000000001e-06 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933121+0000", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00059508200000000003 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933129+0000", 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8639999999999996e-06 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:19.933260+0000", 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000131562 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26791.0:4238 8.1 e718/709 8:86472420:::benchmark_data_smithi159_162467_object4237:head v 718'288, mlcod=718'284)", 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:41:20.862580+0000", 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 141.777072237, 2024-04-15T18:43:42.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22939374100000001, 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:20.862580+0000", 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:20.862580+0000", 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:20.862581+0000", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3499999999999996e-07 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:20.862641+0000", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0507999999999997e-05 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091262+0000", 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22862103 2024-04-15T18:43:42.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091268+0000", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5709999999999997e-06 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091318+0000", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0482000000000002e-05 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091340+0000", 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1679e-05 2024-04-15T18:43:42.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091933+0000", 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00059356000000000003 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:41:21.091974+0000", 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0275999999999999e-05 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:66889 8.c 8:359bc8cc:::benchmark_data_smithi159_162467_object66888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:07.496695+0000", 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 95.142956945999998, 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.215056417, 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 66889 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496695+0000", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496695+0000", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496697+0000", 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7749999999999999e-06 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496719+0000", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2289999999999998e-05 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496732+0000", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3005e-05 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496741+0000", 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3999999999999992e-06 2024-04-15T18:43:42.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496805+0000", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4443999999999995e-05 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496927+0000", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012180099999999999 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.497027+0000", 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000100349 2024-04-15T18:43:42.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.498986+0000", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019590250000000001 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.499750+0000", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00076314500000000001 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.711684+0000", 2024-04-15T18:43:42.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21193403299999999 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.711721+0000", 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7407000000000003e-05 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.711752+0000", 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0743e-05 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:81457 8.c 8:3379a503:::benchmark_data_smithi159_162467_object81456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:21.918997+0000", 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 80.720655610999998, 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17019098299999999, 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 81457 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.918997+0000", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.918997+0000", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.918998+0000", 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.513e-06 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919015+0000", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6864999999999999e-05 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919026+0000", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1471000000000001e-05 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919032+0000", 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0079999999999999e-06 2024-04-15T18:43:42.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919084+0000", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2006000000000003e-05 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919231+0000", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000146365 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.919398+0000", 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016669800000000001 2024-04-15T18:43:42.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.921548+0000", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021499819999999999 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:21.921618+0000", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0038999999999998e-05 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:22.089117+0000", 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16749946099999999 2024-04-15T18:43:42.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:22.089158+0000", 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0645000000000001e-05 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:22.089188+0000", 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.993e-05 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.721 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:91030 8.c 8:383baa1d:::benchmark_data_smithi159_162467_object91029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e740)", 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:31.618601+0000", 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.021051118000003, 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.17552311500000001, 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 91030 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618601+0000", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618601+0000", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618603+0000", 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4100000000000001e-06 2024-04-15T18:43:42.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618627+0000", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4221000000000001e-05 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618641+0000", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4541e-05 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618650+0000", 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3149999999999992e-06 2024-04-15T18:43:42.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618762+0000", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112252 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618868+0000", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000105837 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618970+0000", 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001019 2024-04-15T18:43:42.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.621398+0000", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024286770000000002 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.793559+0000", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.172161017 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.794006+0000", 2024-04-15T18:43:42.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044706800000000001 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.794071+0000", 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.457e-05 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.794124+0000", 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3307000000000003e-05 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.727 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:91049 8.c 8:3549cb9d:::benchmark_data_smithi159_162467_object91048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e740)", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:31.651170+0000", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.988482016999995, 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.16992465800000001, 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 91049 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651170+0000", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651170+0000", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651172+0000", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.649e-06 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651209+0000", 2024-04-15T18:43:42.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6881000000000001e-05 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651224+0000", 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5537e-05 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651233+0000", 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0100000000000001e-06 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651288+0000", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4613e-05 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651437+0000", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000149307 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,8", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.651577+0000", 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000139624 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.656142+0000", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045654409999999999 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.803275+0000", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14713306200000001 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.820981+0000", 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017705604 2024-04-15T18:43:42.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.821045+0000", 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4216999999999999e-05 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.821095+0000", 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9713000000000003e-05 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.733 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26791.0:109216 8.1 e749/709 8:8d5abe39:::benchmark_data_smithi159_162467_object6911:head v 749'6780, mlcod=749'6777)", 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:59.700051+0000", 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.939601537999998, 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.213823076, 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.700051+0000", 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.700051+0000", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.700051+0000", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8100000000000003e-07 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.700057+0000", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.071e-06 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.911105+0000", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.211047607 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.911110+0000", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5580000000000002e-06 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.911154+0000", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4702e-05 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.911170+0000", 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6198e-05 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.913832+0000", 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002661751 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.913874+0000", 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1607999999999999e-05 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.26791.0:115873 8.1 e751/709 8:867e62ee:::benchmark_data_smithi159_162467_object13568:head v 751'7151, mlcod=751'7148)", 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:10.373342+0000", 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.266310623999999, 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18466870499999999, 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.373342+0000", 2024-04-15T18:43:42.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.373342+0000", 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.373342+0000", 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1500000000000004e-07 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:42.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.373349+0000", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9240000000000001e-06 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.555166+0000", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18181715900000001 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.555172+0000", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3700000000000003e-06 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:42.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.555212+0000", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9771999999999998e-05 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.555228+0000", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6203000000000001e-05 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.557974+0000", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027460010000000001 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:43:42.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:10.558010+0000", 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6560999999999998e-05 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:42.742 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:43:42.742 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:43:42.776 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:42.803 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:43:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:42.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:43:42.907 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:42.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:43 smithi159 ceph-mon[45336]: pgmap v1962: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 453 op/s; 20325/213993 objects degraded (9.498%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:43:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:43 smithi159 ceph-mon[45336]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:43.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:43 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 20325/213993 objects degraded (9.498%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:43.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:43 smithi049 ceph-mon[40087]: pgmap v1962: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 453 op/s; 20325/213993 objects degraded (9.498%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:43:43.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:43 smithi049 ceph-mon[40087]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:43.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:43 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 20325/213993 objects degraded (9.498%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:43.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:43 smithi082 ceph-mon[44206]: pgmap v1962: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 453 op/s; 20325/213993 objects degraded (9.498%); 8.9 MiB/s, 141 objects/s recovering 2024-04-15T18:43:43.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:43 smithi082 ceph-mon[44206]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:43:43.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:43 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 20325/213993 objects degraded (9.498%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:43.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:43:43.996 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:44.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:44.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:44.140 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:44.167 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:44.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:44.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:44.337 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:44.340 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:44.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:44 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:44.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:44 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:44.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:44 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:44.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:44 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:44.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:44 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:44.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:44 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:45.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: pgmap v1963: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 19661/207057 objects degraded (9.495%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:43:45.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: osdmap e757: 12 total, 11 up, 11 in 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:45 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: pgmap v1963: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 19661/207057 objects degraded (9.495%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: osdmap e757: 12 total, 11 up, 11 in 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:43:45.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:45 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: pgmap v1963: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 695 op/s; 19661/207057 objects degraded (9.495%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: osdmap e757: 12 total, 11 up, 11 in 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:43:45.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:43:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:45 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:43:46.219 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:46.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:46.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:46.361 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:46.387 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:46.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:46.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:43:46.544 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:47.317 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:43:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:47.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:43:47.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:43:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:47 smithi049 ceph-mon[40087]: pgmap v1965: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 19661/205926 objects degraded (9.548%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:43:47.469 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:47 smithi049 ceph-mon[40087]: 8.a scrub starts 2024-04-15T18:43:47.469 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:47 smithi082 ceph-mon[44206]: pgmap v1965: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 19661/205926 objects degraded (9.548%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:43:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:47 smithi082 ceph-mon[44206]: 8.a scrub starts 2024-04-15T18:43:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:47 smithi159 ceph-mon[45336]: pgmap v1965: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 19661/205926 objects degraded (9.548%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:43:47.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:47 smithi159 ceph-mon[45336]: 8.a scrub starts 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:138916 8.6 8:65209d90:::benchmark_data_smithi159_162467_object36611:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:47.677517+0000", 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.020681996000000001, 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020706156, 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:47.687 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 138916 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677517+0000", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:47.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677517+0000", 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677519+0000", 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5579999999999999e-06 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677524+0000", 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6710000000000004e-06 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677540+0000", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5540999999999999e-05 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.677548+0000", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5580000000000001e-06 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.678247+0000", 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069871199999999999 2024-04-15T18:43:47.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.678394+0000", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000146501 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.678526+0000", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013282800000000001 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.690701+0000", 2024-04-15T18:43:47.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012174571 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.696529+0000", 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058275339999999997 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:138920 8.5 8:a4d98ee8:::benchmark_data_smithi159_162467_object36615:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:47.686494+0000", 2024-04-15T18:43:47.692 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.011705435, 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011787222, 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 138920 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686494+0000", 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686494+0000", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686495+0000", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686500+0000", 2024-04-15T18:43:47.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5169999999999999e-06 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686511+0000", 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1054e-05 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686519+0000", 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2800000000000003e-06 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686581+0000", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1283000000000003e-05 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686715+0000", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000134206 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686813+0000", 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7659999999999997e-05 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.690615+0000", 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038019389999999998 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.691208+0000", 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00059339500000000003 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:138923 8.6 8:688ccfb8:::benchmark_data_smithi159_162467_object36618:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:47.686608+0000", 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.011591473, 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011723206, 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 138923 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:47.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686608+0000", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686608+0000", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686608+0000", 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7800000000000001e-07 2024-04-15T18:43:47.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686611+0000", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2880000000000001e-06 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686618+0000", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7109999999999998e-06 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.686624+0000", 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8239999999999996e-06 2024-04-15T18:43:47.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.690733+0000", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041089409999999996 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.690862+0000", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000128744 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,7", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.690989+0000", 2024-04-15T18:43:47.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000127348 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.696500+0000", 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055108570000000001 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:138931 8.9 8:99a34494:::benchmark_data_smithi159_162467_object36626:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:43:47.695920+0000", 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.0022793010000000001, 2024-04-15T18:43:47.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024642370000000002, 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 138931 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695920+0000", 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695920+0000", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695921+0000", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1000000000000001e-06 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695926+0000", 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.763e-06 2024-04-15T18:43:47.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695936+0000", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0088999999999999e-05 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695942+0000", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9569999999999999e-06 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.695983+0000", 2024-04-15T18:43:47.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1700999999999999e-05 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.696070+0000", 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6520000000000005e-05 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:43:47.696136+0000", 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5829999999999998e-05 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:43:47.707 INFO:teuthology.orchestra.run.smithi159.stdout: ], 2024-04-15T18:43:47.707 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 4 2024-04-15T18:43:47.707 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:43:48.378 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:48 smithi159 ceph-mon[45336]: 8.a scrub ok 2024-04-15T18:43:48.378 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:48 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 19661/205926 objects degraded (9.548%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:48.415 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:43:48.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:48 smithi049 ceph-mon[40087]: 8.a scrub ok 2024-04-15T18:43:48.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:48 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 19661/205926 objects degraded (9.548%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:48.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:48 smithi082 ceph-mon[44206]: 8.a scrub ok 2024-04-15T18:43:48.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:48 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 19661/205926 objects degraded (9.548%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:48.767 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: pgmap v1966: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 18941/201861 objects degraded (9.383%); 13 MiB/s, 208 objects/s recovering 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:49.305 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:49 smithi159 ceph-mon[45336]: osdmap e758: 12 total, 11 up, 11 in 2024-04-15T18:43:49.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: pgmap v1966: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 18941/201861 objects degraded (9.383%); 13 MiB/s, 208 objects/s recovering 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:49.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:49 smithi049 ceph-mon[40087]: osdmap e758: 12 total, 11 up, 11 in 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: pgmap v1966: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 18941/201861 objects degraded (9.383%); 13 MiB/s, 208 objects/s recovering 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:49 smithi082 ceph-mon[44206]: osdmap e758: 12 total, 11 up, 11 in 2024-04-15T18:43:49.554 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:43:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:49.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:43:49.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:43:49.692 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:50.032 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1962,"stamp":"2024-04-15T18:43:41.628782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4675142200,"num_objects":71331,"num_object_clones":0,"num_object_copies":213993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71331,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":133458,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10047,"num_bytes_recovered":661164288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133416,"ondisk_log_size":133416,"up":51,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18138952,"kb_used_data":12511904,"kb_used_omap":285,"kb_used_meta":5626658,"kb_avail":1013057720,"statfs":{"total":1055945392128,"available":1037371105280,"internally_reserved":0,"allocated":12812189696,"data_stored":12808903881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":5761698005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,7,5,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-238092288,"num_objects":-3633,"num_object_clones":0,"num_object_copies":-10899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":74383360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006944"},"pg_stats":[{"pgid":"8.f","version":"754'8507","reported_seq":11987,"reported_epoch":754,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:37.403188+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:43:37.403188+0000","last_peered":"2024-04-15T18:43:37.403188+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:43:37.403188+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":8507,"log_dups_size":0,"ondisk_log_size":8507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":13731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8507,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":110297088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"754'8372","reported_seq":8537,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:37.391692+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:43:37.391692+0000","last_peered":"2024-04-15T18:43:37.391692+0000","last_clean":"2024-04-15T18:43:37.391692+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:43:37.391692+0000","last_undegraded":"2024-04-15T18:43:37.391692+0000","last_fullsized":"2024-04-15T18:43:37.391692+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":8372,"log_dups_size":0,"ondisk_log_size":8372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297664512,"num_objects":4542,"num_object_clones":0,"num_object_copies":13626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"756'8541","reported_seq":14033,"reported_epoch":756,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:41.472356+0000","last_change":"2024-04-15T18:42:40.129165+0000","last_active":"2024-04-15T18:43:41.472356+0000","last_peered":"2024-04-15T18:43:41.472356+0000","last_clean":"2024-04-15T18:42:15.046200+0000","last_became_active":"2024-04-15T18:42:40.077364+0000","last_became_peered":"2024-04-15T18:42:40.077364+0000","last_unstale":"2024-04-15T18:43:41.472356+0000","last_undegraded":"2024-04-15T18:42:39.778417+0000","last_fullsized":"2024-04-15T18:42:39.749589+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288620568,"num_objects":4405,"num_object_clones":0,"num_object_copies":13215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8541,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2691,"num_bytes_recovered":176357376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":965}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"754'8470","reported_seq":8636,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:37.576170+0000","last_change":"2024-04-15T18:42:35.114708+0000","last_active":"2024-04-15T18:43:37.576170+0000","last_peered":"2024-04-15T18:43:37.576170+0000","last_clean":"2024-04-15T18:43:37.576170+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:37.576170+0000","last_undegraded":"2024-04-15T18:43:37.576170+0000","last_fullsized":"2024-04-15T18:43:37.576170+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5959","last_scrub_stamp":"2024-04-15T18:42:35.114530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.114530+0000","objects_scrubbed":5915,"log_size":8470,"log_dups_size":0,"ondisk_log_size":8470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:58.842544+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297926656,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"755'8415","reported_seq":8583,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-04-15T18:43:40.317366+0000","last_change":"2024-04-15T18:42:32.425776+0000","last_active":"2024-04-15T18:43:40.317366+0000","last_peered":"2024-04-15T18:43:40.317366+0000","last_clean":"2024-04-15T18:43:40.317366+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:43:40.317366+0000","last_undegraded":"2024-04-15T18:43:40.317366+0000","last_fullsized":"2024-04-15T18:43:40.317366+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5725","last_scrub_stamp":"2024-04-15T18:42:32.425642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.425642+0000","objects_scrubbed":5700,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:48.489523+0000","scrub_duration":821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288423936,"num_objects":4401,"num_object_clones":0,"num_object_copies":13203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8415,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"755'8355","reported_seq":8523,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-04-15T18:43:40.296484+0000","last_change":"2024-04-15T18:42:32.253080+0000","last_active":"2024-04-15T18:43:40.296484+0000","last_peered":"2024-04-15T18:43:40.296484+0000","last_clean":"2024-04-15T18:43:40.296484+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:43:40.296484+0000","last_undegraded":"2024-04-15T18:43:40.296484+0000","last_fullsized":"2024-04-15T18:43:40.296484+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5682","last_scrub_stamp":"2024-04-15T18:42:32.252916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.252916+0000","objects_scrubbed":5635,"log_size":8355,"log_dups_size":0,"ondisk_log_size":8355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:44.083963+0000","scrub_duration":1663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286851072,"num_objects":4377,"num_object_clones":0,"num_object_copies":13131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"754'8326","reported_seq":8495,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:37.859428+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:43:37.859428+0000","last_peered":"2024-04-15T18:43:37.859428+0000","last_clean":"2024-04-15T18:43:37.859428+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:43:37.859428+0000","last_undegraded":"2024-04-15T18:43:37.859428+0000","last_fullsized":"2024-04-15T18:43:37.859428+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:57.447580+0000","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292159488,"num_objects":4458,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"754'8314","reported_seq":11242,"reported_epoch":754,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:36.936184+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:43:36.936184+0000","last_peered":"2024-04-15T18:43:36.936184+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:43:36.936184+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289144832,"num_objects":4412,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8314,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":92340224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-217}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"754'8339","reported_seq":12936,"reported_epoch":754,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:38.480609+0000","last_change":"2024-04-15T18:42:39.959765+0000","last_active":"2024-04-15T18:43:38.480609+0000","last_peered":"2024-04-15T18:43:38.480609+0000","last_clean":"2024-04-15T18:42:15.002893+0000","last_became_active":"2024-04-15T18:42:39.939882+0000","last_became_peered":"2024-04-15T18:42:39.939882+0000","last_unstale":"2024-04-15T18:43:38.480609+0000","last_undegraded":"2024-04-15T18:42:39.770656+0000","last_fullsized":"2024-04-15T18:42:39.750615+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":8339,"log_dups_size":0,"ondisk_log_size":8339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286851072,"num_objects":4377,"num_object_clones":0,"num_object_copies":13131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8339,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":147259392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":636}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"754'8158","reported_seq":8323,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:38.480374+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:43:38.480374+0000","last_peered":"2024-04-15T18:43:38.480374+0000","last_clean":"2024-04-15T18:43:38.480374+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:43:38.480374+0000","last_undegraded":"2024-04-15T18:43:38.480374+0000","last_fullsized":"2024-04-15T18:43:38.480374+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:56.743295+0000","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293076992,"num_objects":4472,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"754'8063","reported_seq":8228,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:37.883251+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:43:37.883251+0000","last_peered":"2024-04-15T18:43:37.883251+0000","last_clean":"2024-04-15T18:43:37.883251+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:43:37.883251+0000","last_undegraded":"2024-04-15T18:43:37.883251+0000","last_fullsized":"2024-04-15T18:43:37.883251+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":8063,"log_dups_size":0,"ondisk_log_size":8063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:38.275571+0000","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290914304,"num_objects":4439,"num_object_clones":0,"num_object_copies":13317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8063,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"754'8288","reported_seq":12382,"reported_epoch":754,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:36.933562+0000","last_change":"2024-04-15T18:42:39.993114+0000","last_active":"2024-04-15T18:43:36.933562+0000","last_peered":"2024-04-15T18:43:36.933562+0000","last_clean":"2024-04-15T18:42:15.025871+0000","last_became_active":"2024-04-15T18:42:39.954118+0000","last_became_peered":"2024-04-15T18:42:39.954118+0000","last_unstale":"2024-04-15T18:43:36.933562+0000","last_undegraded":"2024-04-15T18:42:39.885263+0000","last_fullsized":"2024-04-15T18:42:39.747827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":8288,"log_dups_size":0,"ondisk_log_size":8288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297926656,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8288,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":130678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"754'8413","reported_seq":8581,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:38.466777+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:43:38.466777+0000","last_peered":"2024-04-15T18:43:38.466777+0000","last_clean":"2024-04-15T18:43:38.466777+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:43:38.466777+0000","last_undegraded":"2024-04-15T18:43:38.466777+0000","last_fullsized":"2024-04-15T18:43:38.466777+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:44:03.860738+0000","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":13227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"754'8165","reported_seq":8329,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:36.837305+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:43:36.837305+0000","last_peered":"2024-04-15T18:43:36.837305+0000","last_clean":"2024-04-15T18:43:36.837305+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:43:36.837305+0000","last_undegraded":"2024-04-15T18:43:36.837305+0000","last_fullsized":"2024-04-15T18:43:36.837305+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":13467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"754'8307","reported_seq":8473,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:37.934500+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:43:37.934500+0000","last_peered":"2024-04-15T18:43:37.934500+0000","last_clean":"2024-04-15T18:43:37.934500+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:37.934500+0000","last_undegraded":"2024-04-15T18:43:37.934500+0000","last_fullsized":"2024-04-15T18:43:37.934500+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":8307,"log_dups_size":0,"ondisk_log_size":8307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:41.360332+0000","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286982144,"num_objects":4379,"num_object_clones":0,"num_object_copies":13137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"754'8244","reported_seq":8410,"reported_epoch":754,"state":"active+clean","last_fresh":"2024-04-15T18:43:38.415550+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:43:38.415550+0000","last_peered":"2024-04-15T18:43:38.415550+0000","last_clean":"2024-04-15T18:43:38.415550+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:43:38.415550+0000","last_undegraded":"2024-04-15T18:43:38.415550+0000","last_fullsized":"2024-04-15T18:43:38.415550+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:51.069715+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912000,"num_objects":4500,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1875,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-04-15T18:43:41.129670+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:43:41.129670+0000","last_peered":"2024-04-15T18:43:41.129670+0000","last_clean":"2024-04-15T18:43:41.129670+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:43:41.129670+0000","last_undegraded":"2024-04-15T18:43:41.129670+0000","last_fullsized":"2024-04-15T18:43:41.129670+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4674551832,"num_objects":71329,"num_object_clones":0,"num_object_copies":213987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133277,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10031,"num_bytes_recovered":656932864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14298460160,"data_stored":14298447944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133277,"ondisk_log_size":133277,"up":48,"acting":43,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2256172,"kb_used_data":1705764,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91488980,"statfs":{"total":95995035648,"available":93684715520,"internally_reserved":0,"allocated":1746702336,"data_stored":1746407287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":563608000},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514922,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2372980,"kb_used_data":1724012,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91372172,"statfs":{"total":95995035648,"available":93565104128,"internally_reserved":0,"allocated":1765388288,"data_stored":1765089687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1250364,"kb_used_data":849904,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92494788,"statfs":{"total":95995035648,"available":94714662912,"internally_reserved":0,"allocated":870301696,"data_stored":870002294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169009,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033992,"kb_used_data":561340,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92711160,"statfs":{"total":95995035648,"available":94936227840,"internally_reserved":0,"allocated":574812160,"data_stored":574505134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195972,"kb_used_data":33016,"kb_used_omap":111,"kb_used_meta":162832,"kb_avail":93549180,"statfs":{"total":95995035648,"available":95794360320,"internally_reserved":0,"allocated":33808384,"data_stored":33505438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":166740635},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575315,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2369452,"kb_used_data":1746852,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91375700,"statfs":{"total":95995035648,"available":93568716800,"internally_reserved":0,"allocated":1788776448,"data_stored":1788481399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486524,"kb_used_data":845872,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92258628,"statfs":{"total":95995035648,"available":94472835072,"internally_reserved":0,"allocated":866172928,"data_stored":865873526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194087,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958448,"kb_used_data":1516904,"kb_used_omap":114,"kb_used_meta":441421,"kb_avail":91786704,"statfs":{"total":95995035648,"available":93989584896,"internally_reserved":0,"allocated":1553309696,"data_stored":1553010575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":452015345},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609217,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548780,"kb_used_data":860324,"kb_used_omap":44,"kb_used_meta":688403,"kb_avail":92196372,"statfs":{"total":95995035648,"available":94409084928,"internally_reserved":0,"allocated":880971776,"data_stored":880676727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":704925226},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986624,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1781676,"kb_used_data":1182692,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91963476,"statfs":{"total":95995035648,"available":94170599424,"internally_reserved":0,"allocated":1211076608,"data_stored":1210781559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703518,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1884592,"kb_used_data":1485224,"kb_used_omap":4,"kb_used_meta":399355,"kb_avail":91860560,"statfs":{"total":95995035648,"available":94065213440,"internally_reserved":0,"allocated":1520869376,"data_stored":1520570255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":408940225},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":863043584,"data_stored":863043584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1776353280,"data_stored":1776353280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1546457088,"data_stored":1546453016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":876412928,"data_stored":876412928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1202388992,"data_stored":1202388992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1519587328,"data_stored":1519583256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32440320,"data_stored":32440320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":569970688,"data_stored":569966616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":867368960,"data_stored":867368960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1755774976,"data_stored":1755774976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1738539008,"data_stored":1738539008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:43:50.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:43:50.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:43:50.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:43:50.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:43:50.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:43:50.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:43:50.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:43:50.765 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:50.845 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:43:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:50.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:43:50.927 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:43:50.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:50.986 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:51.018 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:51.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:51.124 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:51.160 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:51.266 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:51.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:51 smithi082 ceph-mon[44206]: pgmap v1968: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 18735/197604 objects degraded (9.481%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:43:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:51 smithi082 ceph-mon[44206]: 8.b scrub starts 2024-04-15T18:43:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:51 smithi082 ceph-mon[44206]: osdmap e759: 12 total, 11 up, 11 in 2024-04-15T18:43:51.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:51.427 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:51.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:51 smithi049 ceph-mon[40087]: pgmap v1968: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 18735/197604 objects degraded (9.481%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:43:51.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:51 smithi049 ceph-mon[40087]: 8.b scrub starts 2024-04-15T18:43:51.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:51 smithi049 ceph-mon[40087]: osdmap e759: 12 total, 11 up, 11 in 2024-04-15T18:43:51.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:51 smithi159 ceph-mon[45336]: pgmap v1968: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 18735/197604 objects degraded (9.481%); 12 MiB/s, 198 objects/s recovering 2024-04-15T18:43:51.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:51 smithi159 ceph-mon[45336]: 8.b scrub starts 2024-04-15T18:43:51.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:51 smithi159 ceph-mon[45336]: osdmap e759: 12 total, 11 up, 11 in 2024-04-15T18:43:52.010 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:43:52.010 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:43:52.010 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:43:52.011 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:43:52.011 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:43:52.387 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:52 smithi159 ceph-mon[45336]: 8.b scrub ok 2024-04-15T18:43:52.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:52 smithi082 ceph-mon[44206]: 8.b scrub ok 2024-04-15T18:43:52.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:52 smithi049 ceph-mon[40087]: 8.b scrub ok 2024-04-15T18:43:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:52.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:53.012 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:43:53.046 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:53.074 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:43:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:53.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:43:53.178 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:53.209 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:53.236 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:53.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:53.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:43:53.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:53.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:53 smithi049 ceph-mon[40087]: pgmap v1970: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 18448/195750 objects degraded (9.424%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:43:53.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 18448/195750 objects degraded (9.424%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:53.430 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:53.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:43:53.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:43:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:53 smithi082 ceph-mon[44206]: pgmap v1970: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 18448/195750 objects degraded (9.424%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:43:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 18448/195750 objects degraded (9.424%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:53.552 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:43:53.588 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:53.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:53.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:43:53.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:43:53.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:53 smithi159 ceph-mon[45336]: pgmap v1970: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 4.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 18448/195750 objects degraded (9.424%); 10 MiB/s, 162 objects/s recovering 2024-04-15T18:43:53.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 18448/195750 objects degraded (9.424%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:53.648 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:43:53.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:43:53.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:53.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:43:53.765 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:53.876 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:43:53.902 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:53.907 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:43:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:53.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:53.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:43:53.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:43:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:43:53.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:43:53.989 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:43:54.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:54.074 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:54.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4137460611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:54.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:54 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4137460611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:54.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:54 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.564 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:54.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:54.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4137460611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:43:54.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:54 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:54.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:43:54.734 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:54.773 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:43:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:54.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:43:54.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:43:54.911 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:54.992 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:43:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:54.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:43:55.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:55.127 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:55.156 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:55.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:55.292 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:55.319 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:55.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:55.424 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:43:55.455 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: pgmap v1971: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 17833/188142 objects degraded (9.478%); 14 MiB/s, 228 objects/s recovering 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.456 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.462 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:55.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: pgmap v1971: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 17833/188142 objects degraded (9.478%); 14 MiB/s, 228 objects/s recovering 2024-04-15T18:43:55.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: pgmap v1971: 17 pgs: 1 active+clean+scrubbing, 5 active+recovering+undersized+degraded+remapped, 11 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 17833/188142 objects degraded (9.478%); 14 MiB/s, 228 objects/s recovering 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:43:55.716 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:43:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:55.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:43:55.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:43:55.854 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:55.878 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:43:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:55.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:43:55.982 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:43:56.012 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:56.041 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:43:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:56.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:43:56.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:56.216 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:56.588 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:43:56.588 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:43:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:57 smithi082 ceph-mon[44206]: pgmap v1972: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 17833/187071 objects degraded (9.533%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:43:57.620 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:57 smithi159 ceph-mon[45336]: pgmap v1972: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 17833/187071 objects degraded (9.533%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:43:57.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:57 smithi049 ceph-mon[40087]: pgmap v1972: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 17833/187071 objects degraded (9.533%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:43:57.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:43:58.270 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 17833/187071 objects degraded (9.533%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:58.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 17833/187071 objects degraded (9.533%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:58.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 17833/187071 objects degraded (9.533%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:43:59.402 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:59.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:43:59 smithi049 ceph-mon[40087]: pgmap v1973: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 17014/182913 objects degraded (9.302%); 13 MiB/s, 204 objects/s recovering 2024-04-15T18:43:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:59.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:43:59.507 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:43:59.534 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:43:59 smithi082 ceph-mon[44206]: pgmap v1973: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 17014/182913 objects degraded (9.302%); 13 MiB/s, 204 objects/s recovering 2024-04-15T18:43:59.536 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:59.559 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:43:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:59.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:43:59.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:43:59 smithi159 ceph-mon[45336]: pgmap v1973: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 605 op/s; 17014/182913 objects degraded (9.302%); 13 MiB/s, 204 objects/s recovering 2024-04-15T18:43:59.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:43:59.741 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:43:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:43:59.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:43:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:43:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:43:59.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:00.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:00.046 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:00.067 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:00.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:00.172 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:44:00.204 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:00.234 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:44:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:00.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:44:00.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:00 smithi049 ceph-mon[40087]: 8.c scrub starts 2024-04-15T18:44:00.307 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:00 smithi082 ceph-mon[44206]: 8.c scrub starts 2024-04-15T18:44:00.307 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:00.307 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:00.307 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2637565015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.307 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:00 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:00.353 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:44:00.416 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:00.633 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:00.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:00 smithi159 ceph-mon[45336]: 8.c scrub starts 2024-04-15T18:44:00.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:00.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:00.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2637565015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:00 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:00.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:00.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:00.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:00.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2637565015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.649 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:00 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:00.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:44:00.822 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:00.826 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:44:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:01.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:01.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:01.505 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:01.626 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:01.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:01.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:01 smithi049 ceph-mon[40087]: pgmap v1974: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 16775/178962 objects degraded (9.373%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:01.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:01 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:44:01.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:01 smithi049 ceph-mon[40087]: 8.c scrub ok 2024-04-15T18:44:01.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:01 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:01.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:01 smithi049 ceph-mon[40087]: osdmap e760: 12 total, 11 up, 11 in 2024-04-15T18:44:01.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:01 smithi159 ceph-mon[45336]: pgmap v1974: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 16775/178962 objects degraded (9.373%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:01.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:01 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:44:01.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:01 smithi159 ceph-mon[45336]: 8.c scrub ok 2024-04-15T18:44:01.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:01 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:01.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:01 smithi159 ceph-mon[45336]: osdmap e760: 12 total, 11 up, 11 in 2024-04-15T18:44:01.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:01.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:01 smithi082 ceph-mon[44206]: pgmap v1974: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 16775/178962 objects degraded (9.373%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:01.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:01 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:44:01.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:01 smithi082 ceph-mon[44206]: 8.c scrub ok 2024-04-15T18:44:01.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:01 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:01.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:01 smithi082 ceph-mon[44206]: osdmap e760: 12 total, 11 up, 11 in 2024-04-15T18:44:01.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:44:01.800 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:02.157 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1975,"stamp":"2024-04-15T18:44:00.318092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3909878328,"num_objects":59654,"num_object_clones":0,"num_object_copies":178962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59654,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":145135,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13597,"num_bytes_recovered":893554944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145093,"ondisk_log_size":145093,"up":51,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16259252,"kb_used_data":10543388,"kb_used_omap":285,"kb_used_meta":5715490,"kb_avail":1014937420,"statfs":{"total":1055945392128,"available":1039295918080,"internally_reserved":0,"allocated":10796429312,"data_stored":10793423435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":5852661973},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-407240704,"num_objects":-6214,"num_object_clones":0,"num_object_copies":-18642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1960,"num_bytes_recovered":128385024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.096682"},"pg_stats":[{"pgid":"8.f","version":"759'9299","reported_seq":14132,"reported_epoch":759,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:57.471273+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:43:57.471273+0000","last_peered":"2024-04-15T18:43:57.471273+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:43:57.471273+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":9299,"log_dups_size":0,"ondisk_log_size":9299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248053760,"num_objects":3785,"num_object_clones":0,"num_object_copies":11355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2359,"num_bytes_recovered":154140672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":-14}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"759'9149","reported_seq":9324,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:57.482507+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:43:57.482507+0000","last_peered":"2024-04-15T18:43:57.482507+0000","last_clean":"2024-04-15T18:43:57.482507+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:43:57.482507+0000","last_undegraded":"2024-04-15T18:43:57.482507+0000","last_fullsized":"2024-04-15T18:43:57.482507+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":9149,"log_dups_size":0,"ondisk_log_size":9149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9149,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"759'9124","reported_seq":16345,"reported_epoch":759,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:56.478565+0000","last_change":"2024-04-15T18:42:40.129165+0000","last_active":"2024-04-15T18:43:56.478565+0000","last_peered":"2024-04-15T18:43:56.478565+0000","last_clean":"2024-04-15T18:42:15.046200+0000","last_became_active":"2024-04-15T18:42:40.077364+0000","last_became_peered":"2024-04-15T18:42:40.077364+0000","last_unstale":"2024-04-15T18:43:56.478565+0000","last_undegraded":"2024-04-15T18:42:39.778417+0000","last_fullsized":"2024-04-15T18:42:39.749589+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413080,"num_objects":3822,"num_object_clones":0,"num_object_copies":11466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9124,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3552,"num_bytes_recovered":232783872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1243}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"759'9183","reported_seq":9359,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:57.603289+0000","last_change":"2024-04-15T18:42:35.114708+0000","last_active":"2024-04-15T18:43:57.603289+0000","last_peered":"2024-04-15T18:43:57.603289+0000","last_clean":"2024-04-15T18:43:57.603289+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:57.603289+0000","last_undegraded":"2024-04-15T18:43:57.603289+0000","last_fullsized":"2024-04-15T18:43:57.603289+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5959","last_scrub_stamp":"2024-04-15T18:42:35.114530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.114530+0000","objects_scrubbed":5915,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:43:58.842544+0000","scrub_duration":2183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251199488,"num_objects":3833,"num_object_clones":0,"num_object_copies":11499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"759'8989","reported_seq":9214,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:55.348326+0000","last_change":"2024-04-15T18:43:51.090805+0000","last_active":"2024-04-15T18:43:55.348326+0000","last_peered":"2024-04-15T18:43:55.348326+0000","last_clean":"2024-04-15T18:43:55.348326+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:43:55.348326+0000","last_undegraded":"2024-04-15T18:43:55.348326+0000","last_fullsized":"2024-04-15T18:43:55.348326+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'8832","last_scrub_stamp":"2024-04-15T18:43:51.090667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:51.090667+0000","objects_scrubbed":4005,"log_size":8989,"log_dups_size":0,"ondisk_log_size":8989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:01.768909+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250806272,"num_objects":3827,"num_object_clones":0,"num_object_copies":11481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"759'8926","reported_seq":9151,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:55.327491+0000","last_change":"2024-04-15T18:43:47.259631+0000","last_active":"2024-04-15T18:43:55.327491+0000","last_peered":"2024-04-15T18:43:55.327491+0000","last_clean":"2024-04-15T18:43:55.327491+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:43:55.327491+0000","last_undegraded":"2024-04-15T18:43:55.327491+0000","last_fullsized":"2024-04-15T18:43:55.327491+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'8626","last_scrub_stamp":"2024-04-15T18:43:47.259412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:47.259412+0000","objects_scrubbed":4127,"log_size":8926,"log_dups_size":0,"ondisk_log_size":8926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:16.497039+0000","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249430016,"num_objects":3806,"num_object_clones":0,"num_object_copies":11418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8926,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"759'9058","reported_seq":9237,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:57.833039+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:43:57.833039+0000","last_peered":"2024-04-15T18:43:57.833039+0000","last_clean":"2024-04-15T18:43:57.833039+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:43:57.833039+0000","last_undegraded":"2024-04-15T18:43:57.833039+0000","last_fullsized":"2024-04-15T18:43:57.833039+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":9058,"log_dups_size":0,"ondisk_log_size":9058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9058,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"759'9080","reported_seq":13021,"reported_epoch":759,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:56.999912+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:43:56.999912+0000","last_peered":"2024-04-15T18:43:56.999912+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:43:56.999912+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":9080,"log_dups_size":0,"ondisk_log_size":9080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238944256,"num_objects":3646,"num_object_clones":0,"num_object_copies":10938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9080,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1913,"num_bytes_recovered":125108224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-481}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"759'9074","reported_seq":15268,"reported_epoch":759,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:58.442467+0000","last_change":"2024-04-15T18:42:39.959765+0000","last_active":"2024-04-15T18:43:58.442467+0000","last_peered":"2024-04-15T18:43:58.442467+0000","last_clean":"2024-04-15T18:42:15.002893+0000","last_became_active":"2024-04-15T18:42:39.939882+0000","last_became_peered":"2024-04-15T18:42:39.939882+0000","last_unstale":"2024-04-15T18:43:58.442467+0000","last_undegraded":"2024-04-15T18:42:39.770656+0000","last_fullsized":"2024-04-15T18:42:39.750615+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238682112,"num_objects":3642,"num_object_clones":0,"num_object_copies":10926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":199229440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":694}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"759'8927","reported_seq":9102,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:58.472408+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:43:58.472408+0000","last_peered":"2024-04-15T18:43:58.472408+0000","last_clean":"2024-04-15T18:43:58.472408+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:43:58.472408+0000","last_undegraded":"2024-04-15T18:43:58.472408+0000","last_fullsized":"2024-04-15T18:43:58.472408+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":8927,"log_dups_size":0,"ondisk_log_size":8927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242679808,"num_objects":3703,"num_object_clones":0,"num_object_copies":11109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"759'8829","reported_seq":9004,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:57.752308+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:43:57.752308+0000","last_peered":"2024-04-15T18:43:57.752308+0000","last_clean":"2024-04-15T18:43:57.752308+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:43:57.752308+0000","last_undegraded":"2024-04-15T18:43:57.752308+0000","last_fullsized":"2024-04-15T18:43:57.752308+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240713728,"num_objects":3673,"num_object_clones":0,"num_object_copies":11019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"759'9119","reported_seq":14666,"reported_epoch":759,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:43:57.003654+0000","last_change":"2024-04-15T18:42:39.993114+0000","last_active":"2024-04-15T18:43:57.003654+0000","last_peered":"2024-04-15T18:43:57.003654+0000","last_clean":"2024-04-15T18:42:15.025871+0000","last_became_active":"2024-04-15T18:42:39.954118+0000","last_became_peered":"2024-04-15T18:42:39.954118+0000","last_unstale":"2024-04-15T18:43:57.003654+0000","last_undegraded":"2024-04-15T18:42:39.885263+0000","last_fullsized":"2024-04-15T18:42:39.747827+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":9119,"log_dups_size":0,"ondisk_log_size":9119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243466240,"num_objects":3715,"num_object_clones":0,"num_object_copies":11145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9119,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2717,"num_bytes_recovered":178061312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":393}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"759'9189","reported_seq":9365,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:58.492273+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:43:58.492273+0000","last_peered":"2024-04-15T18:43:58.492273+0000","last_clean":"2024-04-15T18:43:58.492273+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:43:58.492273+0000","last_undegraded":"2024-04-15T18:43:58.492273+0000","last_fullsized":"2024-04-15T18:43:58.492273+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":9189,"log_dups_size":0,"ondisk_log_size":9189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:44:03.860738+0000","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238092288,"num_objects":3633,"num_object_clones":0,"num_object_copies":10899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"759'8955","reported_seq":9128,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:56.950393+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:43:56.950393+0000","last_peered":"2024-04-15T18:43:56.950393+0000","last_clean":"2024-04-15T18:43:56.950393+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:43:56.950393+0000","last_undegraded":"2024-04-15T18:43:56.950393+0000","last_fullsized":"2024-04-15T18:43:56.950393+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242417664,"num_objects":3699,"num_object_clones":0,"num_object_copies":11097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8955,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"759'9023","reported_seq":9199,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:57.919124+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:43:57.919124+0000","last_peered":"2024-04-15T18:43:57.919124+0000","last_clean":"2024-04-15T18:43:57.919124+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:43:57.919124+0000","last_undegraded":"2024-04-15T18:43:57.919124+0000","last_fullsized":"2024-04-15T18:43:57.919124+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":9023,"log_dups_size":0,"ondisk_log_size":9023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240058368,"num_objects":3663,"num_object_clones":0,"num_object_copies":10989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"759'9030","reported_seq":9205,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:58.497585+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:43:58.497585+0000","last_peered":"2024-04-15T18:43:58.497585+0000","last_clean":"2024-04-15T18:43:58.497585+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:43:58.497585+0000","last_undegraded":"2024-04-15T18:43:58.497585+0000","last_fullsized":"2024-04-15T18:43:58.497585+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":9030,"log_dups_size":0,"ondisk_log_size":9030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243400704,"num_objects":3714,"num_object_clones":0,"num_object_copies":11142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1881,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-04-15T18:43:50.237454+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:43:50.237454+0000","last_peered":"2024-04-15T18:43:50.237454+0000","last_clean":"2024-04-15T18:43:50.237454+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:43:50.237454+0000","last_undegraded":"2024-04-15T18:43:50.237454+0000","last_fullsized":"2024-04-15T18:43:50.237454+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3909287960,"num_objects":59652,"num_object_clones":0,"num_object_copies":178956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144954,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13581,"num_bytes_recovered":889323520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12267106304,"data_stored":12267094088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144954,"ondisk_log_size":144954,"up":48,"acting":43,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967728,"kb_used_data":1413160,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91777424,"statfs":{"total":95995035648,"available":93980082176,"internally_reserved":0,"allocated":1447075840,"data_stored":1446804657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514926,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083320,"kb_used_data":1434352,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91661832,"statfs":{"total":95995035648,"available":93861715968,"internally_reserved":0,"allocated":1468776448,"data_stored":1468501713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1134000,"kb_used_data":741864,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92611152,"statfs":{"total":95995035648,"available":94833819648,"internally_reserved":0,"allocated":759668736,"data_stored":759397553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951868,"kb_used_data":487540,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92793284,"statfs":{"total":95995035648,"available":95020322816,"internally_reserved":0,"allocated":499240960,"data_stored":498962153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":192824,"kb_used_data":25712,"kb_used_omap":111,"kb_used_meta":166992,"kb_avail":93552328,"statfs":{"total":95995035648,"available":95797583872,"internally_reserved":0,"allocated":26329088,"data_stored":26054361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":171000475},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575319,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2071984,"kb_used_data":1449384,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91673168,"statfs":{"total":95995035648,"available":93873324032,"internally_reserved":0,"allocated":1484169216,"data_stored":1483898033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383536,"kb_used_data":734568,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92361616,"statfs":{"total":95995035648,"available":94578294784,"internally_reserved":0,"allocated":752197632,"data_stored":751926449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194091,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1755508,"kb_used_data":1273708,"kb_used_omap":114,"kb_used_meta":481677,"kb_avail":91989644,"statfs":{"total":95995035648,"available":94197395456,"internally_reserved":0,"allocated":1304276992,"data_stored":1304001737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":493237489},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609221,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419056,"kb_used_data":722280,"kb_used_omap":44,"kb_used_meta":696723,"kb_avail":92326096,"statfs":{"total":95995035648,"available":94541922304,"internally_reserved":0,"allocated":739614720,"data_stored":739343537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":713444906},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986628,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586352,"kb_used_data":987368,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92158800,"statfs":{"total":95995035648,"available":94370611200,"internally_reserved":0,"allocated":1011064832,"data_stored":1010793649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713076,"kb_used_data":1273452,"kb_used_omap":4,"kb_used_meta":439611,"kb_avail":92032076,"statfs":{"total":95995035648,"available":94240845824,"internally_reserved":0,"allocated":1304014848,"data_stored":1303739593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":450162369},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":746717184,"data_stored":746717184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1468530688,"data_stored":1468530688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1294536704,"data_stored":1294532632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":734396416,"data_stored":734396416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1002373120,"data_stored":1002373120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1293357056,"data_stored":1293352984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":26017792,"data_stored":26017792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":493883392,"data_stored":493879320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":756482048,"data_stored":756482048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1463353344,"data_stored":1463353344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1437335552,"data_stored":1437335552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:44:02.158 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.f is not active+clean 2024-04-15T18:44:02.158 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.f', 'version': "759'9299", 'reported_seq': 14132, 'reported_epoch': 759, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:43:57.471273+0000', 'last_change': '2024-04-15T18:42:39.985597+0000', 'last_active': '2024-04-15T18:43:57.471273+0000', 'last_peered': '2024-04-15T18:43:57.471273+0000', 'last_clean': '2024-04-15T18:42:15.038230+0000', 'last_became_active': '2024-04-15T18:42:39.951861+0000', 'last_became_peered': '2024-04-15T18:42:39.951861+0000', 'last_unstale': '2024-04-15T18:43:57.471273+0000', 'last_undegraded': '2024-04-15T18:42:39.779406+0000', 'last_fullsized': '2024-04-15T18:42:39.748827+0000', 'mapping_epoch': 743, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 9299, 'log_dups_size': 0, 'ondisk_log_size': 9299, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 248053760, 'num_objects': 3785, 'num_object_clones': 0, 'num_object_copies': 11355, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3799, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9299, 'num_write_kb': 418688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2359, 'num_bytes_recovered': 154140672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 4], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': -14}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.d is not active+clean 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.d', 'version': "759'9124", 'reported_seq': 16345, 'reported_epoch': 759, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:43:56.478565+0000', 'last_change': '2024-04-15T18:42:40.129165+0000', 'last_active': '2024-04-15T18:43:56.478565+0000', 'last_peered': '2024-04-15T18:43:56.478565+0000', 'last_clean': '2024-04-15T18:42:15.046200+0000', 'last_became_active': '2024-04-15T18:42:40.077364+0000', 'last_became_peered': '2024-04-15T18:42:40.077364+0000', 'last_unstale': '2024-04-15T18:43:56.478565+0000', 'last_undegraded': '2024-04-15T18:42:39.778417+0000', 'last_fullsized': '2024-04-15T18:42:39.749589+0000', 'mapping_epoch': 743, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 9124, 'log_dups_size': 0, 'ondisk_log_size': 9124, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 250413080, 'num_objects': 3822, 'num_object_clones': 0, 'num_object_copies': 11466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2579, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3822, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9124, 'num_write_kb': 414209, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3552, 'num_bytes_recovered': 232783872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2, 5], 'acting': [7, 2], 'avail_no_missing': ['7', '2'], 'object_location_counts': [{'shards': '2,7', 'objects': 1243}], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.7 is not active+clean 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.7', 'version': "759'9080", 'reported_seq': 13021, 'reported_epoch': 759, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:43:56.999912+0000', 'last_change': '2024-04-15T18:42:39.996076+0000', 'last_active': '2024-04-15T18:43:56.999912+0000', 'last_peered': '2024-04-15T18:43:56.999912+0000', 'last_clean': '2024-04-15T18:42:15.060455+0000', 'last_became_active': '2024-04-15T18:42:39.970014+0000', 'last_became_peered': '2024-04-15T18:42:39.970014+0000', 'last_unstale': '2024-04-15T18:43:56.999912+0000', 'last_undegraded': '2024-04-15T18:42:39.784034+0000', 'last_fullsized': '2024-04-15T18:42:39.747843+0000', 'mapping_epoch': 743, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 9080, 'log_dups_size': 0, 'ondisk_log_size': 9080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 238944256, 'num_objects': 3646, 'num_object_clones': 0, 'num_object_copies': 10938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4127, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9080, 'num_write_kb': 407232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1913, 'num_bytes_recovered': 125108224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 10, 2], 'acting': [1, 10], 'avail_no_missing': ['1', '10'], 'object_location_counts': [{'shards': '1,10', 'objects': -481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.6 is not active+clean 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.6', 'version': "759'9074", 'reported_seq': 15268, 'reported_epoch': 759, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:43:58.442467+0000', 'last_change': '2024-04-15T18:42:39.959765+0000', 'last_active': '2024-04-15T18:43:58.442467+0000', 'last_peered': '2024-04-15T18:43:58.442467+0000', 'last_clean': '2024-04-15T18:42:15.002893+0000', 'last_became_active': '2024-04-15T18:42:39.939882+0000', 'last_became_peered': '2024-04-15T18:42:39.939882+0000', 'last_unstale': '2024-04-15T18:43:58.442467+0000', 'last_undegraded': '2024-04-15T18:42:39.770656+0000', 'last_fullsized': '2024-04-15T18:42:39.750615+0000', 'mapping_epoch': 743, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 9074, 'log_dups_size': 0, 'ondisk_log_size': 9074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 238682112, 'num_objects': 3642, 'num_object_clones': 0, 'num_object_copies': 10926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2948, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9074, 'num_write_kb': 406912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3040, 'num_bytes_recovered': 199229440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [10, 7, 2], 'acting': [10, 7], 'avail_no_missing': ['10', '7'], 'object_location_counts': [{'shards': '7,10', 'objects': 694}], 'blocked_by': [], 'up_primary': 10, 'acting_primary': 10, 'purged_snaps': []} 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.3 is not active+clean 2024-04-15T18:44:02.159 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.3', 'version': "759'9119", 'reported_seq': 14666, 'reported_epoch': 759, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:43:57.003654+0000', 'last_change': '2024-04-15T18:42:39.993114+0000', 'last_active': '2024-04-15T18:43:57.003654+0000', 'last_peered': '2024-04-15T18:43:57.003654+0000', 'last_clean': '2024-04-15T18:42:15.025871+0000', 'last_became_active': '2024-04-15T18:42:39.954118+0000', 'last_became_peered': '2024-04-15T18:42:39.954118+0000', 'last_unstale': '2024-04-15T18:43:57.003654+0000', 'last_undegraded': '2024-04-15T18:42:39.885263+0000', 'last_fullsized': '2024-04-15T18:42:39.747827+0000', 'mapping_epoch': 743, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 9119, 'log_dups_size': 0, 'ondisk_log_size': 9119, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 243466240, 'num_objects': 3715, 'num_object_clones': 0, 'num_object_copies': 11145, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9119, 'num_write_kb': 410688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2717, 'num_bytes_recovered': 178061312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5, 6], 'acting': [1, 5], 'avail_no_missing': ['1', '5'], 'object_location_counts': [{'shards': '1,5', 'objects': 393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T18:44:02.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:44:02.287 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:02.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:02.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:02.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:02.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:02.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:02 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:02.393 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:02.422 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:02.445 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:02.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:02.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:02.600 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:02.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:02.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:02.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:02.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:02 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:02.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:02.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:02.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:02.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:02 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:03.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:44:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: pgmap v1976: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 16519/177285 objects degraded (9.318%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:44:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:03 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 16519/177285 objects degraded (9.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: pgmap v1976: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 16519/177285 objects degraded (9.318%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:03 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 16519/177285 objects degraded (9.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: pgmap v1976: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 16519/177285 objects degraded (9.318%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:03 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 16519/177285 objects degraded (9.318%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:05.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:05 smithi159 ceph-mon[45336]: pgmap v1977: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 15830/170124 objects degraded (9.305%); 13 MiB/s, 200 objects/s recovering 2024-04-15T18:44:05.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:05.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:05.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:05 smithi159 ceph-mon[45336]: osdmap e761: 12 total, 11 up, 11 in 2024-04-15T18:44:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:05 smithi049 ceph-mon[40087]: pgmap v1977: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 15830/170124 objects degraded (9.305%); 13 MiB/s, 200 objects/s recovering 2024-04-15T18:44:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:05.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:05 smithi049 ceph-mon[40087]: osdmap e761: 12 total, 11 up, 11 in 2024-04-15T18:44:05.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:05 smithi082 ceph-mon[44206]: pgmap v1977: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 600 op/s; 15830/170124 objects degraded (9.305%); 13 MiB/s, 200 objects/s recovering 2024-04-15T18:44:05.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:05.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:05.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:05 smithi082 ceph-mon[44206]: osdmap e761: 12 total, 11 up, 11 in 2024-04-15T18:44:07.337 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:07.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:07.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:07.508 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:07.586 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:07.591 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:07.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:07 smithi049 ceph-mon[40087]: pgmap v1979: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 15830/168915 objects degraded (9.372%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:44:07.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:07.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:07 smithi159 ceph-mon[45336]: pgmap v1979: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 15830/168915 objects degraded (9.372%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:44:07.678 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:07.682 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:07.708 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:07.757 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:07.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:07 smithi082 ceph-mon[44206]: pgmap v1979: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 582 op/s; 15830/168915 objects degraded (9.372%); 9.2 MiB/s, 147 objects/s recovering 2024-04-15T18:44:07.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:07.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:07.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:07.948 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:07.971 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:07.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:08.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:08.112 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:08.137 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:08.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:08.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:08.276 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:08.307 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:08.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:08.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:08.480 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:08.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 15830/168915 objects degraded (9.372%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:08.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:08 smithi159 ceph-mon[45336]: pgmap v1980: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 14904/164820 objects degraded (9.043%); 15 MiB/s, 233 objects/s recovering 2024-04-15T18:44:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 15830/168915 objects degraded (9.372%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:08.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:08 smithi049 ceph-mon[40087]: pgmap v1980: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 14904/164820 objects degraded (9.043%); 15 MiB/s, 233 objects/s recovering 2024-04-15T18:44:09.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 15830/168915 objects degraded (9.372%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:09.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:08 smithi082 ceph-mon[44206]: pgmap v1980: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 14904/164820 objects degraded (9.043%); 15 MiB/s, 233 objects/s recovering 2024-04-15T18:44:09.359 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:09.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:09.468 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:09.501 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:09.526 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:09.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:09.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:09.685 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:10.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:10 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:10 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:44:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:44:11.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:10 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:44:11.187 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:11.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:11.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:11.323 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:11.349 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:11.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:11.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:11.503 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:11.526 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:11.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:11.634 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:11.722 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:11.749 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:11.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:11.857 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:11.887 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:11.915 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:11.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:12.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:12.092 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:12.462 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:12.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:12.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:12.602 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:44:12.602 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:44:12.618 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:12.796 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:12.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:12.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:12 smithi049 ceph-mon[40087]: pgmap v1982: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 14353/158973 objects degraded (9.029%); 14 MiB/s, 216 objects/s recovering 2024-04-15T18:44:12.899 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:12 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 14353/158973 objects degraded (9.029%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:12.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:12.963 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:12 smithi082 ceph-mon[44206]: pgmap v1982: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 14353/158973 objects degraded (9.029%); 14 MiB/s, 216 objects/s recovering 2024-04-15T18:44:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:12 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 14353/158973 objects degraded (9.029%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:13.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:12 smithi159 ceph-mon[45336]: pgmap v1982: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 14353/158973 objects degraded (9.029%); 14 MiB/s, 216 objects/s recovering 2024-04-15T18:44:13.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:12 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 14353/158973 objects degraded (9.029%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:13.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:44:13.562 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:13.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:13.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:13.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:13.714 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:13.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:13.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:13.855 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:13.896 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:13.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=379543467,config=1059521453,health=3560426059,kv=4070050579,logm=1204550770}) 2024-04-15T18:44:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3817950881}) 2024-04-15T18:44:13.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127519934}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714883664}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2678989160}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934673570}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336489572}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771296527}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913554860}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333588632}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588493553}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103275584}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410242969}) 2024-04-15T18:44:13.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579556774}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276491340}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176101983}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089998309}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165808241}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311135389}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129342851}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245195448}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317916831}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016575418}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711898986}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68843070}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157579730}) 2024-04-15T18:44:13.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3058686628}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173322297}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432526465}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643245195}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510748177}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288834631}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444754975}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551352427}) 2024-04-15T18:44:13.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090365488}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652777382}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1622157871}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589369566}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439759552}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2754255524}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761637606}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700954158}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062419209}) 2024-04-15T18:44:13.970 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492094568}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117219201}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145958376}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946517425}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1684682988}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3435903432}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69088306}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666566213}) 2024-04-15T18:44:13.971 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174255322}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453833757}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816800578}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914821681}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689855194}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1167817373}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022913204}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538434090}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498467100}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572237311}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1103785448}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656521191}) 2024-04-15T18:44:13.972 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742672992}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542032836}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758900916}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042743967}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3670207985}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110114171}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135727134}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4076090556}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875011466}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884214305}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2713558006}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394098222}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424278299}) 2024-04-15T18:44:13.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201461570}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257447658}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388582928}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892869935}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423873885}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762632823}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737175829}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245352571}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018948675}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=118065096}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416638298}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=9} crc {logm=648454413,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=4158870001,mon_config_key=3372135270}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3893592205}) 2024-04-15T18:44:13.974 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4115102829}) 2024-04-15T18:44:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=1742636950,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=389853763,osdmap=2605961843}) 2024-04-15T18:44:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2688855163}) 2024-04-15T18:44:13.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:13 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=992224798}) 2024-04-15T18:44:14.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:14.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=379543467,config=1059521453,health=3560426059,kv=4070050579,logm=1204550770}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3817950881}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127519934}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714883664}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2678989160}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934673570}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336489572}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771296527}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913554860}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333588632}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588493553}) 2024-04-15T18:44:14.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103275584}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410242969}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579556774}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276491340}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176101983}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089998309}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165808241}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311135389}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129342851}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245195448}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317916831}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016575418}) 2024-04-15T18:44:14.028 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711898986}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68843070}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157579730}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3058686628}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173322297}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432526465}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643245195}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510748177}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288834631}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444754975}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551352427}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090365488}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652777382}) 2024-04-15T18:44:14.029 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1622157871}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589369566}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439759552}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2754255524}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761637606}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700954158}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062419209}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492094568}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117219201}) 2024-04-15T18:44:14.030 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145958376}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946517425}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1684682988}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3435903432}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69088306}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666566213}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174255322}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453833757}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816800578}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914821681}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689855194}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1167817373}) 2024-04-15T18:44:14.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022913204}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538434090}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498467100}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572237311}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1103785448}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656521191}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742672992}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542032836}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758900916}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042743967}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3670207985}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110114171}) 2024-04-15T18:44:14.032 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135727134}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4076090556}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875011466}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884214305}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2713558006}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394098222}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424278299}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201461570}) 2024-04-15T18:44:14.033 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257447658}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388582928}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892869935}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423873885}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762632823}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737175829}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245352571}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018948675}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=118065096}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416638298}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=9} crc {logm=648454413,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=4158870001,mon_config_key=3372135270}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3893592205}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4115102829}) 2024-04-15T18:44:14.034 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=1742636950,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=389853763,osdmap=2605961843}) 2024-04-15T18:44:14.035 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2688855163}) 2024-04-15T18:44:14.035 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:13 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=992224798}) 2024-04-15T18:44:14.035 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:14.062 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:14.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:14.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=379543467,config=1059521453,health=3560426059,kv=4070050579,logm=1204550770}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3817950881}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127519934}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714883664}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2678989160}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2934673570}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3336489572}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3771296527}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913554860}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3333588632}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588493553}) 2024-04-15T18:44:14.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4103275584}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1410242969}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2579556774}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2276491340}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176101983}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089998309}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165808241}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311135389}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129342851}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2245195448}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2317916831}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1016575418}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711898986}) 2024-04-15T18:44:14.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=68843070}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3157579730}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3058686628}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4173322297}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1432526465}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3643245195}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510748177}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4288834631}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2444754975}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3551352427}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1090365488}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2652777382}) 2024-04-15T18:44:14.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1622157871}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589369566}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3439759552}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2754255524}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761637606}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700954158}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2062419209}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492094568}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117219201}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3145958376}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3946517425}) 2024-04-15T18:44:14.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1684682988}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3435903432}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69088306}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666566213}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4174255322}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=453833757}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816800578}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2914821681}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2689855194}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1167817373}) 2024-04-15T18:44:14.149 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1022913204}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=538434090}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=498467100}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1572237311}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1103785448}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656521191}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2742672992}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3542032836}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758900916}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4042743967}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3670207985}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110114171}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135727134}) 2024-04-15T18:44:14.150 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4076090556}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3875011466}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2884214305}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2713558006}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3394098222}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424278299}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2201461570}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257447658}) 2024-04-15T18:44:14.151 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2388582928}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892869935}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423873885}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762632823}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1737175829}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1245352571}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018948675}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=118065096}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3416638298}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=59,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=9} crc {logm=648454413,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=4158870001,mon_config_key=3372135270}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3893592205}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4115102829}) 2024-04-15T18:44:14.152 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=78,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=4} crc {mon_config_key=1742636950,monmap=1029477071,osd_metadata=3464201027,osd_pg_creating=389853763,osdmap=2605961843}) 2024-04-15T18:44:14.153 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2688855163}) 2024-04-15T18:44:14.153 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:13 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=11} crc {osdmap=992224798}) 2024-04-15T18:44:14.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:14.214 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:14.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:14 smithi049 ceph-mon[40087]: pgmap v1983: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 13618/151635 objects degraded (8.981%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:14.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:14.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:15.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:14 smithi082 ceph-mon[44206]: pgmap v1983: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 13618/151635 objects degraded (8.981%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:15.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:15.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:14 smithi159 ceph-mon[45336]: pgmap v1983: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 13618/151635 objects degraded (8.981%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:16.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:15 smithi082 ceph-mon[44206]: osdmap e762: 12 total, 11 up, 11 in 2024-04-15T18:44:16.049 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:15 smithi049 ceph-mon[40087]: osdmap e762: 12 total, 11 up, 11 in 2024-04-15T18:44:16.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:15 smithi159 ceph-mon[45336]: osdmap e762: 12 total, 11 up, 11 in 2024-04-15T18:44:16.594 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:44:16.594 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13618/150576 objects degraded (9.044%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2403,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":762,"num_osds":12,"num_up_osds":11,"osd_up_since":1713206535,"num_in_osds":11,"osd_in_since":1713206557,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering+undersized+degraded+remapped","count":5}],"num_pgs":17,"num_pools":2,"num_objects":50192,"data_bytes":3289776696,"bytes_used":15046635520,"bytes_avail":1040898756608,"bytes_total":1055945392128,"degraded_objects":13618,"degraded_total":150576,"degraded_ratio":0.090439379449580276,"recovering_objects_per_sec":221,"recovering_bytes_per_sec":14479453,"recovering_keys_per_sec":0,"num_objects_recovered":2212,"num_bytes_recovered":144834560,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":611},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":55,"modified":"2024-04-15T18:43:37.627946+0000","services":{}},"progress_events":{}} 2024-04-15T18:44:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:16 smithi082 ceph-mon[44206]: pgmap v1985: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 13618/150576 objects degraded (9.044%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:16 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3575855031' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:44:17.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:16 smithi159 ceph-mon[45336]: pgmap v1985: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 13618/150576 objects degraded (9.044%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:17.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:16 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3575855031' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:44:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:16 smithi049 ceph-mon[40087]: pgmap v1985: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 611 op/s; 13618/150576 objects degraded (9.044%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:17.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:16 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3575855031' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:44:17.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 13618/150576 objects degraded (9.044%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:18.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 13618/150576 objects degraded (9.044%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:18.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 13618/150576 objects degraded (9.044%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:18.314 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:18.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:18.344 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:44:18.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:44:18.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:18.478 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:18.502 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:18.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:18.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:18.640 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:18.662 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:18.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:18.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:18.836 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:18.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:18 smithi049 ceph-mon[40087]: pgmap v1986: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 12675/146634 objects degraded (8.644%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:44:19.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:18 smithi082 ceph-mon[44206]: pgmap v1986: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 12675/146634 objects degraded (8.644%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:44:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:18 smithi159 ceph-mon[45336]: pgmap v1986: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 12675/146634 objects degraded (8.644%); 14 MiB/s, 222 objects/s recovering 2024-04-15T18:44:19.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:44:19.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:44:19.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:44:19.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:44:19.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:44:19.517 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:19.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:44:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:19.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:19.562 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:44:19.603 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:44:19.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:19.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:44:19.682 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:19.702 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:44:19.746 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:44:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:44:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:44:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:44:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:44:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:44:19.777 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:19.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:19.885 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:19.917 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:19.943 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:19.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4229296241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:19.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.047 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:44:20.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4229296241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:20.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.080 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:20.111 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:44:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:20.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:44:20.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4229296241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:20.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:20.218 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:20.255 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:20.255 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:20.256 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:20.275 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:20.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:20.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:20.412 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:20.824 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:44:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: pgmap v1987: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 12476/142539 objects degraded (8.753%); 14 MiB/s, 220 objects/s recovering 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2566410438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:21.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:44:21.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: pgmap v1987: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 12476/142539 objects degraded (8.753%); 14 MiB/s, 220 objects/s recovering 2024-04-15T18:44:21.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2566410438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:21.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: pgmap v1987: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 12476/142539 objects degraded (8.753%); 14 MiB/s, 220 objects/s recovering 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:21.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2566410438' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:21.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:44:22.087 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:22.092 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:21 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:22.092 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:44:22.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:21 smithi049 ceph-mon[40087]: osdmap e763: 12 total, 11 up, 11 in 2024-04-15T18:44:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:22.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:21 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:22.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:44:22.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:21 smithi159 ceph-mon[45336]: osdmap e763: 12 total, 11 up, 11 in 2024-04-15T18:44:22.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:22.226 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:22.226 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:21 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:22.226 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:44:22.226 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:21 smithi082 ceph-mon[44206]: osdmap e763: 12 total, 11 up, 11 in 2024-04-15T18:44:22.252 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:22.356 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:22.388 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:22.410 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:22.516 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:22.547 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:22.575 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:22.679 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:22.714 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:22.738 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:22.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:22.846 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:22 smithi082 ceph-mon[44206]: pgmap v1989: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 12139/140895 objects degraded (8.616%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:44:22.846 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 12139/140895 objects degraded (8.616%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:22.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:22 smithi049 ceph-mon[40087]: pgmap v1989: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 12139/140895 objects degraded (8.616%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:44:22.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 12139/140895 objects degraded (8.616%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:22.878 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:22.903 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:22.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:23.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:23.069 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:22 smithi159 ceph-mon[45336]: pgmap v1989: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 12139/140895 objects degraded (8.616%); 12 MiB/s, 184 objects/s recovering 2024-04-15T18:44:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 12139/140895 objects degraded (8.616%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:23.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:44:23.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:44:24.518 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:24.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:24.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:24.659 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:24.680 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:24.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:24.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:24.847 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:24 smithi082 ceph-mon[44206]: pgmap v1990: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 11366/133773 objects degraded (8.496%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:44:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:24 smithi159 ceph-mon[45336]: pgmap v1990: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 11366/133773 objects degraded (8.496%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:44:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:24 smithi049 ceph-mon[40087]: pgmap v1990: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 11366/133773 objects degraded (8.496%); 16 MiB/s, 252 objects/s recovering 2024-04-15T18:44:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:26 smithi159 ceph-mon[45336]: pgmap v1991: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 11366/132579 objects degraded (8.573%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:44:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:26 smithi159 ceph-mon[45336]: osdmap e764: 12 total, 11 up, 11 in 2024-04-15T18:44:27.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:27.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:27.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:26 smithi049 ceph-mon[40087]: pgmap v1991: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 11366/132579 objects degraded (8.573%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:44:27.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:26 smithi049 ceph-mon[40087]: osdmap e764: 12 total, 11 up, 11 in 2024-04-15T18:44:27.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:27.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:27.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:26 smithi082 ceph-mon[44206]: pgmap v1991: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 599 op/s; 11366/132579 objects degraded (8.573%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:44:27.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:26 smithi082 ceph-mon[44206]: osdmap e764: 12 total, 11 up, 11 in 2024-04-15T18:44:27.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:27.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:27.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:27.717 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:28.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 11366/132579 objects degraded (8.573%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:28.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 11366/132579 objects degraded (8.573%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:28.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 11366/132579 objects degraded (8.573%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:29.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:29.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:29.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:28 smithi049 ceph-mon[40087]: pgmap v1993: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 10305/128343 objects degraded (8.029%); 17 MiB/s, 271 objects/s recovering 2024-04-15T18:44:29.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:28 smithi049 ceph-mon[40087]: osdmap e765: 12 total, 11 up, 11 in 2024-04-15T18:44:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:28 smithi082 ceph-mon[44206]: pgmap v1993: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 10305/128343 objects degraded (8.029%); 17 MiB/s, 271 objects/s recovering 2024-04-15T18:44:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:28 smithi082 ceph-mon[44206]: osdmap e765: 12 total, 11 up, 11 in 2024-04-15T18:44:29.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:29.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:29.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:28 smithi159 ceph-mon[45336]: pgmap v1993: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 10305/128343 objects degraded (8.029%); 17 MiB/s, 271 objects/s recovering 2024-04-15T18:44:29.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:28 smithi159 ceph-mon[45336]: osdmap e765: 12 total, 11 up, 11 in 2024-04-15T18:44:30.005 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:30.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:30.143 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:30.170 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:30.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:29 smithi049 ceph-mon[40087]: osdmap e766: 12 total, 11 up, 11 in 2024-04-15T18:44:30.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:29 smithi082 ceph-mon[44206]: osdmap e766: 12 total, 11 up, 11 in 2024-04-15T18:44:30.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:30.323 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:29 smithi159 ceph-mon[45336]: osdmap e766: 12 total, 11 up, 11 in 2024-04-15T18:44:30.323 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:30.350 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:30.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:30.482 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:30.504 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:30.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:30.638 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:30.663 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:30.752 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:30.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:30.797 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:30.821 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:30.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:30.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:30.970 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:30 smithi159 ceph-mon[45336]: pgmap v1996: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 10055/123972 objects degraded (8.111%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:44:31.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:30 smithi159 ceph-mon[45336]: osdmap e767: 12 total, 11 up, 11 in 2024-04-15T18:44:31.189 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:31.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:31.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:30 smithi049 ceph-mon[40087]: pgmap v1996: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 10055/123972 objects degraded (8.111%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:44:31.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:30 smithi049 ceph-mon[40087]: osdmap e767: 12 total, 11 up, 11 in 2024-04-15T18:44:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:30 smithi082 ceph-mon[44206]: pgmap v1996: 17 pgs: 5 active+recovering+undersized+degraded+remapped, 12 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 10055/123972 objects degraded (8.111%); 14 MiB/s, 218 objects/s recovering 2024-04-15T18:44:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:30 smithi082 ceph-mon[44206]: osdmap e767: 12 total, 11 up, 11 in 2024-04-15T18:44:31.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:31.333 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:31.357 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:31.359 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:31.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:31.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:31.489 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:31.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:31.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:31.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:31.658 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:31.783 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:31.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:31.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:31.951 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:32.191 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:44:32.191 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:44:32.191 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:44:32.192 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:44:32.192 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:44:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:44:32.469 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:44:32.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:32 smithi049 ceph-mon[40087]: pgmap v1998: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 9669/122586 objects degraded (7.888%); 18 MiB/s, 294 objects/s recovering 2024-04-15T18:44:32.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:32 smithi049 ceph-mon[40087]: from='client.17241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:32.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 9669/122586 objects degraded (7.888%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:32 smithi082 ceph-mon[44206]: pgmap v1998: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 9669/122586 objects degraded (7.888%); 18 MiB/s, 294 objects/s recovering 2024-04-15T18:44:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:32 smithi082 ceph-mon[44206]: from='client.17241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 9669/122586 objects degraded (7.888%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:32 smithi159 ceph-mon[45336]: pgmap v1998: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 9669/122586 objects degraded (7.888%); 18 MiB/s, 294 objects/s recovering 2024-04-15T18:44:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:32 smithi159 ceph-mon[45336]: from='client.17241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 9669/122586 objects degraded (7.888%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:34.156 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:34.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:34.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:44:34.295 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:34.319 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:44:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:34.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:44:34.423 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:34.453 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:34.481 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:34.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:34.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:34.618 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:34.651 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:34.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:34.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:34.798 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:34.825 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:34.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:34.931 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:34.939 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:34 smithi082 ceph-mon[44206]: pgmap v1999: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 652 op/s; 8593/116589 objects degraded (7.370%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:44:34.977 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:34.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:44:35.003 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:35.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:35.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:44:35.136 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:35.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:34 smithi159 ceph-mon[45336]: pgmap v1999: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 652 op/s; 8593/116589 objects degraded (7.370%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:44:35.157 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:44:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:35.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:44:35.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:34 smithi049 ceph-mon[40087]: pgmap v1999: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 652 op/s; 8593/116589 objects degraded (7.370%); 18 MiB/s, 285 objects/s recovering 2024-04-15T18:44:35.264 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:35.291 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:35.311 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:35.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:35.415 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:35.446 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:35.470 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:35.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:35.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:35.604 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:35.632 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:35.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:35.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:35.812 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:36.039 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:36.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:36.064 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1998,"stamp":"2024-04-15T18:44:31.652650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2678325816,"num_objects":40862,"num_object_clones":0,"num_object_copies":122586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40862,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":163927,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20703,"num_bytes_recovered":1358795008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":13207220,"kb_used_data":7420340,"kb_used_omap":285,"kb_used_meta":5786274,"kb_avail":1017989452,"statfs":{"total":1055945392128,"available":1042421198848,"internally_reserved":0,"allocated":7598428160,"data_stored":7595134093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":5925144789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,4,8,7,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-218300416,"num_objects":-3331,"num_object_clones":0,"num_object_copies":-9993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1697,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1697,"num_bytes_recovered":111214592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758324"},"pg_stats":[{"pgid":"8.f","version":"764'10427","reported_seq":17464,"reported_epoch":764,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:27.525905+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:44:27.525905+0000","last_peered":"2024-04-15T18:44:27.525905+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:44:27.525905+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"718'400","ondisk_log_start":"718'400","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10427,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":226033664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":-45}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"764'10332","reported_seq":10517,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.488449+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:44:27.488449+0000","last_peered":"2024-04-15T18:44:27.488449+0000","last_clean":"2024-04-15T18:44:27.488449+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:44:27.488449+0000","last_undegraded":"2024-04-15T18:44:27.488449+0000","last_fullsized":"2024-04-15T18:44:27.488449+0000","mapping_epoch":709,"log_start":"718'300","ondisk_log_start":"718'300","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169213952,"num_objects":2582,"num_object_clones":0,"num_object_copies":7746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10332,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"767'10433","reported_seq":22836,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-15T18:44:31.464833+0000","last_change":"2024-04-15T18:44:29.972225+0000","last_active":"2024-04-15T18:44:31.464833+0000","last_peered":"2024-04-15T18:44:31.464833+0000","last_clean":"2024-04-15T18:44:31.464833+0000","last_became_active":"2024-04-15T18:44:29.945219+0000","last_became_peered":"2024-04-15T18:44:29.945219+0000","last_unstale":"2024-04-15T18:44:31.464833+0000","last_undegraded":"2024-04-15T18:44:31.464833+0000","last_fullsized":"2024-04-15T18:44:31.464833+0000","mapping_epoch":766,"log_start":"718'400","ondisk_log_start":"718'400","created":709,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626456,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10433,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6131,"num_bytes_recovered":401801216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"764'10391","reported_seq":10624,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.599037+0000","last_change":"2024-04-15T18:44:00.295852+0000","last_active":"2024-04-15T18:44:27.599037+0000","last_peered":"2024-04-15T18:44:27.599037+0000","last_clean":"2024-04-15T18:44:27.599037+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:44:27.599037+0000","last_undegraded":"2024-04-15T18:44:27.599037+0000","last_fullsized":"2024-04-15T18:44:27.599037+0000","mapping_epoch":709,"log_start":"718'300","ondisk_log_start":"718'300","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'9294","last_scrub_stamp":"2024-04-15T18:44:00.295708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:44:00.295708+0000","objects_scrubbed":3744,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:06.552947+0000","scrub_duration":1069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032000,"num_objects":2625,"num_object_clones":0,"num_object_copies":7875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"767'10254","reported_seq":10495,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-15T18:44:30.325702+0000","last_change":"2024-04-15T18:43:51.090805+0000","last_active":"2024-04-15T18:44:30.325702+0000","last_peered":"2024-04-15T18:44:30.325702+0000","last_clean":"2024-04-15T18:44:30.325702+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:44:30.325702+0000","last_undegraded":"2024-04-15T18:44:30.325702+0000","last_fullsized":"2024-04-15T18:44:30.325702+0000","mapping_epoch":709,"log_start":"718'200","ondisk_log_start":"718'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'8832","last_scrub_stamp":"2024-04-15T18:43:51.090667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:51.090667+0000","objects_scrubbed":4005,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:01.768909+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":7686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10254,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"767'10231","reported_seq":10472,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-15T18:44:30.373002+0000","last_change":"2024-04-15T18:43:47.259631+0000","last_active":"2024-04-15T18:44:30.373002+0000","last_peered":"2024-04-15T18:44:30.373002+0000","last_clean":"2024-04-15T18:44:30.373002+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:44:30.373002+0000","last_undegraded":"2024-04-15T18:44:30.373002+0000","last_fullsized":"2024-04-15T18:44:30.373002+0000","mapping_epoch":709,"log_start":"718'200","ondisk_log_start":"718'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'8626","last_scrub_stamp":"2024-04-15T18:43:47.259412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:47.259412+0000","objects_scrubbed":4127,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:16.497039+0000","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"765'10200","reported_seq":10391,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.945882+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:44:27.945882+0000","last_peered":"2024-04-15T18:44:27.945882+0000","last_clean":"2024-04-15T18:44:27.945882+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:44:27.945882+0000","last_undegraded":"2024-04-15T18:44:27.945882+0000","last_fullsized":"2024-04-15T18:44:27.945882+0000","mapping_epoch":709,"log_start":"718'100","ondisk_log_start":"718'100","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"764'10222","reported_seq":15819,"reported_epoch":764,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:27.155273+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:44:27.155273+0000","last_peered":"2024-04-15T18:44:27.155273+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:44:27.155273+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"718'200","ondisk_log_start":"718'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10222,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-800}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"765'10228","reported_seq":19009,"reported_epoch":765,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:28.516750+0000","last_change":"2024-04-15T18:42:39.959765+0000","last_active":"2024-04-15T18:44:28.516750+0000","last_peered":"2024-04-15T18:44:28.516750+0000","last_clean":"2024-04-15T18:42:15.002893+0000","last_became_active":"2024-04-15T18:42:39.939882+0000","last_became_peered":"2024-04-15T18:42:39.939882+0000","last_unstale":"2024-04-15T18:44:28.516750+0000","last_undegraded":"2024-04-15T18:42:39.770656+0000","last_fullsized":"2024-04-15T18:42:39.750615+0000","mapping_epoch":743,"log_start":"718'200","ondisk_log_start":"718'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163053568,"num_objects":2488,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4327,"num_bytes_recovered":283574272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":827}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"765'10043","reported_seq":10230,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:28.470349+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:44:28.470349+0000","last_peered":"2024-04-15T18:44:28.470349+0000","last_clean":"2024-04-15T18:44:28.470349+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:44:28.470349+0000","last_undegraded":"2024-04-15T18:44:28.470349+0000","last_fullsized":"2024-04-15T18:44:28.470349+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169541632,"num_objects":2587,"num_object_clones":0,"num_object_copies":7761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"764'9957","reported_seq":10144,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.924797+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:44:27.924797+0000","last_peered":"2024-04-15T18:44:27.924797+0000","last_clean":"2024-04-15T18:44:27.924797+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:44:27.924797+0000","last_undegraded":"2024-04-15T18:44:27.924797+0000","last_fullsized":"2024-04-15T18:44:27.924797+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":9957,"log_dups_size":0,"ondisk_log_size":9957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166789120,"num_objects":2545,"num_object_clones":0,"num_object_copies":7635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9957,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"764'10298","reported_seq":18497,"reported_epoch":764,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:27.150320+0000","last_change":"2024-04-15T18:42:39.993114+0000","last_active":"2024-04-15T18:44:27.150320+0000","last_peered":"2024-04-15T18:44:27.150320+0000","last_clean":"2024-04-15T18:42:15.025871+0000","last_became_active":"2024-04-15T18:42:39.954118+0000","last_became_peered":"2024-04-15T18:42:39.954118+0000","last_unstale":"2024-04-15T18:44:27.150320+0000","last_undegraded":"2024-04-15T18:42:39.885263+0000","last_fullsized":"2024-04-15T18:42:39.747827+0000","mapping_epoch":743,"log_start":"718'200","ondisk_log_start":"718'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166199296,"num_objects":2536,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2002,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":264568832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"765'10306","reported_seq":10496,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:28.454467+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:44:28.454467+0000","last_peered":"2024-04-15T18:44:28.454467+0000","last_clean":"2024-04-15T18:44:28.454467+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:44:28.454467+0000","last_undegraded":"2024-04-15T18:44:28.454467+0000","last_fullsized":"2024-04-15T18:44:28.454467+0000","mapping_epoch":709,"log_start":"718'300","ondisk_log_start":"718'300","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164888576,"num_objects":2516,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"764'10066","reported_seq":10249,"reported_epoch":764,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.149148+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:44:27.149148+0000","last_peered":"2024-04-15T18:44:27.149148+0000","last_clean":"2024-04-15T18:44:27.149148+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:44:27.149148+0000","last_undegraded":"2024-04-15T18:44:27.149148+0000","last_fullsized":"2024-04-15T18:44:27.149148+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10066,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"765'10168","reported_seq":10355,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:27.924985+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:44:27.924985+0000","last_peered":"2024-04-15T18:44:27.924985+0000","last_clean":"2024-04-15T18:44:27.924985+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:44:27.924985+0000","last_undegraded":"2024-04-15T18:44:27.924985+0000","last_fullsized":"2024-04-15T18:44:27.924985+0000","mapping_epoch":709,"log_start":"718'100","ondisk_log_start":"718'100","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165019648,"num_objects":2518,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"765'10190","reported_seq":10377,"reported_epoch":765,"state":"active+clean","last_fresh":"2024-04-15T18:44:28.424589+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:44:28.424589+0000","last_peered":"2024-04-15T18:44:28.424589+0000","last_clean":"2024-04-15T18:44:28.424589+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:44:28.424589+0000","last_undegraded":"2024-04-15T18:44:28.424589+0000","last_fullsized":"2024-04-15T18:44:28.424589+0000","mapping_epoch":709,"log_start":"718'100","ondisk_log_start":"718'100","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167378944,"num_objects":2554,"num_object_clones":0,"num_object_copies":7662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10190,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1898,"reported_epoch":767,"state":"active+clean","last_fresh":"2024-04-15T18:44:29.943155+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:44:29.943155+0000","last_peered":"2024-04-15T18:44:29.943155+0000","last_clean":"2024-04-15T18:44:29.943155+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:44:29.943155+0000","last_undegraded":"2024-04-15T18:44:29.943155+0000","last_fullsized":"2024-04-15T18:44:29.943155+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2677735448,"num_objects":40860,"num_object_clones":0,"num_object_copies":122580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163746,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20687,"num_bytes_recovered":1354563584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9066196992,"data_stored":9066184776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1536828,"kb_used_data":973936,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92208324,"statfs":{"total":95995035648,"available":94421323776,"internally_reserved":0,"allocated":997310464,"data_stored":997013113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":576387520},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514932,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1641272,"kb_used_data":996460,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92103880,"statfs":{"total":95995035648,"available":94314373120,"internally_reserved":0,"allocated":1020375040,"data_stored":1020078490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874678,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883588,"kb_used_data":491452,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92861564,"statfs":{"total":95995035648,"available":95090241536,"internally_reserved":0,"allocated":503246848,"data_stored":502945094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169019,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":811424,"kb_used_data":320724,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92933728,"statfs":{"total":95995035648,"available":95164137472,"internally_reserved":0,"allocated":328421376,"data_stored":328107574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208032,"kb_used_data":45072,"kb_used_omap":111,"kb_used_meta":162832,"kb_avail":93537120,"statfs":{"total":95995035648,"available":95782010880,"internally_reserved":0,"allocated":46153728,"data_stored":45844006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":166740635},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575325,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647212,"kb_used_data":1016164,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92097940,"statfs":{"total":95995035648,"available":94308290560,"internally_reserved":0,"allocated":1040551936,"data_stored":1040258938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1132300,"kb_used_data":483324,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92612852,"statfs":{"total":95995035648,"available":94835560448,"internally_reserved":0,"allocated":494923776,"data_stored":494622022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194097,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400244,"kb_used_data":918440,"kb_used_omap":114,"kb_used_meta":481677,"kb_avail":92344908,"statfs":{"total":95995035648,"available":94561185792,"internally_reserved":0,"allocated":940482560,"data_stored":940185490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":493237489},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609227,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1198128,"kb_used_data":501348,"kb_used_omap":44,"kb_used_meta":696723,"kb_avail":92547024,"statfs":{"total":95995035648,"available":94768152576,"internally_reserved":0,"allocated":513380352,"data_stored":513087354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":713444906},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986634,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330028,"kb_used_data":699108,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92415124,"statfs":{"total":95995035648,"available":94633086976,"internally_reserved":0,"allocated":715886592,"data_stored":715593594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703528,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418164,"kb_used_data":974312,"kb_used_omap":4,"kb_used_meta":443771,"kb_avail":92326988,"statfs":{"total":95995035648,"available":94542835712,"internally_reserved":0,"allocated":997695488,"data_stored":997398418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":454422209},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":488898560,"data_stored":488898560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1021968384,"data_stored":1021968384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":925831168,"data_stored":925827096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":506134528,"data_stored":506134528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":707330048,"data_stored":707330048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":992874496,"data_stored":992870424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46137344,"data_stored":46137344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":323227648,"data_stored":323223576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":499318784,"data_stored":499318784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1013383168,"data_stored":1013383168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":990969856,"data_stored":990969856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:44:36.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:36.203 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:37.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:36 smithi049 ceph-mon[40087]: pgmap v2000: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 8593/115365 objects degraded (7.449%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:37.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:36 smithi082 ceph-mon[44206]: pgmap v2000: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 8593/115365 objects degraded (7.449%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:36 smithi159 ceph-mon[45336]: pgmap v2000: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 8593/115365 objects degraded (7.449%); 14 MiB/s, 221 objects/s recovering 2024-04-15T18:44:38.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 8593/115365 objects degraded (7.449%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:38.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 8593/115365 objects degraded (7.449%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 8593/115365 objects degraded (7.449%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:39.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:44:39.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:38 smithi082 ceph-mon[44206]: pgmap v2001: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 8000/111117 objects degraded (7.200%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:44:39.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:39.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:39.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:38 smithi082 ceph-mon[44206]: osdmap e768: 12 total, 11 up, 11 in 2024-04-15T18:44:39.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:38 smithi159 ceph-mon[45336]: pgmap v2001: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 8000/111117 objects degraded (7.200%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:44:39.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:39.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:39.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:38 smithi159 ceph-mon[45336]: osdmap e768: 12 total, 11 up, 11 in 2024-04-15T18:44:39.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:38 smithi049 ceph-mon[40087]: pgmap v2001: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 8000/111117 objects degraded (7.200%); 15 MiB/s, 235 objects/s recovering 2024-04-15T18:44:39.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:39.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:39.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:38 smithi049 ceph-mon[40087]: osdmap e768: 12 total, 11 up, 11 in 2024-04-15T18:44:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:40 smithi082 ceph-mon[44206]: pgmap v2003: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 7709/106872 objects degraded (7.213%); 15 MiB/s, 241 objects/s recovering 2024-04-15T18:44:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:40 smithi159 ceph-mon[45336]: pgmap v2003: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 7709/106872 objects degraded (7.213%); 15 MiB/s, 241 objects/s recovering 2024-04-15T18:44:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:40 smithi049 ceph-mon[40087]: pgmap v2003: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 7709/106872 objects degraded (7.213%); 15 MiB/s, 241 objects/s recovering 2024-04-15T18:44:42.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:44:42.592 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:44:42.645 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:44:42.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:44:42.741 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:44:42.786 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:44:42.833 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:44:42.860 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:42.890 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:44:42.932 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:44:42.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:44:43.013 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:44:43.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:44:43.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:44:43.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:44:43.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:44:43.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:44:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:44:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:44:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:44:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:44:43.017 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:44:43.018 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:44:43.018 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:44:43.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:44:43.062 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:43.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:43.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:43.228 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: pgmap v2004: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 7709/104940 objects degraded (7.346%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1888959858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 7709/104940 objects degraded (7.346%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:43.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1005189523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:43.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:43 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: pgmap v2004: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 7709/104940 objects degraded (7.346%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1888959858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 7709/104940 objects degraded (7.346%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:43.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1005189523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:43.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:43 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:43.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: pgmap v2004: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 7709/104940 objects degraded (7.346%); 12 MiB/s, 195 objects/s recovering 2024-04-15T18:44:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1888959858' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:44:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 7709/104940 objects degraded (7.346%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1005189523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:43 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:44:44.194 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:44.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:44.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:44.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osdmap e769: 12 total, 11 up, 11 in 2024-04-15T18:44:44.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:44.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osdmap e769: 12 total, 11 up, 11 in 2024-04-15T18:44:44.309 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.337 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osdmap e769: 12 total, 11 up, 11 in 2024-04-15T18:44:44.338 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:44:44.341 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:44.370 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:44.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:44.475 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:44:44.503 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:44.532 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:44:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:44.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:44:44.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:44.674 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:44.694 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:44.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:44.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:44.881 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:44.987 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:44.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:45.008 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:45 smithi159 ceph-mon[45336]: pgmap v2006: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 6757/96945 objects degraded (6.970%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:44:45.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:45.128 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:45.151 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:45.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:45.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:45.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:45 smithi082 ceph-mon[44206]: pgmap v2006: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 6757/96945 objects degraded (6.970%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:44:45.312 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:45 smithi049 ceph-mon[40087]: pgmap v2006: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 6757/96945 objects degraded (6.970%); 14 MiB/s, 229 objects/s recovering 2024-04-15T18:44:45.325 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:45.792 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:45.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:45.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:45.935 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:45.956 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:45.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:46.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:46.092 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.114 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:46.219 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:46.247 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.269 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:46.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:46.406 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.434 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:46.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:46.535 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:46.568 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.600 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:46.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:46.735 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.760 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:46.864 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:44:46.899 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:46.922 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:44:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:46.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:44:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:44:47.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:44:47.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:47.103 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:47 smithi082 ceph-mon[44206]: pgmap v2007: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 6757/95757 objects degraded (7.056%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:44:47.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:47 smithi159 ceph-mon[45336]: pgmap v2007: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 6757/95757 objects degraded (7.056%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:44:47.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:47 smithi049 ceph-mon[40087]: pgmap v2007: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 6757/95757 objects degraded (7.056%); 9.7 MiB/s, 155 objects/s recovering 2024-04-15T18:44:47.485 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:47.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:47.572 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:47.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:44:47.664 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:47.695 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:44:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:47.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:44:47.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:47.850 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:47.876 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:47.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:47.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:48.018 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:48.050 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:48.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:48.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:48 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 6757/95757 objects degraded (7.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:48.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:48 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 6757/95757 objects degraded (7.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:48.155 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:44:48.187 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:48.211 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:44:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:48.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:44:48.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:44:48.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:48 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 6757/95757 objects degraded (7.056%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:48.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:48.409 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:48.433 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:44:48.433 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:44:49.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:44:49.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:49 smithi159 ceph-mon[45336]: pgmap v2008: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 6091/91488 objects degraded (6.658%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:44:49.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:49 smithi049 ceph-mon[40087]: pgmap v2008: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 6091/91488 objects degraded (6.658%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:44:49.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:49 smithi082 ceph-mon[44206]: pgmap v2008: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 6091/91488 objects degraded (6.658%); 11 MiB/s, 179 objects/s recovering 2024-04-15T18:44:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:50 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:44:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:50 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:44:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:50 smithi159 ceph-mon[45336]: osdmap e770: 12 total, 11 up, 11 in 2024-04-15T18:44:50.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:50 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:44:50.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:50 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:44:50.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:50 smithi049 ceph-mon[40087]: osdmap e770: 12 total, 11 up, 11 in 2024-04-15T18:44:50.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:50 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:44:50.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:50 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:44:50.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:50 smithi082 ceph-mon[44206]: osdmap e770: 12 total, 11 up, 11 in 2024-04-15T18:44:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:51 smithi159 ceph-mon[45336]: pgmap v2010: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 5829/87114 objects degraded (6.691%); 15 MiB/s, 234 objects/s recovering 2024-04-15T18:44:51.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:51 smithi049 ceph-mon[40087]: pgmap v2010: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 5829/87114 objects degraded (6.691%); 15 MiB/s, 234 objects/s recovering 2024-04-15T18:44:51.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:51 smithi082 ceph-mon[44206]: pgmap v2010: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 5829/87114 objects degraded (6.691%); 15 MiB/s, 234 objects/s recovering 2024-04-15T18:44:51.804 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:51.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:51.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:44:51.964 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:52.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:44:52.461 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:44:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:52.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:44:52.575 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:52.604 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:52.624 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:52.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:52.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:52.792 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:52.932 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:52.933 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:52.934 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:52.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:52.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:52.998 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:44:53.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:44:53.068 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:53.088 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:44:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:53.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:44:53.191 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:44:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:53 smithi049 ceph-mon[40087]: pgmap v2011: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5829/85374 objects degraded (6.828%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:44:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 5829/85374 objects degraded (6.828%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:53.223 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:53.247 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:44:53.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:53.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:44:53.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:44:53.371 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:44:53.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:53 smithi159 ceph-mon[45336]: pgmap v2011: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5829/85374 objects degraded (6.828%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:44:53.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 5829/85374 objects degraded (6.828%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:53.384 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:53.385 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:44:53.416 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:44:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:53.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:44:53.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:44:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:53 smithi082 ceph-mon[44206]: pgmap v2011: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5829/85374 objects degraded (6.828%); 8.8 MiB/s, 140 objects/s recovering 2024-04-15T18:44:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 5829/85374 objects degraded (6.828%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:53.574 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:53.607 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:44:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:53.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:44:53.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:53.792 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:54.067 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:54 smithi049 ceph-mon[40087]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:54.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:54 smithi159 ceph-mon[45336]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:54.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:54 smithi082 ceph-mon[44206]: from='client.17253 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:44:55.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:55 smithi159 ceph-mon[45336]: pgmap v2012: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/77460 objects degraded (6.221%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:55.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:55 smithi049 ceph-mon[40087]: pgmap v2012: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/77460 objects degraded (6.221%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:55.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:55 smithi082 ceph-mon[44206]: pgmap v2012: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/77460 objects degraded (6.221%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:56.593 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2011,"stamp":"2024-04-15T18:44:51.667038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1865417272,"num_objects":28458,"num_object_clones":0,"num_object_copies":85374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28458,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":176331,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24543,"num_bytes_recovered":1610125568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160989,"ondisk_log_size":160989,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11178260,"kb_used_data":5281652,"kb_used_omap":285,"kb_used_meta":5895970,"kb_avail":1020018412,"statfs":{"total":1055945392128,"available":1044498853888,"internally_reserved":0,"allocated":5408411648,"data_stored":5404906344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":6037473493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-307101696,"num_objects":-4686,"num_object_clones":0,"num_object_copies":-14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4686,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":79691776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.665749"},"pg_stats":[{"pgid":"8.f","version":"769'11245","reported_seq":20231,"reported_epoch":769,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:47.581925+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:44:47.581925+0000","last_peered":"2024-04-15T18:44:47.581925+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:44:47.581925+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"723'1200","ondisk_log_start":"723'1200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120520704,"num_objects":1839,"num_object_clones":0,"num_object_copies":5517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4425,"num_bytes_recovered":289538048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":106}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"769'11094","reported_seq":11288,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.552133+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:44:47.552133+0000","last_peered":"2024-04-15T18:44:47.552133+0000","last_clean":"2024-04-15T18:44:47.552133+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:44:47.552133+0000","last_undegraded":"2024-04-15T18:44:47.552133+0000","last_fullsized":"2024-04-15T18:44:47.552133+0000","mapping_epoch":709,"log_start":"722'1000","ondisk_log_start":"722'1000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119275520,"num_objects":1820,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"770'11281","reported_seq":23689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-15T18:44:51.495249+0000","last_change":"2024-04-15T18:44:29.972225+0000","last_active":"2024-04-15T18:44:51.495249+0000","last_peered":"2024-04-15T18:44:51.495249+0000","last_clean":"2024-04-15T18:44:51.495249+0000","last_became_active":"2024-04-15T18:44:29.945219+0000","last_became_peered":"2024-04-15T18:44:29.945219+0000","last_unstale":"2024-04-15T18:44:51.495249+0000","last_undegraded":"2024-04-15T18:44:51.495249+0000","last_fullsized":"2024-04-15T18:44:51.495249+0000","mapping_epoch":766,"log_start":"723'1200","ondisk_log_start":"723'1200","created":709,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051928,"num_objects":1665,"num_object_clones":0,"num_object_copies":4995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11281,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6131,"num_bytes_recovered":401801216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"769'11106","reported_seq":11349,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.600829+0000","last_change":"2024-04-15T18:44:00.295852+0000","last_active":"2024-04-15T18:44:47.600829+0000","last_peered":"2024-04-15T18:44:47.600829+0000","last_clean":"2024-04-15T18:44:47.600829+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:44:47.600829+0000","last_undegraded":"2024-04-15T18:44:47.600829+0000","last_fullsized":"2024-04-15T18:44:47.600829+0000","mapping_epoch":709,"log_start":"722'1100","ondisk_log_start":"722'1100","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'9294","last_scrub_stamp":"2024-04-15T18:44:00.295708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:44:00.295708+0000","objects_scrubbed":3744,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:06.552947+0000","scrub_duration":1069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125173760,"num_objects":1910,"num_object_clones":0,"num_object_copies":5730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11106,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"770'11051","reported_seq":11298,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-15T18:44:50.397860+0000","last_change":"2024-04-15T18:43:51.090805+0000","last_active":"2024-04-15T18:44:50.397860+0000","last_peered":"2024-04-15T18:44:50.397860+0000","last_clean":"2024-04-15T18:44:50.397860+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:44:50.397860+0000","last_undegraded":"2024-04-15T18:44:50.397860+0000","last_fullsized":"2024-04-15T18:44:50.397860+0000","mapping_epoch":709,"log_start":"722'1000","ondisk_log_start":"722'1000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'8832","last_scrub_stamp":"2024-04-15T18:43:51.090667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:51.090667+0000","objects_scrubbed":4005,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:01.768909+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115671040,"num_objects":1765,"num_object_clones":0,"num_object_copies":5295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"770'11036","reported_seq":11283,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-15T18:44:50.381238+0000","last_change":"2024-04-15T18:43:47.259631+0000","last_active":"2024-04-15T18:44:50.381238+0000","last_peered":"2024-04-15T18:44:50.381238+0000","last_clean":"2024-04-15T18:44:50.381238+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:44:50.381238+0000","last_undegraded":"2024-04-15T18:44:50.381238+0000","last_fullsized":"2024-04-15T18:44:50.381238+0000","mapping_epoch":709,"log_start":"722'1000","ondisk_log_start":"722'1000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'8626","last_scrub_stamp":"2024-04-15T18:43:47.259412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:47.259412+0000","objects_scrubbed":4127,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:16.497039+0000","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111149056,"num_objects":1696,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11036,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"769'10988","reported_seq":11187,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.936485+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:44:47.936485+0000","last_peered":"2024-04-15T18:44:47.936485+0000","last_clean":"2024-04-15T18:44:47.936485+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:44:47.936485+0000","last_undegraded":"2024-04-15T18:44:47.936485+0000","last_fullsized":"2024-04-15T18:44:47.936485+0000","mapping_epoch":709,"log_start":"721'900","ondisk_log_start":"721'900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117702656,"num_objects":1796,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"769'10976","reported_seq":17974,"reported_epoch":769,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:47.235786+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:44:47.235786+0000","last_peered":"2024-04-15T18:44:47.235786+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:44:47.235786+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"721'900","ondisk_log_start":"721'900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688000,"num_objects":1750,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10976,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3431,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-859}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"769'10981","reported_seq":22106,"reported_epoch":769,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:48.560482+0000","last_change":"2024-04-15T18:42:39.959765+0000","last_active":"2024-04-15T18:44:48.560482+0000","last_peered":"2024-04-15T18:44:48.560482+0000","last_clean":"2024-04-15T18:42:15.002893+0000","last_became_active":"2024-04-15T18:42:39.939882+0000","last_became_peered":"2024-04-15T18:42:39.939882+0000","last_unstale":"2024-04-15T18:44:48.560482+0000","last_undegraded":"2024-04-15T18:42:39.770656+0000","last_fullsized":"2024-04-15T18:42:39.750615+0000","mapping_epoch":743,"log_start":"721'900","ondisk_log_start":"721'900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113704960,"num_objects":1735,"num_object_clones":0,"num_object_copies":5205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5496,"num_bytes_recovered":360185856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":1243}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"769'10836","reported_seq":11030,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:48.506617+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:44:48.506617+0000","last_peered":"2024-04-15T18:44:48.506617+0000","last_clean":"2024-04-15T18:44:48.506617+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:44:48.506617+0000","last_undegraded":"2024-04-15T18:44:48.506617+0000","last_fullsized":"2024-04-15T18:44:48.506617+0000","mapping_epoch":709,"log_start":"721'800","ondisk_log_start":"721'800","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117571584,"num_objects":1794,"num_object_clones":0,"num_object_copies":5382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10836,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"769'10743","reported_seq":10938,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.901504+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:44:47.901504+0000","last_peered":"2024-04-15T18:44:47.901504+0000","last_clean":"2024-04-15T18:44:47.901504+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:44:47.901504+0000","last_undegraded":"2024-04-15T18:44:47.901504+0000","last_fullsized":"2024-04-15T18:44:47.901504+0000","mapping_epoch":709,"log_start":"720'700","ondisk_log_start":"720'700","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115277824,"num_objects":1759,"num_object_clones":0,"num_object_copies":5277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"769'11034","reported_seq":21258,"reported_epoch":769,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:44:47.254841+0000","last_change":"2024-04-15T18:42:39.993114+0000","last_active":"2024-04-15T18:44:47.254841+0000","last_peered":"2024-04-15T18:44:47.254841+0000","last_clean":"2024-04-15T18:42:15.025871+0000","last_became_active":"2024-04-15T18:42:39.954118+0000","last_became_peered":"2024-04-15T18:42:39.954118+0000","last_unstale":"2024-04-15T18:44:47.254841+0000","last_undegraded":"2024-04-15T18:42:39.885263+0000","last_fullsized":"2024-04-15T18:42:39.747827+0000","mapping_epoch":743,"log_start":"722'1000","ondisk_log_start":"722'1000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11034,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5044,"num_bytes_recovered":330563584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"769'11067","reported_seq":11265,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:48.520575+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:44:48.520575+0000","last_peered":"2024-04-15T18:44:48.520575+0000","last_clean":"2024-04-15T18:44:48.520575+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:44:48.520575+0000","last_undegraded":"2024-04-15T18:44:48.520575+0000","last_fullsized":"2024-04-15T18:44:48.520575+0000","mapping_epoch":709,"log_start":"722'1000","ondisk_log_start":"722'1000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115015680,"num_objects":1755,"num_object_clones":0,"num_object_copies":5265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"769'10838","reported_seq":11032,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.167248+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:44:47.167248+0000","last_peered":"2024-04-15T18:44:47.167248+0000","last_clean":"2024-04-15T18:44:47.167248+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:44:47.167248+0000","last_undegraded":"2024-04-15T18:44:47.167248+0000","last_fullsized":"2024-04-15T18:44:47.167248+0000","mapping_epoch":709,"log_start":"721'800","ondisk_log_start":"721'800","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119013376,"num_objects":1816,"num_object_clones":0,"num_object_copies":5448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10838,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"769'10914","reported_seq":11110,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:47.888109+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:44:47.888109+0000","last_peered":"2024-04-15T18:44:47.888109+0000","last_clean":"2024-04-15T18:44:47.888109+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:44:47.888109+0000","last_undegraded":"2024-04-15T18:44:47.888109+0000","last_fullsized":"2024-04-15T18:44:47.888109+0000","mapping_epoch":709,"log_start":"721'900","ondisk_log_start":"721'900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10914,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"769'10960","reported_seq":11156,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-04-15T18:44:48.538043+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:44:48.538043+0000","last_peered":"2024-04-15T18:44:48.538043+0000","last_clean":"2024-04-15T18:44:48.538043+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:44:48.538043+0000","last_undegraded":"2024-04-15T18:44:48.538043+0000","last_fullsized":"2024-04-15T18:44:48.538043+0000","mapping_epoch":709,"log_start":"722'900","ondisk_log_start":"722'900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116916224,"num_objects":1784,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1904,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-04-15T18:44:49.050571+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:44:49.050571+0000","last_peered":"2024-04-15T18:44:49.050571+0000","last_clean":"2024-04-15T18:44:49.050571+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:44:49.050571+0000","last_undegraded":"2024-04-15T18:44:49.050571+0000","last_fullsized":"2024-04-15T18:44:49.050571+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1864826904,"num_objects":28456,"num_object_clones":0,"num_object_copies":85368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28456,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176150,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24527,"num_bytes_recovered":1605894144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6888894464,"data_stored":6888882248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247400,"kb_used_data":680288,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":92497752,"statfs":{"total":95995035648,"available":94717698048,"internally_reserved":0,"allocated":696614912,"data_stored":696299920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":580647360},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514936,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350640,"kb_used_data":705832,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92394512,"statfs":{"total":95995035648,"available":94611980288,"internally_reserved":0,"allocated":722771968,"data_stored":722453424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874682,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758136,"kb_used_data":339628,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92987016,"statfs":{"total":95995035648,"available":95218704384,"internally_reserved":0,"allocated":347779072,"data_stored":347459698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169023,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713728,"kb_used_data":213304,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93031424,"statfs":{"total":95995035648,"available":95264178176,"internally_reserved":0,"allocated":218423296,"data_stored":218096298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":512424394},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257852,"kb_used_data":54644,"kb_used_omap":111,"kb_used_meta":203088,"kb_avail":93487300,"statfs":{"total":95995035648,"available":95730995200,"internally_reserved":0,"allocated":55955456,"data_stored":55632538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":207962779},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575329,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1352552,"kb_used_data":703584,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92392600,"statfs":{"total":95995035648,"available":94610022400,"internally_reserved":0,"allocated":720470016,"data_stored":720155024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971640,"kb_used_data":330988,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92773512,"statfs":{"total":95995035648,"available":95000076288,"internally_reserved":0,"allocated":338931712,"data_stored":338612338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194101,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193388,"kb_used_data":683812,"kb_used_omap":114,"kb_used_meta":509453,"kb_avail":92551764,"statfs":{"total":95995035648,"available":94773006336,"internally_reserved":0,"allocated":700223488,"data_stored":699904424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":521680113},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609231,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1045480,"kb_used_data":356896,"kb_used_omap":44,"kb_used_meta":688403,"kb_avail":92699672,"statfs":{"total":95995035648,"available":94924464128,"internally_reserved":0,"allocated":365461504,"data_stored":365146512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":704925226},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986638,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144168,"kb_used_data":513184,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92600984,"statfs":{"total":95995035648,"available":94823407616,"internally_reserved":0,"allocated":525500416,"data_stored":525185424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1143276,"kb_used_data":699492,"kb_used_omap":4,"kb_used_meta":443771,"kb_avail":92601876,"statfs":{"total":95995035648,"available":94824321024,"internally_reserved":0,"allocated":716279808,"data_stored":715960744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":454422209},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331350016,"data_stored":331350016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":718209024,"data_stored":718209024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":698945536,"data_stored":698941464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357892096,"data_stored":357892096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":517406720,"data_stored":517406720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":701960192,"data_stored":701956120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":54919168,"data_stored":54919168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":213913600,"data_stored":213909528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":343343104,"data_stored":343343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":712835072,"data_stored":712835072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":687996928,"data_stored":687996928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:44:56.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:44:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:57 smithi049 ceph-mon[40087]: pgmap v2013: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/76254 objects degraded (6.320%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:57 smithi049 ceph-mon[40087]: osdmap e771: 12 total, 11 up, 11 in 2024-04-15T18:44:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:57 smithi159 ceph-mon[45336]: pgmap v2013: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/76254 objects degraded (6.320%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:57 smithi159 ceph-mon[45336]: osdmap e771: 12 total, 11 up, 11 in 2024-04-15T18:44:57.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:57 smithi082 ceph-mon[44206]: pgmap v2013: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 4819/76254 objects degraded (6.320%); 12 MiB/s, 193 objects/s recovering 2024-04-15T18:44:57.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:57 smithi082 ceph-mon[44206]: osdmap e771: 12 total, 11 up, 11 in 2024-04-15T18:44:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:58 smithi159 ceph-mon[45336]: osdmap e772: 12 total, 11 up, 11 in 2024-04-15T18:44:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 4819/76254 objects degraded (6.320%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:58 smithi159 ceph-mon[45336]: osdmap e773: 12 total, 11 up, 11 in 2024-04-15T18:44:58.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:58 smithi049 ceph-mon[40087]: osdmap e772: 12 total, 11 up, 11 in 2024-04-15T18:44:58.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 4819/76254 objects degraded (6.320%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:58.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:58 smithi049 ceph-mon[40087]: osdmap e773: 12 total, 11 up, 11 in 2024-04-15T18:44:58.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:58 smithi082 ceph-mon[44206]: osdmap e772: 12 total, 11 up, 11 in 2024-04-15T18:44:58.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 4819/76254 objects degraded (6.320%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-04-15T18:44:58.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:58 smithi082 ceph-mon[44206]: osdmap e773: 12 total, 11 up, 11 in 2024-04-15T18:44:58.693 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:58.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:44:58.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:58.836 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:58.859 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:58.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:58.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:44:58.994 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:59.020 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:44:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:59.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:44:59.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:44:59.194 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:44:59.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:44:59 smithi159 ceph-mon[45336]: pgmap v2017: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 3801/72660 objects degraded (5.231%); 21 MiB/s, 337 objects/s recovering 2024-04-15T18:44:59.418 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:44:59 smithi049 ceph-mon[40087]: pgmap v2017: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 3801/72660 objects degraded (5.231%); 21 MiB/s, 337 objects/s recovering 2024-04-15T18:44:59.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:44:59 smithi082 ceph-mon[44206]: pgmap v2017: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 3801/72660 objects degraded (5.231%); 21 MiB/s, 337 objects/s recovering 2024-04-15T18:44:59.975 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:44:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:44:59.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:00.083 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:00.114 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:00.140 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:00.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:00.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:00.277 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:00.305 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:00.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:00.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:00.442 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:00.467 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:00.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:00.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:00.601 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:00.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:00.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:00.728 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:00.772 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:00.889 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:00.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:00.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:01.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:01.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:01 smithi049 ceph-mon[40087]: pgmap v2018: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 3662/69063 objects degraded (5.302%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:45:01.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:01.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:01.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:01 smithi049 ceph-mon[40087]: osdmap e774: 12 total, 11 up, 11 in 2024-04-15T18:45:01.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:01 smithi082 ceph-mon[44206]: pgmap v2018: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 3662/69063 objects degraded (5.302%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:45:01.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:01.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:01.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:01 smithi082 ceph-mon[44206]: osdmap e774: 12 total, 11 up, 11 in 2024-04-15T18:45:01.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:45:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:01 smithi159 ceph-mon[45336]: pgmap v2018: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 3662/69063 objects degraded (5.302%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:45:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:01.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:01 smithi159 ceph-mon[45336]: osdmap e774: 12 total, 11 up, 11 in 2024-04-15T18:45:01.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:45:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:01.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:01.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:45:01.999 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:45:02.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:02.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:45:02.058 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.078 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:02.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:45:02.120 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:45:02.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:45:02.181 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:02.190 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:45:02.216 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.216 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:45:02.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:02.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:45:02.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:45:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:45:02.291 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:45:02.291 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:45:02.291 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:45:02.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:02.371 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:02.371 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2164025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:02.371 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:02.373 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.392 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:02.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:02.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2164025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:02.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:02.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:02.525 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:02.525 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2164025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:02.525 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:02.527 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.555 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:02.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:02.685 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.703 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:02.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:02.849 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:02.873 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:02.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:02.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:03.016 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:03.043 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:03.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:03.148 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:03.178 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:03.206 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:03.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:03.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:03.382 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: pgmap v2020: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 3662/67551 objects degraded (5.421%); 13 MiB/s, 207 objects/s recovering 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osdmap e775: 12 total, 11 up, 11 in 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3662/67551 objects degraded (5.421%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:03.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:03 smithi049 ceph-mon[40087]: osdmap e776: 12 total, 11 up, 11 in 2024-04-15T18:45:03.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: pgmap v2020: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 3662/67551 objects degraded (5.421%); 13 MiB/s, 207 objects/s recovering 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osdmap e775: 12 total, 11 up, 11 in 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3662/67551 objects degraded (5.421%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:03.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:03 smithi082 ceph-mon[44206]: osdmap e776: 12 total, 11 up, 11 in 2024-04-15T18:45:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: pgmap v2020: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 3662/67551 objects degraded (5.421%); 13 MiB/s, 207 objects/s recovering 2024-04-15T18:45:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1885378435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osdmap e775: 12 total, 11 up, 11 in 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3662/67551 objects degraded (5.421%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:03.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:03 smithi159 ceph-mon[45336]: osdmap e776: 12 total, 11 up, 11 in 2024-04-15T18:45:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:05 smithi049 ceph-mon[40087]: pgmap v2023: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 2714/59934 objects degraded (4.528%); 11 MiB/s, 181 objects/s recovering 2024-04-15T18:45:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:05 smithi082 ceph-mon[44206]: pgmap v2023: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 2714/59934 objects degraded (4.528%); 11 MiB/s, 181 objects/s recovering 2024-04-15T18:45:05.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:05 smithi159 ceph-mon[45336]: pgmap v2023: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 2714/59934 objects degraded (4.528%); 11 MiB/s, 181 objects/s recovering 2024-04-15T18:45:05.935 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:185128 8.7 e776/743 8:edc9397a:::benchmark_data_smithi159_162467_object82823:head v 776'11577, mlcod=737'5334)", 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:03.320254+0000", 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.013757343999999999, 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013780377, 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:45:05.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320254+0000", 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320254+0000", 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320254+0000", 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.67e-07 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320258+0000", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9820000000000002e-06 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320307+0000", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8801999999999998e-05 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.320309+0000", 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.835e-06 2024-04-15T18:45:05.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.331079+0000", 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010769875 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.331103+0000", 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4022000000000001e-05 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:185135 8.7 e776/743)", 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:03.327377+0000", 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0066337389999999996, 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006696512, 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327377+0000", 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327377+0000", 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327378+0000", 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0380000000000001e-06 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327386+0000", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0090000000000007e-06 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327432+0000", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5459e-05 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327440+0000", 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.772e-06 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:185138 8.f 8.732a3fff (undecoded) ondisk+write+known_if_redirected e776)", 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:03.327762+0000", 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0062492099999999998, 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0063411420000000001, 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:05.943 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185138 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327762+0000", 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327762+0000", 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:45:05.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327762+0000", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9699999999999996e-07 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327765+0000", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7049999999999999e-06 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327787+0000", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1990000000000001e-05 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:45:05.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.327794+0000", 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8560000000000001e-06 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:185139 8.1 e776/709 8:8365ddd5:::benchmark_data_smithi159_162467_object82834:head v 776'11445, mlcod=776'11442)", 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:03.332737+0000", 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001273773, 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001401128, 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:45:05.946 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332737+0000", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332737+0000", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332738+0000", 2024-04-15T18:45:05.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0999999999999997e-07 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332746+0000", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0129999999999993e-06 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332787+0000", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0638e-05 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.332795+0000", 2024-04-15T18:45:05.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2300000000000008e-06 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.333025+0000", 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000230292 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.333047+0000", 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1693999999999999e-05 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:185123 8.f 8:fcf0809e:::benchmark_data_smithi159_162467_object82818:head [delete] snapc 0=[] ondisk+write+known_if_redirected e776)", 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:03.312109+0000", 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.021902292, 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.022067185999999999, 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185123 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312109+0000", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312109+0000", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312110+0000", 2024-04-15T18:45:05.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6130000000000001e-06 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312117+0000", 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0509999999999999e-06 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312136+0000", 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9125000000000001e-05 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:45:05.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312145+0000", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5040000000000002e-06 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312567+0000", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00042194200000000002 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312705+0000", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013831600000000001 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:45:05.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.312846+0000", 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014132700000000001 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.331040+0000", 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.018193331 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:03.331844+0000", 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080406800000000001 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:45:05.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:45:05.955 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:45:05.955 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 5 2024-04-15T18:45:05.955 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:45:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:45:06.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:45:07.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:07 smithi049 ceph-mon[40087]: pgmap v2024: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 2714/58602 objects degraded (4.631%); 9.5 MiB/s, 157 objects/s recovering 2024-04-15T18:45:07.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:07.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:07.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:07 smithi049 ceph-mon[40087]: osdmap e777: 12 total, 11 up, 11 in 2024-04-15T18:45:07.429 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:07.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:07.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:07 smithi082 ceph-mon[44206]: pgmap v2024: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 2714/58602 objects degraded (4.631%); 9.5 MiB/s, 157 objects/s recovering 2024-04-15T18:45:07.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:07.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:07.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:07 smithi082 ceph-mon[44206]: osdmap e777: 12 total, 11 up, 11 in 2024-04-15T18:45:07.534 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:07.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:07 smithi159 ceph-mon[45336]: pgmap v2024: 17 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 2714/58602 objects degraded (4.631%); 9.5 MiB/s, 157 objects/s recovering 2024-04-15T18:45:07.565 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:07.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:07.566 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:07 smithi159 ceph-mon[45336]: osdmap e777: 12 total, 11 up, 11 in 2024-04-15T18:45:07.566 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:07.601 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:07.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:07.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:07.756 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:07.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:07.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:07.907 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:08.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:45:08.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2714/58602 objects degraded (4.631%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:08.473 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:08.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:08.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2714/58602 objects degraded (4.631%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:08.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:45:08.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:08.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2714/58602 objects degraded (4.631%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:08.659 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:08.820 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:08.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:08.893 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:08.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:09.011 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.055 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:09.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:09.160 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:09.187 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.210 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:09.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:45:09.310 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:45:09.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:09 smithi049 ceph-mon[40087]: pgmap v2026: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 2100/54162 objects degraded (3.877%); 14 MiB/s, 260 objects/s recovering 2024-04-15T18:45:09.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:09 smithi049 ceph-mon[40087]: 8.4 scrub starts 2024-04-15T18:45:09.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:09 smithi049 ceph-mon[40087]: 8.4 scrub ok 2024-04-15T18:45:09.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:09 smithi049 ceph-mon[40087]: 8.d scrub ok 2024-04-15T18:45:09.509 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:09.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:09.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:09 smithi082 ceph-mon[44206]: pgmap v2026: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 2100/54162 objects degraded (3.877%); 14 MiB/s, 260 objects/s recovering 2024-04-15T18:45:09.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:09 smithi082 ceph-mon[44206]: 8.4 scrub starts 2024-04-15T18:45:09.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:09 smithi082 ceph-mon[44206]: 8.4 scrub ok 2024-04-15T18:45:09.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:09 smithi082 ceph-mon[44206]: 8.d scrub ok 2024-04-15T18:45:09.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:09.647 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:09 smithi159 ceph-mon[45336]: pgmap v2026: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 2100/54162 objects degraded (3.877%); 14 MiB/s, 260 objects/s recovering 2024-04-15T18:45:09.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:09 smithi159 ceph-mon[45336]: 8.4 scrub starts 2024-04-15T18:45:09.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:09 smithi159 ceph-mon[45336]: 8.4 scrub ok 2024-04-15T18:45:09.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:09 smithi159 ceph-mon[45336]: 8.d scrub ok 2024-04-15T18:45:09.677 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:09.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:09.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:45:09.783 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:09.811 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.840 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:09.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:09.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:09.976 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:09.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:10.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:10.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:10.136 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:10.159 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:10.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:10.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:10.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: 8.b scrub starts 2024-04-15T18:45:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: 8.0 scrub starts 2024-04-15T18:45:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: 8.b scrub ok 2024-04-15T18:45:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: from='client.27056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: 8.0 scrub ok 2024-04-15T18:45:10.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:10 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:45:10.305 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:10.335 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:10.343 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2026,"stamp":"2024-04-15T18:45:07.672245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183580728,"num_objects":18054,"num_object_clones":0,"num_object_copies":54162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18054,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":186735,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28272,"num_bytes_recovered":1840484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160793,"ondisk_log_size":160793,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9435712,"kb_used_data":3496008,"kb_used_omap":285,"kb_used_meta":5938978,"kb_avail":1021760960,"statfs":{"total":1055945392128,"available":1046283223040,"internally_reserved":0,"allocated":3579912192,"data_stored":3576091562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":6081513685},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,0,1,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-292487168,"num_objects":-4463,"num_object_clones":0,"num_object_copies":-13389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1562,"num_bytes_recovered":88866816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002151"},"pg_stats":[{"pgid":"8.f","version":"777'12017","reported_seq":23847,"reported_epoch":777,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:45:07.606400+0000","last_change":"2024-04-15T18:42:39.985597+0000","last_active":"2024-04-15T18:45:07.606400+0000","last_peered":"2024-04-15T18:45:07.606400+0000","last_clean":"2024-04-15T18:42:15.038230+0000","last_became_active":"2024-04-15T18:42:39.951861+0000","last_became_peered":"2024-04-15T18:42:39.951861+0000","last_unstale":"2024-04-15T18:45:07.606400+0000","last_undegraded":"2024-04-15T18:42:39.779406+0000","last_fullsized":"2024-04-15T18:42:39.748827+0000","mapping_epoch":743,"log_start":"726'2000","ondisk_log_start":"726'2000","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":3201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5839,"num_bytes_recovered":371064832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"777'11904","reported_seq":12115,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:07.576043+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:45:07.576043+0000","last_peered":"2024-04-15T18:45:07.576043+0000","last_clean":"2024-04-15T18:45:07.576043+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:45:07.576043+0000","last_undegraded":"2024-04-15T18:45:07.576043+0000","last_fullsized":"2024-04-15T18:45:07.576043+0000","mapping_epoch":709,"log_start":"726'1900","ondisk_log_start":"726'1900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66191360,"num_objects":1010,"num_object_clones":0,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"777'11859","reported_seq":24282,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:06.488567+0000","last_change":"2024-04-15T18:44:29.972225+0000","last_active":"2024-04-15T18:45:06.488567+0000","last_peered":"2024-04-15T18:45:06.488567+0000","last_clean":"2024-04-15T18:45:06.488567+0000","last_became_active":"2024-04-15T18:44:29.945219+0000","last_became_peered":"2024-04-15T18:44:29.945219+0000","last_unstale":"2024-04-15T18:45:06.488567+0000","last_undegraded":"2024-04-15T18:45:06.488567+0000","last_fullsized":"2024-04-15T18:45:06.488567+0000","mapping_epoch":766,"log_start":"726'1800","ondisk_log_start":"726'1800","created":709,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:07.065101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71172120,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11859,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6131,"num_bytes_recovered":401801216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"777'11949","reported_seq":12208,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:07.628806+0000","last_change":"2024-04-15T18:44:00.295852+0000","last_active":"2024-04-15T18:45:07.628806+0000","last_peered":"2024-04-15T18:45:07.628806+0000","last_clean":"2024-04-15T18:45:07.628806+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:45:07.628806+0000","last_undegraded":"2024-04-15T18:45:07.628806+0000","last_fullsized":"2024-04-15T18:45:07.628806+0000","mapping_epoch":709,"log_start":"726'1900","ondisk_log_start":"726'1900","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'9294","last_scrub_stamp":"2024-04-15T18:44:00.295708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:44:00.295708+0000","objects_scrubbed":3744,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1069,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69926912,"num_objects":1067,"num_object_clones":0,"num_object_copies":3201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"776'11660","reported_seq":11919,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:05.383966+0000","last_change":"2024-04-15T18:43:51.090805+0000","last_active":"2024-04-15T18:45:05.383966+0000","last_peered":"2024-04-15T18:45:05.383966+0000","last_clean":"2024-04-15T18:45:05.383966+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:45:05.383966+0000","last_undegraded":"2024-04-15T18:45:05.383966+0000","last_fullsized":"2024-04-15T18:45:05.383966+0000","mapping_epoch":709,"log_start":"724'1600","ondisk_log_start":"724'1600","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'8832","last_scrub_stamp":"2024-04-15T18:43:51.090667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:51.090667+0000","objects_scrubbed":4005,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:06.909563+0000","scrub_duration":1048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759616,"num_objects":1156,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"776'11618","reported_seq":11877,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:05.347258+0000","last_change":"2024-04-15T18:43:47.259631+0000","last_active":"2024-04-15T18:45:05.347258+0000","last_peered":"2024-04-15T18:45:05.347258+0000","last_clean":"2024-04-15T18:45:05.347258+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:45:05.347258+0000","last_undegraded":"2024-04-15T18:45:05.347258+0000","last_fullsized":"2024-04-15T18:45:05.347258+0000","mapping_epoch":709,"log_start":"724'1600","ondisk_log_start":"724'1600","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'8626","last_scrub_stamp":"2024-04-15T18:43:47.259412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:43:47.259412+0000","objects_scrubbed":4127,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:16.497039+0000","scrub_duration":1211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73007104,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11618,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"776'11536","reported_seq":11749,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:02.978894+0000","last_change":"2024-04-15T18:42:36.963819+0000","last_active":"2024-04-15T18:45:02.978894+0000","last_peered":"2024-04-15T18:45:02.978894+0000","last_clean":"2024-04-15T18:45:02.978894+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:45:02.978894+0000","last_undegraded":"2024-04-15T18:45:02.978894+0000","last_fullsized":"2024-04-15T18:45:02.978894+0000","mapping_epoch":709,"log_start":"724'1500","ondisk_log_start":"724'1500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5970","last_scrub_stamp":"2024-04-15T18:42:36.963727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.963727+0000","objects_scrubbed":5942,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11536,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"777'11748","reported_seq":20418,"reported_epoch":777,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:45:07.286920+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:45:07.286920+0000","last_peered":"2024-04-15T18:45:07.286920+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:45:07.286920+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"724'1700","ondisk_log_start":"724'1700","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64094208,"num_objects":978,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11748,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4259,"num_bytes_recovered":277544960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-803}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"776'11530","reported_seq":23662,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:03.483512+0000","last_change":"2024-04-15T18:44:57.505649+0000","last_active":"2024-04-15T18:45:03.483512+0000","last_peered":"2024-04-15T18:45:03.483512+0000","last_clean":"2024-04-15T18:45:03.483512+0000","last_became_active":"2024-04-15T18:44:57.475200+0000","last_became_peered":"2024-04-15T18:44:57.475200+0000","last_unstale":"2024-04-15T18:45:03.483512+0000","last_undegraded":"2024-04-15T18:45:03.483512+0000","last_fullsized":"2024-04-15T18:45:03.483512+0000","mapping_epoch":771,"log_start":"724'1500","ondisk_log_start":"724'1500","created":709,"last_epoch_clean":772,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77725696,"num_objects":1186,"num_object_clones":0,"num_object_copies":3558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11530,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5988,"num_bytes_recovered":392429568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"776'11422","reported_seq":11632,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:03.512770+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:45:03.512770+0000","last_peered":"2024-04-15T18:45:03.512770+0000","last_clean":"2024-04-15T18:45:03.512770+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:45:03.512770+0000","last_undegraded":"2024-04-15T18:45:03.512770+0000","last_fullsized":"2024-04-15T18:45:03.512770+0000","mapping_epoch":709,"log_start":"724'1400","ondisk_log_start":"724'1400","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79167488,"num_objects":1208,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"776'11289","reported_seq":11498,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:02.894152+0000","last_change":"2024-04-15T18:42:36.057482+0000","last_active":"2024-04-15T18:45:02.894152+0000","last_peered":"2024-04-15T18:45:02.894152+0000","last_clean":"2024-04-15T18:45:02.894152+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:45:02.894152+0000","last_undegraded":"2024-04-15T18:45:02.894152+0000","last_fullsized":"2024-04-15T18:45:02.894152+0000","mapping_epoch":709,"log_start":"723'1200","ondisk_log_start":"723'1200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5798","last_scrub_stamp":"2024-04-15T18:42:36.057254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.057254+0000","objects_scrubbed":5760,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79495168,"num_objects":1213,"num_object_clones":0,"num_object_copies":3639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"777'11808","reported_seq":24045,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:07.264809+0000","last_change":"2024-04-15T18:45:02.721543+0000","last_active":"2024-04-15T18:45:07.264809+0000","last_peered":"2024-04-15T18:45:07.264809+0000","last_clean":"2024-04-15T18:45:07.264809+0000","last_became_active":"2024-04-15T18:45:02.674121+0000","last_became_peered":"2024-04-15T18:45:02.674121+0000","last_unstale":"2024-04-15T18:45:07.264809+0000","last_undegraded":"2024-04-15T18:45:07.264809+0000","last_fullsized":"2024-04-15T18:45:07.264809+0000","mapping_epoch":775,"log_start":"726'1800","ondisk_log_start":"726'1800","created":709,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6039,"num_bytes_recovered":393412608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"776'11599","reported_seq":11811,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:03.541141+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:45:03.541141+0000","last_peered":"2024-04-15T18:45:03.541141+0000","last_clean":"2024-04-15T18:45:03.541141+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:45:03.541141+0000","last_undegraded":"2024-04-15T18:45:03.541141+0000","last_fullsized":"2024-04-15T18:45:03.541141+0000","mapping_epoch":709,"log_start":"724'1500","ondisk_log_start":"724'1500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"777'11622","reported_seq":11831,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:07.265360+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:45:07.265360+0000","last_peered":"2024-04-15T18:45:07.265360+0000","last_clean":"2024-04-15T18:45:07.265360+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:45:07.265360+0000","last_undegraded":"2024-04-15T18:45:07.265360+0000","last_fullsized":"2024-04-15T18:45:07.265360+0000","mapping_epoch":709,"log_start":"725'1600","ondisk_log_start":"725'1600","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11622,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"776'11466","reported_seq":11674,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:03.001466+0000","last_change":"2024-04-15T18:42:34.888779+0000","last_active":"2024-04-15T18:45:03.001466+0000","last_peered":"2024-04-15T18:45:03.001466+0000","last_clean":"2024-04-15T18:45:03.001466+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:45:03.001466+0000","last_undegraded":"2024-04-15T18:45:03.001466+0000","last_fullsized":"2024-04-15T18:45:03.001466+0000","mapping_epoch":709,"log_start":"723'1400","ondisk_log_start":"723'1400","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5838","last_scrub_stamp":"2024-04-15T18:42:34.888609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.888609+0000","objects_scrubbed":5787,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79953920,"num_objects":1220,"num_object_clones":0,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11466,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"776'11527","reported_seq":11735,"reported_epoch":776,"state":"active+clean","last_fresh":"2024-04-15T18:45:03.549931+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:45:03.549931+0000","last_peered":"2024-04-15T18:45:03.549931+0000","last_clean":"2024-04-15T18:45:03.549931+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:45:03.549931+0000","last_undegraded":"2024-04-15T18:45:03.549931+0000","last_fullsized":"2024-04-15T18:45:03.549931+0000","mapping_epoch":709,"log_start":"724'1500","ondisk_log_start":"724'1500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79757312,"num_objects":1217,"num_object_clones":0,"num_object_copies":3651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11527,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1918,"reported_epoch":777,"state":"active+clean","last_fresh":"2024-04-15T18:45:06.208615+0000","last_change":"2024-04-15T18:42:41.780628+0000","last_active":"2024-04-15T18:45:06.208615+0000","last_peered":"2024-04-15T18:45:06.208615+0000","last_clean":"2024-04-15T18:45:06.208615+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:45:06.208615+0000","last_undegraded":"2024-04-15T18:45:06.208615+0000","last_fullsized":"2024-04-15T18:45:06.208615+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:41:00.313801+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:41:00.313801+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:45:07.065133+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1182990360,"num_objects":18052,"num_object_clones":0,"num_object_copies":54156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186554,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28256,"num_bytes_recovered":1836253184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5052772352,"data_stored":5052760136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187893,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1049924,"kb_used_data":468980,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92695228,"statfs":{"total":95995035648,"available":94919913472,"internally_reserved":0,"allocated":480235520,"data_stored":479889446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":594868672},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514940,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057940,"kb_used_data":408968,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92687212,"statfs":{"total":95995035648,"available":94911705088,"internally_reserved":0,"allocated":418783232,"data_stored":418429166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874685,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635140,"kb_used_data":224948,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93110012,"statfs":{"total":95995035648,"available":95344652288,"internally_reserved":0,"allocated":230346752,"data_stored":230000678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169026,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647944,"kb_used_data":143360,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93097208,"statfs":{"total":95995035648,"available":95331540992,"internally_reserved":0,"allocated":146800640,"data_stored":146446942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":516684234},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284676,"kb_used_data":73148,"kb_used_omap":111,"kb_used_meta":211408,"kb_avail":93460476,"statfs":{"total":95995035648,"available":95703527424,"internally_reserved":0,"allocated":74903552,"data_stored":74553934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":216482459},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050444,"kb_used_data":401152,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92694708,"statfs":{"total":95995035648,"available":94919380992,"internally_reserved":0,"allocated":410779648,"data_stored":410429134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869820,"kb_used_data":220852,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92875332,"statfs":{"total":95995035648,"available":95104339968,"internally_reserved":0,"allocated":226152448,"data_stored":225806374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194105,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957324,"kb_used_data":439428,"kb_used_omap":114,"kb_used_meta":517773,"kb_avail":92787828,"statfs":{"total":95995035648,"available":95014735872,"internally_reserved":0,"allocated":449974272,"data_stored":449619686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":530199793},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609234,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":937460,"kb_used_data":244840,"kb_used_omap":44,"kb_used_meta":692563,"kb_avail":92807692,"statfs":{"total":95995035648,"available":95035076608,"internally_reserved":0,"allocated":250716160,"data_stored":250374568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":709185066},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986641,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016040,"kb_used_data":385116,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92729112,"statfs":{"total":95995035648,"available":94954610688,"internally_reserved":0,"allocated":394358784,"data_stored":394021653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703535,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929000,"kb_used_data":485216,"kb_used_omap":4,"kb_used_meta":443771,"kb_avail":92816152,"statfs":{"total":95995035648,"available":95043739648,"internally_reserved":0,"allocated":496861184,"data_stored":496519981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":454422209},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219938816,"data_stored":219938816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":407502848,"data_stored":407502848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":434245632,"data_stored":434241560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":246808576,"data_stored":246808576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":396230656,"data_stored":396230656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":484577280,"data_stored":484573208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":71499776,"data_stored":71499776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":142282752,"data_stored":142278680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":225509376,"data_stored":225509376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":405405696,"data_stored":405405696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":468647936,"data_stored":468647936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:45:10.344 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.f is not active+clean 2024-04-15T18:45:10.344 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.f', 'version': "777'12017", 'reported_seq': 23847, 'reported_epoch': 777, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:45:07.606400+0000', 'last_change': '2024-04-15T18:42:39.985597+0000', 'last_active': '2024-04-15T18:45:07.606400+0000', 'last_peered': '2024-04-15T18:45:07.606400+0000', 'last_clean': '2024-04-15T18:42:15.038230+0000', 'last_became_active': '2024-04-15T18:42:39.951861+0000', 'last_became_peered': '2024-04-15T18:42:39.951861+0000', 'last_unstale': '2024-04-15T18:45:07.606400+0000', 'last_undegraded': '2024-04-15T18:42:39.779406+0000', 'last_fullsized': '2024-04-15T18:42:39.748827+0000', 'mapping_epoch': 743, 'log_start': "726'2000", 'ondisk_log_start': "726'2000", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 69926912, 'num_objects': 1067, 'num_object_clones': 0, 'num_object_copies': 3201, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12017, 'num_write_kb': 418688, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5839, 'num_bytes_recovered': 371064832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 4], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 748}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T18:45:10.345 INFO:tasks.cephadm.ceph_manager.ceph:PG 8.7 is not active+clean 2024-04-15T18:45:10.345 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '8.7', 'version': "777'11748", 'reported_seq': 20418, 'reported_epoch': 777, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-04-15T18:45:07.286920+0000', 'last_change': '2024-04-15T18:42:39.996076+0000', 'last_active': '2024-04-15T18:45:07.286920+0000', 'last_peered': '2024-04-15T18:45:07.286920+0000', 'last_clean': '2024-04-15T18:42:15.060455+0000', 'last_became_active': '2024-04-15T18:42:39.970014+0000', 'last_became_peered': '2024-04-15T18:42:39.970014+0000', 'last_unstale': '2024-04-15T18:45:07.286920+0000', 'last_undegraded': '2024-04-15T18:42:39.784034+0000', 'last_fullsized': '2024-04-15T18:42:39.747843+0000', 'mapping_epoch': 743, 'log_start': "724'1700", 'ondisk_log_start': "724'1700", 'created': 709, 'last_epoch_clean': 710, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'last_clean_scrub_stamp': '2024-04-15T18:41:01.807048+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 64094208, 'num_objects': 978, 'num_object_clones': 0, 'num_object_copies': 2934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1781, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11748, 'num_write_kb': 407232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4259, 'num_bytes_recovered': 277544960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 10, 2], 'acting': [1, 10], 'avail_no_missing': ['1', '10'], 'object_location_counts': [{'shards': '1,10', 'objects': -803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T18:45:10.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph status --format=json 2024-04-15T18:45:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:10.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:10.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:10.496 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: 8.b scrub starts 2024-04-15T18:45:10.496 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: 8.0 scrub starts 2024-04-15T18:45:10.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: 8.b scrub ok 2024-04-15T18:45:10.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: from='client.27056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:10.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: 8.0 scrub ok 2024-04-15T18:45:10.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:10 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:45:10.498 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:10.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: 8.b scrub starts 2024-04-15T18:45:10.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: 8.0 scrub starts 2024-04-15T18:45:10.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: 8.b scrub ok 2024-04-15T18:45:10.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: from='client.27056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:10.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: 8.0 scrub ok 2024-04-15T18:45:10.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:10 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:45:11.502 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:11.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:11.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:11.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:11 smithi159 ceph-mon[45336]: pgmap v2027: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2100/49443 objects degraded (4.247%); 11 MiB/s, 208 objects/s recovering 2024-04-15T18:45:11.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:11 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:11 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:11 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:11 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:11 smithi049 ceph-mon[40087]: pgmap v2027: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2100/49443 objects degraded (4.247%); 11 MiB/s, 208 objects/s recovering 2024-04-15T18:45:11.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:11 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:11 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:11 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:11.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:11 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:11 smithi082 ceph-mon[44206]: pgmap v2027: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.0 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2100/49443 objects degraded (4.247%); 11 MiB/s, 208 objects/s recovering 2024-04-15T18:45:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:11 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:11 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:11 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:11 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: 8.c scrub starts 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: 8.c scrub ok 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:45:12.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:12 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: 8.c scrub starts 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: 8.c scrub ok 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:45:12.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:12 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:12.905 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:12.905 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:12.908 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:12.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:13.018 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:13.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: 8.c scrub starts 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: 8.c scrub ok 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:45:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:12 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:45:13.050 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:13.077 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:13.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:13.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:13.220 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:13.250 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:13.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:13.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:13.406 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:13.555 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:13.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:13.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:13.689 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:13.713 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:13.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:13.820 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:13.847 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:13 smithi082 ceph-mon[44206]: pgmap v2028: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 990 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 2100/47481 objects degraded (4.423%); 3.1 MiB/s, 68 objects/s recovering 2024-04-15T18:45:13.847 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:13 smithi082 ceph-mon[44206]: osdmap e778: 12 total, 11 up, 11 in 2024-04-15T18:45:13.847 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:13 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2100/47481 objects degraded (4.423%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:13.847 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:13 smithi082 ceph-mon[44206]: osdmap e779: 12 total, 11 up, 11 in 2024-04-15T18:45:13.849 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:13.873 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:13.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:13.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:13 smithi159 ceph-mon[45336]: pgmap v2028: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 990 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 2100/47481 objects degraded (4.423%); 3.1 MiB/s, 68 objects/s recovering 2024-04-15T18:45:13.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:13 smithi159 ceph-mon[45336]: osdmap e778: 12 total, 11 up, 11 in 2024-04-15T18:45:13.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:13 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2100/47481 objects degraded (4.423%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:13.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:13 smithi159 ceph-mon[45336]: osdmap e779: 12 total, 11 up, 11 in 2024-04-15T18:45:13.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:13 smithi049 ceph-mon[40087]: pgmap v2028: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 990 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 2100/47481 objects degraded (4.423%); 3.1 MiB/s, 68 objects/s recovering 2024-04-15T18:45:13.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:13 smithi049 ceph-mon[40087]: osdmap e778: 12 total, 11 up, 11 in 2024-04-15T18:45:13.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:13 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2100/47481 objects degraded (4.423%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-04-15T18:45:13.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:13 smithi049 ceph-mon[40087]: osdmap e779: 12 total, 11 up, 11 in 2024-04-15T18:45:13.930 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:13.977 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:14.021 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:14.048 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:14.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:14.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:14.187 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:14.217 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:14.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:14.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:14.358 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:14.388 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:14.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:14.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:14.561 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:14.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:14.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:14 smithi049 ceph-mon[40087]: pgmap v2031: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 818 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s; 1575/39252 objects degraded (4.013%); 5.4 MiB/s, 152 objects/s recovering 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:14 smithi049 ceph-mon[40087]: osdmap e780: 12 total, 11 up, 11 in 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:14 smithi082 ceph-mon[44206]: pgmap v2031: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 818 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s; 1575/39252 objects degraded (4.013%); 5.4 MiB/s, 152 objects/s recovering 2024-04-15T18:45:14.680 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:14 smithi082 ceph-mon[44206]: osdmap e780: 12 total, 11 up, 11 in 2024-04-15T18:45:14.694 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:14.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:14.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:14.830 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:14.854 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:14.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:14.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:15.017 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:15.115 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:15.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:14 smithi159 ceph-mon[45336]: pgmap v2031: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 818 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 862 op/s; 1575/39252 objects degraded (4.013%); 5.4 MiB/s, 152 objects/s recovering 2024-04-15T18:45:15.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:14 smithi159 ceph-mon[45336]: osdmap e780: 12 total, 11 up, 11 in 2024-04-15T18:45:15.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:15.262 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:15.286 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:15.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:15.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:15.433 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:15.460 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:15.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:15.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:15.594 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:15.616 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:15.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:15.724 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:15.756 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:15.779 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:15.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:15.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:15.946 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:16.936 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:45:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:45:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:45:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:45:16.937 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:45:16.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:16 smithi049 ceph-mon[40087]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 790 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 1575/37887 objects degraded (4.157%); 2.1 MiB/s, 87 objects/s recovering 2024-04-15T18:45:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:16 smithi082 ceph-mon[44206]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 790 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 1575/37887 objects degraded (4.157%); 2.1 MiB/s, 87 objects/s recovering 2024-04-15T18:45:17.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:16 smithi159 ceph-mon[45336]: pgmap v2033: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 790 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 1575/37887 objects degraded (4.157%); 2.1 MiB/s, 87 objects/s recovering 2024-04-15T18:45:17.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:17 smithi049 ceph-mon[40087]: 8.a scrub starts 2024-04-15T18:45:17.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:17 smithi049 ceph-mon[40087]: 8.a scrub ok 2024-04-15T18:45:17.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:17 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1575/37887 objects degraded (4.157%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:17.967 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:17 smithi049 ceph-mon[40087]: osdmap e781: 12 total, 11 up, 11 in 2024-04-15T18:45:18.016 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:18.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:17 smithi082 ceph-mon[44206]: 8.a scrub starts 2024-04-15T18:45:18.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:17 smithi082 ceph-mon[44206]: 8.a scrub ok 2024-04-15T18:45:18.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:17 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1575/37887 objects degraded (4.157%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:18.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:17 smithi082 ceph-mon[44206]: osdmap e781: 12 total, 11 up, 11 in 2024-04-15T18:45:18.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:18.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:17 smithi159 ceph-mon[45336]: 8.a scrub starts 2024-04-15T18:45:18.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:17 smithi159 ceph-mon[45336]: 8.a scrub ok 2024-04-15T18:45:18.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:17 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1575/37887 objects degraded (4.157%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:18.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:17 smithi159 ceph-mon[45336]: osdmap e781: 12 total, 11 up, 11 in 2024-04-15T18:45:18.151 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:18.175 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:18.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:18.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:18.338 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:18.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:18.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:18.717 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:18.736 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:18.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:18.842 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:18.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:18 smithi049 ceph-mon[40087]: 8.8 scrub starts 2024-04-15T18:45:18.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:18 smithi049 ceph-mon[40087]: 8.8 scrub ok 2024-04-15T18:45:18.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:18 smithi049 ceph-mon[40087]: pgmap v2034: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 684 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1339/32793 objects degraded (4.083%); 4.6 MiB/s, 126 objects/s recovering 2024-04-15T18:45:18.875 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:18.899 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:18.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:19.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:19.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:18 smithi082 ceph-mon[44206]: 8.8 scrub starts 2024-04-15T18:45:19.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:18 smithi082 ceph-mon[44206]: 8.8 scrub ok 2024-04-15T18:45:19.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:18 smithi082 ceph-mon[44206]: pgmap v2034: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 684 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1339/32793 objects degraded (4.083%); 4.6 MiB/s, 126 objects/s recovering 2024-04-15T18:45:19.054 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:18 smithi159 ceph-mon[45336]: 8.8 scrub starts 2024-04-15T18:45:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:18 smithi159 ceph-mon[45336]: 8.8 scrub ok 2024-04-15T18:45:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:18 smithi159 ceph-mon[45336]: pgmap v2034: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 684 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 1339/32793 objects degraded (4.083%); 4.6 MiB/s, 126 objects/s recovering 2024-04-15T18:45:20.224 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:20.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:20.333 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:20.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:20.384 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:20.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:20.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:20.516 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:20.537 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:20.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:20.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:20.690 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:20.775 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:20.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:20.883 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:20.914 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:20.941 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:20.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: pgmap v2036: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 580 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 1339/27831 objects degraded (4.811%); 2.1 MiB/s, 33 objects/s recovering 2024-04-15T18:45:20.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:20 smithi049 ceph-mon[40087]: osdmap e782: 12 total, 11 up, 11 in 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: pgmap v2036: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 580 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 1339/27831 objects degraded (4.811%); 2.1 MiB/s, 33 objects/s recovering 2024-04-15T18:45:21.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:20 smithi082 ceph-mon[44206]: osdmap e782: 12 total, 11 up, 11 in 2024-04-15T18:45:21.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:21.094 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: pgmap v2036: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 580 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 762 op/s; 1339/27831 objects degraded (4.811%); 2.1 MiB/s, 33 objects/s recovering 2024-04-15T18:45:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:20 smithi159 ceph-mon[45336]: osdmap e782: 12 total, 11 up, 11 in 2024-04-15T18:45:21.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:45:21.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:45:21.963 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:45:21.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:45:22.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:45:22.062 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:45:22.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:45:22.154 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:45:22.199 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:45:22.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:45:22.301 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:45:22.310 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:45:22.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:45:22.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:45:22.311 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:45:22.311 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:45:22.772 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:45:22.773 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1339/25788 objects degraded (5.192%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2470,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":783,"num_osds":12,"num_up_osds":11,"osd_up_since":1713206535,"num_in_osds":11,"osd_in_since":1713206557,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":8596,"data_bytes":563741240,"bytes_used":7918673920,"bytes_avail":1048026718208,"bytes_total":1055945392128,"degraded_objects":1339,"degraded_total":25788,"degraded_ratio":0.051923375213277494,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":1935412,"recovering_keys_per_sec":0,"num_objects_recovered":236,"num_bytes_recovered":15466496,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":561},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.49:8443/"}},"servicemap":{"epoch":55,"modified":"2024-04-15T18:43:37.627946+0000","services":{}},"progress_events":{}} 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: pgmap v2038: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 538 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 1339/25788 objects degraded (5.192%); 1.8 MiB/s, 29 objects/s recovering 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3173959854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1339/25788 objects degraded (5.192%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:23.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:22 smithi082 ceph-mon[44206]: osdmap e783: 12 total, 11 up, 11 in 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: pgmap v2038: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 538 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 1339/25788 objects degraded (5.192%); 1.8 MiB/s, 29 objects/s recovering 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3173959854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1339/25788 objects degraded (5.192%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:23.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:22 smithi159 ceph-mon[45336]: osdmap e783: 12 total, 11 up, 11 in 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: pgmap v2038: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 538 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 1339/25788 objects degraded (5.192%); 1.8 MiB/s, 29 objects/s recovering 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3173959854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1339/25788 objects degraded (5.192%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:22 smithi049 ceph-mon[40087]: osdmap e783: 12 total, 11 up, 11 in 2024-04-15T18:45:23.751 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3076161820' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:45:24.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3076161820' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:45:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3076161820' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-15T18:45:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:24.213 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:24.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:24.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:24.386 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:45:24.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:45:24.909 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-15T18:45:24.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: pgmap v2040: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 364 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1099/17475 objects degraded (6.289%); 1023 KiB/s, 39 objects/s recovering 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:24 smithi082 ceph-mon[44206]: osdmap e784: 12 total, 11 up, 11 in 2024-04-15T18:45:25.057 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: pgmap v2040: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 364 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1099/17475 objects degraded (6.289%); 1023 KiB/s, 39 objects/s recovering 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:25.061 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:24 smithi049 ceph-mon[40087]: osdmap e784: 12 total, 11 up, 11 in 2024-04-15T18:45:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:25.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: pgmap v2040: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 364 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 850 op/s; 1099/17475 objects degraded (6.289%); 1023 KiB/s, 39 objects/s recovering 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3131577573' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:24 smithi159 ceph-mon[45336]: osdmap e784: 12 total, 11 up, 11 in 2024-04-15T18:45:25.176 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:25.204 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:25.229 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:25.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:25.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:25.365 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:25.387 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:25.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:25.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:25.549 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:25.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:25.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:45:25.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:25.742 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:25.759 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:25.762 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:25.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:25.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:25.928 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.050 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:26.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:26.191 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.217 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:26.323 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:26.383 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:45:26.417 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:26.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:26.552 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.583 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:26.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:26.715 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.736 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:26.841 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:26.871 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:26.894 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:26.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:27.006 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:27.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:26 smithi082 ceph-mon[44206]: pgmap v2042: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 337 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 1099/16140 objects degraded (6.809%); 1024 KiB/s, 39 objects/s recovering 2024-04-15T18:45:27.043 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:26 smithi159 ceph-mon[45336]: pgmap v2042: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 337 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 1099/16140 objects degraded (6.809%); 1024 KiB/s, 39 objects/s recovering 2024-04-15T18:45:27.051 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:27.066 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:27.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:27.172 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:27.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:27.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:26 smithi049 ceph-mon[40087]: pgmap v2042: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 337 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 1099/16140 objects degraded (6.809%); 1024 KiB/s, 39 objects/s recovering 2024-04-15T18:45:27.232 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:27.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:27.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:27.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:27.365 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:27.394 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:27.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:27.499 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:27.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:45:27.530 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:27.561 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:27.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:27.666 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:27.695 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:27.718 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:27.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:27.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:27.841 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:27 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1099/16140 objects degraded (6.809%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:27.841 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:27 smithi049 ceph-mon[40087]: osdmap e785: 12 total, 11 up, 11 in 2024-04-15T18:45:27.861 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:28.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:27 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1099/16140 objects degraded (6.809%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:28.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:27 smithi082 ceph-mon[44206]: osdmap e785: 12 total, 11 up, 11 in 2024-04-15T18:45:28.026 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:28.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:28.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:27 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1099/16140 objects degraded (6.809%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:27 smithi159 ceph-mon[45336]: osdmap e785: 12 total, 11 up, 11 in 2024-04-15T18:45:28.180 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:28.662 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:45:28.666 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:45:28.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:28 smithi049 ceph-mon[40087]: pgmap v2043: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 242 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 807/11613 objects degraded (6.949%); 772 KiB/s, 66 objects/s recovering 2024-04-15T18:45:28.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:28 smithi049 ceph-mon[40087]: from='client.17286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:29.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:28 smithi082 ceph-mon[44206]: pgmap v2043: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 242 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 807/11613 objects degraded (6.949%); 772 KiB/s, 66 objects/s recovering 2024-04-15T18:45:29.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:28 smithi082 ceph-mon[44206]: from='client.17286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:28 smithi159 ceph-mon[45336]: pgmap v2043: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 242 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 807/11613 objects degraded (6.949%); 772 KiB/s, 66 objects/s recovering 2024-04-15T18:45:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:28 smithi159 ceph-mon[45336]: from='client.17286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:31.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:30 smithi082 ceph-mon[44206]: pgmap v2045: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 149 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 807/7116 objects degraded (11.341%); 0 B/s, 41 objects/s recovering 2024-04-15T18:45:31.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:30 smithi159 ceph-mon[45336]: pgmap v2045: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 149 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 807/7116 objects degraded (11.341%); 0 B/s, 41 objects/s recovering 2024-04-15T18:45:31.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:30 smithi049 ceph-mon[40087]: pgmap v2045: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 149 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 807/7116 objects degraded (11.341%); 0 B/s, 41 objects/s recovering 2024-04-15T18:45:31.233 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2044,"stamp":"2024-04-15T18:45:27.686851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":254083640,"num_objects":3871,"num_object_clones":0,"num_object_copies":11613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":200918,"num_write_kb":6550267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29565,"num_bytes_recovered":1875742976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6826216,"kb_used_data":821368,"kb_used_omap":285,"kb_used_meta":6004258,"kb_avail":1024370456,"statfs":{"total":1055945392128,"available":1048955346944,"internally_reserved":0,"allocated":841080832,"data_stored":836883872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":6148360405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-309657600,"num_objects":-4725,"num_object_clones":0,"num_object_copies":-14175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":532,"num_bytes_recovered":6291456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005171"},"pg_stats":[{"pgid":"8.f","version":"784'12937","reported_seq":25425,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:27.587695+0000","last_change":"2024-04-15T18:45:12.726636+0000","last_active":"2024-04-15T18:45:27.587695+0000","last_peered":"2024-04-15T18:45:27.587695+0000","last_clean":"2024-04-15T18:45:27.587695+0000","last_became_active":"2024-04-15T18:45:12.688571+0000","last_became_peered":"2024-04-15T18:45:12.688571+0000","last_unstale":"2024-04-15T18:45:27.587695+0000","last_undegraded":"2024-04-15T18:45:27.587695+0000","last_fullsized":"2024-04-15T18:45:27.587695+0000","mapping_epoch":778,"log_start":"726'2900","ondisk_log_start":"726'2900","created":709,"last_epoch_clean":779,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12937,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6158,"num_bytes_recovered":371064832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"784'12778","reported_seq":13003,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:27.528959+0000","last_change":"2024-04-15T18:42:32.046550+0000","last_active":"2024-04-15T18:45:27.528959+0000","last_peered":"2024-04-15T18:45:27.528959+0000","last_clean":"2024-04-15T18:45:27.528959+0000","last_became_active":"2024-04-15T18:41:02.884438+0000","last_became_peered":"2024-04-15T18:41:02.884438+0000","last_unstale":"2024-04-15T18:45:27.528959+0000","last_undegraded":"2024-04-15T18:45:27.528959+0000","last_fullsized":"2024-04-15T18:45:27.528959+0000","mapping_epoch":709,"log_start":"726'2700","ondisk_log_start":"726'2700","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5775","last_scrub_stamp":"2024-04-15T18:42:32.046350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:32.046350+0000","objects_scrubbed":5746,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8912896,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"784'12757","reported_seq":25213,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:26.440150+0000","last_change":"2024-04-15T18:45:08.460925+0000","last_active":"2024-04-15T18:45:26.440150+0000","last_peered":"2024-04-15T18:45:26.440150+0000","last_clean":"2024-04-15T18:45:26.440150+0000","last_became_active":"2024-04-15T18:44:29.945219+0000","last_became_peered":"2024-04-15T18:44:29.945219+0000","last_unstale":"2024-04-15T18:45:26.440150+0000","last_undegraded":"2024-04-15T18:45:26.440150+0000","last_fullsized":"2024-04-15T18:45:26.440150+0000","mapping_epoch":766,"log_start":"726'2700","ondisk_log_start":"726'2700","created":709,"last_epoch_clean":767,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'11957","last_scrub_stamp":"2024-04-15T18:45:08.460599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:08.460599+0000","objects_scrubbed":997,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:23.703729+0000","scrub_duration":437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320792,"num_objects":189,"num_object_clones":0,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12757,"num_write_kb":414209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6131,"num_bytes_recovered":401801216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"784'12839","reported_seq":13129,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:27.574517+0000","last_change":"2024-04-15T18:45:11.337839+0000","last_active":"2024-04-15T18:45:27.574517+0000","last_peered":"2024-04-15T18:45:27.574517+0000","last_clean":"2024-04-15T18:45:27.574517+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:45:27.574517+0000","last_undegraded":"2024-04-15T18:45:27.574517+0000","last_fullsized":"2024-04-15T18:45:27.574517+0000","mapping_epoch":709,"log_start":"726'2800","ondisk_log_start":"726'2800","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'12103","last_scrub_stamp":"2024-04-15T18:45:11.337582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:11.337582+0000","objects_scrubbed":918,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:41.298318+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12839,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.b","version":"784'12559","reported_seq":12850,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:25.363798+0000","last_change":"2024-04-15T18:45:09.083515+0000","last_active":"2024-04-15T18:45:25.363798+0000","last_peered":"2024-04-15T18:45:25.363798+0000","last_clean":"2024-04-15T18:45:25.363798+0000","last_became_active":"2024-04-15T18:41:02.837287+0000","last_became_peered":"2024-04-15T18:41:02.837287+0000","last_unstale":"2024-04-15T18:45:25.363798+0000","last_undegraded":"2024-04-15T18:45:25.363798+0000","last_fullsized":"2024-04-15T18:45:25.363798+0000","mapping_epoch":709,"log_start":"726'2500","ondisk_log_start":"726'2500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'11829","last_scrub_stamp":"2024-04-15T18:45:09.083205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:09.083205+0000","objects_scrubbed":989,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:16.170422+0000","scrub_duration":257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16842752,"num_objects":257,"num_object_clones":0,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12559,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"784'12506","reported_seq":12796,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:25.406204+0000","last_change":"2024-04-15T18:45:17.036214+0000","last_active":"2024-04-15T18:45:25.406204+0000","last_peered":"2024-04-15T18:45:25.406204+0000","last_clean":"2024-04-15T18:45:25.406204+0000","last_became_active":"2024-04-15T18:41:02.832268+0000","last_became_peered":"2024-04-15T18:41:02.832268+0000","last_unstale":"2024-04-15T18:45:25.406204+0000","last_undegraded":"2024-04-15T18:45:25.406204+0000","last_fullsized":"2024-04-15T18:45:25.406204+0000","mapping_epoch":709,"log_start":"726'2500","ondisk_log_start":"726'2500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'12120","last_scrub_stamp":"2024-04-15T18:45:17.035846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:17.035846+0000","objects_scrubbed":616,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:43.891443+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12506,"num_write_kb":407424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"783'12427","reported_seq":12669,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:22.983385+0000","last_change":"2024-04-15T18:45:17.135954+0000","last_active":"2024-04-15T18:45:22.983385+0000","last_peered":"2024-04-15T18:45:22.983385+0000","last_clean":"2024-04-15T18:45:22.983385+0000","last_became_active":"2024-04-15T18:41:02.835758+0000","last_became_peered":"2024-04-15T18:41:02.835758+0000","last_unstale":"2024-04-15T18:45:22.983385+0000","last_undegraded":"2024-04-15T18:45:22.983385+0000","last_fullsized":"2024-04-15T18:45:22.983385+0000","mapping_epoch":709,"log_start":"726'2400","ondisk_log_start":"726'2400","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'12177","last_scrub_stamp":"2024-04-15T18:45:17.135822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:17.135822+0000","objects_scrubbed":611,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:24.015781+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23396352,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12427,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"784'12597","reported_seq":23231,"reported_epoch":784,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:45:27.287260+0000","last_change":"2024-04-15T18:42:39.996076+0000","last_active":"2024-04-15T18:45:27.287260+0000","last_peered":"2024-04-15T18:45:27.287260+0000","last_clean":"2024-04-15T18:42:15.060455+0000","last_became_active":"2024-04-15T18:42:39.970014+0000","last_became_peered":"2024-04-15T18:42:39.970014+0000","last_unstale":"2024-04-15T18:45:27.287260+0000","last_undegraded":"2024-04-15T18:42:39.784034+0000","last_fullsized":"2024-04-15T18:42:39.747843+0000","mapping_epoch":743,"log_start":"726'2500","ondisk_log_start":"726'2500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12597,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5233,"num_bytes_recovered":312803328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10"],"object_location_counts":[{"shards":"1,10","objects":-678}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"783'12371","reported_seq":24517,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:23.588996+0000","last_change":"2024-04-15T18:44:57.505649+0000","last_active":"2024-04-15T18:45:23.588996+0000","last_peered":"2024-04-15T18:45:23.588996+0000","last_clean":"2024-04-15T18:45:23.588996+0000","last_became_active":"2024-04-15T18:44:57.475200+0000","last_became_peered":"2024-04-15T18:44:57.475200+0000","last_unstale":"2024-04-15T18:45:23.588996+0000","last_undegraded":"2024-04-15T18:45:23.588996+0000","last_fullsized":"2024-04-15T18:45:23.588996+0000","mapping_epoch":771,"log_start":"726'2300","ondisk_log_start":"726'2300","created":709,"last_epoch_clean":772,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12371,"num_write_kb":406912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5988,"num_bytes_recovered":392429568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"783'12333","reported_seq":12556,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:23.592739+0000","last_change":"2024-04-15T18:42:34.198191+0000","last_active":"2024-04-15T18:45:23.592739+0000","last_peered":"2024-04-15T18:45:23.592739+0000","last_clean":"2024-04-15T18:45:23.592739+0000","last_became_active":"2024-04-15T18:41:02.834244+0000","last_became_peered":"2024-04-15T18:41:02.834244+0000","last_unstale":"2024-04-15T18:45:23.592739+0000","last_undegraded":"2024-04-15T18:45:23.592739+0000","last_fullsized":"2024-04-15T18:45:23.592739+0000","mapping_epoch":709,"log_start":"726'2300","ondisk_log_start":"726'2300","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5780","last_scrub_stamp":"2024-04-15T18:42:34.197966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:34.197966+0000","objects_scrubbed":5725,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1489,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12333,"num_write_kb":404160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"783'12155","reported_seq":12395,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:22.958655+0000","last_change":"2024-04-15T18:45:08.222060+0000","last_active":"2024-04-15T18:45:22.958655+0000","last_peered":"2024-04-15T18:45:22.958655+0000","last_clean":"2024-04-15T18:45:22.958655+0000","last_became_active":"2024-04-15T18:41:02.835687+0000","last_became_peered":"2024-04-15T18:41:02.835687+0000","last_unstale":"2024-04-15T18:45:22.958655+0000","last_undegraded":"2024-04-15T18:45:22.958655+0000","last_fullsized":"2024-04-15T18:45:22.958655+0000","mapping_epoch":709,"log_start":"726'2100","ondisk_log_start":"726'2100","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'11508","last_scrub_stamp":"2024-04-15T18:45:08.221676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:08.221676+0000","objects_scrubbed":997,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:14.602985+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12155,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"784'12667","reported_seq":24918,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:27.322504+0000","last_change":"2024-04-15T18:45:02.721543+0000","last_active":"2024-04-15T18:45:27.322504+0000","last_peered":"2024-04-15T18:45:27.322504+0000","last_clean":"2024-04-15T18:45:27.322504+0000","last_became_active":"2024-04-15T18:45:02.674121+0000","last_became_peered":"2024-04-15T18:45:02.674121+0000","last_unstale":"2024-04-15T18:45:27.322504+0000","last_undegraded":"2024-04-15T18:45:27.322504+0000","last_fullsized":"2024-04-15T18:45:27.322504+0000","mapping_epoch":775,"log_start":"726'2600","ondisk_log_start":"726'2600","created":709,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:41:01.807048+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12667,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6039,"num_bytes_recovered":393412608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"783'12520","reported_seq":12747,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:23.609592+0000","last_change":"2024-04-15T18:42:36.224300+0000","last_active":"2024-04-15T18:45:23.609592+0000","last_peered":"2024-04-15T18:45:23.609592+0000","last_clean":"2024-04-15T18:45:23.609592+0000","last_became_active":"2024-04-15T18:41:02.884465+0000","last_became_peered":"2024-04-15T18:41:02.884465+0000","last_unstale":"2024-04-15T18:45:23.609592+0000","last_undegraded":"2024-04-15T18:45:23.609592+0000","last_fullsized":"2024-04-15T18:45:23.609592+0000","mapping_epoch":709,"log_start":"726'2500","ondisk_log_start":"726'2500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5968","last_scrub_stamp":"2024-04-15T18:42:36.224176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:36.224176+0000","objects_scrubbed":5913,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1575,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":410304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"784'12502","reported_seq":12725,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:27.245271+0000","last_change":"2024-04-15T18:42:30.391160+0000","last_active":"2024-04-15T18:45:27.245271+0000","last_peered":"2024-04-15T18:45:27.245271+0000","last_clean":"2024-04-15T18:45:27.245271+0000","last_became_active":"2024-04-15T18:41:02.836069+0000","last_became_peered":"2024-04-15T18:41:02.836069+0000","last_unstale":"2024-04-15T18:45:27.245271+0000","last_undegraded":"2024-04-15T18:45:27.245271+0000","last_fullsized":"2024-04-15T18:45:27.245271+0000","mapping_epoch":709,"log_start":"726'2500","ondisk_log_start":"726'2500","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'5536","last_scrub_stamp":"2024-04-15T18:42:30.390958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:30.390958+0000","objects_scrubbed":5491,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12502,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"783'12374","reported_seq":12618,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:22.994025+0000","last_change":"2024-04-15T18:45:09.488024+0000","last_active":"2024-04-15T18:45:22.994025+0000","last_peered":"2024-04-15T18:45:22.994025+0000","last_clean":"2024-04-15T18:45:22.994025+0000","last_became_active":"2024-04-15T18:41:02.835657+0000","last_became_peered":"2024-04-15T18:41:02.835657+0000","last_unstale":"2024-04-15T18:45:22.994025+0000","last_undegraded":"2024-04-15T18:45:22.994025+0000","last_fullsized":"2024-04-15T18:45:22.994025+0000","mapping_epoch":709,"log_start":"726'2300","ondisk_log_start":"726'2300","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"777'11748","last_scrub_stamp":"2024-04-15T18:45:09.487904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:45:09.487904+0000","objects_scrubbed":949,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:09.763964+0000","scrub_duration":433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12374,"num_write_kb":405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"783'12415","reported_seq":12637,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-04-15T18:45:23.609814+0000","last_change":"2024-04-15T18:42:35.460106+0000","last_active":"2024-04-15T18:45:23.609814+0000","last_peered":"2024-04-15T18:45:23.609814+0000","last_clean":"2024-04-15T18:45:23.609814+0000","last_became_active":"2024-04-15T18:41:02.884378+0000","last_became_peered":"2024-04-15T18:41:02.884378+0000","last_unstale":"2024-04-15T18:45:23.609814+0000","last_undegraded":"2024-04-15T18:45:23.609814+0000","last_fullsized":"2024-04-15T18:45:23.609814+0000","mapping_epoch":709,"log_start":"726'2400","ondisk_log_start":"726'2400","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'5879","last_scrub_stamp":"2024-04-15T18:42:35.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:41:01.807048+0000","last_clean_scrub_stamp":"2024-04-15T18:42:35.459848+0000","objects_scrubbed":5833,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12415,"num_write_kb":407808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1940,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-04-15T18:45:23.748886+0000","last_change":"2024-04-15T18:45:19.433556+0000","last_active":"2024-04-15T18:45:23.748886+0000","last_peered":"2024-04-15T18:45:23.748886+0000","last_clean":"2024-04-15T18:45:23.748886+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:45:23.748886+0000","last_undegraded":"2024-04-15T18:45:23.748886+0000","last_fullsized":"2024-04-15T18:45:23.748886+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:45:19.433459+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:45:19.433459+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:43.372870+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":253493272,"num_objects":3869,"num_object_clones":0,"num_object_copies":11607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200737,"num_write_kb":6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29549,"num_bytes_recovered":1871511552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2308583424,"data_stored":2308571208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187897,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":701140,"kb_used_data":132680,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93044012,"statfs":{"total":95995035648,"available":95277068288,"internally_reserved":0,"allocated":135864320,"data_stored":135486879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":582089152},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514944,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759524,"kb_used_data":70300,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92985628,"statfs":{"total":95995035648,"available":95217283072,"internally_reserved":0,"allocated":71987200,"data_stored":71601738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":705755584},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874689,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471584,"kb_used_data":53076,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93273568,"statfs":{"total":95995035648,"available":95512133632,"internally_reserved":0,"allocated":54349824,"data_stored":53967914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541164,"kb_used_data":28256,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":93203988,"statfs":{"total":95995035648,"available":95440883712,"internally_reserved":0,"allocated":28934144,"data_stored":28544610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":525203914},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217132,"kb_used_data":18076,"kb_used_omap":111,"kb_used_meta":198928,"kb_avail":93528020,"statfs":{"total":95995035648,"available":95772692480,"internally_reserved":0,"allocated":18509824,"data_stored":18124370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":203702939},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575337,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":715804,"kb_used_data":62612,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93029348,"statfs":{"total":95995035648,"available":95262052352,"internally_reserved":0,"allocated":64114688,"data_stored":63732778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738858,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707292,"kb_used_data":48468,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93037860,"statfs":{"total":95995035648,"available":95270768640,"internally_reserved":0,"allocated":49631232,"data_stored":49249322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194109,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":584608,"kb_used_data":75032,"kb_used_omap":114,"kb_used_meta":509453,"kb_avail":93160544,"statfs":{"total":95995035648,"available":95396397056,"internally_reserved":0,"allocated":76832768,"data_stored":76446786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":521680113},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609238,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762436,"kb_used_data":73980,"kb_used_omap":44,"kb_used_meta":688403,"kb_avail":92982716,"statfs":{"total":95995035648,"available":95214301184,"internally_reserved":0,"allocated":75755520,"data_stored":75382598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":704925226},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986645,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762372,"kb_used_data":123132,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":92982780,"statfs":{"total":95995035648,"available":95214366720,"internally_reserved":0,"allocated":126087168,"data_stored":125714246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703539,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603160,"kb_used_data":135756,"kb_used_omap":4,"kb_used_meta":467387,"kb_avail":93141992,"statfs":{"total":95995035648,"available":95377399808,"internally_reserved":0,"allocated":139014144,"data_stored":138632631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":478604993},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43712512,"data_stored":43712512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60096512,"data_stored":60096512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63246336,"data_stored":63242264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":67764224,"data_stored":67764224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":116129792,"data_stored":116129792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":136253440,"data_stored":136249368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15532032,"data_stored":15532032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":25956352,"data_stored":25952280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":48627712,"data_stored":48627712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":56164352,"data_stored":56164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":124977152,"data_stored":124977152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:45:31.253 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:31.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:31.265 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:31.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:31.387 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:31.537 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:31.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:31.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:31.670 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:31.690 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:31.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:31.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:31.825 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:31.846 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:31.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:31.850 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 102302 objects 2024-04-15T18:45:31.850 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :162.461 2024-04-15T18:45:31.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:31.979 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:31.998 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:32.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:32.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:32.129 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:32.149 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:32.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:32.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:32.285 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:32.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:32.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:32.514 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:32.546 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:32.567 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:32.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:32.572 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:45:32.572 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:45:32.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:32.700 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:32.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:32 smithi159 ceph-mon[45336]: pgmap v2046: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 110 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 508 op/s; 807/5271 objects degraded (15.310%); 0 B/s, 36 objects/s recovering 2024-04-15T18:45:32.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 807/5271 objects degraded (15.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:32 smithi049 ceph-mon[40087]: pgmap v2046: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 110 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 508 op/s; 807/5271 objects degraded (15.310%); 0 B/s, 36 objects/s recovering 2024-04-15T18:45:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 807/5271 objects degraded (15.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:32 smithi082 ceph-mon[44206]: pgmap v2046: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 110 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 508 op/s; 807/5271 objects degraded (15.310%); 0 B/s, 36 objects/s recovering 2024-04-15T18:45:33.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 807/5271 objects degraded (15.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:33.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:45:33.249 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:33.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:33.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:33.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:33.381 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:33.434 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:33.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:33.512 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:33.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:33.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:33.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:34.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:34.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:34.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:45:34.521 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-04-15T18:45:34.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-04-15T18:45:34.986 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:34 smithi082 ceph-mon[44206]: pgmap v2047: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.1 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 585/177 objects degraded (330.508%); 0 B/s, 51 objects/s recovering 2024-04-15T18:45:34.989 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:34 smithi082 ceph-mon[44206]: osdmap e786: 12 total, 11 up, 11 in 2024-04-15T18:45:35.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:34 smithi159 ceph-mon[45336]: pgmap v2047: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.1 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 585/177 objects degraded (330.508%); 0 B/s, 51 objects/s recovering 2024-04-15T18:45:35.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:34 smithi159 ceph-mon[45336]: osdmap e786: 12 total, 11 up, 11 in 2024-04-15T18:45:35.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:34 smithi049 ceph-mon[40087]: pgmap v2047: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 4.1 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 585/177 objects degraded (330.508%); 0 B/s, 51 objects/s recovering 2024-04-15T18:45:35.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:34 smithi049 ceph-mon[40087]: osdmap e786: 12 total, 11 up, 11 in 2024-04-15T18:45:35.628 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:35.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:35.733 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:35.765 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:35.786 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:35.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:35.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:35.918 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:35.993 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:35.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:36.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:36.128 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:36.150 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:36.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:36.188 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:36.240 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:45:36.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:36.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:45:36.282 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:36.291 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:45:36.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:45:36.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:45:36.367 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:45:36.392 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:45:36.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:45:36.440 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:45:36.440 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:45:36.441 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:45:36.441 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:45:36.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:45:36.473 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:45:36.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:45:36.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:45:36.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:45:36.501 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:45:36.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: pgmap v2049: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 585/9 objects degraded (6500.000%); 0 B/s, 27 objects/s recovering 2024-04-15T18:45:36.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/162242918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:36.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:36.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: pgmap v2049: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 585/9 objects degraded (6500.000%); 0 B/s, 27 objects/s recovering 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/162242918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:45:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: pgmap v2049: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 483 op/s; 585/9 objects degraded (6500.000%); 0 B/s, 27 objects/s recovering 2024-04-15T18:45:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/162242918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.304 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:37.432 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:37.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:37.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:37.564 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:37.584 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:37.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:37.689 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:37.718 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:37.739 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:37.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:37.843 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:37.871 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2933808870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 585/9 objects degraded (6500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:45:37.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:37 smithi049 ceph-mon[40087]: osdmap e787: 12 total, 11 up, 11 in 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2933808870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 585/9 objects degraded (6500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:38.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:45:38.027 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:37 smithi082 ceph-mon[44206]: osdmap e787: 12 total, 11 up, 11 in 2024-04-15T18:45:38.069 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:38.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:38.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2933808870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:38.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:45:38.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 585/9 objects degraded (6500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:45:38.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:38.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:45:38.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:37 smithi159 ceph-mon[45336]: osdmap e787: 12 total, 11 up, 11 in 2024-04-15T18:45:38.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:38.205 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:38.226 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:38.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:38.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:38.390 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:45:38.580 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:38.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:38 smithi049 ceph-mon[40087]: pgmap v2051: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:39.001 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:45:39.002 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:45:39.002 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:45:39.002 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:45:39.002 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:45:39.011 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:39.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:39.024 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:38 smithi082 ceph-mon[44206]: pgmap v2051: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:39.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:39.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:38 smithi159 ceph-mon[45336]: pgmap v2051: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:39.144 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:39.246 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:39.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:39.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:39.379 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:39.480 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:39.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:39.585 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:45:39.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:45:39.613 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:39.635 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:39.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:39.664 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:45:39.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:39.769 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:39.843 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:39.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:39.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:39.966 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:39.975 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:39.997 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:40.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:40.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:40.135 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:40.156 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:40.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:40.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:40.290 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:40.311 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:40.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:40.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:40.445 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:40.794 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:40.799 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_6' removed 2024-04-15T18:45:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:40.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:40.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:40.935 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:40.955 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:40.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:40.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:45:41.059 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:41.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:40 smithi159 ceph-mon[45336]: pgmap v2052: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 219 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:41.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:41.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:41.087 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:40 smithi159 ceph-mon[45336]: osdmap e788: 12 total, 11 up, 11 in 2024-04-15T18:45:41.088 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:45:41.089 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:41.110 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:41.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:41.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:40 smithi082 ceph-mon[44206]: pgmap v2052: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 219 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:41.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:41.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:41.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:40 smithi082 ceph-mon[44206]: osdmap e788: 12 total, 11 up, 11 in 2024-04-15T18:45:41.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:45:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:40 smithi049 ceph-mon[40087]: pgmap v2052: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 219 op/s; 88/6 objects degraded (1466.667%); 0 B/s, 89 objects/s recovering 2024-04-15T18:45:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:40 smithi049 ceph-mon[40087]: osdmap e788: 12 total, 11 up, 11 in 2024-04-15T18:45:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:45:41.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:41.242 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:41.341 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:41.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:41.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:41.473 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:41.604 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:41.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:45:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:41.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:41.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:41.736 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:41.831 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:164354 6.d 6:bbcdb5bb:::benchmark_data_smithi159_139342_object74475:head [delete] snapc 0=[] ondisk+write+known_if_redirected e606)", 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 584.123520955, 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29154140200000001, 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.832 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 164354 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718176+0000", 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.833 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718177+0000", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5000000000000002e-07 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718181+0000", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6820000000000001e-06 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718192+0000", 2024-04-15T18:45:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0951000000000001e-05 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718197+0000", 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.587e-06 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718252+0000", 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5090999999999998e-05 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.835 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718368+0000", 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115615 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.718486+0000", 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117946 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.722722+0000", 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0042362509999999999 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.836 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723060+0000", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033780300000000002 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009629+0000", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28656950199999998 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009670+0000", 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0249e-05 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.837 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009718+0000", 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8075000000000003e-05 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26224.0:164354 6.d e606/541 ondisk, result = 0)", 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 584.11863607500004, 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286662904, 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.838 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723061+0000", 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.839 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723062+0000", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.82e-07 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:57.723065+0000", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4290000000000001e-06 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009573+0000", 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.286507593 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.840 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009576+0000", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.501e-06 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009619+0000", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2441999999999998e-05 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009626+0000", 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.058e-06 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.841 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:35:58.009724+0000", 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8399000000000004e-05 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26224.0:177238 6.4 e609/541 6:2ec8682d:::benchmark_data_smithi159_139342_object87359:head v 609'10914, mlcod=609'10912)", 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 563.81849796500001, 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.2949041, 2024-04-15T18:45:41.842 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023199+0000", 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.843 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023200+0000", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3399999999999998e-07 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.023206+0000", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9259999999999997e-06 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313521+0000", 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29031515699999999 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.844 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313528+0000", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8569999999999997e-06 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313582+0000", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3565e-05 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.313607+0000", 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4990999999999998e-05 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.845 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.318066+0000", 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044594719999999999 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:18.318103+0000", 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7397999999999997e-05 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.846 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26224.0:179749 6.d 6:bfc4ccdc:::benchmark_data_smithi159_139342_object89870:head [delete] snapc 0=[] ondisk+write+known_if_redirected e611)", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 560.47324073200002, 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.75209580899999995, 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26224", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/1671137392", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179749 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:45:41.847 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368457+0000", 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368458+0000", 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.483e-06 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.848 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368463+0000", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0189999999999997e-06 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368476+0000", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2588e-05 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368483+0000", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4599999999999997e-06 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.849 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368535+0000", 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1928999999999997e-05 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368651+0000", 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000116334 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.368760+0000", 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108328 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.850 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.373250+0000", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044905420000000001 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:21.383059+0000", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0098083690000000008 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120474+0000", 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.73741578100000005 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.851 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120520+0000", 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5677999999999999e-05 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:36:22.120552+0000", 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2298000000000001e-05 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.852 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31772 7.0 7:002aa88c:::benchmark_data_smithi159_149998_object31771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 510.82022694599999, 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32090146899999999, 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:45:41.853 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31772 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021470+0000", 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.854 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021471+0000", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021582+0000", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110575 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021593+0000", 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.075e-05 2024-04-15T18:45:41.855 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021599+0000", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3060000000000003e-06 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021642+0000", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2583999999999998e-05 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021735+0000", 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2876999999999999e-05 2024-04-15T18:45:41.856 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.021822+0000", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7751999999999996e-05 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.022676+0000", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085419300000000005 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.023361+0000", 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00068498799999999998 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.857 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342304+0000", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31894291299999999 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342340+0000", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5759000000000001e-05 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.342372+0000", 2024-04-15T18:45:41.858 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1615000000000001e-05 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26545.0:31792 7.b e631/621 7:d043e990:::benchmark_data_smithi159_149998_object31791:head v 631'1903, mlcod=631'1900)", 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 510.793554728, 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31352247700000002, 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.859 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048143+0000", 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.23e-07 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.860 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048237+0000", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3766999999999999e-05 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360616+0000", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31237881899999997 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360620+0000", 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.33e-06 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360666+0000", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6100000000000002e-05 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.360686+0000", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0154000000000001e-05 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.361637+0000", 2024-04-15T18:45:41.862 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00095048699999999997 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.361665+0000", 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8096999999999999e-05 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26545.0:45889 7.b e636/621 7:db21d6d0:::benchmark_data_smithi159_149998_object45888:head v 636'2779, mlcod=636'2775)", 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 498.67043272799998, 2024-04-15T18:45:41.863 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.200374936, 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.864 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171265+0000", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9299999999999998e-07 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.171359+0000", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3683000000000004e-05 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369880+0000", 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19852134900000001 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.865 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369887+0000", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9380000000000003e-06 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369944+0000", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7318999999999998e-05 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.369969+0000", 2024-04-15T18:45:41.866 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4663000000000001e-05 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.371598+0000", 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001628675 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:23.371639+0000", 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1715999999999998e-05 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.867 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:48075 7.0 7:0971b1ff:::benchmark_data_smithi159_149998_object48074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e637)", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 496.73993951199998, 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20932758800000001, 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 48075 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.868 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101758+0000", 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101763+0000", 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0880000000000002e-06 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.869 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101788+0000", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5174999999999999e-05 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101804+0000", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5716e-05 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101829+0000", 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5181000000000002e-05 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.870 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101888+0000", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9389999999999999e-05 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.101985+0000", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6218999999999995e-05 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.102075+0000", 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0828999999999994e-05 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.871 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103588+0000", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001512698 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103879+0000", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029104499999999998 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310986+0000", 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20710657299999999 2024-04-15T18:45:41.872 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311043+0000", 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7800999999999998e-05 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311085+0000", 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1872999999999997e-05 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.873 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:48075 7.0 e637/621 ondisk, result = 0)", 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 496.73785239900002, 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.207248614, 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.874 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103845+0000", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.84e-07 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.103847+0000", 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9350000000000001e-06 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.875 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310924+0000", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20707672799999999 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310931+0000", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8929999999999999e-06 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310973+0000", 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2160999999999998e-05 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.876 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.310981+0000", 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0949999999999996e-06 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:25.311093+0000", 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011241799999999999 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.877 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:146592 7.0 7:0ce3da3a:::benchmark_data_smithi159_149998_object50592:head [delete] snapc 0=[] ondisk+write+known_if_redirected e676)", 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:31.723118+0000", 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 370.11857959000002, 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37558565399999999, 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 146592 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.878 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723118+0000", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723118+0000", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723119+0000", 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5430000000000001e-06 2024-04-15T18:45:41.879 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723127+0000", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.515e-06 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723142+0000", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5551000000000001e-05 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723168+0000", 2024-04-15T18:45:41.880 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5409999999999999e-05 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723220+0000", 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2342000000000001e-05 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723376+0000", 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000156467 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.881 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.723526+0000", 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149474 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.727145+0000", 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036191399999999999 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.727197+0000", 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1935000000000003e-05 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.882 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.098629+0000", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.371431711 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.098671+0000", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2298000000000001e-05 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.098703+0000", 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2267999999999997e-05 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.883 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66872 8.4 8:2fd3b0dc:::benchmark_data_smithi159_162467_object66871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 214.364160011, 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.221963629, 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66872 2024-04-15T18:45:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477538+0000", 2024-04-15T18:45:41.885 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7899999999999997e-07 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477567+0000", 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8904999999999999e-05 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477579+0000", 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1773999999999999e-05 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.886 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477586+0000", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8129999999999998e-06 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477626+0000", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0457000000000002e-05 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477716+0000", 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0026999999999997e-05 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.887 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477805+0000", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8660999999999997e-05 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.479626+0000", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001821666 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.532510+0000", 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.052884050000000002 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.888 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699344+0000", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16683369100000001 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699462+0000", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117971 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699501+0000", 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8735000000000001e-05 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.889 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:66872 8.4 e732/709 ondisk, result = 0)", 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 214.36140817899999, 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.219219101, 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:45:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480290+0000", 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0699999999999997e-07 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.891 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480293+0000", 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3100000000000001e-06 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699149+0000", 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.218855771 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699165+0000", 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.632e-05 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.892 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699315+0000", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014995399999999999 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699334+0000", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8638999999999999e-05 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699508+0000", 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017450000000000001 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.893 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66881 8.0 8:006672f8:::benchmark_data_smithi159_162467_object66880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 214.353557378, 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22338060300000001, 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66881 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.894 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488141+0000", 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5939999999999999e-06 2024-04-15T18:45:41.895 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488214+0000", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2871000000000001e-05 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488226+0000", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1223000000000001e-05 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488249+0000", 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3269999999999999e-05 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.896 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488287+0000", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8186999999999999e-05 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488391+0000", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104149 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488496+0000", 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010463600000000001 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.897 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490307+0000", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001811602 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490905+0000", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00059743399999999999 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711410+0000", 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.220504902 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.898 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711475+0000", 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5753999999999994e-05 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711520+0000", 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4981000000000002e-05 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82881 8.0 8:07151622:::benchmark_data_smithi159_162467_object82880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:45:41.899 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 198.41256529099999, 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22421091100000001, 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82881 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.900 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429144+0000", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2394e-05 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429232+0000", 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7344999999999998e-05 2024-04-15T18:45:41.901 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429242+0000", 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0164e-05 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429250+0000", 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7249999999999997e-06 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.902 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431184+0000", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019340550000000001 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431318+0000", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013448299999999999 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431421+0000", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010239900000000001 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.903 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433412+0000", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019919009999999999 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436831+0000", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034181820000000001 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653287+0000", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216456327 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653318+0000", 2024-04-15T18:45:41.904 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1352000000000001e-05 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653343+0000", 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4584e-05 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82886 8.0 8:054a31d8:::benchmark_data_smithi159_162467_object82885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 198.408155806, 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.227292199, 2024-04-15T18:45:41.905 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82886 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.906 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433543+0000", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4470000000000001e-06 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433600+0000", 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.907 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433610+0000", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6560000000000004e-06 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433617+0000", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8720000000000002e-06 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433677+0000", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9969000000000001e-05 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.908 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433781+0000", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103631 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433900+0000", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011869199999999999 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436043+0000", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021429449999999998 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.909 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436987+0000", 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00094478900000000002 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660767+0000", 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22377945399999999 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660806+0000", 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9045999999999999e-05 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.910 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660834+0000", 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7704000000000001e-05 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82902 8.0 8:0b1c4e81:::benchmark_data_smithi159_162467_object82901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 198.397532074, 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22047940599999999, 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.911 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82902 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:45:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444166+0000", 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.147e-06 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444220+0000", 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3282999999999998e-05 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.913 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444229+0000", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7860000000000008e-06 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444235+0000", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.463e-06 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444280+0000", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5045e-05 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.914 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444395+0000", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011479 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444498+0000", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010325800000000001 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.446700+0000", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022018820000000001 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.915 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.447033+0000", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033350600000000002 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664573+0000", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21753967799999999 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664616+0000", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3126e-05 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.916 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664645+0000", 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8442e-05 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(8.d 756/743 [PushOp(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head, version: 726'2744, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head@726'2744, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 119.610592678, 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24113336099999999, 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:45:41.917 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231106+0000", 2024-04-15T18:45:41.918 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.508e-06 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231139+0000", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3120000000000001e-05 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469866+0000", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23872668799999999 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469882+0000", 2024-04-15T18:45:41.919 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5794000000000002e-05 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469963+0000", 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1763999999999998e-05 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469972+0000", 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8380000000000008e-06 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.920 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.472238+0000", 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022656489999999998 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:164234 8.0 8:05b44565:::benchmark_data_smithi159_162467_object61929:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 72.691104525, 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24243994899999999, 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.921 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 164234 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.922 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150595+0000", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.137e-06 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150605+0000", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9299999999999998e-06 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150640+0000", 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5648000000000002e-05 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.923 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150664+0000", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3632e-05 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150715+0000", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1121000000000001e-05 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150811+0000", 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6012000000000002e-05 2024-04-15T18:45:41.924 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150891+0000", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9812000000000001e-05 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.154655+0000", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037635139999999999 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.155576+0000", 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00092136400000000004 2024-04-15T18:45:41.925 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392953+0000", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23737709700000001 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392993+0000", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0268000000000001e-05 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.393033+0000", 2024-04-15T18:45:41.926 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9413999999999999e-05 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:169755 8.0 8:06929281:::benchmark_data_smithi159_162467_object67450:head [delete] snapc 0=[] ondisk+write+known_if_redirected e767)", 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.235348938999998, 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22069882099999999, 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:45:41.927 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 169755 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.928 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606350+0000", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.31e-06 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606356+0000", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0800000000000002e-06 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606369+0000", 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3742e-05 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.929 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606377+0000", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2880000000000004e-06 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606422+0000", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5330999999999997e-05 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606543+0000", 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120671 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.930 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606637+0000", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4759000000000005e-05 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611592+0000", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049540929999999997 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.618261+0000", 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0066691210000000001 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.931 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826959+0000", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20869838900000001 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826999+0000", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9787999999999999e-05 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.827047+0000", 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8248999999999998e-05 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.932 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:169755 8.0 e767/709 ondisk, result = 0)", 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:38.611889+0000", 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 63.229808120000001, 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21516637299999999, 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611889+0000", 2024-04-15T18:45:41.933 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611889+0000", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611890+0000", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0270000000000001e-06 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611897+0000", 2024-04-15T18:45:41.934 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.533e-06 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.822506+0000", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.210609406 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.822513+0000", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4659999999999996e-06 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826948+0000", 2024-04-15T18:45:41.935 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044356279999999996 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826955+0000", 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4459999999999998e-06 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.827055+0000", 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010086699999999999 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:45:41.936 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:45:41.937 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:45:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:41.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:45:41.937 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:41 smithi049 ceph-mon[40087]: osdmap e789: 12 total, 11 up, 11 in 2024-04-15T18:45:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:41.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:42.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:42.068 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:42.089 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:45:42.089 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:41 smithi082 ceph-mon[44206]: osdmap e789: 12 total, 11 up, 11 in 2024-04-15T18:45:42.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/816321299' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:45:42.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:41 smithi159 ceph-mon[45336]: osdmap e789: 12 total, 11 up, 11 in 2024-04-15T18:45:42.337 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-15T18:45:42.337 INFO:tasks.radosbench:Beginning radosbench... 2024-04-15T18:45:42.338 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-04-15T18:45:42.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-04-15T18:45:42.430 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:45:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:42.518 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:42.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:42.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:42.655 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:42.678 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:42.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:42.698 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:42.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:45:42.788 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:45:42.792 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:45:42.792 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:42.812 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:42.832 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:45:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:42.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:45:42.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:42.966 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:43.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:42 smithi049 ceph-mon[40087]: pgmap v2055: 1 pgs: 1 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:43.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:42 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/6 objects degraded (1466.667%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:45:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:42 smithi159 ceph-mon[45336]: pgmap v2055: 1 pgs: 1 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:42 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/6 objects degraded (1466.667%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:45:43.205 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:42 smithi082 ceph-mon[44206]: pgmap v2055: 1 pgs: 1 active+clean; 577 KiB data, 5.7 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:43.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:42 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/6 objects degraded (1466.667%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:45:44.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:43 smithi082 ceph-mon[44206]: from='client.17310 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:44.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:43 smithi159 ceph-mon[45336]: from='client.17310 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:44.175 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:43 smithi049 ceph-mon[40087]: from='client.17310 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:45:44.216 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:44.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:44.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:44.352 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:44.372 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:44.373 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:45:44.373 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:45:44.373 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:45:44.373 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:45:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:44.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:44.476 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:44.509 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:44.530 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:44.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:44.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:44.663 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:44.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:44 smithi049 ceph-mon[40087]: pgmap v2056: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:44.974 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:44 smithi082 ceph-mon[44206]: pgmap v2056: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:44.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:45:45.089 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2055,"stamp":"2024-04-15T18:45:41.688188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6021544,"kb_used_data":12712,"kb_used_omap":285,"kb_used_meta":6008418,"kb_avail":1025175128,"statfs":{"total":1055945392128,"available":1049779331072,"internally_reserved":0,"allocated":13017088,"data_stored":8666091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":6152620245},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-585,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-204605,"num_write_kb":-6547329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-29771,"num_bytes_recovered":-1871511552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001249"},"pg_stats":[{"pgid":"1.0","version":"509'139","reported_seq":1950,"reported_epoch":789,"state":"active+clean","last_fresh":"2024-04-15T18:45:41.489171+0000","last_change":"2024-04-15T18:45:19.433556+0000","last_active":"2024-04-15T18:45:41.489171+0000","last_peered":"2024-04-15T18:45:41.489171+0000","last_clean":"2024-04-15T18:45:41.489171+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:45:41.489171+0000","last_undegraded":"2024-04-15T18:45:41.489171+0000","last_fullsized":"2024-04-15T18:45:41.489171+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:45:19.433459+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:45:19.433459+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:43.372870+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187900,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577784,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93167368,"statfs":{"total":95995035648,"available":95403384832,"internally_reserved":0,"allocated":1028096,"data_stored":637198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514946,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694912,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050240,"statfs":{"total":95995035648,"available":95283445760,"internally_reserved":0,"allocated":1560576,"data_stored":1166126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":710015424},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874692,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407056,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93338096,"statfs":{"total":95995035648,"available":95578210304,"internally_reserved":0,"allocated":1052672,"data_stored":652786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169033,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":497816,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93247336,"statfs":{"total":95995035648,"available":95485272064,"internally_reserved":0,"allocated":1585152,"data_stored":1181714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":508164554},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363880,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200600,"kb_used_data":1548,"kb_used_omap":111,"kb_used_meta":198928,"kb_avail":93544552,"statfs":{"total":95995035648,"available":95789621248,"internally_reserved":0,"allocated":1585152,"data_stored":1181722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":203702939},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575339,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654136,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93091016,"statfs":{"total":95995035648,"available":95325200384,"internally_reserved":0,"allocated":1028096,"data_stored":637198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663888,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081264,"statfs":{"total":95995035648,"available":95315214336,"internally_reserved":0,"allocated":1052672,"data_stored":652786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194111,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523060,"kb_used_data":1004,"kb_used_omap":114,"kb_used_meta":521933,"kb_avail":93222092,"statfs":{"total":95995035648,"available":95459422208,"internally_reserved":0,"allocated":1028096,"data_stored":637198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":534459633},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609241,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693636,"kb_used_data":1016,"kb_used_omap":44,"kb_used_meta":692563,"kb_avail":93051516,"statfs":{"total":95995035648,"available":95284752384,"internally_reserved":0,"allocated":1040384,"data_stored":644967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":709185066},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986648,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631928,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93113224,"statfs":{"total":95995035648,"available":95347941376,"internally_reserved":0,"allocated":1028096,"data_stored":637198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703542,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476728,"kb_used_data":1004,"kb_used_omap":4,"kb_used_meta":475707,"kb_avail":93268424,"statfs":{"total":95995035648,"available":95506866176,"internally_reserved":0,"allocated":1028096,"data_stored":637198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":487124673},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:45:45.090 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T18:45:45.090 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-04-15T18:45:45.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:44 smithi159 ceph-mon[45336]: pgmap v2056: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:45.258 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:45:45.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:45:45.904 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:45.915 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:45.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:45:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:45.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:45.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:45:45.969 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:45:45.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:45:45.996 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:45:46.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:46.048 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:46.075 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/645957129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:46.075 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.076 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:46.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:46.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/645957129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:46.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.169 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/645957129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:46.169 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.187 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:46.213 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:46.412 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:46.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:46.516 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:46.543 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:46.666 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:46.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:46.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:46.799 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:46.819 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:46.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:46.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: pgmap v2057: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.951 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:46 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:46.961 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:45:46.962 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: pgmap v2057: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.988 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:46.988 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:46 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: pgmap v2057: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:46 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:47.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:45:47.802 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:45:47.812 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:45:48.065 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.066 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:47 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.066 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:48.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:47 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:48.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:47 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:48.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:45:48.806 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:48.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:48.818 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_7' created 2024-04-15T18:45:48.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:48.943 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:48.978 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:48 smithi082 ceph-mon[44206]: pgmap v2058: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:48.978 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:48 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:48.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:48.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:48 smithi082 ceph-mon[44206]: osdmap e790: 12 total, 11 up, 11 in 2024-04-15T18:45:48.979 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-15T18:45:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:48 smithi159 ceph-mon[45336]: pgmap v2058: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:48 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:48 smithi159 ceph-mon[45336]: osdmap e790: 12 total, 11 up, 11 in 2024-04-15T18:45:49.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-15T18:45:49.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:48 smithi049 ceph-mon[40087]: pgmap v2058: 1 pgs: 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:49.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:48 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:45:49.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1869820941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:45:49.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:48 smithi049 ceph-mon[40087]: osdmap e790: 12 total, 11 up, 11 in 2024-04-15T18:45:49.203 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-04-15T18:45:49.493 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:49.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:49.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-04-15T18:45:49.530 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:45:49.530 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:45:49.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:49.628 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:49.727 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:49.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:49.831 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:49.858 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:45:49.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:45:49.965 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:50.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:45:50.057 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:50.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-15T18:45:50.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:49 smithi049 ceph-mon[40087]: osdmap e791: 12 total, 11 up, 11 in 2024-04-15T18:45:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-15T18:45:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:49 smithi159 ceph-mon[45336]: osdmap e791: 12 total, 11 up, 11 in 2024-04-15T18:45:50.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:50.192 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:50.220 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4101746070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-04-15T18:45:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:49 smithi082 ceph-mon[44206]: osdmap e791: 12 total, 11 up, 11 in 2024-04-15T18:45:50.324 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:45:50.357 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:50.378 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:45:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:45:50.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:50.510 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:50.529 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:50.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:50.661 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:50.735 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:50.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:45:50.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:50.868 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:50.887 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:50.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:50.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:51.017 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.036 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:51.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:50 smithi159 ceph-mon[45336]: pgmap v2061: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:50 smithi159 ceph-mon[45336]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:45:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:50 smithi159 ceph-mon[45336]: osdmap e792: 12 total, 11 up, 11 in 2024-04-15T18:45:51.164 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:50 smithi082 ceph-mon[44206]: pgmap v2061: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:51.164 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:50 smithi082 ceph-mon[44206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:45:51.164 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:50 smithi082 ceph-mon[44206]: osdmap e792: 12 total, 11 up, 11 in 2024-04-15T18:45:51.166 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:50 smithi049 ceph-mon[40087]: pgmap v2061: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:51.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:50 smithi049 ceph-mon[40087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-15T18:45:51.168 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:50 smithi049 ceph-mon[40087]: osdmap e792: 12 total, 11 up, 11 in 2024-04-15T18:45:51.184 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:51.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:51.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:51.317 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.442 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:51.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:51.574 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.593 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:51.696 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:51.729 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.750 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:51.855 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:51.881 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:51.900 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:51.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:51.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:45:52.003 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:52.033 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:51 smithi159 ceph-mon[45336]: osdmap e793: 12 total, 11 up, 11 in 2024-04-15T18:45:52.035 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.056 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:52.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:52.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:51 smithi049 ceph-mon[40087]: osdmap e793: 12 total, 11 up, 11 in 2024-04-15T18:45:52.192 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:51 smithi082 ceph-mon[44206]: osdmap e793: 12 total, 11 up, 11 in 2024-04-15T18:45:52.293 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:52.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:52.437 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.455 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:52.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:52.587 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.606 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:52.710 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:52.740 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.761 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:52.821 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:52.842 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-04-15T18:45:52.864 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:52.893 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:52.913 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:52.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:52.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:52 smithi049 ceph-mon[40087]: pgmap v2064: 17 pgs: 13 unknown, 4 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:52.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:52 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:45:53.017 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:53.043 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:52 smithi159 ceph-mon[45336]: pgmap v2064: 17 pgs: 13 unknown, 4 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:53.043 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:52 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:45:53.045 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:53.064 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:53.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:53.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:53.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:52 smithi082 ceph-mon[44206]: pgmap v2064: 17 pgs: 13 unknown, 4 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:53.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:52 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T18:45:53.197 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:53.215 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:53.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:53.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:53.349 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:53.449 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:53.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:53.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:53.578 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:45:53.578 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:45:53.590 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:54.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:45:54.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:53 smithi159 ceph-mon[45336]: osdmap e794: 12 total, 11 up, 11 in 2024-04-15T18:45:54.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:45:54.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:53 smithi049 ceph-mon[40087]: osdmap e794: 12 total, 11 up, 11 in 2024-04-15T18:45:54.159 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:54.169 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:54.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:54.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:54.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3807713443' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-15T18:45:54.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:53 smithi082 ceph-mon[44206]: osdmap e794: 12 total, 11 up, 11 in 2024-04-15T18:45:54.307 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:54.329 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:54.433 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:54.465 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:54.487 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:54.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:54.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:54.623 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:54.927 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:54.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:54.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:45:55.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:55.064 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:55.085 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:55.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:55.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:54 smithi049 ceph-mon[40087]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:55.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:54 smithi049 ceph-mon[40087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:45:55.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:54 smithi159 ceph-mon[45336]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:55.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:54 smithi159 ceph-mon[45336]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:45:55.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:45:55.197 DEBUG:teuthology.orchestra.run.smithi159:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-04-15T18:45:55.200 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-15T18:45:55.200 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-15T18:45:55.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:54 smithi082 ceph-mon[44206]: pgmap v2066: 17 pgs: 17 active+clean; 577 KiB data, 5.8 GiB used, 978 GiB / 983 GiB avail 2024-04-15T18:45:55.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:54 smithi082 ceph-mon[44206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-15T18:45:55.220 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:55.241 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:45:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:55.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:45:55.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:hints = 1 2024-04-15T18:45:55.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-15T18:45:55.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object prefix: benchmark_data_smithi159_173188 2024-04-15T18:45:55.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:45:55.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 0 0 0 0 0 0 - 0 2024-04-15T18:45:55.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:45:55.380 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:55.405 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:45:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:55.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:45:55.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:55.541 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:55.686 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:45:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:55.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:45:55.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:55.819 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:55.840 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:55.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:55.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:55.980 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:56.109 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:56.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:56.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:56.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:56.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:56.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:56.213 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:56.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:56.238 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:56.240 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:56.259 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:56.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 1 16 1690 1674 104.62 104.625 0.130377 0.00878277 2024-04-15T18:45:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:56.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:56.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:45:56.395 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:56.947 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:45:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:45:57.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:45:57.090 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:57.113 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:45:57.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:56 smithi082 ceph-mon[44206]: pgmap v2067: 17 pgs: 17 active+clean; 2.9 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 395 KiB/s wr, 6 op/s 2024-04-15T18:45:57.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:56 smithi082 ceph-mon[44206]: osdmap e795: 12 total, 11 up, 11 in 2024-04-15T18:45:57.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:45:57.115 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:45:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:57.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:45:57.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:56 smithi159 ceph-mon[45336]: pgmap v2067: 17 pgs: 17 active+clean; 2.9 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 395 KiB/s wr, 6 op/s 2024-04-15T18:45:57.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:56 smithi159 ceph-mon[45336]: osdmap e795: 12 total, 11 up, 11 in 2024-04-15T18:45:57.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:45:57.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:45:57.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:56 smithi049 ceph-mon[40087]: pgmap v2067: 17 pgs: 17 active+clean; 2.9 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 395 KiB/s wr, 6 op/s 2024-04-15T18:45:57.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:56 smithi049 ceph-mon[40087]: osdmap e795: 12 total, 11 up, 11 in 2024-04-15T18:45:57.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:45:57.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:45:57.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:45:57.245 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:57.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 2 16 3523 3507 109.587 114.562 0.00576542 0.00907333 2024-04-15T18:45:57.269 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:45:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:57.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:45:57.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:45:57.412 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:45:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:57.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:45:57.887 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:45:57.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:45:57.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:45:57.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:45:57.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:45:57.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:45:58.005 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:58.013 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:45:58.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:45:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:58.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:45:58.042 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:45:58.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:45:58.108 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:45:58.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:58.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:45:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:57 smithi159 ceph-mon[45336]: osdmap e796: 12 total, 11 up, 11 in 2024-04-15T18:45:58.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3798811506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:58.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:57 smithi049 ceph-mon[40087]: osdmap e796: 12 total, 11 up, 11 in 2024-04-15T18:45:58.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3798811506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:58.174 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:45:58.195 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:58.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:58.209 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:45:58.210 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:45:58.211 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:45:58.259 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 3 16 4962 4946 103.035 89.9375 0.00476075 0.00962489 2024-04-15T18:45:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:57 smithi082 ceph-mon[44206]: osdmap e796: 12 total, 11 up, 11 in 2024-04-15T18:45:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3798811506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:45:58.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:45:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:58.461 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:45:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:45:58.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:45:58.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:45:58.604 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:45:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: pgmap v2070: 17 pgs: 17 active+clean; 108 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 18 MiB/s wr, 286 op/s 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:58 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: pgmap v2070: 17 pgs: 17 active+clean; 108 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 18 MiB/s wr, 286 op/s 2024-04-15T18:45:59.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:58 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 4 16 6724 6708 104.805 110.125 0.0215903 0.00950175 2024-04-15T18:45:59.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: pgmap v2070: 17 pgs: 17 active+clean; 108 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 18 MiB/s wr, 286 op/s 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:45:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:58 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:00.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:00.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:59 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:00.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:00.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:45:59 smithi159 ceph-mon[45336]: osdmap e797: 12 total, 11 up, 11 in 2024-04-15T18:46:00.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:00.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:59 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:00.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:00.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:45:59 smithi049 ceph-mon[40087]: osdmap e797: 12 total, 11 up, 11 in 2024-04-15T18:46:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:46:00.175 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:46:00.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 5 16 8044 8028 100.343 82.5 0.00374764 0.00994505 2024-04-15T18:46:00.260 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:00.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:00.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:00.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:59 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:00.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3118071473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:00.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:45:59 smithi082 ceph-mon[44206]: osdmap e797: 12 total, 11 up, 11 in 2024-04-15T18:46:00.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:00.399 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:00.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:00 smithi049 ceph-mon[40087]: pgmap v2072: 17 pgs: 17 active+clean; 240 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 637 op/s 2024-04-15T18:46:00.951 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:00.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:01.056 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:01.086 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:01.108 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:01.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:01.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:00 smithi159 ceph-mon[45336]: pgmap v2072: 17 pgs: 17 active+clean; 240 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 637 op/s 2024-04-15T18:46:01.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:01.252 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:01.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 6 16 9806 9790 101.972 110.125 0.00327144 0.00979183 2024-04-15T18:46:01.261 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:46:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:46:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:46:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:46:01.261 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:46:01.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:00 smithi082 ceph-mon[44206]: pgmap v2072: 17 pgs: 17 active+clean; 240 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 637 op/s 2024-04-15T18:46:01.279 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:01.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:01.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:01.416 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:01.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:01.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:01.573 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:01.600 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:01.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:01.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:01.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:01.763 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:01.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:01.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:46:02.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:02.086 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:02.111 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:02.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:02.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:46:02.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:02.249 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:02.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 7 16 11942 11926 106.474 133.5 0.00692024 0.00938165 2024-04-15T18:46:02.277 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:02.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:02.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:02.386 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:02.663 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:02.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:02.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:46:02.767 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:02.800 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:02.855 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:02.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:02.955 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:02 smithi049 ceph-mon[40087]: pgmap v2073: 17 pgs: 17 active+clean; 334 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-04-15T18:46:02.989 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:03.015 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:03.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:03.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:02 smithi159 ceph-mon[45336]: pgmap v2073: 17 pgs: 17 active+clean; 334 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-04-15T18:46:03.148 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:03.170 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:03.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:03.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 8 16 13622 13606 106.289 105 0.004335 0.00939325 2024-04-15T18:46:03.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:03.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:02 smithi082 ceph-mon[44206]: pgmap v2073: 17 pgs: 17 active+clean; 334 MiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-04-15T18:46:03.302 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:03.552 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:03.553 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:03.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:03.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:03.689 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:03.712 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:03.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:03.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:03.847 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:03.874 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:03.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:03.980 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:04.010 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.030 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:04.128 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:04.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:04.163 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:04.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 9 16 14902 14886 103.367 80 0.030802 0.00964956 2024-04-15T18:46:04.286 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:04.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.341 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:04.445 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:04.474 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.498 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:04.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:04.633 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.655 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:04.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:04.792 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.820 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:04.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:04 smithi159 ceph-mon[45336]: pgmap v2074: 17 pgs: 17 active+clean; 769 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-15T18:46:04.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:04.954 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:04.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:04 smithi049 ceph-mon[40087]: pgmap v2074: 17 pgs: 17 active+clean; 769 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-15T18:46:04.982 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:04.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:05.087 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:05.118 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:05.142 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:05.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:05.246 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:05.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 10 16 16632 16616 103.842 108.125 0.00478194 0.00962076 2024-04-15T18:46:05.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:04 smithi082 ceph-mon[44206]: pgmap v2074: 17 pgs: 17 active+clean; 769 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-04-15T18:46:05.286 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:05.312 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:05.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:05.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:05.458 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:05.487 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:05.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:22409 7.6 e627/621 7:637759b3:::benchmark_data_smithi159_149998_object22408:head v 627'1465, mlcod=627'1461)", 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 541.00762936000001, 2024-04-15T18:46:05.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21426896200000001, 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519216+0000", 2024-04-15T18:46:05.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519217+0000", 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.099e-06 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.519266+0000", 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8946999999999997e-05 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732196+0000", 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21293002699999999 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732203+0000", 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2450000000000001e-06 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732297+0000", 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4067000000000006e-05 2024-04-15T18:46:05.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.732318+0000", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1378000000000001e-05 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.733441+0000", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011223870000000001 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.733485+0000", 2024-04-15T18:46:05.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3812e-05 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:22411 7.5 e627/621 7:ae9a4933:::benchmark_data_smithi159_149998_object22410:head v 627'1406, mlcod=627'1403)", 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 541.00369422699998, 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.180674529, 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523151+0000", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4000000000000001e-07 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.523218+0000", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6638999999999997e-05 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702813+0000", 2024-04-15T18:46:05.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17959461600000001 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702818+0000", 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3040000000000004e-06 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702936+0000", 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011765300000000001 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.702962+0000", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5967999999999999e-05 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.703798+0000", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00083654400000000005 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:04.703825+0000", 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7064999999999999e-05 2024-04-15T18:46:05.526 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:24097 7.6 e628/621 7:64a751cc:::benchmark_data_smithi159_149998_object24096:head v 628'1566, mlcod=628'1563)", 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 539.48223467100001, 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22156325900000001, 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044610+0000", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044611+0000", 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5899999999999996e-07 2024-04-15T18:46:05.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.044679+0000", 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8415000000000001e-05 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264832+0000", 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22015254400000001 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264839+0000", 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5619999999999998e-06 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264890+0000", 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0745000000000003e-05 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.264908+0000", 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7991000000000001e-05 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266132+0000", 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001223657 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.266174+0000", 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1686e-05 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31783 7.9 e631/621 7:999e18b7:::benchmark_data_smithi159_149998_object31782:head v 631'2061, mlcod=631'2059)", 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 534.48751358699997, 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30491139699999997, 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039331+0000", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039332+0000", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.039377+0000", 2024-04-15T18:46:05.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4564000000000002e-05 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343028+0000", 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.303650851 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343040+0000", 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2043e-05 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343095+0000", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5083000000000001e-05 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.343111+0000", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6446000000000001e-05 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.344208+0000", 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010970000000000001 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.344243+0000", 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4449000000000002e-05 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:31793 7.4 e631/621 7:2fbec3a3:::benchmark_data_smithi159_149998_object31792:head v 631'2016, mlcod=631'2013)", 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 534.47849237299999, 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31115801599999998, 2024-04-15T18:46:05.536 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:46:05.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048353+0000", 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048444+0000", 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1067000000000001e-05 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358065+0000", 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30962077900000001 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358072+0000", 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8680000000000001e-06 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358119+0000", 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7188999999999998e-05 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.358133+0000", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3766e-05 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359477+0000", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013445880000000001 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.359511+0000", 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3061000000000001e-05 2024-04-15T18:46:05.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:55420 7.4 e639/621 7:2ab922f1:::benchmark_data_smithi159_149998_object55419:head v 639'3531, mlcod=639'3527)", 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 514.33174646099997, 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25497385900000002, 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195098+0000", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195099+0000", 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9100000000000004e-07 2024-04-15T18:46:05.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.195196+0000", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6490000000000006e-05 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448347+0000", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25315195600000001 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448355+0000", 2024-04-15T18:46:05.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9580000000000007e-06 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448408+0000", 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2417000000000003e-05 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.448434+0000", 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5846e-05 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.450038+0000", 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001604329 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:31.450072+0000", 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4272e-05 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77949 7.d 7:b4f21db4:::benchmark_data_smithi159_149998_object77948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 491.01241327700001, 2024-04-15T18:46:05.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18121273600000001, 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77949 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514432+0000", 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.514434+0000", 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.367e-06 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515468+0000", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001033876 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515487+0000", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8987e-05 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515499+0000", 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1762e-05 2024-04-15T18:46:05.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516470+0000", 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00097108399999999999 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516482+0000", 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1817999999999999e-05 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.679949+0000", 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16346764499999999 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.681990+0000", 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002041036 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.682168+0000", 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017791199999999999 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690088+0000", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079198110000000006 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690146+0000", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8103999999999998e-05 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.692919+0000", 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027726740000000001 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.692975+0000", 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6691999999999999e-05 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695644+0000", 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026689679999999999 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.552 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77976 7.d 7:bd2ba41d:::benchmark_data_smithi159_149998_object77975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 491.01132122799999, 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18580798200000001, 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77976 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515524+0000", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515525+0000", 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5000000000000001e-07 2024-04-15T18:46:05.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515594+0000", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9438999999999997e-05 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515604+0000", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0032e-05 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515611+0000", 2024-04-15T18:46:05.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8700000000000003e-06 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516508+0000", 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00089666500000000005 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516514+0000", 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4069999999999998e-06 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687324+0000", 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17081004 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687484+0000", 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016032500000000001 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.687654+0000", 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016935999999999999 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695721+0000", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080673120000000001 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695914+0000", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019322799999999999 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.698278+0000", 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023638140000000001 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.698326+0000", 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7848e-05 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.701332+0000", 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030056919999999999 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.559 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77998 7.d 7:b24126a2:::benchmark_data_smithi159_149998_object77997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e648)", 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 491.01122257700001, 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.207862504, 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77998 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515622+0000", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515623+0000", 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000003e-07 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515686+0000", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3291000000000001e-05 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515695+0000", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1900000000000001e-06 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.515702+0000", 2024-04-15T18:46:05.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7279999999999998e-06 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516689+0000", 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0009865570000000001 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.516698+0000", 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7920000000000004e-06 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690233+0000", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17353442899999999 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690357+0000", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123984 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.690464+0000", 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010748500000000001 2024-04-15T18:46:05.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.695895+0000", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054309470000000002 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.701394+0000", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054990689999999997 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.720843+0000", 2024-04-15T18:46:05.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019448213999999998 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.720907+0000", 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3930999999999995e-05 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:54.723485+0000", 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002578313 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.566 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:83519 7.f e651/621 7:f547c4e5:::benchmark_data_smithi159_149998_object83518:head v 651'5282, mlcod=651'5280)", 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 483.13825869200002, 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.190204965, 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388586+0000", 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388587+0000", 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4600000000000003e-07 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.388688+0000", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100833 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572910+0000", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.184222577 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572918+0000", 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3000000000000004e-06 2024-04-15T18:46:05.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572971+0000", 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3060999999999999e-05 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.572993+0000", 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2608000000000001e-05 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.578738+0000", 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057440590000000001 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:02.578791+0000", 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3681000000000003e-05 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:116803 7.5 e661/621 7:a4d27c37:::benchmark_data_smithi159_149998_object20803:head v 661'7308, mlcod=661'7306)", 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:46:05.571 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 436.28082282299999, 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20000991300000001, 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246022+0000", 2024-04-15T18:46:05.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246023+0000", 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.322e-06 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.246031+0000", 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1969999999999999e-06 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.428454+0000", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18242346000000001 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.428461+0000", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9519999999999997e-06 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.429519+0000", 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00105795 2024-04-15T18:46:05.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.429540+0000", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0656999999999999e-05 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.445995+0000", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016454933000000001 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.446032+0000", 2024-04-15T18:46:05.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7441999999999998e-05 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:124560 7.4 e663/621 7:2a9d76e5:::benchmark_data_smithi159_149998_object28560:head v 663'7929, mlcod=663'7926)", 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 425.36464566699999, 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27284343, 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.576 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162199+0000", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162200+0000", 2024-04-15T18:46:05.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7599999999999996e-07 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.162213+0000", 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2889e-05 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431402+0000", 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26918940000000002 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431409+0000", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8619999999999997e-06 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431461+0000", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2212000000000001e-05 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.431476+0000", 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5014999999999999e-05 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.434997+0000", 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035202620000000001 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:00.435043+0000", 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6013999999999997e-05 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:143179 7.d e671/648)", 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 399.209000726, 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31904452599999999, 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317844+0000", 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317845+0000", 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2099999999999995e-07 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.317852+0000", 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9020000000000002e-06 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636785+0000", 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.318932893 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636790+0000", 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4029999999999998e-06 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636870+0000", 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9237e-05 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.636889+0000", 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9170000000000001e-05 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.584 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:143366 7.4 e672/621 7:21b697b5:::benchmark_data_smithi159_149998_object47366:head v 672'9144, mlcod=672'9142)", 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 398.62025633899998, 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53369277100000001, 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906589+0000", 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906590+0000", 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.717e-06 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:26.906604+0000", 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3473e-05 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435564+0000", 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52895973699999999 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435570+0000", 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7490000000000001e-06 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435679+0000", 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010908600000000001 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.435700+0000", 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0897e-05 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.440236+0000", 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045354540000000004 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:27.440281+0000", 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5658000000000002e-05 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.589 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:191961 7.d e700/672 7:b225a1f7:::benchmark_data_smithi159_149998_object95961:head v 700'11971, mlcod=700'11968)", 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 329.65290038099999, 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43827476300000001, 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9999999999999998e-07 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873949+0000", 2024-04-15T18:46:05.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2289999999999997e-06 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308026+0000", 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43407702100000001 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308036+0000", 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4180000000000003e-06 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308109+0000", 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3162000000000002e-05 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308139+0000", 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0298e-05 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312172+0000", 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004033043 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312219+0000", 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7191999999999998e-05 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:109194 8.9 e749/709 8:93c68f30:::benchmark_data_smithi159_162467_object6889:head v 749'6779, mlcod=749'6776)", 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 185.86686969900001, 2024-04-15T18:46:05.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25230181200000001, 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:46:05.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659976+0000", 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659983+0000", 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6229999999999999e-06 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905691+0000", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24570813499999999 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905702+0000", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0795000000000001e-05 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905778+0000", 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6173000000000004e-05 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905804+0000", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5799000000000002e-05 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.912212+0000", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064086550000000001 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.912277+0000", 2024-04-15T18:46:05.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4852999999999997e-05 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:113660 8.9 e751/709 8:9831ce8f:::benchmark_data_smithi159_162467_object11355:head v 751'7050, mlcod=750'7048)", 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 178.58729679499999, 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17086718400000001, 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:46:05.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4299999999999999e-07 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939555+0000", 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0090000000000001e-06 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106710+0000", 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16715449299999999 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106720+0000", 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0281e-05 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106807+0000", 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6903999999999993e-05 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106833+0000", 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5537e-05 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.110352+0000", 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035193429999999999 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.110415+0000", 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3373999999999993e-05 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:128498 8.e e754/709 8:7c81d7a0:::benchmark_data_smithi159_162467_object26193:head v 754'8122, mlcod=754'8119)", 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 155.02225367299999, 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22033059499999999, 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504592+0000", 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504601+0000", 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5019999999999992e-06 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721186+0000", 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21658566800000001 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721196+0000", 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.73e-06 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721252+0000", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5627000000000001e-05 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721276+0000", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4094999999999999e-05 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.724872+0000", 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035959630000000002 2024-04-15T18:46:05.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.724922+0000", 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0042e-05 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:164219 8.e e766/709 8:73021211:::benchmark_data_smithi159_162467_object61914:head v 766'10389, mlcod=766'10386)", 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 96.447383559000002, 2024-04-15T18:46:05.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.370137667, 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:05.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079463+0000", 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079472+0000", 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7669999999999998e-06 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:05.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444723+0000", 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36525010400000002 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444733+0000", 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9939999999999995e-06 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444801+0000", 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8874000000000005e-05 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444824+0000", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2362e-05 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449560+0000", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004735815 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449599+0000", 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9407999999999999e-05 2024-04-15T18:46:05.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:05.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:05.613 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:46:05.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:05.649 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:05.674 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:05.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:05.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:05.811 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:06.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 11 16 18610 18594 105.64 123.625 0.00182271 0.00945968 2024-04-15T18:46:06.519 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:06.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:06.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:06.659 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:06.684 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:06.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:06.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:06.817 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:07.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:06 smithi082 ceph-mon[44206]: pgmap v2075: 17 pgs: 17 active+clean; 803 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:07.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:06 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:07.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:06 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:07.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:06 smithi159 ceph-mon[45336]: pgmap v2075: 17 pgs: 17 active+clean; 803 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:07.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:06 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:07.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:06 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:07.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:06 smithi049 ceph-mon[40087]: pgmap v2075: 17 pgs: 17 active+clean; 803 MiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:07.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:06 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:07.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:06 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:07.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 12 16 19872 19856 103.409 78.875 0.0717917 0.00958322 2024-04-15T18:46:08.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:07 smithi159 ceph-mon[45336]: osdmap e798: 12 total, 11 up, 11 in 2024-04-15T18:46:08.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:07 smithi049 ceph-mon[40087]: osdmap e798: 12 total, 11 up, 11 in 2024-04-15T18:46:08.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 13 16 21263 21247 102.141 86.9375 0.00310705 0.00977388 2024-04-15T18:46:08.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:07 smithi082 ceph-mon[44206]: osdmap e798: 12 total, 11 up, 11 in 2024-04-15T18:46:08.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:46:09.036 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:09.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:08 smithi049 ceph-mon[40087]: pgmap v2077: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:46:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:09.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:09.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:09.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:08 smithi159 ceph-mon[45336]: pgmap v2077: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:46:09.178 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:08 smithi082 ceph-mon[44206]: pgmap v2077: 17 pgs: 17 active+clean; 1.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-04-15T18:46:09.180 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:09.207 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:09.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:09.260 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 14 16 22703 22687 101.274 90 0.00415642 0.00986862 2024-04-15T18:46:09.313 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:09.345 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:09.374 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:09.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:09.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:09.511 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:10.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 15 16 24468 24452 101.876 110.312 0.00393165 0.00980731 2024-04-15T18:46:10.482 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:10.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:10.590 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:10.620 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:10.647 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:10.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:10.751 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:10.781 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:10.805 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:10.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:10.909 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:10.940 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:10.966 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:10.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:11.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:11.106 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:10 smithi049 ceph-mon[40087]: pgmap v2078: 17 pgs: 17 active+clean; 1.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:46:11.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 16 16 25903 25887 101.113 89.6875 0.0410989 0.00975426 2024-04-15T18:46:11.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:10 smithi082 ceph-mon[44206]: pgmap v2078: 17 pgs: 17 active+clean; 1.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:46:11.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:10 smithi159 ceph-mon[45336]: pgmap v2078: 17 pgs: 17 active+clean; 1.2 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:46:11.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:46:11.503 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:46:11.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:46:11.504 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:46:11.539 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:11.540 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:11.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:11.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:11.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:11.721 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:11.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:11.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:11.873 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:11.893 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:11.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:11.904 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:46:11.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:12.030 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:12.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3123851939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:12.197 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.199 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 17 16 27233 27217 100.055 83.125 0.00355011 0.0099869 2024-04-15T18:46:12.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3123851939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:12.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.279 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:12.384 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:12.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3123851939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:12.424 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:12.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:12.600 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.627 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:12.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:12.759 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.780 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:12.884 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:12.916 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:12.938 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:12.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: pgmap v2079: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: osdmap e799: 12 total, 11 up, 11 in 2024-04-15T18:46:13.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:12 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: pgmap v2079: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:13.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: osdmap e799: 12 total, 11 up, 11 in 2024-04-15T18:46:13.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:12 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:13.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:13.069 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.091 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:13.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:13.233 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.257 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:13.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 18 16 28674 28658 99.4991 90.0625 0.00223069 0.0100443 2024-04-15T18:46:13.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: pgmap v2079: 17 pgs: 17 active+clean; 1.3 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1966468012' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: osdmap e799: 12 total, 11 up, 11 in 2024-04-15T18:46:13.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:12 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:13.399 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.481 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:46:13.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:46:13.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:13.617 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.636 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:13.740 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:13.770 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.790 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:13.894 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:13.919 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:13.938 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:13.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:14.041 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:14.067 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:14.091 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:14.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:14.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:14.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:14.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:13 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:14.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:13 smithi049 ceph-mon[40087]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:46:14.233 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:14.233 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:13 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:14.233 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:13 smithi082 ceph-mon[44206]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:46:14.234 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:14.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 19 16 29798 29782 97.9589 70.25 0.00684121 0.0102027 2024-04-15T18:46:14.263 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:14.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:14.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:14.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:14.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:13 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:14.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:13 smithi159 ceph-mon[45336]: from='mgr.14151 ' entity='mgr.y' 2024-04-15T18:46:14.401 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:14.427 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:14.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:14.533 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:14.566 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:14.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:14.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:14.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:14.736 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:14.938 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:14.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:15.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:15.070 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:15.092 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:15.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:15.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:15.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:14 smithi049 ceph-mon[40087]: pgmap v2081: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-15T18:46:15.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:14 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:15.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:14 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:15.230 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:14 smithi082 ceph-mon[44206]: pgmap v2081: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-15T18:46:15.231 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:14 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:15.231 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:14 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:15.233 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:15.258 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:15.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:46:15.273998+0000 min lat: 0.00106913 max lat: 0.341675 avg lat: 0.0102016 2024-04-15T18:46:15.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:46:15.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 20 16 31365 31349 97.9574 97.9375 0.0111777 0.0102016 2024-04-15T18:46:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:15.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:15.363 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:15.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:14 smithi159 ceph-mon[45336]: pgmap v2081: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-04-15T18:46:15.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:14 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:15.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:14 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:15.398 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:15.405 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:46:15.405 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:46:15.425 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:15.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:15.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:46:15.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:15.607 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:15.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:15.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:15.829 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:15.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:46:15.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:15 smithi049 ceph-mon[40087]: osdmap e800: 12 total, 11 up, 11 in 2024-04-15T18:46:16.182 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:16.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:46:16.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 21 16 33024 33008 98.2294 103.688 0.012645 0.010168 2024-04-15T18:46:16.265 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:16.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:15 smithi082 ceph-mon[44206]: osdmap e800: 12 total, 11 up, 11 in 2024-04-15T18:46:16.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:16.314 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:15 smithi159 ceph-mon[45336]: osdmap e800: 12 total, 11 up, 11 in 2024-04-15T18:46:16.316 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:16.334 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:16.438 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:16.467 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:16.489 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:16.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:16.622 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:16.642 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:16.747 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:16.775 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:16.794 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:16.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:16.932 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:16.959 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:16.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:17.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:17.097 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:16 smithi049 ceph-mon[40087]: pgmap v2083: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:46:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.121 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:17.130 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:17.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:16 smithi159 ceph-mon[45336]: pgmap v2083: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:46:17.167 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:17.236 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:17.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 22 16 34160 34144 96.9914 71 0.015859 0.0103023 2024-04-15T18:46:17.268 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:16 smithi082 ceph-mon[44206]: pgmap v2083: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-15T18:46:17.271 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.294 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:17.398 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:17.432 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.454 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:17.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:17.595 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.620 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:17.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:17.765 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.798 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:17.902 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:17.940 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:17.965 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:17.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:18.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:18.108 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:18.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 23 16 36105 36089 98.0593 121.562 0.0193144 0.0101904 2024-04-15T18:46:18.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:18.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:18.867 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:18.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:18.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:19.008 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:19.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:19.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:19.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:19.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:18 smithi049 ceph-mon[40087]: pgmap v2084: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:19.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:18 smithi049 ceph-mon[40087]: osdmap e801: 12 total, 11 up, 11 in 2024-04-15T18:46:19.174 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:19.261 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 24 16 37406 37390 97.3613 81.3125 0.00371846 0.0102663 2024-04-15T18:46:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:18 smithi082 ceph-mon[44206]: pgmap v2084: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:18 smithi082 ceph-mon[44206]: osdmap e801: 12 total, 11 up, 11 in 2024-04-15T18:46:19.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:18 smithi159 ceph-mon[45336]: pgmap v2084: 17 pgs: 17 active+clean; 1.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-04-15T18:46:19.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:18 smithi159 ceph-mon[45336]: osdmap e801: 12 total, 11 up, 11 in 2024-04-15T18:46:20.029 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:20.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:20.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:20.170 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:20.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 25 16 38847 38831 97.0691 90.0625 0.0132761 0.0102968 2024-04-15T18:46:20.535 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:20.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:20.641 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:20.673 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:20.695 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:20.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:20.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:20.830 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:20.856 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:20.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:20.929 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:20 smithi049 ceph-mon[40087]: pgmap v2086: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:46:20.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:20.989 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:21.011 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:21.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:21.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:21.148 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:21.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 26 16 40237 40221 96.6767 86.875 0.0105088 0.0103326 2024-04-15T18:46:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:20 smithi082 ceph-mon[44206]: pgmap v2086: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:46:21.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:20 smithi159 ceph-mon[45336]: pgmap v2086: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-04-15T18:46:22.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 27 16 41743 41727 96.5819 94.125 0.00947335 0.0103458 2024-04-15T18:46:23.030 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:46:23.067 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:46:23.095 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:46:23.122 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:46:23.148 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:46:23.172 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:46:23.197 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:46:23.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:22 smithi049 ceph-mon[40087]: pgmap v2087: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:46:23.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:46:23.248 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:46:23.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 28 16 43056 43040 96.063 82.0625 0.1283 0.0103745 2024-04-15T18:46:23.273 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:46:23.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:22 smithi082 ceph-mon[44206]: pgmap v2087: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:46:23.298 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:46:23.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:46:23.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:46:23.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:46:23.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:46:23.299 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:46:23.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:22 smithi159 ceph-mon[45336]: pgmap v2087: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:46:23.599 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:23.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:23.707 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:23.741 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:23.767 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:23.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:23.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:23.903 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:24.089 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:24.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:46:24.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1079037513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:23 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1079037513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:23 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.233 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:24.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 29 16 44623 44607 96.1273 97.9375 0.00975102 0.0103982 2024-04-15T18:46:24.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1079037513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:24.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:23 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:24.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:24.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:24.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:24.536 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:24.581 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:24.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:24.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:24.721 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:24.749 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:24.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:24.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:24.810 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:46:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:46:24.810 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:46:24.811 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:46:24.811 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:46:24.853 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:24.883 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:24.905 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:24.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:25.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:25.043 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:25.071 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:25.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:25.090 INFO:tasks.thrashosds.thrasher:Reviving osd 11 2024-04-15T18:46:25.091 INFO:tasks.cephadm.osd.11:Restarting osd.11 (starting--it wasn't running)... 2024-04-15T18:46:25.091 DEBUG:teuthology.orchestra.run.smithi159:osd.11> sudo journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11.service 2024-04-15T18:46:25.093 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl start ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11 2024-04-15T18:46:25.131 INFO:journalctl@ceph.osd.11.smithi159.stdout:-- Logs begin at Mon 2024-04-15 17:01:25 UTC. -- 2024-04-15T18:46:25.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:24 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:24 smithi159 ceph-mon[45336]: pgmap v2088: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-15T18:46:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:24 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:25.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:24 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:25.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:24 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:24 smithi049 ceph-mon[40087]: pgmap v2088: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-15T18:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:24 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:24 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:25.221 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:25.248 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:25.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:25.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 30 16 46186 46170 96.1791 97.6875 0.0097296 0.0103916 2024-04-15T18:46:25.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:24 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:25.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:24 smithi082 ceph-mon[44206]: pgmap v2088: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-04-15T18:46:25.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:24 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:25.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:24 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:25.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:25.389 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:25.644 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:25 smithi159 systemd[1]: Starting Ceph osd.11 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:46:25.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:46:26.044 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:26.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:25 smithi049 ceph-mon[40087]: osdmap e802: 12 total, 11 up, 11 in 2024-04-15T18:46:26.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:26.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:26.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:26.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:26.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:26.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:46:26.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:26.216 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:26.224 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:26.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 31 16 47366 47350 95.4554 73.75 0.00898561 0.0104692 2024-04-15T18:46:26.263 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:26.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:26.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:25 smithi082 ceph-mon[44206]: osdmap e802: 12 total, 11 up, 11 in 2024-04-15T18:46:26.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:26.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:26.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:26.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:26.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:26.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:25 smithi159 ceph-mon[45336]: osdmap e802: 12 total, 11 up, 11 in 2024-04-15T18:46:26.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:26.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:26.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:26.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:26.423 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:26.923 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:26.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:26.972 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:46:27.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:27.059 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:27.080 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:27.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:27.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:26 smithi049 ceph-mon[40087]: pgmap v2090: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:46:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:26 smithi049 ceph-mon[40087]: osdmap e803: 12 total, 11 up, 11 in 2024-04-15T18:46:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:27.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:46:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:27.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 32 16 48682 48666 95.0425 82.25 0.0137076 0.0105169 2024-04-15T18:46:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:26 smithi082 ceph-mon[44206]: pgmap v2090: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:46:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:26 smithi082 ceph-mon[44206]: osdmap e803: 12 total, 11 up, 11 in 2024-04-15T18:46:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:46:27.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:26 smithi159 ceph-mon[45336]: pgmap v2090: 17 pgs: 17 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-04-15T18:46:27.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:26 smithi159 ceph-mon[45336]: osdmap e803: 12 total, 11 up, 11 in 2024-04-15T18:46:27.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:27.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:27.288 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:46:27.288 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-15T18:46:27.288 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-11 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-15T18:46:27.288 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-15T18:46:27.289 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-15T18:46:27.289 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-11/block 2024-04-15T18:46:27.289 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-15T18:46:27.289 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:26 smithi159 bash[174305]: --> ceph-volume raw activate successful for osd ID: 11 2024-04-15T18:46:28.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:27 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:28.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:46:28.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:27 smithi049 ceph-mon[40087]: osdmap e804: 12 total, 11 up, 11 in 2024-04-15T18:46:28.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:27 smithi049 ceph-mon[40087]: osdmap e805: 12 total, 11 up, 11 in 2024-04-15T18:46:28.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 33 16 50010 49994 94.6773 83 0.0084574 0.0105573 2024-04-15T18:46:28.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:27 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:28.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:46:28.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:27 smithi082 ceph-mon[44206]: osdmap e804: 12 total, 11 up, 11 in 2024-04-15T18:46:28.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:27 smithi082 ceph-mon[44206]: osdmap e805: 12 total, 11 up, 11 in 2024-04-15T18:46:28.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:27 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:28.368 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3741964768' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:46:28.369 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:27 smithi159 ceph-mon[45336]: osdmap e804: 12 total, 11 up, 11 in 2024-04-15T18:46:28.369 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:27 smithi159 ceph-mon[45336]: osdmap e805: 12 total, 11 up, 11 in 2024-04-15T18:46:28.369 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:28 smithi159 bash[174537]: 84f82972c3895f36f9a1a3f0a0fda0873c7679fc58f42c60d7da6ca83c6ec0ba 2024-04-15T18:46:28.375 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:46:28.483 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:28.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:28.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:28.626 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:28.644 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:28 smithi159 systemd[1]: Started Ceph osd.11 for 22c655ba-fb52-11ee-bc8f-c7b262605968. 2024-04-15T18:46:28.796 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:46:28.890 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:29.092 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:28 smithi159 ceph-mon[45336]: pgmap v2093: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-15T18:46:29.092 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:28.838+0000 7f737f9e47c0 -1 Falling back to public interface 2024-04-15T18:46:29.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:28 smithi049 ceph-mon[40087]: pgmap v2093: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-15T18:46:29.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 34 16 51537 51521 94.6995 95.4375 0.0125792 0.0105539 2024-04-15T18:46:29.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:28 smithi082 ceph-mon[44206]: pgmap v2093: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-04-15T18:46:29.379 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:29.486 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:29.518 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:29.549 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:29.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:29.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:29.707 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:29.740 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:29.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:29.751 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:46:29.751 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:46:29.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:24044 7.b 7:d6248a3d:::benchmark_data_smithi159_149998_object24043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e628)", 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:05.995665+0000", 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 563.76505868599997, 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26526930100000001, 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:29.752 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 24044 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995665+0000", 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995665+0000", 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995667+0000", 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.483e-06 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995723+0000", 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6116999999999998e-05 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995736+0000", 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3441999999999999e-05 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995743+0000", 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.426e-06 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995811+0000", 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8391000000000003e-05 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.995925+0000", 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011385199999999999 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.996033+0000", 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107911 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.997382+0000", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013493279999999999 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:05.997700+0000", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00031827500000000002 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.260813+0000", 2024-04-15T18:46:29.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.263112927 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.260869+0000", 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6159000000000002e-05 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:06.260934+0000", 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4989999999999999e-05 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.758 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:31792 7.b 7:d043e990:::benchmark_data_smithi159_149998_object31791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.047681+0000", 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 558.71304261199998, 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.314196323, 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:29.759 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 31792 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047681+0000", 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047681+0000", 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047682+0000", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1519999999999999e-06 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047774+0000", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1718999999999994e-05 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047785+0000", 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0465000000000001e-05 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047790+0000", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6899999999999997e-06 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047831+0000", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0367000000000002e-05 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.047935+0000", 2024-04-15T18:46:29.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104318 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.048029+0000", 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4004000000000003e-05 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.049021+0000", 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00099244799999999994 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.049251+0000", 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022960999999999999 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.361794+0000", 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31254316599999998 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.361833+0000", 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8683000000000002e-05 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.361878+0000", 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4700999999999998e-05 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:31802 7.d 7:b4a14b96:::benchmark_data_smithi159_149998_object31801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.064582+0000", 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 558.69614213800003, 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28987720500000003, 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.765 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 31802 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064582+0000", 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064582+0000", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064583+0000", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.378e-06 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064643+0000", 2024-04-15T18:46:29.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9775000000000003e-05 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064654+0000", 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1168999999999999e-05 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064660+0000", 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6710000000000004e-06 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064698+0000", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8454000000000001e-05 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064809+0000", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110517 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.064901+0000", 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2325999999999997e-05 2024-04-15T18:46:29.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.065925+0000", 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001024451 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.066254+0000", 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032833500000000002 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354385+0000", 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.288131313 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354425+0000", 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9708e-05 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354459+0000", 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4107999999999999e-05 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.771 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26545.0:31802 7.d e631/621 ondisk, result = 0)", 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:11.066217+0000", 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 558.69450717300003, 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.288250599, 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.066217+0000", 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.066217+0000", 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.066217+0000", 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2900000000000003e-07 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.066221+0000", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8330000000000004e-06 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354316+0000", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28809494099999999 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354324+0000", 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6990000000000003e-06 2024-04-15T18:46:29.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354373+0000", 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9288e-05 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354381+0000", 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2400000000000007e-06 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:11.354467+0000", 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5969000000000003e-05 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26545.0:77956 7.b 7:d2b02bbb:::benchmark_data_smithi159_149998_object77955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e647)", 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:37:52.504099+0000", 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 517.25662510799998, 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.59386945899999999, 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.776 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26545", 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 77956 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504099+0000", 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504099+0000", 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504099+0000", 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4299999999999998e-07 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504177+0000", 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8017999999999997e-05 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504190+0000", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2493999999999999e-05 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.504197+0000", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5349999999999999e-06 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.506691+0000", 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002493725 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.506884+0000", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019353299999999999 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.507056+0000", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000171597 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.509763+0000", 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027068779999999998 2024-04-15T18:46:29.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:52.509913+0000", 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149741 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:53.097882+0000", 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.58796908800000003 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:53.097923+0000", 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1072000000000002e-05 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:37:53.097968+0000", 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5334999999999999e-05 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "replica_scrub(pg: 7.d,from:0'0,to:675'9070,epoch:676/672,start:7:b3d02dcb:::benchmark_data_smithi159_149998_object55481:0,end:7:b470ce8e:::benchmark_data_smithi159_149998_object73560:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-15T18:46:29.782 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:31.621230+0000", 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 418.13949423100001, 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54691076800000005, 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.621230+0000", 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.621230+0000", 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.621231+0000", 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4980000000000001e-06 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.621240+0000", 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.167988+0000", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54674728399999994 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.168005+0000", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6877000000000001e-05 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.168092+0000", 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7737000000000004e-05 2024-04-15T18:46:29.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.168111+0000", 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9055e-05 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.168140+0000", 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8946999999999999e-05 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86279 8.7 8:e097fc19:::benchmark_data_smithi159_162467_object86278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 242.983866613, 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60823186699999998, 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86279 2024-04-15T18:46:29.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776858+0000", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.27e-07 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776884+0000", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6044e-05 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776889+0000", 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6459999999999998e-06 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776893+0000", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9539999999999998e-06 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776945+0000", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1909000000000001e-05 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.777096+0000", 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000150371 2024-04-15T18:46:29.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.777265+0000", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016954200000000001 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.779828+0000", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025631930000000001 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.384972+0000", 2024-04-15T18:46:29.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60514323999999997 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.385012+0000", 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0969e-05 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.385089+0000", 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6571999999999998e-05 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86309 8.7 8:e02e5510:::benchmark_data_smithi159_162467_object86308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 242.89381157099999, 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52165454600000005, 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86309 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.793 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866919+0000", 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9240000000000001e-06 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866972+0000", 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2488999999999998e-05 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866984+0000", 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2003e-05 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866990+0000", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8710000000000002e-06 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867033+0000", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3448000000000001e-05 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867143+0000", 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109533 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867249+0000", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106251 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.869488+0000", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022394400000000001 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388441+0000", 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51895257500000003 2024-04-15T18:46:29.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388496+0000", 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4972999999999997e-05 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388567+0000", 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1038999999999995e-05 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86345 8.7 8:e9b0afce:::benchmark_data_smithi159_162467_object86344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 242.748166129, 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.376124766, 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86345 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.799 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012560+0000", 2024-04-15T18:46:29.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.911e-06 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012594+0000", 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4558999999999999e-05 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012614+0000", 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0091000000000001e-05 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012623+0000", 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3049999999999993e-06 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012692+0000", 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9944000000000007e-05 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012880+0000", 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018745 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012992+0000", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111853 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.015515+0000", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025232330000000002 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388593+0000", 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37307769600000001 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388624+0000", 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1769e-05 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388682+0000", 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7955000000000003e-05 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.804 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(8.7 775/743 [PushReplyOp(8:ee0ea841:::benchmark_data_smithi159_162467_object84326:head)])", 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.424958+0000", 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.335765456000004, 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25462447700000002, 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.424958+0000", 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.424958+0000", 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.424959+0000", 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.61e-07 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.424965+0000", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.446e-06 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.678209+0000", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.253244419 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.678222+0000", 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3258000000000001e-05 2024-04-15T18:46:29.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.678305+0000", 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2488e-05 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.679583+0000", 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012781050000000001 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:184701 8.1 8:8f058534:::benchmark_data_smithi159_162467_object82396:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.425333+0000", 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.335390855, 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26057097200000001, 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 184701 2024-04-15T18:46:29.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425333+0000", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425333+0000", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425335+0000", 2024-04-15T18:46:29.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.823e-06 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425340+0000", 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4770000000000002e-06 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425352+0000", 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1702000000000001e-05 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425359+0000", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9920000000000001e-06 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425426+0000", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7535999999999998e-05 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425582+0000", 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000155403 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,9", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.425727+0000", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000145358 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.430572+0000", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004844671 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.444875+0000", 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014303062 2024-04-15T18:46:29.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685831+0000", 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.240956067 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685868+0000", 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7001e-05 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685904+0000", 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5880000000000002e-05 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26791.0:184701 8.1 e775/709 ondisk, result = 0)", 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.429762+0000", 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.330961655999999, 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25614594699999998, 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:46:29.815 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.429762+0000", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.429762+0000", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.429763+0000", 2024-04-15T18:46:29.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.429775+0000", 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1426999999999999e-05 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685752+0000", 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.255977026 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685758+0000", 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3550000000000001e-06 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685820+0000", 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2526000000000004e-05 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685827+0000", 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.392e-06 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.685908+0000", 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1260000000000005e-05 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:184711 8.5 e775/709 8:aa5d4b26:::benchmark_data_smithi159_162467_object82406:head v 775'11384, mlcod=775'11380)", 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.455457+0000", 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.305266908999997, 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25636925900000002, 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.455457+0000", 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.455457+0000", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.455458+0000", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.68e-07 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.455464+0000", 2024-04-15T18:46:29.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5320000000000003e-06 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.708002+0000", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25253800799999998 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.708006+0000", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.335e-06 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.708076+0000", 2024-04-15T18:46:29.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9783000000000006e-05 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.708093+0000", 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6385e-05 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.711792+0000", 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036992940000000001 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.711826+0000", 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4254000000000001e-05 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:186539 8.7 8:ee8c98d9:::benchmark_data_smithi159_162467_object84234:head [delete] snapc 0=[] ondisk+write+known_if_redirected e776)", 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:05.323210+0000", 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 84.437514234000005, 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24676619, 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.824 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186539 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323210+0000", 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323210+0000", 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323212+0000", 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1550000000000001e-06 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323219+0000", 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4340000000000003e-06 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323232+0000", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3112000000000001e-05 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.323242+0000", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0186e-05 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.325358+0000", 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002115745 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.325459+0000", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101259 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.325552+0000", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2077999999999995e-05 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.335908+0000", 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010356582 2024-04-15T18:46:29.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.358018+0000", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.022109911999999999 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.568822+0000", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.210804135 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.568882+0000", 2024-04-15T18:46:29.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9528000000000003e-05 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.569976+0000", 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010940640000000001 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:186557 8.7 8:e0ea21b2:::benchmark_data_smithi159_162467_object84252:head [delete] snapc 0=[] ondisk+write+known_if_redirected e776)", 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:05.349917+0000", 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 84.410807239999997, 2024-04-15T18:46:29.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23856976499999999, 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186557 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349917+0000", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349917+0000", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349918+0000", 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349922+0000", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7199999999999997e-06 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349933+0000", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0611000000000001e-05 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349940+0000", 2024-04-15T18:46:29.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0779999999999998e-06 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.349997+0000", 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6629999999999998e-05 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.350136+0000", 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000139388 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,10", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.350275+0000", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013867999999999999 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.355119+0000", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048443640000000003 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.376887+0000", 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021767861999999999 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.587511+0000", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.210623854 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.587570+0000", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9577999999999998e-05 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:05.588486+0000", 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000916032 2024-04-15T18:46:29.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(8.7 e786,743 8:efc191eb:::benchmark_data_smithi159_162467_object72738:head,769'10956)", 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 52.352901731999999, 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.723278898, 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407823+0000", 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3399999999999999e-07 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407826+0000", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7890000000000001e-06 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130788+0000", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72296189200000005 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130797+0000", 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2900000000000002e-06 2024-04-15T18:46:29.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130888+0000", 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1656000000000005e-05 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.131101+0000", 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021273700000000001 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:20512 9.2 9:4609eba9:::benchmark_data_smithi159_173188_object20511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:07.599679+0000", 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.161044490999998, 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23149657000000001, 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 20512 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599679+0000", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599679+0000", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599680+0000", 2024-04-15T18:46:29.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0699999999999996e-07 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599765+0000", 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5187999999999998e-05 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599789+0000", 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3825e-05 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.599796+0000", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2479999999999999e-06 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600094+0000", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029735999999999999 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600232+0000", 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000138245 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600396+0000", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000163788 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.601133+0000", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00073755000000000001 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.601293+0000", 2024-04-15T18:46:29.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015979000000000001 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.831101+0000", 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22980814999999999 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.831139+0000", 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7367000000000003e-05 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.831176+0000", 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7252000000000002e-05 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:49724 9.0 9:02cec66c:::benchmark_data_smithi159_173188_object49723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:27.913988+0000", 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.8467357199999999, 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22630937600000001, 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.847 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49724 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.913988+0000", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.913988+0000", 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.913989+0000", 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2699999999999998e-07 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914056+0000", 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7189999999999999e-05 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914069+0000", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2439999999999999e-05 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914076+0000", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3420000000000002e-06 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914121+0000", 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5077999999999998e-05 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914225+0000", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010362899999999999 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,8", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.914311+0000", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6567e-05 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.915683+0000", 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001371616 2024-04-15T18:46:29.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.994891+0000", 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.079207771999999996 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140224+0000", 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14533352299999999 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140261+0000", 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6918000000000001e-05 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140297+0000", 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6473999999999997e-05 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.27025.0:49724 9.0 e805/791 ondisk, result = 0)", 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:27.916243+0000", 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.84448089, 2024-04-15T18:46:29.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22406013299999999, 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.916243+0000", 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.916243+0000", 2024-04-15T18:46:29.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.916243+0000", 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6100000000000001e-07 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.916249+0000", 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5300000000000004e-06 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140181+0000", 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223932043 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140185+0000", 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.155e-06 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140216+0000", 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0545e-05 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140221+0000", 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-06 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.140303+0000", 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2548999999999999e-05 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:49733 9.e e805/791 9:79a4aebf:::benchmark_data_smithi159_173188_object49732:head v 805'3136, mlcod=805'3133)", 2024-04-15T18:46:29.857 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:27.919415+0000", 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.84130854, 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.223844878, 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.919415+0000", 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:46:29.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.919415+0000", 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.919416+0000", 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1100000000000001e-07 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:27.919514+0000", 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8136999999999999e-05 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.141384+0000", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22187030899999999 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.141388+0000", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.794e-06 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.141431+0000", 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3513999999999997e-05 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.141453+0000", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1297e-05 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.143201+0000", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017481619999999999 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:28.143260+0000", 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9253999999999999e-05 2024-04-15T18:46:29.861 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:29.862 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:29.862 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:46:29.862 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:29.902 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:30.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 35 16 53064 53048 94.7203 95.4375 0.00536821 0.0105526 2024-04-15T18:46:30.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:46:31.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:30 smithi049 ceph-mon[40087]: pgmap v2095: 17 pgs: 17 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:46:31.262 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 36 16 54316 54300 94.2627 78.25 0.0230023 0.0106028 2024-04-15T18:46:31.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:30 smithi082 ceph-mon[44206]: pgmap v2095: 17 pgs: 17 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:46:31.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:30 smithi159 ceph-mon[45336]: pgmap v2095: 17 pgs: 17 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-04-15T18:46:31.964 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:31.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:32.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:32.114 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:32.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 37 16 55644 55628 93.9581 83 0.00494617 0.0106384 2024-04-15T18:46:32.394 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:31.977+0000 7f737f9e47c0 -1 osd.11 733 log_to_monitors true 2024-04-15T18:46:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:46:32.911 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:46:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:46:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:46:32.912 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:46:32.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.722+0000 7f736c7c7700 -1 osd.11 733 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f735b611700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f7358e0c700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f7358e0c700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f7358e0c700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f7358e0c700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.972 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.741+0000 7f7358e0c700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735b611700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735b611700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735b611700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735b611700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735c613700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735c613700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735c613700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735c613700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735c613700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735be12700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735be12700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735be12700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735be12700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.973 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f735be12700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7359e0e700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7359e0e700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7359e0e700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7359e0e700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7359e0e700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7357e0a700 -1 osd.11 739 failed to load OSD map for epoch 734, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7357e0a700 -1 osd.11 739 failed to load OSD map for epoch 735, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7357e0a700 -1 osd.11 739 failed to load OSD map for epoch 736, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7357e0a700 -1 osd.11 739 failed to load OSD map for epoch 737, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.742+0000 7f7357e0a700 -1 osd.11 739 failed to load OSD map for epoch 738, got 0 bytes 2024-04-15T18:46:32.974 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:46:32.790+0000 7f7370e3c700 -1 osd.11 806 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T18:46:32.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:46:33.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:46:33.034 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:46:33.067 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:46:33.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:46:33.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:46:33.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:46:33.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:46:33.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:46:33.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:46:33.100 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-04-15T18:46:33.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 38 16 56723 56707 93.26 67.4375 0.106234 0.010699 2024-04-15T18:46:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: pgmap v2096: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: osdmap e806: 12 total, 11 up, 11 in 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1701580557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:33.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: pgmap v2096: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: osdmap e806: 12 total, 11 up, 11 in 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1701580557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:33.378 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:33.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: pgmap v2096: 17 pgs: 17 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-04-15T18:46:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-15T18:46:33.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-15T18:46:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: osdmap e806: 12 total, 11 up, 11 in 2024-04-15T18:46:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi159", "root=default"]}]: dispatch 2024-04-15T18:46:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1701580557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:33.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:46:33.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:33.537 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:33.591 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:33.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:33.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:33.730 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:33.790 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:33.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792] boot 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: osdmap e807: 12 total, 12 up, 11 in 2024-04-15T18:46:33.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:33 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-15T18:46:34.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-15T18:46:34.073 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:34.083 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:34.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:34.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:34.223 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:34.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 39 16 57963 57947 92.8557 77.5 0.0142348 0.0107637 2024-04-15T18:46:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792] boot 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: osdmap e807: 12 total, 12 up, 11 in 2024-04-15T18:46:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:33 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:34.326 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:34.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osd.11 [v2:172.21.15.159:6824/905169792,v1:172.21.15.159:6825/905169792] boot 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: osdmap e807: 12 total, 12 up, 11 in 2024-04-15T18:46:34.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:33 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-15T18:46:34.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:34.466 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:34.891 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:46:34.891 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:46:34.891 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:46:34.891 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:46:35.018 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:35.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:35.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:35.159 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:35.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:35.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:35.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:35.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:34 smithi082 ceph-mon[44206]: pgmap v2098: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-15T18:46:35.175 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:34 smithi082 ceph-mon[44206]: osdmap e808: 12 total, 12 up, 11 in 2024-04-15T18:46:35.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:46:35.275779+0000 min lat: 0.00106913 max lat: 0.341675 avg lat: 0.0107916 2024-04-15T18:46:35.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:46:35.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 40 16 59296 59280 92.617 83.3125 0.00515423 0.0107916 2024-04-15T18:46:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:34 smithi159 ceph-mon[45336]: pgmap v2098: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-15T18:46:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:34 smithi159 ceph-mon[45336]: osdmap e808: 12 total, 12 up, 11 in 2024-04-15T18:46:35.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:35.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:35.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:35.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:34 smithi049 ceph-mon[40087]: pgmap v2098: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-04-15T18:46:35.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:34 smithi049 ceph-mon[40087]: osdmap e808: 12 total, 12 up, 11 in 2024-04-15T18:46:35.614 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:35.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:35.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:35.764 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:35.829 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:35.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:35.933 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:35.969 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:36.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:46:36.117 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:46:36.118 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:46:36.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 41 16 60210 60194 91.7512 57.125 0.03459 0.0108906 2024-04-15T18:46:36.317 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:46:36.318 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-15T18:46:36.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:46:36.322 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:36.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:36.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:36.489 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:36.494 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:36.532 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:36.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:36.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:36.675 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:36.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:46:36.991 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:46:37.185 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:37.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:37.249 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:37.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:36 smithi049 ceph-mon[40087]: pgmap v2101: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-04-15T18:46:37.260 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:37.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 42 16 61253 61237 91.1186 65.1875 0.0105664 0.0109697 2024-04-15T18:46:37.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:36 smithi082 ceph-mon[44206]: pgmap v2101: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-04-15T18:46:37.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:37.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:46:37.323 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:36 smithi159 ceph-mon[45336]: pgmap v2101: 17 pgs: 17 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-04-15T18:46:37.323 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:37.325 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:37.354 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:46:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:37.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:46:37.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:37.493 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:37.518 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:37.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:37.623 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:37.654 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:37.787 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:37.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:37.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:37.929 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:38.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:37 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:38.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:38.217 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:37 smithi082 ceph-mon[44206]: osdmap e809: 12 total, 12 up, 11 in 2024-04-15T18:46:38.218 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:37 smithi082 ceph-mon[44206]: osdmap e810: 12 total, 12 up, 11 in 2024-04-15T18:46:38.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 43 16 62422 62406 90.6986 73.0625 0.0115973 0.0110209 2024-04-15T18:46:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:37 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:37 smithi159 ceph-mon[45336]: osdmap e809: 12 total, 12 up, 11 in 2024-04-15T18:46:38.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:37 smithi159 ceph-mon[45336]: osdmap e810: 12 total, 12 up, 11 in 2024-04-15T18:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:37 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1541009180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:37 smithi049 ceph-mon[40087]: osdmap e809: 12 total, 12 up, 11 in 2024-04-15T18:46:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:37 smithi049 ceph-mon[40087]: osdmap e810: 12 total, 12 up, 11 in 2024-04-15T18:46:38.986 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:39.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:38 smithi049 ceph-mon[40087]: pgmap v2103: 17 pgs: 17 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:46:39.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 44 16 63744 63728 90.5145 82.625 0.0079772 0.0110419 2024-04-15T18:46:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:38 smithi082 ceph-mon[44206]: pgmap v2103: 17 pgs: 17 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:46:39.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:38 smithi159 ceph-mon[45336]: pgmap v2103: 17 pgs: 17 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:46:39.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:46:39.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:46:39.858 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:46:39.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:46:39.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:46:39.947 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:46:39.978 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:46:40.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:46:40.040 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:46:40.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:46:40.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:46:40.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:46:40.126 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:46:40.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:46:40.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:46:40.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:46:40.128 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:46:40.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:46:40.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2148569067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:40.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 45 16 65069 65053 90.3432 82.8125 0.00284543 0.0110588 2024-04-15T18:46:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2148569067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:40.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2148569067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:41.090 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:41.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:41.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:41.234 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:41.263 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 46 16 66169 66153 89.8737 68.75 0.00929466 0.0111212 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: pgmap v2105: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: pgmap v2105: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: pgmap v2105: 17 pgs: 17 active+clean; 3.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 962 op/s 2024-04-15T18:46:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:41.609 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:41.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:41.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:41.761 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:41.790 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:41.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:41.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.047 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:42.091 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:42.153 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:42.186 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.213 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:42.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 47 16 67444 67428 89.6568 79.6875 0.00324557 0.0111491 2024-04-15T18:46:42.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:42.351 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.382 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:42.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:42.537 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.622 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:42.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:42.778 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.814 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:42.920 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:42.932 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:46:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:46:42.932 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:46:42.933 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:46:42.933 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:46:42.951 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:42.974 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:42.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:43.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:43.115 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:43.141 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:43.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:43.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:43.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 48 16 68798 68782 89.5518 84.625 0.00341751 0.0111612 2024-04-15T18:46:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:43 smithi082 ceph-mon[44206]: pgmap v2106: 17 pgs: 17 active+clean; 3.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-15T18:46:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:43.285 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:43.311 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:43.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:43 smithi159 ceph-mon[45336]: pgmap v2106: 17 pgs: 17 active+clean; 3.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-15T18:46:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:43.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:43.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:43 smithi049 ceph-mon[40087]: pgmap v2106: 17 pgs: 17 active+clean; 3.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 62 MiB/s wr, 984 op/s 2024-04-15T18:46:43.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:43.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:43.455 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:46:43.580 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:46:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 811, 2024-04-15T18:46:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:46:43.580 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:46:43.007759+0000", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:46:33.707597+0000", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:42:37.738384+0000", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:46:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 31, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "jewel", 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:46:43.582 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:46:43.583 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:46:43.584 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:46:43.585 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:46:43.586 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:46:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:46:43.588 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-04-15T18:46:43.589 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:45:48.486814+0000", 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:46:43.590 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.591 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "794", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:46:43.592 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.593 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:46:43.594 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.89999997615814209, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-04-15T18:46:43.595 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.596 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 671, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:46:43.597 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.598 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.599 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.600 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:46:43.601 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.602 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:46:43.603 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:46:43.604 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.605 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:46:43.607 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.608 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 778, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:46:43.609 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.610 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.612 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:46:43.613 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:46:43.614 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 675, 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:46:43.616 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.617 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.618 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:46:43.619 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.620 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.621 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.622 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.623 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:46:43.625 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.626 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:46:43.627 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:46:43.628 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:46:43.629 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:46:43.630 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:46:43.631 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:46:43.632 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 676, 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:46:43.634 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:46:43.635 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:46:43.636 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.637 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.638 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.641 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:46:43.642 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.644 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.645 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.649 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.650 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.651 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.652 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.653 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:46:43.654 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.660 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 94, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 733, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 807, 2024-04-15T18:46:43.661 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 709, 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 734, 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.662 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.663 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.664 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.665 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/905169792", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/905169792", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/905169792", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/905169792", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:46:43.666 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:14.066612+0000", 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:24.999679+0000", 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.668 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:17.477088+0000", 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.669 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:45:54.968336+0000", 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:26.080822+0000", 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:25.278313+0000", 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.671 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:45:39.752918+0000", 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:06.071694+0000", 2024-04-15T18:46:43.672 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:45:55.888654+0000", 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.673 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:42.010318+0000", 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.674 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:24.786207+0000", 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:32.997786+0000", 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 733 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.675 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:46:43.676 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:46:43.677 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:46:43.678 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:46:43.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:46:44.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:44 smithi082 ceph-mon[44206]: osdmap e811: 12 total, 12 up, 11 in 2024-04-15T18:46:44.236 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1274894877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:46:44.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 49 15 69998 69983 89.256 75.0625 0.00725394 0.0111983 2024-04-15T18:46:44.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:46:44.317 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:44.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:44.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:44 smithi049 ceph-mon[40087]: osdmap e811: 12 total, 12 up, 11 in 2024-04-15T18:46:44.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1274894877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:46:44.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:44 smithi159 ceph-mon[45336]: osdmap e811: 12 total, 12 up, 11 in 2024-04-15T18:46:44.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1274894877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:46:44.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:44.457 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:44.563 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:44.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:44.665 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:44.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:44.699 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:44.723 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:44.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:44.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:44.866 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:45.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:45 smithi082 ceph-mon[44206]: pgmap v2108: 17 pgs: 17 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:46:45.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:45 smithi049 ceph-mon[40087]: pgmap v2108: 17 pgs: 17 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:46:45.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 50 16 71219 71203 88.9958 76.25 0.0115773 0.0112317 2024-04-15T18:46:45.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:45 smithi159 ceph-mon[45336]: pgmap v2108: 17 pgs: 17 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-04-15T18:46:46.155 INFO:tasks.thrashosds.thrasher:j is {'epoch': 811, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:46:43.007759+0000', 'last_up_change': '2024-04-15T18:46:33.707597+0000', 'last_in_change': '2024-04-15T18:42:37.738384+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 31, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-04-15T18:45:48.486814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '794', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.8999999761581421, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 671, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 778, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 675, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 791, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 676, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 94, 'last_clean_end': 733, 'up_from': 807, 'up_thru': 709, 'down_at': 734, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 905169792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 905169792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 905169792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 905169792}]}, 'public_addr': '172.21.15.159:6825/905169792', 'cluster_addr': '172.21.15.159:6827/905169792', 'heartbeat_back_addr': '172.21.15.159:6831/905169792', 'heartbeat_front_addr': '172.21.15.159:6829/905169792', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:14.066612+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:24.999679+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:17.477088+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:45:54.968336+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:26.080822+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:25.278313+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:45:39.752918+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:06.071694+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:45:55.888654+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:42.010318+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:24.786207+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-15T18:46:32.997786+0000', 'dead_epoch': 733}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:46:46.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:46:46.192 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:46:46.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 51 16 72394 72378 88.6906 73.4375 0.0145743 0.0112705 2024-04-15T18:46:46.265 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:46 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:46:46.385 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:46.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:46 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:46:46.456 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:46.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:46 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:46:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:46.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:46.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:46.603 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:46.639 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:46.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:46.666 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:46:46.666 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31782 7.1 7:82fc4c0d:::benchmark_data_smithi159_149998_object31781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.63932908200002, 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31042012400000002, 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.667 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31782 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.668 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036383+0000", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0600000000000001e-07 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036412+0000", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8798000000000001e-05 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036418+0000", 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7520000000000002e-06 2024-04-15T18:46:46.669 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.036421+0000", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1470000000000001e-06 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037561+0000", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011398739999999999 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037642+0000", 2024-04-15T18:46:46.670 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1481000000000001e-05 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.037728+0000", 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5646e-05 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038470+0000", 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000741729 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.671 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.038630+0000", 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016056800000000001 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346734+0000", 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30810352899999999 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346773+0000", 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8949999999999998e-05 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.672 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346803+0000", 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0343999999999999e-05 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31793 7.4 7:2fbec3a3:::benchmark_data_smithi159_149998_object31792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.62787540600004, 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.31189694899999998, 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.673 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31793 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.674 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047837+0000", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047839+0000", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0530000000000001e-06 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047884+0000", 2024-04-15T18:46:46.675 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5325999999999999e-05 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047911+0000", 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7457000000000002e-05 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.047932+0000", 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0888999999999999e-05 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.676 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048077+0000", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144361 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048189+0000", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112435 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048271+0000", 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1411999999999999e-05 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.677 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.048888+0000", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000617872 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.049960+0000", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001071935 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359672+0000", 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30971173899999999 2024-04-15T18:46:46.678 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359707+0000", 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4965000000000002e-05 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.359734+0000", 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6505000000000001e-05 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.679 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:31795 7.1 7:81918b20:::benchmark_data_smithi159_149998_object31794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e631)", 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.62444183299999, 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32741436800000001, 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 31795 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.680 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051270+0000", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051271+0000", 2024-04-15T18:46:46.681 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1900000000000001e-07 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051305+0000", 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3918000000000003e-05 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051315+0000", 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0376e-05 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.682 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051323+0000", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8329999999999994e-06 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051397+0000", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4245999999999994e-05 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051487+0000", 2024-04-15T18:46:46.683 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0140000000000001e-05 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.051573+0000", 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5922999999999997e-05 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052330+0000", 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075673500000000002 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.684 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.053094+0000", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076432699999999998 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378621+0000", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.32552697800000002 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378663+0000", 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1357999999999999e-05 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.685 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.378685+0000", 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1715000000000001e-05 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:31782 7.1 e631/621 ondisk, result = 0)", 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 575.62271670500002, 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29381337200000002, 2024-04-15T18:46:46.686 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052995+0000", 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.687 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052996+0000", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.41e-07 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.052997+0000", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.265e-06 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346692+0000", 2024-04-15T18:46:46.688 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.29369515000000002 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346695+0000", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3160000000000001e-06 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346724+0000", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8614000000000001e-05 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346730+0000", 2024-04-15T18:46:46.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2009999999999996e-06 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:11.346809+0000", 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8584999999999994e-05 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55371 7.1 7:830c46a9:::benchmark_data_smithi159_149998_object55370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.52500482899995, 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25659491200000001, 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.691 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55371 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150707+0000", 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150708+0000", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1909999999999999e-06 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150765+0000", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7090999999999999e-05 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150775+0000", 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0016e-05 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150781+0000", 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5960000000000003e-06 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150813+0000", 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2066000000000001e-05 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150906+0000", 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3035999999999995e-05 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.150998+0000", 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1699000000000004e-05 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152490+0000", 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014920739999999999 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.152917+0000", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00042668799999999999 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407183+0000", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25426654300000001 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407248+0000", 2024-04-15T18:46:46.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4925999999999995e-05 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.407302+0000", 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3986000000000001e-05 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55375 7.1 7:8b006e6f:::benchmark_data_smithi159_149998_object55374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:46:46.697 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.52206381799999, 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.28137763399999999, 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55375 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153648+0000", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153649+0000", 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8899999999999998e-07 2024-04-15T18:46:46.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153727+0000", 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8459000000000002e-05 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153742+0000", 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4202000000000001e-05 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153748+0000", 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3269999999999997e-06 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153791+0000", 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3341999999999999e-05 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.153937+0000", 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014536700000000001 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.154086+0000", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000149674 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.155698+0000", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016121659999999999 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.156469+0000", 2024-04-15T18:46:46.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00077094200000000005 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434910+0000", 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27844058500000002 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434974+0000", 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3758999999999997e-05 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435026+0000", 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2021999999999998e-05 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55403 7.1 7:8aca0ab0:::benchmark_data_smithi159_149998_object55402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.49923914299995, 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27940837899999998, 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.704 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55403 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176473+0000", 2024-04-15T18:46:46.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176474+0000", 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3319999999999999e-06 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176505+0000", 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0821000000000002e-05 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176519+0000", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3957999999999999e-05 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176527+0000", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.79e-06 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176587+0000", 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9973000000000003e-05 2024-04-15T18:46:46.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176743+0000", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000155814 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.176908+0000", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165198 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.178583+0000", 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001675073 2024-04-15T18:46:46.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179015+0000", 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043260999999999999 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455781+0000", 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27676578499999999 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455836+0000", 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4837999999999999e-05 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.455881+0000", 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5187e-05 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:55375 7.1 e639/621 ondisk, result = 0)", 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:46:46.710 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.49637572999995, 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25569712700000002, 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179336+0000", 2024-04-15T18:46:46.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179337+0000", 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8100000000000002e-07 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.179341+0000", 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2080000000000002e-06 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434841+0000", 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25549990900000003 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434848+0000", 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5019999999999998e-06 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434894+0000", 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6836999999999999e-05 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.434904+0000", 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3519999999999996e-06 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.435033+0000", 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012963800000000001 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.714 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55417 7.4 7:2aa6faee:::benchmark_data_smithi159_149998_object55416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.48537705399997, 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26001807999999998, 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55417 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190335+0000", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190336+0000", 2024-04-15T18:46:46.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3629999999999999e-06 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190388+0000", 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1354000000000003e-05 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190402+0000", 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4353e-05 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190409+0000", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9110000000000004e-06 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190462+0000", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3458999999999998e-05 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190560+0000", 2024-04-15T18:46:46.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7677999999999996e-05 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.190645+0000", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4790000000000006e-05 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192252+0000", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016068009999999999 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.192667+0000", 2024-04-15T18:46:46.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000415189 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450236+0000", 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25756942999999999 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450301+0000", 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5079000000000005e-05 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.720 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.450353+0000", 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1672999999999998e-05 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:55420 7.4 7:2ab922f1:::benchmark_data_smithi159_149998_object55419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e639)", 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 555.48099094899999, 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27355053699999998, 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.721 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 55420 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194721+0000", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194722+0000", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0699999999999996e-07 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194774+0000", 2024-04-15T18:46:46.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2434e-05 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194785+0000", 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0767e-05 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194793+0000", 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6389999999999992e-06 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194835+0000", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2561000000000001e-05 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.194931+0000", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5377999999999994e-05 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.195017+0000", 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6224000000000007e-05 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196339+0000", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013224020000000001 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.196756+0000", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00041645199999999999 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468144+0000", 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27138785100000001 2024-04-15T18:46:46.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468225+0000", 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1422999999999996e-05 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:37:31.468272+0000", 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6499000000000003e-05 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:124560 7.4 7:2a9d76e5:::benchmark_data_smithi159_149998_object28560:head [delete] snapc 0=[] ondisk+write+known_if_redirected e663)", 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 466.51377560899999, 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27331726099999998, 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 124560 2024-04-15T18:46:46.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161936+0000", 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161937+0000", 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5700000000000001e-07 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161941+0000", 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0520000000000003e-06 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161950+0000", 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8769999999999999e-06 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161958+0000", 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3780000000000003e-06 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.161994+0000", 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6823999999999998e-05 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162070+0000", 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5834999999999995e-05 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.162135+0000", 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4783000000000006e-05 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.165059+0000", 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029242880000000001 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.168016+0000", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029565440000000002 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435166+0000", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26715059699999999 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435204+0000", 2024-04-15T18:46:46.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7344e-05 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:00.435254+0000", 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9882000000000001e-05 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:143366 7.4 7:21b697b5:::benchmark_data_smithi159_149998_object47366:head [delete] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-04-15T18:46:46.734 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 439.76972977100002, 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53450922700000003, 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 143366 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905982+0000", 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905984+0000", 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9429999999999999e-06 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.905995+0000", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0349999999999999e-05 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906022+0000", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7248999999999999e-05 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906037+0000", 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5150000000000001e-05 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906143+0000", 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106461 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906269+0000", 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012599200000000001 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 0,1", 2024-04-15T18:46:46.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.906398+0000", 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012908 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.909755+0000", 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033568209999999999 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:26.910370+0000", 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061510700000000005 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440411+0000", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53004094499999999 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440446+0000", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4193000000000003e-05 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:27.440491+0000", 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5936000000000002e-05 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.740 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDScrubReserve(7.3,RELEASE,e:676,from:2,reservation_nonce:0)", 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 434.888055173, 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30058055299999997, 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787657+0000", 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8300000000000001e-07 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:31.787659+0000", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.886e-06 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088016+0000", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30035709100000002 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088030+0000", 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4496e-05 2024-04-15T18:46:46.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088171+0000", 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140505 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088209+0000", 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8426000000000001e-05 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:39:32.088237+0000", 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7966e-05 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26545.0:191990 7.1 7:816ba94e:::benchmark_data_smithi159_149998_object95990:head [delete] snapc 0=[] ondisk+write+known_if_redirected e700)", 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 370.773200304, 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.272141566, 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.745 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26545", 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 191990 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902512+0000", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902514+0000", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9269999999999999e-06 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902520+0000", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1419999999999998e-06 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902535+0000", 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5251e-05 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902543+0000", 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8790000000000003e-06 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902601+0000", 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8553e-05 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902725+0000", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012331500000000001 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,10", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.902841+0000", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011651500000000001 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907458+0000", 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0046170109999999999 2024-04-15T18:46:46.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907670+0000", 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000212017 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174581+0000", 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26691115799999998 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174620+0000", 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8198000000000003e-05 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174653+0000", 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3599999999999997e-05 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26545.0:191990 7.1 e700/621 ondisk, result = 0)", 2024-04-15T18:46:46.751 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 370.76846249599998, 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26740929600000002, 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907249+0000", 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907250+0000", 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3900000000000004e-07 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:35.907256+0000", 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5400000000000003e-06 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174508+0000", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26725248200000001 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174514+0000", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7400000000000001e-06 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174569+0000", 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5276999999999999e-05 2024-04-15T18:46:46.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174577+0000", 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8979999999999996e-06 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:40:36.174659+0000", 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1520000000000006e-05 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26791.0:164269 8.b e766/709 8:dfeda397:::benchmark_data_smithi159_162467_object61964:head v 766'10242, mlcod=766'10239)", 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 137.08754851800001, 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.39892428099999999, 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588164+0000", 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4000000000000001e-07 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588172+0000", 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6190000000000002e-06 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982886+0000", 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.39471447500000001 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982892+0000", 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0209999999999996e-06 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982983+0000", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0558999999999998e-05 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.983005+0000", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2504e-05 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.987059+0000", 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004054017 2024-04-15T18:46:46.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.987088+0000", 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8345999999999999e-05 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26791.0:184710 8.b e775/709 8:d50196f7:::benchmark_data_smithi159_162467_object82405:head v 775'11522, mlcod=775'11519)", 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 104.224884649, 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.314097931, 2024-04-15T18:46:46.760 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450828+0000", 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4099999999999998e-07 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450833+0000", 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6619999999999996e-06 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.762 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760596+0000", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.309763546 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760604+0000", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5959999999999998e-06 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760689+0000", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4753000000000007e-05 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760704+0000", 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5449000000000001e-05 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.764881+0000", 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041766659999999999 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.764925+0000", 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4617999999999999e-05 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:17995 9.c 9:36be814e:::benchmark_data_smithi159_173188_object17994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e797)", 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 40.827643526999999, 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21212373800000001, 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 17995 2024-04-15T18:46:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848070+0000", 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2500000000000001e-06 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848189+0000", 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119659 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848205+0000", 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6042999999999999e-05 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848214+0000", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8279999999999992e-06 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848281+0000", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7001000000000004e-05 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848399+0000", 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118068 2024-04-15T18:46:46.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848492+0000", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2875999999999996e-05 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.849504+0000", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010113139999999999 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.849694+0000", 2024-04-15T18:46:46.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019046499999999999 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060121+0000", 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.210427165 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060153+0000", 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2155e-05 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060192+0000", 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8914e-05 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:55185 9.4 e805/791 9:28d7fa2c:::benchmark_data_smithi159_173188_object55184:head v 805'3532, mlcod=805'3529)", 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:31.825539+0000", 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.850172854, 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.159565127, 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.771 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.825539+0000", 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.825539+0000", 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.825540+0000", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0800000000000003e-07 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.825635+0000", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5210999999999994e-05 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.982923+0000", 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15728751599999999 2024-04-15T18:46:46.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.982929+0000", 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5259999999999999e-06 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.983007+0000", 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7810000000000002e-05 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.983027+0000", 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9409999999999999e-05 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.985053+0000", 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0020266759999999998 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:31.985104+0000", 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1069999999999997e-05 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.775 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:65069 9.c 9:3a853d5d:::benchmark_data_smithi159_173188_object65068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e810)", 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:40.269339+0000", 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 6.4063731529999997, 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14569507600000001, 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:46:46.776 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 65069 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269339+0000", 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269339+0000", 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269340+0000", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.514e-06 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269382+0000", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2131e-05 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269396+0000", 2024-04-15T18:46:46.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3111e-05 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269401+0000", 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8370000000000002e-06 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269440+0000", 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8503e-05 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:46:46.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269541+0000", 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100956 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.269625+0000", 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4116999999999994e-05 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.271487+0000", 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00186206 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.271728+0000", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00024117400000000001 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.414969+0000", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.143240857 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.415006+0000", 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6687000000000003e-05 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:46:46.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:40.415034+0000", 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8129000000000001e-05 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:46:46.782 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:46:46.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:46:46.812 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:46.838 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:46:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:46.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:46:46.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:46.975 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:46.998 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:47.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:47.141 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:47.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:47 smithi159 ceph-mon[45336]: pgmap v2109: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:46:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:47 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:47 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:47.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:47 smithi159 ceph-mon[45336]: osdmap e812: 12 total, 12 up, 11 in 2024-04-15T18:46:47.218 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:47.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 52 16 73450 73434 88.2542 66 0.010108 0.0113233 2024-04-15T18:46:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:47 smithi082 ceph-mon[44206]: pgmap v2109: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:46:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:47 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:47 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:47 smithi082 ceph-mon[44206]: osdmap e812: 12 total, 12 up, 11 in 2024-04-15T18:46:47.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:47 smithi049 ceph-mon[40087]: pgmap v2109: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-04-15T18:46:47.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:47 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:46:47.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:47 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4122884828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:46:47.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:47 smithi049 ceph-mon[40087]: osdmap e812: 12 total, 12 up, 11 in 2024-04-15T18:46:47.324 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:47.370 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:47.407 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:47.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:46:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:47.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:47.560 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:47.582 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:47.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:47.731 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:47.777 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:47.824 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:47.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:47.914 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:47.936 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:47.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:48.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:48.082 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:48.113 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:48.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:48.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:48.261 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:48.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 53 16 74598 74582 87.9427 71.75 0.00678813 0.0113656 2024-04-15T18:46:48.300 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:48.368 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:48.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:48.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:48.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:48.587 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:48.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:48.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:48.721 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:48.745 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:48.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:48.850 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:48.878 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:48.908 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:48.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:49.015 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:49.050 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:49.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:49 smithi159 ceph-mon[45336]: pgmap v2111: 17 pgs: 17 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:46:49.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:46:49.155 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:49.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:49.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:49 smithi049 ceph-mon[40087]: pgmap v2111: 17 pgs: 17 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:46:49.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:49.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 54 15 75537 75522 87.402 58.75 0.013147 0.0114361 2024-04-15T18:46:49.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:49 smithi082 ceph-mon[44206]: pgmap v2111: 17 pgs: 17 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-04-15T18:46:49.320 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:49.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:46:49.407 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:46:49.442 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:46:49.477 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:46:49.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:46:49.546 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:46:49.582 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:46:49.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:46:49.654 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:46:49.693 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:46:49.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:46:49.765 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:46:49.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:46:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:46:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:46:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:46:49.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:46:49.989 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:46:49.989 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:46:50.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3884580790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:50.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 55 15 76768 76753 87.2116 76.9375 0.0668139 0.0114616 2024-04-15T18:46:50.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3884580790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.315 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:50.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3884580790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:50.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:50.453 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:50.479 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:50.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:50.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:50.622 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:50.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:46:50.874 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:50.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:50.981 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:51.017 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:51.042 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:51.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:51.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: pgmap v2112: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:46:51.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:51 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.180 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:51.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 56 16 77841 77825 86.8506 67 0.0169262 0.0115096 2024-04-15T18:46:51.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: pgmap v2112: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:46:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:51 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: pgmap v2112: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-04-15T18:46:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:51 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:46:51.740 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:51.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:51.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:51.890 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:51.934 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:51.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:52.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:46:52.080 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:52.145 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:52.145 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:46:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:52.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:46:52.250 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:46:52.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 57 16 78997 78981 86.5944 72.25 0.02567 0.0115431 2024-04-15T18:46:52.288 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:52.314 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:46:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:52.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:46:52.420 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:46:52.456 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:52.729 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:46:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:46:53.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 58 16 80278 80262 86.4817 80.0625 0.0120377 0.0115587 2024-04-15T18:46:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:53 smithi159 ceph-mon[45336]: pgmap v2113: 17 pgs: 17 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:46:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:53 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:46:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:53 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:46:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:53 smithi049 ceph-mon[40087]: pgmap v2113: 17 pgs: 17 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:46:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:53 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:46:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:53 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:46:53.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:53 smithi082 ceph-mon[44206]: pgmap v2113: 17 pgs: 17 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-04-15T18:46:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:53 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:46:53.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:53 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:46:53.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:46:54.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 59 16 81318 81302 86.1175 65 0.0121332 0.0116077 2024-04-15T18:46:54.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:54 smithi049 ceph-mon[40087]: from='client.17418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:46:54.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:54 smithi159 ceph-mon[45336]: from='client.17418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:46:54.503 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:54 smithi082 ceph-mon[44206]: from='client.17418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:46:54.513 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:46:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:54.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:46:54.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:54.651 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:54.683 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:54.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:54.788 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:46:54.829 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:54.855 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:46:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:54.861 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:46:54.905 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2113,"stamp":"2024-04-15T18:46:51.701580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4900258336,"num_objects":74765,"num_object_clones":0,"num_object_copies":224295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74765,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":74944,"num_write_kb":4787770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74902,"ondisk_log_size":74902,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":9,"kb":1031196672,"kb_used":20689408,"kb_used_data":14286008,"kb_used_omap":285,"kb_used_meta":6402722,"kb_avail":1010507264,"statfs":{"total":1055945392128,"available":1034759438336,"internally_reserved":0,"allocated":14628872192,"data_stored":14625901488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292651,"internal_metadata":6556387541},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612958208,"num_objects":9353,"num_object_clones":0,"num_object_copies":28059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":598592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.684324"},"pg_stats":[{"pgid":"9.f","version":"812'4599","reported_seq":4651,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.996697+0000","last_change":"2024-04-15T18:45:49.842084+0000","last_active":"2024-04-15T18:46:47.996697+0000","last_peered":"2024-04-15T18:46:47.996697+0000","last_clean":"2024-04-15T18:46:47.996697+0000","last_became_active":"2024-04-15T18:45:49.841805+0000","last_became_peered":"2024-04-15T18:45:49.841805+0000","last_unstale":"2024-04-15T18:46:47.996697+0000","last_undegraded":"2024-04-15T18:46:47.996697+0000","last_fullsized":"2024-04-15T18:46:47.996697+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:02.123042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":13797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":294336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"812'4981","reported_seq":5033,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:51.518401+0000","last_change":"2024-04-15T18:45:49.845239+0000","last_active":"2024-04-15T18:46:51.518401+0000","last_peered":"2024-04-15T18:46:51.518401+0000","last_clean":"2024-04-15T18:46:51.518401+0000","last_became_active":"2024-04-15T18:45:49.844972+0000","last_became_peered":"2024-04-15T18:45:49.844972+0000","last_unstale":"2024-04-15T18:46:51.518401+0000","last_undegraded":"2024-04-15T18:46:51.518401+0000","last_fullsized":"2024-04-15T18:46:51.518401+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:54.605537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326434816,"num_objects":4981,"num_object_clones":0,"num_object_copies":14943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":318784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"812'4930","reported_seq":4982,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:51.510864+0000","last_change":"2024-04-15T18:45:49.845316+0000","last_active":"2024-04-15T18:46:51.510864+0000","last_peered":"2024-04-15T18:46:51.510864+0000","last_clean":"2024-04-15T18:46:51.510864+0000","last_became_active":"2024-04-15T18:45:49.844992+0000","last_became_peered":"2024-04-15T18:45:49.844992+0000","last_unstale":"2024-04-15T18:46:51.510864+0000","last_undegraded":"2024-04-15T18:46:51.510864+0000","last_fullsized":"2024-04-15T18:46:51.510864+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:06.847559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323092480,"num_objects":4930,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":315520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"812'4577","reported_seq":4628,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.690214+0000","last_change":"2024-04-15T18:45:49.844200+0000","last_active":"2024-04-15T18:46:47.690214+0000","last_peered":"2024-04-15T18:46:47.690214+0000","last_clean":"2024-04-15T18:46:47.690214+0000","last_became_active":"2024-04-15T18:45:49.843864+0000","last_became_peered":"2024-04-15T18:45:49.843864+0000","last_unstale":"2024-04-15T18:46:47.690214+0000","last_undegraded":"2024-04-15T18:46:47.690214+0000","last_fullsized":"2024-04-15T18:46:47.690214+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:59.581177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":13731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":292928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"812'4779","reported_seq":4831,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:50.377285+0000","last_change":"2024-04-15T18:45:49.842321+0000","last_active":"2024-04-15T18:46:50.377285+0000","last_peered":"2024-04-15T18:46:50.377285+0000","last_clean":"2024-04-15T18:46:50.377285+0000","last_became_active":"2024-04-15T18:45:49.841997+0000","last_became_peered":"2024-04-15T18:45:49.841997+0000","last_unstale":"2024-04-15T18:46:50.377285+0000","last_undegraded":"2024-04-15T18:46:50.377285+0000","last_fullsized":"2024-04-15T18:46:50.377285+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:11.192058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313196544,"num_objects":4779,"num_object_clones":0,"num_object_copies":14337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4779,"num_write_kb":305856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"812'4691","reported_seq":4742,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:48.019001+0000","last_change":"2024-04-15T18:45:49.844191+0000","last_active":"2024-04-15T18:46:48.019001+0000","last_peered":"2024-04-15T18:46:48.019001+0000","last_clean":"2024-04-15T18:46:48.019001+0000","last_became_active":"2024-04-15T18:45:49.843914+0000","last_became_peered":"2024-04-15T18:45:49.843914+0000","last_unstale":"2024-04-15T18:46:48.019001+0000","last_undegraded":"2024-04-15T18:46:48.019001+0000","last_fullsized":"2024-04-15T18:46:48.019001+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:50.910136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307429376,"num_objects":4691,"num_object_clones":0,"num_object_copies":14073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4691,"num_write_kb":300224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"812'4678","reported_seq":4730,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:48.576739+0000","last_change":"2024-04-15T18:45:49.843139+0000","last_active":"2024-04-15T18:46:48.576739+0000","last_peered":"2024-04-15T18:46:48.576739+0000","last_clean":"2024-04-15T18:46:48.576739+0000","last_became_active":"2024-04-15T18:45:49.842804+0000","last_became_peered":"2024-04-15T18:45:49.842804+0000","last_unstale":"2024-04-15T18:46:48.576739+0000","last_undegraded":"2024-04-15T18:46:48.576739+0000","last_fullsized":"2024-04-15T18:46:48.576739+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:09.507228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306577408,"num_objects":4678,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":299392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"812'4690","reported_seq":4742,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:48.576661+0000","last_change":"2024-04-15T18:45:49.841390+0000","last_active":"2024-04-15T18:46:48.576661+0000","last_peered":"2024-04-15T18:46:48.576661+0000","last_clean":"2024-04-15T18:46:48.576661+0000","last_became_active":"2024-04-15T18:45:49.841033+0000","last_became_peered":"2024-04-15T18:45:49.841033+0000","last_unstale":"2024-04-15T18:46:48.576661+0000","last_undegraded":"2024-04-15T18:46:48.576661+0000","last_fullsized":"2024-04-15T18:46:48.576661+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:18.640406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363840,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":300160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"812'4652","reported_seq":4704,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:48.026286+0000","last_change":"2024-04-15T18:45:49.841903+0000","last_active":"2024-04-15T18:46:48.026286+0000","last_peered":"2024-04-15T18:46:48.026286+0000","last_clean":"2024-04-15T18:46:48.026286+0000","last_became_active":"2024-04-15T18:45:49.841652+0000","last_became_peered":"2024-04-15T18:45:49.841652+0000","last_unstale":"2024-04-15T18:46:48.026286+0000","last_undegraded":"2024-04-15T18:46:48.026286+0000","last_fullsized":"2024-04-15T18:46:48.026286+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:11.874030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"812'4604","reported_seq":4656,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:48.033021+0000","last_change":"2024-04-15T18:45:49.844236+0000","last_active":"2024-04-15T18:46:48.033021+0000","last_peered":"2024-04-15T18:46:48.033021+0000","last_clean":"2024-04-15T18:46:48.033021+0000","last_became_active":"2024-04-15T18:45:49.843864+0000","last_became_peered":"2024-04-15T18:45:49.843864+0000","last_unstale":"2024-04-15T18:46:48.033021+0000","last_undegraded":"2024-04-15T18:46:48.033021+0000","last_fullsized":"2024-04-15T18:46:48.033021+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:02.579206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4604,"num_write_kb":294656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"812'4633","reported_seq":4685,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.304348+0000","last_change":"2024-04-15T18:45:49.845048+0000","last_active":"2024-04-15T18:46:47.304348+0000","last_peered":"2024-04-15T18:46:47.304348+0000","last_clean":"2024-04-15T18:46:47.304348+0000","last_became_active":"2024-04-15T18:45:49.844745+0000","last_became_peered":"2024-04-15T18:45:49.844745+0000","last_unstale":"2024-04-15T18:46:47.304348+0000","last_undegraded":"2024-04-15T18:46:47.304348+0000","last_fullsized":"2024-04-15T18:46:47.304348+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:52.144489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303628288,"num_objects":4633,"num_object_clones":0,"num_object_copies":13899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4633,"num_write_kb":296512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"812'4568","reported_seq":4620,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.567119+0000","last_change":"2024-04-15T18:45:49.842669+0000","last_active":"2024-04-15T18:46:47.567119+0000","last_peered":"2024-04-15T18:46:47.567119+0000","last_clean":"2024-04-15T18:46:47.567119+0000","last_became_active":"2024-04-15T18:45:49.842017+0000","last_became_peered":"2024-04-15T18:45:49.842017+0000","last_unstale":"2024-04-15T18:46:47.567119+0000","last_undegraded":"2024-04-15T18:46:47.567119+0000","last_fullsized":"2024-04-15T18:46:47.567119+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:00.210145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299368448,"num_objects":4568,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4568,"num_write_kb":292352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"812'4556","reported_seq":4607,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.261967+0000","last_change":"2024-04-15T18:45:49.836537+0000","last_active":"2024-04-15T18:46:47.261967+0000","last_peered":"2024-04-15T18:46:47.261967+0000","last_clean":"2024-04-15T18:46:47.261967+0000","last_became_active":"2024-04-15T18:45:49.836251+0000","last_became_peered":"2024-04-15T18:45:49.836251+0000","last_unstale":"2024-04-15T18:46:47.261967+0000","last_undegraded":"2024-04-15T18:46:47.261967+0000","last_fullsized":"2024-04-15T18:46:47.261967+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:11.688900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298582016,"num_objects":4556,"num_object_clones":0,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4556,"num_write_kb":291584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"812'4599","reported_seq":4651,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.632996+0000","last_change":"2024-04-15T18:45:50.181052+0000","last_active":"2024-04-15T18:46:47.632996+0000","last_peered":"2024-04-15T18:46:47.632996+0000","last_clean":"2024-04-15T18:46:47.632996+0000","last_became_active":"2024-04-15T18:45:50.180630+0000","last_became_peered":"2024-04-15T18:45:50.180630+0000","last_unstale":"2024-04-15T18:46:47.632996+0000","last_undegraded":"2024-04-15T18:46:47.632996+0000","last_fullsized":"2024-04-15T18:46:47.632996+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:59.837149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":13797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":294336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"812'4658","reported_seq":4710,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.329437+0000","last_change":"2024-04-15T18:45:50.180656+0000","last_active":"2024-04-15T18:46:47.329437+0000","last_peered":"2024-04-15T18:46:47.329437+0000","last_clean":"2024-04-15T18:46:47.329437+0000","last_became_active":"2024-04-15T18:45:50.180276+0000","last_became_peered":"2024-04-15T18:45:50.180276+0000","last_unstale":"2024-04-15T18:46:47.329437+0000","last_undegraded":"2024-04-15T18:46:47.329437+0000","last_fullsized":"2024-04-15T18:46:47.329437+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:12.761180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305266688,"num_objects":4658,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4658,"num_write_kb":298112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"812'4568","reported_seq":4620,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:47.606808+0000","last_change":"2024-04-15T18:45:49.842941+0000","last_active":"2024-04-15T18:46:47.606808+0000","last_peered":"2024-04-15T18:46:47.606808+0000","last_clean":"2024-04-15T18:46:47.606808+0000","last_became_active":"2024-04-15T18:45:49.842612+0000","last_became_peered":"2024-04-15T18:45:49.842612+0000","last_unstale":"2024-04-15T18:46:47.606808+0000","last_undegraded":"2024-04-15T18:46:47.606808+0000","last_fullsized":"2024-04-15T18:46:47.606808+0000","mapping_epoch":791,"log_start":"0'0","ondisk_log_start":"0'0","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:47:09.260162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299368448,"num_objects":4568,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4568,"num_write_kb":292352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":1996,"reported_epoch":812,"state":"active+clean","last_fresh":"2024-04-15T18:46:46.037649+0000","last_change":"2024-04-15T18:45:19.433556+0000","last_active":"2024-04-15T18:46:46.037649+0000","last_peered":"2024-04-15T18:46:46.037649+0000","last_clean":"2024-04-15T18:46:46.037649+0000","last_became_active":"2024-04-15T18:42:41.780248+0000","last_became_peered":"2024-04-15T18:42:41.780248+0000","last_unstale":"2024-04-15T18:46:46.037649+0000","last_undegraded":"2024-04-15T18:46:46.037649+0000","last_fullsized":"2024-04-15T18:46:46.037649+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:45:19.433459+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:45:19.433459+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:46:48.930296+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4899667968,"num_objects":74763,"num_object_clones":0,"num_object_copies":224289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74763,"num_write_kb":4784832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14726922240,"data_stored":14726922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74763,"ondisk_log_size":74763,"up":48,"acting":48,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4231424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":3466038607876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187914,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088424,"kb_used_data":1483872,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":91656728,"statfs":{"total":95995035648,"available":93856489472,"internally_reserved":0,"allocated":1519484928,"data_stored":1519215832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":619051456},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514960,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2463216,"kb_used_data":1742056,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91281936,"statfs":{"total":95995035648,"available":93472702464,"internally_reserved":0,"allocated":1783865344,"data_stored":1783592696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874706,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354984,"kb_used_data":922592,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92390168,"statfs":{"total":95995035648,"available":94607532032,"internally_reserved":0,"allocated":944734208,"data_stored":944465112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169048,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2445360,"kb_used_data":1864424,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91299792,"statfs":{"total":95995035648,"available":93490987008,"internally_reserved":0,"allocated":1909170176,"data_stored":1908897528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363894,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826800,"kb_used_data":619432,"kb_used_omap":111,"kb_used_meta":207248,"kb_avail":92918352,"statfs":{"total":95995035648,"available":95148392448,"internally_reserved":0,"allocated":634298368,"data_stored":634025728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":212222619},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575353,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3390388,"kb_used_data":2633056,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":90354764,"statfs":{"total":95995035648,"available":92523278336,"internally_reserved":0,"allocated":2696249344,"data_stored":2695980248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681652,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063500,"statfs":{"total":95995035648,"available":95297024000,"internally_reserved":0,"allocated":688128,"data_stored":419032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194125,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524212,"kb_used_data":672,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220940,"statfs":{"total":95995035648,"available":95458242560,"internally_reserved":0,"allocated":688128,"data_stored":419032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609256,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2483944,"kb_used_data":1763552,"kb_used_omap":44,"kb_used_meta":720339,"kb_avail":91261208,"statfs":{"total":95995035648,"available":93451476992,"internally_reserved":0,"allocated":1805877248,"data_stored":1805608152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":737627690},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240948,"kb_used_data":587744,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92504204,"statfs":{"total":95995035648,"available":94724304896,"internally_reserved":0,"allocated":601849856,"data_stored":601580760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703557,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3189480,"kb_used_data":2667936,"kb_used_omap":4,"kb_used_meta":521531,"kb_avail":90555672,"statfs":{"total":95995035648,"available":92729008128,"internally_reserved":0,"allocated":2731966464,"data_stored":2731697368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":534048449},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2721644544,"data_stored":2721644544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1813381120,"data_stored":1813381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":605814784,"data_stored":605814784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2732457984,"data_stored":2732457984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":639696896,"data_stored":639696896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1924399104,"data_stored":1924399104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":952041472,"data_stored":952041472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1799225344,"data_stored":1799225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1538260992,"data_stored":1538260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:46:54.907 INFO:tasks.thrashosds.thrasher:Setting 9.d to [5, 1, 8, 11, 9, 10] 2024-04-15T18:46:54.907 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '9.d', '5', '1', '8', '11', '9', '10'] 2024-04-15T18:46:54.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pg-upmap-items 9.d 5 1 8 11 9 10 2024-04-15T18:46:54.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:46:55.003 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:55.114 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:46:55.117 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:55 smithi082 ceph-mon[44206]: pgmap v2114: 17 pgs: 17 active+clean; 4.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:46:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:55.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:46:55.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:55 smithi049 ceph-mon[40087]: pgmap v2114: 17 pgs: 17 active+clean; 4.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:46:55.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:46:55.255 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:55.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:46:55.277523+0000 min lat: 0.00106913 max lat: 0.341675 avg lat: 0.0116294 2024-04-15T18:46:55.264 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:46:55.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 60 16 82521 82505 85.9353 75.1875 0.0310525 0.0116294 2024-04-15T18:46:55.275 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:55.283 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:46:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:55.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:46:55.328 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:46:55.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:55.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:55 smithi159 ceph-mon[45336]: pgmap v2114: 17 pgs: 17 active+clean; 4.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-04-15T18:46:55.430 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:55.457 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:46:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:55.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:46:55.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:46:55.581 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:46:55.581 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:46:55.624 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:55.809 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:46:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:55.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:46:55.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:46:56.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:56 smithi049 ceph-mon[40087]: 9.a scrub starts 2024-04-15T18:46:56.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:56 smithi049 ceph-mon[40087]: 9.e scrub starts 2024-04-15T18:46:56.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:46:56.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:46:56.085 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:56 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:46:56.087 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:56.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 61 16 83363 83347 85.3891 52.625 0.00984519 0.011704 2024-04-15T18:46:56.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:56 smithi159 ceph-mon[45336]: 9.a scrub starts 2024-04-15T18:46:56.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:56 smithi159 ceph-mon[45336]: 9.e scrub starts 2024-04-15T18:46:56.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:46:56.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:46:56.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:56 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:46:56.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:56 smithi082 ceph-mon[44206]: 9.a scrub starts 2024-04-15T18:46:56.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:56 smithi082 ceph-mon[44206]: 9.e scrub starts 2024-04-15T18:46:56.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:46:56.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:46:56.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:56 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:46:57.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 62 16 84376 84360 85.033 63.3125 0.0135956 0.011755 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: 9.4 scrub starts 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: 9.a scrub ok 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: 9.4 scrub starts 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: 9.a scrub ok 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:57.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:46:57.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: 9.4 scrub starts 2024-04-15T18:46:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: 9.a scrub ok 2024-04-15T18:46:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:46:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:46:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:46:57.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:46:58.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:46:58.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 63 16 85317 85301 84.6167 58.8125 0.0211471 0.0118134 2024-04-15T18:46:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:58 smithi082 ceph-mon[44206]: 9.4 scrub ok 2024-04-15T18:46:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:58 smithi082 ceph-mon[44206]: 9.e scrub ok 2024-04-15T18:46:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:58 smithi159 ceph-mon[45336]: 9.4 scrub ok 2024-04-15T18:46:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:58 smithi159 ceph-mon[45336]: 9.e scrub ok 2024-04-15T18:46:58.430 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:46:58.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:58 smithi049 ceph-mon[40087]: 9.4 scrub ok 2024-04-15T18:46:58.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:58 smithi049 ceph-mon[40087]: 9.e scrub ok 2024-04-15T18:46:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:46:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:58.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:46:59.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:46:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:46:59.185 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:46:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:59.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:46:59.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 64 16 86395 86379 84.3472 67.375 0.00357222 0.0118512 2024-04-15T18:46:59.294 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:59.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:59 smithi159 ceph-mon[45336]: pgmap v2116: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:46:59.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:46:59.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:46:59.326 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:59 smithi159 ceph-mon[45336]: osdmap e813: 12 total, 12 up, 11 in 2024-04-15T18:46:59.327 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:46:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:59.327 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:59.351 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:59.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:59.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:59 smithi082 ceph-mon[44206]: pgmap v2116: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:46:59.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:46:59.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:46:59.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:59 smithi082 ceph-mon[44206]: osdmap e813: 12 total, 12 up, 11 in 2024-04-15T18:46:59.413 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:46:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:59.455 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:46:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:59 smithi049 ceph-mon[40087]: pgmap v2116: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-04-15T18:46:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:46:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:46:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:59 smithi049 ceph-mon[40087]: osdmap e813: 12 total, 12 up, 11 in 2024-04-15T18:46:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:46:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:46:59.489 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:46:59.513 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:46:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:46:59.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:46:59.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:46:59.663 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:00.117 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:47:00.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:47:00.152 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:47:00.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:47:00.226 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:47:00.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 65 16 87587 87571 84.1956 74.5 0.0051456 0.0118669 2024-04-15T18:47:00.325 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:00.327 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:00 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:00.327 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:00 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:47:00.327 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:00.327 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:00 smithi082 ceph-mon[44206]: osdmap e814: 12 total, 12 up, 11 in 2024-04-15T18:47:00.327 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/550360844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:00.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:00.369 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:47:00.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:00 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:00 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:47:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:00 smithi159 ceph-mon[45336]: osdmap e814: 12 total, 12 up, 11 in 2024-04-15T18:47:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/550360844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:00.428 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:47:00.428 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:47:00.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:00.455 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:47:00.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:00 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:00.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:00 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:47:00.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3026538393' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:00.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:00 smithi049 ceph-mon[40087]: osdmap e814: 12 total, 12 up, 11 in 2024-04-15T18:47:00.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/550360844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:00.494 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:00.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:47:00.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:47:00.548 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:00.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:00.572 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:47:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:47:00.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:47:00.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:47:00.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:00.691 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:00.713 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:00.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:00.817 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:00.878 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:01.082 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:01.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:01.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:01.229 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:132423 8.d 8:b478b48f:::benchmark_data_smithi159_162467_object30118:head [delete] snapc 0=[] ondisk+write+known_if_redirected e754)", 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:36.813707+0000", 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 204.45088829299999, 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19150832300000001, 2024-04-15T18:47:01.260 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 132423 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813707+0000", 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.261 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813707+0000", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813709+0000", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3519999999999999e-06 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813715+0000", 2024-04-15T18:47:01.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6939999999999998e-06 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813730+0000", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.453e-05 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813737+0000", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.559e-06 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813789+0000", 2024-04-15T18:47:01.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1523000000000001e-05 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813886+0000", 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7621000000000006e-05 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.813975+0000", 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8745999999999994e-05 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.817729+0000", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003753915 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.826526+0000", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0087964770000000005 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.002956+0000", 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.176430156 2024-04-15T18:47:01.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.003016+0000", 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0077000000000001e-05 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.005216+0000", 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021996730000000001 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:132423 8.d e754/743 ondisk, result = 0)", 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:36.818721+0000", 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 204.445874608, 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18650923699999999, 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.818721+0000", 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.818721+0000", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.818722+0000", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.47e-07 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.818727+0000", 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.604e-06 2024-04-15T18:47:01.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.999972+0000", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18124441799999999 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:36.999976+0000", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7790000000000002e-06 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.002940+0000", 2024-04-15T18:47:01.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029644889999999998 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.002949+0000", 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8380000000000008e-06 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:37.005230+0000", 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022811620000000002 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:138088 8.d 8:b4d05b61:::benchmark_data_smithi159_162467_object35783:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.88554229799999, 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20772660300000001, 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138088 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.271 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379056+0000", 2024-04-15T18:47:01.272 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6919999999999998e-06 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379064+0000", 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0190000000000006e-06 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379082+0000", 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8031000000000001e-05 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.273 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379093+0000", 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0688e-05 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384072+0000", 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049789359999999998 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384222+0000", 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015064500000000001 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.274 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384334+0000", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111799 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.390640+0000", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006306056 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.398802+0000", 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081622269999999993 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.275 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585518+0000", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18671573499999999 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585599+0000", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0866999999999997e-05 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.586780+0000", 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001180908 2024-04-15T18:47:01.276 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:138088 8.d e757/743 ondisk, result = 0)", 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:46.392678+0000", 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.87191749799999, 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19410935300000001, 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.277 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.392678+0000", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.392678+0000", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.392679+0000", 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8899999999999998e-07 2024-04-15T18:47:01.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.392687+0000", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2430000000000005e-06 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.581464+0000", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18877722499999999 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.581470+0000", 2024-04-15T18:47:01.279 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1769999999999997e-06 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585498+0000", 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040281249999999996 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585509+0000", 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1715e-05 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.280 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.586787+0000", 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001277879 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPushReply(8.d 757/743 [PushReplyOp(8:bf238105:::benchmark_data_smithi159_162467_object46772:head)])", 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:46.396352+0000", 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.86824329000001, 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18505421, 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-04-15T18:47:01.281 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.396352+0000", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.396352+0000", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.396352+0000", 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8099999999999999e-07 2024-04-15T18:47:01.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.396356+0000", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.569e-06 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.579085+0000", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18272898900000001 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.579094+0000", 2024-04-15T18:47:01.283 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3519999999999996e-06 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.579149+0000", 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5189000000000003e-05 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.581406+0000", 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022569299999999999 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:168061 8.d 8:b73dc4a0:::benchmark_data_smithi159_162467_object65756:head [delete] snapc 0=[] ondisk+write+known_if_redirected e767)", 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:36.014510+0000", 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 145.25008552099999, 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.200849633, 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 168061 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.285 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014510+0000", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014510+0000", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014512+0000", 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2869999999999999e-06 2024-04-15T18:47:01.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014525+0000", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3152999999999999e-05 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014558+0000", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2675999999999997e-05 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014575+0000", 2024-04-15T18:47:01.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7113e-05 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014691+0000", 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115338 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014899+0000", 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00020853500000000001 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.014982+0000", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2740999999999999e-05 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.018919+0000", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0039373819999999997 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.019266+0000", 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00034717100000000001 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215255+0000", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.195988366 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215309+0000", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.414e-05 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215360+0000", 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0730999999999999e-05 2024-04-15T18:47:01.290 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:168061 8.d e767/766 ondisk, result = 0)", 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:36.018182+0000", 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 145.246413666, 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.197184475, 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.291 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.018182+0000", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.018182+0000", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.018183+0000", 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.111e-06 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.018188+0000", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9860000000000002e-06 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.214307+0000", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19611909199999999 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.214316+0000", 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7180000000000002e-06 2024-04-15T18:47:01.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215239+0000", 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00092332999999999998 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215248+0000", 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8089999999999999e-06 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:36.215366+0000", 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118429 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:186366 8.d 8:b5235b6d:::benchmark_data_smithi159_162467_object84061:head [delete] snapc 0=[] ondisk+write+known_if_redirected e776)", 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:45:04.993532+0000", 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.271063595, 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20154651300000001, 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.295 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 186366 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993532+0000", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993532+0000", 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993534+0000", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8500000000000001e-06 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993543+0000", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8449999999999992e-06 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993562+0000", 2024-04-15T18:47:01.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8947e-05 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993571+0000", 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.628e-06 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993630+0000", 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8743000000000003e-05 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993728+0000", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7905000000000005e-05 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.993803+0000", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5135999999999993e-05 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997581+0000", 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037783449999999998 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997674+0000", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3000999999999999e-05 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.194966+0000", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19729211099999999 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.195025+0000", 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8114e-05 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.195078+0000", 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3888000000000003e-05 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:186366 8.d e776/766 ondisk, result = 0)", 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:45:04.997861+0000", 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.266734492, 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19722389700000001, 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.301 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997861+0000", 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997861+0000", 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997862+0000", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6599999999999996e-07 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:04.997866+0000", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7049999999999998e-06 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.194881+0000", 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19701491900000001 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.194888+0000", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7100000000000001e-06 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.194950+0000", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2076999999999995e-05 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.194959+0000", 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2720000000000003e-06 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:05.195085+0000", 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012564799999999999 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:7962 9.7 e797/791 9:ea3d6191:::benchmark_data_smithi159_173188_object7961:head v 797'518, mlcod=797'514)", 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:00.041956+0000", 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.222639057000002, 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18805058399999999, 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.305 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.041956+0000", 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.041956+0000", 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.041957+0000", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3600000000000003e-07 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.042015+0000", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8053999999999997e-05 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.229283+0000", 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18726752199999999 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.229288+0000", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1869999999999996e-06 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.229374+0000", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5790000000000004e-05 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.229388+0000", 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4243000000000001e-05 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.229975+0000", 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058743399999999996 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.230007+0000", 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1717999999999997e-05 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.309 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:7967 9.e 9:747b10a8:::benchmark_data_smithi159_173188_object7966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e797)", 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:00.046784+0000", 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.217811681000001, 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.199061508, 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 7967 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046784+0000", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046784+0000", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046785+0000", 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2799999999999995e-07 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046806+0000", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1682999999999999e-05 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046816+0000", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0175e-05 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046822+0000", 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3489999999999999e-06 2024-04-15T18:47:01.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046873+0000", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1622999999999997e-05 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.046999+0000", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000125673 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.047139+0000", 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000140276 2024-04-15T18:47:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.047676+0000", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00053684299999999996 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.047980+0000", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00030416599999999997 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245754+0000", 2024-04-15T18:47:01.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19777346400000001 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245801+0000", 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7589000000000001e-05 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245845+0000", 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3939000000000001e-05 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:7967 9.e e797/791 ondisk, result = 0)", 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:00.048216+0000", 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 61.216379883999998, 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.19763689500000001, 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.048216+0000", 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.048216+0000", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.048216+0000", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3700000000000001e-07 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.048219+0000", 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7769999999999998e-06 2024-04-15T18:47:01.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245677+0000", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.197458519 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245689+0000", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2033e-05 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245739+0000", 2024-04-15T18:47:01.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9799999999999998e-05 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245748+0000", 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8920000000000006e-06 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.245852+0000", 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104437 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:25901 9.d 9:bb4f7edf:::benchmark_data_smithi159_173188_object25900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.173067629000002, 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25750107, 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 25901 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.320 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091529+0000", 2024-04-15T18:47:01.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0559999999999999e-06 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091583+0000", 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3999000000000003e-05 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091593+0000", 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0366e-05 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091599+0000", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7119999999999997e-06 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091645+0000", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6227000000000003e-05 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091737+0000", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1941999999999995e-05 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:47:01.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091825+0000", 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7906999999999997e-05 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.092734+0000", 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090859999999999997 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.095365+0000", 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002631235 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.348923+0000", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25355768299999998 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.348983+0000", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0720000000000001e-05 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.349029+0000", 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5623e-05 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.325 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:25903 9.d 9:be043def:::benchmark_data_smithi159_173188_object25902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 50.170968354000003, 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26125103199999999, 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.326 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 25903 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093628+0000", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2099999999999996e-07 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093682+0000", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.427e-05 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093692+0000", 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5959999999999993e-06 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093697+0000", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9439999999999996e-06 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093735+0000", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7926000000000002e-05 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093829+0000", 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4678000000000004e-05 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093929+0000", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100207 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.094599+0000", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066987899999999996 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.344374+0000", 2024-04-15T18:47:01.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24977512199999999 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354775+0000", 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010400510999999999 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354834+0000", 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9039000000000002e-05 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354878+0000", 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4138999999999999e-05 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:47135 9.8 e802/791 9:1510fdf2:::benchmark_data_smithi159_173188_object47134:head v 802'2916, mlcod=802'2913)", 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 35.368381221, 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.239493337, 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.332 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896215+0000", 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6799999999999999e-07 2024-04-15T18:47:01.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896371+0000", 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000155895 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.133944+0000", 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23757313799999999 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.133951+0000", 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0979999999999996e-06 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.134004+0000", 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2614000000000001e-05 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.134024+0000", 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0179000000000001e-05 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.135662+0000", 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016382370000000001 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.135708+0000", 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5408000000000003e-05 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:49727 9.e 9:7fa008f6:::benchmark_data_smithi159_173188_object49726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:47:01.336 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.348314932000001, 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22725898, 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 49727 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:47:01.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916282+0000", 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1659999999999999e-06 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916343+0000", 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1583999999999999e-05 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916354+0000", 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0944999999999999e-05 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916362+0000", 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8809999999999996e-06 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916421+0000", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8409000000000003e-05 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916569+0000", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014799100000000001 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916718+0000", 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014955599999999999 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918468+0000", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017494419999999999 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918763+0000", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029521300000000001 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143447+0000", 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22468469399999999 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143509+0000", 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2036e-05 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143540+0000", 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0063e-05 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:49732 9.d 9:bce94f75:::benchmark_data_smithi159_173188_object49731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.918852+0000", 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.345743079999998, 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.187974688, 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 49732 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918852+0000", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918852+0000", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918853+0000", 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8999999999999995e-07 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918936+0000", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2942999999999995e-05 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918945+0000", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5550000000000003e-06 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918950+0000", 2024-04-15T18:47:01.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.135e-06 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918994+0000", 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4076000000000003e-05 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919112+0000", 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011824000000000001 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:47:01.346 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919212+0000", 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100196 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920650+0000", 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001437773 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920909+0000", 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00025922599999999999 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106721+0000", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18581119500000001 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106780+0000", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8981000000000003e-05 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106827+0000", 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7478000000000002e-05 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.348 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:49733 9.e 9:79a4aebf:::benchmark_data_smithi159_173188_object49732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.345632537999997, 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23282433, 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 49733 2024-04-15T18:47:01.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1900000000000003e-07 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919017+0000", 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3041000000000003e-05 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919022+0000", 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0490000000000003e-06 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919025+0000", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6849999999999999e-06 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919060+0000", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5142999999999999e-05 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919193+0000", 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013278899999999999 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919330+0000", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013709900000000001 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920633+0000", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013026629999999999 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920748+0000", 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115322 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151675+0000", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23092637099999999 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151739+0000", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4160999999999998e-05 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151787+0000", 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8488000000000001e-05 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.354 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:49732 9.d e805/791 ondisk, result = 0)", 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.920983+0000", 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.343612284999999, 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18584994199999999, 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920983+0000", 2024-04-15T18:47:01.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920983+0000", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920984+0000", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.08e-07 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920986+0000", 2024-04-15T18:47:01.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1519999999999999e-06 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106579+0000", 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.185593177 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106591+0000", 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2563e-05 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106690+0000", 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8067999999999997e-05 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106703+0000", 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3743e-05 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.106833+0000", 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012983100000000001 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.358 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:49727 9.e e805/791 ondisk, result = 0)", 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 33.343477814000003, 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22242645699999999, 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:47:01.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4400000000000001e-07 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921120+0000", 2024-04-15T18:47:01.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.796e-06 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143401+0000", 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.222281222 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143405+0000", 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9160000000000003e-06 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:47:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143439+0000", 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4301000000000002e-05 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143444+0000", 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7380000000000002e-06 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143544+0000", 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010024 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.362 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.363 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:47:01.363 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:47:01.363 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:47:01.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: pgmap v2119: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:01.365 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:01.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:01.366 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 66 16 88653 88637 83.9293 66.625 0.0304937 0.0119102 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: pgmap v2119: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:47:01.499 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: pgmap v2119: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:47:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:01.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:47:01.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:01.904 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:02.036 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:02.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:02.142 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:02.174 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:02.199 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:02.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:02.239 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:02.239 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:02 smithi082 ceph-mon[44206]: 9.1 scrub starts 2024-04-15T18:47:02.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 67 16 89552 89536 83.5152 56.1875 0.00477883 0.0119689 2024-04-15T18:47:02.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:02.342 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:02.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:02.365 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:02 smithi049 ceph-mon[40087]: 9.1 scrub starts 2024-04-15T18:47:02.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:02.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:02 smithi159 ceph-mon[45336]: 9.1 scrub starts 2024-04-15T18:47:02.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:47:02.899 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:02.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:02.908 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:47:03.005 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:03.041 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:03.066 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:03.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:03.119 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:03.170 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:03.215 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:03 smithi159 ceph-mon[45336]: pgmap v2120: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-15T18:47:03.215 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:03 smithi159 ceph-mon[45336]: 9.1 scrub ok 2024-04-15T18:47:03.217 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:03.258 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:03.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 68 16 90596 90580 83.2465 65.25 0.0142303 0.0120074 2024-04-15T18:47:03.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:03.267 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:03.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:03 smithi082 ceph-mon[44206]: pgmap v2120: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-15T18:47:03.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:03 smithi082 ceph-mon[44206]: 9.1 scrub ok 2024-04-15T18:47:03.367 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:03.402 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:03.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:03 smithi049 ceph-mon[40087]: pgmap v2120: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 43 MiB/s wr, 690 op/s 2024-04-15T18:47:03.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:03 smithi049 ceph-mon[40087]: 9.1 scrub ok 2024-04-15T18:47:03.759 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:03.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:03.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:47:03.902 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:03.927 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:03.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:04.031 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:04.069 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:04.083 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2892942554' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:04.100 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.d pg_upmap_items mapping to [5->1,8->11,9->10] 2024-04-15T18:47:04.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 69 16 91644 91628 82.9892 65.5 0.00795984 0.0120449 2024-04-15T18:47:04.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2892942554' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:04.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:04 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:04.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2892942554' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:04.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:04 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:04.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:47:04.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:04 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]: dispatch 2024-04-15T18:47:05.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 70 16 92642 92626 82.6946 62.375 0.00336678 0.0120822 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: pgmap v2121: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: 9.3 scrub starts 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: 9.f scrub starts 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]': finished 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: osdmap e815: 12 total, 12 up, 11 in 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:05.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:05 smithi159 ceph-mon[45336]: 9.c scrub starts 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: pgmap v2121: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: 9.3 scrub starts 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: 9.f scrub starts 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]': finished 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: osdmap e815: 12 total, 12 up, 11 in 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:05 smithi049 ceph-mon[40087]: 9.c scrub starts 2024-04-15T18:47:05.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: pgmap v2121: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-04-15T18:47:05.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: 9.3 scrub starts 2024-04-15T18:47:05.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: 9.f scrub starts 2024-04-15T18:47:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.d", "id": [5, 1, 8, 11, 9, 10]}]': finished 2024-04-15T18:47:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: osdmap e815: 12 total, 12 up, 11 in 2024-04-15T18:47:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:05.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:05 smithi082 ceph-mon[44206]: 9.c scrub starts 2024-04-15T18:47:06.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:06.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:06.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:06.173 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:06.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:06 smithi049 ceph-mon[40087]: osdmap e816: 12 total, 12 up, 11 in 2024-04-15T18:47:06.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:06 smithi049 ceph-mon[40087]: 9.3 scrub ok 2024-04-15T18:47:06.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:06 smithi049 ceph-mon[40087]: 9.f scrub ok 2024-04-15T18:47:06.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 71 16 93298 93282 82.1073 41 0.00872342 0.0121742 2024-04-15T18:47:06.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:06 smithi159 ceph-mon[45336]: osdmap e816: 12 total, 12 up, 11 in 2024-04-15T18:47:06.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:06 smithi159 ceph-mon[45336]: 9.3 scrub ok 2024-04-15T18:47:06.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:06 smithi159 ceph-mon[45336]: 9.f scrub ok 2024-04-15T18:47:06.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:06 smithi082 ceph-mon[44206]: osdmap e816: 12 total, 12 up, 11 in 2024-04-15T18:47:06.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:06 smithi082 ceph-mon[44206]: 9.3 scrub ok 2024-04-15T18:47:06.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:06 smithi082 ceph-mon[44206]: 9.f scrub ok 2024-04-15T18:47:06.886 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:07.121 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:47:07.265 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 72 15 94040 94025 81.6118 46.4375 0.00474628 0.0122479 2024-04-15T18:47:07.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:07 smithi159 ceph-mon[45336]: pgmap v2124: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:47:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:07 smithi159 ceph-mon[45336]: 9.c scrub ok 2024-04-15T18:47:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:47:07.432 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:07.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:07.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:07 smithi049 ceph-mon[40087]: pgmap v2124: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:47:07.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:07 smithi049 ceph-mon[40087]: 9.c scrub ok 2024-04-15T18:47:07.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:47:07.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:07 smithi082 ceph-mon[44206]: pgmap v2124: 17 pgs: 17 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 53 MiB/s wr, 841 op/s 2024-04-15T18:47:07.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:07 smithi082 ceph-mon[44206]: 9.c scrub ok 2024-04-15T18:47:07.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:47:07.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:07.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:07.619 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:07.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:07.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:07.758 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:08.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:08 smithi049 ceph-mon[40087]: 9.d scrub starts 2024-04-15T18:47:08.115 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:08.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:08.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:08.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 73 16 94825 94809 81.1651 49 0.0127901 0.0123164 2024-04-15T18:47:08.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:08 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:08.277 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:47:08.278 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:08 smithi049 ceph-mon[40087]: osdmap e817: 12 total, 12 up, 11 in 2024-04-15T18:47:08.282 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:08.361 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:08.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:08.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:08 smithi159 ceph-mon[45336]: 9.d scrub starts 2024-04-15T18:47:08.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:08 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:08.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:47:08.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:08 smithi159 ceph-mon[45336]: osdmap e817: 12 total, 12 up, 11 in 2024-04-15T18:47:08.470 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:08.512 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:08.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:08 smithi082 ceph-mon[44206]: 9.d scrub starts 2024-04-15T18:47:08.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:08 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:08.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/738515842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:47:08.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:08 smithi082 ceph-mon[44206]: osdmap e817: 12 total, 12 up, 11 in 2024-04-15T18:47:08.615 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:08.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:08.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:08.775 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:09.033 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:09.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:09.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:09.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:09 smithi049 ceph-mon[40087]: pgmap v2126: 17 pgs: 17 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:47:09.185 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:09.216 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:09.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:09.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 74 16 95966 95950 81.0318 71.3125 0.00402832 0.0123359 2024-04-15T18:47:09.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:09.361 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:09.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:09 smithi159 ceph-mon[45336]: pgmap v2126: 17 pgs: 17 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:47:09.474 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:09.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:09.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:09 smithi082 ceph-mon[44206]: pgmap v2126: 17 pgs: 17 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-04-15T18:47:09.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:47:09.566 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:47:09.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:09.597 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:47:09.613 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:09.619 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:47:09.619 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:47:09.619 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:47:09.619 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:47:09.636 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:47:09.640 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:09.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:09.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:47:09.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:47:09.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:47:09.747 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:09.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:47:09.781 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:09.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:47:09.815 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:47:09.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:47:09.877 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:47:09.880 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1286813973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.126 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.127 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1286813973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 75 16 96918 96902 80.7447 59.5 0.00815318 0.0123798 2024-04-15T18:47:10.437 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:10.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1286813973' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:10.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:10.577 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:10.882 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:47:10.936 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:10.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:10.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:47:11.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:11.078 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:11.111 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:11.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:11.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:11.253 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:11.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 76 16 97976 97960 80.5522 66.125 0.0152971 0.0124081 2024-04-15T18:47:11.282 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:11.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:11.289 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:11.298 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:11.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:11.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: pgmap v2127: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 46 MiB/s wr, 732 op/s 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:11 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: pgmap v2127: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 46 MiB/s wr, 732 op/s 2024-04-15T18:47:11.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:11 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.422 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: pgmap v2127: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 46 MiB/s wr, 732 op/s 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:11 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:11.552 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:11.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:11.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:11.694 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:11.731 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:11.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:11.842 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:11.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6] out_osds: [11] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:47:11.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:47:11.865 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-04-15T18:47:11.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd in 11 2024-04-15T18:47:11.879 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:12.132 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:12.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:12.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:12.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 77 16 99097 99081 80.4159 70.0625 0.0134274 0.0124304 2024-04-15T18:47:12.280 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:12.543 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:12.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:12.657 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:12.701 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:12.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:47:12.857 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:12.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:12.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:13.138 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:13.140 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:13 smithi159 ceph-mon[45336]: pgmap v2128: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 38 MiB/s wr, 613 op/s 2024-04-15T18:47:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:13.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:13.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:13 smithi049 ceph-mon[40087]: pgmap v2128: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 38 MiB/s wr, 613 op/s 2024-04-15T18:47:13.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:13.254 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:47:13.255 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:47:13.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 78 15 100139 100124 80.2206 65.1875 0.00720706 0.0124609 2024-04-15T18:47:13.284 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:13.316 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:13.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:13.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:13.471 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:13.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:13 smithi082 ceph-mon[44206]: pgmap v2128: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 38 MiB/s wr, 613 op/s 2024-04-15T18:47:13.552 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:13.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:13.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:13.690 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:13.721 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:13.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:13.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:47:13.869 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:13.891 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:13.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:13.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:14.039 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:14.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:14.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:14.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 79 16 101178 101162 80.0263 64.875 0.0137426 0.0124881 2024-04-15T18:47:14.287 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:14.318 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:14.319 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:14.675 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:47:14.739 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:14.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:14.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:14.887 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:14.920 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:14.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:15.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:15.056 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:15.063 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:15.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:15 smithi082 ceph-mon[44206]: pgmap v2129: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-15T18:47:15.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/813441425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.161 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:15 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.164 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:47:15.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:2024-04-15T18:47:15.279209+0000 min lat: 0.00106913 max lat: 0.341675 avg lat: 0.0125432 2024-04-15T18:47:15.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-15T18:47:15.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 80 16 102024 102008 79.6869 52.875 0.00703664 0.0125432 2024-04-15T18:47:15.274 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:15.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:15 smithi049 ceph-mon[40087]: pgmap v2129: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-15T18:47:15.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/813441425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:15 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:15.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:15.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:15.498 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:15.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:15.601 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:15 smithi159 ceph-mon[45336]: pgmap v2129: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-04-15T18:47:15.601 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/813441425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.601 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:15 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:15.604 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:15.642 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:15.853 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:15.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:15.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:15.996 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:16.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 81 16 103129 103113 79.5556 69.0625 0.00682473 0.0125642 2024-04-15T18:47:16.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:16 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:16.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:16 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:16.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:16 smithi049 ceph-mon[40087]: osdmap e818: 12 total, 12 up, 11 in 2024-04-15T18:47:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:16 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:16 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:16 smithi082 ceph-mon[44206]: osdmap e818: 12 total, 12 up, 11 in 2024-04-15T18:47:16.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:16 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:16.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:16 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:16.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:16 smithi159 ceph-mon[45336]: osdmap e818: 12 total, 12 up, 11 in 2024-04-15T18:47:16.959 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:16.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:17.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:17.108 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:17.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 82 16 104134 104118 79.3514 62.8125 0.0155039 0.0125981 2024-04-15T18:47:17.289 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:17 smithi159 ceph-mon[45336]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:47:17.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:17 smithi049 ceph-mon[40087]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:47:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:17 smithi082 ceph-mon[44206]: pgmap v2131: 17 pgs: 17 active+clean; 6.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-04-15T18:47:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:17.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:17.671 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:17.719 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:17.751 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:17.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:17.856 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:17.894 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:17.917 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:17.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:18.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:18.069 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:18.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:47:18.211 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:47:18.211 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:47:18.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 83 16 105199 105183 79.1973 66.5625 0.0203862 0.0126219 2024-04-15T18:47:18.837 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:18.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:18.942 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:18.955 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:47:18.980 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:18.988 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:47:19.007 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:19.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:19.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:47:19.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:47:19.067 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:47:19.103 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:47:19.110 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:19.125 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:19.140 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:19.155 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:47:19.167 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:19.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:19.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-mon[40087]: pgmap v2132: 17 pgs: 17 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-04-15T18:47:19.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:19.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:19.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2895663277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:19.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:19.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:47:19.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:47:19.264 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:47:19.266 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 84 16 106220 106204 79.0141 63.8125 0.0111987 0.0126508 2024-04-15T18:47:19.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:19 smithi082 ceph-mon[44206]: pgmap v2132: 17 pgs: 17 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-04-15T18:47:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:19 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:19 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:19.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2895663277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:19.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:19.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:47:19.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:19 smithi159 ceph-mon[45336]: pgmap v2132: 17 pgs: 17 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 976 op/s 2024-04-15T18:47:19.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:19 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:19.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:19 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:19.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2895663277' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:19.307 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:19.310 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:19.335 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:47:19.338 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:19.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:19.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:47:19.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:47:19.412 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:19.445 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:47:19.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:19 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:47:19.282+0000 7f8015e7e700 -1 mon.a@0(leader).osd e819 definitely_dead 0 2024-04-15T18:47:19.476 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:19.509 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:19.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:19.613 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:47:19.662 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:19.693 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:19.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:19.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:19.859 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:20.202 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.11. 2024-04-15T18:47:20.267 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 85 16 107196 107180 78.802 61 0.00432628 0.0126773 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osdmap e819: 12 total, 12 up, 11 in 2024-04-15T18:47:20.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/961361291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:20 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osdmap e819: 12 total, 12 up, 11 in 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/961361291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:20 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osdmap e819: 12 total, 12 up, 11 in 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/961361291' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-04-15T18:47:20.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:20.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:20 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:21.267 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 86 16 107223 107207 77.9053 1.6875 0.0052097 0.0127183 2024-04-15T18:47:21.324 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:21.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:21.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:21.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:21 smithi049 ceph-mon[40087]: pgmap v2134: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:47:21.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:21 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-15T18:47:21.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:21 smithi049 ceph-mon[40087]: osdmap e820: 12 total, 12 up, 12 in 2024-04-15T18:47:21.465 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:21 smithi159 ceph-mon[45336]: pgmap v2134: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:47:21.465 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:21 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-15T18:47:21.465 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:21 smithi159 ceph-mon[45336]: osdmap e820: 12 total, 12 up, 12 in 2024-04-15T18:47:21.470 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:21.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:21 smithi082 ceph-mon[44206]: pgmap v2134: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-04-15T18:47:21.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:21 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-04-15T18:47:21.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:21 smithi082 ceph-mon[44206]: osdmap e820: 12 total, 12 up, 12 in 2024-04-15T18:47:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:21.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:21.886 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:47:21.886 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:47:21.886 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:47:21.886 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:47:21.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:47:21.956 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:22.267 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 87 16 107223 107207 77.0099 0 - 0.0127183 2024-04-15T18:47:22.458 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:22.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:22.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:22 smithi049 ceph-mon[40087]: osdmap e821: 12 total, 12 up, 12 in 2024-04-15T18:47:22.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:22 smithi049 ceph-mon[40087]: osdmap e822: 12 total, 12 up, 12 in 2024-04-15T18:47:22.482 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-04-15T18:47:22.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:22.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:22.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:22 smithi082 ceph-mon[44206]: osdmap e821: 12 total, 12 up, 12 in 2024-04-15T18:47:22.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:22 smithi082 ceph-mon[44206]: osdmap e822: 12 total, 12 up, 12 in 2024-04-15T18:47:22.540 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:22.540 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:22.540 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:22 smithi159 ceph-mon[45336]: osdmap e821: 12 total, 12 up, 12 in 2024-04-15T18:47:22.541 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:22 smithi159 ceph-mon[45336]: osdmap e822: 12 total, 12 up, 12 in 2024-04-15T18:47:22.566 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:47:22.806 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:47:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:22.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:47:22.911 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:22.980 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:23.011 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:23.032 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:23.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:23.141 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:23.171 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:23.192 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:23.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:23.267 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 88 16 107482 107466 76.3187 8.09375 0.00468468 0.0130967 2024-04-15T18:47:23.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:23.308 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:23 smithi159 ceph-mon[45336]: pgmap v2137: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-15T18:47:23.331 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:23.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:23 smithi049 ceph-mon[40087]: pgmap v2137: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-15T18:47:23.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:23 smithi082 ceph-mon[44206]: pgmap v2137: 17 pgs: 17 active+clean; 6.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 843 op/s 2024-04-15T18:47:23.533 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:23.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:23.638 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:23.668 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:23.688 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:23.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:23.794 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:23.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:47:23.852 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:23.929 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:23.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:24.034 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:24.064 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:24.086 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:24.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:24.194 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:24.259 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:24.266 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:24.268 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 89 16 108118 108102 75.9065 39.75 0.0172017 0.0131683 2024-04-15T18:47:24.379 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:24.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:24.485 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:24.557 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:24.769 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:47:24.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:24.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:24.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:24 smithi159 ceph-mon[45336]: pgmap v2139: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.5 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 13459/321873 objects degraded (4.181%); 13350/321873 objects misplaced (4.148%); 226 KiB/s, 3 objects/s recovering 2024-04-15T18:47:24.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:24 smithi159 ceph-mon[45336]: osdmap e823: 12 total, 12 up, 12 in 2024-04-15T18:47:24.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4033572136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:24.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:24 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:24.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:24 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 13459/321873 objects degraded (4.181%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:24 smithi049 ceph-mon[40087]: pgmap v2139: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.5 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 13459/321873 objects degraded (4.181%); 13350/321873 objects misplaced (4.148%); 226 KiB/s, 3 objects/s recovering 2024-04-15T18:47:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:24 smithi049 ceph-mon[40087]: osdmap e823: 12 total, 12 up, 12 in 2024-04-15T18:47:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4033572136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:24 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:24 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 13459/321873 objects degraded (4.181%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:24.976 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:25.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:24 smithi082 ceph-mon[44206]: pgmap v2139: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.5 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 39 MiB/s wr, 623 op/s; 13459/321873 objects degraded (4.181%); 13350/321873 objects misplaced (4.148%); 226 KiB/s, 3 objects/s recovering 2024-04-15T18:47:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:24 smithi082 ceph-mon[44206]: osdmap e823: 12 total, 12 up, 12 in 2024-04-15T18:47:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4033572136' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:24 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:47:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:24 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 13459/321873 objects degraded (4.181%), 2 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:25.196 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:47:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:47:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:47:25.197 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:47:25.197 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:47:25.268 INFO:tasks.radosbench.radosbench.2.smithi159.stdout: 90 13 108630 108617 75.4206 32.1875 0.0180289 0.0132527 2024-04-15T18:47:25.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:25 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:25.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:25 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:47:25.882 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:25 smithi159 ceph-mon[45336]: osdmap e824: 12 total, 12 up, 12 in 2024-04-15T18:47:25.921 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:47:25.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:25 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:25.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:25 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:47:25.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:25 smithi049 ceph-mon[40087]: osdmap e824: 12 total, 12 up, 12 in 2024-04-15T18:47:26.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:25 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:47:26.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:25 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:47:26.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:25 smithi082 ceph-mon[44206]: osdmap e824: 12 total, 12 up, 12 in 2024-04-15T18:47:26.243 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total time run: 90.8001 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Total writes made: 108630 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Write size: 65536 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Object size: 65536 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Bandwidth (MB/sec): 74.7727 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Bandwidth: 23.1039 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max bandwidth (MB/sec): 133.5 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min bandwidth (MB/sec): 0 2024-04-15T18:47:26.269 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average IOPS: 1196 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev IOPS: 369.679 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max IOPS: 2136 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min IOPS: 0 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Average Latency(s): 0.0132808 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Stddev Latency(s): 0.0336185 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Max latency(s): 2.69945 2024-04-15T18:47:26.270 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Min latency(s): 0.00106913 2024-04-15T18:47:26.280 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Cleaning up (deleting benchmark objects) 2024-04-15T18:47:26.731 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:26.757 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:26 smithi159 ceph-mon[45336]: pgmap v2142: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 13459/322104 objects degraded (4.178%); 13350/322104 objects misplaced (4.145%); 227 KiB/s, 3 objects/s recovering 2024-04-15T18:47:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:26.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:26.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:26.897 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:26.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:26 smithi049 ceph-mon[40087]: pgmap v2142: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 13459/322104 objects degraded (4.178%); 13350/322104 objects misplaced (4.145%); 227 KiB/s, 3 objects/s recovering 2024-04-15T18:47:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:26 smithi082 ceph-mon[44206]: pgmap v2142: 17 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 activating+remapped, 1 remapped+peering, 12 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 568 op/s; 13459/322104 objects degraded (4.178%); 13350/322104 objects misplaced (4.145%); 227 KiB/s, 3 objects/s recovering 2024-04-15T18:47:27.483 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:47:27.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:47:27.484 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-15T18:47:27.484 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:47:27.484 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:47:27.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-04-15T18:47:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:27 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13459/322104 objects degraded (4.178%), 2 pgs degraded) 2024-04-15T18:47:28.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:27 smithi159 ceph-mon[45336]: osdmap e825: 12 total, 12 up, 12 in 2024-04-15T18:47:28.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:27 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13459/322104 objects degraded (4.178%), 2 pgs degraded) 2024-04-15T18:47:28.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:27 smithi049 ceph-mon[40087]: osdmap e825: 12 total, 12 up, 12 in 2024-04-15T18:47:28.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:27 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13459/322104 objects degraded (4.178%), 2 pgs degraded) 2024-04-15T18:47:28.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:27 smithi082 ceph-mon[44206]: osdmap e825: 12 total, 12 up, 12 in 2024-04-15T18:47:28.351 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:28.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:28.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:28.438 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:28.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:28.492 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:28.701 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:28.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:28.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:47:28.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:28.854 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:28.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:28.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:28.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:28 smithi049 ceph-mon[40087]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 35 MiB/s wr, 634 op/s; 26431/323193 objects misplaced (8.178%); 4.2 MiB/s, 65 objects/s recovering 2024-04-15T18:47:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:28 smithi159 ceph-mon[45336]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 35 MiB/s wr, 634 op/s; 26431/323193 objects misplaced (8.178%); 4.2 MiB/s, 65 objects/s recovering 2024-04-15T18:47:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:28 smithi082 ceph-mon[44206]: pgmap v2143: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 35 MiB/s wr, 634 op/s; 26431/323193 objects misplaced (8.178%); 4.2 MiB/s, 65 objects/s recovering 2024-04-15T18:47:31.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:30 smithi159 ceph-mon[45336]: pgmap v2145: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 14 MiB/s wr, 375 op/s; 26309/323160 objects misplaced (8.141%); 5.3 MiB/s, 83 objects/s recovering 2024-04-15T18:47:31.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:30 smithi049 ceph-mon[40087]: pgmap v2145: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 14 MiB/s wr, 375 op/s; 26309/323160 objects misplaced (8.141%); 5.3 MiB/s, 83 objects/s recovering 2024-04-15T18:47:31.217 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:31.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:31.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:30 smithi082 ceph-mon[44206]: pgmap v2145: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 14 MiB/s wr, 375 op/s; 26309/323160 objects misplaced (8.141%); 5.3 MiB/s, 83 objects/s recovering 2024-04-15T18:47:31.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:47:31.361 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:31.389 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:47:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:31.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:47:31.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:47:31.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:47:31.494 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:31.505 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:47:31.529 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:31.531 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:47:31.554 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:31.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:31.558 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:47:31.615 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:47:31.615 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:47:31.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:47:31.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:31.667 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:47:31.702 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:31.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:47:31.748 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:47:31.787 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:47:31.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:47:31.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:47:31.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:47:31.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:47:31.836 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:31.836 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:31.836 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: osdmap e826: 12 total, 12 up, 12 in 2024-04-15T18:47:31.836 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2008792643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:31.837 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:31.837 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: osdmap e826: 12 total, 12 up, 12 in 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2008792643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:32.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:32.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:32.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: osdmap e826: 12 total, 12 up, 12 in 2024-04-15T18:47:32.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2008792643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:32.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:32.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:32.780 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:32.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:32.885 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:32.941 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:32.971 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:32.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:33.075 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:33.110 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.110 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: pgmap v2147: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 12 MiB/s wr, 348 op/s; 26309/322665 objects misplaced (8.154%); 4.6 MiB/s, 72 objects/s recovering 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.111 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.112 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:33.147 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:33.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: pgmap v2147: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 12 MiB/s wr, 348 op/s; 26309/322665 objects misplaced (8.154%); 4.6 MiB/s, 72 objects/s recovering 2024-04-15T18:47:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: pgmap v2147: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 B/s rd, 12 MiB/s wr, 348 op/s; 26309/322665 objects misplaced (8.154%); 4.6 MiB/s, 72 objects/s recovering 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:33.286 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:33.495 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:47:33.495 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:47:34.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:33 smithi049 ceph-mon[40087]: Health check failed: Degraded data redundancy: 55/313848 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:47:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:33 smithi082 ceph-mon[44206]: Health check failed: Degraded data redundancy: 55/313848 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:47:34.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:33 smithi159 ceph-mon[45336]: Health check failed: Degraded data redundancy: 55/313848 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-04-15T18:47:34.419 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:34.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:34.532 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:34.603 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:34.852 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:34.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:34.896 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:47:34.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:35.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:34 smithi159 ceph-mon[45336]: pgmap v2148: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.9 MiB/s wr, 660 op/s; 55/313848 objects degraded (0.018%); 25735/313848 objects misplaced (8.200%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:47:35.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:34 smithi159 ceph-mon[45336]: 9.b scrub starts 2024-04-15T18:47:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:34 smithi049 ceph-mon[40087]: pgmap v2148: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.9 MiB/s wr, 660 op/s; 55/313848 objects degraded (0.018%); 25735/313848 objects misplaced (8.200%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:47:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:34 smithi049 ceph-mon[40087]: 9.b scrub starts 2024-04-15T18:47:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:34 smithi082 ceph-mon[44206]: pgmap v2148: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.9 MiB/s wr, 660 op/s; 55/313848 objects degraded (0.018%); 25735/313848 objects misplaced (8.200%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:47:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:34 smithi082 ceph-mon[44206]: 9.b scrub starts 2024-04-15T18:47:35.335 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:35.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:47:35.863 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:35.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:35.969 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:47:36.000 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:36.770 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:47:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:36.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:47:36.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:36.969 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:47:36.969 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:47:36.969 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:47:36.993 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:36 smithi159 ceph-mon[45336]: pgmap v2149: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 527 op/s; 55/313302 objects degraded (0.018%); 25735/313302 objects misplaced (8.214%); 5.0 MiB/s, 80 objects/s recovering 2024-04-15T18:47:36.993 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:36 smithi159 ceph-mon[45336]: 9.b scrub ok 2024-04-15T18:47:37.092 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:37.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:37.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:37.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:36 smithi049 ceph-mon[40087]: pgmap v2149: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 527 op/s; 55/313302 objects degraded (0.018%); 25735/313302 objects misplaced (8.214%); 5.0 MiB/s, 80 objects/s recovering 2024-04-15T18:47:37.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:36 smithi049 ceph-mon[40087]: 9.b scrub ok 2024-04-15T18:47:37.221 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:36 smithi082 ceph-mon[44206]: pgmap v2149: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 527 op/s; 55/313302 objects degraded (0.018%); 25735/313302 objects misplaced (8.214%); 5.0 MiB/s, 80 objects/s recovering 2024-04-15T18:47:37.221 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:36 smithi082 ceph-mon[44206]: 9.b scrub ok 2024-04-15T18:47:37.224 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:37.248 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:37.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:37.352 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:37.403 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:38.086 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:38.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:38.192 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:38.249 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:38.382 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:38.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:38.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:38.520 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:38.632 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:38.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:38.739 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:38.789 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:38.919 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:38 smithi159 ceph-mon[45336]: pgmap v2150: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 518 op/s; 99/310068 objects degraded (0.032%); 25398/310068 objects misplaced (8.191%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:47:39.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:38 smithi049 ceph-mon[40087]: pgmap v2150: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 518 op/s; 99/310068 objects degraded (0.032%); 25398/310068 objects misplaced (8.191%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:47:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:38 smithi082 ceph-mon[44206]: pgmap v2150: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 518 op/s; 99/310068 objects degraded (0.032%); 25398/310068 objects misplaced (8.191%); 5.9 MiB/s, 93 objects/s recovering 2024-04-15T18:47:39.817 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:47:39.817 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:47:39.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2998540879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:39.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:39 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:39.933 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:47:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2998540879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:39 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:40.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2998540879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:40.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:39 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:47:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: pgmap v2151: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 221/305358 objects degraded (0.072%); 25042/305358 objects misplaced (8.201%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 221/305358 objects degraded (0.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:41.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: osdmap e827: 12 total, 12 up, 12 in 2024-04-15T18:47:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:41.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:41.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: pgmap v2151: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 221/305358 objects degraded (0.072%); 25042/305358 objects misplaced (8.201%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:41.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 221/305358 objects degraded (0.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:41.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: osdmap e827: 12 total, 12 up, 12 in 2024-04-15T18:47:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:41.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: pgmap v2151: 17 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 221/305358 objects degraded (0.072%); 25042/305358 objects misplaced (8.201%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 221/305358 objects degraded (0.072%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: osdmap e827: 12 total, 12 up, 12 in 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:41.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:47:41.519 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:47:41.551 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:47:41.584 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:47:41.622 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:47:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:47:41.686 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:47:41.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:47:41.753 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:47:41.788 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:47:41.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:47:41.864 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:47:41.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:47:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:47:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:47:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:47:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:47:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:47:41.959 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:41.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:41.974 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:47:42.069 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:42.156 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:41 smithi049 ceph-mon[40087]: osdmap e828: 12 total, 12 up, 12 in 2024-04-15T18:47:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:41 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3752327539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:41 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.233 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:42.242 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:41 smithi159 ceph-mon[45336]: osdmap e828: 12 total, 12 up, 12 in 2024-04-15T18:47:42.242 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:41 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3752327539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:42.242 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.242 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:41 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:42.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:41 smithi082 ceph-mon[44206]: osdmap e828: 12 total, 12 up, 12 in 2024-04-15T18:47:42.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:41 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3752327539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:47:42.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:41 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.343 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:42.373 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:42.398 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:42.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:42.481 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:42.503 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:42.531 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:42.559 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:42.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:42.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:42.691 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:42.713 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:42.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:42.817 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:47:42.857 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: pgmap v2154: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 376 op/s; 221/304821 objects degraded (0.073%); 25042/304821 objects misplaced (8.215%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:42.941 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: pgmap v2154: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 376 op/s; 221/304821 objects degraded (0.073%); 25042/304821 objects misplaced (8.215%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: pgmap v2154: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 376 op/s; 221/304821 objects degraded (0.073%); 25042/304821 objects misplaced (8.215%); 4.1 MiB/s, 65 objects/s recovering 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:47:43.417 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:47:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:43.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:47:43.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:47:43.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:47:43.438 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-15T18:47:43.438 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:47:43.439 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:47:43.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-15T18:47:43.524 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:43.622 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:43.708 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:43.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:43.823 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:47:43.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:47:43.835 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:43.876 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:44.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:47:44.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:44.963 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:47:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:44.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:47:45.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:45.102 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:44 smithi049 ceph-mon[40087]: pgmap v2155: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 320/297513 objects degraded (0.108%); 24507/297513 objects misplaced (8.237%); 7.5 MiB/s, 120 objects/s recovering 2024-04-15T18:47:45.105 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:45.185 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:45.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:45.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:44 smithi082 ceph-mon[44206]: pgmap v2155: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 320/297513 objects degraded (0.108%); 24507/297513 objects misplaced (8.237%); 7.5 MiB/s, 120 objects/s recovering 2024-04-15T18:47:45.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:47:45.323 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:45.388 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:44 smithi159 ceph-mon[45336]: pgmap v2155: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 320/297513 objects degraded (0.108%); 24507/297513 objects misplaced (8.237%); 7.5 MiB/s, 120 objects/s recovering 2024-04-15T18:47:45.452 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:47:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:45.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:47:45.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:45.587 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:45.611 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:45.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:45.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:47:45.744 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:45.768 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:47:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:45.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:47:45.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:45.932 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:45.998 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:46 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 393/295746 objects degraded (0.133%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:46.014 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:46.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:46.119 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:47:46.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:46 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 393/295746 objects degraded (0.133%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:46.176 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:46.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:46 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 393/295746 objects degraded (0.133%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:46.944 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:47:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:46.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:47:47.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:47.094 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:47.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:47 smithi082 ceph-mon[44206]: pgmap v2156: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 393/295746 objects degraded (0.133%); 24319/295746 objects misplaced (8.223%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:47:47.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:47 smithi159 ceph-mon[45336]: pgmap v2156: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 393/295746 objects degraded (0.133%); 24319/295746 objects misplaced (8.223%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:47:47.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:47 smithi049 ceph-mon[40087]: pgmap v2156: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 393/295746 objects degraded (0.133%); 24319/295746 objects misplaced (8.223%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:47:47.593 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:47:47.594 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:47:47.594 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:47:47.594 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:47:47.594 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:47:49.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:47:49.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:49 smithi082 ceph-mon[44206]: pgmap v2157: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 456/292485 objects degraded (0.156%); 23972/292485 objects misplaced (8.196%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:49.278 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:49 smithi159 ceph-mon[45336]: pgmap v2157: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 456/292485 objects degraded (0.156%); 23972/292485 objects misplaced (8.196%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:49.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:49 smithi049 ceph-mon[40087]: pgmap v2157: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 456/292485 objects degraded (0.156%); 23972/292485 objects misplaced (8.196%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:47:49.650 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:47:50.732 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:47:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:47:50.732 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:47:51.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:51 smithi082 ceph-mon[44206]: pgmap v2158: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 625/287619 objects degraded (0.217%); 23583/287619 objects misplaced (8.199%); 6.7 MiB/s, 107 objects/s recovering 2024-04-15T18:47:51.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:51.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:51 smithi082 ceph-mon[44206]: osdmap e829: 12 total, 12 up, 12 in 2024-04-15T18:47:51.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:51 smithi159 ceph-mon[45336]: pgmap v2158: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 625/287619 objects degraded (0.217%); 23583/287619 objects misplaced (8.199%); 6.7 MiB/s, 107 objects/s recovering 2024-04-15T18:47:51.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:51.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:51.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:51 smithi159 ceph-mon[45336]: osdmap e829: 12 total, 12 up, 12 in 2024-04-15T18:47:51.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:51 smithi049 ceph-mon[40087]: pgmap v2158: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 625/287619 objects degraded (0.217%); 23583/287619 objects misplaced (8.199%); 6.7 MiB/s, 107 objects/s recovering 2024-04-15T18:47:51.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:51.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:51.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:51 smithi049 ceph-mon[40087]: osdmap e829: 12 total, 12 up, 12 in 2024-04-15T18:47:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:52.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:52.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:52.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:52 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:52.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:52 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:53 smithi082 ceph-mon[44206]: pgmap v2160: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 625/287124 objects degraded (0.218%); 23583/287124 objects misplaced (8.214%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:47:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 625/287124 objects degraded (0.218%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:53 smithi082 ceph-mon[44206]: osdmap e830: 12 total, 12 up, 12 in 2024-04-15T18:47:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:53 smithi159 ceph-mon[45336]: pgmap v2160: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 625/287124 objects degraded (0.218%); 23583/287124 objects misplaced (8.214%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:47:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 625/287124 objects degraded (0.218%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:53 smithi159 ceph-mon[45336]: osdmap e830: 12 total, 12 up, 12 in 2024-04-15T18:47:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:53 smithi049 ceph-mon[40087]: pgmap v2160: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.8 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 625/287124 objects degraded (0.218%); 23583/287124 objects misplaced (8.214%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:47:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 625/287124 objects degraded (0.218%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:53.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:53 smithi049 ceph-mon[40087]: osdmap e830: 12 total, 12 up, 12 in 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:143340 7.2 7:4c09de55:::benchmark_data_smithi159_149998_object47340:head [delete] snapc 0=[] ondisk+write+known_if_redirected e672)", 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 506.84674688299998, 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.343721899, 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 143340 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757869+0000", 2024-04-15T18:47:53.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757871+0000", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.689e-06 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757878+0000", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2830000000000004e-06 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757893+0000", 2024-04-15T18:47:53.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5038000000000001e-05 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757901+0000", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6909999999999997e-06 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.757952+0000", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.758042+0000", 2024-04-15T18:47:53.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0453999999999998e-05 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.758114+0000", 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2093000000000002e-05 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.761392+0000", 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032774060000000001 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763286+0000", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018943740000000001 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101516+0000", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33822963499999997 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101556+0000", 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0268000000000001e-05 2024-04-15T18:47:53.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101591+0000", 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4824999999999999e-05 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26545.0:143340 7.2 e672/621 ondisk, result = 0)", 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:47:53.679 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 506.84151143600002, 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33849178099999999, 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763104+0000", 2024-04-15T18:47:53.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763105+0000", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7700000000000005e-07 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:26.763108+0000", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.506e-06 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101426+0000", 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33831763300000001 2024-04-15T18:47:53.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101431+0000", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2469999999999999e-06 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101503+0000", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1979999999999999e-05 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101511+0000", 2024-04-15T18:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7770000000000002e-06 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:27.101596+0000", 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5161000000000007e-05 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "replica_scrub(pg: 7.1,from:0'0,to:675'9066,epoch:676/621,start:7:8923feff:::benchmark_data_smithi159_149998_object80750:0,end:7:89aa5995:::benchmark_data_smithi159_149998_object63737:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 501.85836973900001, 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.40023066699999998, 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.683 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746246+0000", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746247+0000", 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8100000000000003e-07 2024-04-15T18:47:53.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.746253+0000", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5989999999999998e-06 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146191+0000", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39993743199999998 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146210+0000", 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9602999999999998e-05 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146395+0000", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018475300000000001 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146422+0000", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7135999999999999e-05 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.146477+0000", 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4562999999999999e-05 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.686 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26545.0:146597 7.2 7:40f54c50:::benchmark_data_smithi159_149998_object50597:head [delete] snapc 0=[] ondisk+write+known_if_redirected e676)", 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 501.83674832899999, 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33914392500000001, 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.687 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26545", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/3618244140", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 146597 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767867+0000", 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767869+0000", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.637e-06 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767875+0000", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9209999999999997e-06 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767889+0000", 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3542e-05 2024-04-15T18:47:53.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767895+0000", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9500000000000004e-06 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.767943+0000", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7043999999999999e-05 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.768031+0000", 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8159999999999996e-05 2024-04-15T18:47:53.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,3", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.768102+0000", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1759000000000002e-05 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.771302+0000", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031990529999999999 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:31.772670+0000", 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013685870000000001 2024-04-15T18:47:53.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.106933+0000", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33426301200000003 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.106975+0000", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1527999999999999e-05 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:39:32.107011+0000", 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6732e-05 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86268 8.5 8:abb429cc:::benchmark_data_smithi159_162467_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 326.84190757300001, 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.615438708, 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86268 2024-04-15T18:47:53.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762709+0000", 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0800000000000004e-07 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762726+0000", 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7192000000000001e-05 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762736+0000", 2024-04-15T18:47:53.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3810000000000005e-06 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762741+0000", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5260000000000001e-06 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762778+0000", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7024000000000003e-05 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762876+0000", 2024-04-15T18:47:53.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7795999999999997e-05 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762962+0000", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6310999999999994e-05 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766713+0000", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037510239999999999 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766745+0000", 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1340000000000001e-05 2024-04-15T18:47:53.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378055+0000", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61131082199999998 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378089+0000", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3828999999999997e-05 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378147+0000", 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7655000000000003e-05 2024-04-15T18:47:53.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86270 8.5 8:a09049c9:::benchmark_data_smithi159_162467_object86269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 326.84001454600002, 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61363609500000005, 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.699 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86270 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764602+0000", 2024-04-15T18:47:53.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764625+0000", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2901e-05 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764635+0000", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9059999999999997e-06 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764643+0000", 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2800000000000003e-06 2024-04-15T18:47:53.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765036+0000", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000392407 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765142+0000", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106304 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765229+0000", 2024-04-15T18:47:53.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7107000000000005e-05 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766840+0000", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016110740000000001 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.767079+0000", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023867499999999999 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378176+0000", 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61109754500000002 2024-04-15T18:47:53.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378206+0000", 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9281999999999998e-05 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378237+0000", 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1773999999999998e-05 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.704 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86284 8.5 8:a9db63aa:::benchmark_data_smithi159_162467_object86283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 326.81241224799999, 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59497596799999997, 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86284 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.705 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:47:53.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4399999999999998e-07 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792225+0000", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0562999999999999e-05 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792235+0000", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4800000000000007e-06 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792240+0000", 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:47:53.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792282+0000", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1910999999999999e-05 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792403+0000", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012128799999999999 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792502+0000", 2024-04-15T18:47:53.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8987000000000006e-05 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.794820+0000", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023174480000000002 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.795082+0000", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026202700000000002 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387105+0000", 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59202341400000003 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387136+0000", 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1027e-05 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387179+0000", 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3510999999999997e-05 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86369 8.5 8:ace0c49e:::benchmark_data_smithi159_162467_object86368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:47:53.710 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 326.557630292, 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34036048600000002, 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86369 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046987+0000", 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3850000000000001e-06 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047012+0000", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4839000000000001e-05 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047021+0000", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0410000000000003e-06 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047028+0000", 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1849999999999998e-06 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047078+0000", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0288999999999999e-05 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047193+0000", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114578 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047292+0000", 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9225e-05 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.049881+0000", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002589058 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.049898+0000", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6453999999999999e-05 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387236+0000", 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33733875600000002 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387289+0000", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2482000000000003e-05 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387346+0000", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7194000000000002e-05 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86387 8.5 8:a9104f57:::benchmark_data_smithi159_162467_object86386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:47:53.716 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 326.47171998300001, 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25521748500000002, 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86387 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132898+0000", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9929999999999998e-06 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132926+0000", 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8184999999999999e-05 2024-04-15T18:47:53.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132946+0000", 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132955+0000", 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7779999999999996e-06 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133023+0000", 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7885000000000005e-05 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133158+0000", 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013503300000000001 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133267+0000", 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000108964 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.135567+0000", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022998680000000001 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.135790+0000", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022382000000000001 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388004+0000", 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25221387299999998 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388059+0000", 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5041999999999999e-05 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388113+0000", 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3947999999999999e-05 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:109194 8.9 8:93c68f30:::benchmark_data_smithi159_162467_object6889:head [delete] snapc 0=[] ondisk+write+known_if_redirected e749)", 2024-04-15T18:47:53.722 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 293.94737544899999, 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25696971299999999, 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109194 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657241+0000", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8100000000000002e-07 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657245+0000", 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0770000000000001e-06 2024-04-15T18:47:53.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657255+0000", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7620000000000007e-06 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657260+0000", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3539999999999999e-06 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659686+0000", 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024256019999999998 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659814+0000", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012830799999999999 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659900+0000", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6269999999999999e-05 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.664407+0000", 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045067909999999996 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.667244+0000", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028369189999999998 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914135+0000", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24689090599999999 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914165+0000", 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0035e-05 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914210+0000", 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4907999999999998e-05 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(8.6 762/743 [PushReplyOp(8:61976d2a:::benchmark_data_smithi159_162467_object63438:head)])", 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 213.404804918, 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.288127721, 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.728 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199812+0000", 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4799999999999997e-07 2024-04-15T18:47:53.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199820+0000", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2300000000000008e-06 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485841+0000", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.286021566 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485848+0000", 2024-04-15T18:47:53.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6359999999999997e-06 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485898+0000", 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9755000000000001e-05 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.487939+0000", 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020406859999999999 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:164212 8.9 8:9021f52a:::benchmark_data_smithi159_162467_object61907:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:47:53.731 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 204.53385983999999, 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30788343899999998, 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164212 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:47:53.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070757+0000", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1289999999999999e-06 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070763+0000", 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6620000000000002e-06 2024-04-15T18:47:53.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070773+0000", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669000000000001e-05 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070779+0000", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7969999999999997e-06 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070827+0000", 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7729000000000003e-05 2024-04-15T18:47:53.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070959+0000", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013182499999999999 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.071095+0000", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013583900000000001 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.075673+0000", 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045782289999999996 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.075708+0000", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5200000000000002e-05 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378569+0000", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30286143100000001 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378607+0000", 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7573000000000001e-05 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378639+0000", 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2356e-05 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26791.0:164212 8.9 e766/709 ondisk, result = 0)", 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 204.52969381899999, 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30372249400000001, 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.737 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074923+0000", 2024-04-15T18:47:53.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074930+0000", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0010000000000004e-06 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378463+0000", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.303533834 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378469+0000", 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2850000000000002e-06 2024-04-15T18:47:53.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378560+0000", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0997999999999998e-05 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378565+0000", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9280000000000001e-06 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378644+0000", 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9771999999999994e-05 2024-04-15T18:47:53.740 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:184704 8.5 8:ad05c937:::benchmark_data_smithi159_162467_object82399:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 171.17342438, 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.280460137, 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 184704 2024-04-15T18:47:53.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431192+0000", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431196+0000", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431206+0000", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0339999999999999e-05 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431211+0000", 2024-04-15T18:47:53.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3079999999999998e-06 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431254+0000", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2358999999999999e-05 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431331+0000", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7669999999999993e-05 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431414+0000", 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2330000000000006e-05 2024-04-15T18:47:53.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.435054+0000", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036404190000000002 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.435521+0000", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000466778 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711589+0000", 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27606850399999999 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711623+0000", 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3442999999999998e-05 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711652+0000", 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8693000000000001e-05 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.746 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:184711 8.5 8:aa5d4b26:::benchmark_data_smithi159_162467_object82406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 171.149494043, 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28251888400000003, 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 184711 2024-04-15T18:47:53.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455123+0000", 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.387e-06 2024-04-15T18:47:53.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455130+0000", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455143+0000", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3356999999999999e-05 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455150+0000", 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7020000000000004e-06 2024-04-15T18:47:53.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455199+0000", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9261000000000002e-05 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455289+0000", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9815999999999995e-05 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455373+0000", 2024-04-15T18:47:53.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4107e-05 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.459099+0000", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037253949999999998 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.459489+0000", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039004399999999998 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737569+0000", 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27808035199999998 2024-04-15T18:47:53.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737606+0000", 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7478999999999998e-05 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737641+0000", 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4150999999999999e-05 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:25896 9.7 9:ee1ba830:::benchmark_data_smithi159_173188_object25895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:47:53.752 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 102.522208854, 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25357581800000001, 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 25896 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082408+0000", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2500000000000004e-07 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082427+0000", 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8703999999999999e-05 2024-04-15T18:47:53.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082437+0000", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0243e-05 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082442+0000", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4299999999999997e-06 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082480+0000", 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7805000000000001e-05 2024-04-15T18:47:53.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082568+0000", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8391000000000001e-05 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082656+0000", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7449999999999998e-05 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.083527+0000", 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00087103799999999998 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.083775+0000", 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024838200000000001 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335900+0000", 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25212455 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335933+0000", 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3467999999999999e-05 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335983+0000", 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9431999999999997e-05 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107183 9.7 9:eca3b222:::benchmark_data_smithi159_173188_object107182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-04-15T18:47:53.758 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.409425693999999, 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.448199395, 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107183 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195191+0000", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1009999999999999e-06 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195212+0000", 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0451000000000001e-05 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195220+0000", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5299999999999996e-06 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195225+0000", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1710000000000001e-06 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195261+0000", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5707999999999998e-05 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195379+0000", 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000117794 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195472+0000", 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2781999999999994e-05 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.198905+0000", 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003433611 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.201194+0000", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022885560000000002 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643272+0000", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44207840900000001 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643336+0000", 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4119000000000001e-05 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643389+0000", 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3162999999999999e-05 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:107183 9.7 e820/791 ondisk, result = 0)", 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.382944903000002, 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42172676999999997, 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221672+0000", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4799999999999997e-07 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221682+0000", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7399999999999999e-06 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643194+0000", 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42151264199999999 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643200+0000", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0589999999999999e-06 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643254+0000", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3507000000000001e-05 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643265+0000", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1498000000000001e-05 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643398+0000", 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013237599999999999 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107200 9.9 9.8b51f119 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.003368213000002, 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60121708600000001, 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.768 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107200 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601249+0000", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5859999999999999e-06 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601275+0000", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6095999999999999e-05 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601292+0000", 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6702999999999999e-05 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601300+0000", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1499999999999999e-06 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601374+0000", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3526e-05 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601384+0000", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0386e-05 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.202433+0000", 2024-04-15T18:47:53.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60104948999999996 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.202465+0000", 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1149000000000003e-05 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107200 9.9 9:988f8ad1:::benchmark_data_smithi159_173188_object107199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 31.369567425, 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64774168700000001, 2024-04-15T18:47:53.772 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107200 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235050+0000", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8339999999999999e-06 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235078+0000", 2024-04-15T18:47:53.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8215999999999999e-05 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235094+0000", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5279000000000001e-05 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235106+0000", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2765999999999999e-05 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.238843+0000", 2024-04-15T18:47:53.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037361489999999998 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.238855+0000", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2471e-05 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851501+0000", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61264580599999996 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851645+0000", 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014372199999999999 2024-04-15T18:47:53.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851743+0000", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8747999999999997e-05 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860277+0000", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.008534023 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860337+0000", 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9802000000000001e-05 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.877703+0000", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017365734000000001 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.877768+0000", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5407000000000006e-05 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.882790+0000", 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050217300000000003 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.778 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.779 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.779 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:47:53.779 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:47:53.779 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:47:55.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:55 smithi082 ceph-mon[44206]: pgmap v2162: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 732/280104 objects degraded (0.261%); 23063/280104 objects misplaced (8.234%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:47:55.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:55 smithi159 ceph-mon[45336]: pgmap v2162: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 732/280104 objects degraded (0.261%); 23063/280104 objects misplaced (8.234%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:47:55.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:55 smithi049 ceph-mon[40087]: pgmap v2162: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 732/280104 objects degraded (0.261%); 23063/280104 objects misplaced (8.234%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:47:56.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:47:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:57 smithi049 ceph-mon[40087]: pgmap v2163: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 769/278517 objects degraded (0.276%); 22889/278517 objects misplaced (8.218%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:47:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:57 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:47:57.294 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:57.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:57.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:57 smithi159 ceph-mon[45336]: pgmap v2163: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 769/278517 objects degraded (0.276%); 22889/278517 objects misplaced (8.218%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:47:57.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:57 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:47:57.404 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:57.434 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:57 smithi082 ceph-mon[44206]: pgmap v2163: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 769/278517 objects degraded (0.276%); 22889/278517 objects misplaced (8.218%); 6.0 MiB/s, 96 objects/s recovering 2024-04-15T18:47:57.434 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:57 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:47:57.437 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:57.462 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:57.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:57.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:47:57.599 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:58.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:58 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:47:58.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:58 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:47:58.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:58 smithi049 ceph-mon[40087]: osdmap e831: 12 total, 12 up, 12 in 2024-04-15T18:47:58.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 769/278517 objects degraded (0.276%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:47:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:58.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:47:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:58 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:47:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:58 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:47:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:58 smithi159 ceph-mon[45336]: osdmap e831: 12 total, 12 up, 12 in 2024-04-15T18:47:58.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 769/278517 objects degraded (0.276%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:58.406 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:47:58.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:47:58.505 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:58.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:58 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:47:58.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:58 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:47:58.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:58 smithi082 ceph-mon[44206]: osdmap e831: 12 total, 12 up, 12 in 2024-04-15T18:47:58.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 769/278517 objects degraded (0.276%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:47:58.585 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:47:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:58.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:47:58.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:58.721 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:58.752 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:58.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:58.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:58.886 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:58.909 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:58.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:59.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:59.042 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.064 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:59.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:59.197 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:47:59 smithi049 ceph-mon[40087]: pgmap v2165: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 807/275208 objects degraded (0.293%); 22538/275208 objects misplaced (8.189%); 7.0 MiB/s, 112 objects/s recovering 2024-04-15T18:47:59.220 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:59.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:59.360 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:47:59 smithi082 ceph-mon[44206]: pgmap v2165: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 807/275208 objects degraded (0.293%); 22538/275208 objects misplaced (8.189%); 7.0 MiB/s, 112 objects/s recovering 2024-04-15T18:47:59.362 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.389 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:59.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:47:59 smithi159 ceph-mon[45336]: pgmap v2165: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 807/275208 objects degraded (0.293%); 22538/275208 objects misplaced (8.189%); 7.0 MiB/s, 112 objects/s recovering 2024-04-15T18:47:59.492 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:59.528 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.549 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:59.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:47:59.682 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.705 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:47:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:47:59.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:47:59.844 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:47:59.867 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:47:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:47:59.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:47:59.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:00.010 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:00.099 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:00.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:00.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:00.264 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:00.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:48:00.309 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:00.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:00.317 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:48:00.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:00.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:48:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:00.471 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:00.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:00.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:00.616 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:01.377 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:01.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:01 smithi049 ceph-mon[40087]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 912/270279 objects degraded (0.337%); 22172/270279 objects misplaced (8.203%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:48:01.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:01 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:01.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:48:01.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:01 smithi049 ceph-mon[40087]: osdmap e832: 12 total, 12 up, 12 in 2024-04-15T18:48:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:01.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:01 smithi159 ceph-mon[45336]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 912/270279 objects degraded (0.337%); 22172/270279 objects misplaced (8.203%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:48:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:01 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:48:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:01 smithi159 ceph-mon[45336]: osdmap e832: 12 total, 12 up, 12 in 2024-04-15T18:48:01.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:01.522 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:01 smithi082 ceph-mon[44206]: pgmap v2166: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 912/270279 objects degraded (0.337%); 22172/270279 objects misplaced (8.203%); 8.7 MiB/s, 140 objects/s recovering 2024-04-15T18:48:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:01 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1916971442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:48:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:01 smithi082 ceph-mon[44206]: osdmap e832: 12 total, 12 up, 12 in 2024-04-15T18:48:02.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:48:02.316 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:48:02.316 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-15T18:48:02.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:48:02.879 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:02.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:02.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:48:03.029 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:48:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:03 smithi049 ceph-mon[40087]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 433 op/s; 912/269694 objects degraded (0.338%); 22172/269694 objects misplaced (8.221%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:48:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:03 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:03 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:03 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 912/269694 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:03.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:03 smithi159 ceph-mon[45336]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 433 op/s; 912/269694 objects degraded (0.338%); 22172/269694 objects misplaced (8.221%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:48:03.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:03 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:03.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:03 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:03.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:03 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 912/269694 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:03.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:03 smithi082 ceph-mon[44206]: pgmap v2168: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 433 op/s; 912/269694 objects degraded (0.338%); 22172/269694 objects misplaced (8.221%); 5.5 MiB/s, 88 objects/s recovering 2024-04-15T18:48:03.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:03 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:03.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:03 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:03.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:03 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 912/269694 objects degraded (0.338%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:04.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:04 smithi049 ceph-mon[40087]: osdmap e833: 12 total, 12 up, 12 in 2024-04-15T18:48:04.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:04 smithi159 ceph-mon[45336]: osdmap e833: 12 total, 12 up, 12 in 2024-04-15T18:48:04.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:04 smithi082 ceph-mon[44206]: osdmap e833: 12 total, 12 up, 12 in 2024-04-15T18:48:05.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:05 smithi049 ceph-mon[40087]: pgmap v2170: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1043/261969 objects degraded (0.398%); 21617/261969 objects misplaced (8.252%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:05.297 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:48:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:05.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:48:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:131962 9.3 e833/791 9:cc79c7c7:::benchmark_data_smithi159_173188_object23329:head v 833'8201, mlcod=833'8198)", 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:05.385464+0000", 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0057791020000000004, 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058114860000000003, 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:05.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385464+0000", 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385464+0000", 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385465+0000", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7000000000000004e-07 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385471+0000", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4989999999999999e-06 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385489+0000", 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7495000000000001e-05 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385495+0000", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1279999999999996e-06 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385541+0000", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6514999999999998e-05 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.385556+0000", 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4450000000000001e-05 2024-04-15T18:48:05.382 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:131963 9.3 e833/791 9:ce77c172:::benchmark_data_smithi159_173188_object23330:head v 833'8202, mlcod=833'8198)", 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:05.387894+0000", 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0033491660000000002, 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034523850000000001, 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387894+0000", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387894+0000", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387895+0000", 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5099999999999998e-07 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387915+0000", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0032999999999999e-05 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387978+0000", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2916000000000005e-05 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.387987+0000", 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.448e-06 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.389107+0000", 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00111972 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:05.389126+0000", 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9224000000000001e-05 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:05.386 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:05.387 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-04-15T18:48:05.387 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:48:05.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:05 smithi159 ceph-mon[45336]: pgmap v2170: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1043/261969 objects degraded (0.398%); 21617/261969 objects misplaced (8.252%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:05.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:05.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:05 smithi082 ceph-mon[44206]: pgmap v2170: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 1043/261969 objects degraded (0.398%); 21617/261969 objects misplaced (8.252%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:06.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:06.278 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:06.314 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:06.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:06.465 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:06.493 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:06.603 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:06.631 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:06.655 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:06.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:06.802 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:06.833 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:06.937 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:06.967 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:06.989 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:06.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:07.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:48:07.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:07.141 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:07.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:07 smithi082 ceph-mon[44206]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 1087/260307 objects degraded (0.418%); 21447/260307 objects misplaced (8.239%); 6.3 MiB/s, 101 objects/s recovering 2024-04-15T18:48:07.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:07 smithi159 ceph-mon[45336]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 1087/260307 objects degraded (0.418%); 21447/260307 objects misplaced (8.239%); 6.3 MiB/s, 101 objects/s recovering 2024-04-15T18:48:07.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:07 smithi049 ceph-mon[40087]: pgmap v2171: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 1087/260307 objects degraded (0.418%); 21447/260307 objects misplaced (8.239%); 6.3 MiB/s, 101 objects/s recovering 2024-04-15T18:48:08.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1087/260307 objects degraded (0.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:08.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1087/260307 objects degraded (0.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:08.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1087/260307 objects degraded (0.418%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:09.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:48:09.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:48:09.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:09 smithi049 ceph-mon[40087]: pgmap v2172: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1134/256983 objects degraded (0.441%); 21110/256983 objects misplaced (8.215%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:48:09.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:09 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/677351864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:09.146 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:48:09.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:48:09.219 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:48:09.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:48:09.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:48:09.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:48:09.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:48:09.396 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:48:09.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:09 smithi159 ceph-mon[45336]: pgmap v2172: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1134/256983 objects degraded (0.441%); 21110/256983 objects misplaced (8.215%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:48:09.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:09 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/677351864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:09.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:48:09.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:48:09.484 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:48:09.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:48:09.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:09 smithi082 ceph-mon[44206]: pgmap v2172: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1134/256983 objects degraded (0.441%); 21110/256983 objects misplaced (8.215%); 6.5 MiB/s, 104 objects/s recovering 2024-04-15T18:48:09.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:09 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:09.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/677351864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:10.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:48:10.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:10 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:10 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:10.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:10 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:11.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:11 smithi159 ceph-mon[45336]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1250/251658 objects degraded (0.497%); 20730/251658 objects misplaced (8.237%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:48:11.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:48:11.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:48:11.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:11.409 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:11.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:11 smithi049 ceph-mon[40087]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1250/251658 objects degraded (0.497%); 20730/251658 objects misplaced (8.237%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:48:11.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:48:11.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:48:11.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:11.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:11.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:11 smithi082 ceph-mon[44206]: pgmap v2173: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 643 op/s; 1250/251658 objects degraded (0.497%); 20730/251658 objects misplaced (8.237%); 7.1 MiB/s, 114 objects/s recovering 2024-04-15T18:48:11.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:48:11.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:48:11.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:48:11.529 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:11.567 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:11.595 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:11.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:11.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:11.746 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:11.792 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:11.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:11.840 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:11.847 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:11.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:11.966 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:12.028 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:12.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:12.133 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:12.194 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:12.224 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:12.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:12.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:12.357 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:12.391 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:12.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:12.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:12.556 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:12.580 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:12.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:12.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:12.721 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:13.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:13 smithi049 ceph-mon[40087]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s; 1250/251031 objects degraded (0.498%); 20730/251031 objects misplaced (8.258%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:48:13.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:13 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1250/251031 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:13.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:13 smithi159 ceph-mon[45336]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s; 1250/251031 objects degraded (0.498%); 20730/251031 objects misplaced (8.258%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:48:13.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:13 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1250/251031 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:13.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:13 smithi082 ceph-mon[44206]: pgmap v2174: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s; 1250/251031 objects degraded (0.498%); 20730/251031 objects misplaced (8.258%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:48:13.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:13 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1250/251031 objects degraded (0.498%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:14.267 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:14.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:14.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:14.400 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:14.429 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:14.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:14.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:14.564 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:14.591 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:14.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:14.698 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:48:14.751 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:14.782 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:48:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:14.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:48:14.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:14.920 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:14.943 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:14.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:15.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:15.085 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:15.108 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:15.108 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:15 smithi159 ceph-mon[45336]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1381/243399 objects degraded (0.567%); 20144/243399 objects misplaced (8.276%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:48:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:15.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:15.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:15 smithi049 ceph-mon[40087]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1381/243399 objects degraded (0.567%); 20144/243399 objects misplaced (8.276%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:48:15.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:48:15.262 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:15.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:15 smithi082 ceph-mon[44206]: pgmap v2175: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 656 op/s; 1381/243399 objects degraded (0.567%); 20144/243399 objects misplaced (8.276%); 8.4 MiB/s, 134 objects/s recovering 2024-04-15T18:48:16.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 833, 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:48:03.068911+0000", 2024-04-15T18:48:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:46:33.707597+0000", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:47:20.193442+0000", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 31, 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:48:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:48:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:48:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:48:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:48:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:48:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:48:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-04-15T18:48:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:45:48.486814+0000", 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:48:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "794", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:48:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:48:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:48:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8799999952316284, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-15T18:48:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 671, 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.081 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.082 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.086 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.087 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 778, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.088 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:48:17.089 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:48:17.090 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.092 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 675, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:48:17.093 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.094 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.096 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:48:17.097 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:48:17.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:17.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:48:17.103 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:48:17.104 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.106 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.108 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 676, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:48:17.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.113 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 823, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.114 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:48:17.118 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:48:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.119 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.121 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:48:17.126 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:48:17.127 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.128 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.130 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.132 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:48:17.133 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:48:17.134 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 94, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 733, 2024-04-15T18:48:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 807, 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 820, 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 734, 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:48:17.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:48:17.140 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.141 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:17.142 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/905169792", 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/905169792", 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/905169792", 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/905169792", 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:48:17.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:21.107720+0000", 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.144 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:40.053179+0000", 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:19.174021+0000", 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.145 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:18.700712+0000", 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.146 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:51.456661+0000", 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:49.958135+0000", 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:48:17.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:46:57.843303+0000", 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.148 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:30.214236+0000", 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:04.125183+0000", 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:56.756529+0000", 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:48:17.150 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:27.113465+0000", 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:17.151 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:02.524072+0000", 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 733 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 5, 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.0", 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.5", 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:48:17.153 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:17.154 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.9", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:17.155 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:17.156 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:17.157 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:48:17.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:17 smithi159 ceph-mon[45336]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1481/241788 objects degraded (0.613%); 19935/241788 objects misplaced (8.245%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:48:17.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2848290574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:17.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:17 smithi049 ceph-mon[40087]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1481/241788 objects degraded (0.613%); 19935/241788 objects misplaced (8.245%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:48:17.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2848290574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:17 smithi082 ceph-mon[44206]: pgmap v2176: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1481/241788 objects degraded (0.613%); 19935/241788 objects misplaced (8.245%); 6.4 MiB/s, 103 objects/s recovering 2024-04-15T18:48:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2848290574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:18.114 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:18 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1481/241788 objects degraded (0.613%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:18.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:18 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1481/241788 objects degraded (0.613%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:18 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1481/241788 objects degraded (0.613%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:18.784 INFO:tasks.thrashosds.thrasher:j is {'epoch': 833, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:48:03.068911+0000', 'last_up_change': '2024-04-15T18:46:33.707597+0000', 'last_in_change': '2024-04-15T18:47:20.193442+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 31, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-04-15T18:45:48.486814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '794', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 671, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 778, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 675, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 791, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 676, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 94, 'last_clean_end': 733, 'up_from': 807, 'up_thru': 820, 'down_at': 734, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 905169792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 905169792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 905169792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 905169792}]}, 'public_addr': '172.21.15.159:6825/905169792', 'cluster_addr': '172.21.15.159:6827/905169792', 'heartbeat_back_addr': '172.21.15.159:6831/905169792', 'heartbeat_front_addr': '172.21.15.159:6829/905169792', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:21.107720+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:40.053179+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:19.174021+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:18.700712+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:51.456661+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:49.958135+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:46:57.843303+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:30.214236+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:04.125183+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:56.756529+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:27.113465+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:02.524072+0000', 'dead_epoch': 733}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '9.d', 'mappings': [{'from': 5, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6]}, {'pgid': '9.5', 'osds': [1, 5, 7]}, {'pgid': '9.8', 'osds': [9, 5]}, {'pgid': '9.9', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:48:18.785 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-15T18:48:19.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:19 smithi049 ceph-mon[40087]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1505/238488 objects degraded (0.631%); 19603/238488 objects misplaced (8.220%); 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:48:19.238 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:48:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:19.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:48:19.269 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:19.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:19.371 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:19 smithi082 ceph-mon[44206]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1505/238488 objects degraded (0.631%); 19603/238488 objects misplaced (8.220%); 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:48:19.372 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:19.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:19 smithi159 ceph-mon[45336]: pgmap v2177: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 1505/238488 objects degraded (0.631%); 19603/238488 objects misplaced (8.220%); 7.4 MiB/s, 118 objects/s recovering 2024-04-15T18:48:19.396 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:19.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:19.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:19.540 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:19.758 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:19.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:19.863 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:19.922 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:19.975 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:19.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:20.084 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:48:20.143 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:20.169 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:48:20.172 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:20.172 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:20.172 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:20 smithi159 ceph-mon[45336]: osdmap e834: 12 total, 12 up, 12 in 2024-04-15T18:48:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:20.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:48:20.274 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:20.303 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:20.327 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:20.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:20.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:20.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:20 smithi049 ceph-mon[40087]: osdmap e834: 12 total, 12 up, 12 in 2024-04-15T18:48:20.466 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:20.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:20.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:20.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:20 smithi082 ceph-mon[44206]: osdmap e834: 12 total, 12 up, 12 in 2024-04-15T18:48:21.367 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:21.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:21.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:21 smithi049 ceph-mon[40087]: pgmap v2179: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 1566/233394 objects degraded (0.671%); 19243/233394 objects misplaced (8.245%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:21.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:21.513 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:21 smithi159 ceph-mon[45336]: pgmap v2179: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 1566/233394 objects degraded (0.671%); 19243/233394 objects misplaced (8.245%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:21.519 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:21.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:21 smithi082 ceph-mon[44206]: pgmap v2179: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 1566/233394 objects degraded (0.671%); 19243/233394 objects misplaced (8.245%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:21.560 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:21.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:21.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:21.700 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:21.726 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:21.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:21.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:21.866 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:21.896 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:21.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:22.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:48:22.037 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:22.543 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:48:22.543 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:48:23.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:23 smithi049 ceph-mon[40087]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 1566/232908 objects degraded (0.672%); 19243/232908 objects misplaced (8.262%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:23.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1566/232908 objects degraded (0.672%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:23.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:23 smithi082 ceph-mon[44206]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 1566/232908 objects degraded (0.672%); 19243/232908 objects misplaced (8.262%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:23.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1566/232908 objects degraded (0.672%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:23 smithi159 ceph-mon[45336]: pgmap v2180: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 1566/232908 objects degraded (0.672%); 19243/232908 objects misplaced (8.262%); 7.3 MiB/s, 117 objects/s recovering 2024-04-15T18:48:23.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1566/232908 objects degraded (0.672%), 3 pgs degraded (PG_DEGRADED) 2024-04-15T18:48:23.785 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:48:23.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:48:23.785 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-04-15T18:48:23.786 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-15T18:48:23.786 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-15T18:48:23.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-04-15T18:48:24.211 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:48:24.300 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:48:24.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:24.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:48:24.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:48:24.405 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:24.433 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:24.457 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:24.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:24.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:24.591 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:24.617 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:24.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:24.724 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:24.761 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:25.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:25 smithi049 ceph-mon[40087]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 1663/225561 objects degraded (0.737%); 18696/225561 objects misplaced (8.289%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:48:25.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:25.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:25.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:25 smithi049 ceph-mon[40087]: osdmap e835: 12 total, 12 up, 12 in 2024-04-15T18:48:25.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:25 smithi082 ceph-mon[44206]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 1663/225561 objects degraded (0.737%); 18696/225561 objects misplaced (8.289%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:48:25.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:25.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:25.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:25 smithi082 ceph-mon[44206]: osdmap e835: 12 total, 12 up, 12 in 2024-04-15T18:48:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:25 smithi159 ceph-mon[45336]: pgmap v2181: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 1663/225561 objects degraded (0.737%); 18696/225561 objects misplaced (8.289%); 7.3 MiB/s, 116 objects/s recovering 2024-04-15T18:48:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:25.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:25 smithi159 ceph-mon[45336]: osdmap e835: 12 total, 12 up, 12 in 2024-04-15T18:48:25.970 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:25.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:26.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:48:26.113 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:26.208 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:48:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:48:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:26.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:48:26.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3512057026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:26.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:26 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:26.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3512057026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:26.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:26 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:26.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:26.473 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3512057026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:26 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:48:27.042 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:27.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:27.147 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:27.198 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:48:27.199 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:27.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:48:27.239 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:27.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:27.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:27.257 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:48:27.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:48:27.318 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:48:27.344 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:27.355 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:48:27.379 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:27.383 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:48:27.408 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:27.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:27.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:48:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:27 smithi049 ceph-mon[40087]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1748/223902 objects degraded (0.781%); 18488/223902 objects misplaced (8.257%); 6.8 MiB/s, 108 objects/s recovering 2024-04-15T18:48:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:27 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:27 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:48:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:27 smithi049 ceph-mon[40087]: osdmap e836: 12 total, 12 up, 12 in 2024-04-15T18:48:27.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/168785121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:27.462 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:48:27.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:48:27.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:27.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:27 smithi082 ceph-mon[44206]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1748/223902 objects degraded (0.781%); 18488/223902 objects misplaced (8.257%); 6.8 MiB/s, 108 objects/s recovering 2024-04-15T18:48:27.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:27 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:27.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:27 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:48:27.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:27 smithi082 ceph-mon[44206]: osdmap e836: 12 total, 12 up, 12 in 2024-04-15T18:48:27.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/168785121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:27.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:48:27.549 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:27.591 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:48:27.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:48:27.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:48:27.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:48:27.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:48:27.594 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:48:27.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:48:27.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:27 smithi159 ceph-mon[45336]: pgmap v2183: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 1748/223902 objects degraded (0.781%); 18488/223902 objects misplaced (8.257%); 6.8 MiB/s, 108 objects/s recovering 2024-04-15T18:48:27.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:27 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:27.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:27 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:48:27.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:27 smithi159 ceph-mon[45336]: osdmap e836: 12 total, 12 up, 12 in 2024-04-15T18:48:27.605 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/168785121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:27.909 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:27.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:28.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:28.042 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:28.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:28.172 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:28.220 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:48:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-15T18:48:28.220 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:48:28.233 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:28.262 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:28.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:28.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:28.401 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.402 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1748/223902 objects degraded (0.781%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:28.407 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:28.429 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:28.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:28.438 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1748/223902 objects degraded (0.781%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:28.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:28.576 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:28.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1748/223902 objects degraded (0.781%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:28.730 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:28.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:28.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:28.885 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:29.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:29 smithi049 ceph-mon[40087]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 1799/220464 objects degraded (0.816%); 18103/220464 objects misplaced (8.211%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:48:29.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:29 smithi082 ceph-mon[44206]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 1799/220464 objects degraded (0.816%); 18103/220464 objects misplaced (8.211%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:48:29.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:29 smithi159 ceph-mon[45336]: pgmap v2185: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 1799/220464 objects degraded (0.816%); 18103/220464 objects misplaced (8.211%); 7.1 MiB/s, 113 objects/s recovering 2024-04-15T18:48:29.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:48:30.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:48:30.552 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:30.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:30.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:48:30.706 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:30.745 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:48:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:30.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:48:30.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:48:30.911 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:30.913 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:30.942 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:30.997 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:48:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:48:31.114 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:31.166 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:31.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:31 smithi049 ceph-mon[40087]: pgmap v2186: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 1871/215475 objects degraded (0.868%); 17744/215475 objects misplaced (8.235%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:31.199 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:31 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:31.199 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:31 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:31.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:31.346 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:31.423 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:31.425 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:31 smithi159 ceph-mon[45336]: pgmap v2186: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 1871/215475 objects degraded (0.868%); 17744/215475 objects misplaced (8.235%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:31.425 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:31 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:31.426 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:31 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:31.462 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:31 smithi082 ceph-mon[44206]: pgmap v2186: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 1871/215475 objects degraded (0.868%); 17744/215475 objects misplaced (8.235%); 9.3 MiB/s, 149 objects/s recovering 2024-04-15T18:48:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:31 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:31 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:31.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:31.628 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:31.653 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:31.762 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:31.798 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:31.823 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:31.930 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:31.961 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:31.984 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:31.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:32.092 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:32.120 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:32.146 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:32.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:32.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:32.290 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:32.318 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:32.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:32.421 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:32.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:32 smithi082 ceph-mon[44206]: osdmap e837: 12 total, 12 up, 12 in 2024-04-15T18:48:32.452 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:32.473 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:32.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:32.578 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:32.608 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:32.632 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:32.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:32.654 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:32 smithi159 ceph-mon[45336]: osdmap e837: 12 total, 12 up, 12 in 2024-04-15T18:48:32.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:32 smithi049 ceph-mon[40087]: osdmap e837: 12 total, 12 up, 12 in 2024-04-15T18:48:32.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:32.779 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:32.931 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:48:32.931 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:116782 7.1 e661/621 7:897aa864:::benchmark_data_smithi159_149998_object20782:head v 661'7226, mlcod=661'7224)", 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 583.74845504999996, 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22267819999999999, 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193473+0000", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193474+0000", 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4900000000000005e-07 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.193480+0000", 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9850000000000003e-06 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408763+0000", 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.215283789 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408779+0000", 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5925000000000001e-05 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408841+0000", 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1650999999999996e-05 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.408864+0000", 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3337e-05 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.416103+0000", 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072389539999999997 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:38:49.416151+0000", 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7809999999999998e-05 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.936 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(7.c 670/648 [PushReplyOp(7:329499ac:::benchmark_data_smithi159_149998_object70303:head)])", 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 549.25418279799999, 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41282126800000002, 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687745+0000", 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687746+0000", 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6499999999999999e-07 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:23.687750+0000", 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7500000000000003e-06 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099805+0000", 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41205476099999999 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099816+0000", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1031e-05 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.099882+0000", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6072e-05 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:24.100566+0000", 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00068408900000000003 2024-04-15T18:48:32.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:146592 7.0 e676/621 7:0ce3da3a:::benchmark_data_smithi159_149998_object50592:head v 676'9102, mlcod=676'9099)", 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 541.218308052, 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37473794599999999, 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723620+0000", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723621+0000", 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4099999999999997e-07 2024-04-15T18:48:32.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:31.723626+0000", 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.485e-06 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094806+0000", 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37117979200000001 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094816+0000", 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0121e-05 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094906+0000", 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9442000000000002e-05 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.094928+0000", 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2731999999999999e-05 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.098320+0000", 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033914890000000001 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:39:32.098358+0000", 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7944000000000001e-05 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:82886 8.0 e736/709 8:054a31d8:::benchmark_data_smithi159_162467_object82885:head v 736'5140, mlcod=736'5134)", 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 369.50793724200003, 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21918625799999999, 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7700000000000004e-07 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.434090+0000", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9130999999999997e-05 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650659+0000", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21656877999999999 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650669+0000", 2024-04-15T18:48:32.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3389999999999999e-06 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650736+0000", 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7150000000000006e-05 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650757+0000", 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1081e-05 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.653135+0000", 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023781499999999999 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.653177+0000", 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2049999999999999e-05 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:82902 8.0 e736/709 8:0b1c4e81:::benchmark_data_smithi159_162467_object82901:head v 736'5141, mlcod=736'5136)", 2024-04-15T18:48:32.950 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 369.497306385, 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21600396299999999, 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444623+0000", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2300000000000001e-06 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444696+0000", 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2739999999999998e-05 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654441+0000", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20974567099999999 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654449+0000", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9249999999999995e-06 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654512+0000", 2024-04-15T18:48:32.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3359000000000001e-05 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654533+0000", 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0837000000000001e-05 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.660566+0000", 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060331960000000002 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.660625+0000", 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9005000000000002e-05 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:129871 8.8 e754/709 8:1dc83888:::benchmark_data_smithi159_162467_object27566:head v 754'8136, mlcod=754'8132)", 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:32.758469+0000", 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 300.18345859800002, 2024-04-15T18:48:32.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16002148699999999, 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.758469+0000", 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.758469+0000", 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.758471+0000", 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1829999999999999e-06 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.758479+0000", 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6519999999999995e-06 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.913802+0000", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15532257599999999 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.913813+0000", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.097e-05 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.913879+0000", 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5947999999999999e-05 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.913906+0000", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6874000000000001e-05 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.918430+0000", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045240970000000004 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.918491+0000", 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1186999999999995e-05 2024-04-15T18:48:32.959 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:129872 8.0 e754/709 8:0270eb88:::benchmark_data_smithi159_162467_object27567:head v 754'8126, mlcod=754'8123)", 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:32.760557+0000", 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 300.18137062400001, 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.167687114, 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.760557+0000", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.760557+0000", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.760558+0000", 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8100000000000002e-07 2024-04-15T18:48:32.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.760566+0000", 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3410000000000003e-06 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.923837+0000", 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16327056100000001 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.923844+0000", 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4399999999999999e-06 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.923903+0000", 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8909999999999997e-05 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.923927+0000", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4049999999999998e-05 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.928196+0000", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042686060000000003 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:32.928244+0000", 2024-04-15T18:48:32.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8424999999999998e-05 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:164234 8.0 e766/709 8:05b44565:::benchmark_data_smithi159_162467_object61929:head v 766'10204, mlcod=766'10202)", 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 243.790952172, 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24182635299999999, 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.965 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150977+0000", 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7699999999999992e-07 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150986+0000", 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4159999999999993e-06 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388667+0000", 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.237680745 2024-04-15T18:48:32.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388681+0000", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.431e-05 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388828+0000", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014656599999999999 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388874+0000", 2024-04-15T18:48:32.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6641999999999998e-05 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.392758+0000", 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038836330000000001 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.392802+0000", 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4063999999999997e-05 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:19854 9.9 e798/791 9:9bfb6eb2:::benchmark_data_smithi159_173188_object19853:head v 798'1261, mlcod=798'1259)", 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:07.124902+0000", 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.817026191, 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15735824600000001, 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.124902+0000", 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.124902+0000", 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.124902+0000", 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2600000000000002e-07 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.124998+0000", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6181999999999995e-05 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.281395+0000", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15639655 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.281400+0000", 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2240000000000003e-06 2024-04-15T18:48:32.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.281447+0000", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7231999999999998e-05 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.281470+0000", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2566e-05 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.282228+0000", 2024-04-15T18:48:32.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075762400000000004 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.282260+0000", 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2342000000000003e-05 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:20512 9.2 e798/791 9:4609eba9:::benchmark_data_smithi159_173188_object20511:head v 798'1310, mlcod=798'1306)", 2024-04-15T18:48:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.34144481600001, 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.230495805, 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600484+0000", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8400000000000004e-07 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600504+0000", 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0466999999999999e-05 2024-04-15T18:48:32.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.829983+0000", 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22947854200000001 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.829989+0000", 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8619999999999999e-06 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830052+0000", 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3346e-05 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830078+0000", 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5901000000000002e-05 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830952+0000", 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00087385800000000003 2024-04-15T18:48:32.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830979+0000", 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7144999999999998e-05 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:55688 9.1 e805/791 9:8e075c20:::benchmark_data_smithi159_173188_object55687:head v 805'3456, mlcod=805'3453)", 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 120.639074771, 2024-04-15T18:48:32.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22574402499999999, 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302854+0000", 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302973+0000", 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118841 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525198+0000", 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.222224858 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525214+0000", 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6704000000000001e-05 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525270+0000", 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5557999999999999e-05 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525284+0000", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4691999999999999e-05 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.528555+0000", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003270649 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.528597+0000", 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2046999999999999e-05 2024-04-15T18:48:32.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:55744 9.2 e805/791 9:4e4066a2:::benchmark_data_smithi159_173188_object55743:head v 805'3456, mlcod=805'3453)", 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:32.341879+0000", 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 120.600048523, 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.189504803, 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.341879+0000", 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.341879+0000", 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.341881+0000", 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1650000000000001e-06 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.341906+0000", 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5846e-05 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.529638+0000", 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.187731709 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.529645+0000", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2980000000000003e-06 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.529710+0000", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4549999999999997e-05 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.529728+0000", 2024-04-15T18:48:32.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8221000000000001e-05 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.531334+0000", 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001606293 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.531384+0000", 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9721e-05 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:88878 9.2 e814/791 9:48f0ea8d:::benchmark_data_smithi159_173188_object88877:head v 814'5542, mlcod=814'5539)", 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 91.443378229000004, 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21614793800000001, 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498551+0000", 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4900000000000004e-07 2024-04-15T18:48:32.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498579+0000", 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.836e-05 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711601+0000", 2024-04-15T18:48:32.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21302242900000001 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711608+0000", 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9450000000000004e-06 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711689+0000", 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0228000000000005e-05 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711715+0000", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6690999999999999e-05 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.714657+0000", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029415869999999998 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.714698+0000", 2024-04-15T18:48:32.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0748999999999999e-05 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:107185 9.1 e820/791 9:8e8a4364:::benchmark_data_smithi159_173188_object107184:head v 820'6735, mlcod=819'6731)", 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 72.721097553999996, 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35684397299999998, 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.994 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:32.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220831+0000", 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1899999999999999e-07 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220861+0000", 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.078e-05 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574570+0000", 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35370858199999999 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574577+0000", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8240000000000002e-06 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574630+0000", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3201000000000001e-05 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574648+0000", 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8457999999999998e-05 2024-04-15T18:48:32.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.577632+0000", 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029834140000000002 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.577674+0000", 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2494999999999999e-05 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:32.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:108615 9.3 e824/791 9:c0fefe84:::benchmark_data_smithi159_173188_object108614:head v 824'6735, mlcod=824'6733)", 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 67.694132035999999, 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.364883597, 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:32.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247797+0000", 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.23e-07 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247874+0000", 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6927000000000004e-05 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609793+0000", 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36191985100000001 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609805+0000", 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1585e-05 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609881+0000", 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6261999999999996e-05 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609904+0000", 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2286e-05 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.612643+0000", 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027391479999999998 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.612679+0000", 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6715000000000003e-05 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:108620 9.2 e824/791 9:4e35ac8e:::benchmark_data_smithi159_173188_object108619:head v 824'6814, mlcod=824'6811)", 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 67.687836081, 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28632161, 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254093+0000", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254118+0000", 2024-04-15T18:48:33.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5588e-05 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536803+0000", 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28268433599999998 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536813+0000", 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.081e-05 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536876+0000", 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2095999999999996e-05 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536894+0000", 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8898000000000001e-05 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540350+0000", 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034555509999999998 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540413+0000", 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3449999999999997e-05 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:110369 9.1 e825/791 9:88154153:::benchmark_data_smithi159_173188_object1736:head v 825'6930, mlcod=825'6927)", 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:28.912216+0000", 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 64.029712224999997, 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16045667599999999, 2024-04-15T18:48:33.008 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:28.912216+0000", 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:28.912216+0000", 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:28.912216+0000", 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4300000000000003e-07 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:28.912222+0000", 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1970000000000004e-06 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.068836+0000", 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15661489200000001 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.068846+0000", 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3239999999999992e-06 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.068926+0000", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0018000000000005e-05 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.068952+0000", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5928999999999999e-05 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.072617+0000", 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036651090000000002 2024-04-15T18:48:33.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:29.072672+0000", 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5563999999999999e-05 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:123464 9.c e829/791 9:393c6a59:::benchmark_data_smithi159_173188_object14831:head v 829'7686, mlcod=829'7683)", 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:48:33.013 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 41.675507289999999, 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24320899900000001, 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8999999999999996e-07 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266428+0000", 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502266+0000", 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.235837662 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502272+0000", 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2469999999999997e-06 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502319+0000", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6671000000000001e-05 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502335+0000", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5634e-05 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.509589+0000", 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072545300000000004 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.509630+0000", 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0528000000000002e-05 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:130391 9.7 e832/791 9:e9635ffd:::benchmark_data_smithi159_173188_object21758:head v 832'8140, mlcod=832'8137)", 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 30.235550885999999, 2024-04-15T18:48:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20858686800000001, 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706378+0000", 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7999999999999996e-07 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706382+0000", 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6589999999999998e-06 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910843+0000", 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20446082900000001 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910849+0000", 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8019999999999997e-06 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910904+0000", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4707999999999999e-05 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910920+0000", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6875e-05 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.914922+0000", 2024-04-15T18:48:33.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040018579999999996 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.914964+0000", 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1656999999999998e-05 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:147946 9.c e837/791 9:3a288080:::benchmark_data_smithi159_173188_object39313:head v 837'9251, mlcod=837'9248)", 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:32.034609+0000", 2024-04-15T18:48:33.023 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.90731843300000004, 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17657189400000001, 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.034609+0000", 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:48:33.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.034609+0000", 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.034611+0000", 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.869e-06 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.034624+0000", 2024-04-15T18:48:33.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2323999999999999e-05 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.207205+0000", 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17258132400000001 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.207212+0000", 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4390000000000003e-06 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.207605+0000", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039280499999999998 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.207631+0000", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5295000000000001e-05 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.211149+0000", 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003518223 2024-04-15T18:48:33.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:32.211181+0000", 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2614999999999999e-05 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:33.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:33.029 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:48:33.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:33 smithi082 ceph-mon[44206]: pgmap v2188: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 419 op/s; 1871/214923 objects degraded (0.871%); 17744/214923 objects misplaced (8.256%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:48:33.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:33 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1871/214923 objects degraded (0.871%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:33 smithi159 ceph-mon[45336]: pgmap v2188: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 419 op/s; 1871/214923 objects degraded (0.871%); 17744/214923 objects misplaced (8.256%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:48:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:33 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1871/214923 objects degraded (0.871%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:33.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:33 smithi049 ceph-mon[40087]: pgmap v2188: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 419 op/s; 1871/214923 objects degraded (0.871%); 17744/214923 objects misplaced (8.256%); 5.1 MiB/s, 82 objects/s recovering 2024-04-15T18:48:33.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:33 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1871/214923 objects degraded (0.871%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:34.265 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:34.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:34.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:48:34.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:34.430 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:34.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:34 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:34.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:34 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:34.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:34 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:34.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:34 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:34.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:34 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:34.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:34 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:34.706 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:34.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:34.820 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:34.871 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:34.903 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:34.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:35.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:35.046 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:35.368 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:35.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:35 smithi082 ceph-mon[44206]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1940/207897 objects degraded (0.933%); 17187/207897 objects misplaced (8.267%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:48:35.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:35 smithi082 ceph-mon[44206]: osdmap e838: 12 total, 12 up, 12 in 2024-04-15T18:48:35.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:35 smithi159 ceph-mon[45336]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1940/207897 objects degraded (0.933%); 17187/207897 objects misplaced (8.267%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:48:35.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:35 smithi159 ceph-mon[45336]: osdmap e838: 12 total, 12 up, 12 in 2024-04-15T18:48:35.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:35 smithi049 ceph-mon[40087]: pgmap v2189: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 1940/207897 objects degraded (0.933%); 17187/207897 objects misplaced (8.267%); 8.6 MiB/s, 138 objects/s recovering 2024-04-15T18:48:35.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:35 smithi049 ceph-mon[40087]: osdmap e838: 12 total, 12 up, 12 in 2024-04-15T18:48:36.284 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:48:36.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:48:36.285 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-15T18:48:36.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:48:37.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:37 smithi082 ceph-mon[44206]: pgmap v2191: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1960/206253 objects degraded (0.950%); 17020/206253 objects misplaced (8.252%); 7.2 MiB/s, 115 objects/s recovering 2024-04-15T18:48:37.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:37 smithi159 ceph-mon[45336]: pgmap v2191: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1960/206253 objects degraded (0.950%); 17020/206253 objects misplaced (8.252%); 7.2 MiB/s, 115 objects/s recovering 2024-04-15T18:48:37.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:37 smithi049 ceph-mon[40087]: pgmap v2191: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1960/206253 objects degraded (0.950%); 17020/206253 objects misplaced (8.252%); 7.2 MiB/s, 115 objects/s recovering 2024-04-15T18:48:38.131 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:38.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:38.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:38.273 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:38.294 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:38.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:38.402 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:38.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:38 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1960/206253 objects degraded (0.950%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:38.472 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:38.504 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:38.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:38.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:38 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1960/206253 objects degraded (0.950%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:38.614 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:38.643 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:38.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:38 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1960/206253 objects degraded (0.950%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:38.664 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:38.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:38.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:38.808 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:38.835 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:38.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:38.940 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:38.989 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:39.026 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:39.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:39.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:39.167 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:39.193 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:39.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:39.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:39.329 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:39.357 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:39.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:39.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:39.491 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:39 smithi082 ceph-mon[44206]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 1976/202863 objects degraded (0.974%); 16626/202863 objects misplaced (8.196%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:39.494 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:39.525 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:39.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:39.632 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:39.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:39 smithi159 ceph-mon[45336]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 1976/202863 objects degraded (0.974%); 16626/202863 objects misplaced (8.196%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:39.660 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:39.686 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:39.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:39.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:39 smithi049 ceph-mon[40087]: pgmap v2192: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 1976/202863 objects degraded (0.974%); 16626/202863 objects misplaced (8.196%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:39.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:48:39.836 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:40.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:40.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:40.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:40.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:40.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:40.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:41.096 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:48:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:41.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:48:41.203 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:41.257 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:41.289 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:41.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:41.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:41 smithi049 ceph-mon[40087]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2063/197619 objects degraded (1.044%); 16259/197619 objects misplaced (8.227%); 9.5 MiB/s, 151 objects/s recovering 2024-04-15T18:48:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:41 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:41 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:41.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:41 smithi049 ceph-mon[40087]: osdmap e839: 12 total, 12 up, 12 in 2024-04-15T18:48:41.487 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:48:41.615 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:48:41.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:48:41.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:41 smithi159 ceph-mon[45336]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2063/197619 objects degraded (1.044%); 16259/197619 objects misplaced (8.227%); 9.5 MiB/s, 151 objects/s recovering 2024-04-15T18:48:41.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:41 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:41.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:41 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:41.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:41 smithi159 ceph-mon[45336]: osdmap e839: 12 total, 12 up, 12 in 2024-04-15T18:48:41.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:48:41.702 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:48:41.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:48:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:48:41.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:41 smithi082 ceph-mon[44206]: pgmap v2193: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2063/197619 objects degraded (1.044%); 16259/197619 objects misplaced (8.227%); 9.5 MiB/s, 151 objects/s recovering 2024-04-15T18:48:41.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:41 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:41.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:41 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:41.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:41 smithi082 ceph-mon[44206]: osdmap e839: 12 total, 12 up, 12 in 2024-04-15T18:48:41.798 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:48:41.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:48:41.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:48:41.931 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:48:41.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:48:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:48:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:48:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:48:42.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3844042070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:42 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3844042070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:42 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3844042070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:42.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:42 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.359 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:43.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:43.466 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:48:43.514 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:43.545 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:48:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:43.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:48:43.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: pgmap v2195: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 452 op/s; 2063/197037 objects degraded (1.047%); 16259/197037 objects misplaced (8.252%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:48:43.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:43 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2063/197037 objects degraded (1.047%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:43.649 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:43.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.673 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: pgmap v2195: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 452 op/s; 2063/197037 objects degraded (1.047%); 16259/197037 objects misplaced (8.252%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:48:43.674 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:43 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2063/197037 objects degraded (1.047%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:43.675 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:43.694 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:43.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: pgmap v2195: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 452 op/s; 2063/197037 objects degraded (1.047%); 16259/197037 objects misplaced (8.252%); 6.3 MiB/s, 100 objects/s recovering 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:48:43.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:43 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2063/197037 objects degraded (1.047%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:43.797 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:43.844 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:43.872 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:43.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:43.976 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:44.006 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:44.035 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:44.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:44.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:44.211 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:44.246 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:44.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:44.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:44 smithi049 ceph-mon[40087]: osdmap e840: 12 total, 12 up, 12 in 2024-04-15T18:48:44.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:48:44.427 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:44.443 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:44.591 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:48:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:44.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:48:44.620 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:48:44.620 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:48:44.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:44 smithi159 ceph-mon[45336]: osdmap e840: 12 total, 12 up, 12 in 2024-04-15T18:48:44.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:48:44.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:44.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:44 smithi082 ceph-mon[44206]: osdmap e840: 12 total, 12 up, 12 in 2024-04-15T18:48:45.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:48:45.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:45 smithi049 ceph-mon[40087]: pgmap v2197: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2145/189549 objects degraded (1.132%); 15672/189549 objects misplaced (8.268%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:48:45.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3273760579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:45.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:45 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:45.379 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:48:45.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:45 smithi159 ceph-mon[45336]: pgmap v2197: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2145/189549 objects degraded (1.132%); 15672/189549 objects misplaced (8.268%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:48:45.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3273760579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:45.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:45 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:45.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:45 smithi082 ceph-mon[44206]: pgmap v2197: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 2145/189549 objects degraded (1.132%); 15672/189549 objects misplaced (8.268%); 9.1 MiB/s, 145 objects/s recovering 2024-04-15T18:48:45.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3273760579' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:45.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:45 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:48:46.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:46 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:46.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:46 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:48:46.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:46 smithi159 ceph-mon[45336]: osdmap e841: 12 total, 12 up, 12 in 2024-04-15T18:48:46.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:46 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:46.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:46 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:48:46.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:46 smithi049 ceph-mon[40087]: osdmap e841: 12 total, 12 up, 12 in 2024-04-15T18:48:46.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:46 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:48:46.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:46 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:48:46.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:46 smithi082 ceph-mon[44206]: osdmap e841: 12 total, 12 up, 12 in 2024-04-15T18:48:47.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:47 smithi159 ceph-mon[45336]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 2246/188349 objects degraded (1.192%); 15453/188349 objects misplaced (8.204%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:48:47.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:47 smithi049 ceph-mon[40087]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 2246/188349 objects degraded (1.192%); 15453/188349 objects misplaced (8.204%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:48:47.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:47 smithi082 ceph-mon[44206]: pgmap v2199: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 2246/188349 objects degraded (1.192%); 15453/188349 objects misplaced (8.204%); 6.5 MiB/s, 103 objects/s recovering 2024-04-15T18:48:48.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:48 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2246/188349 objects degraded (1.192%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:48.652 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:48:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:48.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:48:48.659 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:48 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2246/188349 objects degraded (1.192%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:48.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:48.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:48 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2246/188349 objects degraded (1.192%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:48.795 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:48.821 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:48.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:48.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:48.956 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:48.983 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:48.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:49.089 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:49.118 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:49.141 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:49.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:49.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:48:49.245 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:49.271 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:49.299 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:49.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:49.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:49.437 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:49 smithi082 ceph-mon[44206]: pgmap v2200: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 2246/186753 objects degraded (1.203%); 15453/186753 objects misplaced (8.275%); 5.2 MiB/s, 83 objects/s recovering 2024-04-15T18:48:49.437 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:49.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:49 smithi049 ceph-mon[40087]: pgmap v2200: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 2246/186753 objects degraded (1.203%); 15453/186753 objects misplaced (8.275%); 5.2 MiB/s, 83 objects/s recovering 2024-04-15T18:48:49.462 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:49.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:49.569 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:49.603 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:49.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:49.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:49 smithi159 ceph-mon[45336]: pgmap v2200: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 2246/186753 objects degraded (1.203%); 15453/186753 objects misplaced (8.275%); 5.2 MiB/s, 83 objects/s recovering 2024-04-15T18:48:49.734 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:49.781 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:49.817 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:49.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:49.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:49.965 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:50.001 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:50.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:50.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:50.124 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:50.173 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:50.296 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:50.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:50.400 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:50.429 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:50.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:50.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:50.602 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:50.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:50.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:50.793 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:50.823 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:50.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:50.833 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:50.930 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:51.010 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.053 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:51.162 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:48:51.220 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.271 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:48:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:48:51.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:48:51.415 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.436 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:48:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:48:51.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:51.571 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.592 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:51.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:48:51.651 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:51 smithi159 ceph-mon[45336]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2316/179646 objects degraded (1.289%); 14728/179646 objects misplaced (8.198%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:48:51.686 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:51 smithi049 ceph-mon[40087]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2316/179646 objects degraded (1.289%); 14728/179646 objects misplaced (8.198%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:48:51.695 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:48:51.742 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.773 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:48:51.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:51 smithi082 ceph-mon[44206]: pgmap v2201: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 2316/179646 objects degraded (1.289%); 14728/179646 objects misplaced (8.198%); 10 MiB/s, 159 objects/s recovering 2024-04-15T18:48:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:48:51.878 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:51.907 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:51.928 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:51.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:52.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:48:52.100 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:52.129 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:48:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:52.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:48:52.233 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:52.261 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:52.281 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:52.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:52.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:48:52.461 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:52.750 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:48:52.750 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 841, 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:48:45.324806+0000", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:46:33.707597+0000", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:47:20.193442+0000", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:48:52.751 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 31, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:48:52.752 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:48:52.753 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:48:52.754 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:48:52.755 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:48:52.756 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:48:52.757 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:48:52.758 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:45:48.486814+0000", 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:48:52.759 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:48:52.760 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "794", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:48:52.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:48:52.762 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:48:52.763 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8799999952316284, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-15T18:48:52.764 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.765 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 671, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.766 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.768 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.769 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.771 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.772 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:48:52.773 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.775 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:48:52.776 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.777 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:48:52.778 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.779 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 778, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.780 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.781 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.782 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:48:52.783 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.785 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:48:52.786 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 675, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.787 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:48:52.788 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.791 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.793 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.794 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.795 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.796 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.797 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.798 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.799 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.800 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:48:52.801 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:48:52.802 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:52.803 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:48:52.804 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.807 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:48:52.808 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.809 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.810 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 676, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.811 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.812 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.813 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:48:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:48:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 823, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.818 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.819 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.823 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:48:52.825 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.829 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.830 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:48:52.831 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.832 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:48:52.833 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.835 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.836 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:48:52.837 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.839 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.840 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:48:52.841 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.844 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.845 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:48:52.847 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.848 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:48:52.849 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 94, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 733, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 807, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 820, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 734, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:48:52.852 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.853 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/905169792", 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/905169792", 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/905169792", 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/905169792", 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:48:52.856 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:42.386174+0000", 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.857 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:40.053179+0000", 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:40.027037+0000", 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:48:52.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:39.756500+0000", 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:48:52.859 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:51.456661+0000", 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.860 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:49.958135+0000", 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:19.018160+0000", 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:48:52.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:33.398820+0000", 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.862 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:23.980112+0000", 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.863 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:47:56.756529+0000", 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:30.291194+0000", 2024-04-15T18:48:52.864 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:02.524072+0000", 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 733 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:48:52.865 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 5, 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:48:52.866 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.0", 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.5", 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-04-15T18:48:52.867 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.9", 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-04-15T18:48:52.868 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:52.869 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:48:52.870 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:48:52.871 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:48:53.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:53 smithi159 ceph-mon[45336]: pgmap v2202: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2316/179061 objects degraded (1.293%); 14728/179061 objects misplaced (8.225%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:48:53.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:53 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2316/179061 objects degraded (1.293%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:53.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/367929548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:53.668 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:48:53.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:53 smithi049 ceph-mon[40087]: pgmap v2202: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2316/179061 objects degraded (1.293%); 14728/179061 objects misplaced (8.225%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:48:53.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:53 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2316/179061 objects degraded (1.293%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:53.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/367929548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:53.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:48:53.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:53.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:53 smithi082 ceph-mon[44206]: pgmap v2202: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 2316/179061 objects degraded (1.293%); 14728/179061 objects misplaced (8.225%); 5.7 MiB/s, 91 objects/s recovering 2024-04-15T18:48:53.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:53 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2316/179061 objects degraded (1.293%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:53.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/367929548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:48:53.805 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:53.832 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:53.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:53.936 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:48:53.970 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:54.000 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:48:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:54.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:48:54.107 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:48:54.168 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:54.214 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:48:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:54.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:48:54.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:54.357 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:55.332 INFO:tasks.thrashosds.thrasher:j is {'epoch': 841, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:48:45.324806+0000', 'last_up_change': '2024-04-15T18:46:33.707597+0000', 'last_in_change': '2024-04-15T18:47:20.193442+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 31, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-04-15T18:45:48.486814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '794', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 671, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 778, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 675, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 791, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 676, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 94, 'last_clean_end': 733, 'up_from': 807, 'up_thru': 820, 'down_at': 734, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 905169792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 905169792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 905169792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 905169792}]}, 'public_addr': '172.21.15.159:6825/905169792', 'cluster_addr': '172.21.15.159:6827/905169792', 'heartbeat_back_addr': '172.21.15.159:6831/905169792', 'heartbeat_front_addr': '172.21.15.159:6829/905169792', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:42.386174+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:40.053179+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:40.027037+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:39.756500+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:51.456661+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:49.958135+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:19.018160+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:33.398820+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:23.980112+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:47:56.756529+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:30.291194+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:02.524072+0000', 'dead_epoch': 733}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '9.d', 'mappings': [{'from': 5, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6]}, {'pgid': '9.5', 'osds': [1, 5, 7]}, {'pgid': '9.8', 'osds': [9, 5]}, {'pgid': '9.9', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:48:55.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:48:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:55 smithi159 ceph-mon[45336]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 2398/171756 objects degraded (1.396%); 14139/171756 objects misplaced (8.232%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:48:55.703 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:55 smithi049 ceph-mon[40087]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 2398/171756 objects degraded (1.396%); 14139/171756 objects misplaced (8.232%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:48:55.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:55 smithi082 ceph-mon[44206]: pgmap v2203: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 2398/171756 objects degraded (1.396%); 14139/171756 objects misplaced (8.232%); 8.0 MiB/s, 127 objects/s recovering 2024-04-15T18:48:56.649 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:56 smithi159 ceph-mon[45336]: 9.c scrub starts 2024-04-15T18:48:56.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:56 smithi049 ceph-mon[40087]: 9.c scrub starts 2024-04-15T18:48:56.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:56 smithi082 ceph-mon[44206]: 9.c scrub starts 2024-04-15T18:48:57.644 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:57 smithi159 ceph-mon[45336]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2402/170127 objects degraded (1.412%); 13978/170127 objects misplaced (8.216%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:57.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:57 smithi159 ceph-mon[45336]: 9.c scrub ok 2024-04-15T18:48:57.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:57 smithi049 ceph-mon[40087]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2402/170127 objects degraded (1.412%); 13978/170127 objects misplaced (8.216%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:57.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:57 smithi049 ceph-mon[40087]: 9.c scrub ok 2024-04-15T18:48:57.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:57 smithi082 ceph-mon[44206]: pgmap v2204: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 2402/170127 objects degraded (1.412%); 13978/170127 objects misplaced (8.216%); 7.9 MiB/s, 126 objects/s recovering 2024-04-15T18:48:57.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:57 smithi082 ceph-mon[44206]: 9.c scrub ok 2024-04-15T18:48:58.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:58 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2402/170127 objects degraded (1.412%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:58.638 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:58.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:58.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:58 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2402/170127 objects degraded (1.412%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:58.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:58.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:58 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2402/170127 objects degraded (1.412%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:48:58.781 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:58.806 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:58.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:58.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:48:58.957 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.103 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:48:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:48:59.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:48:59.241 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.268 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:48:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:48:59.373 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:59.403 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.424 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:59.473 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:48:59.473 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:48:59.473 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:48:59.474 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:48:59.474 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:48:59.515 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:48:59.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:48:59.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:59 smithi082 ceph-mon[44206]: pgmap v2205: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 541 op/s; 2402/168837 objects degraded (1.423%); 13978/168837 objects misplaced (8.279%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:48:59.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1326109151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:59.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:48:59 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:48:59.530 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:48:59.530 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:48:59.556 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.566 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:48:59.586 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:48:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:48:59.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:48:59.642 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:48:59.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:59 smithi159 ceph-mon[45336]: pgmap v2205: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 541 op/s; 2402/168837 objects degraded (1.423%); 13978/168837 objects misplaced (8.279%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:48:59.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1326109151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:59.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.643 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:48:59 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:48:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:48:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:48:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:48:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:48:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:48:59.684 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:48:59.691 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:59.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:59 smithi049 ceph-mon[40087]: pgmap v2205: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 541 op/s; 2402/168837 objects degraded (1.423%); 13978/168837 objects misplaced (8.279%); 6.9 MiB/s, 109 objects/s recovering 2024-04-15T18:48:59.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1326109151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:48:59.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:48:59 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:48:59.734 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.789 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:48:59.895 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:48:59.941 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:48:59.971 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:48:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:48:59.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:00.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:00.139 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:00.169 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:00.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:00.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:00.578 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:00 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:00.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:00.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.712 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.713 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.713 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:00 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.714 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:00.734 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:00.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:00.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:00.874 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:00 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:00.897 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:00.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:01.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:01.036 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.080 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:01.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:01.218 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.241 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:01.345 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:01.403 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.433 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:01.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:01 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:01.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:01 smithi049 ceph-mon[40087]: pgmap v2206: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2553/161295 objects degraded (1.583%); 13155/161295 objects misplaced (8.156%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:49:01.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:01.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:01 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:01 smithi082 ceph-mon[44206]: pgmap v2206: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2553/161295 objects degraded (1.583%); 13155/161295 objects misplaced (8.156%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:49:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:01.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:01.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:01.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.592 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:01.599 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:01.695 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:49:01.731 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.755 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:49:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:49:01.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:01.891 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:01.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:01 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:01.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:01 smithi159 ceph-mon[45336]: pgmap v2206: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2553/161295 objects degraded (1.583%); 13155/161295 objects misplaced (8.156%); 10 MiB/s, 165 objects/s recovering 2024-04-15T18:49:01.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:01.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:01.918 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:01.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:02.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:02.065 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:02.169 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:02.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:02.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:02.315 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:02.576 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:02.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:02 smithi049 ceph-mon[40087]: osdmap e842: 12 total, 12 up, 12 in 2024-04-15T18:49:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:02.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:02.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:02.728 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:02.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:02 smithi082 ceph-mon[44206]: osdmap e842: 12 total, 12 up, 12 in 2024-04-15T18:49:02.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:02 smithi159 ceph-mon[45336]: osdmap e842: 12 total, 12 up, 12 in 2024-04-15T18:49:03.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:03.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:03.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:49:03.410 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:03.478 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:03.509 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:03.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:03.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:03.657 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:03.685 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:03.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:03.694 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:03 smithi159 ceph-mon[45336]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 2553/160680 objects degraded (1.589%); 13155/160680 objects misplaced (8.187%); 8.3 MiB/s, 133 objects/s recovering 2024-04-15T18:49:03.694 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:03 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2553/160680 objects degraded (1.589%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:03.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:03 smithi049 ceph-mon[40087]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 2553/160680 objects degraded (1.589%); 13155/160680 objects misplaced (8.187%); 8.3 MiB/s, 133 objects/s recovering 2024-04-15T18:49:03.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:03 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2553/160680 objects degraded (1.589%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:03 smithi082 ceph-mon[44206]: pgmap v2208: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 2553/160680 objects degraded (1.589%); 13155/160680 objects misplaced (8.187%); 8.3 MiB/s, 133 objects/s recovering 2024-04-15T18:49:03.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:03 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2553/160680 objects degraded (1.589%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:03.795 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:49:03.830 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:03.859 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:49:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:03.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:49:03.971 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:04.024 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:04.069 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:04.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:04.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:04.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:49:04.211 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:04.264 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:04.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:04.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:04.406 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:04.426 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:04.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:04.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:04.587 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:04.627 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:04.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:04.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:04.768 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:05.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:05 smithi049 ceph-mon[40087]: pgmap v2209: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 2651/153423 objects degraded (1.728%); 12557/153423 objects misplaced (8.185%); 8.3 MiB/s, 132 objects/s recovering 2024-04-15T18:49:05.727 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:05.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:05.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:05 smithi082 ceph-mon[44206]: pgmap v2209: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 2651/153423 objects degraded (1.728%); 12557/153423 objects misplaced (8.185%); 8.3 MiB/s, 132 objects/s recovering 2024-04-15T18:49:05.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:05.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:05 smithi159 ceph-mon[45336]: pgmap v2209: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 2651/153423 objects degraded (1.728%); 12557/153423 objects misplaced (8.185%); 8.3 MiB/s, 132 objects/s recovering 2024-04-15T18:49:05.905 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:07.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:07 smithi049 ceph-mon[40087]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2669/151845 objects degraded (1.758%); 12360/151845 objects misplaced (8.140%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:07.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:07 smithi049 ceph-mon[40087]: 9.b scrub starts 2024-04-15T18:49:07.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:07 smithi082 ceph-mon[44206]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2669/151845 objects degraded (1.758%); 12360/151845 objects misplaced (8.140%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:07.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:07 smithi082 ceph-mon[44206]: 9.b scrub starts 2024-04-15T18:49:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:07 smithi159 ceph-mon[45336]: pgmap v2210: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 2669/151845 objects degraded (1.758%); 12360/151845 objects misplaced (8.140%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:07.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:07 smithi159 ceph-mon[45336]: 9.b scrub starts 2024-04-15T18:49:08.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:08 smithi049 ceph-mon[40087]: 9.b scrub ok 2024-04-15T18:49:08.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2669/151845 objects degraded (1.758%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:08.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:08 smithi082 ceph-mon[44206]: 9.b scrub ok 2024-04-15T18:49:08.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2669/151845 objects degraded (1.758%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:08.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:08 smithi159 ceph-mon[45336]: 9.b scrub ok 2024-04-15T18:49:08.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2669/151845 objects degraded (1.758%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:08.992 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:08.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:09.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:09.138 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:09.186 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:09.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:09.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:09.332 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:09.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:09 smithi049 ceph-mon[40087]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 2669/150747 objects degraded (1.771%); 12360/150747 objects misplaced (8.199%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:09.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:09 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:09.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:09 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:09.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:09 smithi082 ceph-mon[44206]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 2669/150747 objects degraded (1.771%); 12360/150747 objects misplaced (8.199%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:09.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:09 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:09.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:09 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:09.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:09 smithi159 ceph-mon[45336]: pgmap v2211: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 2669/150747 objects degraded (1.771%); 12360/150747 objects misplaced (8.199%); 8.4 MiB/s, 135 objects/s recovering 2024-04-15T18:49:09.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:09 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:09.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:09 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:10.073 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:10.089 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:10.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:10.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:10.233 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:10.489 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:10.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:10.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:10.667 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:10.760 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:10.766 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:49:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:10.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:10 smithi049 ceph-mon[40087]: osdmap e843: 12 total, 12 up, 12 in 2024-04-15T18:49:10.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:10.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:10 smithi082 ceph-mon[44206]: osdmap e843: 12 total, 12 up, 12 in 2024-04-15T18:49:10.783 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:49:10.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:10.905 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:10 smithi159 ceph-mon[45336]: osdmap e843: 12 total, 12 up, 12 in 2024-04-15T18:49:10.943 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:11.138 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:11.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:11.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:11.369 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:11.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2743/143583 objects degraded (1.910%); 11577/143583 objects misplaced (8.063%); 10 MiB/s, 166 objects/s recovering 2024-04-15T18:49:11.491 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:11.492 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3769603521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.492 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.492 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:11.492 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:11 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:11.507 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2743/143583 objects degraded (1.910%); 11577/143583 objects misplaced (8.063%); 10 MiB/s, 166 objects/s recovering 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3769603521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:11.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:11 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: pgmap v2213: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2743/143583 objects degraded (1.910%); 11577/143583 objects misplaced (8.063%); 10 MiB/s, 166 objects/s recovering 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3769603521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:11.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:11 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:12.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:12 smithi049 ceph-mon[40087]: from='client.17526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:12.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:12 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:49:12.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:12 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:12.760 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:12 smithi049 ceph-mon[40087]: osdmap e844: 12 total, 12 up, 12 in 2024-04-15T18:49:12.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:12 smithi082 ceph-mon[44206]: from='client.17526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:12.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:12 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:49:12.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:12 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:12.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:12 smithi082 ceph-mon[44206]: osdmap e844: 12 total, 12 up, 12 in 2024-04-15T18:49:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:12 smithi159 ceph-mon[45336]: from='client.17526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:12 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:49:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:12 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:12.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:12 smithi159 ceph-mon[45336]: osdmap e844: 12 total, 12 up, 12 in 2024-04-15T18:49:13.513 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:13 smithi049 ceph-mon[40087]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 432 op/s; 2743/143052 objects degraded (1.917%); 11577/143052 objects misplaced (8.093%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:49:13.514 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:13 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2743/143052 objects degraded (1.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:13.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:13 smithi082 ceph-mon[44206]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 432 op/s; 2743/143052 objects degraded (1.917%); 11577/143052 objects misplaced (8.093%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:49:13.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:13 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2743/143052 objects degraded (1.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:13.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:13 smithi159 ceph-mon[45336]: pgmap v2215: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 432 op/s; 2743/143052 objects degraded (1.917%); 11577/143052 objects misplaced (8.093%); 6.9 MiB/s, 110 objects/s recovering 2024-04-15T18:49:13.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:13 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2743/143052 objects degraded (1.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:14.341 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2213,"stamp":"2024-04-15T18:49:09.735641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3137012280,"num_objects":47861,"num_object_clones":0,"num_object_copies":143583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2743,"num_objects_misplaced":11577,"num_objects_unfound":0,"num_objects_dirty":47861,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":169584,"num_write_kb":6955259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":822190880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":17023436,"kb_used_data":10065764,"kb_used_omap":430,"kb_used_meta":6956881,"kb_avail":1107918388,"statfs":{"total":1151940427776,"available":1134508429312,"internally_reserved":0,"allocated":10307342336,"data_stored":10303679788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":440519,"internal_metadata":7123846969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,8,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-373489664,"num_objects":-5699,"num_object_clones":0,"num_object_copies":-17097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":190,"num_objects_misplaced":-1578,"num_objects_unfound":0,"num_objects_dirty":-5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5699,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":90701824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.318749"},"pg_stats":[{"pgid":"9.f","version":"842'10478","reported_seq":10655,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:08.089172+0000","last_change":"2024-04-15T18:47:05.724961+0000","last_active":"2024-04-15T18:49:08.089172+0000","last_peered":"2024-04-15T18:49:08.089172+0000","last_clean":"2024-04-15T18:49:08.089172+0000","last_became_active":"2024-04-15T18:45:49.841805+0000","last_became_peered":"2024-04-15T18:45:49.841805+0000","last_unstale":"2024-04-15T18:49:08.089172+0000","last_undegraded":"2024-04-15T18:49:08.089172+0000","last_fullsized":"2024-04-15T18:49:08.089172+0000","mapping_epoch":791,"log_start":"797'400","ondisk_log_start":"797'400","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5699","last_scrub_stamp":"2024-04-15T18:47:05.724825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.724825+0000","objects_scrubbed":5653,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10478,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"842'10712","reported_seq":10889,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:06.720169+0000","last_change":"2024-04-15T18:46:57.710845+0000","last_active":"2024-04-15T18:49:06.720169+0000","last_peered":"2024-04-15T18:49:06.720169+0000","last_clean":"2024-04-15T18:49:06.720169+0000","last_became_active":"2024-04-15T18:45:49.844972+0000","last_became_peered":"2024-04-15T18:45:49.844972+0000","last_unstale":"2024-04-15T18:49:06.720169+0000","last_undegraded":"2024-04-15T18:49:06.720169+0000","last_fullsized":"2024-04-15T18:49:06.720169+0000","mapping_epoch":791,"log_start":"797'700","ondisk_log_start":"797'700","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5375","last_scrub_stamp":"2024-04-15T18:46:57.710704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.710704+0000","objects_scrubbed":5329,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198967296,"num_objects":3036,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10712,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"842'10609","reported_seq":10735,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:06.704489+0000","last_change":"2024-04-15T18:47:07.167264+0000","last_active":"2024-04-15T18:49:06.704489+0000","last_peered":"2024-04-15T18:49:06.704489+0000","last_clean":"2024-04-15T18:49:06.704489+0000","last_became_active":"2024-04-15T18:45:49.844992+0000","last_became_peered":"2024-04-15T18:45:49.844992+0000","last_unstale":"2024-04-15T18:49:06.704489+0000","last_undegraded":"2024-04-15T18:49:06.704489+0000","last_fullsized":"2024-04-15T18:49:06.704489+0000","mapping_epoch":791,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":899,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195544,"num_objects":3071,"num_object_clones":0,"num_object_copies":9213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10609,"num_write_kb":437697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"842'10557","reported_seq":10778,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:07.665273+0000","last_change":"2024-04-15T18:48:56.919239+0000","last_active":"2024-04-15T18:49:07.665273+0000","last_peered":"2024-04-15T18:49:07.665273+0000","last_clean":"2024-04-15T18:49:07.665273+0000","last_became_active":"2024-04-15T18:45:49.843864+0000","last_became_peered":"2024-04-15T18:45:49.843864+0000","last_unstale":"2024-04-15T18:49:07.665273+0000","last_undegraded":"2024-04-15T18:49:07.665273+0000","last_fullsized":"2024-04-15T18:49:07.665273+0000","mapping_epoch":791,"log_start":"797'500","ondisk_log_start":"797'500","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'10164","last_scrub_stamp":"2024-04-15T18:48:56.918932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:48:56.918932+0000","objects_scrubbed":3417,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:09.262458+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196149248,"num_objects":2993,"num_object_clones":0,"num_object_copies":8979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"842'10341","reported_seq":10525,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:05.398634+0000","last_change":"2024-04-15T18:47:35.941528+0000","last_active":"2024-04-15T18:49:05.398634+0000","last_peered":"2024-04-15T18:49:05.398634+0000","last_clean":"2024-04-15T18:49:05.398634+0000","last_became_active":"2024-04-15T18:45:49.841997+0000","last_became_peered":"2024-04-15T18:45:49.841997+0000","last_unstale":"2024-04-15T18:49:05.398634+0000","last_undegraded":"2024-04-15T18:49:05.398634+0000","last_fullsized":"2024-04-15T18:49:05.398634+0000","mapping_epoch":791,"log_start":"796'300","ondisk_log_start":"796'300","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"826'7037","last_scrub_stamp":"2024-04-15T18:47:35.941267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:35.941267+0000","objects_scrubbed":6333,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196149248,"num_objects":2993,"num_object_clones":0,"num_object_copies":8979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"842'10582","reported_seq":10755,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:08.081408+0000","last_change":"2024-04-15T18:46:55.950320+0000","last_active":"2024-04-15T18:49:08.081408+0000","last_peered":"2024-04-15T18:49:08.081408+0000","last_clean":"2024-04-15T18:49:08.081408+0000","last_became_active":"2024-04-15T18:45:49.843914+0000","last_became_peered":"2024-04-15T18:45:49.843914+0000","last_unstale":"2024-04-15T18:49:08.081408+0000","last_undegraded":"2024-04-15T18:49:08.081408+0000","last_fullsized":"2024-04-15T18:49:08.081408+0000","mapping_epoch":791,"log_start":"797'500","ondisk_log_start":"797'500","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5238","last_scrub_stamp":"2024-04-15T18:46:55.950094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:55.950094+0000","objects_scrubbed":5218,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194641920,"num_objects":2970,"num_object_clones":0,"num_object_copies":8910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"842'10648","reported_seq":16497,"reported_epoch":842,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:08.894567+0000","last_change":"2024-04-15T18:47:30.615221+0000","last_active":"2024-04-15T18:49:08.894567+0000","last_peered":"2024-04-15T18:49:08.894567+0000","last_clean":"2024-04-15T18:47:20.186030+0000","last_became_active":"2024-04-15T18:47:22.838451+0000","last_became_peered":"2024-04-15T18:47:22.838451+0000","last_unstale":"2024-04-15T18:49:08.894567+0000","last_undegraded":"2024-04-15T18:47:30.608553+0000","last_fullsized":"2024-04-15T18:47:22.203522+0000","mapping_epoch":821,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194117632,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":977,"num_objects_misplaced":2877,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10648,"num_write_kb":435520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2866,"num_bytes_recovered":187367424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,5],"avail_no_missing":["10","3","5"],"object_location_counts":[{"shards":"3,5,10","objects":2877},{"shards":"5,10","objects":-892}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"842'10607","reported_seq":10720,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:08.938297+0000","last_change":"2024-04-15T18:45:49.841390+0000","last_active":"2024-04-15T18:49:08.938297+0000","last_peered":"2024-04-15T18:49:08.938297+0000","last_clean":"2024-04-15T18:49:08.938297+0000","last_became_active":"2024-04-15T18:45:49.841033+0000","last_became_peered":"2024-04-15T18:45:49.841033+0000","last_unstale":"2024-04-15T18:49:08.938297+0000","last_undegraded":"2024-04-15T18:49:08.938297+0000","last_fullsized":"2024-04-15T18:49:08.938297+0000","mapping_epoch":791,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":8787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"842'10625","reported_seq":10738,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:08.037842+0000","last_change":"2024-04-15T18:45:49.841903+0000","last_active":"2024-04-15T18:49:08.037842+0000","last_peered":"2024-04-15T18:49:08.037842+0000","last_clean":"2024-04-15T18:49:08.037842+0000","last_became_active":"2024-04-15T18:45:49.841652+0000","last_became_peered":"2024-04-15T18:45:49.841652+0000","last_unstale":"2024-04-15T18:49:08.037842+0000","last_undegraded":"2024-04-15T18:49:08.037842+0000","last_fullsized":"2024-04-15T18:49:08.037842+0000","mapping_epoch":791,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193003520,"num_objects":2945,"num_object_clones":0,"num_object_copies":8835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"842'10535","reported_seq":18195,"reported_epoch":842,"state":"active+recovering+remapped","last_fresh":"2024-04-15T18:49:07.786019+0000","last_change":"2024-04-15T18:47:22.914564+0000","last_active":"2024-04-15T18:49:07.786019+0000","last_peered":"2024-04-15T18:49:07.786019+0000","last_clean":"2024-04-15T18:47:20.156259+0000","last_became_active":"2024-04-15T18:47:22.844939+0000","last_became_peered":"2024-04-15T18:47:22.844939+0000","last_unstale":"2024-04-15T18:49:07.786019+0000","last_undegraded":"2024-04-15T18:49:07.786019+0000","last_fullsized":"2024-04-15T18:49:07.786019+0000","mapping_epoch":821,"log_start":"797'500","ondisk_log_start":"797'500","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191823872,"num_objects":2927,"num_object_clones":0,"num_object_copies":8781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2873,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":430784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3773,"num_bytes_recovered":247136256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[1,5,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":54}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.4","version":"842'10763","reported_seq":10935,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:07.769095+0000","last_change":"2024-04-15T18:46:57.262366+0000","last_active":"2024-04-15T18:49:07.769095+0000","last_peered":"2024-04-15T18:49:07.769095+0000","last_clean":"2024-04-15T18:49:07.769095+0000","last_became_active":"2024-04-15T18:45:49.844745+0000","last_became_peered":"2024-04-15T18:45:49.844745+0000","last_unstale":"2024-04-15T18:49:07.769095+0000","last_undegraded":"2024-04-15T18:49:07.769095+0000","last_fullsized":"2024-04-15T18:49:07.769095+0000","mapping_epoch":791,"log_start":"797'700","ondisk_log_start":"797'700","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5319","last_scrub_stamp":"2024-04-15T18:46:57.262149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.262149+0000","objects_scrubbed":5273,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":9081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":441280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"842'10489","reported_seq":10669,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:07.893553+0000","last_change":"2024-04-15T18:47:05.437565+0000","last_active":"2024-04-15T18:49:07.893553+0000","last_peered":"2024-04-15T18:49:07.893553+0000","last_clean":"2024-04-15T18:49:07.893553+0000","last_became_active":"2024-04-15T18:45:49.842017+0000","last_became_peered":"2024-04-15T18:45:49.842017+0000","last_unstale":"2024-04-15T18:49:07.893553+0000","last_undegraded":"2024-04-15T18:49:07.893553+0000","last_fullsized":"2024-04-15T18:49:07.893553+0000","mapping_epoch":791,"log_start":"797'400","ondisk_log_start":"797'400","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5799","last_scrub_stamp":"2024-04-15T18:47:05.437371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.437371+0000","objects_scrubbed":5752,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195362816,"num_objects":2981,"num_object_clones":0,"num_object_copies":8943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"842'10591","reported_seq":10703,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:07.725635+0000","last_change":"2024-04-15T18:45:49.836537+0000","last_active":"2024-04-15T18:49:07.725635+0000","last_peered":"2024-04-15T18:49:07.725635+0000","last_clean":"2024-04-15T18:49:07.725635+0000","last_became_active":"2024-04-15T18:45:49.836251+0000","last_became_peered":"2024-04-15T18:45:49.836251+0000","last_unstale":"2024-04-15T18:49:07.725635+0000","last_undegraded":"2024-04-15T18:49:07.725635+0000","last_fullsized":"2024-04-15T18:49:07.725635+0000","mapping_epoch":791,"log_start":"797'500","ondisk_log_start":"797'500","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"842'10605","reported_seq":10782,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:07.885850+0000","last_change":"2024-04-15T18:47:01.868689+0000","last_active":"2024-04-15T18:49:07.885850+0000","last_peered":"2024-04-15T18:49:07.885850+0000","last_clean":"2024-04-15T18:49:07.885850+0000","last_became_active":"2024-04-15T18:45:50.180630+0000","last_became_peered":"2024-04-15T18:45:50.180630+0000","last_unstale":"2024-04-15T18:49:07.885850+0000","last_undegraded":"2024-04-15T18:49:07.885850+0000","last_fullsized":"2024-04-15T18:49:07.885850+0000","mapping_epoch":791,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'5601","last_scrub_stamp":"2024-04-15T18:47:01.868503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:01.868503+0000","objects_scrubbed":5580,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":9147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"842'10692","reported_seq":16909,"reported_epoch":842,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:07.781848+0000","last_change":"2024-04-15T18:47:32.874160+0000","last_active":"2024-04-15T18:49:07.781848+0000","last_peered":"2024-04-15T18:49:07.781848+0000","last_clean":"2024-04-15T18:47:20.171208+0000","last_became_active":"2024-04-15T18:47:22.829093+0000","last_became_peered":"2024-04-15T18:47:22.829093+0000","last_unstale":"2024-04-15T18:49:07.781848+0000","last_undegraded":"2024-04-15T18:47:32.870404+0000","last_fullsized":"2024-04-15T18:47:22.201052+0000","mapping_epoch":821,"log_start":"797'600","ondisk_log_start":"797'600","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":2933,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3050,"num_bytes_recovered":199032832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6],"avail_no_missing":["1","6","8"],"object_location_counts":[{"shards":"1,6","objects":-693},{"shards":"1,6,8","objects":2933}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"842'10569","reported_seq":16316,"reported_epoch":842,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:07.915191+0000","last_change":"2024-04-15T18:47:32.882544+0000","last_active":"2024-04-15T18:49:07.915191+0000","last_peered":"2024-04-15T18:49:07.915191+0000","last_clean":"2024-04-15T18:47:20.189472+0000","last_became_active":"2024-04-15T18:47:22.835987+0000","last_became_peered":"2024-04-15T18:47:22.835987+0000","last_unstale":"2024-04-15T18:49:07.915191+0000","last_undegraded":"2024-04-15T18:47:32.878238+0000","last_fullsized":"2024-04-15T18:47:22.203408+0000","mapping_epoch":821,"log_start":"797'500","ondisk_log_start":"797'500","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197984256,"num_objects":3021,"num_object_clones":0,"num_object_copies":9063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":978,"num_objects_misplaced":2894,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10569,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2815,"num_bytes_recovered":183894016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,5],"avail_no_missing":["9","5","7"],"object_location_counts":[{"shards":"5,7,9","objects":2894},{"shards":"5,9","objects":-851}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":2080,"reported_epoch":842,"state":"active+clean","last_fresh":"2024-04-15T18:49:01.423423+0000","last_change":"2024-04-15T18:47:24.773236+0000","last_active":"2024-04-15T18:49:01.423423+0000","last_peered":"2024-04-15T18:49:01.423423+0000","last_clean":"2024-04-15T18:49:01.423423+0000","last_became_active":"2024-04-15T18:47:24.772930+0000","last_became_peered":"2024-04-15T18:47:24.772930+0000","last_unstale":"2024-04-15T18:49:01.423423+0000","last_undegraded":"2024-04-15T18:49:01.423423+0000","last_fullsized":"2024-04-15T18:49:01.423423+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:46:52.053607+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:46:52.053607+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3136421912,"num_objects":47859,"num_object_clones":0,"num_object_copies":143577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2743,"num_objects_misplaced":11577,"num_objects_unfound":0,"num_objects_dirty":47859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169403,"num_write_kb":6952321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12504,"num_bytes_recovered":817430528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10181292032,"data_stored":10181279816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":48,"acting":45,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":807,"seq":3466038607904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":240144,"kb_used_data":77320,"kb_used_omap":144,"kb_used_meta":162671,"kb_avail":93505008,"statfs":{"total":95995035648,"available":95749128192,"internally_reserved":0,"allocated":79175680,"data_stored":78868810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147868,"internal_metadata":166575716},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187942,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1587144,"kb_used_data":946496,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92158008,"statfs":{"total":95995035648,"available":94369800192,"internally_reserved":0,"allocated":969211904,"data_stored":968908598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":656013760},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514988,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1916296,"kb_used_data":1154688,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":91828856,"statfs":{"total":95995035648,"available":94032748544,"internally_reserved":0,"allocated":1182400512,"data_stored":1182097206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874733,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280840,"kb_used_data":826240,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92464312,"statfs":{"total":95995035648,"available":94683455488,"internally_reserved":0,"allocated":846069760,"data_stored":845766454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169075,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2024788,"kb_used_data":1403596,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":91720364,"statfs":{"total":95995035648,"available":93921652736,"internally_reserved":0,"allocated":1437282304,"data_stored":1436971374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":636090826},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363921,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641104,"kb_used_data":401800,"kb_used_omap":111,"kb_used_meta":239184,"kb_avail":93104048,"statfs":{"total":95995035648,"available":95338545152,"internally_reserved":0,"allocated":411443200,"data_stored":411136350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":244925083},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575381,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2548236,"kb_used_data":1745156,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91196916,"statfs":{"total":95995035648,"available":93385641984,"internally_reserved":0,"allocated":1787039744,"data_stored":1786732366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681676,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063476,"statfs":{"total":95995035648,"available":95296999424,"internally_reserved":0,"allocated":786432,"data_stored":483126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524236,"kb_used_data":768,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220916,"statfs":{"total":95995035648,"available":95458217984,"internally_reserved":0,"allocated":786432,"data_stored":483126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609284,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2167752,"kb_used_data":1398720,"kb_used_omap":44,"kb_used_meta":768915,"kb_avail":91577400,"statfs":{"total":95995035648,"available":93775257600,"internally_reserved":0,"allocated":1432289280,"data_stored":1431985974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":787369514},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986691,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053896,"kb_used_data":386880,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92691256,"statfs":{"total":95995035648,"available":94915846144,"internally_reserved":0,"allocated":396165120,"data_stored":395861814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703585,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2357324,"kb_used_data":1723332,"kb_used_omap":4,"kb_used_meta":633979,"kb_avail":91387828,"statfs":{"total":95995035648,"available":93581135872,"internally_reserved":0,"allocated":1764691968,"data_stored":1764384590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":649195201},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1761873920,"data_stored":1761869848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1423769600,"data_stored":1423769600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":394526720,"data_stored":394526720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1743785984,"data_stored":1743781912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":407961600,"data_stored":407961600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1429934080,"data_stored":1429930008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":843776000,"data_stored":843776000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1177485312,"data_stored":1177485312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":959053824,"data_stored":959053824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":39124992,"data_stored":39124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:49:14.342 INFO:tasks.thrashosds.thrasher:Setting 9.7 to [6, 1, 2, 8, 7, 10] 2024-04-15T18:49:14.342 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '9.7', '6', '1', '2', '8', '7', '10'] 2024-04-15T18:49:14.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pg-upmap-items 9.7 6 1 2 8 7 10 2024-04-15T18:49:14.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=2738869719,config=1059521453,health=2030214595,kv=3102977241,logm=3810290515}) 2024-04-15T18:49:14.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959930492}) 2024-04-15T18:49:14.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3387927874}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068576816}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87325794}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770205200}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545026422}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:49:14.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2308001018}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050083571}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127561891}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800634988}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121235019}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003186586}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618228792}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511530211}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294789653}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847432028}) 2024-04-15T18:49:14.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297482198}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1138372748}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626759527}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959579734}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843472416}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1401312312}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264129123}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620110711}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1512641223}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412689250}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641833005}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551423793}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1339131726}) 2024-04-15T18:49:14.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251259560}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2710343997}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999045990}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411895744}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714588453}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639164779}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4052584823}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762290732}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214080111}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349028094}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863064999}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045006433}) 2024-04-15T18:49:14.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2072371466}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774205728}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969837977}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455486357}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604592446}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646222127}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266483820}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769038967}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034595885}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400399681}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127003039}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903667738}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271100047}) 2024-04-15T18:49:14.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169238110}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054317438}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292322259}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791158004}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1097014250}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424565015}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018707933}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631589955}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328643077}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500009954}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932889687}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2273900238}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139305738}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066389958}) 2024-04-15T18:49:14.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304656534}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372133217}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800277383}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020357453}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807994280}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006787326}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408828087}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550740854}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=169342472}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087125729}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3158622313}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447819373}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104949776}) 2024-04-15T18:49:14.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412881252}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642266530}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705860896}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=14} crc {logm=3515700301,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=3964216675,mon_config_key=1738221538}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2243623742}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3618260380}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=655766421,monmap=1029477071,osd_metadata=2065572459,osd_pg_creating=3751948937,osdmap=630877908}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3872359915}) 2024-04-15T18:49:14.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:14 smithi049 ceph-mon[40087]: scrub ok on 0,1,2: ScrubResult(keys {} crc {}) 2024-04-15T18:49:14.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=2738869719,config=1059521453,health=2030214595,kv=3102977241,logm=3810290515}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959930492}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3387927874}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068576816}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87325794}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770205200}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545026422}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:49:14.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2308001018}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050083571}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127561891}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800634988}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121235019}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003186586}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618228792}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511530211}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294789653}) 2024-04-15T18:49:14.777 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847432028}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297482198}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1138372748}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626759527}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959579734}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843472416}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1401312312}) 2024-04-15T18:49:14.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264129123}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620110711}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1512641223}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412689250}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641833005}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551423793}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1339131726}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251259560}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2710343997}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999045990}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411895744}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714588453}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639164779}) 2024-04-15T18:49:14.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4052584823}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762290732}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214080111}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349028094}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863064999}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045006433}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2072371466}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774205728}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969837977}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455486357}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604592446}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646222127}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266483820}) 2024-04-15T18:49:14.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769038967}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034595885}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400399681}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127003039}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903667738}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271100047}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169238110}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054317438}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292322259}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791158004}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1097014250}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424565015}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018707933}) 2024-04-15T18:49:14.781 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631589955}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328643077}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500009954}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932889687}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2273900238}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139305738}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066389958}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304656534}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372133217}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800277383}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020357453}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807994280}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006787326}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408828087}) 2024-04-15T18:49:14.782 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550740854}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=169342472}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087125729}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3158622313}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447819373}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104949776}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412881252}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642266530}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705860896}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=14} crc {logm=3515700301,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=3964216675,mon_config_key=1738221538}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2243623742}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3618260380}) 2024-04-15T18:49:14.783 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=655766421,monmap=1029477071,osd_metadata=2065572459,osd_pg_creating=3751948937,osdmap=630877908}) 2024-04-15T18:49:14.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3872359915}) 2024-04-15T18:49:14.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:14 smithi082 ceph-mon[44206]: scrub ok on 0,1,2: ScrubResult(keys {} crc {}) 2024-04-15T18:49:14.836 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:14.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=53,logm=6} crc {auth=2738869719,config=1059521453,health=2030214595,kv=3102977241,logm=3810290515}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1959930492}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3387927874}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068576816}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87325794}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770205200}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545026422}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094618109}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173762113}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=153940417}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2269812244}) 2024-04-15T18:49:14.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224632653}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=355552310}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557464675}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692134090}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=269127477}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267646857}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119036273}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2308001018}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050083571}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127561891}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2800634988}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121235019}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3003186586}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618228792}) 2024-04-15T18:49:14.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1511530211}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1294789653}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3847432028}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2297482198}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1138372748}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626759527}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=959579734}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=843472416}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1401312312}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264129123}) 2024-04-15T18:49:14.899 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2620110711}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1512641223}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412689250}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641833005}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2551423793}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1339131726}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1251259560}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2710343997}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999045990}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411895744}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714588453}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639164779}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4052584823}) 2024-04-15T18:49:14.900 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3762290732}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214080111}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349028094}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=863064999}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1045006433}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2072371466}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3774205728}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969837977}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=455486357}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604592446}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646222127}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=266483820}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769038967}) 2024-04-15T18:49:14.901 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2034595885}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2400399681}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127003039}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903667738}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271100047}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169238110}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054317438}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1292322259}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791158004}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1097014250}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424565015}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4018707933}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3631589955}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328643077}) 2024-04-15T18:49:14.902 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=500009954}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3932889687}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2273900238}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4139305738}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066389958}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1304656534}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2372133217}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=800277383}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2020357453}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807994280}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006787326}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408828087}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550740854}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=169342472}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2087125729}) 2024-04-15T18:49:14.903 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3158622313}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447819373}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1104949776}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412881252}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1642266530}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2705860896}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8,mon_config_key=14} crc {logm=3515700301,mdsmap=2840952586,mgr=103319482,mgr_command_descs=4152233241,mgr_metadata=1481864419,mgrstat=3964216675,mon_config_key=1738221538}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2243623742}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3618260380}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=655766421,monmap=1029477071,osd_metadata=2065572459,osd_pg_creating=3751948937,osdmap=630877908}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3872359915}) 2024-04-15T18:49:14.904 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:14 smithi159 ceph-mon[45336]: scrub ok on 0,1,2: ScrubResult(keys {} crc {}) 2024-04-15T18:49:14.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:14.990 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:15.052 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:15.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:15.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:15.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:49:15.197 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:15.228 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:15.241 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:49:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:15.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:15.272 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:49:15.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:49:15.343 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:49:15.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:15.376 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:49:15.393 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:15.418 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:49:15.557 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:15.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:15.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:15.667 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:15.718 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:15.772 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: pgmap v2216: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2756/135807 objects degraded (2.029%); 11016/135807 objects misplaced (8.112%); 9.8 MiB/s, 157 objects/s recovering 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3665613265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:15.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:15 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: pgmap v2216: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2756/135807 objects degraded (2.029%); 11016/135807 objects misplaced (8.112%); 9.8 MiB/s, 157 objects/s recovering 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3665613265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:15.780 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.781 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:15 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:15.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: pgmap v2216: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2756/135807 objects degraded (2.029%); 11016/135807 objects misplaced (8.112%); 9.8 MiB/s, 157 objects/s recovering 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3665613265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.840 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:15 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:15.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:15.918 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:15.942 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:15.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:16.049 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:16.122 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:16.145 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:16.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:16.251 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:16.255 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:49:16.291 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:49:16.302 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:16.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:49:16.338 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:16.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:16.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:49:16.403 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:49:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:49:16.444 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:16.492 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:16.522 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:16.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:16.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:16.678 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:16.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:16 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:16 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:16.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:49:17.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:16 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 2767/134118 objects degraded (2.063%); 10815/134118 objects misplaced (8.064%); 11 MiB/s, 180 objects/s recovering 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:17.897 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:17 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 2767/134118 objects degraded (2.063%); 10815/134118 objects misplaced (8.064%); 11 MiB/s, 180 objects/s recovering 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:17.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:17 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: pgmap v2217: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 692 op/s; 2767/134118 objects degraded (2.063%); 10815/134118 objects misplaced (8.064%); 11 MiB/s, 180 objects/s recovering 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:18.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:17 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:18.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:18 smithi159 ceph-mon[45336]: osdmap e845: 12 total, 12 up, 12 in 2024-04-15T18:49:18.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:18 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2767/134118 objects degraded (2.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:18.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:18 smithi049 ceph-mon[40087]: osdmap e845: 12 total, 12 up, 12 in 2024-04-15T18:49:18.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:18 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2767/134118 objects degraded (2.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:19.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:18 smithi082 ceph-mon[44206]: osdmap e845: 12 total, 12 up, 12 in 2024-04-15T18:49:19.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:18 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2767/134118 objects degraded (2.063%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:19.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:19 smithi159 ceph-mon[45336]: pgmap v2219: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2767/132831 objects degraded (2.083%); 10815/132831 objects misplaced (8.142%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:49:19.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:19 smithi049 ceph-mon[40087]: pgmap v2219: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2767/132831 objects degraded (2.083%); 10815/132831 objects misplaced (8.142%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:49:20.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:19 smithi082 ceph-mon[44206]: pgmap v2219: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 2767/132831 objects degraded (2.083%); 10815/132831 objects misplaced (8.142%); 5.8 MiB/s, 92 objects/s recovering 2024-04-15T18:49:20.790 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:49:20.790 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:49:21.344 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:21.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:21.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:21.481 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:21.507 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:21.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:21.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:21.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:21 smithi049 ceph-mon[40087]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 2844/125466 objects degraded (2.267%); 9970/125466 objects misplaced (7.946%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:49:21.653 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:21.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:21 smithi082 ceph-mon[44206]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 2844/125466 objects degraded (2.267%); 9970/125466 objects misplaced (7.946%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:49:21.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:21 smithi159 ceph-mon[45336]: pgmap v2220: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 712 op/s; 2844/125466 objects degraded (2.267%); 9970/125466 objects misplaced (7.946%); 11 MiB/s, 183 objects/s recovering 2024-04-15T18:49:22.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:49:23.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:49:23.393 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:23.853 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:23 smithi082 ceph-mon[44206]: pgmap v2221: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2844/124869 objects degraded (2.278%); 9970/124869 objects misplaced (7.984%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:49:23.853 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2844/124869 objects degraded (2.278%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:23.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:23 smithi159 ceph-mon[45336]: pgmap v2221: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2844/124869 objects degraded (2.278%); 9970/124869 objects misplaced (7.984%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:49:23.893 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2844/124869 objects degraded (2.278%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:23 smithi049 ceph-mon[40087]: pgmap v2221: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2844/124869 objects degraded (2.278%); 9970/124869 objects misplaced (7.984%); 9.4 MiB/s, 150 objects/s recovering 2024-04-15T18:49:23.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2844/124869 objects degraded (2.278%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:24.894 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:24.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:25.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:49:25.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:49:25.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:49:25.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:49:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:49:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:49:25.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:49:25.898 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:25 smithi159 ceph-mon[45336]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 2909/117333 objects degraded (2.479%); 9334/117333 objects misplaced (7.955%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:25.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:25 smithi049 ceph-mon[40087]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 2909/117333 objects degraded (2.479%); 9334/117333 objects misplaced (7.955%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:26.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:25 smithi082 ceph-mon[44206]: pgmap v2222: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 2909/117333 objects degraded (2.479%); 9334/117333 objects misplaced (7.955%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:26.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:26 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:26.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:26 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:26.654 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:26 smithi049 ceph-mon[40087]: osdmap e846: 12 total, 12 up, 12 in 2024-04-15T18:49:26.904 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:26.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:26.911 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:27.012 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:26 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:26 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:26 smithi082 ceph-mon[44206]: osdmap e846: 12 total, 12 up, 12 in 2024-04-15T18:49:27.057 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:26 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:27.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:26 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:27.058 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:26 smithi159 ceph-mon[45336]: osdmap e846: 12 total, 12 up, 12 in 2024-04-15T18:49:27.062 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:27.100 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:27.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:27.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:27.243 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:27.267 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:27.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:27.378 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:27.425 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:27.471 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:27.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:27.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:27.625 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:27.649 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:27.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:27.660 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.7 pg_upmap_items mapping to [6->1,2->8,7->10] 2024-04-15T18:49:27.679 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:49:27.755 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:27.794 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:27.822 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:27.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:27.899 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:49:27.899 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:49:27.925 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:27 smithi082 ceph-mon[44206]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2909/115497 objects degraded (2.519%); 9119/115497 objects misplaced (7.895%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:49:27.925 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:49:27.925 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:27 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]: dispatch 2024-04-15T18:49:27.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:27.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:27 smithi049 ceph-mon[40087]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2909/115497 objects degraded (2.519%); 9119/115497 objects misplaced (7.895%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:49:27.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:49:27.957 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:27 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]: dispatch 2024-04-15T18:49:27.960 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:28.087 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:28.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:27 smithi159 ceph-mon[45336]: pgmap v2224: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 2909/115497 objects degraded (2.519%); 9119/115497 objects misplaced (7.895%); 12 MiB/s, 190 objects/s recovering 2024-04-15T18:49:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:49:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:27 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]: dispatch 2024-04-15T18:49:28.192 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:28.218 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:28.239 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:28.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:28.343 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:28.371 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:28.392 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:28.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:28.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:28.527 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:28.659 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:28.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:28.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:28.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:28 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:28.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:49:28.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]': finished 2024-04-15T18:49:28.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:28 smithi049 ceph-mon[40087]: osdmap e847: 12 total, 12 up, 12 in 2024-04-15T18:49:28.802 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2909/115497 objects degraded (2.519%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:28.805 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:28 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:49:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]': finished 2024-04-15T18:49:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:28 smithi082 ceph-mon[44206]: osdmap e847: 12 total, 12 up, 12 in 2024-04-15T18:49:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2909/115497 objects degraded (2.519%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:29.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:28 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:29.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1233687084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:49:29.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1369906417' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "9.7", "id": [6, 1, 2, 8, 7, 10]}]': finished 2024-04-15T18:49:29.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:28 smithi159 ceph-mon[45336]: osdmap e847: 12 total, 12 up, 12 in 2024-04-15T18:49:29.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2909/115497 objects degraded (2.519%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:29.470 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:29.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:29.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:29.607 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:29.866 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:29.871 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:29 smithi049 ceph-mon[40087]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 2909/114312 objects degraded (2.545%); 9119/114312 objects misplaced (7.977%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:49:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:29.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:29.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:30.011 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:30.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:29 smithi082 ceph-mon[44206]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 2909/114312 objects degraded (2.545%); 9119/114312 objects misplaced (7.977%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:49:30.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:49:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:29 smithi159 ceph-mon[45336]: pgmap v2226: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 2909/114312 objects degraded (2.545%); 9119/114312 objects misplaced (7.977%); 6.1 MiB/s, 98 objects/s recovering 2024-04-15T18:49:30.366 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:30.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:30.407 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:30.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:30.501 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:30.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:49:30.710 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:30.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:30.815 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:30.852 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:30.882 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:30.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:30.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:31.042 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:31.078 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:31.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:31.164 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:49:31.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:31.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:49:31.229 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:31.239 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:49:31.261 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:31.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:31.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:49:31.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:49:31.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:49:31.365 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:31.394 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:49:31.395 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:31.428 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:49:31.468 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:49:31.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:49:31.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:49:31.587 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:49:31.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:49:31.853 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:31.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:31.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2955/106878 objects degraded (2.765%); 8247/106878 objects misplaced (7.716%); 13 MiB/s, 201 objects/s recovering 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/292896097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:31 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2955/106878 objects degraded (2.765%); 8247/106878 objects misplaced (7.716%); 13 MiB/s, 201 objects/s recovering 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/292896097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.987 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:31 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:31.991 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:32.097 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:32.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: pgmap v2227: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 749 op/s; 2955/106878 objects degraded (2.765%); 8247/106878 objects misplaced (7.716%); 13 MiB/s, 201 objects/s recovering 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/292896097' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:31 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:32.254 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:49:32.471 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:49:32.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.776 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:32 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:32.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:32 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:32 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:33.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:49:33.567 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:33.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:33 smithi049 ceph-mon[40087]: pgmap v2228: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 2955/106320 objects degraded (2.779%); 8247/106320 objects misplaced (7.757%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:49:33.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:33 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2955/106320 objects degraded (2.779%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:33 smithi082 ceph-mon[44206]: pgmap v2228: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 2955/106320 objects degraded (2.779%); 8247/106320 objects misplaced (7.757%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:49:34.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:33 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2955/106320 objects degraded (2.779%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:34.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:33 smithi159 ceph-mon[45336]: pgmap v2228: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 2955/106320 objects degraded (2.779%); 8247/106320 objects misplaced (7.757%); 8.1 MiB/s, 130 objects/s recovering 2024-04-15T18:49:34.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:33 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2955/106320 objects degraded (2.779%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:34.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:49:34.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:49:34.506 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-15T18:49:34.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:49:35.014 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:35.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:35.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:35.155 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:35.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:35.319 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:35.353 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:35.387 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:35.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:35.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:35.539 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:35.566 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:35.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:35.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66872 8.4 8:2fd3b0dc:::benchmark_data_smithi159_162467_object66871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:49:35.689 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 448.21772673499999, 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.221963629, 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66872 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.690 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.691 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477538+0000", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7899999999999997e-07 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477567+0000", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8904999999999999e-05 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477579+0000", 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1773999999999999e-05 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.692 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477586+0000", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8129999999999998e-06 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477626+0000", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0457000000000002e-05 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477716+0000", 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0026999999999997e-05 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.693 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477805+0000", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8660999999999997e-05 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.479626+0000", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001821666 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.532510+0000", 2024-04-15T18:49:35.694 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.052884050000000002 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699344+0000", 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16683369100000001 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699462+0000", 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117971 2024-04-15T18:49:35.695 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699501+0000", 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8735000000000001e-05 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:66872 8.4 e732/709 ondisk, result = 0)", 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 448.21497490299998, 2024-04-15T18:49:35.696 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.219219101, 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.697 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480290+0000", 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0699999999999997e-07 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480293+0000", 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3100000000000001e-06 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.698 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699149+0000", 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.218855771 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699165+0000", 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.632e-05 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699315+0000", 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014995399999999999 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.699 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699334+0000", 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8638999999999999e-05 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699508+0000", 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017450000000000001 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.700 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66881 8.0 8:006672f8:::benchmark_data_smithi159_162467_object66880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 448.20712410200002, 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22338060300000001, 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66881 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.701 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488141+0000", 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5939999999999999e-06 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.702 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488214+0000", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2871000000000001e-05 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488226+0000", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1223000000000001e-05 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488249+0000", 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3269999999999999e-05 2024-04-15T18:49:35.703 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488287+0000", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8186999999999999e-05 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488391+0000", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104149 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488496+0000", 2024-04-15T18:49:35.704 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010463600000000001 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490307+0000", 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001811602 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490905+0000", 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00059743399999999999 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.705 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711410+0000", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.220504902 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711475+0000", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5753999999999994e-05 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711520+0000", 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4981000000000002e-05 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.706 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82881 8.0 8:07151622:::benchmark_data_smithi159_162467_object82880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 432.26613201499998, 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22421091100000001, 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.707 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82881 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.708 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429144+0000", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2394e-05 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429232+0000", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7344999999999998e-05 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429242+0000", 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0164e-05 2024-04-15T18:49:35.709 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429250+0000", 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7249999999999997e-06 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431184+0000", 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019340550000000001 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.710 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431318+0000", 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013448299999999999 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431421+0000", 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010239900000000001 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433412+0000", 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019919009999999999 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.711 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436831+0000", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034181820000000001 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653287+0000", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216456327 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653318+0000", 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1352000000000001e-05 2024-04-15T18:49:35.712 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653343+0000", 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4584e-05 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82886 8.0 8:054a31d8:::benchmark_data_smithi159_162467_object82885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 432.26172252999999, 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.227292199, 2024-04-15T18:49:35.713 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82886 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.714 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433543+0000", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4470000000000001e-06 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433600+0000", 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:49:35.715 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433610+0000", 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6560000000000004e-06 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433617+0000", 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8720000000000002e-06 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.716 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433677+0000", 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9969000000000001e-05 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433781+0000", 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103631 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433900+0000", 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011869199999999999 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.717 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436043+0000", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021429449999999998 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436987+0000", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00094478900000000002 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660767+0000", 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22377945399999999 2024-04-15T18:49:35.718 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660806+0000", 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9045999999999999e-05 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660834+0000", 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7704000000000001e-05 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.719 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82902 8.0 8:0b1c4e81:::benchmark_data_smithi159_162467_object82901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 432.25109879799999, 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22047940599999999, 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82902 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.720 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.721 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444166+0000", 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.147e-06 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444220+0000", 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3282999999999998e-05 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444229+0000", 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7860000000000008e-06 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444235+0000", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.463e-06 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444280+0000", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5045e-05 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444395+0000", 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011479 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.723 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444498+0000", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010325800000000001 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.446700+0000", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022018820000000001 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.447033+0000", 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033350600000000002 2024-04-15T18:49:35.724 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664573+0000", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21753967799999999 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664616+0000", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3126e-05 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664645+0000", 2024-04-15T18:49:35.725 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8442e-05 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(8.d 756/743 [PushOp(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head, version: 726'2744, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head@726'2744, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 353.46415940200001, 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24113336099999999, 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.726 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231106+0000", 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.508e-06 2024-04-15T18:49:35.727 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231139+0000", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3120000000000001e-05 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469866+0000", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23872668799999999 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469882+0000", 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5794000000000002e-05 2024-04-15T18:49:35.728 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469963+0000", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1763999999999998e-05 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469972+0000", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8380000000000008e-06 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.472238+0000", 2024-04-15T18:49:35.729 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022656489999999998 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:164234 8.0 8:05b44565:::benchmark_data_smithi159_162467_object61929:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 306.54467124899998, 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24243994899999999, 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.730 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 164234 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.731 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150595+0000", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.137e-06 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150605+0000", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9299999999999998e-06 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150640+0000", 2024-04-15T18:49:35.732 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5648000000000002e-05 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150664+0000", 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3632e-05 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150715+0000", 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1121000000000001e-05 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.733 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150811+0000", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6012000000000002e-05 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150891+0000", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9812000000000001e-05 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.154655+0000", 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037635139999999999 2024-04-15T18:49:35.734 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.155576+0000", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00092136400000000004 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392953+0000", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23737709700000001 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392993+0000", 2024-04-15T18:49:35.735 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0268000000000001e-05 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.393033+0000", 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9413999999999999e-05 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:169755 8.0 8:06929281:::benchmark_data_smithi159_162467_object67450:head [delete] snapc 0=[] ondisk+write+known_if_redirected e767)", 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:49:35.736 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 297.08891566300002, 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22069882099999999, 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 169755 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:49:35.737 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606350+0000", 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.31e-06 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.738 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606356+0000", 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0800000000000002e-06 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606369+0000", 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3742e-05 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606377+0000", 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2880000000000004e-06 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.739 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606422+0000", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5330999999999997e-05 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606543+0000", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120671 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606637+0000", 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4759000000000005e-05 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.740 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611592+0000", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049540929999999997 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.618261+0000", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0066691210000000001 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826959+0000", 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20869838900000001 2024-04-15T18:49:35.741 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826999+0000", 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9787999999999999e-05 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.827047+0000", 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8248999999999998e-05 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:93297 9.6 9:6f080384:::benchmark_data_smithi159_173188_object93296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 149.42009629, 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24058054700000001, 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93297 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.743 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.744 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275169+0000", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.491e-06 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275227+0000", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7577000000000001e-05 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275241+0000", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.454e-05 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.745 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275250+0000", 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2309999999999996e-06 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275308+0000", 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7991000000000001e-05 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275442+0000", 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000134315 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.746 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275577+0000", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135537 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.278398+0000", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028205539999999999 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.278472+0000", 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3807999999999995e-05 2024-04-15T18:49:35.747 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515675+0000", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23720350500000001 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515715+0000", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9694000000000003e-05 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515748+0000", 2024-04-15T18:49:35.748 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3303999999999998e-05 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:93315 9.6 9:6a62fa81:::benchmark_data_smithi159_173188_object93314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:06.307554+0000", 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 149.38770968399999, 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21603122, 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.749 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93315 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307554+0000", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307554+0000", 2024-04-15T18:49:35.750 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307555+0000", 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9599999999999999e-07 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307609+0000", 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4080999999999999e-05 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.751 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307616+0000", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1910000000000003e-06 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307621+0000", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.792e-06 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307670+0000", 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9484999999999998e-05 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.752 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307773+0000", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010302900000000001 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.307865+0000", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1460999999999997e-05 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.310181+0000", 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0023157770000000002 2024-04-15T18:49:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.311170+0000", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00098952000000000003 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.523493+0000", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21232306300000001 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.523546+0000", 2024-04-15T18:49:35.754 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3204000000000001e-05 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.523586+0000", 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9121e-05 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107186 9.a 9:5ca3234a:::benchmark_data_smithi159_173188_object107185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:49:35.755 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.473334983, 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.497663148, 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107186 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.756 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221931+0000", 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6819999999999999e-06 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.757 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221998+0000", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7490000000000006e-05 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222013+0000", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4367e-05 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222020+0000", 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8620000000000003e-06 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.758 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222080+0000", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9579e-05 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222224+0000", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014447900000000001 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222326+0000", 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102008 2024-04-15T18:49:35.759 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225366+0000", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030397940000000002 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.414130+0000", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.188763655 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719483+0000", 2024-04-15T18:49:35.760 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30535280599999998 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719541+0000", 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8579999999999998e-05 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719592+0000", 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0846000000000001e-05 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107188 9.f 9:ff26e8fa:::benchmark_data_smithi159_173188_object107187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.47019430399999, 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.62629479399999999, 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107188 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.762 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.763 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225071+0000", 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.623e-06 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225131+0000", 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9784999999999998e-05 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225144+0000", 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2860000000000001e-05 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.764 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225154+0000", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5899999999999997e-06 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225208+0000", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4045000000000002e-05 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225336+0000", 2024-04-15T18:49:35.765 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012817199999999999 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225442+0000", 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106579 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228413+0000", 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002971137 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.766 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228608+0000", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019488599999999999 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851256+0000", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.62264738900000005 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851318+0000", 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2151999999999997e-05 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.767 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851364+0000", 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6576000000000003e-05 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107186 9.a e820/791 ondisk, result = 0)", 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 135.46627335700001, 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49061027400000001, 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.768 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:49:35.769 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228996+0000", 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4610000000000001e-06 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719405+0000", 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49040913600000002 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.770 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719410+0000", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8559999999999998e-06 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719466+0000", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6671e-05 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719475+0000", 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7690000000000002e-06 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.771 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719601+0000", 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012567200000000001 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107208 9.f 9:f46dc3ae:::benchmark_data_smithi159_173188_object107207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.97979490399999, 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41139816699999998, 2024-04-15T18:49:35.772 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107208 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715471+0000", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9700000000000002e-06 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715584+0000", 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112747 2024-04-15T18:49:35.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715600+0000", 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6232999999999999e-05 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715608+0000", 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8460000000000008e-06 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.775 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715672+0000", 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4622000000000006e-05 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715842+0000", 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016921899999999999 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.716014+0000", 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017243599999999999 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.776 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.718882+0000", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028680239999999998 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719280+0000", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039826500000000002 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126718+0000", 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.407437938 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.777 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126796+0000", 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7824000000000006e-05 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126867+0000", 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1043000000000004e-05 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.778 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107208 9.f e820/791 ondisk, result = 0)", 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.84418662300001, 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27579915900000002, 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.779 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851079+0000", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8729999999999999e-06 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851089+0000", 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7459999999999995e-06 2024-04-15T18:49:35.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126561+0000", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27547190900000001 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126576+0000", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4956999999999999e-05 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126699+0000", 2024-04-15T18:49:35.781 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012357700000000001 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126711+0000", 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2028000000000001e-05 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126877+0000", 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165069 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:110348 9.f 9:feb2cf5a:::benchmark_data_smithi159_173188_object1715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 126.839044229, 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22072198600000001, 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110348 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856221+0000", 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1489999999999999e-06 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.784 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856225+0000", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4340000000000002e-06 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856235+0000", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5859999999999994e-06 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856241+0000", 2024-04-15T18:49:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9320000000000001e-06 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856291+0000", 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0559000000000002e-05 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856421+0000", 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129987 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:49:35.786 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856549+0000", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012783200000000001 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.859056+0000", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025069419999999999 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.873184+0000", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01412771 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.787 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076858+0000", 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203673839 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076889+0000", 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1689000000000001e-05 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076942+0000", 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2327000000000002e-05 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.788 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:110350 9.a 9:521ea892:::benchmark_data_smithi159_173188_object1717:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 126.834788395, 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22033312399999999, 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110350 2024-04-15T18:49:35.789 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.790 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.75e-07 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860479+0000", 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0680000000000001e-06 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860483+0000", 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3409999999999996e-06 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.791 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860487+0000", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1310000000000002e-06 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860522+0000", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5845e-05 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860600+0000", 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7152999999999998e-05 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.792 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860658+0000", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8718999999999998e-05 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.863725+0000", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030663330000000001 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.863801+0000", 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6598000000000001e-05 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.793 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080741+0000", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21694023400000001 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080772+0000", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0395999999999998e-05 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080809+0000", 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6831e-05 2024-04-15T18:49:35.794 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:185486 9.a 9:551b3fde:::benchmark_data_smithi159_173188_object76853:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 1.3104859069999999, 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26204080299999999, 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.795 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185486 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.796 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384780+0000", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4410000000000001e-06 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384785+0000", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4410000000000001e-06 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384800+0000", 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4829e-05 2024-04-15T18:49:35.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384808+0000", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8820000000000001e-06 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384861+0000", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3816000000000001e-05 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385000+0000", 2024-04-15T18:49:35.798 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013899600000000001 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385134+0000", 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133702 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.389490+0000", 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043563209999999998 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.799 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.396059+0000", 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0065680110000000003 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646717+0000", 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25065870299999998 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646774+0000", 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7104000000000001e-05 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.800 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646819+0000", 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4557e-05 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:185492 9.a 9:536494b0:::benchmark_data_smithi159_173188_object76859:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 1.3037645929999999, 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27240242599999998, 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:49:35.801 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185492 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:49:35.802 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391501+0000", 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.482e-06 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391508+0000", 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1670000000000002e-06 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.803 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391522+0000", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4003e-05 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391532+0000", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6860000000000001e-06 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391591+0000", 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8789000000000002e-05 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.804 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391706+0000", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115592 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391798+0000", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2268999999999994e-05 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.395521+0000", 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037226590000000001 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.805 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.396129+0000", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00060791299999999996 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663834+0000", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26770484300000003 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663868+0000", 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4471000000000003e-05 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:49:35.806 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663902+0000", 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3552e-05 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:49:35.807 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:49:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:35.808 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:35.809 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:35.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:35.815 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:35.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:35.940 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:35.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:35 smithi049 ceph-mon[40087]: pgmap v2229: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 2980/99318 objects degraded (3.000%); 7620/99318 objects misplaced (7.672%); 11 MiB/s, 176 objects/s recovering 2024-04-15T18:49:36.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:35 smithi082 ceph-mon[44206]: pgmap v2229: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 2980/99318 objects degraded (3.000%); 7620/99318 objects misplaced (7.672%); 11 MiB/s, 176 objects/s recovering 2024-04-15T18:49:36.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:35 smithi159 ceph-mon[45336]: pgmap v2229: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 2980/99318 objects degraded (3.000%); 7620/99318 objects misplaced (7.672%); 11 MiB/s, 176 objects/s recovering 2024-04-15T18:49:36.297 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:36.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:36.405 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:36.449 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:49:36.479 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:36.519 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:36.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:36.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:49:36.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:36.669 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:36.709 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:36.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:36.721 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:49:36.808 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:36.814 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:36.845 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:36.867 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:36.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:36.895 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:36.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:36.972 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:37.015 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:49:37.047 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:37.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:37.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:37.184 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:37.707 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:37 smithi082 ceph-mon[44206]: pgmap v2230: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2980/97590 objects degraded (3.054%); 7411/97590 objects misplaced (7.594%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:37.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:37 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:37.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:37.708 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:37 smithi082 ceph-mon[44206]: osdmap e848: 12 total, 12 up, 12 in 2024-04-15T18:49:37.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:37 smithi049 ceph-mon[40087]: pgmap v2230: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2980/97590 objects degraded (3.054%); 7411/97590 objects misplaced (7.594%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:37.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:37 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:37.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:37.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:37 smithi049 ceph-mon[40087]: osdmap e848: 12 total, 12 up, 12 in 2024-04-15T18:49:38.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:37 smithi159 ceph-mon[45336]: pgmap v2230: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 2980/97590 objects degraded (3.054%); 7411/97590 objects misplaced (7.594%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:38.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:37 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:38.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3232739993' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:49:38.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:37 smithi159 ceph-mon[45336]: osdmap e848: 12 total, 12 up, 12 in 2024-04-15T18:49:38.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:38 smithi082 ceph-mon[44206]: pgmap v2232: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 2980/96450 objects degraded (3.090%); 7411/96450 objects misplaced (7.684%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:38.715 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:38 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2980/97590 objects degraded (3.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:38.858 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:49:38.858 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:49:38.858 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:49:38.858 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:49:38.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:38 smithi049 ceph-mon[40087]: pgmap v2232: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 2980/96450 objects degraded (3.090%); 7411/96450 objects misplaced (7.684%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:38.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:38 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2980/97590 objects degraded (3.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:38 smithi159 ceph-mon[45336]: pgmap v2232: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 2980/96450 objects degraded (3.090%); 7411/96450 objects misplaced (7.684%); 10 MiB/s, 163 objects/s recovering 2024-04-15T18:49:39.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:38 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2980/97590 objects degraded (3.054%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:39.445 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:39.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:39.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:39.551 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:39.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:49:39.593 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:39.635 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:39.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:39.740 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:49:39.783 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:39.828 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:49:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:49:39.935 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:49:39.938 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:39.973 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:40.182 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:49:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:40.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:49:40.290 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:40.342 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:40.406 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:40.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:40.515 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:40.581 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:40.625 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:40.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:40.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:40.791 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:40.819 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:40.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:40.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:40 smithi159 ceph-mon[45336]: pgmap v2233: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/89154 objects degraded (3.357%); 6501/89154 objects misplaced (7.292%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:40.896 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4053254822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:49:40.929 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:49:40.966 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:40.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:49:41.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:40 smithi049 ceph-mon[40087]: pgmap v2233: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/89154 objects degraded (3.357%); 6501/89154 objects misplaced (7.292%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:41.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4053254822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 848, 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:49:36.712398+0000", 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:46:33.707597+0000", 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:47:20.193442+0000", 2024-04-15T18:49:41.231 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 31, 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:49:41.232 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:49:41.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:49:41.234 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:49:41.235 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:49:41.236 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.237 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:49:41.238 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:49:41.239 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:45:48.486814+0000", 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:49:41.240 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:49:41.241 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "794", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:49:41.242 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:49:41.243 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:49:41.244 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-04-15T18:49:41.245 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8799999952316284, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:49:41.246 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 671, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.247 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:49:41.251 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.252 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:49:41.253 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.254 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.255 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:49:41.257 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.258 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 778, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:49:41.262 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:49:41.263 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:49:41.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 675, 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.270 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.271 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.274 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.276 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.277 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.281 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 676, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.283 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.284 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.285 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:49:41.286 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.287 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 823, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.288 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.289 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.290 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:49:41.292 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:49:41.293 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:49:41.298 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:49:41.299 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.300 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.303 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.305 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:49:41.306 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:49:41.307 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:49:41.308 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:49:41.309 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:49:41.310 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 94, 2024-04-15T18:49:41.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 733, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 807, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 820, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 734, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/905169792", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/905169792", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/905169792", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/905169792", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:49:41.316 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:42.386174+0000", 2024-04-15T18:49:41.317 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:09.239457+0000", 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:49:41.318 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:40.027037+0000", 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.319 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:39.756500+0000", 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:10.803432+0000", 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:00.551410+0000", 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.321 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:19.018160+0000", 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.322 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:33.398820+0000", 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:23.980112+0000", 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:24.861484+0000", 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.324 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:48:30.291194+0000", 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:16.730516+0000", 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 733 2024-04-15T18:49:41.325 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.7", 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 10 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 5, 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.0", 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:49:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.5", 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-04-15T18:49:41.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.9", 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:49:41.329 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:49:41.330 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:49:41.331 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:49:41.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:40 smithi082 ceph-mon[44206]: pgmap v2233: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/89154 objects degraded (3.357%); 6501/89154 objects misplaced (7.292%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:41.333 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4053254822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:49:41.890 INFO:tasks.thrashosds.thrasher:j is {'epoch': 848, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:49:36.712398+0000', 'last_up_change': '2024-04-15T18:46:33.707597+0000', 'last_in_change': '2024-04-15T18:47:20.193442+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 31, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-04-15T18:45:48.486814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '794', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 671, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 778, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 675, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 791, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 676, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 94, 'last_clean_end': 733, 'up_from': 807, 'up_thru': 820, 'down_at': 734, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 905169792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 905169792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 905169792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 905169792}]}, 'public_addr': '172.21.15.159:6825/905169792', 'cluster_addr': '172.21.15.159:6827/905169792', 'heartbeat_back_addr': '172.21.15.159:6831/905169792', 'heartbeat_front_addr': '172.21.15.159:6829/905169792', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:42.386174+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:09.239457+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:40.027037+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:39.756500+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:10.803432+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:00.551410+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:19.018160+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:33.398820+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:23.980112+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:24.861484+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:48:30.291194+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:16.730516+0000', 'dead_epoch': 733}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '9.7', 'mappings': [{'from': 7, 'to': 10}]}, {'pgid': '9.d', 'mappings': [{'from': 5, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6]}, {'pgid': '9.5', 'osds': [1, 5, 7]}, {'pgid': '9.8', 'osds': [9, 5]}, {'pgid': '9.9', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:49:41.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:49:42.829 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:49:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:42.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:49:42.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:42.970 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:42 smithi159 ceph-mon[45336]: pgmap v2234: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/88587 objects degraded (3.379%); 6501/88587 objects misplaced (7.339%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:42 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2993/88587 objects degraded (3.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:43.149 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:42 smithi082 ceph-mon[44206]: pgmap v2234: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/88587 objects degraded (3.379%); 6501/88587 objects misplaced (7.339%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:43.149 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:42 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2993/88587 objects degraded (3.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:43.180 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:43.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:43.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:42 smithi049 ceph-mon[40087]: pgmap v2234: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 2993/88587 objects degraded (3.379%); 6501/88587 objects misplaced (7.339%); 11 MiB/s, 170 objects/s recovering 2024-04-15T18:49:43.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:42 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2993/88587 objects degraded (3.379%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:43.234 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:43.242 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:43.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:43.321 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:43.363 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:43.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:43.468 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:43.515 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:43.542 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:43.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:43.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:43.697 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:43.778 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:49:43.778 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:49:43.778 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:49:43.779 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:49:43.779 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:49:43.810 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:49:43.827 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:43.827 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:43.827 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:43 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2817826446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:43.827 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:43 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:43.849 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:49:43.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:49:43.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:49:43.949 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:49:43.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:43.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:43.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:43 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2817826446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:43.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:43 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:43.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:49:44.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:49:44.043 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:49:44.070 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:49:44.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:49:44.150 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:49:44.188 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:49:44.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:44.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:44.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:43 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2817826446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:44.191 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:43 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:49:44.192 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:49:44.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:49:44.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: pgmap v2235: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 2993/80637 objects degraded (3.712%); 5851/80637 objects misplaced (7.256%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:49:44.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osdmap e849: 12 total, 12 up, 12 in 2024-04-15T18:49:44.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:44 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:44.914 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:49:44.951 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:44.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:45.057 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:45.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: pgmap v2235: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 2993/80637 objects degraded (3.712%); 5851/80637 objects misplaced (7.256%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:49:45.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osdmap e849: 12 total, 12 up, 12 in 2024-04-15T18:49:45.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:44 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.126 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:45.158 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:45.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:45.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: pgmap v2235: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 2993/80637 objects degraded (3.712%); 5851/80637 objects misplaced (7.256%); 11 MiB/s, 175 objects/s recovering 2024-04-15T18:49:45.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osdmap e849: 12 total, 12 up, 12 in 2024-04-15T18:49:45.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:44 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:45.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:45.331 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:45.397 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:45.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:45.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:45.561 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:45.814 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:45.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:45.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:49:45.952 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:46.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:45 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:45 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:45 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:46.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:46 smithi049 ceph-mon[40087]: pgmap v2237: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 2993/78819 objects degraded (3.797%); 5658/78819 objects misplaced (7.178%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:49:46.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:46.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:47.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:46 smithi082 ceph-mon[44206]: pgmap v2237: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 2993/78819 objects degraded (3.797%); 5658/78819 objects misplaced (7.178%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:49:47.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:47.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:47.109 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:49:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:47.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:49:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:46 smithi159 ceph-mon[45336]: pgmap v2237: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 2993/78819 objects degraded (3.797%); 5658/78819 objects misplaced (7.178%); 12 MiB/s, 192 objects/s recovering 2024-04-15T18:49:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:47.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:47.215 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:49:47.261 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:47.302 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:49:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:47.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:49:47.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:47.454 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:47.560 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:47.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:47.667 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:47.744 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:47.778 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:47.780 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-04-15T18:49:47.780 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:49:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:47.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:47.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:49:47.888 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:47.939 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:47.982 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:47.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:48.090 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:48.139 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:47 smithi159 ceph-mon[45336]: osdmap e850: 12 total, 12 up, 12 in 2024-04-15T18:49:48.139 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:47 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2993/78819 objects degraded (3.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:48.141 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:48.167 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:48.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:48.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:47 smithi049 ceph-mon[40087]: osdmap e850: 12 total, 12 up, 12 in 2024-04-15T18:49:48.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:47 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2993/78819 objects degraded (3.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:48.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:48.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:47 smithi082 ceph-mon[44206]: osdmap e850: 12 total, 12 up, 12 in 2024-04-15T18:49:48.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:47 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2993/78819 objects degraded (3.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:48.324 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:48.369 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:48.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:48.478 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:48.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:49:48.521 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:48.546 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:48.548 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:49:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:48.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:48.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:48.692 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:48.869 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:48 smithi082 ceph-mon[44206]: pgmap v2239: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 2993/77559 objects degraded (3.859%); 5658/77559 objects misplaced (7.295%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:49:48.869 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:48 smithi082 ceph-mon[44206]: from='client.17562 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:48.930 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:48.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:48.963 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:49:49.038 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:49.076 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:49.094 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:49.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:49.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:48 smithi159 ceph-mon[45336]: pgmap v2239: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 2993/77559 objects degraded (3.859%); 5658/77559 objects misplaced (7.295%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:49:49.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:48 smithi159 ceph-mon[45336]: from='client.17562 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:49.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:49.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:48 smithi049 ceph-mon[40087]: pgmap v2239: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 2993/77559 objects degraded (3.859%); 5658/77559 objects misplaced (7.295%); 6.6 MiB/s, 105 objects/s recovering 2024-04-15T18:49:49.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:48 smithi049 ceph-mon[40087]: from='client.17562 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:49:49.225 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:49.246 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:49.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:49.350 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:49.371 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:49:49.417 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:49.541 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:49.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:49.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:49.683 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:50.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:50.144 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:50.193 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:50.340 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:50.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:50.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:49:50.505 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:50.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:50 smithi159 ceph-mon[45336]: pgmap v2240: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3016/69378 objects degraded (4.347%); 4791/69378 objects misplaced (6.906%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:49:50.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4096324071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:50.880 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:50 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:50.900 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:49:50.990 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:49:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:50.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:49:51.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:49:51.123 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:51.145 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:49:51.148 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:50 smithi082 ceph-mon[44206]: pgmap v2240: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3016/69378 objects degraded (4.347%); 4791/69378 objects misplaced (6.906%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:49:51.148 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4096324071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:51.148 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:50 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:51.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:49:51.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:50 smithi049 ceph-mon[40087]: pgmap v2240: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 3016/69378 objects degraded (4.347%); 4791/69378 objects misplaced (6.906%); 13 MiB/s, 210 objects/s recovering 2024-04-15T18:49:51.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4096324071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:51.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:50 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:49:51.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:49:51.281 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:51.311 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:49:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:51.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:49:51.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:51.452 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:51.809 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:51.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:51.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:51.949 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:52.008 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:49:52.008 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:49:52.034 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:52.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:52.054 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2239,"stamp":"2024-04-15T18:49:47.743921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1694695992,"num_objects":25853,"num_object_clones":0,"num_object_copies":77559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2993,"num_objects_misplaced":5658,"num_objects_unfound":0,"num_objects_dirty":25853,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":191592,"num_write_kb":6955259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18191,"num_bytes_recovered":1193452320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161050,"ondisk_log_size":161050,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":13066536,"kb_used_data":6059128,"kb_used_omap":430,"kb_used_meta":7006865,"kb_avail":1111875288,"statfs":{"total":1151940427776,"available":1138560294912,"internally_reserved":0,"allocated":6204547072,"data_stored":6201086040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":440519,"internal_metadata":7175030585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,8,2,3,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-253296640,"num_objects":-3865,"num_object_clones":0,"num_object_copies":-11595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-843,"num_objects_unfound":0,"num_objects_dirty":-3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":55246848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001777"},"pg_stats":[{"pgid":"9.f","version":"848'11764","reported_seq":11954,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:43.091888+0000","last_change":"2024-04-15T18:47:05.724961+0000","last_active":"2024-04-15T18:49:43.091888+0000","last_peered":"2024-04-15T18:49:43.091888+0000","last_clean":"2024-04-15T18:49:43.091888+0000","last_became_active":"2024-04-15T18:45:49.841805+0000","last_became_peered":"2024-04-15T18:45:49.841805+0000","last_unstale":"2024-04-15T18:49:43.091888+0000","last_undegraded":"2024-04-15T18:49:43.091888+0000","last_fullsized":"2024-04-15T18:49:43.091888+0000","mapping_epoch":791,"log_start":"799'1700","ondisk_log_start":"799'1700","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5699","last_scrub_stamp":"2024-04-15T18:47:05.724825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.724825+0000","objects_scrubbed":5653,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105119744,"num_objects":1604,"num_object_clones":0,"num_object_copies":4812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"849'12258","reported_seq":12450,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-04-15T18:49:46.731573+0000","last_change":"2024-04-15T18:46:57.710845+0000","last_active":"2024-04-15T18:49:46.731573+0000","last_peered":"2024-04-15T18:49:46.731573+0000","last_clean":"2024-04-15T18:49:46.731573+0000","last_became_active":"2024-04-15T18:45:49.844972+0000","last_became_peered":"2024-04-15T18:45:49.844972+0000","last_unstale":"2024-04-15T18:49:46.731573+0000","last_undegraded":"2024-04-15T18:49:46.731573+0000","last_fullsized":"2024-04-15T18:49:46.731573+0000","mapping_epoch":791,"log_start":"800'2200","ondisk_log_start":"800'2200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5375","last_scrub_stamp":"2024-04-15T18:46:57.710704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.710704+0000","objects_scrubbed":5329,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"849'12203","reported_seq":12343,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-04-15T18:49:46.724737+0000","last_change":"2024-04-15T18:47:07.167264+0000","last_active":"2024-04-15T18:49:46.724737+0000","last_peered":"2024-04-15T18:49:46.724737+0000","last_clean":"2024-04-15T18:49:46.724737+0000","last_became_active":"2024-04-15T18:45:49.844992+0000","last_became_peered":"2024-04-15T18:45:49.844992+0000","last_unstale":"2024-04-15T18:49:46.724737+0000","last_undegraded":"2024-04-15T18:49:46.724737+0000","last_fullsized":"2024-04-15T18:49:46.724737+0000","mapping_epoch":791,"log_start":"800'2200","ondisk_log_start":"800'2200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":899,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731160,"num_objects":1477,"num_object_clones":0,"num_object_copies":4431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12203,"num_write_kb":437697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"848'11889","reported_seq":12122,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:42.742193+0000","last_change":"2024-04-15T18:48:56.919239+0000","last_active":"2024-04-15T18:49:42.742193+0000","last_peered":"2024-04-15T18:49:42.742193+0000","last_clean":"2024-04-15T18:49:42.742193+0000","last_became_active":"2024-04-15T18:45:49.843864+0000","last_became_peered":"2024-04-15T18:45:49.843864+0000","last_unstale":"2024-04-15T18:49:42.742193+0000","last_undegraded":"2024-04-15T18:49:42.742193+0000","last_fullsized":"2024-04-15T18:49:42.742193+0000","mapping_epoch":791,"log_start":"799'1800","ondisk_log_start":"799'1800","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'10164","last_scrub_stamp":"2024-04-15T18:48:56.918932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:48:56.918932+0000","objects_scrubbed":3417,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:09.262458+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108855296,"num_objects":1661,"num_object_clones":0,"num_object_copies":4983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11889,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"849'11848","reported_seq":12083,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-04-15T18:49:45.506068+0000","last_change":"2024-04-15T18:49:06.655574+0000","last_active":"2024-04-15T18:49:45.506068+0000","last_peered":"2024-04-15T18:49:45.506068+0000","last_clean":"2024-04-15T18:49:45.506068+0000","last_became_active":"2024-04-15T18:45:49.841997+0000","last_became_peered":"2024-04-15T18:45:49.841997+0000","last_unstale":"2024-04-15T18:49:45.506068+0000","last_undegraded":"2024-04-15T18:49:45.506068+0000","last_fullsized":"2024-04-15T18:49:45.506068+0000","mapping_epoch":791,"log_start":"799'1800","ondisk_log_start":"799'1800","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"842'10385","last_scrub_stamp":"2024-04-15T18:49:06.655307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:49:06.655307+0000","objects_scrubbed":2956,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:26.435085+0000","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97386496,"num_objects":1486,"num_object_clones":0,"num_object_copies":4458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11848,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"848'11973","reported_seq":12158,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:43.047415+0000","last_change":"2024-04-15T18:46:55.950320+0000","last_active":"2024-04-15T18:49:43.047415+0000","last_peered":"2024-04-15T18:49:43.047415+0000","last_clean":"2024-04-15T18:49:43.047415+0000","last_became_active":"2024-04-15T18:45:49.843914+0000","last_became_peered":"2024-04-15T18:45:49.843914+0000","last_unstale":"2024-04-15T18:49:43.047415+0000","last_undegraded":"2024-04-15T18:49:43.047415+0000","last_fullsized":"2024-04-15T18:49:43.047415+0000","mapping_epoch":791,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5238","last_scrub_stamp":"2024-04-15T18:46:55.950094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:55.950094+0000","objects_scrubbed":5218,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103481344,"num_objects":1579,"num_object_clones":0,"num_object_copies":4737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"849'11990","reported_seq":20677,"reported_epoch":849,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:44.017808+0000","last_change":"2024-04-15T18:47:30.615221+0000","last_active":"2024-04-15T18:49:44.017808+0000","last_peered":"2024-04-15T18:49:44.017808+0000","last_clean":"2024-04-15T18:47:20.186030+0000","last_became_active":"2024-04-15T18:47:22.838451+0000","last_became_peered":"2024-04-15T18:47:22.838451+0000","last_unstale":"2024-04-15T18:49:44.017808+0000","last_undegraded":"2024-04-15T18:47:30.608553+0000","last_fullsized":"2024-04-15T18:47:22.203522+0000","mapping_epoch":821,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1007,"num_objects_misplaced":1435,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":435520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4278,"num_bytes_recovered":279904256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,5],"avail_no_missing":["10","3","5"],"object_location_counts":[{"shards":"3,5,10","objects":1435},{"shards":"5,10","objects":-822}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"849'11944","reported_seq":12071,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-04-15T18:49:43.993215+0000","last_change":"2024-04-15T18:45:49.841390+0000","last_active":"2024-04-15T18:49:43.993215+0000","last_peered":"2024-04-15T18:49:43.993215+0000","last_clean":"2024-04-15T18:49:43.993215+0000","last_became_active":"2024-04-15T18:45:49.841033+0000","last_became_peered":"2024-04-15T18:45:49.841033+0000","last_unstale":"2024-04-15T18:49:43.993215+0000","last_undegraded":"2024-04-15T18:49:43.993215+0000","last_fullsized":"2024-04-15T18:49:43.993215+0000","mapping_epoch":791,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104333312,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"848'11940","reported_seq":12065,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:43.078581+0000","last_change":"2024-04-15T18:45:49.841903+0000","last_active":"2024-04-15T18:49:43.078581+0000","last_peered":"2024-04-15T18:49:43.078581+0000","last_clean":"2024-04-15T18:49:43.078581+0000","last_became_active":"2024-04-15T18:45:49.841652+0000","last_became_peered":"2024-04-15T18:45:49.841652+0000","last_unstale":"2024-04-15T18:49:43.078581+0000","last_undegraded":"2024-04-15T18:49:43.078581+0000","last_fullsized":"2024-04-15T18:49:43.078581+0000","mapping_epoch":791,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11940,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"848'11769","reported_seq":22912,"reported_epoch":848,"state":"active+recovering+remapped","last_fresh":"2024-04-15T18:49:42.867042+0000","last_change":"2024-04-15T18:47:22.914564+0000","last_active":"2024-04-15T18:49:42.867042+0000","last_peered":"2024-04-15T18:49:42.867042+0000","last_clean":"2024-04-15T18:47:20.156259+0000","last_became_active":"2024-04-15T18:47:22.844939+0000","last_became_peered":"2024-04-15T18:47:22.844939+0000","last_unstale":"2024-04-15T18:49:42.867042+0000","last_undegraded":"2024-04-15T18:49:42.867042+0000","last_fullsized":"2024-04-15T18:49:42.867042+0000","mapping_epoch":821,"log_start":"799'1700","ondisk_log_start":"799'1700","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110952448,"num_objects":1693,"num_object_clones":0,"num_object_copies":5079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1138,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":430784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5508,"num_bytes_recovered":360841216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[1,5,7],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":555}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.4","version":"848'12073","reported_seq":12258,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:42.887228+0000","last_change":"2024-04-15T18:46:57.262366+0000","last_active":"2024-04-15T18:49:42.887228+0000","last_peered":"2024-04-15T18:49:42.887228+0000","last_clean":"2024-04-15T18:49:42.887228+0000","last_became_active":"2024-04-15T18:45:49.844745+0000","last_became_peered":"2024-04-15T18:45:49.844745+0000","last_unstale":"2024-04-15T18:49:42.887228+0000","last_undegraded":"2024-04-15T18:49:42.887228+0000","last_fullsized":"2024-04-15T18:49:42.887228+0000","mapping_epoch":791,"log_start":"800'2000","ondisk_log_start":"800'2000","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5319","last_scrub_stamp":"2024-04-15T18:46:57.262149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.262149+0000","objects_scrubbed":5273,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112525312,"num_objects":1717,"num_object_clones":0,"num_object_copies":5151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12073,"num_write_kb":441280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"848'11863","reported_seq":12055,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:42.859575+0000","last_change":"2024-04-15T18:47:05.437565+0000","last_active":"2024-04-15T18:49:42.859575+0000","last_peered":"2024-04-15T18:49:42.859575+0000","last_clean":"2024-04-15T18:49:42.859575+0000","last_became_active":"2024-04-15T18:45:49.842017+0000","last_became_peered":"2024-04-15T18:45:49.842017+0000","last_unstale":"2024-04-15T18:49:42.859575+0000","last_undegraded":"2024-04-15T18:49:42.859575+0000","last_fullsized":"2024-04-15T18:49:42.859575+0000","mapping_epoch":791,"log_start":"799'1800","ondisk_log_start":"799'1800","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5799","last_scrub_stamp":"2024-04-15T18:47:05.437371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.437371+0000","objects_scrubbed":5752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105316352,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"848'11928","reported_seq":12053,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:42.795654+0000","last_change":"2024-04-15T18:45:49.836537+0000","last_active":"2024-04-15T18:49:42.795654+0000","last_peered":"2024-04-15T18:49:42.795654+0000","last_clean":"2024-04-15T18:49:42.795654+0000","last_became_active":"2024-04-15T18:45:49.836251+0000","last_became_peered":"2024-04-15T18:45:49.836251+0000","last_unstale":"2024-04-15T18:49:42.795654+0000","last_undegraded":"2024-04-15T18:49:42.795654+0000","last_fullsized":"2024-04-15T18:49:42.795654+0000","mapping_epoch":791,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111411200,"num_objects":1700,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"848'11991","reported_seq":12180,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-04-15T18:49:42.916111+0000","last_change":"2024-04-15T18:47:01.868689+0000","last_active":"2024-04-15T18:49:42.916111+0000","last_peered":"2024-04-15T18:49:42.916111+0000","last_clean":"2024-04-15T18:49:42.916111+0000","last_became_active":"2024-04-15T18:45:50.180630+0000","last_became_peered":"2024-04-15T18:45:50.180630+0000","last_unstale":"2024-04-15T18:49:42.916111+0000","last_undegraded":"2024-04-15T18:49:42.916111+0000","last_fullsized":"2024-04-15T18:49:42.916111+0000","mapping_epoch":791,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'5601","last_scrub_stamp":"2024-04-15T18:47:01.868503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:01.868503+0000","objects_scrubbed":5580,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11991,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"848'12068","reported_seq":20855,"reported_epoch":848,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:42.882764+0000","last_change":"2024-04-15T18:47:32.874160+0000","last_active":"2024-04-15T18:49:42.882764+0000","last_peered":"2024-04-15T18:49:42.882764+0000","last_clean":"2024-04-15T18:47:20.171208+0000","last_became_active":"2024-04-15T18:47:22.829093+0000","last_became_peered":"2024-04-15T18:47:22.829093+0000","last_unstale":"2024-04-15T18:49:42.882764+0000","last_undegraded":"2024-04-15T18:47:32.870404+0000","last_fullsized":"2024-04-15T18:47:22.201052+0000","mapping_epoch":821,"log_start":"800'2000","ondisk_log_start":"800'2000","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":893,"num_objects_misplaced":1549,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12068,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4329,"num_bytes_recovered":282787840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6],"avail_no_missing":["1","6","8"],"object_location_counts":[{"shards":"1,6","objects":-790},{"shards":"1,6,8","objects":1549}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"848'11910","reported_seq":20154,"reported_epoch":848,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:49:42.956114+0000","last_change":"2024-04-15T18:47:32.882544+0000","last_active":"2024-04-15T18:49:42.956114+0000","last_peered":"2024-04-15T18:49:42.956114+0000","last_clean":"2024-04-15T18:47:20.189472+0000","last_became_active":"2024-04-15T18:47:22.835987+0000","last_became_peered":"2024-04-15T18:47:22.835987+0000","last_unstale":"2024-04-15T18:49:42.956114+0000","last_undegraded":"2024-04-15T18:47:32.878238+0000","last_fullsized":"2024-04-15T18:47:22.203408+0000","mapping_epoch":821,"log_start":"799'1900","ondisk_log_start":"799'1900","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":1536,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4058,"num_bytes_recovered":265158656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,5],"avail_no_missing":["9","5","7"],"object_location_counts":[{"shards":"5,7,9","objects":1536},{"shards":"5,9","objects":-949}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":2094,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-04-15T18:49:43.835895+0000","last_change":"2024-04-15T18:47:24.773236+0000","last_active":"2024-04-15T18:49:43.835895+0000","last_peered":"2024-04-15T18:49:43.835895+0000","last_clean":"2024-04-15T18:49:43.835895+0000","last_became_active":"2024-04-15T18:47:24.772930+0000","last_became_peered":"2024-04-15T18:47:24.772930+0000","last_unstale":"2024-04-15T18:49:43.835895+0000","last_undegraded":"2024-04-15T18:49:43.835895+0000","last_fullsized":"2024-04-15T18:49:43.835895+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:46:52.053607+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:46:52.053607+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1694105624,"num_objects":25851,"num_object_clones":0,"num_object_copies":77553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2993,"num_objects_misplaced":5658,"num_objects_unfound":0,"num_objects_dirty":25851,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191411,"num_write_kb":6952321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18173,"num_bytes_recovered":1188691968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6154432512,"data_stored":6154420296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160911,"ondisk_log_size":160911,"up":48,"acting":45,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":807,"seq":3466038607911,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":285344,"kb_used_data":78104,"kb_used_omap":144,"kb_used_meta":207087,"kb_avail":93459808,"statfs":{"total":95995035648,"available":95702843392,"internally_reserved":0,"allocated":79978496,"data_stored":79690939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147868,"internal_metadata":212057700},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187949,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160424,"kb_used_data":511452,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92584728,"statfs":{"total":95995035648,"available":94806761472,"internally_reserved":0,"allocated":523726848,"data_stored":523438535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":76,"seq":326417514995,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1396312,"kb_used_data":639056,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92348840,"statfs":{"total":95995035648,"available":94565212160,"internally_reserved":0,"allocated":654393344,"data_stored":654109351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874741,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1085480,"kb_used_data":635036,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92659672,"statfs":{"total":95995035648,"available":94883504128,"internally_reserved":0,"allocated":650276864,"data_stored":649988551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169083,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521904,"kb_used_data":904872,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92223248,"statfs":{"total":95995035648,"available":94436605952,"internally_reserved":0,"allocated":926588928,"data_stored":926292991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":631830986},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363929,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435116,"kb_used_data":199972,"kb_used_omap":111,"kb_used_meta":235024,"kb_avail":93310036,"statfs":{"total":95995035648,"available":95549476864,"internally_reserved":0,"allocated":204771328,"data_stored":204479471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":240665243},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575388,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1770652,"kb_used_data":955092,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91974500,"statfs":{"total":95995035648,"available":94181888000,"internally_reserved":0,"allocated":978014208,"data_stored":977726143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681640,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063512,"statfs":{"total":95995035648,"available":95297036288,"internally_reserved":0,"allocated":749568,"data_stored":461255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524212,"kb_used_data":744,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220940,"statfs":{"total":95995035648,"available":95458242560,"internally_reserved":0,"allocated":761856,"data_stored":469223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609291,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725272,"kb_used_data":964624,"kb_used_omap":44,"kb_used_meta":760595,"kb_avail":92019880,"statfs":{"total":95995035648,"available":94228357120,"internally_reserved":0,"allocated":987774976,"data_stored":987490983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":778849834},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986698,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881176,"kb_used_data":218320,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92863976,"statfs":{"total":95995035648,"available":95092711424,"internally_reserved":0,"allocated":223559680,"data_stored":223275687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703592,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1599004,"kb_used_data":951124,"kb_used_omap":4,"kb_used_meta":647867,"kb_avail":92146148,"statfs":{"total":95995035648,"available":94357655552,"internally_reserved":0,"allocated":973950976,"data_stored":973662911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":663416513},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":976031744,"data_stored":976027672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":980221952,"data_stored":980221952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":221773824,"data_stored":221773824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":957943808,"data_stored":957939736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":200867840,"data_stored":200867840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":923144192,"data_stored":923140120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":645922816,"data_stored":645922816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":649527296,"data_stored":649527296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":521666560,"data_stored":521666560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":77332480,"data_stored":77332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:49:52.056 INFO:tasks.thrashosds.thrasher:Setting 9.3 to [11, 4, 8] 2024-04-15T18:49:52.056 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '9.3', '11', '4', '8'] 2024-04-15T18:49:52.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pg-upmap 9.3 11 4 8 2024-04-15T18:49:52.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:51 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:52.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:51 smithi049 ceph-mon[40087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:49:52.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:51 smithi049 ceph-mon[40087]: osdmap e851: 12 total, 12 up, 12 in 2024-04-15T18:49:52.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:51 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:52.123 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:51 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:52.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:51 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:51 smithi159 ceph-mon[45336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:49:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:51 smithi159 ceph-mon[45336]: osdmap e851: 12 total, 12 up, 12 in 2024-04-15T18:49:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:51 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:51 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:52.185 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:51 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:49:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:51 smithi082 ceph-mon[44206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:49:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:51 smithi082 ceph-mon[44206]: osdmap e851: 12 total, 12 up, 12 in 2024-04-15T18:49:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:51 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:52.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:51 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:52.456 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:52.457 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:52.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:52.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:49:52.596 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:52.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:49:52.798 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:49:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:52.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:49:52.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:49:52.941 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:52 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:52 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:52 smithi159 ceph-mon[45336]: pgmap v2242: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 3016/68775 objects degraded (4.385%); 4791/68775 objects misplaced (6.966%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:52 smithi159 ceph-mon[45336]: osdmap e852: 12 total, 12 up, 12 in 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:52 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3016/68775 objects degraded (4.385%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:52 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:53.148 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:52 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:53.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:52 smithi049 ceph-mon[40087]: pgmap v2242: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 3016/68775 objects degraded (4.385%); 4791/68775 objects misplaced (6.966%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:49:53.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:52 smithi049 ceph-mon[40087]: osdmap e852: 12 total, 12 up, 12 in 2024-04-15T18:49:53.149 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:52 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3016/68775 objects degraded (4.385%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:53.174 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:49:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:52 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:52 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:52 smithi082 ceph-mon[44206]: pgmap v2242: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 3016/68775 objects degraded (4.385%); 4791/68775 objects misplaced (6.966%); 8.2 MiB/s, 131 objects/s recovering 2024-04-15T18:49:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:52 smithi082 ceph-mon[44206]: osdmap e852: 12 total, 12 up, 12 in 2024-04-15T18:49:53.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:52 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3016/68775 objects degraded (4.385%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:53.419 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:54.093 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:49:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:54.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:49:54.197 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:54.245 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:54.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:49:54.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:54 smithi049 ceph-mon[40087]: pgmap v2244: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s; 3016/60711 objects degraded (4.968%); 4088/60711 objects misplaced (6.734%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:49:54.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:54 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:49:54.924 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:54 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:49:55.214 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:54 smithi159 ceph-mon[45336]: pgmap v2244: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s; 3016/60711 objects degraded (4.968%); 4088/60711 objects misplaced (6.734%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:49:55.214 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:54 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:49:55.214 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:54 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:49:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:54 smithi082 ceph-mon[44206]: pgmap v2244: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s; 3016/60711 objects degraded (4.968%); 4088/60711 objects misplaced (6.734%); 14 MiB/s, 224 objects/s recovering 2024-04-15T18:49:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:54 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:49:55.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:54 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:49:55.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:49:55.460 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:49:55.487 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:49:55.524 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:49:55.566 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:49:55.597 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:49:55.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:49:55.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:49:55.728 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:49:55.770 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:49:55.804 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:49:55.833 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:49:55.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:49:55.836 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:49:55.836 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: osdmap e853: 12 total, 12 up, 12 in 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2638860870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:55.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:55 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: osdmap e853: 12 total, 12 up, 12 in 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2638860870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:55 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: osdmap e853: 12 total, 12 up, 12 in 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2638860870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:55 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.915 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:56.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.938 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: pgmap v2246: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 3016/58938 objects degraded (5.117%); 3876/58938 objects misplaced (6.576%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:56.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:56.939 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:56 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: pgmap v2246: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 3016/58938 objects degraded (5.117%); 3876/58938 objects misplaced (6.576%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:57.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:56 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: pgmap v2246: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 3016/58938 objects degraded (5.117%); 3876/58938 objects misplaced (6.576%); 9.5 MiB/s, 152 objects/s recovering 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:56 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:49:57.518 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:57.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:57.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:49:57.679 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:57.946 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]: dispatch 2024-04-15T18:49:57.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:57 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3016/58938 objects degraded (5.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:57.952 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.3 pg_upmap mapping to [11,4,8] 2024-04-15T18:49:58.042 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:49:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:58.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:49:58.146 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:58.176 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:58.206 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:49:58.212 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]: dispatch 2024-04-15T18:49:58.212 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:57 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3016/58938 objects degraded (5.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:58.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:49:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]: dispatch 2024-04-15T18:49:58.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:57 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3016/58938 objects degraded (5.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:49:58.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:49:58.340 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:58.366 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:49:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:49:58.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:49:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:58.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:49:58.732 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:49:58.732 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:49:58.732 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:49:58.732 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:49:58.732 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:49:59.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:58 smithi049 ceph-mon[40087]: pgmap v2247: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3016/57735 objects degraded (5.224%); 3876/57735 objects misplaced (6.713%); 8.4 MiB/s, 133 objects/s recovering 2024-04-15T18:49:59.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]': finished 2024-04-15T18:49:59.018 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:58 smithi049 ceph-mon[40087]: osdmap e854: 12 total, 12 up, 12 in 2024-04-15T18:49:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:58 smithi159 ceph-mon[45336]: pgmap v2247: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3016/57735 objects degraded (5.224%); 3876/57735 objects misplaced (6.713%); 8.4 MiB/s, 133 objects/s recovering 2024-04-15T18:49:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]': finished 2024-04-15T18:49:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:58 smithi159 ceph-mon[45336]: osdmap e854: 12 total, 12 up, 12 in 2024-04-15T18:49:59.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:58 smithi082 ceph-mon[44206]: pgmap v2247: 17 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 3016/57735 objects degraded (5.224%); 3876/57735 objects misplaced (6.713%); 8.4 MiB/s, 133 objects/s recovering 2024-04-15T18:49:59.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/712312831' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.3", "id": [11, 4, 8]}]': finished 2024-04-15T18:49:59.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:58 smithi082 ceph-mon[44206]: osdmap e854: 12 total, 12 up, 12 in 2024-04-15T18:49:59.361 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:49:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:49:59.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:49:59.466 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:49:59.499 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:49:59.667 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:49:59.956 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:50:00.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:59 smithi049 ceph-mon[40087]: osdmap e855: 12 total, 12 up, 12 in 2024-04-15T18:50:00.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:49:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:00.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:59 smithi082 ceph-mon[44206]: osdmap e855: 12 total, 12 up, 12 in 2024-04-15T18:50:00.278 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:49:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:59 smithi159 ceph-mon[45336]: osdmap e855: 12 total, 12 up, 12 in 2024-04-15T18:50:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:49:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:00.453 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:00.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:00.559 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:00.586 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:00.605 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:00.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:00.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:00.962 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: pgmap v2250: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 986 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/47295 objects degraded (6.440%); 2802/47295 objects misplaced (5.925%); 12 MiB/s, 209 objects/s recovering 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: osdmap e856: 12 total, 12 up, 12 in 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: 9.3 starting backfill to osd.4 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: Health detail: HEALTH_WARN Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: [WRN] PG_DEGRADED: Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: pg 9.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,6] 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: pg 9.8 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [9,5] 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: pg 9.9 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [10,5] 2024-04-15T18:50:00.963 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:00 smithi082 ceph-mon[44206]: 9.3 starting backfill to osd.11 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:50:01.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: pgmap v2250: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 986 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/47295 objects degraded (6.440%); 2802/47295 objects misplaced (5.925%); 12 MiB/s, 209 objects/s recovering 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: osdmap e856: 12 total, 12 up, 12 in 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: 9.3 starting backfill to osd.4 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: Health detail: HEALTH_WARN Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: [WRN] PG_DEGRADED: Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: pg 9.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,6] 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: pg 9.8 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [9,5] 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: pg 9.9 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [10,5] 2024-04-15T18:50:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:00 smithi049 ceph-mon[40087]: 9.3 starting backfill to osd.11 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: pgmap v2250: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 986 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/47295 objects degraded (6.440%); 2802/47295 objects misplaced (5.925%); 12 MiB/s, 209 objects/s recovering 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2446994340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: osdmap e856: 12 total, 12 up, 12 in 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: 9.3 starting backfill to osd.4 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: 9.3 starting backfill to osd.8 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: Health detail: HEALTH_WARN Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: [WRN] PG_DEGRADED: Degraded data redundancy: 3046/47295 objects degraded (6.440%), 3 pgs degraded, 3 pgs undersized 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: pg 9.0 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [1,6] 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: pg 9.8 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [9,5] 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: pg 9.9 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [10,5] 2024-04-15T18:50:01.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:00 smithi159 ceph-mon[45336]: 9.3 starting backfill to osd.11 from (0'0,0'0] MAX to 853'12491 2024-04-15T18:50:01.401 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:01.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:01.456 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:50:01.509 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:01.564 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:01.785 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:01.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:01.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:01.926 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.060 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:02.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:01 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:02.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:01 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:02.146 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:01 smithi159 ceph-mon[45336]: osdmap e857: 12 total, 12 up, 12 in 2024-04-15T18:50:02.164 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:02.201 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.227 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:02.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:01 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:02.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:01 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:02.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:01 smithi082 ceph-mon[44206]: osdmap e857: 12 total, 12 up, 12 in 2024-04-15T18:50:02.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:02.366 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:50:02.398 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:02.432 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:50:02.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:01 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:02.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:01 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:02.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:01 smithi049 ceph-mon[40087]: osdmap e857: 12 total, 12 up, 12 in 2024-04-15T18:50:02.462 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:50:02.492 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:50:02.504 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:02.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:50:02.541 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:50:02.571 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:02.679 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:02.713 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.741 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:02.801 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:50:02.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:02.888 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:02.918 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:02.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:02.932 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:50:02.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:50:03.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:50:03.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:03.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:50:03.060 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.089 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:03.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: pgmap v2253: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/46977 objects degraded (6.484%); 2802/46977 objects misplaced (5.965%); 15 MiB/s, 260 objects/s recovering 2024-04-15T18:50:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1520029342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3046/46977 objects degraded (6.484%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:03.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:02 smithi159 ceph-mon[45336]: osdmap e858: 12 total, 12 up, 12 in 2024-04-15T18:50:03.192 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:03.193 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:50:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:50:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:50:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:50:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:50:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:50:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:50:03.228 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.260 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: pgmap v2253: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/46977 objects degraded (6.484%); 2802/46977 objects misplaced (5.965%); 15 MiB/s, 260 objects/s recovering 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1520029342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3046/46977 objects degraded (6.484%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:03.262 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:02 smithi082 ceph-mon[44206]: osdmap e858: 12 total, 12 up, 12 in 2024-04-15T18:50:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:03.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:03.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: pgmap v2253: 17 pgs: 1 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3046/46977 objects degraded (6.484%); 2802/46977 objects misplaced (5.965%); 15 MiB/s, 260 objects/s recovering 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1520029342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:03.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3046/46977 objects degraded (6.484%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:03.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:02 smithi049 ceph-mon[40087]: osdmap e858: 12 total, 12 up, 12 in 2024-04-15T18:50:03.402 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.435 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:03.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:03.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:03.639 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.666 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:03.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:03.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set noscrub 2024-04-15T18:50:03.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:03.804 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.831 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:03.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:03.935 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:03.937 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-04-15T18:50:03.937 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:50:03.970 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:03.999 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:04.075 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.075 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.075 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:04.076 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:04 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:04.102 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:04.134 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:04.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:04 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:04.137 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:04.158 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:04.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:04.298 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:04.332 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:04.333 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:04 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:04.379 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 4, 5, 7, 8, 9, 10, 3, 2, 6, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 7, 8, 9, 10, 3, 2, 5, 6, 11] 2024-04-15T18:50:04.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-15T18:50:04.379 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-15T18:50:04.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-15T18:50:04.426 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:04.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:04.562 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:04.583 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:04.686 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:04.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-15T18:50:04.722 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:04.742 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:04.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:04.874 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:04.907 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:04.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:05.004 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:05 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:05.004 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:05 smithi159 ceph-mon[45336]: pgmap v2255: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 959 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 2.97k op/s; 3046/46005 objects degraded (6.621%); 4254/46005 objects misplaced (9.247%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:50:05.004 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:05 smithi159 ceph-mon[45336]: osdmap e859: 12 total, 12 up, 12 in 2024-04-15T18:50:05.004 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:05 smithi159 ceph-mon[45336]: 9.d scrub starts 2024-04-15T18:50:05.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:05.049 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:05.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:05 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:05.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:05 smithi082 ceph-mon[44206]: pgmap v2255: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 959 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 2.97k op/s; 3046/46005 objects degraded (6.621%); 4254/46005 objects misplaced (9.247%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:50:05.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:05 smithi082 ceph-mon[44206]: osdmap e859: 12 total, 12 up, 12 in 2024-04-15T18:50:05.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:05 smithi082 ceph-mon[44206]: 9.d scrub starts 2024-04-15T18:50:05.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:05 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:05 smithi049 ceph-mon[40087]: pgmap v2255: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 12 active+clean; 959 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 2.97k op/s; 3046/46005 objects degraded (6.621%); 4254/46005 objects misplaced (9.247%); 14 MiB/s, 231 objects/s recovering 2024-04-15T18:50:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:05 smithi049 ceph-mon[40087]: osdmap e859: 12 total, 12 up, 12 in 2024-04-15T18:50:05.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:05 smithi049 ceph-mon[40087]: 9.d scrub starts 2024-04-15T18:50:05.595 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:05.605 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:05.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:05.715 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:05.754 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:05.782 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:05.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:05.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:50:05.903 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:05.945 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:05.981 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:05.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:06.002 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:06 smithi082 ceph-mon[44206]: 9.d scrub ok 2024-04-15T18:50:06.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:50:06.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:06.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:06 smithi049 ceph-mon[40087]: 9.d scrub ok 2024-04-15T18:50:06.189 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:06.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:06 smithi159 ceph-mon[45336]: 9.d scrub ok 2024-04-15T18:50:07.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:07 smithi082 ceph-mon[44206]: pgmap v2257: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 921 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 2.51k op/s; 3046/44205 objects degraded (6.891%); 3826/44205 objects misplaced (8.655%); 13 MiB/s, 229 objects/s recovering 2024-04-15T18:50:07.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:07 smithi159 ceph-mon[45336]: pgmap v2257: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 921 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 2.51k op/s; 3046/44205 objects degraded (6.891%); 3826/44205 objects misplaced (8.655%); 13 MiB/s, 229 objects/s recovering 2024-04-15T18:50:07.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:07 smithi049 ceph-mon[40087]: pgmap v2257: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 921 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 2.51k op/s; 3046/44205 objects degraded (6.891%); 3826/44205 objects misplaced (8.655%); 13 MiB/s, 229 objects/s recovering 2024-04-15T18:50:07.887 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:07.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:07.992 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:08.059 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:08.102 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:08.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:08.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:08.243 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:08.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:08 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3046/44205 objects degraded (6.891%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:08.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:08 smithi082 ceph-mon[44206]: osdmap e860: 12 total, 12 up, 12 in 2024-04-15T18:50:08.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:08 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3046/44205 objects degraded (6.891%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:08.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:08 smithi159 ceph-mon[45336]: osdmap e860: 12 total, 12 up, 12 in 2024-04-15T18:50:08.447 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:08.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:08 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3046/44205 objects degraded (6.891%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:08.451 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:08 smithi049 ceph-mon[40087]: osdmap e860: 12 total, 12 up, 12 in 2024-04-15T18:50:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:08.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:08.553 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:08.611 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:08.654 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:08.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:08.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:50:08.809 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:08.841 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:08.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:08.951 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:08.990 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:09.029 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:09.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:09.053 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-04-15T18:50:09.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:09.163 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:09.190 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:09.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:09.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:09 smithi082 ceph-mon[44206]: pgmap v2258: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 895 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 2.19k op/s; 3046/42963 objects degraded (7.090%); 3826/42963 objects misplaced (8.905%); 11 MiB/s, 196 objects/s recovering 2024-04-15T18:50:09.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:09 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:50:09.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:09.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:09 smithi049 ceph-mon[40087]: pgmap v2258: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 895 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 2.19k op/s; 3046/42963 objects degraded (7.090%); 3826/42963 objects misplaced (8.905%); 11 MiB/s, 196 objects/s recovering 2024-04-15T18:50:09.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:09 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:50:09.328 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:09.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:09 smithi159 ceph-mon[45336]: pgmap v2258: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 895 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 2.19k op/s; 3046/42963 objects degraded (7.090%); 3826/42963 objects misplaced (8.905%); 11 MiB/s, 196 objects/s recovering 2024-04-15T18:50:09.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:09 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-15T18:50:09.881 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:09.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:09.987 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:10.032 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:10.059 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:10.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:10.169 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:10 smithi049 ceph-mon[40087]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:50:10.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:10 smithi049 ceph-mon[40087]: osdmap e861: 12 total, 12 up, 12 in 2024-04-15T18:50:10.229 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:10.272 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:10.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:10 smithi082 ceph-mon[44206]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:10.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:50:10.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:10 smithi082 ceph-mon[44206]: osdmap e861: 12 total, 12 up, 12 in 2024-04-15T18:50:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:10.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:10.285 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:10 smithi159 ceph-mon[45336]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:10.285 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1305265142' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-15T18:50:10.285 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:10 smithi159 ceph-mon[45336]: osdmap e861: 12 total, 12 up, 12 in 2024-04-15T18:50:10.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:10.430 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:10.992 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:11.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:11.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:11.134 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:11.164 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:11.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:11.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:11.305 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:11 smithi049 ceph-mon[40087]: pgmap v2261: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 731 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3046/35067 objects degraded (8.686%); 2136/35067 objects misplaced (6.091%); 12 MiB/s, 223 objects/s recovering 2024-04-15T18:50:11.309 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:11.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:11.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:11 smithi159 ceph-mon[45336]: pgmap v2261: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 731 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3046/35067 objects degraded (8.686%); 2136/35067 objects misplaced (6.091%); 12 MiB/s, 223 objects/s recovering 2024-04-15T18:50:11.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:11.491 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:11.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:11 smithi082 ceph-mon[44206]: pgmap v2261: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 731 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 3046/35067 objects degraded (8.686%); 2136/35067 objects misplaced (6.091%); 12 MiB/s, 223 objects/s recovering 2024-04-15T18:50:11.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:11.809 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:50:11.809 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 861, 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "22c655ba-fb52-11ee-bc8f-c7b262605968", 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-04-15T18:02:11.588977+0000", 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-04-15T18:50:09.027397+0000", 2024-04-15T18:50:11.931 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-04-15T18:46:33.707597+0000", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-04-15T18:47:20.193442+0000", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 31, 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-04-15T18:50:11.932 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 12, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:05:43.035801+0000", 2024-04-15T18:50:11.933 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-04-15T18:50:11.934 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:50:11.935 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "21", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:50:11.936 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:50:11.937 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:50:11.938 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-04-15T18:50:11.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 10, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 10, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.30000001192092896, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_7", 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-04-15T18:45:48.486814+0000", 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-04-15T18:50:11.941 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-04-15T18:50:11.942 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "794", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-04-15T18:50:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-04-15T18:50:11.944 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-04-15T18:50:11.945 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 3, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8799999952316284, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.83333325386047363, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 1, 2024-04-15T18:50:11.946 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.82999998331069946, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "error_message": "pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "c6740db9-c8a9-4083-9ef3-6ac938f861a9", 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 671, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-04-15T18:50:11.948 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.949 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.950 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.951 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 306402154 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/306402154", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/306402154", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/306402154", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/306402154", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a5f0a887-6af8-4537-b37c-3d9ed493b595", 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 854, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.953 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.954 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.955 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.956 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1404755046 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.957 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1404755046", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1404755046", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1404755046", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1404755046", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "57bad304-da93-49aa-9320-13cf3d1ab1a8", 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.958 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 307, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 361, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 778, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 308, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-04-15T18:50:11.959 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-04-15T18:50:11.960 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-04-15T18:50:11.961 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.962 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2410603388 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/2410603388", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/2410603388", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/2410603388", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/2410603388", 2024-04-15T18:50:11.963 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8db13188-c0af-4b82-9333-e0659a026aa2", 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 26, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 177, 2024-04-15T18:50:11.964 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 226, 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 855, 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 178, 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.966 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-04-15T18:50:11.967 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-04-15T18:50:11.968 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1036848877 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1036848877", 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1036848877", 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1036848877", 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1036848877", 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.969 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6f66dfc2-7e1a-4545-81f8-246ff27e5928", 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 31, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.970 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6800", 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6801", 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.971 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6802", 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6803", 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6806", 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6807", 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6804", 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6805", 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1223630649 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.974 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6801/1223630649", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6803/1223630649", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6807/1223630649", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6805/1223630649", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "fafc8df6-bd72-4306-9b73-7ce797f5744a", 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.975 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 363, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 376, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 364, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6808", 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6809", 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6810", 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6811", 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6814", 2024-04-15T18:50:11.978 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6815", 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6812", 2024-04-15T18:50:11.979 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6813", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1630882481 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6809/1630882481", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6811/1630882481", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6815/1630882481", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6813/1630882481", 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.980 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2ac61adb-a5b8-4e27-a648-756453c35783", 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 47, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 561, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 631, 2024-04-15T18:50:11.981 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 676, 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 562, 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6816", 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6817", 2024-04-15T18:50:11.982 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6818", 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6819", 2024-04-15T18:50:11.983 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6822", 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6823", 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6820", 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6821", 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 256106271 2024-04-15T18:50:11.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6817/256106271", 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6819/256106271", 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6823/256106271", 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6821/256106271", 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.986 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "a8940a14-d060-46ed-aeb9-d5c43ff421c2", 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 56, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 823, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6824", 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6825", 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6826", 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6827", 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6830", 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6831", 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6828", 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.82:6829", 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 457966805 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.82:6825/457966805", 2024-04-15T18:50:11.991 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.82:6827/457966805", 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.82:6831/457966805", 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.82:6829/457966805", 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "07798de8-8c74-4948-8860-a3e8c07d7e59", 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.992 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 65, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 791, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6800", 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.993 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6801", 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6802", 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6803", 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.995 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6806", 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6807", 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6804", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6805", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 403815130 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6801/403815130", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6803/403815130", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6807/403815130", 2024-04-15T18:50:11.997 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6805/403815130", 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "43cab3c4-4f27-44ba-b8a4-0f6909e3001d", 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:11.998 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 76, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6808", 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:11.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6809", 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6810", 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.000 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6811", 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6814", 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6815", 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6812", 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6813", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1558317233 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6809/1558317233", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6811/1558317233", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6815/1558317233", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6813/1558317233", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:12.003 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:12.004 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.004 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:50:12.004 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "e9731146-d494-4a3c-a205-37064c685816", 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 86, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 821, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6816", 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6817", 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:12.006 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6818", 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6819", 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.007 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6822", 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6823", 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.008 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6820", 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6821", 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2653197187 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.009 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6817/2653197187", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6819/2653197187", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6823/2653197187", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6821/2653197187", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "f8de423a-9e36-45ae-97e7-0384858a8118", 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-04-15T18:50:12.010 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 94, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 733, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 807, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 855, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 734, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.011 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6824", 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6825", 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.012 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6826", 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6827", 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6830", 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6831", 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-04-15T18:50:12.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6828", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.159:6829", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 905169792 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.159:6825/905169792", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.159:6827/905169792", 2024-04-15T18:50:12.015 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.159:6831/905169792", 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.159:6829/905169792", 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.016 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:54.076876+0000", 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:09.239457+0000", 2024-04-15T18:50:12.017 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:51.284500+0000", 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 307 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.018 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:51.440397+0000", 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 177 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.019 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:10.803432+0000", 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:50:03.269991+0000", 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 363 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:46.553303+0000", 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 561 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.021 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:50:03.429060+0000", 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 8, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:42.878894+0000", 2024-04-15T18:50:12.022 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 9, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:24.861484+0000", 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 10, 2024-04-15T18:50:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:50:00.599349+0000", 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 11, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540701547738038271, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-04-15T18:50:12.024 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-04-15T18:49:16.730516+0000", 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 733 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.3", 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: 11, 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: 8 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:12.025 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.7", 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 10 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.d", 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.026 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 5, 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.0", 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.3", 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: 9 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.8", 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: 9, 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.9", 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: 10, 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3052240311": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:50:12.029 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2417929619": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/769044034": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1867631398": "2024-04-16T18:03:08.925868+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3639973481": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2669325064": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/530504890": "2024-04-16T18:02:40.629638+0000", 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/453126905": "2024-04-16T18:02:40.629638+0000" 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-04-15T18:50:12.030 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-04-15T18:50:12.033 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-04-15T18:50:12.033 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-04-15T18:50:12.033 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-04-15T18:50:12.034 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-04-15T18:50:12.035 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-04-15T18:50:12.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:12.035 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:50:12.057 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:12.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:12.161 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:12.244 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:12.280 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:12.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:12.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:12.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3183813149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:50:12.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3183813149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:50:12.423 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:12.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3183813149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-15T18:50:12.771 INFO:tasks.thrashosds.thrasher:j is {'epoch': 861, 'fsid': '22c655ba-fb52-11ee-bc8f-c7b262605968', 'created': '2024-04-15T18:02:11.588977+0000', 'modified': '2024-04-15T18:50:09.027397+0000', 'last_up_change': '2024-04-15T18:46:33.707597+0000', 'last_in_change': '2024-04-15T18:47:20.193442+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 31, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T18:05:43.035801+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-04-15T18:45:48.486814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '794', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8333332538604736, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8299999833106995, 'error_message': 'pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': 'c6740db9-c8a9-4083-9ef3-6ac938f861a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 671, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 306402154}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 306402154}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 306402154}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 306402154}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 306402154}]}, 'public_addr': '172.21.15.49:6803/306402154', 'cluster_addr': '172.21.15.49:6805/306402154', 'heartbeat_back_addr': '172.21.15.49:6809/306402154', 'heartbeat_front_addr': '172.21.15.49:6807/306402154', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a5f0a887-6af8-4537-b37c-3d9ed493b595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 854, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1404755046}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1404755046}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1404755046}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1404755046}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1404755046}]}, 'public_addr': '172.21.15.49:6811/1404755046', 'cluster_addr': '172.21.15.49:6813/1404755046', 'heartbeat_back_addr': '172.21.15.49:6817/1404755046', 'heartbeat_front_addr': '172.21.15.49:6815/1404755046', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '57bad304-da93-49aa-9320-13cf3d1ab1a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 307, 'up_from': 361, 'up_thru': 778, 'down_at': 308, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 2410603388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 2410603388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 2410603388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 2410603388}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 2410603388}]}, 'public_addr': '172.21.15.49:6819/2410603388', 'cluster_addr': '172.21.15.49:6821/2410603388', 'heartbeat_back_addr': '172.21.15.49:6825/2410603388', 'heartbeat_front_addr': '172.21.15.49:6823/2410603388', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8db13188-c0af-4b82-9333-e0659a026aa2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 26, 'last_clean_end': 177, 'up_from': 226, 'up_thru': 855, 'down_at': 178, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1036848877}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1036848877}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1036848877}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1036848877}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1036848877}]}, 'public_addr': '172.21.15.49:6827/1036848877', 'cluster_addr': '172.21.15.49:6829/1036848877', 'heartbeat_back_addr': '172.21.15.49:6833/1036848877', 'heartbeat_front_addr': '172.21.15.49:6831/1036848877', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6f66dfc2-7e1a-4545-81f8-246ff27e5928', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6800', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6801', 'nonce': 1223630649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6802', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6803', 'nonce': 1223630649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6806', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6807', 'nonce': 1223630649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6804', 'nonce': 1223630649}, {'type': 'v1', 'addr': '172.21.15.82:6805', 'nonce': 1223630649}]}, 'public_addr': '172.21.15.82:6801/1223630649', 'cluster_addr': '172.21.15.82:6803/1223630649', 'heartbeat_back_addr': '172.21.15.82:6807/1223630649', 'heartbeat_front_addr': '172.21.15.82:6805/1223630649', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fafc8df6-bd72-4306-9b73-7ce797f5744a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 363, 'up_from': 376, 'up_thru': 791, 'down_at': 364, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6808', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6809', 'nonce': 1630882481}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6810', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6811', 'nonce': 1630882481}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6814', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6815', 'nonce': 1630882481}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6812', 'nonce': 1630882481}, {'type': 'v1', 'addr': '172.21.15.82:6813', 'nonce': 1630882481}]}, 'public_addr': '172.21.15.82:6809/1630882481', 'cluster_addr': '172.21.15.82:6811/1630882481', 'heartbeat_back_addr': '172.21.15.82:6815/1630882481', 'heartbeat_front_addr': '172.21.15.82:6813/1630882481', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2ac61adb-a5b8-4e27-a648-756453c35783', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 47, 'last_clean_end': 561, 'up_from': 631, 'up_thru': 676, 'down_at': 562, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6816', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6817', 'nonce': 256106271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6818', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6819', 'nonce': 256106271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6822', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6823', 'nonce': 256106271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6820', 'nonce': 256106271}, {'type': 'v1', 'addr': '172.21.15.82:6821', 'nonce': 256106271}]}, 'public_addr': '172.21.15.82:6817/256106271', 'cluster_addr': '172.21.15.82:6819/256106271', 'heartbeat_back_addr': '172.21.15.82:6823/256106271', 'heartbeat_front_addr': '172.21.15.82:6821/256106271', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'a8940a14-d060-46ed-aeb9-d5c43ff421c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 56, 'up_thru': 823, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6824', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6825', 'nonce': 457966805}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6826', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6827', 'nonce': 457966805}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6830', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6831', 'nonce': 457966805}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6828', 'nonce': 457966805}, {'type': 'v1', 'addr': '172.21.15.82:6829', 'nonce': 457966805}]}, 'public_addr': '172.21.15.82:6825/457966805', 'cluster_addr': '172.21.15.82:6827/457966805', 'heartbeat_back_addr': '172.21.15.82:6831/457966805', 'heartbeat_front_addr': '172.21.15.82:6829/457966805', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '07798de8-8c74-4948-8860-a3e8c07d7e59', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 65, 'up_thru': 791, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6800', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6801', 'nonce': 403815130}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6803', 'nonce': 403815130}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6807', 'nonce': 403815130}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 403815130}, {'type': 'v1', 'addr': '172.21.15.159:6805', 'nonce': 403815130}]}, 'public_addr': '172.21.15.159:6801/403815130', 'cluster_addr': '172.21.15.159:6803/403815130', 'heartbeat_back_addr': '172.21.15.159:6807/403815130', 'heartbeat_front_addr': '172.21.15.159:6805/403815130', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '43cab3c4-4f27-44ba-b8a4-0f6909e3001d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6809', 'nonce': 1558317233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6811', 'nonce': 1558317233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6815', 'nonce': 1558317233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1558317233}, {'type': 'v1', 'addr': '172.21.15.159:6813', 'nonce': 1558317233}]}, 'public_addr': '172.21.15.159:6809/1558317233', 'cluster_addr': '172.21.15.159:6811/1558317233', 'heartbeat_back_addr': '172.21.15.159:6815/1558317233', 'heartbeat_front_addr': '172.21.15.159:6813/1558317233', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e9731146-d494-4a3c-a205-37064c685816', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 821, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6817', 'nonce': 2653197187}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6818', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6819', 'nonce': 2653197187}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6822', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6823', 'nonce': 2653197187}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6820', 'nonce': 2653197187}, {'type': 'v1', 'addr': '172.21.15.159:6821', 'nonce': 2653197187}]}, 'public_addr': '172.21.15.159:6817/2653197187', 'cluster_addr': '172.21.15.159:6819/2653197187', 'heartbeat_back_addr': '172.21.15.159:6823/2653197187', 'heartbeat_front_addr': '172.21.15.159:6821/2653197187', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f8de423a-9e36-45ae-97e7-0384858a8118', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 94, 'last_clean_end': 733, 'up_from': 807, 'up_thru': 855, 'down_at': 734, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6824', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6825', 'nonce': 905169792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6826', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6827', 'nonce': 905169792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6830', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6831', 'nonce': 905169792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6828', 'nonce': 905169792}, {'type': 'v1', 'addr': '172.21.15.159:6829', 'nonce': 905169792}]}, 'public_addr': '172.21.15.159:6825/905169792', 'cluster_addr': '172.21.15.159:6827/905169792', 'heartbeat_back_addr': '172.21.15.159:6831/905169792', 'heartbeat_front_addr': '172.21.15.159:6829/905169792', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:54.076876+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:09.239457+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:51.284500+0000', 'dead_epoch': 307}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:51.440397+0000', 'dead_epoch': 177}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:10.803432+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:50:03.269991+0000', 'dead_epoch': 363}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:46.553303+0000', 'dead_epoch': 561}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:50:03.429060+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:42.878894+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:24.861484+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:50:00.599349+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-15T18:49:16.730516+0000', 'dead_epoch': 733}], 'pg_upmap': [{'pgid': '9.3', 'osds': [11, 4, 8]}], 'pg_upmap_items': [{'pgid': '9.7', 'mappings': [{'from': 7, 'to': 10}]}, {'pgid': '9.d', 'mappings': [{'from': 5, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6]}, {'pgid': '9.3', 'osds': [3, 5, 9]}, {'pgid': '9.8', 'osds': [9, 5]}, {'pgid': '9.9', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/3052240311': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/2417929619': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:0/769044034': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6800/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/1867631398': '2024-04-16T18:03:08.925868+0000', '172.21.15.49:6801/3639973481': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/2669325064': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/530504890': '2024-04-16T18:02:40.629638+0000', '172.21.15.49:0/453126905': '2024-04-16T18:02:40.629638+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-15T18:50:12.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:50:13.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-15T18:50:13.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-15T18:50:13.180 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:13.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:13.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:13.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:13.319 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:13 smithi082 ceph-mon[44206]: pgmap v2262: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 719 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 3046/34485 objects degraded (8.833%); 2136/34485 objects misplaced (6.194%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:50:13.319 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:13 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 3046/34485 objects degraded (8.833%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:13.319 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:13 smithi082 ceph-mon[44206]: osdmap e862: 12 total, 12 up, 12 in 2024-04-15T18:50:13.345 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:13.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:13.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:13 smithi159 ceph-mon[45336]: pgmap v2262: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 719 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 3046/34485 objects degraded (8.833%); 2136/34485 objects misplaced (6.194%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:50:13.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:13 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 3046/34485 objects degraded (8.833%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:13.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:13 smithi159 ceph-mon[45336]: osdmap e862: 12 total, 12 up, 12 in 2024-04-15T18:50:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:13 smithi049 ceph-mon[40087]: pgmap v2262: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 719 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 3046/34485 objects degraded (8.833%); 2136/34485 objects misplaced (6.194%); 9.6 MiB/s, 153 objects/s recovering 2024-04-15T18:50:13.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:13 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 3046/34485 objects degraded (8.833%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:13.440 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:13 smithi049 ceph-mon[40087]: osdmap e862: 12 total, 12 up, 12 in 2024-04-15T18:50:13.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:13.499 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:13.529 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:13.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:13.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:13.666 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:13.687 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:13.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:13.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:13.822 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:14.076 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:14.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:14.105 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:14.111 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:14.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:14.223 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:14.249 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:14.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:14.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:14.404 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:14.436 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:14.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:14.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:14.576 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:14.598 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:14.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:14.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:14.746 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:15.184 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:15 smithi049 ceph-mon[40087]: pgmap v2264: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 563 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 2938/27027 objects degraded (10.871%); 744/27027 objects misplaced (2.753%); 17 MiB/s, 283 objects/s recovering 2024-04-15T18:50:15.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:15 smithi159 ceph-mon[45336]: pgmap v2264: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 563 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 2938/27027 objects degraded (10.871%); 744/27027 objects misplaced (2.753%); 17 MiB/s, 283 objects/s recovering 2024-04-15T18:50:15.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:15 smithi082 ceph-mon[44206]: pgmap v2264: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 563 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 2938/27027 objects degraded (10.871%); 744/27027 objects misplaced (2.753%); 17 MiB/s, 283 objects/s recovering 2024-04-15T18:50:16.310 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:16.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:16.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:16 smithi159 ceph-mon[45336]: osdmap e863: 12 total, 12 up, 12 in 2024-04-15T18:50:16.416 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:16.457 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:16.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:16 smithi049 ceph-mon[40087]: osdmap e863: 12 total, 12 up, 12 in 2024-04-15T18:50:16.497 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:16.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:16.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:16 smithi082 ceph-mon[44206]: osdmap e863: 12 total, 12 up, 12 in 2024-04-15T18:50:16.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:16.641 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:17.286 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:50:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:50:17.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:17 smithi159 ceph-mon[45336]: pgmap v2266: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 514 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2938/24654 objects degraded (11.917%); 480/24654 objects misplaced (1.947%); 11 MiB/s, 197 objects/s recovering 2024-04-15T18:50:17.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:17 smithi159 ceph-mon[45336]: osdmap e864: 12 total, 12 up, 12 in 2024-04-15T18:50:17.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2943037605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:17.413 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:50:17.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:17 smithi049 ceph-mon[40087]: pgmap v2266: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 514 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2938/24654 objects degraded (11.917%); 480/24654 objects misplaced (1.947%); 11 MiB/s, 197 objects/s recovering 2024-04-15T18:50:17.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:17 smithi049 ceph-mon[40087]: osdmap e864: 12 total, 12 up, 12 in 2024-04-15T18:50:17.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2943037605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:17 smithi082 ceph-mon[44206]: pgmap v2266: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 514 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 2938/24654 objects degraded (11.917%); 480/24654 objects misplaced (1.947%); 11 MiB/s, 197 objects/s recovering 2024-04-15T18:50:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:17 smithi082 ceph-mon[44206]: osdmap e864: 12 total, 12 up, 12 in 2024-04-15T18:50:17.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2943037605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:17.568 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:50:17.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:50:17.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:50:17.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:50:17.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:50:17.572 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2938/24654 objects degraded (11.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:18.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:18 smithi159 ceph-mon[45336]: osdmap e865: 12 total, 12 up, 12 in 2024-04-15T18:50:18.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2938/24654 objects degraded (11.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:18.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:18 smithi049 ceph-mon[40087]: osdmap e865: 12 total, 12 up, 12 in 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:18.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2938/24654 objects degraded (11.917%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:18.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:18 smithi082 ceph-mon[44206]: osdmap e865: 12 total, 12 up, 12 in 2024-04-15T18:50:19.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:19 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:19.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:19 smithi049 ceph-mon[40087]: pgmap v2268: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 493 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 2938/23622 objects degraded (12.438%); 480/23622 objects misplaced (2.032%); 9.8 MiB/s, 175 objects/s recovering 2024-04-15T18:50:19.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:19 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:19.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:19 smithi159 ceph-mon[45336]: pgmap v2268: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 493 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 2938/23622 objects degraded (12.438%); 480/23622 objects misplaced (2.032%); 9.8 MiB/s, 175 objects/s recovering 2024-04-15T18:50:19.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:19 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:19.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:19 smithi082 ceph-mon[44206]: pgmap v2268: 17 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 493 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 2938/23622 objects degraded (12.438%); 480/23622 objects misplaced (2.032%); 9.8 MiB/s, 175 objects/s recovering 2024-04-15T18:50:20.329 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:20.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:20.418 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:50:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:50:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:50:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:50:20.418 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:50:20.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:20.474 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:20.678 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:20.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:20.782 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:20.816 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:20.841 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:20.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:20.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:20.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-15T18:50:20.983 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:21.016 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:21.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:21.102 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:21 smithi082 ceph-mon[44206]: pgmap v2270: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 348 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2143/16662 objects degraded (12.862%); 5.7 MiB/s, 223 objects/s recovering 2024-04-15T18:50:21.103 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:21.121 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-04-15T18:50:21.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:21.156 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:21.180 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:21.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:21.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:21.319 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:21.319 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:21 smithi049 ceph-mon[40087]: pgmap v2270: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 348 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2143/16662 objects degraded (12.862%); 5.7 MiB/s, 223 objects/s recovering 2024-04-15T18:50:21.319 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:21.346 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:50:21.352 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:50:21.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:21 smithi159 ceph-mon[45336]: pgmap v2270: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 348 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2143/16662 objects degraded (12.862%); 5.7 MiB/s, 223 objects/s recovering 2024-04-15T18:50:21.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:22.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-15T18:50:22.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:22 smithi159 ceph-mon[45336]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:22.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:22.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:22 smithi159 ceph-mon[45336]: osdmap e866: 12 total, 12 up, 12 in 2024-04-15T18:50:22.393 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:22 smithi159 ceph-mon[45336]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:50:22.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:22 smithi049 ceph-mon[40087]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:22.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:22.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:22 smithi049 ceph-mon[40087]: osdmap e866: 12 total, 12 up, 12 in 2024-04-15T18:50:22.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:22 smithi049 ceph-mon[40087]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:50:22.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:22 smithi082 ceph-mon[44206]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:22.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2155099505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:22.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:22 smithi082 ceph-mon[44206]: osdmap e866: 12 total, 12 up, 12 in 2024-04-15T18:50:22.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:22 smithi082 ceph-mon[44206]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:50:23.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:23 smithi159 ceph-mon[45336]: pgmap v2272: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 335 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 2143/16068 objects degraded (13.337%); 3.0 MiB/s, 179 objects/s recovering 2024-04-15T18:50:23.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:23 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 2143/16068 objects degraded (13.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:23.397 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:23 smithi159 ceph-mon[45336]: osdmap e867: 12 total, 12 up, 12 in 2024-04-15T18:50:23.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:23 smithi049 ceph-mon[40087]: pgmap v2272: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 335 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 2143/16068 objects degraded (13.337%); 3.0 MiB/s, 179 objects/s recovering 2024-04-15T18:50:23.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:23 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 2143/16068 objects degraded (13.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:23.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:23 smithi049 ceph-mon[40087]: osdmap e867: 12 total, 12 up, 12 in 2024-04-15T18:50:23.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:23 smithi082 ceph-mon[44206]: pgmap v2272: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 335 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 476 op/s; 2143/16068 objects degraded (13.337%); 3.0 MiB/s, 179 objects/s recovering 2024-04-15T18:50:23.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:23 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 2143/16068 objects degraded (13.337%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:23.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:23 smithi082 ceph-mon[44206]: osdmap e867: 12 total, 12 up, 12 in 2024-04-15T18:50:23.983 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:23.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:24.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:24.122 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:24.154 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:24.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:24.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:24.318 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:24.345 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:24.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:24.398 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:24 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:24.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:24.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:24 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:24.485 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:24 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:24.487 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:24.700 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:24.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:24.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:24.858 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:25.121 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:25 smithi159 ceph-mon[45336]: pgmap v2274: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 177 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1596/8472 objects degraded (18.839%); 3.0 MiB/s, 271 objects/s recovering 2024-04-15T18:50:25.238 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:25.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:25.343 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:25.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:25 smithi082 ceph-mon[44206]: pgmap v2274: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 177 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1596/8472 objects degraded (18.839%); 3.0 MiB/s, 271 objects/s recovering 2024-04-15T18:50:25.389 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:25.436 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:25.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:25.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:25 smithi049 ceph-mon[40087]: pgmap v2274: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 177 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1596/8472 objects degraded (18.839%); 3.0 MiB/s, 271 objects/s recovering 2024-04-15T18:50:25.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:25.569 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:25.589 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:25.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:25.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:25.720 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:25.739 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:25.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:25.843 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:25.873 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:26.214 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Removed 108630 objects 2024-04-15T18:50:26.215 INFO:tasks.radosbench.radosbench.2.smithi159.stdout:Clean up completed and total clean up time :179.935 2024-04-15T18:50:26.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:26.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:26 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:27.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:26 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:27.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:27.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:26 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:27.254 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2271,"stamp":"2024-04-15T18:50:21.126935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":351404600,"num_objects":5356,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":212089,"num_write_kb":6955259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25366,"num_bytes_recovered":1589748512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160847,"ondisk_log_size":160847,"up":51,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":10,"kb":1124941824,"kb_used":9625928,"kb_used_data":2451896,"kb_used_omap":430,"kb_used_meta":7173457,"kb_avail":1115315896,"statfs":{"total":1151940427776,"available":1142083477504,"internally_reserved":0,"allocated":2510741504,"data_stored":2506451318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":440519,"internal_metadata":7345620793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-199557120,"num_objects":-3045,"num_object_clones":0,"num_object_copies":-9135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-795,"num_objects_misplaced":-480,"num_objects_unfound":0,"num_objects_dirty":-3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3045,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.036435"},"pg_stats":[{"pgid":"9.f","version":"865'13035","reported_seq":13259,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.110926+0000","last_change":"2024-04-15T18:47:05.724961+0000","last_active":"2024-04-15T18:50:18.110926+0000","last_peered":"2024-04-15T18:50:18.110926+0000","last_clean":"2024-04-15T18:50:18.110926+0000","last_became_active":"2024-04-15T18:45:49.841805+0000","last_became_peered":"2024-04-15T18:45:49.841805+0000","last_unstale":"2024-04-15T18:50:18.110926+0000","last_undegraded":"2024-04-15T18:50:18.110926+0000","last_fullsized":"2024-04-15T18:50:18.110926+0000","mapping_epoch":791,"log_start":"803'3000","ondisk_log_start":"803'3000","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5699","last_scrub_stamp":"2024-04-15T18:47:05.724825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.724825+0000","objects_scrubbed":5653,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21823488,"num_objects":333,"num_object_clones":0,"num_object_copies":999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13035,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"864'13373","reported_seq":13595,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-04-15T18:50:16.751236+0000","last_change":"2024-04-15T18:46:57.710845+0000","last_active":"2024-04-15T18:50:16.751236+0000","last_peered":"2024-04-15T18:50:16.751236+0000","last_clean":"2024-04-15T18:50:16.751236+0000","last_became_active":"2024-04-15T18:45:49.844972+0000","last_became_peered":"2024-04-15T18:45:49.844972+0000","last_unstale":"2024-04-15T18:50:16.751236+0000","last_undegraded":"2024-04-15T18:50:16.751236+0000","last_fullsized":"2024-04-15T18:50:16.751236+0000","mapping_epoch":791,"log_start":"805'3300","ondisk_log_start":"805'3300","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5375","last_scrub_stamp":"2024-04-15T18:46:57.710704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.710704+0000","objects_scrubbed":5329,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:21.628248+0000","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24576000,"num_objects":375,"num_object_clones":0,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13373,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"864'13285","reported_seq":13471,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-04-15T18:50:16.663807+0000","last_change":"2024-04-15T18:50:05.066233+0000","last_active":"2024-04-15T18:50:16.663807+0000","last_peered":"2024-04-15T18:50:16.663807+0000","last_clean":"2024-04-15T18:50:16.663807+0000","last_became_active":"2024-04-15T18:45:49.844992+0000","last_became_peered":"2024-04-15T18:45:49.844992+0000","last_unstale":"2024-04-15T18:50:16.663807+0000","last_undegraded":"2024-04-15T18:50:16.663807+0000","last_fullsized":"2024-04-15T18:50:16.663807+0000","mapping_epoch":791,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'12840","last_scrub_stamp":"2024-04-15T18:50:05.065900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:50:05.065900+0000","objects_scrubbed":843,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:51:33.814092+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821208,"num_objects":395,"num_object_clones":0,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13285,"num_write_kb":437697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"865'13193","reported_seq":13460,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:17.775806+0000","last_change":"2024-04-15T18:48:56.919239+0000","last_active":"2024-04-15T18:50:17.775806+0000","last_peered":"2024-04-15T18:50:17.775806+0000","last_clean":"2024-04-15T18:50:17.775806+0000","last_became_active":"2024-04-15T18:45:49.843864+0000","last_became_peered":"2024-04-15T18:45:49.843864+0000","last_unstale":"2024-04-15T18:50:17.775806+0000","last_undegraded":"2024-04-15T18:50:17.775806+0000","last_fullsized":"2024-04-15T18:50:17.775806+0000","mapping_epoch":791,"log_start":"805'3100","ondisk_log_start":"805'3100","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'10164","last_scrub_stamp":"2024-04-15T18:48:56.918932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:48:56.918932+0000","objects_scrubbed":3417,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:21.699277+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23396352,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13193,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"865'13101","reported_seq":13370,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:20.454826+0000","last_change":"2024-04-15T18:49:06.655574+0000","last_active":"2024-04-15T18:50:20.454826+0000","last_peered":"2024-04-15T18:50:20.454826+0000","last_clean":"2024-04-15T18:50:20.454826+0000","last_became_active":"2024-04-15T18:45:49.841997+0000","last_became_peered":"2024-04-15T18:45:49.841997+0000","last_unstale":"2024-04-15T18:50:20.454826+0000","last_undegraded":"2024-04-15T18:50:20.454826+0000","last_fullsized":"2024-04-15T18:50:20.454826+0000","mapping_epoch":791,"log_start":"805'3100","ondisk_log_start":"805'3100","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"842'10385","last_scrub_stamp":"2024-04-15T18:49:06.655307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:49:06.655307+0000","objects_scrubbed":2956,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:50:26.435085+0000","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13101,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"865'13208","reported_seq":13428,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.110301+0000","last_change":"2024-04-15T18:46:55.950320+0000","last_active":"2024-04-15T18:50:18.110301+0000","last_peered":"2024-04-15T18:50:18.110301+0000","last_clean":"2024-04-15T18:50:18.110301+0000","last_became_active":"2024-04-15T18:45:49.843914+0000","last_became_peered":"2024-04-15T18:45:49.843914+0000","last_unstale":"2024-04-15T18:50:18.110301+0000","last_undegraded":"2024-04-15T18:50:18.110301+0000","last_fullsized":"2024-04-15T18:50:18.110301+0000","mapping_epoch":791,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5238","last_scrub_stamp":"2024-04-15T18:46:55.950094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:55.950094+0000","objects_scrubbed":5218,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22544384,"num_objects":344,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13208,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"865'13288","reported_seq":25467,"reported_epoch":865,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:50:19.065668+0000","last_change":"2024-04-15T18:47:30.615221+0000","last_active":"2024-04-15T18:50:19.065668+0000","last_peered":"2024-04-15T18:50:19.065668+0000","last_clean":"2024-04-15T18:47:20.186030+0000","last_became_active":"2024-04-15T18:47:22.838451+0000","last_became_peered":"2024-04-15T18:47:22.838451+0000","last_unstale":"2024-04-15T18:50:19.065668+0000","last_undegraded":"2024-04-15T18:47:30.608553+0000","last_fullsized":"2024-04-15T18:47:22.203522+0000","mapping_epoch":821,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":322,"num_object_clones":0,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":435520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6008,"num_bytes_recovered":373948416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,5],"avail_no_missing":["10","3","5"],"object_location_counts":[{"shards":"5,10","objects":-390}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"865'13233","reported_seq":13392,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:19.029396+0000","last_change":"2024-04-15T18:45:49.841390+0000","last_active":"2024-04-15T18:50:19.029396+0000","last_peered":"2024-04-15T18:50:19.029396+0000","last_clean":"2024-04-15T18:50:19.029396+0000","last_became_active":"2024-04-15T18:45:49.841033+0000","last_became_peered":"2024-04-15T18:45:49.841033+0000","last_unstale":"2024-04-15T18:50:19.029396+0000","last_undegraded":"2024-04-15T18:50:19.029396+0000","last_fullsized":"2024-04-15T18:50:19.029396+0000","mapping_epoch":791,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13233,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"865'13212","reported_seq":13372,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.096628+0000","last_change":"2024-04-15T18:45:49.841903+0000","last_active":"2024-04-15T18:50:18.096628+0000","last_peered":"2024-04-15T18:50:18.096628+0000","last_clean":"2024-04-15T18:50:18.096628+0000","last_became_active":"2024-04-15T18:45:49.841652+0000","last_became_peered":"2024-04-15T18:45:49.841652+0000","last_unstale":"2024-04-15T18:50:18.096628+0000","last_undegraded":"2024-04-15T18:50:18.096628+0000","last_fullsized":"2024-04-15T18:50:18.096628+0000","mapping_epoch":791,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"865'13164","reported_seq":26502,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.856173+0000","last_change":"2024-04-15T18:50:00.817963+0000","last_active":"2024-04-15T18:50:18.856173+0000","last_peered":"2024-04-15T18:50:18.856173+0000","last_clean":"2024-04-15T18:50:18.856173+0000","last_became_active":"2024-04-15T18:50:00.775637+0000","last_became_peered":"2024-04-15T18:50:00.775637+0000","last_unstale":"2024-04-15T18:50:18.856173+0000","last_undegraded":"2024-04-15T18:50:18.856173+0000","last_fullsized":"2024-04-15T18:50:18.856173+0000","mapping_epoch":855,"log_start":"804'3100","ondisk_log_start":"804'3100","created":791,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":298,"num_object_clones":0,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":430784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6646,"num_bytes_recovered":420675584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"865'13410","reported_seq":13627,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:17.901723+0000","last_change":"2024-04-15T18:46:57.262366+0000","last_active":"2024-04-15T18:50:17.901723+0000","last_peered":"2024-04-15T18:50:17.901723+0000","last_clean":"2024-04-15T18:50:17.901723+0000","last_became_active":"2024-04-15T18:45:49.844745+0000","last_became_peered":"2024-04-15T18:45:49.844745+0000","last_unstale":"2024-04-15T18:50:17.901723+0000","last_undegraded":"2024-04-15T18:50:17.901723+0000","last_fullsized":"2024-04-15T18:50:17.901723+0000","mapping_epoch":791,"log_start":"805'3400","ondisk_log_start":"805'3400","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"812'5319","last_scrub_stamp":"2024-04-15T18:46:57.262149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:46:57.262149+0000","objects_scrubbed":5273,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":380,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13410,"num_write_kb":441280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"865'13219","reported_seq":16126,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.898856+0000","last_change":"2024-04-15T18:50:16.169408+0000","last_active":"2024-04-15T18:50:18.898856+0000","last_peered":"2024-04-15T18:50:18.898856+0000","last_clean":"2024-04-15T18:50:18.898856+0000","last_became_active":"2024-04-15T18:50:16.105836+0000","last_became_peered":"2024-04-15T18:50:16.105836+0000","last_unstale":"2024-04-15T18:50:18.898856+0000","last_undegraded":"2024-04-15T18:50:18.898856+0000","last_fullsized":"2024-04-15T18:50:18.898856+0000","mapping_epoch":863,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":864,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'5799","last_scrub_stamp":"2024-04-15T18:47:05.437371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:05.437371+0000","objects_scrubbed":5752,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13219,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"865'13258","reported_seq":13418,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:17.896096+0000","last_change":"2024-04-15T18:45:49.836537+0000","last_active":"2024-04-15T18:50:17.896096+0000","last_peered":"2024-04-15T18:50:17.896096+0000","last_clean":"2024-04-15T18:50:17.896096+0000","last_became_active":"2024-04-15T18:45:49.836251+0000","last_became_peered":"2024-04-15T18:45:49.836251+0000","last_unstale":"2024-04-15T18:50:17.896096+0000","last_undegraded":"2024-04-15T18:50:17.896096+0000","last_fullsized":"2024-04-15T18:50:17.896096+0000","mapping_epoch":791,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"865'13307","reported_seq":13531,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-04-15T18:50:18.026561+0000","last_change":"2024-04-15T18:47:01.868689+0000","last_active":"2024-04-15T18:50:18.026561+0000","last_peered":"2024-04-15T18:50:18.026561+0000","last_clean":"2024-04-15T18:50:18.026561+0000","last_became_active":"2024-04-15T18:45:50.180630+0000","last_became_peered":"2024-04-15T18:45:50.180630+0000","last_unstale":"2024-04-15T18:50:18.026561+0000","last_undegraded":"2024-04-15T18:50:18.026561+0000","last_fullsized":"2024-04-15T18:50:18.026561+0000","mapping_epoch":791,"log_start":"805'3300","ondisk_log_start":"805'3300","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"814'5601","last_scrub_stamp":"2024-04-15T18:47:01.868503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:47:01.868503+0000","objects_scrubbed":5580,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13307,"num_write_kb":436928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"865'13391","reported_seq":26080,"reported_epoch":865,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:50:17.908938+0000","last_change":"2024-04-15T18:47:32.874160+0000","last_active":"2024-04-15T18:50:17.908938+0000","last_peered":"2024-04-15T18:50:17.908938+0000","last_clean":"2024-04-15T18:47:20.171208+0000","last_became_active":"2024-04-15T18:47:22.829093+0000","last_became_peered":"2024-04-15T18:47:22.829093+0000","last_unstale":"2024-04-15T18:50:17.908938+0000","last_undegraded":"2024-04-15T18:47:32.870404+0000","last_fullsized":"2024-04-15T18:47:22.201052+0000","mapping_epoch":821,"log_start":"805'3300","ondisk_log_start":"805'3300","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13391,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6263,"num_bytes_recovered":382795776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6],"avail_no_missing":["1","6","8"],"object_location_counts":[{"shards":"1,6","objects":-179}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"865'13231","reported_seq":24923,"reported_epoch":865,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-15T18:50:18.039713+0000","last_change":"2024-04-15T18:47:32.882544+0000","last_active":"2024-04-15T18:50:18.039713+0000","last_peered":"2024-04-15T18:50:18.039713+0000","last_clean":"2024-04-15T18:47:20.189472+0000","last_became_active":"2024-04-15T18:47:22.835987+0000","last_became_peered":"2024-04-15T18:47:22.835987+0000","last_unstale":"2024-04-15T18:50:18.039713+0000","last_undegraded":"2024-04-15T18:47:32.878238+0000","last_fullsized":"2024-04-15T18:47:22.203408+0000","mapping_epoch":821,"log_start":"805'3200","ondisk_log_start":"805'3200","created":791,"last_epoch_clean":792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T18:45:48.819687+0000","last_clean_scrub_stamp":"2024-04-15T18:45:48.819687+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23527424,"num_objects":359,"num_object_clones":0,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13231,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5764,"num_bytes_recovered":363855872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,5],"avail_no_missing":["9","5","7"],"object_location_counts":[{"shards":"5,9","objects":-564}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"509'139","reported_seq":2125,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-04-15T18:50:16.127071+0000","last_change":"2024-04-15T18:47:24.773236+0000","last_active":"2024-04-15T18:50:16.127071+0000","last_peered":"2024-04-15T18:50:16.127071+0000","last_clean":"2024-04-15T18:50:16.127071+0000","last_became_active":"2024-04-15T18:47:24.772930+0000","last_became_peered":"2024-04-15T18:47:24.772930+0000","last_unstale":"2024-04-15T18:50:16.127071+0000","last_undegraded":"2024-04-15T18:50:16.127071+0000","last_fullsized":"2024-04-15T18:50:16.127071+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:46:52.053607+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:46:52.053607+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":350814232,"num_objects":5354,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211908,"num_write_kb":6952321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25348,"num_bytes_recovered":1584988160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2437230592,"data_stored":2437218376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":48,"acting":45,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":807,"seq":3466038607918,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344044,"kb_used_data":96548,"kb_used_omap":144,"kb_used_meta":247343,"kb_avail":93401108,"statfs":{"total":95995035648,"available":95642734592,"internally_reserved":0,"allocated":98865152,"data_stored":98504158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147868,"internal_metadata":253279844},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187957,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768876,"kb_used_data":110172,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92976276,"statfs":{"total":95995035648,"available":95207706624,"internally_reserved":0,"allocated":112816128,"data_stored":112458698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":674494912},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886440,"kb_used_data":115292,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92858712,"statfs":{"total":95995035648,"available":95087321088,"internally_reserved":0,"allocated":118059008,"data_stored":117701578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":789641664},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874748,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":932260,"kb_used_data":481820,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92812892,"statfs":{"total":95995035648,"available":95040401408,"internally_reserved":0,"allocated":493383680,"data_stored":493026250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169089,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161960,"kb_used_data":557404,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92583192,"statfs":{"total":95995035648,"available":94805188608,"internally_reserved":0,"allocated":570781696,"data_stored":570421012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":619051466},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363936,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274220,"kb_used_data":30756,"kb_used_omap":111,"kb_used_meta":243344,"kb_avail":93470932,"statfs":{"total":95995035648,"available":95714234368,"internally_reserved":0,"allocated":31494144,"data_stored":31133170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":249184923},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575395,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064028,"kb_used_data":234580,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92681124,"statfs":{"total":95995035648,"available":94905470976,"internally_reserved":0,"allocated":240209920,"data_stored":239852788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681836,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063316,"statfs":{"total":95995035648,"available":95296835584,"internally_reserved":0,"allocated":946176,"data_stored":588746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524384,"kb_used_data":912,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220768,"statfs":{"total":95995035648,"available":95458066432,"internally_reserved":0,"allocated":933888,"data_stored":580828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354912,"kb_used_data":549840,"kb_used_omap":44,"kb_used_meta":805011,"kb_avail":92390240,"statfs":{"total":95995035648,"available":94607605760,"internally_reserved":0,"allocated":563036160,"data_stored":562683100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":824331818},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749340,"kb_used_data":68432,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":92995812,"statfs":{"total":95995035648,"available":95227711488,"internally_reserved":0,"allocated":70074368,"data_stored":69721308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703599,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":883628,"kb_used_data":205216,"kb_used_omap":4,"kb_used_meta":678395,"kb_avail":92861524,"statfs":{"total":95995035648,"available":95090200576,"internally_reserved":0,"allocated":210141184,"data_stored":209779682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":694677185},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":216403968,"data_stored":216399896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":554041344,"data_stored":554041344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":68419584,"data_stored":68419584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":202379264,"data_stored":202375192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27590656,"data_stored":27590656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":567742464,"data_stored":567738392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":488964096,"data_stored":488964096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":114950144,"data_stored":114950144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":99614720,"data_stored":99614720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":97124352,"data_stored":97124352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:50:27.256 INFO:tasks.thrashosds.thrasher:Setting 9.a to [7, 11, 9] 2024-04-15T18:50:27.256 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '9.a', '7', '11', '9'] 2024-04-15T18:50:27.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pg-upmap 9.a 7 11 9 2024-04-15T18:50:27.684 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:27.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:27 smithi049 ceph-mon[40087]: pgmap v2275: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 105 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 1274/5013 objects degraded (25.414%); 2.2 MiB/s, 244 objects/s recovering 2024-04-15T18:50:27.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:27 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:27.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:27 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:27.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:27 smithi049 ceph-mon[40087]: osdmap e868: 12 total, 12 up, 12 in 2024-04-15T18:50:28.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:27 smithi082 ceph-mon[44206]: pgmap v2275: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 105 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 1274/5013 objects degraded (25.414%); 2.2 MiB/s, 244 objects/s recovering 2024-04-15T18:50:28.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:27 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:28.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:27 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:28.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:27 smithi082 ceph-mon[44206]: osdmap e868: 12 total, 12 up, 12 in 2024-04-15T18:50:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:27 smithi159 ceph-mon[45336]: pgmap v2275: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 105 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 1274/5013 objects degraded (25.414%); 2.2 MiB/s, 244 objects/s recovering 2024-04-15T18:50:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:27 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:27 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:28.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:27 smithi159 ceph-mon[45336]: osdmap e868: 12 total, 12 up, 12 in 2024-04-15T18:50:28.545 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:28.549 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:28.725 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:28.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:28.829 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:28.858 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:28.878 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:28.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:28.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:28 smithi049 ceph-mon[40087]: osdmap e869: 12 total, 12 up, 12 in 2024-04-15T18:50:28.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:28 smithi049 ceph-mon[40087]: pgmap v2278: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 85 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1274/4047 objects degraded (31.480%); 0 B/s, 144 objects/s recovering 2024-04-15T18:50:28.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:28 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 1274/5013 objects degraded (25.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:28.982 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:29.005 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:28 smithi159 ceph-mon[45336]: osdmap e869: 12 total, 12 up, 12 in 2024-04-15T18:50:29.005 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:28 smithi159 ceph-mon[45336]: pgmap v2278: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 85 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1274/4047 objects degraded (31.480%); 0 B/s, 144 objects/s recovering 2024-04-15T18:50:29.005 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:28 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 1274/5013 objects degraded (25.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:29.008 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:28 smithi082 ceph-mon[44206]: osdmap e869: 12 total, 12 up, 12 in 2024-04-15T18:50:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:28 smithi082 ceph-mon[44206]: pgmap v2278: 17 pgs: 3 active+recovering+undersized+degraded+remapped, 14 active+clean; 85 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 1274/4047 objects degraded (31.480%); 0 B/s, 144 objects/s recovering 2024-04-15T18:50:29.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:28 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 1274/5013 objects degraded (25.414%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-04-15T18:50:29.027 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:29.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:29.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:29.166 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.187 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:29.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:29.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:29.322 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.342 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:29.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:29.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:29.477 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.532 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:29.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:29.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:29.664 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:50:29.685 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:29.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:29.760 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-04-15T18:50:29.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-04-15T18:50:29.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: osdmap e870: 12 total, 12 up, 12 in 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:50:29.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:50:29.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:29.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:29.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:29.963 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: osdmap e870: 12 total, 12 up, 12 in 2024-04-15T18:50:30.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:30.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.087 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:30.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: osdmap e870: 12 total, 12 up, 12 in 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:50:30.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:50:30.191 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:30.220 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:30.241 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:30.345 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:30.373 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:30.391 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:30.495 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:30.526 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:30.547 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:30.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:30.680 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:30.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:30 smithi082 ceph-mon[44206]: pgmap v2280: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 470 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 234 objects/s recovering 2024-04-15T18:50:30.775 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:30 smithi082 ceph-mon[44206]: osdmap e871: 12 total, 12 up, 12 in 2024-04-15T18:50:30.803 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:30.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:30 smithi049 ceph-mon[40087]: pgmap v2280: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 470 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 234 objects/s recovering 2024-04-15T18:50:30.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:30 smithi049 ceph-mon[40087]: osdmap e871: 12 total, 12 up, 12 in 2024-04-15T18:50:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:30.910 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:30.942 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:30.964 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:30.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:31.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:31.096 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:30 smithi159 ceph-mon[45336]: pgmap v2280: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 470 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 234 objects/s recovering 2024-04-15T18:50:31.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:30 smithi159 ceph-mon[45336]: osdmap e871: 12 total, 12 up, 12 in 2024-04-15T18:50:31.195 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:31.238 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:31.298 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:31.326 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.345 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:31.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:31.475 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.493 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:31.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:31.630 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.652 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:31.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:50:31.783 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.802 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:31.906 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:31.932 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:31.951 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:31.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:32.009 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:50:32.009 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:50:32.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:50:32.009 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:50:32.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26545.0:191961 7.d e700/672 7:b225a1f7:::benchmark_data_smithi159_149998_object95961:head v 700'11971, mlcod=700'11968)", 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 596.14770299500003, 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43827476300000001, 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873945+0000", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9999999999999998e-07 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:35.873949+0000", 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2289999999999997e-06 2024-04-15T18:50:32.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308026+0000", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.43407702100000001 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308036+0000", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4180000000000003e-06 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308109+0000", 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3162000000000002e-05 2024-04-15T18:50:32.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.308139+0000", 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0298e-05 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312172+0000", 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004033043 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:32.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:40:36.312219+0000", 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7191999999999998e-05 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:109194 8.9 e749/709 8:93c68f30:::benchmark_data_smithi159_162467_object6889:head v 749'6779, mlcod=749'6776)", 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 452.36167231299999, 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25230181200000001, 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:32.014 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659975+0000", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659976+0000", 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-04-15T18:50:32.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.659983+0000", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6229999999999999e-06 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905691+0000", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24570813499999999 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905702+0000", 2024-04-15T18:50:32.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0795000000000001e-05 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905778+0000", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6173000000000004e-05 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.905804+0000", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5799000000000002e-05 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.912212+0000", 2024-04-15T18:50:32.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064086550000000001 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:59.912277+0000", 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4852999999999997e-05 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:113660 8.9 e751/709 8:9831ce8f:::benchmark_data_smithi159_162467_object11355:head v 751'7050, mlcod=750'7048)", 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 445.08209940900002, 2024-04-15T18:50:32.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17086718400000001, 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939548+0000", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4299999999999999e-07 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:06.939555+0000", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0090000000000001e-06 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106710+0000", 2024-04-15T18:50:32.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16715449299999999 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106720+0000", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0281e-05 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106807+0000", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6903999999999993e-05 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.106833+0000", 2024-04-15T18:50:32.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5537e-05 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.110352+0000", 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035193429999999999 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:07.110415+0000", 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3373999999999993e-05 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:128498 8.e e754/709 8:7c81d7a0:::benchmark_data_smithi159_162467_object26193:head v 754'8122, mlcod=754'8119)", 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 421.517056287, 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22033059499999999, 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504591+0000", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504592+0000", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.504601+0000", 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5019999999999992e-06 2024-04-15T18:50:32.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721186+0000", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21658566800000001 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721196+0000", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.73e-06 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721252+0000", 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5627000000000001e-05 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.721276+0000", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4094999999999999e-05 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.724872+0000", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035959630000000002 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:43:30.724922+0000", 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0042e-05 2024-04-15T18:50:32.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:164219 8.e e766/709 8:73021211:::benchmark_data_smithi159_162467_object61914:head v 766'10389, mlcod=766'10386)", 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 362.94218617299998, 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.370137667, 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:32.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079461+0000", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079463+0000", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3430000000000001e-06 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:32.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079472+0000", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7669999999999998e-06 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444723+0000", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36525010400000002 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444733+0000", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9939999999999995e-06 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444801+0000", 2024-04-15T18:50:32.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8874000000000005e-05 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.444824+0000", 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2362e-05 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449560+0000", 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004735815 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:32.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449599+0000", 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9407999999999999e-05 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:32.031 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:50:32.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:32.086 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:32.106 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:32.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:32.210 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:32.241 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:32.262 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:32.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:32.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3 2024-04-15T18:50:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:32.606 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T18:50:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:32.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.3' 2024-04-15T18:50:32.718 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:32.747 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:32.768 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:32.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:32.777 INFO:teuthology.orchestra.run.smithi049.stderr:set 9.a pg_upmap mapping to [7,11,9] 2024-04-15T18:50:32.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:32.904 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.002 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:33.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:32 smithi082 ceph-mon[44206]: pgmap v2282: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 216 objects/s recovering 2024-04-15T18:50:33.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]: dispatch 2024-04-15T18:50:33.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:32 smithi082 ceph-mon[44206]: Health check update: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:50:33.109 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:50:33.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:32 smithi159 ceph-mon[45336]: pgmap v2282: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 216 objects/s recovering 2024-04-15T18:50:33.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]: dispatch 2024-04-15T18:50:33.135 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:32 smithi159 ceph-mon[45336]: Health check update: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:50:33.137 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.157 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:33.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:32 smithi049 ceph-mon[40087]: pgmap v2282: 17 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 332 op/s; 189/6 objects degraded (3150.000%); 0 B/s, 216 objects/s recovering 2024-04-15T18:50:33.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]: dispatch 2024-04-15T18:50:33.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:32 smithi049 ceph-mon[40087]: Health check update: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-04-15T18:50:33.261 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:33.288 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.307 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:33.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:33.414 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:33.443 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-15T18:50:33.542 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:33.647 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:33.675 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.694 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:33.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:33.829 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:33.850 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:33.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:33.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:33.984 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]': finished 2024-04-15T18:50:34.192 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:33 smithi159 ceph-mon[45336]: osdmap e872: 12 total, 12 up, 12 in 2024-04-15T18:50:34.192 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:33 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:34.192 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:33 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:34.193 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:34.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]': finished 2024-04-15T18:50:34.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:33 smithi049 ceph-mon[40087]: osdmap e872: 12 total, 12 up, 12 in 2024-04-15T18:50:34.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:33 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:34.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:33 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/138334888' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "9.a", "id": [7, 11, 9]}]': finished 2024-04-15T18:50:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:33 smithi082 ceph-mon[44206]: osdmap e872: 12 total, 12 up, 12 in 2024-04-15T18:50:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:33 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:34.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:33 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:34.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:34.302 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:50:34.302 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:50:34.302 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:50:34.302 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:50:34.309 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:50:34.326 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:50:34.348 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:34.359 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:50:34.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:34.483 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.504 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:34.608 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:34.637 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.657 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:34.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:34.794 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.816 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:34.920 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:34.949 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:34.969 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:34.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:35.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: pgmap v2284: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 0 B/s, 212 objects/s recovering 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osdmap e873: 12 total, 12 up, 12 in 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3750014407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.114 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.115 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:34 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.116 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:35.134 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:35.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:35.187 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:50:35.211 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:50:35.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: pgmap v2284: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 0 B/s, 212 objects/s recovering 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osdmap e873: 12 total, 12 up, 12 in 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3750014407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:34 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.236 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:50:35.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:35.260 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:50:35.264 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:35.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: pgmap v2284: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 224 op/s; 0 B/s, 212 objects/s recovering 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 189/6 objects degraded (3150.000%), 1 pg degraded, 1 pg undersized) 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osdmap e873: 12 total, 12 up, 12 in 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3750014407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:34 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.283 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:35.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:35.286 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:50:35.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:50:35.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:50:35.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:50:35.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:50:35.289 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:50:35.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:35.415 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:35.540 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:35.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:35.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:35.672 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:35.690 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:35.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:35.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:35.829 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:35.851 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:35.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:35.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:35.979 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.980 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.980 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: osdmap e874: 12 total, 12 up, 12 in 2024-04-15T18:50:35.980 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:35.980 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:35.980 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:35 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:35.981 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.000 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:36.104 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:36.134 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.134 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.134 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: osdmap e874: 12 total, 12 up, 12 in 2024-04-15T18:50:36.135 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:36.135 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:36.135 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:35 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.137 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.159 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: osdmap e874: 12 total, 12 up, 12 in 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: 9.a starting backfill to osd.9 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: 9.a starting backfill to osd.11 from (0'0,0'0] MAX to 867'13552 2024-04-15T18:50:36.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:35 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:36.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:36.292 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.311 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:36.415 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:36.442 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.461 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:36.565 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:36.595 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.616 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:36.721 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10 2024-04-15T18:50:36.747 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:36.766 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10" found: no such container 2024-04-15T18:50:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:36.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.10' 2024-04-15T18:50:36.810 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:50:36.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:36.904 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: pgmap v2287: 17 pgs: 17 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 31 objects/s recovering 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: osdmap e875: 12 total, 12 up, 12 in 2024-04-15T18:50:37.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:36 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:37.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.077 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: pgmap v2287: 17 pgs: 17 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 31 objects/s recovering 2024-04-15T18:50:37.078 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: osdmap e875: 12 total, 12 up, 12 in 2024-04-15T18:50:37.078 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:36 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:37.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: pgmap v2287: 17 pgs: 17 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 31 objects/s recovering 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: osdmap e875: 12 total, 12 up, 12 in 2024-04-15T18:50:37.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:36 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:37.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:38.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:37 smithi049 ceph-mon[40087]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:38.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:38.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:37 smithi049 ceph-mon[40087]: osdmap e876: 12 total, 12 up, 12 in 2024-04-15T18:50:38.084 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:37 smithi049 ceph-mon[40087]: osdmap e877: 12 total, 12 up, 12 in 2024-04-15T18:50:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:37 smithi159 ceph-mon[45336]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:37 smithi159 ceph-mon[45336]: osdmap e876: 12 total, 12 up, 12 in 2024-04-15T18:50:38.145 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:37 smithi159 ceph-mon[45336]: osdmap e877: 12 total, 12 up, 12 in 2024-04-15T18:50:38.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:37 smithi082 ceph-mon[44206]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-15T18:50:38.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4278304915' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:38.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:37 smithi082 ceph-mon[44206]: osdmap e876: 12 total, 12 up, 12 in 2024-04-15T18:50:38.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:37 smithi082 ceph-mon[44206]: osdmap e877: 12 total, 12 up, 12 in 2024-04-15T18:50:39.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-15T18:50:39.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:38 smithi049 ceph-mon[40087]: pgmap v2290: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:39.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:38 smithi049 ceph-mon[40087]: 9.5 scrub starts 2024-04-15T18:50:39.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:38 smithi049 ceph-mon[40087]: 9.5 scrub ok 2024-04-15T18:50:39.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:38 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:39.131 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:38 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:38 smithi159 ceph-mon[45336]: pgmap v2290: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:38 smithi159 ceph-mon[45336]: 9.5 scrub starts 2024-04-15T18:50:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:38 smithi159 ceph-mon[45336]: 9.5 scrub ok 2024-04-15T18:50:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:38 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:39.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:38 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:38 smithi082 ceph-mon[44206]: pgmap v2290: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:38 smithi082 ceph-mon[44206]: 9.5 scrub starts 2024-04-15T18:50:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:38 smithi082 ceph-mon[44206]: 9.5 scrub ok 2024-04-15T18:50:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:38 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:39.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:38 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:39.491 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:39.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:39.498 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:50:39.498 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:50:39.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2 2024-04-15T18:50:39.627 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:39.775 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T18:50:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:39.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.2' 2024-04-15T18:50:39.879 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:39.906 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:39.925 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:39.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:40.029 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4 2024-04-15T18:50:40.055 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:40.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:39 smithi049 ceph-mon[40087]: 9.a scrub starts 2024-04-15T18:50:40.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:39 smithi049 ceph-mon[40087]: 9.a scrub ok 2024-04-15T18:50:40.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:39 smithi049 ceph-mon[40087]: osdmap e878: 12 total, 12 up, 12 in 2024-04-15T18:50:40.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:39 smithi049 ceph-mon[40087]: 9.3 scrub starts 2024-04-15T18:50:40.063 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:39 smithi049 ceph-mon[40087]: 9.3 scrub ok 2024-04-15T18:50:40.073 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T18:50:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:40.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.4' 2024-04-15T18:50:40.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:50:40.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:39 smithi159 ceph-mon[45336]: 9.a scrub starts 2024-04-15T18:50:40.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:39 smithi159 ceph-mon[45336]: 9.a scrub ok 2024-04-15T18:50:40.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:39 smithi159 ceph-mon[45336]: osdmap e878: 12 total, 12 up, 12 in 2024-04-15T18:50:40.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:39 smithi159 ceph-mon[45336]: 9.3 scrub starts 2024-04-15T18:50:40.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:39 smithi159 ceph-mon[45336]: 9.3 scrub ok 2024-04-15T18:50:40.177 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:40.209 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:40.230 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:40.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:39 smithi082 ceph-mon[44206]: 9.a scrub starts 2024-04-15T18:50:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:39 smithi082 ceph-mon[44206]: 9.a scrub ok 2024-04-15T18:50:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:39 smithi082 ceph-mon[44206]: osdmap e878: 12 total, 12 up, 12 in 2024-04-15T18:50:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:39 smithi082 ceph-mon[44206]: 9.3 scrub starts 2024-04-15T18:50:40.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:39 smithi082 ceph-mon[44206]: 9.3 scrub ok 2024-04-15T18:50:40.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:40.362 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:40.610 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:40.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:40.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-15T18:50:40.715 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7 2024-04-15T18:50:40.746 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:40.767 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T18:50:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:40.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.7' 2024-04-15T18:50:40.872 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5 2024-04-15T18:50:40.901 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:40.921 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T18:50:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:40.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.5' 2024-04-15T18:50:40.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 9.9 scrub starts 2024-04-15T18:50:40.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 9.9 scrub ok 2024-04-15T18:50:40.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:50:40.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: pgmap v2293: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:40.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:50:40.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 9.b scrub starts 2024-04-15T18:50:40.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:40 smithi049 ceph-mon[40087]: 9.b scrub ok 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 9.9 scrub starts 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 9.9 scrub ok 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: pgmap v2293: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 9.b scrub starts 2024-04-15T18:50:41.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:40 smithi082 ceph-mon[44206]: 9.b scrub ok 2024-04-15T18:50:41.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6 2024-04-15T18:50:41.053 INFO:teuthology.orchestra.run.smithi082.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.073 INFO:teuthology.orchestra.run.smithi082.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T18:50:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi082 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.6' 2024-04-15T18:50:41.087 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 9.9 scrub starts 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 9.9 scrub ok 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: pgmap v2293: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 9.b scrub starts 2024-04-15T18:50:41.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:40 smithi159 ceph-mon[45336]: 9.b scrub ok 2024-04-15T18:50:41.176 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:41.208 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.229 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:41.333 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:41.364 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.385 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:41.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:41.518 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.556 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:41.660 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8 2024-04-15T18:50:41.692 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.713 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8" found: no such container 2024-04-15T18:50:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.8' 2024-04-15T18:50:41.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0 2024-04-15T18:50:41.846 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:41.884 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T18:50:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:41.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.0' 2024-04-15T18:50:41.987 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9 2024-04-15T18:50:42.016 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:41 smithi159 ceph-mon[45336]: 9.7 scrub starts 2024-04-15T18:50:42.016 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:41 smithi159 ceph-mon[45336]: 9.7 scrub ok 2024-04-15T18:50:42.018 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:42.039 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9" found: no such container 2024-04-15T18:50:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:42.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.9' 2024-04-15T18:50:42.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:41 smithi049 ceph-mon[40087]: 9.7 scrub starts 2024-04-15T18:50:42.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:41 smithi049 ceph-mon[40087]: 9.7 scrub ok 2024-04-15T18:50:42.143 DEBUG:teuthology.orchestra.run.smithi159:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11 2024-04-15T18:50:42.171 INFO:teuthology.orchestra.run.smithi159.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:42.191 INFO:teuthology.orchestra.run.smithi159.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11" found: no such container 2024-04-15T18:50:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:42.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi159 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.11' 2024-04-15T18:50:42.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:41 smithi082 ceph-mon[44206]: 9.7 scrub starts 2024-04-15T18:50:42.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:41 smithi082 ceph-mon[44206]: 9.7 scrub ok 2024-04-15T18:50:42.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1 2024-04-15T18:50:42.326 INFO:teuthology.orchestra.run.smithi049.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-04-15T18:50:42.867 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_7' removed 2024-04-15T18:50:43.032 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:43.032 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: pgmap v2294: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: 9.0 scrub starts 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: 9.0 scrub ok 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:50:43.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:42 smithi159 ceph-mon[45336]: osdmap e879: 12 total, 12 up, 12 in 2024-04-15T18:50:43.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: pgmap v2294: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:43.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: 9.0 scrub starts 2024-04-15T18:50:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: 9.0 scrub ok 2024-04-15T18:50:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:50:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:50:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:42 smithi049 ceph-mon[40087]: osdmap e879: 12 total, 12 up, 12 in 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: pgmap v2294: 17 pgs: 1 peering, 16 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: 9.0 scrub starts 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: 9.0 scrub ok 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1640828904' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-04-15T18:50:43.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:42 smithi082 ceph-mon[44206]: osdmap e879: 12 total, 12 up, 12 in 2024-04-15T18:50:43.777 INFO:teuthology.orchestra.run.smithi049.stderr:Error: no container with name or ID "ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T18:50:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-04-15T18:50:43.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 125: 'sudo docker kill -s 1 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd.1' 2024-04-15T18:50:43.788 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-15T18:50:43.797 INFO:tasks.thrashosds:joining thrashosds 2024-04-15T18:50:44.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:43 smithi049 ceph-mon[40087]: 9.8 scrub starts 2024-04-15T18:50:44.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:43 smithi049 ceph-mon[40087]: 9.8 scrub ok 2024-04-15T18:50:44.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:43 smithi049 ceph-mon[40087]: 9.e scrub starts 2024-04-15T18:50:44.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:43 smithi049 ceph-mon[40087]: 9.e scrub ok 2024-04-15T18:50:44.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:43 smithi159 ceph-mon[45336]: 9.8 scrub starts 2024-04-15T18:50:44.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:43 smithi159 ceph-mon[45336]: 9.8 scrub ok 2024-04-15T18:50:44.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:43 smithi159 ceph-mon[45336]: 9.e scrub starts 2024-04-15T18:50:44.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:43 smithi159 ceph-mon[45336]: 9.e scrub ok 2024-04-15T18:50:44.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:43 smithi082 ceph-mon[44206]: 9.8 scrub starts 2024-04-15T18:50:44.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:43 smithi082 ceph-mon[44206]: 9.8 scrub ok 2024-04-15T18:50:44.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:43 smithi082 ceph-mon[44206]: 9.e scrub starts 2024-04-15T18:50:44.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:43 smithi082 ceph-mon[44206]: 9.e scrub ok 2024-04-15T18:50:45.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-15T18:50:45.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:44 smithi049 ceph-mon[40087]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:45.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:44 smithi159 ceph-mon[45336]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:45.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:44 smithi082 ceph-mon[44206]: pgmap v2296: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:46.490 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-04-15T18:50:46.491 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-15T18:50:46.692 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:47.075 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:46 smithi049 ceph-mon[40087]: pgmap v2297: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:47.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-15T18:50:47.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:46 smithi159 ceph-mon[45336]: pgmap v2297: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:47.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:46 smithi082 ceph-mon[44206]: pgmap v2297: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:48.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-04-15T18:50:48.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-04-15T18:50:48.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-04-15T18:50:48.781 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-04-15T18:50:48.819 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-04-15T18:50:49.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:48 smithi082 ceph-mon[44206]: pgmap v2298: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:49.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3684303869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:49.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:48 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:49.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:48 smithi049 ceph-mon[40087]: pgmap v2298: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:49.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3684303869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:49.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:48 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:49.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-04-15T18:50:49.074 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-04-15T18:50:49.106 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-04-15T18:50:49.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8: {} 2024-04-15T18:50:49.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:48 smithi159 ceph-mon[45336]: pgmap v2298: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:49.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3684303869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-15T18:50:49.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:48 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:49.156 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9: {} 2024-04-15T18:50:49.189 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10: {} 2024-04-15T18:50:49.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11: {} 2024-04-15T18:50:49.215 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-15T18:50:49.215 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-15T18:50:49.215 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-15T18:50:49.216 INFO:teuthology.orchestra.run.smithi049.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-15T18:50:49.857 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:50.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osdmap e880: 12 total, 12 up, 12 in 2024-04-15T18:50:50.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:49 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osdmap e880: 12 total, 12 up, 12 in 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:49 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:50:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osdmap e880: 12 total, 12 up, 12 in 2024-04-15T18:50:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:50.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:49 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.003 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: pgmap v2300: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: Cluster is now healthy 2024-04-15T18:50:51.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:51.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:50 smithi082 ceph-mon[44206]: osdmap e881: 12 total, 12 up, 12 in 2024-04-15T18:50:51.059 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-04-15T18:50:51.075 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-04-15T18:50:51.090 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-04-15T18:50:51.091 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:50:51.091 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:50:51.092 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: pgmap v2300: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: Cluster is now healthy 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:51.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:50 smithi049 ceph-mon[40087]: osdmap e881: 12 total, 12 up, 12 in 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-04-15T18:50:51.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: pgmap v2300: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-15T18:50:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: Cluster is now healthy 2024-04-15T18:50:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2161439691' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:50:51.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:50 smithi159 ceph-mon[45336]: osdmap e881: 12 total, 12 up, 12 in 2024-04-15T18:50:52.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:51 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2456471953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:50:52.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:51 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2456471953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:50:52.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 0 1 2024-04-15T18:50:52.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:51 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2456471953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T18:50:52.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset noscrub 2024-04-15T18:50:52.803 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:52 smithi049 ceph-mon[40087]: pgmap v2302: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:52.876 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:53.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:52 smithi082 ceph-mon[44206]: pgmap v2302: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:53.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:52 smithi159 ceph-mon[45336]: pgmap v2302: 1 pgs: 1 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:53.579 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86279 8.7 8:e097fc19:::benchmark_data_smithi159_162467_object86278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 506.81300362399998, 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60823186699999998, 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.580 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86279 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776857+0000", 2024-04-15T18:50:53.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776858+0000", 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.27e-07 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776884+0000", 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6044e-05 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776889+0000", 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6459999999999998e-06 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776893+0000", 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9539999999999998e-06 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.776945+0000", 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1909000000000001e-05 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.777096+0000", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000150371 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.777265+0000", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016954200000000001 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.779828+0000", 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025631930000000001 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.384972+0000", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60514323999999997 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.385012+0000", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0969e-05 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.385089+0000", 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6571999999999998e-05 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.585 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86309 8.7 8:e02e5510:::benchmark_data_smithi159_162467_object86308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 506.72294858200001, 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.52165454600000005, 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:50:53.586 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86309 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866912+0000", 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866919+0000", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9240000000000001e-06 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866972+0000", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2488999999999998e-05 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866984+0000", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2003e-05 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.866990+0000", 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8710000000000002e-06 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867033+0000", 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3448000000000001e-05 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867143+0000", 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109533 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.867249+0000", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106251 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:26.869488+0000", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022394400000000001 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388441+0000", 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.51895257500000003 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388496+0000", 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4972999999999997e-05 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388567+0000", 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1038999999999995e-05 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.591 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:86345 8.7 8:e9b0afce:::benchmark_data_smithi159_162467_object86344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 506.57730314000003, 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.376124766, 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 86345 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012558+0000", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012560+0000", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.911e-06 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012594+0000", 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4558999999999999e-05 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012614+0000", 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0091000000000001e-05 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012623+0000", 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3049999999999993e-06 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012692+0000", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9944000000000007e-05 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012880+0000", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018745 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 10", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.012992+0000", 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111853 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.015515+0000", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025232330000000002 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388593+0000", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37307769600000001 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388624+0000", 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1769e-05 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:27.388682+0000", 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7955000000000003e-05 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDeleteReply(8.7 e786,743 8:efc191eb:::benchmark_data_smithi159_162467_object72738:head,769'10956)", 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 316.18203874300002, 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.723278898, 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:50:53.597 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407822+0000", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407823+0000", 2024-04-15T18:50:53.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3399999999999999e-07 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:37.407826+0000", 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7890000000000001e-06 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130788+0000", 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.72296189200000005 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130797+0000", 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2900000000000002e-06 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.130888+0000", 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1656000000000005e-05 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.131101+0000", 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021273700000000001 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:93314 9.2 9:4a92ac12:::benchmark_data_smithi159_173188_object93313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:06.306756+0000", 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 227.28310442599999, 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27835642700000002, 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 93314 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.601 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306756+0000", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306756+0000", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306757+0000", 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2999999999999999e-07 2024-04-15T18:50:53.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306809+0000", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1941000000000003e-05 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306819+0000", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0152999999999999e-05 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306827+0000", 2024-04-15T18:50:53.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9270000000000005e-06 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306873+0000", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5648000000000001e-05 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.306984+0000", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110771 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.307074+0000", 2024-04-15T18:50:53.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0593000000000005e-05 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.309863+0000", 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002788212 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.311990+0000", 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021278899999999999 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.584989+0000", 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27299846300000002 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.585056+0000", 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7214999999999999e-05 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.585113+0000", 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6684000000000001e-05 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.606 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.27025.0:93314 9.2 e816/791 ondisk, result = 0)", 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:06.309596+0000", 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 227.280264786, 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27552457600000002, 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.309596+0000", 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.309596+0000", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.309597+0000", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0699999999999997e-07 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.309600+0000", 2024-04-15T18:50:53.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.681e-06 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.582773+0000", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27317259199999999 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.582779+0000", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3820000000000001e-06 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.584974+0000", 2024-04-15T18:50:53.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021948720000000001 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.584981+0000", 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5760000000000002e-06 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:06.585121+0000", 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013986599999999999 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:107188 9.f e820/791 9:ff26e8fa:::benchmark_data_smithi159_173188_object107187:head v 820'6581, mlcod=819'6578)", 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:20.225585+0000", 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 213.36427616500001, 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49377124, 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.225585+0000", 2024-04-15T18:50:53.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.225585+0000", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.225585+0000", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-07 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.225673+0000", 2024-04-15T18:50:53.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7714999999999996e-05 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.716368+0000", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.490695351 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.716375+0000", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0260000000000002e-06 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.716437+0000", 2024-04-15T18:50:53.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1583999999999999e-05 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.716457+0000", 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0387e-05 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.719304+0000", 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002847259 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.719356+0000", 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1433e-05 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107193 9.5 9:a5269f66:::benchmark_data_smithi159_173188_object107192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.248213+0000", 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.34164826200001, 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66172050400000004, 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.615 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107193 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248213+0000", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248213+0000", 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248214+0000", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7379999999999999e-06 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248358+0000", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014355800000000001 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248371+0000", 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2988e-05 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248381+0000", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0506000000000001e-05 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252098+0000", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037162340000000001 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252112+0000", 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4093999999999999e-05 2024-04-15T18:50:53.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.854488+0000", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60237639600000004 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.856372+0000", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018836090000000001 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.856491+0000", 2024-04-15T18:50:53.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011883399999999999 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.864468+0000", 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079779259999999998 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.864486+0000", 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7286999999999999e-05 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.864528+0000", 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2412e-05 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.907049+0000", 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.042521064999999997 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.907138+0000", 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8399999999999994e-05 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.909933+0000", 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027954569999999999 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107196 9.5 9:ad84200b:::benchmark_data_smithi159_173188_object107195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.248399+0000", 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.341461452, 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.66565533399999999, 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.622 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107196 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248399+0000", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248399+0000", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248400+0000", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9999999999999998e-07 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248427+0000", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6973000000000001e-05 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248432+0000", 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5330000000000002e-06 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248450+0000", 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8065000000000001e-05 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252148+0000", 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036980170000000001 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252154+0000", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4110000000000004e-06 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.861340+0000", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.60918669599999997 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.861462+0000", 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012170799999999999 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.861574+0000", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111994 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871608+0000", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010033602000000001 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871750+0000", 2024-04-15T18:50:53.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142306 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871837+0000", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6725999999999996e-05 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.910028+0000", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.038191055000000002 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.910109+0000", 2024-04-15T18:50:53.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1482000000000004e-05 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.914055+0000", 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039453659999999996 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107213 9.5 9:a452a4a2:::benchmark_data_smithi159_173188_object107212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.248457+0000", 2024-04-15T18:50:53.629 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.34140385399999, 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.67323196699999999, 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107213 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248457+0000", 2024-04-15T18:50:53.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248457+0000", 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248457+0000", 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6e-07 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248483+0000", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5462000000000001e-05 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248488+0000", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1780000000000002e-06 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248492+0000", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.262e-06 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252183+0000", 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036908319999999998 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252188+0000", 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6949999999999999e-06 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.864663+0000", 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61247558800000002 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.864840+0000", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017719799999999999 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.865008+0000", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016784300000000001 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871695+0000", 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066871580000000003 2024-04-15T18:50:53.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871906+0000", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000210194 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871971+0000", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5153000000000005e-05 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.918349+0000", 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.046377822999999999 2024-04-15T18:50:53.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.918410+0000", 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1253999999999993e-05 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.921689+0000", 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032790670000000001 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107214 9.5 9:a1b79672:::benchmark_data_smithi159_173188_object107213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.248497+0000", 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.34136358000001, 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.67594278299999999, 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107214 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248497+0000", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248497+0000", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248498+0000", 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3000000000000002e-07 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248532+0000", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4629999999999999e-05 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248540+0000", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4189999999999996e-06 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248543+0000", 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7100000000000001e-06 2024-04-15T18:50:53.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252211+0000", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003667376 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252217+0000", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6850000000000004e-06 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.868371+0000", 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61615350700000004 2024-04-15T18:50:53.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.868557+0000", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000186007 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.868749+0000", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019233799999999999 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878732+0000", 2024-04-15T18:50:53.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0099824449999999995 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878751+0000", 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9785000000000001e-05 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878799+0000", 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7036000000000002e-05 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.921781+0000", 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.042982157 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.921856+0000", 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5716999999999994e-05 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.924440+0000", 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025836409999999998 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107215 9.5 9:a3da4265:::benchmark_data_smithi159_173188_object107214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.248549+0000", 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.34131160000001, 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.67864455400000001, 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107215 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.644 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248549+0000", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248549+0000", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248550+0000", 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2800000000000003e-07 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248624+0000", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.415e-05 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248629+0000", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9060000000000001e-06 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.248632+0000", 2024-04-15T18:50:53.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6839999999999998e-06 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252264+0000", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036314720000000002 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.252268+0000", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6299999999999997e-06 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.872046+0000", 2024-04-15T18:50:53.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.61977774600000002 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.872262+0000", 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000216092 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.872420+0000", 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000157567 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878855+0000", 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064355660000000002 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878872+0000", 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6608000000000001e-05 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.878902+0000", 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0504000000000001e-05 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.924499+0000", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.045596980000000002 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.924540+0000", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0825999999999998e-05 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.927194+0000", 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026534950000000001 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.650 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107173 9.0 9:0f460aa7:::benchmark_data_smithi159_173188_object107172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.337792+0000", 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.25206850999999, 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49972326900000003, 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.651 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107173 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337792+0000", 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337792+0000", 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337794+0000", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7850000000000001e-06 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337825+0000", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1145000000000001e-05 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337840+0000", 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.449e-05 2024-04-15T18:50:53.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337851+0000", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0787e-05 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341501+0000", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036507570000000001 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341511+0000", 2024-04-15T18:50:53.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7540000000000001e-06 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.837428+0000", 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49591731900000002 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.837516+0000", 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7231999999999994e-05 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107174 9.0 9:01ea3564:::benchmark_data_smithi159_173188_object107173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.337870+0000", 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.251990643, 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50249582100000001, 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107174 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337870+0000", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337870+0000", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337871+0000", 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6899999999999998e-07 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337893+0000", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2860000000000001e-05 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337900+0000", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3960000000000003e-06 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337904+0000", 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.476e-06 2024-04-15T18:50:53.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341553+0000", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00364899 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341566+0000", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2313000000000001e-05 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.839748+0000", 2024-04-15T18:50:53.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.498182124 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.840366+0000", 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00061829300000000001 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107194 9.0 9:01f46091:::benchmark_data_smithi159_173188_object107193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.337910+0000", 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.25195089299999, 2024-04-15T18:50:53.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.53401485199999998, 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107194 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337910+0000", 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337910+0000", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337910+0000", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7799999999999997e-07 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337936+0000", 2024-04-15T18:50:53.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5432999999999999e-05 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337941+0000", 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7239999999999997e-06 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337946+0000", 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1810000000000003e-06 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341599+0000", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036535550000000002 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341604+0000", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.532e-06 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.840435+0000", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49883096799999999 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.840615+0000", 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018045600000000001 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.840844+0000", 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022850400000000001 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.850244+0000", 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094002319999999997 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.850288+0000", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4441999999999999e-05 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.868371+0000", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.018083194 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.868441+0000", 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9202999999999995e-05 2024-04-15T18:50:53.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.871925+0000", 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034841500000000001 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:107217 9.0 9:0e1ac76d:::benchmark_data_smithi159_173188_object107216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:22.337950+0000", 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 211.25191051799999, 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54245532699999999, 2024-04-15T18:50:53.667 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 107217 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337950+0000", 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337950+0000", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.337951+0000", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9499999999999998e-07 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.338000+0000", 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9607000000000001e-05 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.338007+0000", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.815e-06 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.338012+0000", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9219999999999996e-06 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341649+0000", 2024-04-15T18:50:53.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003636986 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.341659+0000", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0451999999999999e-05 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.843766+0000", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.502106634 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.843936+0000", 2024-04-15T18:50:53.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016986099999999999 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.844055+0000", 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118948 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.850321+0000", 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062665050000000003 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.850333+0000", 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1211e-05 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.875430+0000", 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.025097791000000001 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.875500+0000", 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9387999999999994e-05 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:22.880406+0000", 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049059120000000001 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:108614 9.0 9:093b5500:::benchmark_data_smithi159_173188_object108613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.246713+0000", 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 208.34314773200001, 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26801438100000002, 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.674 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 108614 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246713+0000", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246713+0000", 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246714+0000", 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1699999999999997e-07 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246781+0000", 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7254000000000004e-05 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246793+0000", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2133999999999999e-05 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.246800+0000", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3969999999999999e-06 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254714+0000", 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079140159999999994 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254872+0000", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015794100000000001 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 6,11", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.255026+0000", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015469600000000001 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.258134+0000", 2024-04-15T18:50:53.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031077710000000001 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.258178+0000", 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4027999999999999e-05 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.511016+0000", 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25283743800000003 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.511066+0000", 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0039e-05 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.514727+0000", 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036618499999999999 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:108620 9.2 9:4e35ac8e:::benchmark_data_smithi159_173188_object108619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.253570+0000", 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 208.336290663, 2024-04-15T18:50:53.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28709153199999998, 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 108620 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253570+0000", 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253570+0000", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253572+0000", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.457e-06 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253619+0000", 2024-04-15T18:50:53.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7718e-05 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253632+0000", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3108999999999999e-05 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253640+0000", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000003e-06 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253697+0000", 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6572e-05 2024-04-15T18:50:53.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.253864+0000", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016740099999999999 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254002+0000", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013755899999999999 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.257408+0000", 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034061550000000001 2024-04-15T18:50:53.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.260066+0000", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026587609999999999 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540566+0000", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28049944599999999 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540617+0000", 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1468999999999998e-05 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540662+0000", 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4356e-05 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:130356 9.5 9:aaab5d27:::benchmark_data_smithi159_173188_object21723:head [delete] snapc 0=[] ondisk+write+known_if_redirected e832)", 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:02.658543+0000", 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 170.93131749099999, 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24890409599999999, 2024-04-15T18:50:53.686 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 130356 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658543+0000", 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658543+0000", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658545+0000", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3030000000000001e-06 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658551+0000", 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.981e-06 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.688 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658563+0000", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-05 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.658570+0000", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6120000000000001e-06 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.660030+0000", 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014599700000000001 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.660134+0000", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103974 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7,11", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.660233+0000", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8690000000000005e-05 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.664645+0000", 2024-04-15T18:50:53.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044123449999999998 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.664675+0000", 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9638999999999999e-05 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.684446+0000", 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019771555 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.904859+0000", 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22041229500000001 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.904929+0000", 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0202000000000003e-05 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.907447+0000", 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00251855 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.692 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:217252 9.2 9:4e35ac8e:::benchmark_data_smithi159_173188_object108619:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:25.525511+0000", 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 28.064350043000001, 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.42069907099999998, 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 217252 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:50:53.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525511+0000", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525511+0000", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525512+0000", 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-07 2024-04-15T18:50:53.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525516+0000", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1679999999999997e-06 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525525+0000", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1949999999999992e-06 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525531+0000", 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6450000000000002e-06 2024-04-15T18:50:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525576+0000", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5392999999999997e-05 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525700+0000", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012434300000000001 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525838+0000", 2024-04-15T18:50:53.696 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000137966 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.530763+0000", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049251440000000002 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.646859+0000", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11609604499999999 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.946086+0000", 2024-04-15T18:50:53.697 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29922696700000001 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.946157+0000", 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0201000000000001e-05 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.946210+0000", 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3226000000000002e-05 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:50:53.698 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:50:53.699 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:50:55.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:54 smithi082 ceph-mon[44206]: pgmap v2303: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:55.119 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:54 smithi049 ceph-mon[40087]: pgmap v2303: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:55.120 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:55.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:54 smithi159 ceph-mon[45336]: pgmap v2303: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:55.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-15T18:50:55.432 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:50:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-04-15T18:50:56.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:56 smithi082 ceph-mon[44206]: pgmap v2304: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:56.964 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:56 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:50:57.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:56 smithi159 ceph-mon[45336]: pgmap v2304: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:57.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:56 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:50:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:56 smithi049 ceph-mon[40087]: pgmap v2304: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:57.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:56 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:50:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:50:57.306 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:50:57.306 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:50:57.306 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:50:57.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-15T18:50:57.793 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-04-15T18:50:57.864 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:57 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:50:57.864 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:57 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:50:57.864 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:50:57.864 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:57 smithi082 ceph-mon[44206]: osdmap e882: 12 total, 12 up, 12 in 2024-04-15T18:50:57.864 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:57 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:57.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-15T18:50:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:57 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:50:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:57 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:50:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:50:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:57 smithi049 ceph-mon[40087]: osdmap e882: 12 total, 12 up, 12 in 2024-04-15T18:50:57.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:57 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:57 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:50:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:57 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:50:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/380595401' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:50:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:57 smithi159 ceph-mon[45336]: osdmap e882: 12 total, 12 up, 12 in 2024-04-15T18:50:58.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:57 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-15T18:50:58.152 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:50:58.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-15T18:50:58.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:58.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:58 smithi049 ceph-mon[40087]: pgmap v2306: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:58.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:58 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:58.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:50:58 smithi049 ceph-mon[40087]: osdmap e883: 12 total, 12 up, 12 in 2024-04-15T18:50:59.048 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:58 smithi082 ceph-mon[44206]: pgmap v2306: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:59.049 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:58 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:59.049 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:50:58 smithi082 ceph-mon[44206]: osdmap e883: 12 total, 12 up, 12 in 2024-04-15T18:50:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:58 smithi159 ceph-mon[45336]: pgmap v2306: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:50:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:58 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/204946151' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-15T18:50:59.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:50:58 smithi159 ceph-mon[45336]: osdmap e883: 12 total, 12 up, 12 in 2024-04-15T18:50:59.900 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:50:59.977 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:50:59.977 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:50:59.977 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:50:59.977 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:50:59.977 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:51:00.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-15T18:51:00.871 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:51:01.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:00 smithi082 ceph-mon[44206]: pgmap v2308: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:01.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:00 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:01.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:00 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:01.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:00 smithi159 ceph-mon[45336]: pgmap v2308: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:01.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:00 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:01.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:00 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:00 smithi049 ceph-mon[40087]: pgmap v2308: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:00 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:01.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:00 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-15T18:51:02.009 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:01 smithi082 ceph-mon[44206]: osdmap e884: 12 total, 12 up, 12 in 2024-04-15T18:51:02.010 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:01 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:51:02.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:01 smithi159 ceph-mon[45336]: osdmap e884: 12 total, 12 up, 12 in 2024-04-15T18:51:02.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:01 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:51:02.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:01 smithi049 ceph-mon[40087]: osdmap e884: 12 total, 12 up, 12 in 2024-04-15T18:51:02.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:01 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-15T18:51:02.550 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:51:02.550 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:51:02.550 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:51:02.550 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26791.0:164269 8.b e766/709 8:dfeda397:::benchmark_data_smithi159_162467_object61964:head v 766'10242, mlcod=766'10239)", 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 392.97296731400002, 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.39892428099999999, 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.551 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588163+0000", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588164+0000", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4000000000000001e-07 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.588172+0000", 2024-04-15T18:51:02.552 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6190000000000002e-06 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982886+0000", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.39471447500000001 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982892+0000", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0209999999999996e-06 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.982983+0000", 2024-04-15T18:51:02.553 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0558999999999998e-05 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.983005+0000", 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2504e-05 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.987059+0000", 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004054017 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.554 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.987088+0000", 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8345999999999999e-05 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.26791.0:184710 8.b e775/709 8:d50196f7:::benchmark_data_smithi159_162467_object82405:head v 775'11522, mlcod=775'11519)", 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 360.110303445, 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.314097931, 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.555 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450827+0000", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450828+0000", 2024-04-15T18:51:02.556 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4099999999999998e-07 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.450833+0000", 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6619999999999996e-06 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760596+0000", 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.309763546 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.557 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760604+0000", 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5959999999999998e-06 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760689+0000", 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4753000000000007e-05 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.760704+0000", 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5449000000000001e-05 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.558 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.764881+0000", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041766659999999999 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:45:02.764925+0000", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4617999999999999e-05 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:17995 9.c 9:36be814e:::benchmark_data_smithi159_173188_object17994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e797)", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:51:02.559 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 296.71306232299997, 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21212373800000001, 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 17995 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:51:02.560 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848068+0000", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848070+0000", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2500000000000001e-06 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848189+0000", 2024-04-15T18:51:02.561 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000119659 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848205+0000", 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6042999999999999e-05 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848214+0000", 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8279999999999992e-06 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.562 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848281+0000", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7001000000000004e-05 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848399+0000", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000118068 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.848492+0000", 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2875999999999996e-05 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.563 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.849504+0000", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010113139999999999 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:05.849694+0000", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019046499999999999 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060121+0000", 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.210427165 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.564 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060153+0000", 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2155e-05 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:06.060192+0000", 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8914e-05 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:92936 9.c 9:395c82b1:::benchmark_data_smithi159_173188_object92935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:51:02.565 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:05.753773+0000", 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 236.80735750299999, 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17519844500000001, 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 92936 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.566 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753773+0000", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753773+0000", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753774+0000", 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7899999999999997e-07 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.567 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753829+0000", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5201000000000002e-05 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753840+0000", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0638999999999999e-05 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753846+0000", 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8409999999999996e-06 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.568 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753878+0000", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2317000000000002e-05 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.753980+0000", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010203 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.754084+0000", 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103931 2024-04-15T18:51:02.569 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.756260+0000", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021758419999999999 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.756632+0000", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00037207800000000002 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.928893+0000", 2024-04-15T18:51:02.570 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17226112599999999 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.928931+0000", 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7601000000000001e-05 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.928972+0000", 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0960000000000001e-05 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.571 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "rep_scrubmap(9.c e791 from shard 3)", 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:05.764245+0000", 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 236.79688541600001, 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203232142, 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.764245+0000", 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.572 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.764245+0000", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.764246+0000", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1800000000000005e-07 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.764286+0000", 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0054e-05 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.573 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.967229+0000", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20294231800000001 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.967235+0000", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0399999999999998e-06 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.967277+0000", 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1974000000000002e-05 2024-04-15T18:51:02.574 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.967289+0000", 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.254e-05 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:05.967478+0000", 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018839799999999999 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107189 9.c 9:3e2c1a6c:::benchmark_data_smithi159_173188_object107188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:51:02.575 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.225473+0000", 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 222.33565798800001, 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.54017878600000002, 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107189 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225473+0000", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225473+0000", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225474+0000", 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6610000000000001e-06 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.577 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225516+0000", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1406000000000003e-05 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225534+0000", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7971000000000001e-05 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225543+0000", 2024-04-15T18:51:02.578 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3549999999999994e-06 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225600+0000", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6932999999999999e-05 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225701+0000", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101055 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225797+0000", 2024-04-15T18:51:02.579 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6002999999999995e-05 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228685+0000", 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028873750000000002 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228911+0000", 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000226641 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.580 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765562+0000", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53665059599999998 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765603+0000", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0951999999999997e-05 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765652+0000", 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8838000000000002e-05 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.581 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107189 9.c e820/791 ondisk, result = 0)", 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.229284+0000", 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 222.33184665799999, 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53637416400000004, 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.229284+0000", 2024-04-15T18:51:02.582 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.229284+0000", 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.229285+0000", 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4099999999999999e-07 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.583 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.229287+0000", 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6510000000000001e-06 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765477+0000", 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.53618969800000005 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765484+0000", 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5750000000000003e-06 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.584 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765551+0000", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6733e-05 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765558+0000", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3720000000000001e-06 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.765658+0000", 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100694 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.585 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:108627 9.c 9:325038b8:::benchmark_data_smithi159_173188_object108626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:25.263871+0000", 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 217.297260013, 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.211909088, 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 108627 2024-04-15T18:51:02.586 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263871+0000", 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263871+0000", 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.587 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263872+0000", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0139999999999999e-06 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263950+0000", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8384e-05 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263960+0000", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5859999999999994e-06 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.588 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.263965+0000", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5600000000000001e-06 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.264009+0000", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3834000000000001e-05 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.264096+0000", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6682999999999997e-05 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.264184+0000", 2024-04-15T18:51:02.589 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7918999999999996e-05 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.266316+0000", 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021323169999999999 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.267185+0000", 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00086917299999999999 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.590 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.475709+0000", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.208523928 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.475750+0000", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0470999999999999e-05 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:25.475780+0000", 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0219e-05 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.591 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:123464 9.c 9:393c6a59:::benchmark_data_smithi159_173188_object14831:head [delete] snapc 0=[] ondisk+write+known_if_redirected e829)", 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:51.265952+0000", 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 191.29517847700001, 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.243852703, 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 123464 2024-04-15T18:51:02.592 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265952+0000", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265952+0000", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265954+0000", 2024-04-15T18:51:02.593 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5719999999999999e-06 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265961+0000", 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4490000000000002e-06 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265977+0000", 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5591e-05 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.594 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.265987+0000", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7189999999999996e-06 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.266049+0000", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2602999999999997e-05 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.266184+0000", 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013458 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.595 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.266303+0000", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011901 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.269928+0000", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0036252020000000001 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.270729+0000", 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00080050900000000003 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.596 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.509745+0000", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.239016587 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.509776+0000", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.074e-05 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:51.509805+0000", 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9141000000000001e-05 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.597 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:130911 9.4 e833/791 9:23d95a66:::benchmark_data_smithi159_173188_object22278:head v 833'8276, mlcod=833'8273)", 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:03.690632+0000", 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 178.870499107, 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.164091557, 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.598 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.690632+0000", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.690632+0000", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.690633+0000", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1289999999999999e-06 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.599 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.690641+0000", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1610000000000003e-06 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.851013+0000", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.160372025 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.851023+0000", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8230000000000006e-06 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.600 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.851086+0000", 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3454000000000006e-05 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.851110+0000", 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3986e-05 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.854689+0000", 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035784329999999998 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.601 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:03.854723+0000", 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4545999999999998e-05 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:146534 9.c 9:3b767719:::benchmark_data_smithi159_173188_object37901:head [delete] snapc 0=[] ondisk+write+known_if_redirected e836)", 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:29.556408+0000", 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 153.00472253199999, 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17065797099999999, 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.602 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 146534 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556408+0000", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556408+0000", 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.603 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556410+0000", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5409999999999999e-06 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556416+0000", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5359999999999998e-06 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556431+0000", 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4863e-05 2024-04-15T18:51:02.604 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556441+0000", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.414e-06 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556502+0000", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1359999999999995e-05 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556598+0000", 2024-04-15T18:51:02.605 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5921000000000005e-05 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.556670+0000", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2150000000000005e-05 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.560264+0000", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035941340000000001 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.561105+0000", 2024-04-15T18:51:02.606 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084048599999999999 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.726987+0000", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16588256500000001 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.727023+0000", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5960999999999997e-05 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.727066+0000", 2024-04-15T18:51:02.607 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3040000000000001e-05 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:146534 9.c e836/791 ondisk, result = 0)", 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:29.560674+0000", 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 153.00045665100001, 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16639696900000001, 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.608 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.560674+0000", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.560674+0000", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.560676+0000", 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.703e-06 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.609 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.560686+0000", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0293e-05 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.726935+0000", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16624934099999999 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.726940+0000", 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7230000000000004e-06 2024-04-15T18:51:02.610 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.726978+0000", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8127000000000003e-05 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.726983+0000", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1749999999999996e-06 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.727071+0000", 2024-04-15T18:51:02.611 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7607000000000003e-05 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:146558 9.c 9:3533922d:::benchmark_data_smithi159_173188_object37925:head [delete] snapc 0=[] ondisk+write+known_if_redirected e836)", 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:29.635307+0000", 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.92582395900001, 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17310344899999999, 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.612 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 146558 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635307+0000", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635307+0000", 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.613 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635309+0000", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.885e-06 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635316+0000", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7449999999999995e-06 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635336+0000", 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9449e-05 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.614 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635347+0000", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0958000000000001e-05 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635401+0000", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4268999999999998e-05 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635495+0000", 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3579999999999995e-05 2024-04-15T18:51:02.615 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.635561+0000", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6709000000000001e-05 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.639227+0000", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003665583 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.639633+0000", 2024-04-15T18:51:02.616 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00040568400000000001 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.808339+0000", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16870590999999999 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.808375+0000", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6145e-05 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.808410+0000", 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5531999999999998e-05 2024-04-15T18:51:02.617 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:147946 9.c 9:3a288080:::benchmark_data_smithi159_173188_object39313:head [delete] snapc 0=[] ondisk+write+known_if_redirected e837)", 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:32.034116+0000", 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 150.527014974, 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18218298099999999, 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.618 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 147946 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034116+0000", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034116+0000", 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.619 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034118+0000", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.73e-06 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034126+0000", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1510000000000004e-06 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034142+0000", 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6021000000000001e-05 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.620 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034151+0000", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.3340000000000008e-06 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034214+0000", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3433000000000001e-05 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034346+0000", 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000131023 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.621 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.034472+0000", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000126482 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.038988+0000", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045157399999999999 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.043561+0000", 2024-04-15T18:51:02.622 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045728189999999997 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.216228+0000", 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.172667495 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.216265+0000", 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6721000000000003e-05 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.623 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:32.216299+0000", 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4032000000000002e-05 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:171610 9.c 9:31edfcd7:::benchmark_data_smithi159_173188_object62977:head [delete] snapc 0=[] ondisk+write+known_if_redirected e844)", 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:11.532483+0000", 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 111.028648042, 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16449165199999999, 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.624 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 171610 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532483+0000", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532483+0000", 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.625 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532484+0000", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.141e-06 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532488+0000", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5090000000000004e-06 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532498+0000", 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8749999999999995e-06 2024-04-15T18:51:02.626 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532504+0000", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8860000000000001e-06 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532543+0000", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9054999999999998e-05 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532660+0000", 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117088 2024-04-15T18:51:02.627 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.532740+0000", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9578000000000003e-05 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.536751+0000", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040112960000000001 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.538016+0000", 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0012647380000000001 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.696873+0000", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15885750800000001 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.696925+0000", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1325000000000001e-05 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:11.696974+0000", 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9653e-05 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.629 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:190108 9.c 9:3cb23ace:::benchmark_data_smithi159_173188_object81475:head [delete] snapc 0=[] ondisk+write+known_if_redirected e848)", 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:41.699472+0000", 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 80.861658993999995, 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16346147899999999, 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.630 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 190108 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699472+0000", 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699472+0000", 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.631 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699473+0000", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.282e-06 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699478+0000", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0939999999999998e-06 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699490+0000", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1838000000000001e-05 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.632 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699499+0000", 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.132e-06 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699557+0000", 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7705999999999999e-05 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699697+0000", 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013969500000000001 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.633 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.699827+0000", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000130643 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.704198+0000", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043713049999999998 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.704782+0000", 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00058366400000000001 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.634 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.862855+0000", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15807275300000001 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.862891+0000", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5725000000000001e-05 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:41.862933+0000", 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2642000000000003e-05 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.635 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:217259 9.c 9:325038b8:::benchmark_data_smithi159_173188_object108626:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:25.531896+0000", 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.029235127, 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23176110799999999, 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 217259 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.636 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531896+0000", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531896+0000", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531896+0000", 2024-04-15T18:51:02.637 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6899999999999996e-07 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531900+0000", 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8319999999999999e-06 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531909+0000", 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6969999999999999e-06 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.531916+0000", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5529999999999998e-06 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.536728+0000", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0048121040000000002 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.536813+0000", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5204000000000006e-05 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:51:02.639 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.536885+0000", 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1746000000000001e-05 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.541125+0000", 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004240794 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.547396+0000", 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062710300000000004 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.640 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763577+0000", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21618071 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763620+0000", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3250000000000001e-05 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763657+0000", 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6418999999999998e-05 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.641 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:217259 9.c e867/791 ondisk, result = 0)", 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:25.540716+0000", 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 37.020415165999999, 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22294640900000001, 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.540716+0000", 2024-04-15T18:51:02.642 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.540716+0000", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.540717+0000", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.595e-06 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.540724+0000", 2024-04-15T18:51:02.643 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0110000000000003e-06 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763487+0000", 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22276252299999999 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763499+0000", 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2614e-05 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:02.644 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763563+0000", 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3688000000000004e-05 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763572+0000", 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.67e-06 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.763662+0000", 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0308000000000003e-05 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.645 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:02.646 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:02.646 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:51:02.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 1 1 2024-04-15T18:51:02.813 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-04-15T18:51:02.876 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:02 smithi082 ceph-mon[44206]: pgmap v2310: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:02.876 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:51:02.876 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:02 smithi082 ceph-mon[44206]: osdmap e885: 12 total, 12 up, 12 in 2024-04-15T18:51:02.876 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:02 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:51:02.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:02 smithi049 ceph-mon[40087]: pgmap v2310: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:02.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:51:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:02 smithi049 ceph-mon[40087]: osdmap e885: 12 total, 12 up, 12 in 2024-04-15T18:51:02.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:02 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:51:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:02 smithi159 ceph-mon[45336]: pgmap v2310: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1074891864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-15T18:51:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:02 smithi159 ceph-mon[45336]: osdmap e885: 12 total, 12 up, 12 in 2024-04-15T18:51:03.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:02 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-15T18:51:03.171 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-15T18:51:03.437 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:51:03.983 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:04.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:51:04.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:03 smithi082 ceph-mon[44206]: osdmap e886: 12 total, 12 up, 12 in 2024-04-15T18:51:04.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:51:04.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:03 smithi159 ceph-mon[45336]: osdmap e886: 12 total, 12 up, 12 in 2024-04-15T18:51:04.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3441281179' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-15T18:51:04.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:03 smithi049 ceph-mon[40087]: osdmap e886: 12 total, 12 up, 12 in 2024-04-15T18:51:05.038 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:51:05.038 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:51:05.038 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:51:05.038 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:51:05.067 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:04 smithi082 ceph-mon[44206]: pgmap v2313: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:05.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:04 smithi159 ceph-mon[45336]: pgmap v2313: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:05.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:04 smithi049 ceph-mon[40087]: pgmap v2313: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:05.726 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-15T18:51:05.994 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:51:06.823 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-04-15T18:51:07.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:06 smithi082 ceph-mon[44206]: pgmap v2314: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:07.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:06 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:07.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:06 smithi159 ceph-mon[45336]: pgmap v2314: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:07.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:06 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:07.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:06 smithi049 ceph-mon[40087]: pgmap v2314: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:07.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:06 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:07.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-15T18:51:07.550 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:51:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:51:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:51:07.550 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:51:07.551 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:51:07.788 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:08.083 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:08.083 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:07 smithi082 ceph-mon[44206]: osdmap e887: 12 total, 12 up, 12 in 2024-04-15T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:08.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:07 smithi049 ceph-mon[40087]: osdmap e887: 12 total, 12 up, 12 in 2024-04-15T18:51:08.113 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-15T18:51:08.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4048808570' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:08.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:07 smithi159 ceph-mon[45336]: osdmap e887: 12 total, 12 up, 12 in 2024-04-15T18:51:08.378 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:51:08.892 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:08 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:08.893 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:08 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:08.893 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:08 smithi082 ceph-mon[44206]: pgmap v2316: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:08.893 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:08 smithi082 ceph-mon[44206]: osdmap e888: 12 total, 12 up, 12 in 2024-04-15T18:51:08.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:08 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:08.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:08 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:08.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:08 smithi049 ceph-mon[40087]: pgmap v2316: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:08.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:08 smithi049 ceph-mon[40087]: osdmap e888: 12 total, 12 up, 12 in 2024-04-15T18:51:09.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:08 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:09.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:08 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:09.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:08 smithi159 ceph-mon[45336]: pgmap v2316: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:09.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:08 smithi159 ceph-mon[45336]: osdmap e888: 12 total, 12 up, 12 in 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:138088 8.d 8:b4d05b61:::benchmark_data_smithi159_162467_object35783:head [delete] snapc 0=[] ondisk+write+known_if_redirected e757)", 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 443.71587159900002, 2024-04-15T18:51:10.085 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20772660300000001, 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 138088 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.086 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379053+0000", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379056+0000", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6919999999999998e-06 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379064+0000", 2024-04-15T18:51:10.087 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0190000000000006e-06 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379082+0000", 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8031000000000001e-05 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.379093+0000", 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0688e-05 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.088 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384072+0000", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049789359999999998 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384222+0000", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015064500000000001 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 2,5", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.384334+0000", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111799 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.089 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.390640+0000", 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006306056 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.398802+0000", 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0081622269999999993 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585518+0000", 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.18671573499999999 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.090 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.585599+0000", 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0866999999999997e-05 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:46.586780+0000", 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001180908 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.091 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:25901 9.d 9:bb4f7edf:::benchmark_data_smithi159_173188_object25900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 299.00339693000001, 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25750107, 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 25901 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.092 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091528+0000", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091529+0000", 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0559999999999999e-06 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.093 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091583+0000", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3999000000000003e-05 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091593+0000", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0366e-05 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091599+0000", 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7119999999999997e-06 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.094 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091645+0000", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6227000000000003e-05 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091737+0000", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1941999999999995e-05 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.091825+0000", 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7906999999999997e-05 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.095 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.092734+0000", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00090859999999999997 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.095365+0000", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002631235 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.348923+0000", 2024-04-15T18:51:10.096 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25355768299999998 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.348983+0000", 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0720000000000001e-05 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.349029+0000", 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5623e-05 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.097 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:25903 9.d 9:be043def:::benchmark_data_smithi159_173188_object25902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 299.00129765499997, 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26125103199999999, 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 25903 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.098 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093627+0000", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093628+0000", 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2099999999999996e-07 2024-04-15T18:51:10.099 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093682+0000", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.427e-05 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093692+0000", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5959999999999993e-06 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093697+0000", 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9439999999999996e-06 2024-04-15T18:51:10.100 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093735+0000", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7926000000000002e-05 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093829+0000", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4678000000000004e-05 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.093929+0000", 2024-04-15T18:51:10.101 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000100207 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.094599+0000", 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066987899999999996 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.344374+0000", 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24977512199999999 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.102 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354775+0000", 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010400510999999999 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354834+0000", 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9039000000000002e-05 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:11.354878+0000", 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4138999999999999e-05 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.103 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:47135 9.8 e802/791 9:1510fdf2:::benchmark_data_smithi159_173188_object47134:head v 802'2916, mlcod=802'2913)", 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 284.198710522, 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.239493337, 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896214+0000", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896215+0000", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6799999999999999e-07 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:25.896371+0000", 2024-04-15T18:51:10.105 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000155895 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.133944+0000", 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23757313799999999 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.133951+0000", 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0979999999999996e-06 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.106 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.134004+0000", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2614000000000001e-05 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.134024+0000", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0179000000000001e-05 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.135662+0000", 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016382370000000001 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.107 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:26.135708+0000", 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5408000000000003e-05 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:49727 9.e 9:7fa008f6:::benchmark_data_smithi159_173188_object49726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 282.178644233, 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22725898, 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.108 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 49727 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916281+0000", 2024-04-15T18:51:10.109 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916282+0000", 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1659999999999999e-06 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916343+0000", 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1583999999999999e-05 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.110 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916354+0000", 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0944999999999999e-05 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916362+0000", 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8809999999999996e-06 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916421+0000", 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8409000000000003e-05 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.111 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916569+0000", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014799100000000001 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.916718+0000", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014955599999999999 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918468+0000", 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017494419999999999 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.112 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918763+0000", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00029521300000000001 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143447+0000", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22468469399999999 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143509+0000", 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2036e-05 2024-04-15T18:51:10.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143540+0000", 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0063e-05 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:49733 9.e 9:79a4aebf:::benchmark_data_smithi159_173188_object49732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 282.17596183900002, 2024-04-15T18:51:10.114 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23282433, 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 49733 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.115 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.918963+0000", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1900000000000003e-07 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919017+0000", 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3041000000000003e-05 2024-04-15T18:51:10.116 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919022+0000", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0490000000000003e-06 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919025+0000", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6849999999999999e-06 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919060+0000", 2024-04-15T18:51:10.117 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5142999999999999e-05 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919193+0000", 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013278899999999999 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.919330+0000", 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013709900000000001 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.118 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920633+0000", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013026629999999999 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.920748+0000", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115322 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151675+0000", 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23092637099999999 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.119 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151739+0000", 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4160999999999998e-05 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.151787+0000", 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8488000000000001e-05 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:49727 9.e e805/791 ondisk, result = 0)", 2024-04-15T18:51:10.120 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 282.17380711499999, 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22242645699999999, 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:51:10.121 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921118+0000", 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4400000000000001e-07 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:27.921120+0000", 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.796e-06 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.122 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143401+0000", 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.222281222 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143405+0000", 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9160000000000003e-06 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143439+0000", 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4301000000000002e-05 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.123 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143444+0000", 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7380000000000002e-06 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:28.143544+0000", 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010024 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:88861 9.e 9:7bb67731:::benchmark_data_smithi159_173188_object88860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e814)", 2024-04-15T18:51:10.124 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:01.483605+0000", 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 248.611320087, 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.205956102, 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 88861 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.125 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483605+0000", 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483605+0000", 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483606+0000", 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.037e-06 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.126 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483666+0000", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0217999999999998e-05 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483677+0000", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1039000000000001e-05 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483684+0000", 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.956e-06 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.127 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483735+0000", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1227999999999998e-05 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.483868+0000", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013298300000000001 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.484004+0000", 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013569799999999999 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.128 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.486384+0000", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002379833 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.487389+0000", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00100552 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.689448+0000", 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20205915499999999 2024-04-15T18:51:10.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.689515+0000", 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6180999999999995e-05 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:01.689561+0000", 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6254000000000001e-05 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.130 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:93292 9.e 9:7b0514e4:::benchmark_data_smithi159_173188_object93291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:06.270355+0000", 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 243.82457022099999, 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24007503099999999, 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93292 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.131 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270355+0000", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270355+0000", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270355+0000", 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0599999999999999e-07 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.132 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270433+0000", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7292999999999993e-05 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270444+0000", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1715999999999999e-05 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270459+0000", 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4239e-05 2024-04-15T18:51:10.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270522+0000", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3273000000000002e-05 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270643+0000", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012139200000000001 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.270771+0000", 2024-04-15T18:51:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012716800000000001 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.273735+0000", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029641960000000001 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.274873+0000", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001138383 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.510325+0000", 2024-04-15T18:51:10.135 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23545144700000001 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.510385+0000", 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0321000000000001e-05 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.510430+0000", 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4697000000000003e-05 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:93302 9.e 9:75ce773d:::benchmark_data_smithi159_173188_object93301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:06.285367+0000", 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 243.80955766100001, 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23294820999999999, 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93302 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.137 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285367+0000", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285367+0000", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285368+0000", 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5000000000000001e-07 2024-04-15T18:51:10.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285477+0000", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000108551 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285490+0000", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3634e-05 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285499+0000", 2024-04-15T18:51:10.139 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2619999999999998e-06 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285552+0000", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3597000000000002e-05 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285706+0000", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000154096 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.285868+0000", 2024-04-15T18:51:10.140 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000161638 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.288538+0000", 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0026698780000000001 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.288692+0000", 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015467899999999999 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.141 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.518214+0000", 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.229522005 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.518273+0000", 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8289999999999999e-05 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.518315+0000", 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2629999999999997e-05 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.142 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107181 9.e 9:7769b37f:::benchmark_data_smithi159_173188_object107180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.194529+0000", 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 229.90039578899999, 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.37433277100000001, 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107181 2024-04-15T18:51:10.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194529+0000", 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194529+0000", 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.144 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194530+0000", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0449999999999999e-06 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194586+0000", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5476000000000002e-05 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194594+0000", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8270000000000004e-06 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194599+0000", 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1560000000000003e-06 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194631+0000", 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1656000000000003e-05 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194730+0000", 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.8479000000000004e-05 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.194827+0000", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7581e-05 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.199131+0000", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004303861 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.226851+0000", 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.027719957 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568686+0000", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34183472100000001 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568788+0000", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102135 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568862+0000", 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3876999999999998e-05 2024-04-15T18:51:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107181 9.e e820/791 ondisk, result = 0)", 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.222119+0000", 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 229.87280608699999, 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34674978400000001, 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222119+0000", 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222119+0000", 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222119+0000", 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5600000000000005e-07 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.150 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222123+0000", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0269999999999997e-06 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568510+0000", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.346386521 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568524+0000", 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4286e-05 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.151 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568659+0000", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000134745 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568673+0000", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4433000000000001e-05 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.568868+0000", 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019511600000000001 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.152 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107191 9.d 9:b2732e22:::benchmark_data_smithi159_173188_object107190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.228316+0000", 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 229.86660831500001, 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34500964899999997, 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.153 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107191 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228316+0000", 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228316+0000", 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.154 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228318+0000", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.269e-06 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228347+0000", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9427000000000001e-05 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228359+0000", 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1728e-05 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.155 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228365+0000", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8440000000000003e-06 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228422+0000", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6863000000000001e-05 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228566+0000", 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000144857 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.156 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228668+0000", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000101502 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.231536+0000", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028675829999999999 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.415378+0000", 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.183842591 2024-04-15T18:51:10.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573214+0000", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15783636000000001 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573278+0000", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3230000000000003e-05 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573326+0000", 2024-04-15T18:51:10.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8395e-05 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107191 9.d e820/791 ondisk, result = 0)", 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.231831+0000", 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 229.86309374699999, 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34150278499999998, 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.231831+0000", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.231831+0000", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.231832+0000", 2024-04-15T18:51:10.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.16e-07 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.231837+0000", 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2410000000000003e-06 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573151+0000", 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.34131393599999998 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573156+0000", 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7149999999999997e-06 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573200+0000", 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3676e-05 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573208+0000", 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7630000000000006e-06 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.573334+0000", 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012553799999999999 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:167888 9.e 9:778641c3:::benchmark_data_smithi159_173188_object59255:head [delete] snapc 0=[] ondisk+write+known_if_redirected e842)", 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:05.045491+0000", 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 125.04943370399999, 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21340773199999999, 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.163 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 167888 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045491+0000", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045491+0000", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045492+0000", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.879999999999999e-07 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045496+0000", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3220000000000003e-06 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045507+0000", 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1058e-05 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045513+0000", 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6899999999999997e-06 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045557+0000", 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4248000000000001e-05 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045648+0000", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0346000000000005e-05 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.045723+0000", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4808000000000006e-05 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.049750+0000", 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0040273260000000003 2024-04-15T18:51:10.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.050884+0000", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0011346310000000001 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258797+0000", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.207912349 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258854+0000", 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7420999999999998e-05 2024-04-15T18:51:10.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258899+0000", 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4545000000000001e-05 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:167888 9.e e842/791 ondisk, result = 0)", 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:05.050595+0000", 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 125.0443294, 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.208309784, 2024-04-15T18:51:10.169 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.050595+0000", 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.050595+0000", 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.050596+0000", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.85e-07 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.050600+0000", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7009999999999998e-06 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258725+0000", 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20812517 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258732+0000", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.199e-06 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258784+0000", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1644999999999998e-05 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258791+0000", 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4340000000000003e-06 2024-04-15T18:51:10.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:05.258905+0000", 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011385 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:185486 9.a e847/791 9:551b3fde:::benchmark_data_smithi159_173188_object76853:head v 847'11613, mlcod=847'11609)", 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.385233+0000", 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.709691866, 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26043307999999998, 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.173 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385233+0000", 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385233+0000", 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385234+0000", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7400000000000002e-07 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385250+0000", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6694999999999999e-05 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.641493+0000", 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25624229799999998 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.641500+0000", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.075e-06 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.641553+0000", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3025000000000001e-05 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.641578+0000", 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5327999999999999e-05 2024-04-15T18:51:10.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.645608+0000", 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004029866 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.645666+0000", 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7918999999999999e-05 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:185492 9.a e847/791 9:536494b0:::benchmark_data_smithi159_173188_object76859:head v 847'11614, mlcod=847'11609)", 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.391875+0000", 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 95.703049542000002, 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26378669300000002, 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391875+0000", 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391875+0000", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391876+0000", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9500000000000001e-07 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391885+0000", 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6300000000000004e-06 2024-04-15T18:51:10.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.649957+0000", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25807205100000002 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.649964+0000", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5039999999999996e-06 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.650040+0000", 2024-04-15T18:51:10.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5686999999999995e-05 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.650058+0000", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8391000000000001e-05 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.655616+0000", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0055581629999999996 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.655662+0000", 2024-04-15T18:51:10.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5472000000000001e-05 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:200809 9.7 e854/791 9:eeabe9e9:::benchmark_data_smithi159_173188_object92176:head v 854'12487, mlcod=854'12485)", 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:58.800018+0000", 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 71.294906466, 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.75524249600000004, 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.182 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:58.800018+0000", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:58.800018+0000", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:58.800019+0000", 2024-04-15T18:51:10.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.062e-06 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:58.800028+0000", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8799999999999997e-06 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.550837+0000", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.75080901099999997 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.550854+0000", 2024-04-15T18:51:10.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6756e-05 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.550945+0000", 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0984999999999997e-05 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.550966+0000", 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1101999999999999e-05 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.555225+0000", 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004258848 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:59.555261+0000", 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5852000000000002e-05 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:217244 9.d 9:b7f77d1e:::benchmark_data_smithi159_173188_object108611:head [delete] snapc 0=[] ondisk+write+known_if_redirected e867)", 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:25.514623+0000", 2024-04-15T18:51:10.186 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 44.580301736999999, 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.70820681200000002, 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 217244 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514623+0000", 2024-04-15T18:51:10.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514623+0000", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514624+0000", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.353e-06 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514630+0000", 2024-04-15T18:51:10.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.801e-06 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514646+0000", 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5617e-05 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514654+0000", 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7600000000000002e-06 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:51:10.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514715+0000", 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1445999999999994e-05 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514837+0000", 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000121952 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,5", 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.514948+0000", 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000111097 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.521142+0000", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0061936609999999996 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.522967+0000", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001825578 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.222753+0000", 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.69978604200000005 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.222796+0000", 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2185999999999999e-05 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.222830+0000", 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4319000000000001e-05 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:51:10.192 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:51:10.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-04-15T18:51:10.848 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-15T18:51:10.971 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:11.025 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:10 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:10 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:10 smithi082 ceph-mon[44206]: pgmap v2318: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:10 smithi082 ceph-mon[44206]: osdmap e889: 12 total, 12 up, 12 in 2024-04-15T18:51:11.026 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:10 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:10 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:10 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:10 smithi159 ceph-mon[45336]: pgmap v2318: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:10 smithi159 ceph-mon[45336]: osdmap e889: 12 total, 12 up, 12 in 2024-04-15T18:51:11.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:10 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:10 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:10 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:10 smithi049 ceph-mon[40087]: pgmap v2318: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:10 smithi049 ceph-mon[40087]: osdmap e889: 12 total, 12 up, 12 in 2024-04-15T18:51:11.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:10 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-15T18:51:11.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 4 1 2024-04-15T18:51:11.802 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:12.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:12.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:11 smithi159 ceph-mon[45336]: osdmap e890: 12 total, 12 up, 12 in 2024-04-15T18:51:12.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:12.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:11 smithi049 ceph-mon[40087]: osdmap e890: 12 total, 12 up, 12 in 2024-04-15T18:51:12.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2611903864' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-15T18:51:12.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:11 smithi082 ceph-mon[44206]: osdmap e890: 12 total, 12 up, 12 in 2024-04-15T18:51:12.629 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:12.629 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:12.630 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:51:12.630 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:12.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:12 smithi049 ceph-mon[40087]: pgmap v2321: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:12.978 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:12 smithi159 ceph-mon[45336]: pgmap v2321: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:13.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:12 smithi082 ceph-mon[44206]: pgmap v2321: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:13.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-04-15T18:51:13.550 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:13.862 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-04-15T18:51:14.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:14.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:14.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:14.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:14.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:14.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:14.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:14.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:14.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:14.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-15T18:51:14.929 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:14.946 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:14 smithi159 ceph-mon[45336]: pgmap v2322: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:14.946 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:14.946 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:14 smithi159 ceph-mon[45336]: osdmap e891: 12 total, 12 up, 12 in 2024-04-15T18:51:14.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:14 smithi049 ceph-mon[40087]: pgmap v2322: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:14.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:14.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:14 smithi049 ceph-mon[40087]: osdmap e891: 12 total, 12 up, 12 in 2024-04-15T18:51:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:15.206 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:15.206 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:51:15.206 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:51:15.206 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:15.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:14 smithi082 ceph-mon[44206]: pgmap v2322: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:15.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4226496350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:15.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:14 smithi082 ceph-mon[44206]: osdmap e891: 12 total, 12 up, 12 in 2024-04-15T18:51:15.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-04-15T18:51:16.165 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:17.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:16 smithi159 ceph-mon[45336]: pgmap v2324: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:17.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:16 smithi049 ceph-mon[40087]: pgmap v2324: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:17.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:16 smithi082 ceph-mon[44206]: pgmap v2324: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:66867 8.a 8:57e4a7a7:::benchmark_data_smithi159_162467_object66866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:07.473100+0000", 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 550.34575402500002, 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.236742288, 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.809 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 66867 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473100+0000", 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473100+0000", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473102+0000", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473124+0000", 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1382999999999998e-05 2024-04-15T18:51:17.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473139+0000", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5812e-05 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473148+0000", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5299999999999996e-06 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473212+0000", 2024-04-15T18:51:17.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3924000000000007e-05 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473353+0000", 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014084199999999999 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,9", 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.473510+0000", 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015773299999999999 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.475825+0000", 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00231438 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496708+0000", 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882759000000001 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.709779+0000", 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21307192799999999 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.709816+0000", 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6146999999999998e-05 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.709843+0000", 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7109000000000001e-05 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:66887 8.a 8:5eddc5e8:::benchmark_data_smithi159_162467_object66886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:51:17.815 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:07.493713+0000", 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 550.32514185499997, 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.22076227700000001, 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 66887 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493713+0000", 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493713+0000", 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493714+0000", 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.376e-06 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493736+0000", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1906e-05 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493751+0000", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4761e-05 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493759+0000", 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0269999999999995e-06 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493816+0000", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7064000000000001e-05 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.493921+0000", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000105667 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,9", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.494013+0000", 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1934000000000004e-05 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.496288+0000", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022750510000000002 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.497246+0000", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000958036 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.714368+0000", 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.217121913 2024-04-15T18:51:17.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.714428+0000", 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9919e-05 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:07.714475+0000", 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6622999999999997e-05 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:91008 8.a 8:5652b0a2:::benchmark_data_smithi159_162467_object91007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e740)", 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:31.587560+0000", 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 526.23129429300002, 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21803751699999999, 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 91008 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587560+0000", 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587560+0000", 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587561+0000", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9600000000000008e-07 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587585+0000", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4133000000000001e-05 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587597+0000", 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1528999999999999e-05 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587605+0000", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1699999999999997e-06 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587656+0000", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1365e-05 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587798+0000", 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014148200000000001 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 2,9", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.587963+0000", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000165248 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.590459+0000", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024963020000000002 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.618611+0000", 2024-04-15T18:51:17.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.028151533999999999 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.805509+0000", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.18689820100000001 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.805545+0000", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6140000000000003e-05 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:31.805598+0000", 2024-04-15T18:51:17.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2417000000000003e-05 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:164237 8.b 8:d84c213b:::benchmark_data_smithi159_162467_object61932:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.165591+0000", 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 408.65326302099999, 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.222185994, 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.828 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164237 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165591+0000", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165591+0000", 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165593+0000", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4959999999999999e-06 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165599+0000", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1519999999999997e-06 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165612+0000", 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3275000000000001e-05 2024-04-15T18:51:17.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165621+0000", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9819999999999997e-06 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165673+0000", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1699000000000001e-05 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165763+0000", 2024-04-15T18:51:17.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9628000000000003e-05 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.165838+0000", 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5716999999999994e-05 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.169982+0000", 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041434999999999996 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.170446+0000", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046401899999999998 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.387711+0000", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21726511400000001 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.387747+0000", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5790000000000001e-05 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.387777+0000", 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0621999999999999e-05 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:164269 8.b 8:dfeda397:::benchmark_data_smithi159_162467_object61964:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.587810+0000", 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 408.231044849, 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39951788799999999, 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.834 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164269 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587810+0000", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587810+0000", 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587811+0000", 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8470000000000001e-06 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587819+0000", 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.469e-06 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587836+0000", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7453e-05 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587845+0000", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1899999999999995e-06 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587898+0000", 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3489000000000002e-05 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.587995+0000", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6830999999999995e-05 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.588076+0000", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0863000000000002e-05 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.591814+0000", 2024-04-15T18:51:17.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037380960000000002 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.593692+0000", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018778289999999999 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.987262+0000", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.39357024600000001 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.987298+0000", 2024-04-15T18:51:17.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6196999999999999e-05 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.987327+0000", 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9377999999999999e-05 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:184710 8.b 8:d50196f7:::benchmark_data_smithi159_162467_object82405:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:45:02.450452+0000", 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 375.36840274600002, 2024-04-15T18:51:17.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.31473824, 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 184710 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450452+0000", 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450452+0000", 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450453+0000", 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5899999999999995e-07 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450457+0000", 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7539999999999997e-06 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450467+0000", 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4450000000000002e-06 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450475+0000", 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7440000000000007e-06 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450524+0000", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9004e-05 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450618+0000", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4759999999999994e-05 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.450698+0000", 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9351999999999995e-05 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.454295+0000", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035977610000000001 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455117+0000", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00082190900000000001 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.765115+0000", 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30999761300000001 2024-04-15T18:51:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.765153+0000", 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8052000000000001e-05 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.765190+0000", 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6986999999999997e-05 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:7950 9.b 9:df12bf65:::benchmark_data_smithi159_173188_object7949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e797)", 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:00.031138+0000", 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 317.78771600900001, 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.214587374, 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 7950 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031138+0000", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031138+0000", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031140+0000", 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1370000000000001e-06 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031161+0000", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1902000000000001e-05 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031173+0000", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1722000000000001e-05 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031181+0000", 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7370000000000006e-06 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031231+0000", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0297999999999999e-05 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031322+0000", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0733e-05 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031426+0000", 2024-04-15T18:51:17.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000104412 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.031953+0000", 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00052641099999999996 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.055737+0000", 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023784502999999999 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.245658+0000", 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.189920955 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.245696+0000", 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8126000000000001e-05 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:00.245726+0000", 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9437999999999998e-05 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:55089 9.b 9:d7c86918:::benchmark_data_smithi159_173188_object55088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e805)", 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:31.751523+0000", 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 286.06733100500003, 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.215103929, 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 55089 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.853 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751523+0000", 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751523+0000", 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751525+0000", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.739e-06 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751550+0000", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4600000000000002e-05 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751563+0000", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.327e-05 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751571+0000", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1839999999999999e-06 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751634+0000", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2992999999999998e-05 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751767+0000", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000132741 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.751910+0000", 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014345499999999999 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.753903+0000", 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019924109999999999 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.762274+0000", 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0083708270000000008 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.966545+0000", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.20427156099999999 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.966589+0000", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4016e-05 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:31.966627+0000", 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8132e-05 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.858 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:88857 9.b 9:d4cc9cdf:::benchmark_data_smithi159_173188_object88856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e814)", 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:01.478115+0000", 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 256.34073964200002, 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24090642600000001, 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 88857 2024-04-15T18:51:17.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478115+0000", 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478115+0000", 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478116+0000", 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.218e-06 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478135+0000", 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9300000000000002e-05 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478146+0000", 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0842000000000001e-05 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478152+0000", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9499999999999998e-06 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478205+0000", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2465e-05 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478328+0000", 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012391100000000001 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.478517+0000", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000188326 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.481398+0000", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028811940000000001 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718741+0000", 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23734344600000001 2024-04-15T18:51:17.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718920+0000", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000178906 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718977+0000", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6178999999999999e-05 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.719021+0000", 2024-04-15T18:51:17.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4688999999999999e-05 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:88857 9.b e814/791 ondisk, result = 0)", 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:01.481695+0000", 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 256.33715894699998, 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23733396300000001, 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.481695+0000", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.481695+0000", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.481696+0000", 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6899999999999996e-07 2024-04-15T18:51:17.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.481701+0000", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9699999999999998e-06 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718852+0000", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23715046200000001 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718858+0000", 2024-04-15T18:51:17.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1820000000000003e-06 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718909+0000", 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0670000000000001e-05 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718917+0000", 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1089999999999998e-06 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.719029+0000", 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112701 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:88857 9.b e814/791 ondisk, result = 0)", 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:01.484270+0000", 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 256.33458444199999, 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23448244900000001, 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.484270+0000", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.484270+0000", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.484271+0000", 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6700000000000002e-07 2024-04-15T18:51:17.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.484276+0000", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718662+0000", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23438594200000001 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718670+0000", 2024-04-15T18:51:17.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1699999999999997e-06 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718727+0000", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6922000000000002e-05 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718736+0000", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4130000000000006e-06 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:01.718752+0000", 2024-04-15T18:51:17.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6806000000000001e-05 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:93299 9.b 9:d3153acc:::benchmark_data_smithi159_173188_object93298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:06.285237+0000", 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 251.53361765899999, 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23058565, 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.873 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93299 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285237+0000", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285237+0000", 2024-04-15T18:51:17.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285238+0000", 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2139999999999999e-06 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285257+0000", 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.872e-05 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285270+0000", 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3244999999999999e-05 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285279+0000", 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9910000000000008e-06 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285341+0000", 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1834999999999993e-05 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285460+0000", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118961 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.285565+0000", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000105034 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.287964+0000", 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023992789999999998 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.296042+0000", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080775910000000003 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.515718+0000", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.219676598 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.515778+0000", 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9636000000000003e-05 2024-04-15T18:51:17.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.515822+0000", 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4545999999999997e-05 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:93309 9.b 9:d9bd977a:::benchmark_data_smithi159_173188_object93308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:06.301864+0000", 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 251.51699018100001, 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21421892400000001, 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.879 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 93309 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301864+0000", 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301864+0000", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301866+0000", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4479999999999999e-06 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301886+0000", 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0032000000000001e-05 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301899+0000", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3348e-05 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301906+0000", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7279999999999998e-06 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.301981+0000", 2024-04-15T18:51:17.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5038999999999998e-05 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.302083+0000", 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010234 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.302194+0000", 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000111296 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.304592+0000", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002397611 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.306095+0000", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001503035 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.515989+0000", 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.209893627 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.516045+0000", 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5887000000000002e-05 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:06.516083+0000", 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8532999999999999e-05 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.885 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:123119 9.b 9:d27ca697:::benchmark_data_smithi159_173188_object14486:head [delete] snapc 0=[] ondisk+write+known_if_redirected e829)", 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:50.605528+0000", 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 207.21332675400001, 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24833705, 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 123119 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605528+0000", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605528+0000", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605530+0000", 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8819999999999999e-06 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605538+0000", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5280000000000003e-06 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605559+0000", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0543e-05 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605569+0000", 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0334e-05 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605640+0000", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1025000000000005e-05 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605794+0000", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000154263 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.605907+0000", 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112853 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.609834+0000", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003926985 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.620836+0000", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01100201 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.853768+0000", 2024-04-15T18:51:17.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23293159899999999 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.853821+0000", 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3635999999999999e-05 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:50.853865+0000", 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3392000000000001e-05 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:167896 9.b 9:d1d1a8fb:::benchmark_data_smithi159_173188_object59263:head [delete] snapc 0=[] ondisk+write+known_if_redirected e842)", 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:05.064190+0000", 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 132.75466420800001, 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.223174965, 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 167896 2024-04-15T18:51:17.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064190+0000", 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064190+0000", 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064192+0000", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8810000000000001e-06 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064201+0000", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1600000000000004e-06 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064221+0000", 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9497e-05 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064232+0000", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1576999999999999e-05 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064314+0000", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1397999999999996e-05 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064474+0000", 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000160049 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.064610+0000", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013606600000000001 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.069740+0000", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051299049999999997 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.070228+0000", 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048870000000000001 2024-04-15T18:51:17.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287263+0000", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21703492099999999 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287309+0000", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5263999999999999e-05 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287365+0000", 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6546999999999999e-05 2024-04-15T18:51:17.897 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:167896 9.b e842/791 ondisk, result = 0)", 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:05.069406+0000", 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 132.74944846, 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21796494599999999, 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.069406+0000", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.069406+0000", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.069407+0000", 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4300000000000001e-06 2024-04-15T18:51:17.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.069417+0000", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9879999999999999e-06 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287200+0000", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.21778210100000001 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287207+0000", 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5950000000000001e-06 2024-04-15T18:51:17.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287250+0000", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2957000000000002e-05 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287258+0000", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:05.287371+0000", 2024-04-15T18:51:17.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113384 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200765 9.b 9:db80d7c7:::benchmark_data_smithi159_173188_object92132:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:58.265719+0000", 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.553135170000004, 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.12885733899999999, 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.902 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200765 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265719+0000", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265719+0000", 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265721+0000", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8989999999999999e-06 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265729+0000", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8790000000000003e-06 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265747+0000", 2024-04-15T18:51:17.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7961999999999998e-05 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265756+0000", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7620000000000001e-06 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265818+0000", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2719999999999996e-05 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.265963+0000", 2024-04-15T18:51:17.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000144807 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.266103+0000", 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000139235 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.270878+0000", 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047754629999999998 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.271262+0000", 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00038427399999999999 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.394504+0000", 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.123242138 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.394543+0000", 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8379e-05 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.394577+0000", 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3821e-05 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200775 9.b 9:d1c3e333:::benchmark_data_smithi159_173188_object92142:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:58.400743+0000", 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.418111729000003, 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.117686891, 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200775 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400743+0000", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400743+0000", 2024-04-15T18:51:17.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400745+0000", 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0320000000000002e-06 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400753+0000", 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3340000000000002e-06 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400772+0000", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9049999999999999e-05 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400783+0000", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1012999999999999e-05 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400840+0000", 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6425000000000003e-05 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.400965+0000", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012544100000000001 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.401086+0000", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012112000000000001 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.405479+0000", 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043927009999999997 2024-04-15T18:51:17.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.405799+0000", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032021300000000002 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.518324+0000", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.11252449 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.518374+0000", 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0315000000000002e-05 2024-04-15T18:51:17.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.518430+0000", 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5757000000000002e-05 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200800 9.b 9:d75ae9cf:::benchmark_data_smithi159_173188_object92167:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:58.555501+0000", 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.263353566000006, 2024-04-15T18:51:17.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.19834163699999999, 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200800 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555501+0000", 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555501+0000", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555502+0000", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.598e-06 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555511+0000", 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1599999999999998e-06 2024-04-15T18:51:17.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555527+0000", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6860000000000001e-05 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555536+0000", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4169999999999992e-06 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555600+0000", 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3962999999999997e-05 2024-04-15T18:51:17.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555744+0000", 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014450800000000001 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,10", 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.555878+0000", 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000133597 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:17.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.560507+0000", 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046289260000000002 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.560944+0000", 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000437544 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.753758+0000", 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.192813702 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.753797+0000", 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9252999999999998e-05 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.753843+0000", 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5108999999999998e-05 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(9.3 856/855 [PushOp(9:c0d8d2c3:::benchmark_data_smithi159_173188_object103316:head, version: 818'6453, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:c0d8d2c3:::benchmark_data_smithi159_173188_object103316:head@818'6453, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:51:17.920 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.756279+0000", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.062575069000005, 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14938590199999999, 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.756279+0000", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.756279+0000", 2024-04-15T18:51:17.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.756280+0000", 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.34e-07 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.756318+0000", 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7929999999999998e-05 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.903087+0000", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.14676898199999999 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.903104+0000", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7246999999999999e-05 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.905456+0000", 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023524660000000001 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.905467+0000", 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.041e-05 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.905665+0000", 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019863300000000001 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:17.924 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:17.925 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-15T18:51:18.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:18.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:18.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-15T18:51:18.454 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-04-15T18:51:18.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 7 1 2024-04-15T18:51:18.716 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:18.954 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:18.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:18 smithi049 ceph-mon[40087]: pgmap v2325: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:18.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:18.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:18 smithi049 ceph-mon[40087]: osdmap e892: 12 total, 12 up, 12 in 2024-04-15T18:51:19.143 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:18 smithi159 ceph-mon[45336]: pgmap v2325: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:19.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:18 smithi159 ceph-mon[45336]: osdmap e892: 12 total, 12 up, 12 in 2024-04-15T18:51:19.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:18 smithi082 ceph-mon[44206]: pgmap v2325: 1 pgs: 1 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:19.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2398540279' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-15T18:51:19.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:18 smithi082 ceph-mon[44206]: osdmap e892: 12 total, 12 up, 12 in 2024-04-15T18:51:20.082 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:19 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:20.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:19 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:20.083 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:19 smithi159 ceph-mon[45336]: osdmap e893: 12 total, 12 up, 12 in 2024-04-15T18:51:20.097 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:19 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:20.097 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:19 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:20.097 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:19 smithi049 ceph-mon[40087]: osdmap e893: 12 total, 12 up, 12 in 2024-04-15T18:51:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:19 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:19 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:20.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:19 smithi082 ceph-mon[44206]: osdmap e893: 12 total, 12 up, 12 in 2024-04-15T18:51:20.316 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:20.316 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:20.316 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:51:20.316 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:20.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-04-15T18:51:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:20 smithi159 ceph-mon[45336]: pgmap v2328: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:20 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:21.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:20 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:21.166 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:21.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:20 smithi049 ceph-mon[40087]: pgmap v2328: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:21.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:20 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:21.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:20 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:20 smithi082 ceph-mon[44206]: pgmap v2328: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:20 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:21.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:20 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:21.890 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-04-15T18:51:22.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:21 smithi049 ceph-mon[40087]: osdmap e894: 12 total, 12 up, 12 in 2024-04-15T18:51:22.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:22.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:21 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:22.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:21 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:22.264 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:21 smithi159 ceph-mon[45336]: osdmap e894: 12 total, 12 up, 12 in 2024-04-15T18:51:22.264 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:22.264 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:21 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:22.265 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:21 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:22.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:21 smithi082 ceph-mon[44206]: osdmap e894: 12 total, 12 up, 12 in 2024-04-15T18:51:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:21 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:22.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:21 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:22.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-15T18:51:22.823 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:51:22.823 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:51:22.824 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:22.943 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:22 smithi159 ceph-mon[45336]: pgmap v2330: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:23.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:22 smithi159 ceph-mon[45336]: osdmap e895: 12 total, 12 up, 12 in 2024-04-15T18:51:23.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:22 smithi049 ceph-mon[40087]: pgmap v2330: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:23.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:23.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:22 smithi049 ceph-mon[40087]: osdmap e895: 12 total, 12 up, 12 in 2024-04-15T18:51:23.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:22 smithi082 ceph-mon[44206]: pgmap v2330: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:23.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3749736606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:23.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:22 smithi082 ceph-mon[44206]: osdmap e895: 12 total, 12 up, 12 in 2024-04-15T18:51:23.472 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-04-15T18:51:23.733 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:25.159 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:24 smithi159 ceph-mon[45336]: pgmap v2332: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:25.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:24 smithi049 ceph-mon[40087]: pgmap v2332: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:25.275 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:24 smithi082 ceph-mon[44206]: pgmap v2332: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:25.445 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:25.445 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:25889 9.8 9:111f0238:::benchmark_data_smithi159_173188_object25888:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:11.044591+0000", 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 314.41074225, 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28933205899999997, 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.446 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 25889 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044591+0000", 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044591+0000", 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044592+0000", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3599999999999999e-06 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044617+0000", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4831999999999999e-05 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044630+0000", 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3151e-05 2024-04-15T18:51:25.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044638+0000", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2069999999999995e-06 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044690+0000", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1700999999999998e-05 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044808+0000", 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118124 2024-04-15T18:51:25.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.044916+0000", 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107872 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.045848+0000", 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00093213499999999995 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.093964+0000", 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048116025 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.333811+0000", 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23984660499999999 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.333872+0000", 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1552999999999998e-05 2024-04-15T18:51:25.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.333923+0000", 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0494000000000001e-05 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:25890 9.1 9:87414256:::benchmark_data_smithi159_173188_object25889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:11.046447+0000", 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 314.408885779, 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30726329699999999, 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.452 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 25890 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046447+0000", 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046447+0000", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046448+0000", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046470+0000", 2024-04-15T18:51:25.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2019999999999999e-05 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046478+0000", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3270000000000001e-06 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046483+0000", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7140000000000001e-06 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046531+0000", 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8655000000000001e-05 2024-04-15T18:51:25.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046657+0000", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012590299999999999 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.046801+0000", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000143688 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.047738+0000", 2024-04-15T18:51:25.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00093650299999999999 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.048137+0000", 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039932500000000002 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.353641+0000", 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30550399700000003 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.353680+0000", 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8705999999999998e-05 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.353710+0000", 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0783e-05 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107178 9.1 9:8835a4fe:::benchmark_data_smithi159_173188_object107177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-04-15T18:51:25.458 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.188918+0000", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 245.26641524499999, 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45326154600000002, 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107178 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188918+0000", 2024-04-15T18:51:25.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188918+0000", 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188918+0000", 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9999999999999996e-07 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188935+0000", 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7062e-05 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188945+0000", 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3330000000000003e-06 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.188950+0000", 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8779999999999997e-06 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.191680+0000", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027298579999999999 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.191776+0000", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6119999999999995e-05 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.191883+0000", 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010746100000000001 2024-04-15T18:51:25.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.194367+0000", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002484315 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.194533+0000", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016536099999999999 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642100+0000", 2024-04-15T18:51:25.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44756749400000001 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642141+0000", 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0476999999999999e-05 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642179+0000", 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8386999999999997e-05 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107185 9.1 9:8e8a4364:::benchmark_data_smithi159_173188_object107184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.220256+0000", 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 245.23507677800001, 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42694738300000001, 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107185 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.465 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220256+0000", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220256+0000", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220258+0000", 2024-04-15T18:51:25.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.418e-06 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220279+0000", 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1517000000000001e-05 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220292+0000", 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3074000000000001e-05 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220300+0000", 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0979999999999994e-06 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220366+0000", 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6237999999999998e-05 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220518+0000", 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015165900000000001 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.220676+0000", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015781500000000001 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.223785+0000", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031087060000000001 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.223800+0000", 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5017999999999999e-05 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.647136+0000", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.423336828 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.647174+0000", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7938000000000002e-05 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.647203+0000", 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9074e-05 2024-04-15T18:51:25.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:107178 9.1 e820/791 ondisk, result = 0)", 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.225149+0000", 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 245.230183858, 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41703599899999999, 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.225149+0000", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.225149+0000", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.225150+0000", 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0900000000000001e-07 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.225153+0000", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8639999999999998e-06 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642013+0000", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41685919900000001 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642018+0000", 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4960000000000004e-06 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642089+0000", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0554000000000002e-05 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642096+0000", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.588e-06 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.642185+0000", 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8689000000000004e-05 2024-04-15T18:51:25.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107187 9.8 9:127db936:::benchmark_data_smithi159_173188_object107186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231423+0000", 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.22390959800001, 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.65481730699999996, 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107187 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231423+0000", 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231423+0000", 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231425+0000", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5290000000000001e-06 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231454+0000", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9437999999999998e-05 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231474+0000", 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9641e-05 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231489+0000", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5293000000000001e-05 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234168+0000", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026793149999999998 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234181+0000", 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2415999999999999e-05 2024-04-15T18:51:25.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.850970+0000", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61678896599999999 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851153+0000", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000183134 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851343+0000", 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000189987 2024-04-15T18:51:25.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.867291+0000", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015948117000000001 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878611+0000", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011320076 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.881001+0000", 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023898830000000002 2024-04-15T18:51:25.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.881061+0000", 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9499000000000001e-05 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.886241+0000", 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051800129999999998 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107190 9.8 9:18652d9d:::benchmark_data_smithi159_173188_object107189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231524+0000", 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.223808706, 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.66040817900000004, 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107190 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231524+0000", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231524+0000", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231525+0000", 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7000000000000005e-07 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231547+0000", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1863000000000001e-05 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231556+0000", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2920000000000001e-06 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231563+0000", 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6499999999999999e-06 2024-04-15T18:51:25.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234206+0000", 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026439089999999998 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234214+0000", 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1459999999999999e-06 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.859999+0000", 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62578589799999995 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860193+0000", 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019372500000000001 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860393+0000", 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019990000000000001 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.874944+0000", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014550571 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878619+0000", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036753390000000001 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.886315+0000", 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076959139999999999 2024-04-15T18:51:25.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.886397+0000", 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1600999999999993e-05 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.891932+0000", 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055358009999999999 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107195 9.8 9:19a5bd44:::benchmark_data_smithi159_173188_object107194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231572+0000", 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.223761051, 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67236641500000005, 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107195 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231572+0000", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231572+0000", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231572+0000", 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3000000000000001e-07 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231594+0000", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2248000000000001e-05 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231601+0000", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1210000000000003e-06 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231606+0000", 2024-04-15T18:51:25.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8880000000000002e-06 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234238+0000", 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002631768 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234245+0000", 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7530000000000004e-06 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.864021+0000", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62977632299999997 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.864140+0000", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118502 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.864254+0000", 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113952 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.874977+0000", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010723623999999999 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878626+0000", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036487379999999999 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.899427+0000", 2024-04-15T18:51:25.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020800593999999999 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.899493+0000", 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6662999999999995e-05 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.903938+0000", 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044448109999999999 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107204 9.8 9:196b6e92:::benchmark_data_smithi159_173188_object107203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231615+0000", 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.22371794099999, 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.676700038, 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107204 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231615+0000", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231615+0000", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231615+0000", 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0999999999999999e-07 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231637+0000", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1767e-05 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231646+0000", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4859999999999997e-06 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231651+0000", 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2830000000000001e-06 2024-04-15T18:51:25.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234269+0000", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026178899999999999 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234276+0000", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.024e-06 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.867338+0000", 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63306228499999995 2024-04-15T18:51:25.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.867499+0000", 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016085900000000001 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.867629+0000", 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013042899999999999 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878633+0000", 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011003453999999999 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878687+0000", 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4002000000000002e-05 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.904018+0000", 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025331637000000001 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.904092+0000", 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3252999999999998e-05 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.908315+0000", 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004223259 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107206 9.8 9:12e5e166:::benchmark_data_smithi159_173188_object107205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231658+0000", 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.22367459500001, 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.68603439099999997, 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107206 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.503 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231658+0000", 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231658+0000", 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231659+0000", 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2500000000000001e-07 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231679+0000", 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0318000000000001e-05 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231687+0000", 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1710000000000002e-06 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231692+0000", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7269999999999998e-06 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234312+0000", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026201660000000002 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234320+0000", 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8890000000000002e-06 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.870987+0000", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63666672599999996 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.871163+0000", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000176441 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.871374+0000", 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000210884 2024-04-15T18:51:25.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878954+0000", 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.007579901 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878966+0000", 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2215000000000001e-05 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.912301+0000", 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.033334453 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.912378+0000", 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7257999999999998e-05 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.917693+0000", 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053148170000000003 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107223 9.8 9:1cee0e0c:::benchmark_data_smithi159_173188_object107222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e821)", 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.231700+0000", 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 243.22363295900001, 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.69015187, 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107223 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.510 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231700+0000", 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231700+0000", 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231700+0000", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9000000000000002e-07 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231753+0000", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2274999999999997e-05 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231761+0000", 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6670000000000002e-06 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.231767+0000", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8549999999999998e-06 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234376+0000", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026086759999999999 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.234387+0000", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1725e-05 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.875061+0000", 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64067386800000004 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.875225+0000", 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000163296 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.875339+0000", 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011403 2024-04-15T18:51:25.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.878976+0000", 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036370370000000001 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.880954+0000", 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019783880000000002 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.917765+0000", 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036810470999999997 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.917839+0000", 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4281000000000003e-05 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.921852+0000", 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040129110000000001 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.516 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:108615 9.3 9:c0fefe84:::benchmark_data_smithi159_173188_object108614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:25.247210+0000", 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.20812302100001, 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.36580686400000001, 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 108615 2024-04-15T18:51:25.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247210+0000", 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247210+0000", 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247211+0000", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.497e-06 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247266+0000", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5003000000000002e-05 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247278+0000", 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1352000000000001e-05 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247286+0000", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9850000000000006e-06 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247339+0000", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3414000000000001e-05 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247535+0000", 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000195906 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,5", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.247712+0000", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000177338 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.251635+0000", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039228759999999996 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.257242+0000", 2024-04-15T18:51:25.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056067529999999999 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.612903+0000", 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35566138200000003 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.612966+0000", 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2875999999999999e-05 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.613017+0000", 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0482000000000002e-05 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:108629 9.8 9:1058a7af:::benchmark_data_smithi159_173188_object108628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e824)", 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:25.270450+0000", 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.18488305899999, 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.57777589900000004, 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.523 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 108629 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270450+0000", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270450+0000", 2024-04-15T18:51:25.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270451+0000", 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.674e-06 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270478+0000", 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6230000000000001e-05 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270498+0000", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0169e-05 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.270507+0000", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1470000000000007e-06 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.277783+0000", 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.007275536 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.277961+0000", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000178128 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.278143+0000", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018238600000000001 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.296739+0000", 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018595839999999999 2024-04-15T18:51:25.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.301029+0000", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042897179999999997 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844411+0000", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.54338262199999998 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844481+0000", 2024-04-15T18:51:25.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9572000000000004e-05 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.848226+0000", 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037448770000000002 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:108629 9.8 e824/821 ondisk, result = 0)", 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:25.281707+0000", 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 240.17362587700001, 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56652765800000005, 2024-04-15T18:51:25.529 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.281707+0000", 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.281707+0000", 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.281707+0000", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9700000000000002e-07 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.281710+0000", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8760000000000001e-06 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844328+0000", 2024-04-15T18:51:25.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56261808899999999 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844336+0000", 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9030000000000004e-06 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844392+0000", 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6119000000000003e-05 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.844402+0000", 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.006e-05 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.848235+0000", 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003832214 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:146501 9.1 9:85051db9:::benchmark_data_smithi159_173188_object37868:head [delete] snapc 0=[] ondisk+write+known_if_redirected e836)", 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:48:29.464489+0000", 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 175.99084421699999, 2024-04-15T18:51:25.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27385695100000002, 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 146501 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464489+0000", 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464489+0000", 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464490+0000", 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.68e-06 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464496+0000", 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0959999999999997e-06 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464510+0000", 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3114e-05 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464517+0000", 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2150000000000004e-06 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464581+0000", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4067999999999997e-05 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464728+0000", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014760000000000001 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.464848+0000", 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119172 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.468771+0000", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039235470000000003 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.469841+0000", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010695069999999999 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738239+0000", 2024-04-15T18:51:25.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.268398839 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738297+0000", 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7806999999999997e-05 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738346+0000", 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8306000000000001e-05 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:146501 9.1 e836/791 ondisk, result = 0)", 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:48:29.468790+0000", 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 175.986542969, 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26956364500000002, 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.468790+0000", 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.468790+0000", 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.468791+0000", 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4799999999999997e-07 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.468799+0000", 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2029999999999992e-06 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.737277+0000", 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.26847796299999999 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.737285+0000", 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1669999999999999e-06 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738225+0000", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00093946700000000004 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738233+0000", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9430000000000006e-06 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:29.738354+0000", 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120054 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200963 9.1 9:8e5be07c:::benchmark_data_smithi159_173188_object92330:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.439173+0000", 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.016159678999998, 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32533377499999999, 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:25.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200963 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439173+0000", 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439173+0000", 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439174+0000", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2839999999999999e-06 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439179+0000", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439191+0000", 2024-04-15T18:51:25.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1177e-05 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439199+0000", 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5350000000000004e-06 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439264+0000", 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5104000000000006e-05 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:25.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439401+0000", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013709700000000001 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,6", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.439534+0000", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013275400000000001 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.444090+0000", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045563679999999999 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.444381+0000", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029055299999999998 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.764399+0000", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.320017932 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.764456+0000", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6643e-05 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.764507+0000", 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1371e-05 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.8 e867,821 9:12cd4bfc:::benchmark_data_smithi159_173188_object41347:head,838'9369)", 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:26.303210+0000", 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.152122955999999, 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33283494800000002, 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.550 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:25.551 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.303210+0000", 2024-04-15T18:51:25.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.303210+0000", 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.303211+0000", 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.61e-07 2024-04-15T18:51:25.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.303215+0000", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.746e-06 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.635408+0000", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.332192459 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.635416+0000", 2024-04-15T18:51:25.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1000000000000004e-06 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.635481+0000", 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5165000000000004e-05 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.636045+0000", 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056371700000000004 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.8 e868,821 9:15a31272:::benchmark_data_smithi159_173188_object53520:head,841'10115)", 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:27.188550+0000", 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 58.266783017000002, 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.68028295000000005, 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.188550+0000", 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.188550+0000", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.188551+0000", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4100000000000001e-06 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.188562+0000", 2024-04-15T18:51:25.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0686999999999999e-05 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.868386+0000", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67982416199999995 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.868396+0000", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0259e-05 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.868465+0000", 2024-04-15T18:51:25.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8726000000000006e-05 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.868833+0000", 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00036770599999999998 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.8 e868,821 9:10051af5:::benchmark_data_smithi159_173188_object53547:head,841'10117)", 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:27.191636+0000", 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 58.263696471999999, 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67811133099999998, 2024-04-15T18:51:25.560 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.191636+0000", 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.191636+0000", 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.191637+0000", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9500000000000002e-07 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.191641+0000", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2039999999999999e-06 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.869462+0000", 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.67782081500000002 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.869471+0000", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5660000000000007e-06 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.869528+0000", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7874000000000002e-05 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.869748+0000", 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00021927700000000001 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.563 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:25.564 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:25.564 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:25.902 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-15T18:51:26.060 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:26.087 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-04-15T18:51:26.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:26.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-15T18:51:26.350 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:26.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 8 1 2024-04-15T18:51:26.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:26 smithi159 ceph-mon[45336]: pgmap v2333: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:27.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:26 smithi159 ceph-mon[45336]: osdmap e896: 12 total, 12 up, 12 in 2024-04-15T18:51:27.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:26 smithi049 ceph-mon[40087]: pgmap v2333: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:27.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:27.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:26 smithi049 ceph-mon[40087]: osdmap e896: 12 total, 12 up, 12 in 2024-04-15T18:51:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:26 smithi082 ceph-mon[44206]: pgmap v2333: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2027408544' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-15T18:51:27.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:26 smithi082 ceph-mon[44206]: osdmap e896: 12 total, 12 up, 12 in 2024-04-15T18:51:28.133 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:28.133 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:28.133 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:51:28.133 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:28.799 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-04-15T18:51:29.062 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:29.144 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:28 smithi159 ceph-mon[45336]: pgmap v2335: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:29.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:28 smithi049 ceph-mon[40087]: pgmap v2335: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:29.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:28 smithi082 ceph-mon[44206]: pgmap v2335: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:29.910 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.8 to 1 (10000) 2024-04-15T18:51:30.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:30.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:29 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:30.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:30.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:29 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:30.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:30.306 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:29 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:30.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 8 1 2024-04-15T18:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:30.957 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: pgmap v2336: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: osdmap e897: 12 total, 12 up, 12 in 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:31.187 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:30 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: pgmap v2336: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: osdmap e897: 12 total, 12 up, 12 in 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:31.274 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:30 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: pgmap v2336: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4110874634' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: osdmap e897: 12 total, 12 up, 12 in 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:31.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:30 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:31.581 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-04-15T18:51:31.843 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:51:32.961 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:32 smithi159 ceph-mon[45336]: pgmap v2338: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:33.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:32 smithi049 ceph-mon[40087]: pgmap v2338: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:33.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:32 smithi082 ceph-mon[44206]: pgmap v2338: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86268 8.5 8:abb429cc:::benchmark_data_smithi159_162467_object86267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 546.79592232000005, 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.615438708, 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.549 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86268 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762708+0000", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762709+0000", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0800000000000004e-07 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762726+0000", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7192000000000001e-05 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762736+0000", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3810000000000005e-06 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762741+0000", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5260000000000001e-06 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762778+0000", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7024000000000003e-05 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762876+0000", 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7795999999999997e-05 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.762962+0000", 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6310999999999994e-05 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766713+0000", 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037510239999999999 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766745+0000", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1340000000000001e-05 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378055+0000", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61131082199999998 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378089+0000", 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3828999999999997e-05 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378147+0000", 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7655000000000003e-05 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86270 8.5 8:a09049c9:::benchmark_data_smithi159_162467_object86269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 546.79402929299999, 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61363609500000005, 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.555 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86270 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764601+0000", 2024-04-15T18:51:33.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764602+0000", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764625+0000", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2901e-05 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764635+0000", 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9059999999999997e-06 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.764643+0000", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2800000000000003e-06 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765036+0000", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000392407 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765142+0000", 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106304 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.765229+0000", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7107000000000005e-05 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.766840+0000", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016110740000000001 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.767079+0000", 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023867499999999999 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378176+0000", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61109754500000002 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378206+0000", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9281999999999998e-05 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.378237+0000", 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1773999999999998e-05 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86284 8.5 8:a9db63aa:::benchmark_data_smithi159_162467_object86283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 546.76642699499996, 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59497596799999997, 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86284 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.561 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792204+0000", 2024-04-15T18:51:33.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4399999999999998e-07 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792225+0000", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0562999999999999e-05 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792235+0000", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4800000000000007e-06 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792240+0000", 2024-04-15T18:51:33.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792282+0000", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1910999999999999e-05 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792403+0000", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012128799999999999 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.792502+0000", 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8987000000000006e-05 2024-04-15T18:51:33.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.794820+0000", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023174480000000002 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:26.795082+0000", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026202700000000002 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387105+0000", 2024-04-15T18:51:33.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.59202341400000003 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387136+0000", 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1027e-05 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387179+0000", 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3510999999999997e-05 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86369 8.5 8:ace0c49e:::benchmark_data_smithi159_162467_object86368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 546.51164503899997, 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34036048600000002, 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86369 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046985+0000", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.046987+0000", 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3850000000000001e-06 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047012+0000", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4839000000000001e-05 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047021+0000", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0410000000000003e-06 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047028+0000", 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1849999999999998e-06 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047078+0000", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0288999999999999e-05 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047193+0000", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114578 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.047292+0000", 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9225e-05 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.049881+0000", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002589058 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.049898+0000", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6453999999999999e-05 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387236+0000", 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33733875600000002 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387289+0000", 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2482000000000003e-05 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.387346+0000", 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7194000000000002e-05 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:86387 8.5 8:a9104f57:::benchmark_data_smithi159_162467_object86386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e739)", 2024-04-15T18:51:33.572 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 546.42573473000004, 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25521748500000002, 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 86387 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132896+0000", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132898+0000", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9929999999999998e-06 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132926+0000", 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8184999999999999e-05 2024-04-15T18:51:33.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132946+0000", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.132955+0000", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7779999999999996e-06 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133023+0000", 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7885000000000005e-05 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133158+0000", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013503300000000001 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.133267+0000", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000108964 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.135567+0000", 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022998680000000001 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.135790+0000", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022382000000000001 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388004+0000", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25221387299999998 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388059+0000", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5041999999999999e-05 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:27.388113+0000", 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3947999999999999e-05 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:109194 8.9 8:93c68f30:::benchmark_data_smithi159_162467_object6889:head [delete] snapc 0=[] ondisk+write+known_if_redirected e749)", 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 513.90139019599997, 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25696971299999999, 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.578 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 109194 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657240+0000", 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657241+0000", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8100000000000002e-07 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657245+0000", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0770000000000001e-06 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657255+0000", 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7620000000000007e-06 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.657260+0000", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3539999999999999e-06 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659686+0000", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024256019999999998 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659814+0000", 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012830799999999999 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.659900+0000", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6269999999999999e-05 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.664407+0000", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045067909999999996 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.667244+0000", 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028369189999999998 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914135+0000", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.24689090599999999 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914165+0000", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0035e-05 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:42:59.914210+0000", 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4907999999999998e-05 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(8.6 762/743 [PushReplyOp(8:61976d2a:::benchmark_data_smithi159_162467_object63438:head)])", 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 433.358819665, 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.288127721, 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199811+0000", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199812+0000", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4799999999999997e-07 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.199820+0000", 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2300000000000008e-06 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485841+0000", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.286021566 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485848+0000", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6359999999999997e-06 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.485898+0000", 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9755000000000001e-05 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:20.487939+0000", 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020406859999999999 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:164212 8.9 8:9021f52a:::benchmark_data_smithi159_162467_object61907:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 424.48787458700002, 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30788343899999998, 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.587 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 164212 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070756+0000", 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070757+0000", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1289999999999999e-06 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070763+0000", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6620000000000002e-06 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070773+0000", 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669000000000001e-05 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070779+0000", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7969999999999997e-06 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070827+0000", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7729000000000003e-05 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.070959+0000", 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013182499999999999 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 0,5", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.071095+0000", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013583900000000001 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.075673+0000", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045782289999999996 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.075708+0000", 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5200000000000002e-05 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378569+0000", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30286143100000001 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378607+0000", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7573000000000001e-05 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378639+0000", 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2356e-05 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.592 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.26791.0:164212 8.9 e766/709 ondisk, result = 0)", 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 424.48370856600002, 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.30372249400000001, 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074922+0000", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074923+0000", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.074930+0000", 2024-04-15T18:51:33.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0010000000000004e-06 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378463+0000", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.303533834 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378469+0000", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2850000000000002e-06 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378560+0000", 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0997999999999998e-05 2024-04-15T18:51:33.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378565+0000", 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9280000000000001e-06 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:44:29.378644+0000", 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9771999999999994e-05 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:184704 8.5 8:ad05c937:::benchmark_data_smithi159_162467_object82399:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:51:33.596 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 391.127439127, 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.280460137, 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 184704 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431191+0000", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431192+0000", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431196+0000", 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-04-15T18:51:33.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431206+0000", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0339999999999999e-05 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431211+0000", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3079999999999998e-06 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431254+0000", 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2358999999999999e-05 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431331+0000", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7669999999999993e-05 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.431414+0000", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2330000000000006e-05 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.435054+0000", 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036404190000000002 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.435521+0000", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000466778 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711589+0000", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27606850399999999 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711623+0000", 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3442999999999998e-05 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.601 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.711652+0000", 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8693000000000001e-05 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.26791.0:184711 8.5 8:aa5d4b26:::benchmark_data_smithi159_162467_object82406:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 391.10350878999998, 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28251888400000003, 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.602 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.26791", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 184711 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455122+0000", 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455123+0000", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.387e-06 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455130+0000", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8329999999999996e-06 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455143+0000", 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3356999999999999e-05 2024-04-15T18:51:33.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455150+0000", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7020000000000004e-06 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455199+0000", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9261000000000002e-05 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455289+0000", 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9815999999999995e-05 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 1,4", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.455373+0000", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4107e-05 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.459099+0000", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037253949999999998 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.459489+0000", 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039004399999999998 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737569+0000", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27808035199999998 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737606+0000", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7478999999999998e-05 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:45:02.737641+0000", 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4150999999999999e-05 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.607 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:25896 9.7 9:ee1ba830:::benchmark_data_smithi159_173188_object25895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e798)", 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 322.47622360100002, 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25357581800000001, 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 25896 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.608 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082407+0000", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082408+0000", 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2500000000000004e-07 2024-04-15T18:51:33.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082427+0000", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8703999999999999e-05 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082437+0000", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0243e-05 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082442+0000", 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4299999999999997e-06 2024-04-15T18:51:33.610 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082480+0000", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7805000000000001e-05 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082568+0000", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8391000000000001e-05 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.082656+0000", 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7449999999999998e-05 2024-04-15T18:51:33.611 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.083527+0000", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00087103799999999998 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.083775+0000", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024838200000000001 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335900+0000", 2024-04-15T18:51:33.612 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.25212455 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335933+0000", 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3467999999999999e-05 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:46:11.335983+0000", 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9431999999999997e-05 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.613 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107183 9.7 9:eca3b222:::benchmark_data_smithi159_173188_object107182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e819)", 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 253.36344044099999, 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.448199395, 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107183 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.614 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195190+0000", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195191+0000", 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1009999999999999e-06 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.615 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195212+0000", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0451000000000001e-05 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195220+0000", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5299999999999996e-06 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195225+0000", 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1710000000000001e-06 2024-04-15T18:51:33.616 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195261+0000", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5707999999999998e-05 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195379+0000", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000117794 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.195472+0000", 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2781999999999994e-05 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.617 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.198905+0000", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003433611 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.201194+0000", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022885560000000002 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643272+0000", 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44207840900000001 2024-04-15T18:51:33.618 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643336+0000", 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4119000000000001e-05 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643389+0000", 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3162999999999999e-05 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.619 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(client.27025.0:107183 9.7 e820/791 ondisk, result = 0)", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 253.33695965000001, 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42172676999999997, 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.620 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221671+0000", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221672+0000", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4799999999999997e-07 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.221682+0000", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7399999999999999e-06 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643194+0000", 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.42151264199999999 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643200+0000", 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0589999999999999e-06 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643254+0000", 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3507000000000001e-05 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.622 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643265+0000", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1498000000000001e-05 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.643398+0000", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013237599999999999 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107200 9.9 9.8b51f119 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 252.95738295999999, 2024-04-15T18:51:33.623 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60121708600000001, 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107200 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601248+0000", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601249+0000", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5859999999999999e-06 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601275+0000", 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6095999999999999e-05 2024-04-15T18:51:33.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601292+0000", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6702999999999999e-05 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601300+0000", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1499999999999999e-06 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601374+0000", 2024-04-15T18:51:33.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3526e-05 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.601384+0000", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0386e-05 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.202433+0000", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60104948999999996 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.202465+0000", 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1149000000000003e-05 2024-04-15T18:51:33.627 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107200 9.9 9:988f8ad1:::benchmark_data_smithi159_173188_object107199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e822)", 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 251.32358217199999, 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.64774168700000001, 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:33.628 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107200 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235048+0000", 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.629 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235050+0000", 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8339999999999999e-06 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235078+0000", 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8215999999999999e-05 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235094+0000", 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5279000000000001e-05 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.630 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.235106+0000", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2765999999999999e-05 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.238843+0000", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037361489999999998 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.238855+0000", 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2471e-05 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851501+0000", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.61264580599999996 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851645+0000", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014372199999999999 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,11", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.851743+0000", 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8747999999999997e-05 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860277+0000", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.008534023 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.860337+0000", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9802000000000001e-05 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.877703+0000", 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017365734000000001 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.877768+0000", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5407000000000006e-05 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:22.882790+0000", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050217300000000003 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200809 9.7 9:eeabe9e9:::benchmark_data_smithi159_173188_object92176:head [delete] snapc 0=[] ondisk+write+known_if_redirected e854)", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:58.799596+0000", 2024-04-15T18:51:33.634 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.759034193999994, 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75602029999999998, 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200809 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799596+0000", 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799596+0000", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799598+0000", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1160000000000002e-06 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799608+0000", 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2809999999999997e-06 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799627+0000", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8889000000000002e-05 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799637+0000", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499000000000001e-05 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799700+0000", 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2424999999999999e-05 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799825+0000", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125711 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 3,7", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.799902+0000", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6946000000000005e-05 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.803918+0000", 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004015434 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.804099+0000", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018145699999999999 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.555519+0000", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.75141968199999998 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.555565+0000", 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6440000000000003e-05 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.555617+0000", 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1419999999999999e-05 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.9 e867,821 9:90e818f5:::benchmark_data_smithi159_173188_object39968:head,837'9306)", 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:25.550505+0000", 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 68.008125534000001, 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56835216499999996, 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:25.550505+0000", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:25.550505+0000", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:25.550506+0000", 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6300000000000004e-07 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:25.550511+0000", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4689999999999996e-06 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.118188+0000", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.56767660399999997 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.118213+0000", 2024-04-15T18:51:33.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4867000000000001e-05 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.118394+0000", 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181163 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.118857+0000", 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046319900000000002 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.9 e868,821 9:98eed6ad:::benchmark_data_smithi159_173188_object45148:head,839'9647)", 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:26.785555+0000", 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.773075383000005, 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.487247133, 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.785555+0000", 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.785555+0000", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.785556+0000", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.785561+0000", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5090000000000001e-06 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.272374+0000", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48681303799999998 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.272398+0000", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3703e-05 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.272470+0000", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2259000000000001e-05 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.272802+0000", 2024-04-15T18:51:33.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00033180900000000002 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGRecoveryDeleteReply(9.9 e868,821 9:9c2df3a2:::benchmark_data_smithi159_173188_object45161:head,839'9649)", 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:26.789634+0000", 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.768996282000003, 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48754528000000003, 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:51:33.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.789634+0000", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.789634+0000", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.789635+0000", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0189999999999999e-06 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:26.789642+0000", 2024-04-15T18:51:33.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2940000000000002e-06 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.276898+0000", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48725629500000001 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.276905+0000", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8399999999999997e-06 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.276958+0000", 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3161000000000001e-05 2024-04-15T18:51:33.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:27.277180+0000", 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221671 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:51:33.650 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:51:33.926 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-15T18:51:34.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:33 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:34.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:33 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:34.277 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:33 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-04-15T18:51:34.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-15T18:51:34.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 9 1 2024-04-15T18:51:35.038 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:34 smithi049 ceph-mon[40087]: pgmap v2339: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:35.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:34 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:35.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:34 smithi049 ceph-mon[40087]: osdmap e898: 12 total, 12 up, 12 in 2024-04-15T18:51:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:34 smithi082 ceph-mon[44206]: pgmap v2339: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:34 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:35.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:34 smithi082 ceph-mon[44206]: osdmap e898: 12 total, 12 up, 12 in 2024-04-15T18:51:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:34 smithi159 ceph-mon[45336]: pgmap v2339: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:34 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/368810770' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-04-15T18:51:35.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:34 smithi159 ceph-mon[45336]: osdmap e898: 12 total, 12 up, 12 in 2024-04-15T18:51:36.304 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:37.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:37.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:36 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:37.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:37.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:36 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:37.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:37.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:37.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:36 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:37.413 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:37.413 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:51:37.413 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:51:37.413 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:38.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-15T18:51:38.634 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:39.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:38 smithi049 ceph-mon[40087]: pgmap v2342: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:39.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:38 smithi082 ceph-mon[44206]: pgmap v2342: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:39.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:38 smithi159 ceph-mon[45336]: pgmap v2342: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:40.138 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.9 to 1 (10000) 2024-04-15T18:51:40.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:39 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:40.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:39 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:40.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:39 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:40.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 9 1 2024-04-15T18:51:41.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:40 smithi049 ceph-mon[40087]: pgmap v2343: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:41.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:41.013 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:40 smithi049 ceph-mon[40087]: osdmap e899: 12 total, 12 up, 12 in 2024-04-15T18:51:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:40 smithi082 ceph-mon[44206]: pgmap v2343: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:41.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:40 smithi082 ceph-mon[44206]: osdmap e899: 12 total, 12 up, 12 in 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:41.327 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:40 smithi159 ceph-mon[45336]: pgmap v2343: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/473938343' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:41.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:40 smithi159 ceph-mon[45336]: osdmap e899: 12 total, 12 up, 12 in 2024-04-15T18:51:42.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-15T18:51:43.147 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:43.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:42 smithi049 ceph-mon[40087]: pgmap v2345: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:43.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:42 smithi082 ceph-mon[44206]: pgmap v2345: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:43.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:42 smithi159 ceph-mon[45336]: pgmap v2345: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:44.944 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-15T18:51:45.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:44 smithi049 ceph-mon[40087]: pgmap v2346: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:45.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:44 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:45.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:44 smithi082 ceph-mon[44206]: pgmap v2346: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:45.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:44 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:45.390 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:82886 8.0 e736/709 8:054a31d8:::benchmark_data_smithi159_162467_object82885:head v 736'5140, mlcod=736'5134)", 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 561.96549587899995, 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21918625799999999, 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.433991+0000", 2024-04-15T18:51:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7700000000000004e-07 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.434090+0000", 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9130999999999997e-05 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650659+0000", 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21656877999999999 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650669+0000", 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3389999999999999e-06 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650736+0000", 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7150000000000006e-05 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.650757+0000", 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1081e-05 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.653135+0000", 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023781499999999999 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.653177+0000", 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2049999999999999e-05 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:82902 8.0 e736/709 8:0b1c4e81:::benchmark_data_smithi159_162467_object82901:head v 736'5141, mlcod=736'5136)", 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 561.95486502200004, 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21600396299999999, 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444622+0000", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444623+0000", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2300000000000001e-06 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.444696+0000", 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2739999999999998e-05 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654441+0000", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20974567099999999 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654449+0000", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9249999999999995e-06 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654512+0000", 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3359000000000001e-05 2024-04-15T18:51:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.654533+0000", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0837000000000001e-05 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.660566+0000", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060331960000000002 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:42:23.660625+0000", 2024-04-15T18:51:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9005000000000002e-05 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26791.0:164234 8.0 e766/709 8:05b44565:::benchmark_data_smithi159_162467_object61929:head v 766'10204, mlcod=766'10202)", 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 436.24851080899998, 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24182635299999999, 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150976+0000", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150977+0000", 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7699999999999992e-07 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.150986+0000", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4159999999999993e-06 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388667+0000", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.237680745 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388681+0000", 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.431e-05 2024-04-15T18:51:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388828+0000", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014656599999999999 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.388874+0000", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6641999999999998e-05 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.392758+0000", 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038836330000000001 2024-04-15T18:51:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.392802+0000", 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4063999999999997e-05 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:20512 9.2 e798/791 9:4609eba9:::benchmark_data_smithi159_173188_object20511:head v 798'1310, mlcod=798'1306)", 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 337.79900345300001, 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.230495805, 2024-04-15T18:51:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600483+0000", 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600484+0000", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8400000000000004e-07 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.600504+0000", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0466999999999999e-05 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.829983+0000", 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22947854200000001 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.829989+0000", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8619999999999999e-06 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830052+0000", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3346e-05 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830078+0000", 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5901000000000002e-05 2024-04-15T18:51:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830952+0000", 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00087385800000000003 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:07.830979+0000", 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7144999999999998e-05 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:55688 9.1 e805/791 9:8e075c20:::benchmark_data_smithi159_173188_object55687:head v 805'3456, mlcod=805'3453)", 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 313.096633408, 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22574402499999999, 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302853+0000", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302854+0000", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.302973+0000", 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118841 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525198+0000", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.222224858 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525214+0000", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6704000000000001e-05 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525270+0000", 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5557999999999999e-05 2024-04-15T18:51:45.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.525284+0000", 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4691999999999999e-05 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.528555+0000", 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003270649 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:46:32.528597+0000", 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2046999999999999e-05 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:88878 9.2 e814/791 9:48f0ea8d:::benchmark_data_smithi159_173188_object88877:head v 814'5542, mlcod=814'5539)", 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 283.900936866, 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21614793800000001, 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498550+0000", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498551+0000", 2024-04-15T18:51:45.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4900000000000004e-07 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.498579+0000", 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.836e-05 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711601+0000", 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21302242900000001 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711608+0000", 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9450000000000004e-06 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711689+0000", 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0228000000000005e-05 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.711715+0000", 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6690999999999999e-05 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.714657+0000", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029415869999999998 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:01.714698+0000", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0748999999999999e-05 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:107185 9.1 e820/791 9:8e8a4364:::benchmark_data_smithi159_173188_object107184:head v 820'6735, mlcod=819'6731)", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 265.17865619100002, 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35684397299999998, 2024-04-15T18:51:45.417 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220830+0000", 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220831+0000", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1899999999999999e-07 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.220861+0000", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.078e-05 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574570+0000", 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35370858199999999 2024-04-15T18:51:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574577+0000", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8240000000000002e-06 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574630+0000", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3201000000000001e-05 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.574648+0000", 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8457999999999998e-05 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.577632+0000", 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029834140000000002 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:20.577674+0000", 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2494999999999999e-05 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.421 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:108615 9.3 e824/791 9:c0fefe84:::benchmark_data_smithi159_173188_object108614:head v 824'6735, mlcod=824'6733)", 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 260.15169067300002, 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.364883597, 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247796+0000", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247797+0000", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.23e-07 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.247874+0000", 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6927000000000004e-05 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609793+0000", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.36191985100000001 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609805+0000", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1585e-05 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609881+0000", 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6261999999999996e-05 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.609904+0000", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2286e-05 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.612643+0000", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027391479999999998 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.612679+0000", 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6715000000000003e-05 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.425 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:108620 9.2 e824/791 9:4e35ac8e:::benchmark_data_smithi159_173188_object108619:head v 824'6814, mlcod=824'6811)", 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 260.14539471799998, 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28632161, 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254092+0000", 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254093+0000", 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8100000000000001e-07 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.254118+0000", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5588e-05 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536803+0000", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28268433599999998 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536813+0000", 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.081e-05 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536876+0000", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2095999999999996e-05 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.536894+0000", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8898000000000001e-05 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540350+0000", 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034555509999999998 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:25.540413+0000", 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3449999999999997e-05 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:123464 9.c e829/791 9:393c6a59:::benchmark_data_smithi159_173188_object14831:head v 829'7686, mlcod=829'7683)", 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 234.13306592699999, 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24320899900000001, 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.430 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266421+0000", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8999999999999996e-07 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.266428+0000", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502266+0000", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.235837662 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502272+0000", 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2469999999999997e-06 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502319+0000", 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6671000000000001e-05 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.502335+0000", 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5634e-05 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.509589+0000", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072545300000000004 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:47:51.509630+0000", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0528000000000002e-05 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:130391 9.7 e832/791 9:e9635ffd:::benchmark_data_smithi159_173188_object21758:head v 832'8140, mlcod=832'8137)", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:51:45.434 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 222.693109523, 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20858686800000001, 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706377+0000", 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706378+0000", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7999999999999996e-07 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.706382+0000", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6589999999999998e-06 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910843+0000", 2024-04-15T18:51:45.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20446082900000001 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910849+0000", 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8019999999999997e-06 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910904+0000", 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4707999999999999e-05 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.910920+0000", 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6875e-05 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.914922+0000", 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040018579999999996 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:48:02.914964+0000", 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1656999999999998e-05 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200709 9.3 9:cd2f9431:::benchmark_data_smithi159_173188_object92076:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019576+0000", 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.379910745, 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.214747248, 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200709 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.439 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019576+0000", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019576+0000", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019576+0000", 2024-04-15T18:51:45.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6499999999999999e-07 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019579+0000", 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1999999999999999e-06 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019591+0000", 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1209000000000001e-05 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019598+0000", 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5460000000000003e-06 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019813+0000", 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021502899999999999 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019821+0000", 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5870000000000004e-06 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.209376+0000", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18955483200000001 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.209463+0000", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7274000000000005e-05 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.209546+0000", 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2888999999999999e-05 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.213126+0000", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035806039999999998 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.213788+0000", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066193199999999995 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.213916+0000", 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012803900000000001 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224107+0000", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010190484 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224636+0000", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052909799999999998 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234180+0000", 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095442819999999994 2024-04-15T18:51:45.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234223+0000", 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2901000000000001e-05 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234323+0000", 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9877000000000006e-05 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200714 9.3 9:cd7ddbae:::benchmark_data_smithi159_173188_object92081:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019613+0000", 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.37987333, 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.224576674, 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200714 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019613+0000", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019613+0000", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019614+0000", 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9799999999999999e-07 2024-04-15T18:51:45.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019616+0000", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5399999999999998e-06 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019629+0000", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2678e-05 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019636+0000", 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6599999999999998e-06 2024-04-15T18:51:45.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019856+0000", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000220666 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019865+0000", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9420000000000001e-06 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.213941+0000", 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.194076048 2024-04-15T18:51:45.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.214033+0000", 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1865000000000002e-05 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.214128+0000", 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4668999999999998e-05 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.223997+0000", 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0098693819999999995 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224071+0000", 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4317e-05 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234130+0000", 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010059032000000001 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234512+0000", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038174299999999998 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.243944+0000", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094316629999999999 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244069+0000", 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012514500000000001 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244107+0000", 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7589999999999998e-05 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244190+0000", 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3336000000000003e-05 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200732 9.3 9:c65e8b31:::benchmark_data_smithi159_173188_object92099:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019646+0000", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.37984034599999, 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22483873600000001, 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200732 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019646+0000", 2024-04-15T18:51:45.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019646+0000", 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019647+0000", 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5400000000000002e-07 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019649+0000", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5009999999999999e-06 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019661+0000", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1338000000000001e-05 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019667+0000", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0360000000000003e-06 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019903+0000", 2024-04-15T18:51:45.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023693299999999999 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019912+0000", 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5459999999999992e-06 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.218737+0000", 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19882518599999999 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.218907+0000", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000169471 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.219061+0000", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000154148 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224080+0000", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050192930000000002 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224316+0000", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000235682 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234443+0000", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010127258 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234560+0000", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117366 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244229+0000", 2024-04-15T18:51:45.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.009668289 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244375+0000", 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000146114 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244409+0000", 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4548000000000002e-05 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244485+0000", 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5573000000000003e-05 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200734 9.3 9:c2b1bb18:::benchmark_data_smithi159_173188_object92101:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019697+0000", 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.379789684, 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22506527800000001, 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.462 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200734 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019697+0000", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019697+0000", 2024-04-15T18:51:45.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019697+0000", 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019700+0000", 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9780000000000001e-06 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019709+0000", 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8440000000000004e-06 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019715+0000", 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0190000000000003e-06 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019947+0000", 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023125399999999999 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019951+0000", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.369e-06 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224663+0000", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204712372 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224751+0000", 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7499999999999999e-05 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.224899+0000", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014781500000000001 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234394+0000", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094957559999999993 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234454+0000", 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9485999999999999e-05 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244034+0000", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095806439999999993 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244267+0000", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023238299999999999 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244540+0000", 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027302499999999998 2024-04-15T18:51:45.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244651+0000", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110933 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244687+0000", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5882e-05 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244762+0000", 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5469000000000005e-05 2024-04-15T18:51:45.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200744 9.3 9:c98aa08b:::benchmark_data_smithi159_173188_object92111:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019723+0000", 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.379763455, 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23342075100000001, 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.470 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200744 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019723+0000", 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019723+0000", 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019723+0000", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3599999999999999e-07 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019726+0000", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3860000000000001e-06 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019738+0000", 2024-04-15T18:51:45.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2634e-05 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019745+0000", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1840000000000004e-06 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019964+0000", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000219574 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019968+0000", 2024-04-15T18:51:45.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6069999999999999e-06 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.229711+0000", 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20974335399999999 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.229863+0000", 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015226000000000001 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.230480+0000", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00061647600000000002 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234458+0000", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003978316 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.243988+0000", 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095294509999999995 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244340+0000", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035270599999999999 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244798+0000", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045750100000000002 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244835+0000", 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7571999999999999e-05 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.252981+0000", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081451300000000004 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253032+0000", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1876000000000003e-05 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253144+0000", 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111388 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.477 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200758 9.3 9:c7798815:::benchmark_data_smithi159_173188_object92125:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019752+0000", 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.379734476, 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.233669192, 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200758 2024-04-15T18:51:45.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019752+0000", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019752+0000", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019753+0000", 2024-04-15T18:51:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3200000000000005e-07 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019755+0000", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2749999999999998e-06 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019763+0000", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8299999999999996e-06 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019775+0000", 2024-04-15T18:51:45.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2043e-05 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019981+0000", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020592200000000001 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019987+0000", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5830000000000003e-06 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234603+0000", 2024-04-15T18:51:45.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21461530300000001 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234722+0000", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119707 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.234827+0000", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010492699999999999 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244000+0000", 2024-04-15T18:51:45.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0091726749999999999 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244306+0000", 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030599400000000003 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244612+0000", 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030626399999999999 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244869+0000", 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025640599999999998 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244905+0000", 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6473000000000001e-05 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253205+0000", 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0083001040000000009 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253269+0000", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4028000000000004e-05 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253421+0000", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015212599999999999 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200777 9.3 9:c5727b4c:::benchmark_data_smithi159_173188_object92144:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019809+0000", 2024-04-15T18:51:45.485 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.37967763499999, 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23991992600000001, 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200777 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019809+0000", 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019809+0000", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019809+0000", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3700000000000001e-07 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019812+0000", 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.993e-06 2024-04-15T18:51:45.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019822+0000", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4530000000000008e-06 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019832+0000", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9110000000000005e-06 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.020026+0000", 2024-04-15T18:51:45.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000193816 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.020037+0000", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1548e-05 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.244975+0000", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22493838899999999 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.245066+0000", 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0828000000000005e-05 2024-04-15T18:51:45.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.245154+0000", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7948999999999994e-05 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253462+0000", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0083081960000000003 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253473+0000", 2024-04-15T18:51:45.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0154e-05 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.256650+0000", 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031775739999999999 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.257277+0000", 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062713500000000002 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.259171+0000", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001893739 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.259492+0000", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032123200000000002 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.259560+0000", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7899999999999997e-05 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.259729+0000", 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000168672 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.27025.0:200802 9.3 9:c3ce6622:::benchmark_data_smithi159_173188_object92169:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e856)", 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:00.019849+0000", 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 105.379637243, 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24914549899999999, 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.27025", 2024-04-15T18:51:45.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 200802 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019849+0000", 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019849+0000", 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019850+0000", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9400000000000001e-07 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019852+0000", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6680000000000001e-06 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019861+0000", 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7819999999999999e-06 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.019867+0000", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9070000000000004e-06 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.020083+0000", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021586500000000001 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.020094+0000", 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0644e-05 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.248665+0000", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22857160300000001 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.248761+0000", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5533999999999997e-05 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,5,8,9,11", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.248859+0000", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7848000000000002e-05 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:45.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253478+0000", 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046190229999999999 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.253527+0000", 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8930000000000001e-05 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.256698+0000", 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003171255 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.257319+0000", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062135499999999998 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.259787+0000", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024675259999999998 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.268759+0000", 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0089718909999999992 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.268837+0000", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8458e-05 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:00.268995+0000", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015781599999999999 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.27025.0:217252 9.2 e867/791 9:4e35ac8e:::benchmark_data_smithi159_173188_object108619:head v 867'13628, mlcod=867'13625)", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:50:25.525920+0000", 2024-04-15T18:51:45.500 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 79.873567011999995, 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41998162100000003, 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525920+0000", 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525920+0000", 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525921+0000", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5999999999999991e-07 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.525928+0000", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3719999999999999e-06 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.941822+0000", 2024-04-15T18:51:45.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41589404299999999 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.941828+0000", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1550000000000004e-06 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.941883+0000", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4697999999999997e-05 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.941907+0000", 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3750000000000001e-05 2024-04-15T18:51:45.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.945863+0000", 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039567029999999998 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:50:25.945901+0000", 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7939999999999999e-05 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:45.504 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:45.505 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:44 smithi159 ceph-mon[45336]: pgmap v2346: 1 pgs: 1 active+clean; 577 KiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:45.505 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:44 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-04-15T18:51:45.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 10 1 2024-04-15T18:51:46.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:45 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:46.152 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:45 smithi049 ceph-mon[40087]: osdmap e900: 12 total, 12 up, 12 in 2024-04-15T18:51:46.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:45 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:46.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:45 smithi082 ceph-mon[44206]: osdmap e900: 12 total, 12 up, 12 in 2024-04-15T18:51:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:45 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1213598758' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-04-15T18:51:46.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:45 smithi159 ceph-mon[45336]: osdmap e900: 12 total, 12 up, 12 in 2024-04-15T18:51:46.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-15T18:51:46.540 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:47.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:47.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:47.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: osdmap e901: 12 total, 12 up, 12 in 2024-04-15T18:51:47.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:47.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: osdmap e901: 12 total, 12 up, 12 in 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:47.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:47.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: osdmap e901: 12 total, 12 up, 12 in 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:51:47.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:51:47.799 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:48.956 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.10 to 1 (10000) 2024-04-15T18:51:49.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:48 smithi049 ceph-mon[40087]: pgmap v2350: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:49.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:48 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:49.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:48 smithi082 ceph-mon[44206]: pgmap v2350: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:49.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:48 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:49.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:48 smithi159 ceph-mon[45336]: pgmap v2350: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:49.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:48 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:50.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:49 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:50.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:49 smithi049 ceph-mon[40087]: osdmap e902: 12 total, 12 up, 12 in 2024-04-15T18:51:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:49 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:50.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:49 smithi082 ceph-mon[44206]: osdmap e902: 12 total, 12 up, 12 in 2024-04-15T18:51:50.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 10 1 2024-04-15T18:51:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:49 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1524213814' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:50.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:49 smithi159 ceph-mon[45336]: osdmap e902: 12 total, 12 up, 12 in 2024-04-15T18:51:50.484 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:50.484 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:51:50.484 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-04-15T18:51:50.484 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:51.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:50 smithi049 ceph-mon[40087]: pgmap v2352: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:51.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-15T18:51:51.172 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:51.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:50 smithi082 ceph-mon[44206]: pgmap v2352: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:51.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:50 smithi159 ceph-mon[45336]: pgmap v2352: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:51.968 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:53.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:52 smithi049 ceph-mon[40087]: pgmap v2353: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:53.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:52 smithi082 ceph-mon[44206]: pgmap v2353: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:53.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:52 smithi159 ceph-mon[45336]: pgmap v2353: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:54.106 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-15T18:51:54.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:53 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:54.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:53 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:54.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:53 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-04-15T18:51:54.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 3 1 2024-04-15T18:51:55.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:54 smithi049 ceph-mon[40087]: pgmap v2354: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:55.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:54 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:55.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:54 smithi049 ceph-mon[40087]: osdmap e903: 12 total, 12 up, 12 in 2024-04-15T18:51:55.226 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:55.226 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-04-15T18:51:55.226 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-04-15T18:51:55.226 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-04-15T18:51:55.226 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:55.233 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:55.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:54 smithi082 ceph-mon[44206]: pgmap v2354: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:55.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:54 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:55.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:54 smithi082 ceph-mon[44206]: osdmap e903: 12 total, 12 up, 12 in 2024-04-15T18:51:55.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:54 smithi159 ceph-mon[45336]: pgmap v2354: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:55.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:54 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1643383722' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-04-15T18:51:55.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:54 smithi159 ceph-mon[45336]: osdmap e903: 12 total, 12 up, 12 in 2024-04-15T18:51:56.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-15T18:51:57.005 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:51:57.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:56 smithi049 ceph-mon[40087]: pgmap v2356: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:57.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:56 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:57.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:56 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:56 smithi082 ceph-mon[44206]: pgmap v2356: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:56 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:57.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:56 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:57.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:56 smithi159 ceph-mon[45336]: pgmap v2356: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:57.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:56 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:57.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:56 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:58.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:58 smithi082 ceph-mon[44206]: osdmap e904: 12 total, 12 up, 12 in 2024-04-15T18:51:58.389 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:57 smithi049 ceph-mon[40087]: osdmap e904: 12 total, 12 up, 12 in 2024-04-15T18:51:58.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:58 smithi159 ceph-mon[45336]: osdmap e904: 12 total, 12 up, 12 in 2024-04-15T18:51:58.993 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-04-15T18:51:59.011 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-04-15T18:51:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:164219 8.e 8:73021211:::benchmark_data_smithi159_162467_object61914:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.079132+0000", 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 449.94304760900002, 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.37070884100000001, 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:51:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 164219 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079132+0000", 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079132+0000", 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079134+0000", 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.652e-06 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079140+0000", 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6999999999999996e-06 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079154+0000", 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4073000000000001e-05 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079162+0000", 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6340000000000007e-06 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079209+0000", 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6993000000000002e-05 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079301+0000", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1935999999999995e-05 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.079378+0000", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7220999999999998e-05 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.083331+0000", 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039526259999999999 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.097496+0000", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01416539 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449742+0000", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.35224547 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449792+0000", 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0158999999999999e-05 2024-04-15T18:51:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.449841+0000", 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8986999999999997e-05 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(8.f 766/743 [PushReplyOp(8:fd3ca488:::benchmark_data_smithi159_162467_object63039:head)])", 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:44:29.094491+0000", 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 449.927689203, 2024-04-15T18:51:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33920763799999998, 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.094491+0000", 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.094491+0000", 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.094491+0000", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2499999999999995e-07 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.094497+0000", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6010000000000002e-06 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.430785+0000", 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33628846899999998 2024-04-15T18:51:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.430793+0000", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8590000000000005e-06 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.430862+0000", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9387000000000005e-05 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:44:29.433698+0000", 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028357970000000001 2024-04-15T18:51:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26791.0:184699 8.e 8:7aff4c81:::benchmark_data_smithi159_162467_object82394:head [delete] snapc 0=[] ondisk+write+known_if_redirected e775)", 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.415183+0000", 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.60699731699998, 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32677178299999998, 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-04-15T18:51:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26791", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 184699 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415183+0000", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415183+0000", 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415184+0000", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1570000000000001e-06 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415189+0000", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3789999999999997e-06 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415202+0000", 2024-04-15T18:51:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2714000000000001e-05 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415211+0000", 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9279999999999999e-06 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415275+0000", 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4504000000000004e-05 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415416+0000", 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014110300000000001 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.415520+0000", 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010370300000000001 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.419405+0000", 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038845559999999999 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.419489+0000", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4611999999999996e-05 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741851+0000", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32236137999999998 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741907+0000", 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6824999999999999e-05 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741954+0000", 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6922000000000002e-05 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26791.0:184699 8.e e775/709 ondisk, result = 0)", 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:02.420731+0000", 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.60144880600001, 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32122877900000002, 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.420731+0000", 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.420731+0000", 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.420732+0000", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.420737+0000", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9649999999999999e-06 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741761+0000", 2024-04-15T18:51:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32102393499999998 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741779+0000", 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8544999999999999e-05 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741835+0000", 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6137000000000001e-05 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741844+0000", 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9949999999999994e-06 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:02.741960+0000", 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115364 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(8.7 e786,743 8:ee59f91a:::benchmark_data_smithi159_162467_object53910:head,762'9806)", 2024-04-15T18:51:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:36.215049+0000", 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 382.80713041500002, 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48101633599999999, 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.215049+0000", 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.215049+0000", 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.215050+0000", 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0049999999999999e-06 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.215057+0000", 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1020000000000002e-06 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.695934+0000", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.48087748400000002 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.695943+0000", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9910000000000008e-06 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.695984+0000", 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1294999999999997e-05 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.695991+0000", 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6599999999999998e-06 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:36.696066+0000", 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4799e-05 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(8.7 e786,743 8:e70b0e9d:::benchmark_data_smithi159_162467_object78668:head,770'11328)", 2024-04-15T18:51:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:38.278564+0000", 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 380.74361595599999, 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.41010553799999999, 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.278564+0000", 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.278564+0000", 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.278564+0000", 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8999999999999997e-07 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.278568+0000", 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5130000000000001e-06 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.688528+0000", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.40995984200000002 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.688539+0000", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0869e-05 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.688614+0000", 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5464999999999997e-05 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.688622+0000", 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3389999999999993e-06 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.688669+0000", 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.702e-05 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGRecoveryDelete(8.7 e787,743 8:e4d7c3c5:::benchmark_data_smithi159_162467_object78847:head,770'11342)", 2024-04-15T18:51:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-04-15T18:45:38.708812+0000", 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 380.31336804300003, 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54118186499999998, 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.708812+0000", 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-04-15T18:51:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.708812+0000", 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.708812+0000", 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2900000000000004e-07 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:38.708816+0000", 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0509999999999998e-06 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:39.249838+0000", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.54102129200000004 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:39.249847+0000", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9290000000000004e-06 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:39.249910+0000", 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3596000000000006e-05 2024-04-15T18:51:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:39.249918+0000", 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1410000000000005e-06 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-04-15T18:45:39.249994+0000", 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5327000000000006e-05 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-04-15T18:51:59.044 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-04-15T18:51:59.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:59 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:51:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:59 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:51:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:59 smithi082 ceph-mon[44206]: pgmap v2358: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:59 smithi082 ceph-mon[44206]: osdmap e905: 12 total, 12 up, 12 in 2024-04-15T18:51:59.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:51:59 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:51:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:59 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:51:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:59 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:51:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:59 smithi159 ceph-mon[45336]: pgmap v2358: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:59 smithi159 ceph-mon[45336]: osdmap e905: 12 total, 12 up, 12 in 2024-04-15T18:51:59.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:51:59 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:51:59.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:59 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:51:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:59 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:51:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:59 smithi049 ceph-mon[40087]: pgmap v2358: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:51:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:59 smithi049 ceph-mon[40087]: osdmap e905: 12 total, 12 up, 12 in 2024-04-15T18:51:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:51:59 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:51:59.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-15T18:52:00.170 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:00.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:00 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:00.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:00 smithi049 ceph-mon[40087]: osdmap e906: 12 total, 12 up, 12 in 2024-04-15T18:52:00.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:00 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:00.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:00 smithi082 ceph-mon[44206]: osdmap e906: 12 total, 12 up, 12 in 2024-04-15T18:52:00.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:00 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3117419091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:00.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:00 smithi159 ceph-mon[45336]: osdmap e906: 12 total, 12 up, 12 in 2024-04-15T18:52:00.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-15T18:52:00.736 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:01.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:01 smithi049 ceph-mon[40087]: pgmap v2361: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:01.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:01 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:52:01.228 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:01 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:52:01.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:01 smithi082 ceph-mon[44206]: pgmap v2361: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:01.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:01 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:52:01.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:01 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:52:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:01 smithi159 ceph-mon[45336]: pgmap v2361: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:01 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:52:01.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:01 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:52:01.982 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:01.982 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:52:01.983 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:52:01.983 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:02.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-15T18:52:02.906 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-15T18:52:03.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:03 smithi082 ceph-mon[44206]: pgmap v2362: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:03.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:03 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:52:03.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:03 smithi049 ceph-mon[40087]: pgmap v2362: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:03.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:03 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:52:03.394 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:03 smithi159 ceph-mon[45336]: pgmap v2362: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:03.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:03 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-15T18:52:03.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 2 1 2024-04-15T18:52:04.039 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:04.241 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:04 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:04.241 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:04 smithi082 ceph-mon[44206]: osdmap e907: 12 total, 12 up, 12 in 2024-04-15T18:52:04.261 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:04 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:04.261 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:04 smithi049 ceph-mon[40087]: osdmap e907: 12 total, 12 up, 12 in 2024-04-15T18:52:04.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:04 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1797260366' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-15T18:52:04.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:04 smithi159 ceph-mon[45336]: osdmap e907: 12 total, 12 up, 12 in 2024-04-15T18:52:04.529 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:52:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:52:04.529 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:52:04.529 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:05.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:05 smithi049 ceph-mon[40087]: pgmap v2364: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:05.259 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:05 smithi082 ceph-mon[44206]: pgmap v2364: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:05.288 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-15T18:52:05.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:05 smithi159 ceph-mon[45336]: pgmap v2364: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:05.551 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:07.025 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66872 8.4 8:2fd3b0dc:::benchmark_data_smithi159_162467_object66871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:52:07.158 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 599.688959404, 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.221963629, 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66872 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:52:07.159 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477537+0000", 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477538+0000", 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7899999999999997e-07 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.160 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477567+0000", 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8904999999999999e-05 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477579+0000", 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1773999999999999e-05 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477586+0000", 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8129999999999998e-06 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.161 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477626+0000", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0457000000000002e-05 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477716+0000", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0026999999999997e-05 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 9,10", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.477805+0000", 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8660999999999997e-05 2024-04-15T18:52:07.162 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.479626+0000", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001821666 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.532510+0000", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.052884050000000002 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699344+0000", 2024-04-15T18:52:07.163 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.16683369100000001 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699462+0000", 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000117971 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699501+0000", 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8735000000000001e-05 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.164 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.26791.0:66872 8.4 e732/709 ondisk, result = 0)", 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 599.68620757199994, 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.219219101, 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.165 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480289+0000", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480290+0000", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0699999999999997e-07 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.480293+0000", 2024-04-15T18:52:07.166 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3100000000000001e-06 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699149+0000", 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.218855771 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699165+0000", 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.632e-05 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.167 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699315+0000", 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014995399999999999 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699334+0000", 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8638999999999999e-05 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.699508+0000", 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017450000000000001 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.168 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:66881 8.0 8:006672f8:::benchmark_data_smithi159_162467_object66880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e732)", 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 599.67835677100004, 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22338060300000001, 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 66881 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.169 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488140+0000", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488141+0000", 2024-04-15T18:52:07.170 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5939999999999999e-06 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488214+0000", 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2871000000000001e-05 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488226+0000", 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1223000000000001e-05 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.171 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488249+0000", 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3269999999999999e-05 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488287+0000", 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8186999999999999e-05 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488391+0000", 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000104149 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.172 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.488496+0000", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010463600000000001 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490307+0000", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001811602 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.490905+0000", 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00059743399999999999 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.173 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711410+0000", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.220504902 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711475+0000", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5753999999999994e-05 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:07.711520+0000", 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4981000000000002e-05 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.174 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82881 8.0 8:07151622:::benchmark_data_smithi159_162467_object82880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 583.737364684, 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22421091100000001, 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.175 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82881 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429132+0000", 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.176 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429144+0000", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2394e-05 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429232+0000", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7344999999999998e-05 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429242+0000", 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0164e-05 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.177 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.429250+0000", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7249999999999997e-06 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431184+0000", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019340550000000001 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431318+0000", 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013448299999999999 2024-04-15T18:52:07.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.431421+0000", 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010239900000000001 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433412+0000", 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019919009999999999 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.179 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436831+0000", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034181820000000001 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653287+0000", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.216456327 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653318+0000", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1352000000000001e-05 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.180 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.653343+0000", 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4584e-05 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82886 8.0 8:054a31d8:::benchmark_data_smithi159_162467_object82885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 583.732955199, 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.227292199, 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.181 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82886 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433541+0000", 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433543+0000", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4470000000000001e-06 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433600+0000", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6993999999999997e-05 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433610+0000", 2024-04-15T18:52:07.183 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6560000000000004e-06 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433617+0000", 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8720000000000002e-06 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433677+0000", 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9969000000000001e-05 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.184 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433781+0000", 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103631 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.433900+0000", 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011869199999999999 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436043+0000", 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021429449999999998 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.185 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.436987+0000", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00094478900000000002 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660767+0000", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22377945399999999 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660806+0000", 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9045999999999999e-05 2024-04-15T18:52:07.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.660834+0000", 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7704000000000001e-05 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:82902 8.0 8:0b1c4e81:::benchmark_data_smithi159_162467_object82901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e736)", 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 583.722331467, 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22047940599999999, 2024-04-15T18:52:07.187 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 82902 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.188 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444165+0000", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444166+0000", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.147e-06 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444220+0000", 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3282999999999998e-05 2024-04-15T18:52:07.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444229+0000", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7860000000000008e-06 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444235+0000", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.463e-06 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444280+0000", 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5045e-05 2024-04-15T18:52:07.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444395+0000", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011479 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.444498+0000", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010325800000000001 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.446700+0000", 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022018820000000001 2024-04-15T18:52:07.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.447033+0000", 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00033350600000000002 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664573+0000", 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21753967799999999 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.192 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664616+0000", 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3126e-05 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:42:23.664645+0000", 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8442e-05 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.193 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDPGPush(8.d 756/743 [PushOp(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head, version: 726'2744, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ba8895a8:::benchmark_data_smithi159_162467_object43443:head@726'2744, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 504.93539207100002, 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24113336099999999, 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.194 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231105+0000", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231106+0000", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.508e-06 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.231139+0000", 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3120000000000001e-05 2024-04-15T18:52:07.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469866+0000", 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23872668799999999 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469882+0000", 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5794000000000002e-05 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.196 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469963+0000", 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.1763999999999998e-05 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.469972+0000", 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8380000000000008e-06 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:43:42.472238+0000", 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022656489999999998 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.197 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:164234 8.0 8:05b44565:::benchmark_data_smithi159_162467_object61929:head [delete] snapc 0=[] ondisk+write+known_if_redirected e766)", 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 458.01590391799999, 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24243994899999999, 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.198 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 164234 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150593+0000", 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.199 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150595+0000", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.137e-06 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150605+0000", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9299999999999998e-06 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150640+0000", 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5648000000000002e-05 2024-04-15T18:52:07.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150664+0000", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3632e-05 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150715+0000", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1121000000000001e-05 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150811+0000", 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6012000000000002e-05 2024-04-15T18:52:07.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.150891+0000", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.9812000000000001e-05 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.154655+0000", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037635139999999999 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.155576+0000", 2024-04-15T18:52:07.202 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00092136400000000004 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392953+0000", 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23737709700000001 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.392993+0000", 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0268000000000001e-05 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.203 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:29.393033+0000", 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9413999999999999e-05 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.26791.0:169755 8.0 8:06929281:::benchmark_data_smithi159_162467_object67450:head [delete] snapc 0=[] ondisk+write+known_if_redirected e767)", 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 448.56014833199998, 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22069882099999999, 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.204 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.26791", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/2666727279", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 169755 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606348+0000", 2024-04-15T18:52:07.205 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606350+0000", 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.31e-06 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606356+0000", 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0800000000000002e-06 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.206 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606369+0000", 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3742e-05 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606377+0000", 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2880000000000004e-06 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606422+0000", 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5330999999999997e-05 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.207 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606543+0000", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000120671 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 3,9", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.606637+0000", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4759000000000005e-05 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.611592+0000", 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0049540929999999997 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.208 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.618261+0000", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0066691210000000001 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826959+0000", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20869838900000001 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.826999+0000", 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9787999999999999e-05 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.209 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:44:38.827047+0000", 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8248999999999998e-05 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:93297 9.6 9:6f080384:::benchmark_data_smithi159_173188_object93296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e816)", 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 300.89132895900002, 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.24058054700000001, 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.210 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 93297 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275168+0000", 2024-04-15T18:52:07.211 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275169+0000", 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.491e-06 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275227+0000", 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7577000000000001e-05 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.212 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275241+0000", 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.454e-05 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275250+0000", 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2309999999999996e-06 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275308+0000", 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7991000000000001e-05 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.213 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275442+0000", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000134315 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,10", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.275577+0000", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000135537 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.278398+0000", 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028205539999999999 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.214 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.278472+0000", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3807999999999995e-05 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515675+0000", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.23720350500000001 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515715+0000", 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9694000000000003e-05 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.215 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:06.515748+0000", 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3303999999999998e-05 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107186 9.a 9:5ca3234a:::benchmark_data_smithi159_173188_object107185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.94456765199999, 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.497663148, 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.216 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107186 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.217 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221929+0000", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221931+0000", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6819999999999999e-06 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.221998+0000", 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7490000000000006e-05 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.218 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222013+0000", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4367e-05 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222020+0000", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8620000000000003e-06 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222080+0000", 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9579e-05 2024-04-15T18:52:07.219 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222224+0000", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014447900000000001 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.222326+0000", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000102008 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225366+0000", 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030397940000000002 2024-04-15T18:52:07.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.414130+0000", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.188763655 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719483+0000", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.30535280599999998 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719541+0000", 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8579999999999998e-05 2024-04-15T18:52:07.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719592+0000", 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0846000000000001e-05 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107188 9.f 9:ff26e8fa:::benchmark_data_smithi159_173188_object107187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.94142697299998, 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.62629479399999999, 2024-04-15T18:52:07.222 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107188 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.223 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225070+0000", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225071+0000", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.623e-06 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225131+0000", 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9784999999999998e-05 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.224 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225144+0000", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2860000000000001e-05 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225154+0000", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5899999999999997e-06 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225208+0000", 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4045000000000002e-05 2024-04-15T18:52:07.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225336+0000", 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012817199999999999 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.225442+0000", 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106579 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.226 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228413+0000", 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002971137 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228608+0000", 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00019488599999999999 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851256+0000", 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.62264738900000005 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.227 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851318+0000", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2151999999999997e-05 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851364+0000", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.6576000000000003e-05 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107186 9.a e820/791 ondisk, result = 0)", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:52:07.228 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.93750602599999, 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49061027400000001, 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228991+0000", 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0900000000000001e-07 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.228996+0000", 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4610000000000001e-06 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.230 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719405+0000", 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.49040913600000002 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719410+0000", 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8559999999999998e-06 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719466+0000", 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6671e-05 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.231 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719475+0000", 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7690000000000002e-06 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719601+0000", 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012567200000000001 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:107208 9.f 9:f46dc3ae:::benchmark_data_smithi159_173188_object107207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.45102757299998, 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.41139816699999998, 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 107208 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.233 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715469+0000", 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715471+0000", 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9700000000000002e-06 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.234 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715584+0000", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000112747 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715600+0000", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6232999999999999e-05 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715608+0000", 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8460000000000008e-06 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.235 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715672+0000", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4622000000000006e-05 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.715842+0000", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016921899999999999 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.716014+0000", 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00017243599999999999 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.236 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.718882+0000", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028680239999999998 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.719280+0000", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039826500000000002 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126718+0000", 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.407437938 2024-04-15T18:52:07.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126796+0000", 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7824000000000006e-05 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126867+0000", 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1043000000000004e-05 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.238 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop_reply(client.27025.0:107208 9.f e820/791 ondisk, result = 0)", 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 286.315419292, 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27579915900000002, 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.239 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851077+0000", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851079+0000", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.8729999999999999e-06 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:20.851089+0000", 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7459999999999995e-06 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.240 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126561+0000", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27547190900000001 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126576+0000", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4956999999999999e-05 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126699+0000", 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012357700000000001 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.241 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126711+0000", 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2028000000000001e-05 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:21.126877+0000", 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000165069 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.242 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:110348 9.f 9:feb2cf5a:::benchmark_data_smithi159_173188_object1715:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.31027689799998, 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22072198600000001, 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110348 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.243 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856220+0000", 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856221+0000", 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1489999999999999e-06 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.244 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856225+0000", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4340000000000002e-06 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856235+0000", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5859999999999994e-06 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856241+0000", 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9320000000000001e-06 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.245 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856291+0000", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0559000000000002e-05 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856421+0000", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000129987 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,9", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.856549+0000", 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012783200000000001 2024-04-15T18:52:07.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.859056+0000", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0025069419999999999 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.873184+0000", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01412771 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076858+0000", 2024-04-15T18:52:07.247 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.203673839 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076889+0000", 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1689000000000001e-05 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.076942+0000", 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2327000000000002e-05 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.248 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:110350 9.a 9:521ea892:::benchmark_data_smithi159_173188_object1717:head [delete] snapc 0=[] ondisk+write+known_if_redirected e825)", 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 278.30602106399999, 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22033312399999999, 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 110350 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.249 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860476+0000", 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.75e-07 2024-04-15T18:52:07.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860479+0000", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0680000000000001e-06 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860483+0000", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3409999999999996e-06 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860487+0000", 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1310000000000002e-06 2024-04-15T18:52:07.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860522+0000", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5845e-05 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860600+0000", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7152999999999998e-05 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.860658+0000", 2024-04-15T18:52:07.252 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8718999999999998e-05 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.863725+0000", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0030663330000000001 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:28.863801+0000", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6598000000000001e-05 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080741+0000", 2024-04-15T18:52:07.253 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.21694023400000001 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080772+0000", 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0395999999999998e-05 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:29.080809+0000", 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6831e-05 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:185486 9.a 9:551b3fde:::benchmark_data_smithi159_173188_object76853:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.781718576, 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26204080299999999, 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185486 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.255 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384778+0000", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384780+0000", 2024-04-15T18:52:07.256 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4410000000000001e-06 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384785+0000", 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4410000000000001e-06 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384800+0000", 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4829e-05 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.257 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384808+0000", 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8820000000000001e-06 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.384861+0000", 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.3816000000000001e-05 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385000+0000", 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00013899600000000001 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.258 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.385134+0000", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000133702 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.389490+0000", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043563209999999998 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.396059+0000", 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0065680110000000003 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.259 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646717+0000", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.25065870299999998 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646774+0000", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7104000000000001e-05 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.646819+0000", 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4557e-05 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.260 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.27025.0:185492 9.a 9:536494b0:::benchmark_data_smithi159_173188_object76859:head [delete] snapc 0=[] ondisk+write+known_if_redirected e847)", 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 152.774997262, 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.27240242599999998, 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.27025", 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:07.261 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 185492 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391499+0000", 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.262 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391501+0000", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.482e-06 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391508+0000", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.1670000000000002e-06 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391522+0000", 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4003e-05 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.263 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391532+0000", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6860000000000001e-06 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391591+0000", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8789000000000002e-05 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391706+0000", 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000115592 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.264 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for subops from 1,7", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.391798+0000", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2268999999999994e-05 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "op_commit", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.395521+0000", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037226590000000001 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.396129+0000", 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00060791299999999996 2024-04-15T18:52:07.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663834+0000", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.26770484300000003 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663868+0000", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4471000000000003e-05 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:34.663902+0000", 2024-04-15T18:52:07.266 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3552e-05 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:217244 9.d e867/791 9:b7f77d1e:::benchmark_data_smithi159_173188_object108611:head v 867'13679, mlcod=867'13676)", 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:25.518534+0000", 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 101.647962698, 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.70404940699999996, 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.267 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.518534+0000", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.518534+0000", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.518534+0000", 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.23e-07 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.268 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:25.518538+0000", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9149999999999998e-06 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.218233+0000", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.69969450600000005 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.218249+0000", 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6164e-05 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.269 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.218394+0000", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014489799999999999 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.218413+0000", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9321e-05 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.222543+0000", 2024-04-15T18:52:07.270 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0041302500000000002 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:26.222583+0000", 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0129999999999997e-05 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:07.271 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:07.273 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:07 smithi082 ceph-mon[44206]: pgmap v2365: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:07.273 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:07 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:07.273 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:07 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:07.274 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:07 smithi082 ceph-mon[44206]: osdmap e908: 12 total, 12 up, 12 in 2024-04-15T18:52:07.274 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:07 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:07.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:07 smithi049 ceph-mon[40087]: pgmap v2365: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:07.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:07 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:07.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:07 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:07.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:07 smithi049 ceph-mon[40087]: osdmap e908: 12 total, 12 up, 12 in 2024-04-15T18:52:07.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:07 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:07 smithi159 ceph-mon[45336]: pgmap v2365: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:07 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:07 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:07 smithi159 ceph-mon[45336]: osdmap e908: 12 total, 12 up, 12 in 2024-04-15T18:52:07.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:07 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:07.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-15T18:52:07.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-15T18:52:08.070 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:08.176 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:08.276 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:08 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:08.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:08 smithi082 ceph-mon[44206]: osdmap e909: 12 total, 12 up, 12 in 2024-04-15T18:52:08.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:08 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:08.302 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:08 smithi049 ceph-mon[40087]: osdmap e909: 12 total, 12 up, 12 in 2024-04-15T18:52:08.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:08 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1281443974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:08.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:08 smithi159 ceph-mon[45336]: osdmap e909: 12 total, 12 up, 12 in 2024-04-15T18:52:09.364 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:09 smithi082 ceph-mon[44206]: pgmap v2368: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:09.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:09 smithi159 ceph-mon[45336]: pgmap v2368: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:09.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:09 smithi049 ceph-mon[40087]: pgmap v2368: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:09.982 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:09.983 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:52:09.983 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-04-15T18:52:09.983 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:10.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-15T18:52:10.909 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-15T18:52:11.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:11 smithi082 ceph-mon[44206]: pgmap v2369: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:11.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:11.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:11 smithi159 ceph-mon[45336]: pgmap v2369: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:11.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:11.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:11 smithi049 ceph-mon[40087]: pgmap v2369: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:11.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-15T18:52:11.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 5 1 2024-04-15T18:52:12.160 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:12.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:12 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:12.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:12 smithi049 ceph-mon[40087]: osdmap e910: 12 total, 12 up, 12 in 2024-04-15T18:52:12.374 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:12 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:12.374 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:12 smithi082 ceph-mon[44206]: osdmap e910: 12 total, 12 up, 12 in 2024-04-15T18:52:12.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:12 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1125095951' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-15T18:52:12.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:12 smithi159 ceph-mon[45336]: osdmap e910: 12 total, 12 up, 12 in 2024-04-15T18:52:12.622 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:12.622 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-04-15T18:52:12.622 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-04-15T18:52:12.622 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-04-15T18:52:12.622 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:13.258 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:13 smithi082 ceph-mon[44206]: pgmap v2371: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:13.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:13 smithi049 ceph-mon[40087]: pgmap v2371: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:13.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-15T18:52:13.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:13 smithi159 ceph-mon[45336]: pgmap v2371: 1 pgs: 1 active+clean; 577 KiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:13.548 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.b/config 2024-04-15T18:52:14.066 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-04-15T18:52:14.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:14.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:14.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:14.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-15T18:52:15.031 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:15.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:15 smithi082 ceph-mon[44206]: pgmap v2372: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:15.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:15 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:15.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:15 smithi082 ceph-mon[44206]: osdmap e911: 12 total, 12 up, 12 in 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:7974 9.1 e797/791 9:8a8d9b8c:::benchmark_data_smithi159_173188_object7973:head v 797'472, mlcod=797'470)", 2024-04-15T18:52:15.286 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:00.059885+0000", 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 375.23537723099997, 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.184842215, 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.059885+0000", 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.287 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.059885+0000", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.059886+0000", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4600000000000004e-07 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.059922+0000", 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5601e-05 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.288 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244032+0000", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.184110092 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244037+0000", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5230000000000003e-06 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244086+0000", 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8661e-05 2024-04-15T18:52:15.289 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244101+0000", 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5500000000000001e-05 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244649+0000", 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054783999999999996 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.290 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:00.244727+0000", 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8251999999999995e-05 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:55688 9.1 e805/791 9:8e075c20:::benchmark_data_smithi159_173188_object55687:head v 805'3456, mlcod=805'3453)", 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:46:32.302892+0000", 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 342.99237068399998, 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22428397799999999, 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.291 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.302892+0000", 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.302892+0000", 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.292 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.302893+0000", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.164e-06 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.302962+0000", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9543999999999997e-05 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.525165+0000", 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.22220241099999999 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.293 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.525174+0000", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8619999999999992e-06 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.525242+0000", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7838999999999999e-05 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.525265+0000", 2024-04-15T18:52:15.294 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3578999999999999e-05 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.527124+0000", 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001858478 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:46:32.527176+0000", 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2101000000000001e-05 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.295 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:124776 9.1 e830/791 9:8d607086:::benchmark_data_smithi159_173188_object16143:head v 830'7823, mlcod=830'7820)", 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:53.513754+0000", 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 261.78150849399998, 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.099879706999999998, 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.513754+0000", 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.296 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.513754+0000", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.513755+0000", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6200000000000006e-07 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.513761+0000", 2024-04-15T18:52:15.297 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4289999999999998e-06 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.609739+0000", 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.095977489999999999 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.609748+0000", 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.0059999999999998e-06 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.298 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.609848+0000", 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9897999999999998e-05 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.609871+0000", 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3200000000000001e-05 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.613581+0000", 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003710185 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.299 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:53.613634+0000", 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2537000000000002e-05 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:125067 9.1 e830/791 9:871a6daf:::benchmark_data_smithi159_173188_object16434:head v 830'7840, mlcod=830'7837)", 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:47:54.164979+0000", 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 261.13028379600001, 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.20320354400000001, 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.300 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.164979+0000", 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.164979+0000", 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.301 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.164980+0000", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.02e-06 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.164987+0000", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9940000000000003e-06 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.365222+0000", 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.200235196 2024-04-15T18:52:15.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.365227+0000", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4269999999999999e-06 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.365301+0000", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3987999999999997e-05 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.365317+0000", 2024-04-15T18:52:15.303 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5489999999999999e-05 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.368138+0000", 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0028213439999999999 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:47:54.368182+0000", 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4085000000000002e-05 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.304 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:146607 9.0 e836/821 9:0ac79120:::benchmark_data_smithi159_173188_object37974:head v 836'9223, mlcod=801'2369)", 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:29.817731+0000", 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.47753186, 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.093391627000000005, 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.305 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.817731+0000", 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.817731+0000", 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.817731+0000", 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5799999999999998e-07 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.306 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.817737+0000", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4569999999999996e-06 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.907209+0000", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.089471676999999999 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.907217+0000", 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4260000000000003e-06 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.307 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.907277+0000", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9871999999999999e-05 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.907305+0000", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8283999999999999e-05 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.911074+0000", 2024-04-15T18:52:15.308 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037686809999999999 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.911122+0000", 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8371999999999997e-05 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:146690 9.0 e836/821 9:0888a5b9:::benchmark_data_smithi159_173188_object38057:head v 836'9227, mlcod=801'2373)", 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:48:29.965847+0000", 2024-04-15T18:52:15.309 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 225.32941532800001, 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.096628599999999995, 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.965847+0000", 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.310 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.965847+0000", 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.965848+0000", 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1459999999999999e-06 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:29.965856+0000", 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0460000000000006e-06 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.311 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.059224+0000", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.093367427000000003 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.059230+0000", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.959e-06 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.059285+0000", 2024-04-15T18:52:15.312 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.5447e-05 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.059313+0000", 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8211000000000001e-05 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.062437+0000", 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031240590000000002 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.313 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:48:30.062476+0000", 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8305e-05 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:165374 9.1 e841/791 9:8a7c94a5:::benchmark_data_smithi159_173188_object56741:head v 841'10352, mlcod=841'10349)", 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:00.846957+0000", 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 194.448305554, 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.10399594600000001, 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.314 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.846957+0000", 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.846957+0000", 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.315 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.846958+0000", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0380000000000001e-06 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.846966+0000", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8150000000000006e-06 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.947420+0000", 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.100454417 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.316 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.947427+0000", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7159999999999997e-06 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.947485+0000", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8578000000000001e-05 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.947502+0000", 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6795000000000001e-05 2024-04-15T18:52:15.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.950917+0000", 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034146810000000001 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:00.950953+0000", 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.5905999999999998e-05 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.318 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:172885 9.1 e844/791 9:8e50881d:::benchmark_data_smithi159_173188_object64252:head v 844'10819, mlcod=844'10816)", 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:13.561897+0000", 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 181.733365784, 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.100689441, 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.561897+0000", 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.319 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.561897+0000", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.561897+0000", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.99e-07 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.561902+0000", 2024-04-15T18:52:15.320 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9640000000000002e-06 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.658999+0000", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.097096777999999995 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.659007+0000", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6389999999999992e-06 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.659063+0000", 2024-04-15T18:52:15.321 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6607000000000002e-05 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.659088+0000", 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4802000000000001e-05 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.662543+0000", 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034551349999999998 2024-04-15T18:52:15.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:13.662586+0000", 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.2817e-05 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:187014 9.1 e848/791 9:8b9bf739:::benchmark_data_smithi159_173188_object78381:head v 848'11720, mlcod=848'11717)", 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:36.958538+0000", 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 158.336723998, 2024-04-15T18:52:15.323 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.11708265499999999, 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:36.958538+0000", 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:36.958538+0000", 2024-04-15T18:52:15.324 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:36.958539+0000", 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.5499999999999997e-07 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:36.958546+0000", 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4309999999999999e-06 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.325 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.071909+0000", 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.113363667 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.071920+0000", 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0214000000000001e-05 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.071987+0000", 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7181999999999995e-05 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.326 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.072014+0000", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6978000000000002e-05 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.075587+0000", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003573138 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:37.075621+0000", 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.4190000000000003e-05 2024-04-15T18:52:15.327 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:200253 9.1 e853/791 9:8202d970:::benchmark_data_smithi159_173188_object91620:head v 853'12573, mlcod=853'12570)", 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:57.268548+0000", 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 138.02671403900001, 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.13841995600000001, 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.328 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.268548+0000", 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.268548+0000", 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.268550+0000", 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.823e-06 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.329 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.268560+0000", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.7929999999999992e-06 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.403392+0000", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.134832339 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.403403+0000", 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.045e-05 2024-04-15T18:52:15.330 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.403477+0000", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3771999999999998e-05 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.403497+0000", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0041e-05 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.406934+0000", 2024-04-15T18:52:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034377589999999999 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.406968+0000", 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3979000000000001e-05 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:200486 9.0 e853/821 9:046ec469:::benchmark_data_smithi159_173188_object91853:head v 853'12662, mlcod=816'5897)", 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:49:57.657163+0000", 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 137.63809946399999, 2024-04-15T18:52:15.332 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.10596657499999999, 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.657163+0000", 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.333 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.657163+0000", 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.657164+0000", 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.201e-06 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.657170+0000", 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3130000000000004e-06 2024-04-15T18:52:15.334 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.759459+0000", 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.102288771 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.759468+0000", 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4970000000000001e-06 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.335 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.759518+0000", 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0705000000000003e-05 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.759540+0000", 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1701000000000001e-05 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.763080+0000", 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035402469999999998 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.336 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:49:57.763130+0000", 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.914e-05 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:200963 9.1 e856/791 9:8e5be07c:::benchmark_data_smithi159_173188_object92330:head v 856'12626, mlcod=856'12624)", 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:00.439624+0000", 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.85563836, 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.15187864200000001, 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.337 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.439624+0000", 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.439624+0000", 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.338 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.439626+0000", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.545e-06 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.439636+0000", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0903999999999999e-05 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587939+0000", 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.14830284299999999 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.339 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587946+0000", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9330000000000004e-06 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587996+0000", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9421e-05 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.588012+0000", 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6597999999999999e-05 2024-04-15T18:52:15.340 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.591465+0000", 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003452826 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.591503+0000", 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7571999999999999e-05 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.341 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:200997 9.1 e856/791 9:8774002f:::benchmark_data_smithi159_173188_object92364:head v 856'12627, mlcod=856'12624)", 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:00.587908+0000", 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 134.70735389399999, 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17614518000000001, 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587908+0000", 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.342 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587908+0000", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587910+0000", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0780000000000001e-06 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.587918+0000", 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4800000000000001e-06 2024-04-15T18:52:15.343 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.760533+0000", 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.17261522700000001 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.760540+0000", 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6109999999999999e-06 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.344 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.760592+0000", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.2043000000000003e-05 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.760609+0000", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.6659000000000001e-05 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.764016+0000", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0034072410000000001 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.345 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:00.764054+0000", 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7840999999999999e-05 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:201811 9.0 e857/821 9:0cf2d6cb:::benchmark_data_smithi159_173188_object93178:head v 857'12767, mlcod=817'6263)", 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:02.047088+0000", 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 133.248174541, 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.097117785999999998, 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.346 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.047088+0000", 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.047088+0000", 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.347 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.047089+0000", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.114e-06 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.047098+0000", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8200000000000003e-06 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139640+0000", 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.092542652000000003 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.348 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139648+0000", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.0639999999999996e-06 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139699+0000", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1060000000000002e-05 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139720+0000", 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0945999999999999e-05 2024-04-15T18:52:15.349 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.144155+0000", 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044356220000000002 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.144206+0000", 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0507999999999998e-05 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.350 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:201875 9.0 e857/821 9:0ad0f11e:::benchmark_data_smithi159_173188_object93242:head v 857'12768, mlcod=817'6264)", 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:02.139610+0000", 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 133.155652046, 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.101925474, 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139610+0000", 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.351 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139610+0000", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139611+0000", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0869999999999999e-06 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.139620+0000", 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7460000000000006e-06 2024-04-15T18:52:15.352 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.236871+0000", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.097250989999999995 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.236879+0000", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3339999999999996e-06 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.236939+0000", 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0489999999999999e-05 2024-04-15T18:52:15.353 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.236967+0000", 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8007000000000001e-05 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.241498+0000", 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045309850000000004 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.354 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:02.241536+0000", 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.7834999999999999e-05 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:212022 9.0 e865/821 9:0ec3d922:::benchmark_data_smithi159_173188_object103389:head v 865'13413, mlcod=833'8812)", 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:18.405937+0000", 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.88932536, 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.087495748999999998, 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.355 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.405937+0000", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.405937+0000", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.405938+0000", 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.08e-06 2024-04-15T18:52:15.356 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.405946+0000", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8709999999999997e-06 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.488681+0000", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.082734905999999997 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.488696+0000", 2024-04-15T18:52:15.357 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5203e-05 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.488764+0000", 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8048999999999998e-05 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.488797+0000", 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2446000000000001e-05 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.358 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.493389+0000", 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0045927959999999997 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.493433+0000", 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3398e-05 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:212105 9.1 e865/791 9:893026f6:::benchmark_data_smithi159_173188_object103472:head v 865'13334, mlcod=865'13331)", 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:18.561480+0000", 2024-04-15T18:52:15.359 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.73378238799999, 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.068039279999999994, 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.561480+0000", 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.561480+0000", 2024-04-15T18:52:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.561481+0000", 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1200000000000001e-06 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.561489+0000", 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3489999999999992e-06 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.361 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.625701+0000", 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.064212004000000003 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.625710+0000", 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8769999999999999e-06 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.625765+0000", 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.4685000000000002e-05 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.362 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.625788+0000", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2665000000000001e-05 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.629468+0000", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003679959 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.629519+0000", 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1620999999999999e-05 2024-04-15T18:52:15.363 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:212237 9.0 e865/821 9:0deca2e5:::benchmark_data_smithi159_173188_object103604:head v 865'13428, mlcod=835'9006)", 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:18.734982+0000", 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.560280408, 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.062408144999999998, 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.364 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.734982+0000", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.734982+0000", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.734983+0000", 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.2699999999999999e-07 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.365 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.734988+0000", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9799999999999998e-06 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.792839+0000", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.057851726999999999 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.792844+0000", 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9509999999999997e-06 2024-04-15T18:52:15.366 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.792892+0000", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7830000000000001e-05 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.792916+0000", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3964999999999998e-05 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.797358+0000", 2024-04-15T18:52:15.367 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0044415330000000001 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.797390+0000", 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.2431999999999997e-05 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_repop(client.27025.0:212312 9.0 e865/821 9:00267135:::benchmark_data_smithi159_173188_object103679:head v 865'13432, mlcod=835'9019)", 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:50:18.825872+0000", 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 116.469390309, 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.057703928000000002, 2024-04-15T18:52:15.368 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.825872+0000", 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.825872+0000", 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.369 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.825874+0000", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4279999999999999e-06 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.825884+0000", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.0859999999999999e-05 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.879134+0000", 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.053249816999999998 2024-04-15T18:52:15.370 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.879143+0000", 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3930000000000008e-06 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.879208+0000", 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.5226000000000002e-05 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.371 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.879234+0000", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6183999999999999e-05 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.883531+0000", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.004296556 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:50:18.883576+0000", 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.5463999999999997e-05 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.372 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDScrubReserve(1.0,RELEASE,e:906,from:7,reservation_nonce:0)", 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-04-15T18:52:00.419606+0000", 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 14.875656331, 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000198937, 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "continuous": false, 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-04-15T18:52:15.373 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419606+0000", 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419606+0000", 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419606+0000", 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.37e-07 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.374 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419609+0000", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.367e-06 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419691+0000", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2082000000000005e-05 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419698+0000", 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6629999999999993e-06 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.375 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419777+0000", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8937999999999995e-05 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419791+0000", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.3689000000000001e-05 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-04-15T18:52:00.419805+0000", 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4061e-05 2024-04-15T18:52:15.376 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.377 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.377 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.377 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-04-15T18:52:15.377 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-15T18:52:15.377 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-15T18:52:15.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:15 smithi049 ceph-mon[40087]: pgmap v2372: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:15.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:15 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:15.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:15 smithi049 ceph-mon[40087]: osdmap e911: 12 total, 12 up, 12 in 2024-04-15T18:52:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:15 smithi159 ceph-mon[45336]: pgmap v2372: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:15 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/737827221' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:15.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:15 smithi159 ceph-mon[45336]: osdmap e911: 12 total, 12 up, 12 in 2024-04-15T18:52:15.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-04-15T18:52:16.254 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:52:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:16 smithi082 ceph-mon[44206]: osdmap e912: 12 total, 12 up, 12 in 2024-04-15T18:52:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:16 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:16.277 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:16 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:16.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:16.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:16.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:16 smithi049 ceph-mon[40087]: osdmap e912: 12 total, 12 up, 12 in 2024-04-15T18:52:16.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:16 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:16.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:16 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:16 smithi159 ceph-mon[45336]: osdmap e912: 12 total, 12 up, 12 in 2024-04-15T18:52:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:16 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:16.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:16 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:17.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:17 smithi159 ceph-mon[45336]: pgmap v2375: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:17.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:17 smithi159 ceph-mon[45336]: osdmap e913: 12 total, 12 up, 12 in 2024-04-15T18:52:17.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:17 smithi049 ceph-mon[40087]: pgmap v2375: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:17.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:17 smithi049 ceph-mon[40087]: osdmap e913: 12 total, 12 up, 12 in 2024-04-15T18:52:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:17 smithi082 ceph-mon[44206]: pgmap v2375: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:17.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:17 smithi082 ceph-mon[44206]: osdmap e913: 12 total, 12 up, 12 in 2024-04-15T18:52:18.026 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:52:18.026 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:52:18.026 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-04-15T18:52:18.026 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:52:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-15T18:52:18.336 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:18 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:18.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:18 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:18.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:18 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-15T18:52:18.736 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-04-15T18:52:18.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 6 1 2024-04-15T18:52:19.001 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:52:19.068 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:19.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:19 smithi049 ceph-mon[40087]: pgmap v2377: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:19.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:19 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:19.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:19 smithi049 ceph-mon[40087]: osdmap e914: 12 total, 12 up, 12 in 2024-04-15T18:52:19.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:19 smithi159 ceph-mon[45336]: pgmap v2377: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:19.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:19 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:19.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:19 smithi159 ceph-mon[45336]: osdmap e914: 12 total, 12 up, 12 in 2024-04-15T18:52:19.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:19 smithi082 ceph-mon[44206]: pgmap v2377: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:19.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:19 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2427362003' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-15T18:52:19.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:19 smithi082 ceph-mon[44206]: osdmap e914: 12 total, 12 up, 12 in 2024-04-15T18:52:20.681 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:52:20.681 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-04-15T18:52:20.681 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-04-15T18:52:20.681 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-04-15T18:52:20.681 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:52:21.091 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-04-15T18:52:21.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:21 smithi159 ceph-mon[45336]: pgmap v2379: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:21.381 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:21 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:21.408 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-04-15T18:52:21.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:21 smithi049 ceph-mon[40087]: pgmap v2379: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:21.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:21 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:21.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:21 smithi082 ceph-mon[44206]: pgmap v2379: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:21.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:21 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:21.669 INFO:teuthology.orchestra.run.smithi159.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.c/config 2024-04-15T18:52:21.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-15T18:52:22.169 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:22.213 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:22.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:22 smithi049 ceph-mon[40087]: osdmap e915: 12 total, 12 up, 12 in 2024-04-15T18:52:22.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:22.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:22 smithi159 ceph-mon[45336]: osdmap e915: 12 total, 12 up, 12 in 2024-04-15T18:52:22.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/983897874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:22.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:22 smithi082 ceph-mon[44206]: osdmap e915: 12 total, 12 up, 12 in 2024-04-15T18:52:23.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:23 smithi049 ceph-mon[40087]: pgmap v2381: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107193 9.5 9.66f964a5 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.233055+0000", 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 303.14889256999999, 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.970903827, 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107193 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233055+0000", 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233055+0000", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233056+0000", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.17e-06 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233129+0000", 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2589000000000006e-05 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233139+0000", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9990000000000003e-06 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233147+0000", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3550000000000005e-06 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233198+0000", 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1523000000000001e-05 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233207+0000", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6519999999999995e-06 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.203942+0000", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.97073532900000004 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.203959+0000", 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6209999999999999e-05 2024-04-15T18:52:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107196 9.5 9.d00421b5 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.233163+0000", 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 303.148784691, 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.97097403599999998, 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107196 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233163+0000", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233163+0000", 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233163+0000", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8799999999999998e-07 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233199+0000", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6124000000000002e-05 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233205+0000", 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3159999999999996e-06 2024-04-15T18:52:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233209+0000", 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6439999999999999e-06 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233233+0000", 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3669999999999999e-05 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.233236+0000", 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1370000000000002e-06 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204125+0000", 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.97088917600000002 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204137+0000", 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1581e-05 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107213 9.5 9.45254a25 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.735159+0000", 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 302.64678816700001, 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46902947900000003, 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107213 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735159+0000", 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735159+0000", 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735161+0000", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.629e-06 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735215+0000", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3990000000000003e-05 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735231+0000", 2024-04-15T18:52:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6395000000000001e-05 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735239+0000", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9249999999999995e-06 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735292+0000", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2450000000000001e-05 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.735298+0000", 2024-04-15T18:52:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4559999999999997e-06 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204181+0000", 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46888342599999999 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204189+0000", 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2080000000000003e-06 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107214 9.5 9.4e69ed85 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.766125+0000", 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 302.61582252699998, 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43811705000000001, 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107214 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766125+0000", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766125+0000", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766126+0000", 2024-04-15T18:52:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9400000000000004e-07 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766167+0000", 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.155e-05 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766179+0000", 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1233000000000001e-05 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766185+0000", 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.438e-06 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766242+0000", 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6743000000000002e-05 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.766252+0000", 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0626e-05 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204234+0000", 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.43798132400000001 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204242+0000", 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.242e-06 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:107215 9.5 9.a6425bc5 (undecoded) ondisk+write+known_if_redirected e820)", 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:20.851903+0000", 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 302.530044766, 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35237188899999999, 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 107215 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851903+0000", 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851903+0000", 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851904+0000", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7540000000000001e-06 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851952+0000", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7998999999999999e-05 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851967+0000", 2024-04-15T18:52:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4714e-05 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.851976+0000", 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6999999999999997e-06 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.852046+0000", 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0276000000000003e-05 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:20.852058+0000", 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2139e-05 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204266+0000", 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.35220760499999998 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:21.204274+0000", 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7020000000000007e-06 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(9.5 824/821 [PushOp(9:a22ab0eb:::benchmark_data_smithi159_173188_object965:head, version: 795'66, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:a22ab0eb:::benchmark_data_smithi159_173188_object965:head@795'66, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:47:25.282894+0000", 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 298.09905370000001, 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23861591900000001, 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.282894+0000", 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.282894+0000", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.282895+0000", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9800000000000002e-07 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.282939+0000", 2024-04-15T18:52:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4734999999999998e-05 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.519384+0000", 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23644482 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.519395+0000", 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0623e-05 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.519466+0000", 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-05 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.519477+0000", 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1420000000000001e-05 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:47:25.521510+0000", 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020325930000000001 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(9.9 832/821 [PushOp(9:96db27c6:::benchmark_data_smithi159_173188_object21858:head, version: 798'1370, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:96db27c6:::benchmark_data_smithi159_173188_object21858:head@798'1370, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:48:02.748208+0000", 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 260.63373916299997, 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23507151800000001, 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.748208+0000", 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.748208+0000", 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.748210+0000", 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9039999999999999e-06 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.748242+0000", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1596e-05 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.980416+0000", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.23217380000000001 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.980434+0000", 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8178000000000001e-05 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.980527+0000", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2930999999999995e-05 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.980540+0000", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:48:02.983280+0000", 2024-04-15T18:52:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027393399999999998 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(9.9 854/821 [PushOp(9:958e27af:::benchmark_data_smithi159_173188_object94074:head, version: 817'5864, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:958e27af:::benchmark_data_smithi159_173188_object94074:head@817'5864, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:58.414388+0000", 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 144.967559032, 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34341197800000001, 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.414388+0000", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.414388+0000", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.414389+0000", 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-04-15T18:52:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.414417+0000", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.756e-05 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.754488+0000", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34007157599999999 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.754501+0000", 2024-04-15T18:52:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2227e-05 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.754591+0000", 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0838e-05 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.754606+0000", 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4644000000000001e-05 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:58.757800+0000", 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031942440000000002 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200814 9.5 9:a9a847ae:::benchmark_data_smithi159_173188_object92181:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e855)", 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:49:59.971757+0000", 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.41019073999999, 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.85079943599999996, 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200814 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971757+0000", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971757+0000", 2024-04-15T18:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971758+0000", 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7150000000000001e-06 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971764+0000", 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9680000000000003e-06 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971778+0000", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3380999999999999e-05 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971787+0000", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8179999999999993e-06 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971901+0000", 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011441 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:49:59.971912+0000", 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1396000000000001e-05 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.813458+0000", 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.84154562499999996 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.813592+0000", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013433999999999999 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.813738+0000", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014547099999999999 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822376+0000", 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.008638583 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822396+0000", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9185999999999999e-05 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822448+0000", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2299999999999997e-05 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822507+0000", 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8940999999999997e-05 2024-04-15T18:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822556+0000", 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9302000000000003e-05 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200835 9.5 9:a49a6a96:::benchmark_data_smithi159_173188_object92202:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.234346+0000", 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.14760124700001, 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.60050618300000003, 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200835 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234346+0000", 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234346+0000", 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234348+0000", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9149999999999999e-06 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234361+0000", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.278e-05 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234377+0000", 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6351999999999999e-05 2024-04-15T18:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234387+0000", 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.028e-05 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234451+0000", 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3435999999999994e-05 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.234460+0000", 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3589999999999997e-06 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.818129+0000", 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.58366878300000002 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.818238+0000", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109413 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.818342+0000", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103446 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822622+0000", 2024-04-15T18:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042799919999999998 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834611+0000", 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011988896000000001 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834741+0000", 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000130042 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834801+0000", 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0393999999999998e-05 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834852+0000", 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1094999999999998e-05 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200858 9.5 9:a34e2f84:::benchmark_data_smithi159_173188_object92225:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.270178+0000", 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.111768998, 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.586742557, 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200858 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270178+0000", 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270178+0000", 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270180+0000", 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8250000000000001e-06 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270188+0000", 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3849999999999996e-06 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270203+0000", 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5370999999999999e-05 2024-04-15T18:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270211+0000", 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5229999999999995e-06 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270273+0000", 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1624999999999994e-05 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.270282+0000", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0089999999999996e-06 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822747+0000", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.55246488599999999 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.822874+0000", 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012676199999999999 2024-04-15T18:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.823010+0000", 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013580600000000001 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834672+0000", 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011662216 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834697+0000", 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4839000000000001e-05 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856818+0000", 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022121023 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856873+0000", 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5538000000000003e-05 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856921+0000", 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7749e-05 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200917 9.5 9:ab57ebd7:::benchmark_data_smithi159_173188_object92284:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.359460+0000", 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.02248722900001, 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.50605150700000001, 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200917 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359460+0000", 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359460+0000", 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359461+0000", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3009999999999999e-06 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359468+0000", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9739999999999996e-06 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359482+0000", 2024-04-15T18:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3847e-05 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359491+0000", 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7460000000000006e-06 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359544+0000", 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3483999999999998e-05 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.359551+0000", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3969999999999999e-06 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.826962+0000", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46741135499999997 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.827085+0000", 2024-04-15T18:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012270900000000001 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.827216+0000", 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000131285 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834702+0000", 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074855529999999998 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834898+0000", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000196014 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865412+0000", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.030514030000000001 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865470+0000", 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8328000000000001e-05 2024-04-15T18:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865512+0000", 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1483999999999998e-05 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200926 9.5 9:af429fd8:::benchmark_data_smithi159_173188_object92293:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.363924+0000", 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.01802339899999, 2024-04-15T18:52:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.50173679100000002, 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200926 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363924+0000", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363924+0000", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363925+0000", 2024-04-15T18:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3149999999999999e-06 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363933+0000", 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9219999999999997e-06 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363944+0000", 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1275e-05 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363950+0000", 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2440000000000001e-06 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.363995+0000", 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5096999999999999e-05 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.364001+0000", 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5270000000000004e-06 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.830382+0000", 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.466381037 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.830510+0000", 2024-04-15T18:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012743699999999999 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.830639+0000", 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012889299999999999 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834944+0000", 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043048959999999999 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.841974+0000", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070300600000000003 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865584+0000", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.023610407999999999 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865625+0000", 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1161000000000001e-05 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865661+0000", 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5519000000000003e-05 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200932 9.5 9:a8820267:::benchmark_data_smithi159_173188_object92299:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.375268+0000", 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.006678951, 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.49053707499999999, 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200932 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375268+0000", 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375268+0000", 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375270+0000", 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.95e-06 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375276+0000", 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9769999999999997e-06 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375290+0000", 2024-04-15T18:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3322999999999999e-05 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375299+0000", 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1959999999999997e-06 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375367+0000", 2024-04-15T18:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8466999999999993e-05 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.375375+0000", 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5140000000000004e-06 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.834978+0000", 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45960279799999998 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.835111+0000", 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013384000000000001 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.835256+0000", 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014469999999999999 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845404+0000", 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010147661000000001 2024-04-15T18:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845420+0000", 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5871000000000001e-05 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865733+0000", 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020313332999999999 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865774+0000", 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1418999999999998e-05 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865805+0000", 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1025999999999997e-05 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200935 9.5 9:a0efeee6:::benchmark_data_smithi159_173188_object92302:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.378402+0000", 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.003545699, 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.48755444199999998, 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200935 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378402+0000", 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378402+0000", 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378404+0000", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4279999999999999e-06 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378412+0000", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6930000000000007e-06 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378428+0000", 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5865000000000001e-05 2024-04-15T18:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.378436+0000", 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5539999999999998e-06 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.380462+0000", 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002025312 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.380470+0000", 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4649999999999994e-06 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.838563+0000", 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45809284500000003 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.838676+0000", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113152 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.838800+0000", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012404699999999999 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845445+0000", 2024-04-15T18:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0066453090000000003 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.849362+0000", 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039169139999999996 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865877+0000", 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016515019999999998 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865920+0000", 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3001999999999999e-05 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865956+0000", 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5836e-05 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200938 9.5 9:a0056b13:::benchmark_data_smithi159_173188_object92305:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.387409+0000", 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.994538038, 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.47869982900000002, 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200938 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387409+0000", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387409+0000", 2024-04-15T18:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387411+0000", 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6020000000000001e-06 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387418+0000", 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6590000000000001e-06 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387430+0000", 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2819999999999999e-05 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387438+0000", 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9920000000000007e-06 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387503+0000", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5074999999999996e-05 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.387514+0000", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0244999999999999e-05 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.842016+0000", 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45450274499999999 2024-04-15T18:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.842127+0000", 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000110093 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.842241+0000", 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114128 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856771+0000", 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014529996 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865550+0000", 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087792019999999998 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866030+0000", 2024-04-15T18:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048038699999999998 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866074+0000", 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3288999999999998e-05 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866109+0000", 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5596000000000003e-05 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200946 9.5 9:a75e68c0:::benchmark_data_smithi159_173188_object92313:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.400743+0000", 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.98120391, 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.46547971900000001, 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200946 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400743+0000", 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400743+0000", 2024-04-15T18:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400745+0000", 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4300000000000001e-06 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400751+0000", 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5320000000000003e-06 2024-04-15T18:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400765+0000", 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3726e-05 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400773+0000", 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2099999999999993e-06 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400830+0000", 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6379999999999999e-05 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.400839+0000", 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9800000000000004e-06 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845472+0000", 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.44463319099999998 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845581+0000", 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000109165 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.845657+0000", 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5728000000000004e-05 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856930+0000", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011273101000000001 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865695+0000", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087649330000000008 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866148+0000", 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00045273500000000001 2024-04-15T18:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866191+0000", 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3562000000000001e-05 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866223+0000", 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2045999999999998e-05 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200966 9.5 9:ad91b4a5:::benchmark_data_smithi159_173188_object92333:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.447404+0000", 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.934543373, 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.41892621400000002, 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200966 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447404+0000", 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447404+0000", 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447405+0000", 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.471e-06 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447412+0000", 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6950000000000003e-06 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447425+0000", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2668e-05 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447433+0000", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0309999999999998e-06 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447486+0000", 2024-04-15T18:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3059999999999997e-05 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.447493+0000", 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9990000000000002e-06 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.849410+0000", 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.401917101 2024-04-15T18:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.849539+0000", 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000128486 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.849667+0000", 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012841099999999999 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856935+0000", 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072677590000000004 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865842+0000", 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0089077719999999996 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866260+0000", 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00041781899999999999 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866300+0000", 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9807e-05 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.866330+0000", 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0134999999999999e-05 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200983 9.5 9:ac364f19:::benchmark_data_smithi159_173188_object92350:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.534281+0000", 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.847666205, 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34661932000000001, 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200983 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534281+0000", 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534281+0000", 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534283+0000", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4410000000000001e-06 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534290+0000", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1980000000000004e-06 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534303+0000", 2024-04-15T18:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3699999999999999e-05 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534313+0000", 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.679e-06 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534363+0000", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999e-05 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.534369+0000", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6930000000000004e-06 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.853226+0000", 2024-04-15T18:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.31885738499999999 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.853345+0000", 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119182 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.853449+0000", 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103823 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.865995+0000", 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012546170000000001 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.874615+0000", 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086191870000000004 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.880797+0000", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061820809999999999 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.880851+0000", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3962000000000003e-05 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.880900+0000", 2024-04-15T18:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9820000000000001e-05 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.27025.0:200994 9.5 9:a90d540b:::benchmark_data_smithi159_173188_object92361:head [delete] snapc 0=[] ondisk+write+known_if_redirected e856)", 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-04-15T18:50:00.570977+0000", 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 142.81097080800001, 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.31011555000000002, 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-04-15T18:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.27025", 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4042619606", 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 200994 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.570977+0000", 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.570977+0000", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.570978+0000", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.159e-06 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.570993+0000", 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5014000000000001e-05 2024-04-15T18:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.571005+0000", 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2017e-05 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.571011+0000", 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6409999999999996e-06 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.571065+0000", 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3591000000000002e-05 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for peered", 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.571073+0000", 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4670000000000004e-06 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-04-15T18:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.856959+0000", 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.28588598999999998 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.857075+0000", 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000115602 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5,7", 2024-04-15T18:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.857201+0000", 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012617300000000001 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.874659+0000", 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017457883 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.874670+0000", 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1168999999999999e-05 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.881010+0000", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063397560000000002 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.881055+0000", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5226999999999999e-05 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-04-15T18:50:00.881092+0000", 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860999999999998e-05 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-04-15T18:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-04-15T18:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-04-15T18:52:23.507 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:23 smithi159 ceph-mon[45336]: pgmap v2381: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:23.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:23 smithi082 ceph-mon[44206]: pgmap v2381: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:24.107 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-15T18:52:24.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:24 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:24.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:24 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:24.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:24 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-15T18:52:24.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd reweight 11 1 2024-04-15T18:52:25.157 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:25.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:25 smithi049 ceph-mon[40087]: pgmap v2382: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:25.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:25.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:25 smithi049 ceph-mon[40087]: osdmap e916: 12 total, 12 up, 12 in 2024-04-15T18:52:25.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:25 smithi159 ceph-mon[45336]: pgmap v2382: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:25.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:25.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:25 smithi159 ceph-mon[45336]: osdmap e916: 12 total, 12 up, 12 in 2024-04-15T18:52:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:25 smithi082 ceph-mon[44206]: pgmap v2382: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3178350338' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-15T18:52:25.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:25 smithi082 ceph-mon[44206]: osdmap e916: 12 total, 12 up, 12 in 2024-04-15T18:52:27.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:27 smithi159 ceph-mon[45336]: pgmap v2384: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:27.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:27 smithi049 ceph-mon[40087]: pgmap v2384: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:27.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:27 smithi082 ceph-mon[44206]: pgmap v2384: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:28.118 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.11 to 1 (10000) 2024-04-15T18:52:28.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:28 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:28 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:28 smithi159 ceph-mon[45336]: osdmap e917: 12 total, 12 up, 12 in 2024-04-15T18:52:28.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:28 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:28.463 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:28 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:28.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:28 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:28.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:28 smithi049 ceph-mon[40087]: osdmap e917: 12 total, 12 up, 12 in 2024-04-15T18:52:28.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:28 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:28.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:28 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:28.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:28 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:28.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:28 smithi082 ceph-mon[44206]: osdmap e917: 12 total, 12 up, 12 in 2024-04-15T18:52:28.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:28 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:28.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd primary-affinity 11 1 2024-04-15T18:52:29.062 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:29.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:29 smithi159 ceph-mon[45336]: pgmap v2386: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:29.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:29 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:29.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:29 smithi159 ceph-mon[45336]: osdmap e918: 12 total, 12 up, 12 in 2024-04-15T18:52:29.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:29 smithi049 ceph-mon[40087]: pgmap v2386: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:29.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:29 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:29.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:29 smithi049 ceph-mon[40087]: osdmap e918: 12 total, 12 up, 12 in 2024-04-15T18:52:29.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:29 smithi082 ceph-mon[44206]: pgmap v2386: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:29.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:29 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/332640074' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:29.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:29 smithi082 ceph-mon[44206]: osdmap e918: 12 total, 12 up, 12 in 2024-04-15T18:52:30.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:30 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:30.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:30 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:30.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:30 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:30.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:30 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:30 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:30.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:30 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:31.129 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-15T18:52:31.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:31 smithi159 ceph-mon[45336]: pgmap v2388: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:31.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:31 smithi159 ceph-mon[45336]: osdmap e919: 12 total, 12 up, 12 in 2024-04-15T18:52:31.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:31 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:31.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:31 smithi049 ceph-mon[40087]: pgmap v2388: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:31.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:31 smithi049 ceph-mon[40087]: osdmap e919: 12 total, 12 up, 12 in 2024-04-15T18:52:31.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:31 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:31 smithi082 ceph-mon[44206]: pgmap v2388: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:31 smithi082 ceph-mon[44206]: osdmap e919: 12 total, 12 up, 12 in 2024-04-15T18:52:31.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:31 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-04-15T18:52:31.854 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-15T18:52:31.854 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-15T18:52:31.854 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-15T18:52:31.854 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-15T18:52:31.855 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T18:52:31.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:52:32.138 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:32.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:32.395 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:32 smithi049 ceph-mon[40087]: osdmap e920: 12 total, 12 up, 12 in 2024-04-15T18:52:32.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:32.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:32 smithi159 ceph-mon[45336]: osdmap e920: 12 total, 12 up, 12 in 2024-04-15T18:52:32.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4200402610' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-04-15T18:52:32.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:32 smithi082 ceph-mon[44206]: osdmap e920: 12 total, 12 up, 12 in 2024-04-15T18:52:33.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:33 smithi159 ceph-mon[45336]: pgmap v2391: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:33.428 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:33 smithi049 ceph-mon[40087]: pgmap v2391: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:33.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:33 smithi082 ceph-mon[44206]: pgmap v2391: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:34.179 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:52:34.179 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":920,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:52:31.134606+0000","last_up_change":"2024-04-15T18:46:33.707597+0000","last_in_change":"2024-04-15T18:47:20.193442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":671,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":868,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":307,"up_from":361,"up_thru":778,"down_at":308,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6819","nonce":2410603388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6821","nonce":2410603388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6825","nonce":2410603388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6823","nonce":2410603388}]},"public_addr":"172.21.15.49:6819/2410603388","cluster_addr":"172.21.15.49:6821/2410603388","heartbeat_back_addr":"172.21.15.49:6825/2410603388","heartbeat_front_addr":"172.21.15.49:6823/2410603388","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":177,"up_from":226,"up_thru":862,"down_at":178,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6827","nonce":1036848877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6829","nonce":1036848877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6833","nonce":1036848877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6831","nonce":1036848877}]},"public_addr":"172.21.15.49:6827/1036848877","cluster_addr":"172.21.15.49:6829/1036848877","heartbeat_back_addr":"172.21.15.49:6833/1036848877","heartbeat_front_addr":"172.21.15.49:6831/1036848877","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":363,"up_from":376,"up_thru":791,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6809","nonce":1630882481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6811","nonce":1630882481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6815","nonce":1630882481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6813","nonce":1630882481}]},"public_addr":"172.21.15.82:6809/1630882481","cluster_addr":"172.21.15.82:6811/1630882481","heartbeat_back_addr":"172.21.15.82:6815/1630882481","heartbeat_front_addr":"172.21.15.82:6813/1630882481","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":47,"last_clean_end":561,"up_from":631,"up_thru":676,"down_at":562,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6817","nonce":256106271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6819","nonce":256106271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6823","nonce":256106271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6821","nonce":256106271}]},"public_addr":"172.21.15.82:6817/256106271","cluster_addr":"172.21.15.82:6819/256106271","heartbeat_back_addr":"172.21.15.82:6823/256106271","heartbeat_front_addr":"172.21.15.82:6821/256106271","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":875,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":870,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":870,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":94,"last_clean_end":733,"up_from":807,"up_thru":863,"down_at":734,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6825","nonce":905169792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6827","nonce":905169792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6831","nonce":905169792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6829","nonce":905169792}]},"public_addr":"172.21.15.159:6825/905169792","cluster_addr":"172.21.15.159:6827/905169792","heartbeat_back_addr":"172.21.15.159:6831/905169792","heartbeat_front_addr":"172.21.15.159:6829/905169792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:21.074852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:05.801984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:15.550167+0000","dead_epoch":307},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:18.294307+0000","dead_epoch":177},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:45.015610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:20.365233+0000","dead_epoch":363},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:26.879258+0000","dead_epoch":561},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:09.617893+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:14.903121+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:56.566946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:29.502745+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:57.583574+0000","dead_epoch":733}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:52:34.788 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T18:52:34.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T18:52:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:35.395 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:35 smithi159 ceph-mon[45336]: pgmap v2392: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:35.396 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:35 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/546614615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:35.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:35 smithi049 ceph-mon[40087]: pgmap v2392: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:35.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:35 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/546614615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:35.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:35 smithi082 ceph-mon[44206]: pgmap v2392: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:35.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:35 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/546614615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:52:37.089 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":921,"fsid":"22c655ba-fb52-11ee-bc8f-c7b262605968","created":"2024-04-15T18:02:11.588977+0000","modified":"2024-04-15T18:52:36.150673+0000","last_up_change":"2024-04-15T18:46:33.707597+0000","last_in_change":"2024-04-15T18:47:20.193442+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T18:05:43.035801+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c6740db9-c8a9-4083-9ef3-6ac938f861a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":671,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6803","nonce":306402154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6805","nonce":306402154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6809","nonce":306402154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":306402154},{"type":"v1","addr":"172.21.15.49:6807","nonce":306402154}]},"public_addr":"172.21.15.49:6803/306402154","cluster_addr":"172.21.15.49:6805/306402154","heartbeat_back_addr":"172.21.15.49:6809/306402154","heartbeat_front_addr":"172.21.15.49:6807/306402154","state":["exists","up"]},{"osd":1,"uuid":"a5f0a887-6af8-4537-b37c-3d9ed493b595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":868,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6811","nonce":1404755046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6813","nonce":1404755046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6817","nonce":1404755046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1404755046},{"type":"v1","addr":"172.21.15.49:6815","nonce":1404755046}]},"public_addr":"172.21.15.49:6811/1404755046","cluster_addr":"172.21.15.49:6813/1404755046","heartbeat_back_addr":"172.21.15.49:6817/1404755046","heartbeat_front_addr":"172.21.15.49:6815/1404755046","state":["exists","up"]},{"osd":2,"uuid":"57bad304-da93-49aa-9320-13cf3d1ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":307,"up_from":361,"up_thru":778,"down_at":308,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6819","nonce":2410603388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6821","nonce":2410603388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6825","nonce":2410603388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2410603388},{"type":"v1","addr":"172.21.15.49:6823","nonce":2410603388}]},"public_addr":"172.21.15.49:6819/2410603388","cluster_addr":"172.21.15.49:6821/2410603388","heartbeat_back_addr":"172.21.15.49:6825/2410603388","heartbeat_front_addr":"172.21.15.49:6823/2410603388","state":["exists","up"]},{"osd":3,"uuid":"8db13188-c0af-4b82-9333-e0659a026aa2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":177,"up_from":226,"up_thru":862,"down_at":178,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6827","nonce":1036848877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6829","nonce":1036848877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6833","nonce":1036848877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1036848877},{"type":"v1","addr":"172.21.15.49:6831","nonce":1036848877}]},"public_addr":"172.21.15.49:6827/1036848877","cluster_addr":"172.21.15.49:6829/1036848877","heartbeat_back_addr":"172.21.15.49:6833/1036848877","heartbeat_front_addr":"172.21.15.49:6831/1036848877","state":["exists","up"]},{"osd":4,"uuid":"6f66dfc2-7e1a-4545-81f8-246ff27e5928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6801","nonce":1223630649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6803","nonce":1223630649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6807","nonce":1223630649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1223630649},{"type":"v1","addr":"172.21.15.82:6805","nonce":1223630649}]},"public_addr":"172.21.15.82:6801/1223630649","cluster_addr":"172.21.15.82:6803/1223630649","heartbeat_back_addr":"172.21.15.82:6807/1223630649","heartbeat_front_addr":"172.21.15.82:6805/1223630649","state":["exists","up"]},{"osd":5,"uuid":"fafc8df6-bd72-4306-9b73-7ce797f5744a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":363,"up_from":376,"up_thru":791,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6809","nonce":1630882481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6811","nonce":1630882481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6815","nonce":1630882481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":1630882481},{"type":"v1","addr":"172.21.15.82:6813","nonce":1630882481}]},"public_addr":"172.21.15.82:6809/1630882481","cluster_addr":"172.21.15.82:6811/1630882481","heartbeat_back_addr":"172.21.15.82:6815/1630882481","heartbeat_front_addr":"172.21.15.82:6813/1630882481","state":["exists","up"]},{"osd":6,"uuid":"2ac61adb-a5b8-4e27-a648-756453c35783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":47,"last_clean_end":561,"up_from":631,"up_thru":676,"down_at":562,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6817","nonce":256106271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6819","nonce":256106271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6823","nonce":256106271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":256106271},{"type":"v1","addr":"172.21.15.82:6821","nonce":256106271}]},"public_addr":"172.21.15.82:6817/256106271","cluster_addr":"172.21.15.82:6819/256106271","heartbeat_back_addr":"172.21.15.82:6823/256106271","heartbeat_front_addr":"172.21.15.82:6821/256106271","state":["exists","up"]},{"osd":7,"uuid":"a8940a14-d060-46ed-aeb9-d5c43ff421c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":875,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6825","nonce":457966805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6827","nonce":457966805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6831","nonce":457966805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":457966805},{"type":"v1","addr":"172.21.15.82:6829","nonce":457966805}]},"public_addr":"172.21.15.82:6825/457966805","cluster_addr":"172.21.15.82:6827/457966805","heartbeat_back_addr":"172.21.15.82:6831/457966805","heartbeat_front_addr":"172.21.15.82:6829/457966805","state":["exists","up"]},{"osd":8,"uuid":"07798de8-8c74-4948-8860-a3e8c07d7e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6801","nonce":403815130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6803","nonce":403815130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6807","nonce":403815130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":403815130},{"type":"v1","addr":"172.21.15.159:6805","nonce":403815130}]},"public_addr":"172.21.15.159:6801/403815130","cluster_addr":"172.21.15.159:6803/403815130","heartbeat_back_addr":"172.21.15.159:6807/403815130","heartbeat_front_addr":"172.21.15.159:6805/403815130","state":["exists","up"]},{"osd":9,"uuid":"43cab3c4-4f27-44ba-b8a4-0f6909e3001d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":870,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6809","nonce":1558317233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6811","nonce":1558317233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6815","nonce":1558317233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1558317233},{"type":"v1","addr":"172.21.15.159:6813","nonce":1558317233}]},"public_addr":"172.21.15.159:6809/1558317233","cluster_addr":"172.21.15.159:6811/1558317233","heartbeat_back_addr":"172.21.15.159:6815/1558317233","heartbeat_front_addr":"172.21.15.159:6813/1558317233","state":["exists","up"]},{"osd":10,"uuid":"e9731146-d494-4a3c-a205-37064c685816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":870,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6817","nonce":2653197187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6818","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6819","nonce":2653197187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6822","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6823","nonce":2653197187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6820","nonce":2653197187},{"type":"v1","addr":"172.21.15.159:6821","nonce":2653197187}]},"public_addr":"172.21.15.159:6817/2653197187","cluster_addr":"172.21.15.159:6819/2653197187","heartbeat_back_addr":"172.21.15.159:6823/2653197187","heartbeat_front_addr":"172.21.15.159:6821/2653197187","state":["exists","up"]},{"osd":11,"uuid":"f8de423a-9e36-45ae-97e7-0384858a8118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":94,"last_clean_end":733,"up_from":807,"up_thru":863,"down_at":734,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6824","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6825","nonce":905169792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6826","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6827","nonce":905169792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6830","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6831","nonce":905169792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6828","nonce":905169792},{"type":"v1","addr":"172.21.15.159:6829","nonce":905169792}]},"public_addr":"172.21.15.159:6825/905169792","cluster_addr":"172.21.15.159:6827/905169792","heartbeat_back_addr":"172.21.15.159:6831/905169792","heartbeat_front_addr":"172.21.15.159:6829/905169792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:21.074852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:05.801984+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:15.550167+0000","dead_epoch":307},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:18.294307+0000","dead_epoch":177},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:45.015610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:20.365233+0000","dead_epoch":363},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:26.879258+0000","dead_epoch":561},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:36.065406+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:14.903121+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:56.566946+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:52:29.502745+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T18:51:57.583574+0000","dead_epoch":733}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3052240311":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/2417929619":"2024-04-16T18:03:08.925868+0000","172.21.15.49:0/769044034":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6800/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/1867631398":"2024-04-16T18:03:08.925868+0000","172.21.15.49:6801/3639973481":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/2669325064":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/530504890":"2024-04-16T18:02:40.629638+0000","172.21.15.49:0/453126905":"2024-04-16T18:02:40.629638+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T18:52:37.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:37 smithi049 ceph-mon[40087]: pgmap v2393: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:37.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:37 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:37.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:37 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:37.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:37 smithi049 ceph-mon[40087]: osdmap e921: 12 total, 12 up, 12 in 2024-04-15T18:52:37.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:37 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3097736150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:37.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:37 smithi082 ceph-mon[44206]: pgmap v2393: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:37.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:37 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:37.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:37 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:37.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:37 smithi082 ceph-mon[44206]: osdmap e921: 12 total, 12 up, 12 in 2024-04-15T18:52:37.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:37 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3097736150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:37.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:37 smithi159 ceph-mon[45336]: pgmap v2393: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:37.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:37 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:37.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:37 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:37.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:37 smithi159 ceph-mon[45336]: osdmap e921: 12 total, 12 up, 12 in 2024-04-15T18:52:37.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:37 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3097736150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T18:52:37.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T18:52:37.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T18:52:37.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T18:52:37.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T18:52:37.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T18:52:37.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T18:52:37.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-15T18:52:37.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-15T18:52:37.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-15T18:52:37.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-15T18:52:37.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-15T18:52:37.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-15T18:52:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.682 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.688 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.688 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.699 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.744 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.792 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.792 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:38.807 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:39.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:39 smithi049 ceph-mon[40087]: pgmap v2395: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:39.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:39 smithi082 ceph-mon[44206]: pgmap v2395: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:39.534 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:39.535 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:52:39.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:39 smithi159 ceph-mon[45336]: pgmap v2395: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:41.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:41 smithi049 ceph-mon[40087]: pgmap v2396: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:41.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:41 smithi082 ceph-mon[44206]: pgmap v2396: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:41.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:41 smithi159 ceph-mon[45336]: pgmap v2396: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:43 smithi049 ceph-mon[40087]: pgmap v2397: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:43.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:43 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:43.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:43 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:43.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:43 smithi082 ceph-mon[44206]: pgmap v2397: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:43.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:43 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:43.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:43 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:43.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:43 smithi159 ceph-mon[45336]: pgmap v2397: 1 pgs: 1 active+clean; 577 KiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:43.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:43 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:43.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:43 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:44.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:44 smithi049 ceph-mon[40087]: osdmap e922: 12 total, 12 up, 12 in 2024-04-15T18:52:44.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:44 smithi082 ceph-mon[44206]: osdmap e922: 12 total, 12 up, 12 in 2024-04-15T18:52:44.645 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:44 smithi159 ceph-mon[45336]: osdmap e922: 12 total, 12 up, 12 in 2024-04-15T18:52:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:45 smithi049 ceph-mon[40087]: pgmap v2399: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:45 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:45.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:45 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:45 smithi082 ceph-mon[44206]: pgmap v2399: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:45.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:45 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:45.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:45 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:45 smithi159 ceph-mon[45336]: pgmap v2399: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:45 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:45.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:45 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:46.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:46 smithi049 ceph-mon[40087]: osdmap e923: 12 total, 12 up, 12 in 2024-04-15T18:52:46.450 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:46 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:52:46.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:46 smithi082 ceph-mon[44206]: osdmap e923: 12 total, 12 up, 12 in 2024-04-15T18:52:46.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:46 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:52:46.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:46 smithi159 ceph-mon[45336]: osdmap e923: 12 total, 12 up, 12 in 2024-04-15T18:52:46.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:46 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:52:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:47 smithi082 ceph-mon[44206]: pgmap v2401: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:52:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:52:47.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:47 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:52:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:47 smithi159 ceph-mon[45336]: pgmap v2401: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:52:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:52:47.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:47 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:52:47.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:47 smithi049 ceph-mon[40087]: pgmap v2401: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:47.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:52:47.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:52:47.653 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:47 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:52:49.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:49 smithi049 ceph-mon[40087]: pgmap v2402: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:49.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:49 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:49.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:49 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:49.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:49 smithi049 ceph-mon[40087]: osdmap e924: 12 total, 12 up, 12 in 2024-04-15T18:52:49.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:49 smithi082 ceph-mon[44206]: pgmap v2402: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:49.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:49 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:49.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:49 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:49.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:49 smithi082 ceph-mon[44206]: osdmap e924: 12 total, 12 up, 12 in 2024-04-15T18:52:49.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:49 smithi159 ceph-mon[45336]: pgmap v2402: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:49.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:49 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:49.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:49 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:49.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:49 smithi159 ceph-mon[45336]: osdmap e924: 12 total, 12 up, 12 in 2024-04-15T18:52:51.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:51 smithi049 ceph-mon[40087]: pgmap v2404: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:51.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:51 smithi082 ceph-mon[44206]: pgmap v2404: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:51.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:51 smithi159 ceph-mon[45336]: pgmap v2404: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:52.143 INFO:teuthology.orchestra.run.smithi049.stdout:2710124363966 2024-04-15T18:52:52.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-15T18:52:53.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:53 smithi049 ceph-mon[40087]: pgmap v2405: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:53.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:53 smithi082 ceph-mon[44206]: pgmap v2405: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:53.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:53 smithi159 ceph-mon[45336]: pgmap v2405: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:55.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:55 smithi082 ceph-mon[44206]: pgmap v2406: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:55.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:55 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:52:55.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:55 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:52:55.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:55 smithi082 ceph-mon[44206]: osdmap e925: 12 total, 12 up, 12 in 2024-04-15T18:52:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:55 smithi159 ceph-mon[45336]: pgmap v2406: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:55 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:52:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:55 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:52:55.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:55 smithi159 ceph-mon[45336]: osdmap e925: 12 total, 12 up, 12 in 2024-04-15T18:52:55.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:55 smithi049 ceph-mon[40087]: pgmap v2406: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:55.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:55 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:52:55.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:55 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:52:55.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:55 smithi049 ceph-mon[40087]: osdmap e925: 12 total, 12 up, 12 in 2024-04-15T18:52:56.716 INFO:teuthology.orchestra.run.smithi049.stdout:1614907703631 2024-04-15T18:52:56.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T18:52:57.015 INFO:teuthology.orchestra.run.smithi049.stdout:34359738948 2024-04-15T18:52:57.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T18:52:57.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:57 smithi049 ceph-mon[40087]: pgmap v2408: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:57.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:57 smithi082 ceph-mon[44206]: pgmap v2408: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:57.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:57 smithi159 ceph-mon[45336]: pgmap v2408: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:57.881 INFO:teuthology.orchestra.run.smithi049.stdout:279172874780 2024-04-15T18:52:57.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-15T18:52:58.768 INFO:teuthology.orchestra.run.smithi049.stdout:55834575427 2024-04-15T18:52:58.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T18:52:59.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:52:59 smithi049 ceph-mon[40087]: pgmap v2409: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:59.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:52:59 smithi082 ceph-mon[44206]: pgmap v2409: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:52:59.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:52:59 smithi159 ceph-mon[45336]: pgmap v2409: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:00.331 INFO:teuthology.orchestra.run.smithi049.stdout:369367187989 2024-04-15T18:53:00.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-15T18:53:01.464 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:01 smithi049 ceph-mon[40087]: pgmap v2410: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:01.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:01 smithi082 ceph-mon[44206]: pgmap v2410: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:01.565 INFO:teuthology.orchestra.run.smithi049.stdout:133143986737 2024-04-15T18:53:01.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T18:53:01.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:01 smithi159 ceph-mon[45336]: pgmap v2410: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:02.447 INFO:teuthology.orchestra.run.smithi049.stdout:326417515035 2024-04-15T18:53:02.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-15T18:53:03.437 INFO:teuthology.orchestra.run.smithi049.stdout:3466038607951 2024-04-15T18:53:03.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-15T18:53:03.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:03 smithi082 ceph-mon[44206]: pgmap v2411: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:03.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:03 smithi159 ceph-mon[45336]: pgmap v2411: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:03.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:03 smithi049 ceph-mon[40087]: pgmap v2411: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:03.765 INFO:teuthology.orchestra.run.smithi049.stdout:240518169122 2024-04-15T18:53:03.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-15T18:53:04.803 INFO:teuthology.orchestra.run.smithi049.stdout:1550483194201 2024-04-15T18:53:04.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T18:53:04.825 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:04.833 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:05.036 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:05.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:05 smithi049 ceph-mon[40087]: pgmap v2412: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:05.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:05 smithi082 ceph-mon[44206]: pgmap v2412: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:05.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:05 smithi159 ceph-mon[45336]: pgmap v2412: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:07.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:07 smithi049 ceph-mon[40087]: pgmap v2413: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:07.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:07 smithi082 ceph-mon[44206]: pgmap v2413: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:07.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:07 smithi159 ceph-mon[45336]: pgmap v2413: 1 pgs: 1 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:08.445 INFO:teuthology.orchestra.run.smithi049.stdout:970662609332 2024-04-15T18:53:08.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T18:53:08.447 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:08.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:08.450 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:08.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:08.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:08.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:09.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:09 smithi049 ceph-mon[40087]: pgmap v2414: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:09.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:09 smithi082 ceph-mon[44206]: pgmap v2414: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:09.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:09 smithi159 ceph-mon[45336]: pgmap v2414: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:10.715 INFO:teuthology.orchestra.run.smithi049.stdout:2710124363970 2024-04-15T18:53:11.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:11 smithi082 ceph-mon[44206]: pgmap v2415: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:11.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:11 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2672536096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:53:11.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:11 smithi159 ceph-mon[45336]: pgmap v2415: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:11.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:11 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2672536096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:53:11.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:11 smithi049 ceph-mon[40087]: pgmap v2415: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:11.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:11 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2672536096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T18:53:12.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:12 smithi049 ceph-mon[40087]: 1.0 scrub starts 2024-04-15T18:53:12.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:12 smithi049 ceph-mon[40087]: 1.0 scrub ok 2024-04-15T18:53:12.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:12 smithi082 ceph-mon[44206]: 1.0 scrub starts 2024-04-15T18:53:12.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:12 smithi082 ceph-mon[44206]: 1.0 scrub ok 2024-04-15T18:53:12.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:12 smithi159 ceph-mon[45336]: 1.0 scrub starts 2024-04-15T18:53:12.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:12 smithi159 ceph-mon[45336]: 1.0 scrub ok 2024-04-15T18:53:12.725 INFO:teuthology.orchestra.run.smithi049.stdout:279172874784 2024-04-15T18:53:13.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:13 smithi049 ceph-mon[40087]: pgmap v2416: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:13.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:13 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:13.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:13 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:13.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:13 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/518996533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:53:13.487 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874780 got 279172874784 for osd.8 2024-04-15T18:53:13.487 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:13 smithi082 ceph-mon[44206]: pgmap v2416: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:13 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:13 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:13.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:13 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/518996533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:53:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:13 smithi159 ceph-mon[45336]: pgmap v2416: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:13 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:13 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:13.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:13 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/518996533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-15T18:53:13.956 INFO:teuthology.orchestra.run.smithi049.stdout:55834575430 2024-04-15T18:53:14.493 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2710124363966 got 2710124363970 for osd.6 2024-04-15T18:53:14.493 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:14 smithi082 ceph-mon[44206]: osdmap e926: 12 total, 12 up, 12 in 2024-04-15T18:53:14.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:14 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2753068640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:53:14.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:14 smithi049 ceph-mon[40087]: osdmap e926: 12 total, 12 up, 12 in 2024-04-15T18:53:14.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:14 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2753068640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:53:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:14 smithi159 ceph-mon[45336]: osdmap e926: 12 total, 12 up, 12 in 2024-04-15T18:53:14.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:14 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2753068640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T18:53:14.918 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575427 got 55834575430 for osd.1 2024-04-15T18:53:14.918 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:14.929 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:15.545 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:15 smithi049 ceph-mon[40087]: pgmap v2418: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:15.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:15 smithi159 ceph-mon[45336]: pgmap v2418: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:15.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:15 smithi082 ceph-mon[44206]: pgmap v2418: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:16.975 INFO:teuthology.orchestra.run.smithi049.stdout:1614907703635 2024-04-15T18:53:17.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:17 smithi159 ceph-mon[45336]: pgmap v2419: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:17.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3508438555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:53:17.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:17 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3692278003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:53:17.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:17 smithi049 ceph-mon[40087]: pgmap v2419: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:17.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3508438555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:53:17.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:17 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3692278003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:53:17.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:17 smithi082 ceph-mon[44206]: pgmap v2419: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:17.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3508438555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T18:53:17.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:17 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3692278003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T18:53:18.533 INFO:teuthology.orchestra.run.smithi049.stdout:34359738953 2024-04-15T18:53:19.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:19 smithi049 ceph-mon[40087]: pgmap v2420: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:19.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:19 smithi159 ceph-mon[45336]: pgmap v2420: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:19.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:19 smithi082 ceph-mon[44206]: pgmap v2420: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:20.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:20 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:20.647 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738948 got 34359738953 for osd.0 2024-04-15T18:53:20.647 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:20.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:53:20.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:53:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:53:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:20.648 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:20 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:20.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:20.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:20 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:21.622 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:21.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1614907703631 got 1614907703635 for osd.5 2024-04-15T18:53:21.656 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:21.840 INFO:teuthology.orchestra.run.smithi049.stdout:3466038607954 2024-04-15T18:53:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:22 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:22 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:22 smithi049 ceph-mon[40087]: osdmap e927: 12 total, 12 up, 12 in 2024-04-15T18:53:22.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:22 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/310226999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:53:22.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:22 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:22.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:22 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:22.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:22 smithi159 ceph-mon[45336]: osdmap e927: 12 total, 12 up, 12 in 2024-04-15T18:53:22.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:22 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/310226999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:53:22.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:22 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:22 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:22 smithi082 ceph-mon[44206]: osdmap e927: 12 total, 12 up, 12 in 2024-04-15T18:53:22.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:22 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/310226999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-15T18:53:23.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:23 smithi049 ceph-mon[40087]: pgmap v2423: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:23.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:23 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:23.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:23 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:23.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:23 smithi159 ceph-mon[45336]: pgmap v2423: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:23.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:23 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:23.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:23 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:23.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:23 smithi082 ceph-mon[44206]: pgmap v2423: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:23.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:23 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:23.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:23 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:23.934 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3466038607951 got 3466038607954 for osd.11 2024-04-15T18:53:23.934 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:24.327 INFO:teuthology.orchestra.run.smithi049.stdout:133143986743 2024-04-15T18:53:24.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:24 smithi049 ceph-mon[40087]: osdmap e928: 12 total, 12 up, 12 in 2024-04-15T18:53:24.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:24 smithi159 ceph-mon[45336]: osdmap e928: 12 total, 12 up, 12 in 2024-04-15T18:53:24.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:24 smithi082 ceph-mon[44206]: osdmap e928: 12 total, 12 up, 12 in 2024-04-15T18:53:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:240518169127 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: pgmap v2425: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: osdmap e929: 12 total, 12 up, 12 in 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/3842870887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/167592042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:53:25.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:25 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4053004125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:53:25.694 INFO:teuthology.orchestra.run.smithi049.stdout:1550483194205 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: pgmap v2425: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: osdmap e929: 12 total, 12 up, 12 in 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/3842870887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:53:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/167592042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:53:25.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:25 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4053004125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:53:25.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: pgmap v2425: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:25.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:25.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: osdmap e929: 12 total, 12 up, 12 in 2024-04-15T18:53:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/3842870887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T18:53:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/167592042' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T18:53:25.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:25 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4053004125' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-15T18:53:26.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986737 got 133143986743 for osd.4 2024-04-15T18:53:26.189 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:26.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:26 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1698989332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:53:26.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:26 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1698989332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:53:26.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:26 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1698989332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T18:53:26.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518169122 got 240518169127 for osd.7 2024-04-15T18:53:26.962 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:27.006 INFO:teuthology.orchestra.run.smithi049.stdout:369367187994 2024-04-15T18:53:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:27.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:27 smithi159 ceph-mon[45336]: pgmap v2427: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:27.662 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:27 smithi049 ceph-mon[40087]: pgmap v2427: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:27.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:27 smithi082 ceph-mon[44206]: pgmap v2427: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:28.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1550483194201 got 1550483194205 for osd.2 2024-04-15T18:53:28.059 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:29.394 INFO:teuthology.orchestra.run.smithi049.stdout:326417515040 2024-04-15T18:53:29.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:29 smithi049 ceph-mon[40087]: pgmap v2428: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:29.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:29 smithi159 ceph-mon[45336]: pgmap v2428: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:29.662 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187989 got 369367187994 for osd.10 2024-04-15T18:53:29.662 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:29.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:29 smithi082 ceph-mon[44206]: pgmap v2428: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:30.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417515035 got 326417515040 for osd.9 2024-04-15T18:53:30.234 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:30.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:30 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/1870242194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:53:30.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:30 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/1870242194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:53:30.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:30 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/1870242194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-15T18:53:31.392 INFO:teuthology.orchestra.run.smithi049.stdout:970662609337 2024-04-15T18:53:31.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:31 smithi159 ceph-mon[45336]: pgmap v2429: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:31 smithi049 ceph-mon[40087]: pgmap v2429: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:31.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:31 smithi082 ceph-mon[44206]: pgmap v2429: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:32.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 970662609332 got 970662609337 for osd.3 2024-04-15T18:53:32.079 DEBUG:teuthology.parallel:result is None 2024-04-15T18:53:32.079 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-15T18:53:32.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:53:32.360 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:32.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:32 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/4134131849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:53:32.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:32 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/4134131849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:53:32.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:32 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/4134131849' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T18:53:33.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:33 smithi049 ceph-mon[40087]: pgmap v2430: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:33.646 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:33 smithi159 ceph-mon[45336]: pgmap v2430: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:33.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:33 smithi082 ceph-mon[44206]: pgmap v2430: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:34.409 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:53:34.411 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:53:35.100 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2432,"stamp":"2024-04-15T18:53:34.354958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":4,"kb":1124941824,"kb_used":7396140,"kb_used_data":173512,"kb_used_omap":430,"kb_used_meta":7222033,"kb_avail":1117545684,"statfs":{"total":1151940427776,"available":1144366780416,"internally_reserved":0,"allocated":177676288,"data_stored":173912664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":440519,"internal_metadata":7395362617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.027840"},"pg_stats":[{"pgid":"1.0","version":"509'139","reported_seq":2278,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-04-15T18:53:25.191097+0000","last_change":"2024-04-15T18:53:11.184793+0000","last_active":"2024-04-15T18:53:25.191097+0000","last_peered":"2024-04-15T18:53:25.191097+0000","last_clean":"2024-04-15T18:53:25.191097+0000","last_became_active":"2024-04-15T18:47:24.772930+0000","last_became_peered":"2024-04-15T18:47:24.772930+0000","last_unstale":"2024-04-15T18:53:25.191097+0000","last_undegraded":"2024-04-15T18:53:25.191097+0000","last_fullsized":"2024-04-15T18:53:25.191097+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:53:11.184634+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:53:11.184634+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:54:28.796418+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":807,"seq":3466038607958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254316,"kb_used_data":1244,"kb_used_omap":144,"kb_used_meta":252911,"kb_avail":93490836,"statfs":{"total":95995035648,"available":95734616064,"internally_reserved":0,"allocated":1273856,"data_stored":957545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147868,"internal_metadata":258981476},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":667748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93077404,"statfs":{"total":95995035648,"available":95311261696,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":683014592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":780196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964956,"statfs":{"total":95995035648,"available":95196114944,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":798161344},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":455332,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93289820,"statfs":{"total":95995035648,"available":95528775680,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712740,"kb_used_data":81820,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93032412,"statfs":{"total":95995035648,"available":95265189888,"internally_reserved":0,"allocated":83783680,"data_stored":83467381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363975,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":248872,"kb_used_data":1244,"kb_used_omap":111,"kb_used_meta":247504,"kb_avail":93496280,"statfs":{"total":95995035648,"available":95740190720,"internally_reserved":0,"allocated":1273856,"data_stored":957565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":253444763},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":826020,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919132,"statfs":{"total":95995035648,"available":95149191168,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681636,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063516,"statfs":{"total":95995035648,"available":95297040384,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524196,"kb_used_data":724,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220956,"statfs":{"total":95995035648,"available":95458258944,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880156,"kb_used_data":83412,"kb_used_omap":44,"kb_used_meta":796691,"kb_avail":92864996,"statfs":{"total":95995035648,"available":95093755904,"internally_reserved":0,"allocated":85413888,"data_stored":85101141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":815812138},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":689952,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93055200,"statfs":{"total":95995035648,"available":95288524800,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":674976,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":674235,"kb_avail":93070176,"statfs":{"total":95995035648,"available":95303860224,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":690417345},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:53:35.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T18:53:35.379 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:35.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:35 smithi049 ceph-mon[40087]: pgmap v2431: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:35.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:35 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:35.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:35 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:35.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:35 smithi049 ceph-mon[40087]: osdmap e930: 12 total, 12 up, 12 in 2024-04-15T18:53:35.618 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:35 smithi049 ceph-mon[40087]: from='client.17961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:35.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:35 smithi159 ceph-mon[45336]: pgmap v2431: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:35.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:35 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:35.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:35 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:35.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:35 smithi159 ceph-mon[45336]: osdmap e930: 12 total, 12 up, 12 in 2024-04-15T18:53:35.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:35 smithi159 ceph-mon[45336]: from='client.17961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:35 smithi082 ceph-mon[44206]: pgmap v2431: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:35 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:35.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:35 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:35.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:35 smithi082 ceph-mon[44206]: osdmap e930: 12 total, 12 up, 12 in 2024-04-15T18:53:35.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:35 smithi082 ceph-mon[44206]: from='client.17961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:37.319 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-15T18:53:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-15T18:53:37.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:37 smithi159 ceph-mon[45336]: pgmap v2433: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:37.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:37 smithi049 ceph-mon[40087]: pgmap v2433: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:37.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:37 smithi082 ceph-mon[44206]: pgmap v2433: 1 pgs: 1 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:38.031 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":2433,"stamp":"2024-04-15T18:53:35.792535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":4,"kb":1124941824,"kb_used":7396156,"kb_used_data":173536,"kb_used_omap":430,"kb_used_meta":7222033,"kb_avail":1117545668,"statfs":{"total":1151940427776,"available":1144366764032,"internally_reserved":0,"allocated":177700864,"data_stored":173927514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":440519,"internal_metadata":7395362617},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002203"},"pg_stats":[{"pgid":"1.0","version":"509'139","reported_seq":2278,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-04-15T18:53:25.191097+0000","last_change":"2024-04-15T18:53:11.184793+0000","last_active":"2024-04-15T18:53:25.191097+0000","last_peered":"2024-04-15T18:53:25.191097+0000","last_clean":"2024-04-15T18:53:25.191097+0000","last_became_active":"2024-04-15T18:47:24.772930+0000","last_became_peered":"2024-04-15T18:47:24.772930+0000","last_unstale":"2024-04-15T18:53:25.191097+0000","last_undegraded":"2024-04-15T18:53:25.191097+0000","last_fullsized":"2024-04-15T18:53:25.191097+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'139","last_scrub_stamp":"2024-04-15T18:53:11.184634+0000","last_deep_scrub":"509'139","last_deep_scrub_stamp":"2024-04-15T18:36:05.552905+0000","last_clean_scrub_stamp":"2024-04-15T18:53:11.184634+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-15T18:54:28.796418+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4760352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":807,"seq":3466038607958,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254316,"kb_used_data":1244,"kb_used_omap":144,"kb_used_meta":252911,"kb_avail":93490836,"statfs":{"total":95995035648,"available":95734616064,"internally_reserved":0,"allocated":1273856,"data_stored":957545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":147868,"internal_metadata":258981476},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":86,"seq":369367187997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":667748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93077404,"statfs":{"total":95995035648,"available":95311261696,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":683014592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":76,"seq":326417515042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":780196,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964956,"statfs":{"total":95995035648,"available":95196114944,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":798161344},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":65,"seq":279172874789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":455340,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93289812,"statfs":{"total":95995035648,"available":95528767488,"internally_reserved":0,"allocated":753664,"data_stored":436054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":56,"seq":240518169129,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712740,"kb_used_data":81820,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93032412,"statfs":{"total":95995035648,"available":95265189888,"internally_reserved":0,"allocated":83783680,"data_stored":83467381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":631,"seq":2710124363975,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":248872,"kb_used_data":1244,"kb_used_omap":111,"kb_used_meta":247504,"kb_avail":93496280,"statfs":{"total":95995035648,"available":95740190720,"internally_reserved":0,"allocated":1273856,"data_stored":957565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114021,"internal_metadata":253444763},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":826020,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92919132,"statfs":{"total":95995035648,"available":95149191168,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":681644,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063508,"statfs":{"total":95995035648,"available":95297032192,"internally_reserved":0,"allocated":753664,"data_stored":436054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":361,"seq":1550483194207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":524196,"kb_used_data":724,"kb_used_omap":114,"kb_used_meta":523341,"kb_avail":93220956,"statfs":{"total":95995035648,"available":95458258944,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":117519,"internal_metadata":535901425},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":226,"seq":970662609338,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880156,"kb_used_data":83412,"kb_used_omap":44,"kb_used_meta":796691,"kb_avail":92864996,"statfs":{"total":95995035648,"available":95093755904,"internally_reserved":0,"allocated":85413888,"data_stored":85101141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45526,"internal_metadata":815812138},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":689952,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93055200,"statfs":{"total":95995035648,"available":95288524800,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":376,"seq":1614907703639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":674976,"kb_used_data":724,"kb_used_omap":4,"kb_used_meta":674235,"kb_avail":93070176,"statfs":{"total":95995035648,"available":95303860224,"internally_reserved":0,"allocated":741376,"data_stored":428629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4415,"internal_metadata":690417345},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T18:53:38.032 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-15T18:53:38.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c2d8c3155d854ee204cb7c7261232571229d1d54 shell --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-15T18:53:38.311 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T18:53:38.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:38 smithi159 ceph-mon[45336]: from='client.17967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:38.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:38 smithi049 ceph-mon[40087]: from='client.17967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:38.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:38 smithi082 ceph-mon[44206]: from='client.17967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T18:53:39.537 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:39 smithi049 ceph-mon[40087]: pgmap v2434: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:39.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:39 smithi159 ceph-mon[45336]: pgmap v2434: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:39.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:39 smithi082 ceph-mon[44206]: pgmap v2434: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:40.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: purged_snaps scrub starts 2024-04-15T18:53:40.611 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: purged_snaps scrub ok 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: osdmap e931: 12 total, 12 up, 12 in 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:40.612 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:40 smithi049 ceph-mon[40087]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: purged_snaps scrub starts 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: purged_snaps scrub ok 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: osdmap e931: 12 total, 12 up, 12 in 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:40.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:40 smithi159 ceph-mon[45336]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:40.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: purged_snaps scrub starts 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: purged_snaps scrub ok 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: osdmap e931: 12 total, 12 up, 12 in 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='client.? 172.21.15.49:0/2734856389' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T18:53:40.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:40 smithi082 ceph-mon[44206]: from='mgr.14151 172.21.15.49:0/46346707' entity='mgr.y' 2024-04-15T18:53:40.913 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-15T18:53:40.922 INFO:tasks.cephadm:Teardown begin 2024-04-15T18:53:40.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:53:40.990 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:53:41.043 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:53:41.097 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-15T18:53:41.098 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T18:53:41.140 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T18:53:41.178 DEBUG:teuthology.orchestra.run.smithi159:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T18:53:41.208 INFO:tasks.cephadm:Stopping all daemons... 2024-04-15T18:53:41.208 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-15T18:53:41.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a 2024-04-15T18:53:41.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:41 smithi159 ceph-mon[45336]: pgmap v2436: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:41.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:41 smithi049 systemd[1]: Stopping Ceph mon.a for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:41.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:41 smithi049 ceph-mon[40087]: pgmap v2436: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:41.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:41 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:53:41.488+0000 7f801b689700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:41.716 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:41 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a[40064]: 2024-04-15T18:53:41.488+0000 7f801b689700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-15T18:53:41.778 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:41 smithi082 ceph-mon[44206]: pgmap v2436: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-15T18:53:42.215 INFO:journalctl@ceph.mon.a.smithi049.stdout:Apr 15 18:53:42 smithi049 bash[340100]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-a 2024-04-15T18:53:42.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.a.service' 2024-04-15T18:53:42.518 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:42.519 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-15T18:53:42.519 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-15T18:53:42.519 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.b 2024-04-15T18:53:42.832 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:42 smithi082 systemd[1]: Stopping Ceph mon.b for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:43.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-b[44183]: 2024-04-15T18:53:42.838+0000 7fe1dc16e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:43.279 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-b[44183]: 2024-04-15T18:53:42.838+0000 7fe1dc16e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-15T18:53:43.779 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 15 18:53:43 smithi082 bash[192792]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-b 2024-04-15T18:53:43.890 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.b.service' 2024-04-15T18:53:43.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:43.960 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-15T18:53:43.960 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-15T18:53:43.960 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.c 2024-04-15T18:53:44.253 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:44 smithi159 systemd[1]: Stopping Ceph mon.c for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:44.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-c[45296]: 2024-04-15T18:53:44.259+0000 7f365991a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:44.647 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-c[45296]: 2024-04-15T18:53:44.259+0000 7f365991a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-15T18:53:45.147 INFO:journalctl@ceph.mon.c.smithi159.stdout:Apr 15 18:53:44 smithi159 bash[187058]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mon-c 2024-04-15T18:53:45.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mon.c.service' 2024-04-15T18:53:45.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:45.271 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-15T18:53:45.271 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-15T18:53:45.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y 2024-04-15T18:53:45.575 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:53:45 smithi049 systemd[1]: Stopping Ceph mgr.y for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:45.575 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:53:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:53:45.559+0000 7fb029c36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:45.575 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:53:45 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y[40340]: 2024-04-15T18:53:45.559+0000 7fb029c36700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-15T18:53:46.215 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Apr 15 18:53:46 smithi049 bash[340244]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-y 2024-04-15T18:53:46.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.y.service' 2024-04-15T18:53:46.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:46.620 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-15T18:53:46.620 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-15T18:53:46.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.x 2024-04-15T18:53:47.029 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:53:46 smithi082 systemd[1]: Stopping Ceph mgr.x for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:47.517 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:53:47 smithi082 bash[192948]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-mgr-x 2024-04-15T18:53:47.671 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:53:47 smithi082 systemd[1]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-15T18:53:47.671 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 15 18:53:47 smithi082 systemd[1]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-15T18:53:47.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@mgr.x.service' 2024-04-15T18:53:47.707 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:47.707 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-15T18:53:47.707 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-15T18:53:47.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.0 2024-04-15T18:53:48.215 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:53:47 smithi049 systemd[1]: Stopping Ceph osd.0 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:48.216 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:53:47 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0[59449]: 2024-04-15T18:53:47.987+0000 7f7a394d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:48.216 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:53:47 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0[59449]: 2024-04-15T18:53:47.987+0000 7f7a394d6700 -1 osd.0 931 *** Got signal Terminated *** 2024-04-15T18:53:48.216 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:53:47 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0[59449]: 2024-04-15T18:53:47.987+0000 7f7a394d6700 -1 osd.0 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:53:54.216 INFO:journalctl@ceph.osd.0.smithi049.stdout:Apr 15 18:53:54 smithi049 bash[340388]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-0 2024-04-15T18:53:56.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-15T18:53:56.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:53:56.395 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-15T18:53:56.395 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-15T18:53:56.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.1 2024-04-15T18:53:56.694 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:53:56 smithi049 systemd[1]: Stopping Ceph osd.1 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:53:56.965 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:53:56 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-1[66694]: 2024-04-15T18:53:56.701+0000 7fdd75a34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:53:56.966 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:53:56 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-1[66694]: 2024-04-15T18:53:56.701+0000 7fdd75a34700 -1 osd.1 931 *** Got signal Terminated *** 2024-04-15T18:53:56.966 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:53:56 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-1[66694]: 2024-04-15T18:53:56.701+0000 7fdd75a34700 -1 osd.1 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:02.966 INFO:journalctl@ceph.osd.1.smithi049.stdout:Apr 15 18:54:02 smithi049 bash[340654]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-1 2024-04-15T18:54:04.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-15T18:54:04.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:04.617 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-15T18:54:04.618 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-15T18:54:04.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2 2024-04-15T18:54:04.871 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:54:04 smithi049 systemd[1]: Stopping Ceph osd.2 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:05.216 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:54:04 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:54:04.878+0000 7fc11c0d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:05.216 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:54:04 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:54:04.878+0000 7fc11c0d1700 -1 osd.2 931 *** Got signal Terminated *** 2024-04-15T18:54:05.216 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:54:04 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2[176316]: 2024-04-15T18:54:04.878+0000 7fc11c0d1700 -1 osd.2 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:10.966 INFO:journalctl@ceph.osd.2.smithi049.stdout:Apr 15 18:54:10 smithi049 bash[340923]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-2 2024-04-15T18:54:12.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-15T18:54:12.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:12.508 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-15T18:54:12.509 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-15T18:54:12.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3 2024-04-15T18:54:12.716 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:54:12 smithi049 systemd[1]: Stopping Ceph osd.3 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:13.216 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:54:12 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:54:12.819+0000 7f6c6be69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:13.216 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:54:12 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:54:12.819+0000 7f6c6be69700 -1 osd.3 931 *** Got signal Terminated *** 2024-04-15T18:54:13.216 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:54:12 smithi049 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3[131985]: 2024-04-15T18:54:12.819+0000 7f6c6be69700 -1 osd.3 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:15.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:15.285+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:16.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:16.242+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:17.530 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:17 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:17.103+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:17.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:17.284+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:17.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:17.210+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:18.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:18 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:18.125+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:18.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:18 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:18.234+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:18.647 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:18.186+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:18.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:18.293+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:19.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:18 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:18.633+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:19.466 INFO:journalctl@ceph.osd.3.smithi049.stdout:Apr 15 18:54:19 smithi049 bash[341184]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-3 2024-04-15T18:54:19.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:19 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:19.129+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:19.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:19 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:19.218+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:19.647 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:19.168+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:19.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:19.303+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:20.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:19 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:19.644+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:20.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:20.141+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:20.398 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:20.353+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:20.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:20 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:20.080+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:20.529 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:20 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:20.242+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:20.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-15T18:54:20.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:20.963 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-15T18:54:20.963 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-15T18:54:20.963 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.4 2024-04-15T18:54:20.995 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:20 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:20.656+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:21.275 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:21 smithi082 systemd[1]: Stopping Ceph osd.4 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:21.275 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:21.122+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:21.529 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:21.283+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:21.530 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:21.478+0000 7f7040c5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:21.530 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:21.478+0000 7f7040c5d700 -1 osd.4 931 *** Got signal Terminated *** 2024-04-15T18:54:21.530 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:21.478+0000 7f7040c5d700 -1 osd.4 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:21.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:21.179+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:21.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:21.359+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:22.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:21 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:21.619+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:22.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:22.087+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:22.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:22.237+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:22.647 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:22.226+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:22.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:22.408+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:23.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:22 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:22.667+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:23.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:23.103+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:23.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:23.189+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:23.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:23 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:23.411+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:23.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:23 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:23.267+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:23.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:23 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:23.267+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:24.020 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:23.694+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:24.020 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:23 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:23.694+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:24.279 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:24.161+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:24.280 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:24.127+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:24.280 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:24.127+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.276905+0000 front 2024-04-15T18:53:59.276998+0000 (oldest deadline 2024-04-15T18:54:23.976811+0000) 2024-04-15T18:54:24.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:24.028+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:24.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:24 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:24.397+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:24.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:24 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:24.218+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:24.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:24 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:24.218+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:25.017 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:24.647+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:25.018 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:24 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:24.647+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:25.279 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:25.120+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:25.280 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:25.161+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:25.280 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:25.161+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.276905+0000 front 2024-04-15T18:53:59.276998+0000 (oldest deadline 2024-04-15T18:54:23.976811+0000) 2024-04-15T18:54:25.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:25.025+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:25.484 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:25 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:25.381+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:25.485 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:25 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:25.193+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:25.485 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:25 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:25.193+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:25.485 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:25 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:25.203+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:25.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:25 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:25.492+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:26.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:25.660+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:26.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:25 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:25.660+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:26.449 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:26.253+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:26.449 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:26.187+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:26.449 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:26.187+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:26.449 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:26.379+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:26.449 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:26.379+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:26.529 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:26.118+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:49.375148+0000 front 2024-04-15T18:53:49.375300+0000 (oldest deadline 2024-04-15T18:54:16.375716+0000) 2024-04-15T18:54:26.530 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4[49709]: 2024-04-15T18:54:26.118+0000 7f703d251700 -1 osd.4 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.276905+0000 front 2024-04-15T18:53:59.276998+0000 (oldest deadline 2024-04-15T18:54:23.976811+0000) 2024-04-15T18:54:26.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:26.072+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:26.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:26.162+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:26.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:26.162+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:26.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:26 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:26.456+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:27.030 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:26.701+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:27.030 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:26 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:26.701+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:27.338 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:27.037+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:27.338 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:27.179+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:27.338 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:27.179+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:27.464 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:27.212+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:27.464 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:27.212+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:27.464 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:27.223+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:27.464 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:27.355+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:27.464 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:27.355+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:27.701 INFO:journalctl@ceph.osd.4.smithi082.stdout:Apr 15 18:54:27 smithi082 bash[193098]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-4 2024-04-15T18:54:27.897 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:27 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:27.471+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:27.971 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:27.709+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:27.972 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:27.709+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:28.279 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:28.202+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:28.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:28.202+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:28.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:27 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:27.991+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:28.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:28.429+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:28.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:28.182+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:28.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:28.261+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:28.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:28.261+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:28.649 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:28.395+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:28.649 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:28 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:28.395+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:28.944 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:28.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:28.945 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:28.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:29.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:29.163+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:29.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:29.163+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:29.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:28 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:28.981+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:29.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-15T18:54:29.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:29.452 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-15T18:54:29.452 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-15T18:54:29.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5 2024-04-15T18:54:29.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:29.422+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:29.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:29.190+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:29.649 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:29.234+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:29.649 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:29.234+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:29.649 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:29.422+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:29.649 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:29 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:29.422+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:29.703 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 systemd[1]: Stopping Ceph osd.5 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:29.985 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:29.710+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:29.985 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:29.710+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:29.985 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:29.755+0000 7f6a85deb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:29.985 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:29.755+0000 7f6a85deb700 -1 osd.5 931 *** Got signal Terminated *** 2024-04-15T18:54:29.985 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:29.755+0000 7f6a85deb700 -1 osd.5 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:30.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:30 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:30.121+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:30.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:30 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:30.121+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:30.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:29 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:29.993+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:30.648 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:30.193+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:30.649 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:30.193+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:30.650 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:30.397+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:30.650 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:30.200+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:30.650 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:30.397+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:30.650 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:30 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:30.397+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:30.963 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:30 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:30.666+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:30.963 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:30 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:30.666+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:31.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:31.082+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:31.282 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:31.082+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:31.283 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:30 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:30.970+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:31.434 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:31.155+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:31.434 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:31.155+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:31.434 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:31.359+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:31.435 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:31.208+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:31.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:31.441+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:31.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:31 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:31.441+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:31.913 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:31.665+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:31.914 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:31.665+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:32.279 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:32.078+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:32.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:32.078+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:32.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:32.078+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:32.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:31 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:31.922+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:32.473 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:32.201+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:32.474 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:32.201+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:32.474 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:32.386+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:32.474 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:32.386+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:32.474 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:32.241+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:32.475 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:32.241+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:32.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:32.480+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:32.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:32 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:32.480+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:33.030 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:32.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:33.030 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:32.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:33.030 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:32 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:32.879+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:33.230+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:33.230+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:33.230+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:33.403+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:33.403+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:33.272+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:33.480 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:33.272+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:33.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:33.100+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:33.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:33.100+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:33.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:33.100+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:33.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:33.487+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:33.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:33 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:33.487+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:34.029 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:33.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:34.030 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:33.702+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:34.030 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:33 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:33.928+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:34.467 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:34.186+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:34.467 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:34.186+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:34.467 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:34.186+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:34.467 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:34.388+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:34.468 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:34.388+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:34.468 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:34.260+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:34.468 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:34.260+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:34.529 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:34.075+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:34.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:34.075+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:34.530 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:34.075+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:34.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:34.474+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:34.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:34 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:34.474+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:35.020 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:34.919+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:35.021 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:34.919+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:35.021 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:34.744+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.908794+0000 front 2024-04-15T18:53:51.908856+0000 (oldest deadline 2024-04-15T18:54:17.709772+0000) 2024-04-15T18:54:35.021 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:34.744+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:57.710493+0000 front 2024-04-15T18:53:57.710539+0000 (oldest deadline 2024-04-15T18:54:23.610338+0000) 2024-04-15T18:54:35.021 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:34 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5[120529]: 2024-04-15T18:54:34.744+0000 7f6a823df700 -1 osd.5 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:09.312633+0000 front 2024-04-15T18:54:09.312575+0000 (oldest deadline 2024-04-15T18:54:34.012679+0000) 2024-04-15T18:54:35.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:35 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:35.028+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:35.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:35 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:35.028+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:35.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:35 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:35.028+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:35.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:35.153+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:35.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:35.153+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:35.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:35.153+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:35.398 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:35.368+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:35.398 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:35.368+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:35.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:35.296+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:35.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:35.296+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:35.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:35.477+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:35.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:35 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:35.477+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:35.949 INFO:journalctl@ceph.osd.5.smithi082.stdout:Apr 15 18:54:35 smithi082 bash[193360]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-5 2024-04-15T18:54:36.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:36.010+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:36.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:36.010+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:36.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:36.010+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:36.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:35 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:35.955+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:36.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:35 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:35.955+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:36.166+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:36.166+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:36.166+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:36.411+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:36.411+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:36.343+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:36.507 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:36.343+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:36.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:36.514+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:36.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:36 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:36.514+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:37.264 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:36.956+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:37.264 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:36 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:36.956+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:37.264 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.001+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:37.264 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.001+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:37.264 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.001+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:37.446 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:37.371+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:37.446 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:37.371+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:37.446 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:37.196+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:37.446 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:37.196+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:37.447 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:37.196+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:37.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-15T18:54:37.699 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:37.699 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-15T18:54:37.699 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-15T18:54:37.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6 2024-04-15T18:54:37.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:37.453+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:37.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:37.453+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:37.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:37.471+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:37.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:37 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:37.471+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:37.965 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 systemd[1]: Stopping Ceph osd.6 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:37.965 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.972+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:37.965 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.972+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:37.998+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:37.998+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:37 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:37.972+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.015+0000 7f06fb2a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.015+0000 7f06fb2a8700 -1 osd.6 931 *** Got signal Terminated *** 2024-04-15T18:54:38.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.015+0000 7f06fb2a8700 -1 osd.6 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:38.419 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:38.170+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:38.419 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:38.170+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:38.420 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:38.170+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:38.420 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:38.405+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:38.420 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:38.405+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:38.420 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:38.330+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:38.420 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:38.330+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:38.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:38.426+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:38.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:38 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:38.426+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:39.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.954+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:39.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.954+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:39.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:38 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:38.954+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:39.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:39.031+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:39.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:39.031+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:39.439 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:39.161+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:39.439 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:39.161+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:39.439 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:39.161+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:39.439 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:39.406+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:39.439 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:39.406+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:39.440 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:39.365+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:39.440 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:39.365+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:39.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:39.446+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:39.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:39 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:39.446+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:39.924+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:39.924+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:39.924+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:39 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:39.924+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.591262+0000 front 2024-04-15T18:54:17.591237+0000 (oldest deadline 2024-04-15T18:54:39.291217+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:40.026+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:40.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:40.026+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:40.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:40.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:40.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:40.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:40.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:40.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:40.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:40.355+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:40.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:40.355+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:40.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:40.355+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:40.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:40.410+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:40.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:40.410+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:40.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:40.491+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:40.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:40.491+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:40.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:40 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:40.491+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:40.943+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:40.943+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:40.943+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:40 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:40.943+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.591262+0000 front 2024-04-15T18:54:17.591237+0000 (oldest deadline 2024-04-15T18:54:39.291217+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:41.003+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:41.003+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:41.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:41.003+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:41.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:41.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:41.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:41.125+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:41.347+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:41.347+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:41.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:41.347+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:41.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:41.412+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:41.898 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:41.412+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:41.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:41.515+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:41.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:41.515+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:41.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:41 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:41.515+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:41.912+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:41.912+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:41.912+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:41 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:41.912+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.591262+0000 front 2024-04-15T18:54:17.591237+0000 (oldest deadline 2024-04-15T18:54:39.291217+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:42.037+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:42.037+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:42.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:42.037+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:42.368 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:42.082+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:42.368 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:42.082+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:42.368 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:42.082+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:42.368 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:42.082+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:42.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:42.485+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:42.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:42.485+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:42.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:42.485+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:42.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:42.413+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:42.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:42.413+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:42.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:42.375+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:42.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:42.375+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:42.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:42 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:42.375+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:43 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:43.075+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:43 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:43.075+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:43 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:43.075+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:42.952+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.888549+0000 front 2024-04-15T18:53:52.888506+0000 (oldest deadline 2024-04-15T18:54:17.588368+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:42.952+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.889338+0000 front 2024-04-15T18:53:59.889505+0000 (oldest deadline 2024-04-15T18:54:25.789421+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:42.952+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.790072+0000 front 2024-04-15T18:54:05.790192+0000 (oldest deadline 2024-04-15T18:54:31.690091+0000) 2024-04-15T18:54:43.280 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:42 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6[153890]: 2024-04-15T18:54:42.952+0000 7f06f789c700 -1 osd.6 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.591262+0000 front 2024-04-15T18:54:17.591237+0000 (oldest deadline 2024-04-15T18:54:39.291217+0000) 2024-04-15T18:54:43.353 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:43.035+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:43.353 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:43.035+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:43.353 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:43.035+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:43.353 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:43.035+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:43.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:43.525+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:43.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:43.525+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:43.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:43.525+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:43.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:43.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:43.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:43.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:43.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:43.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:43.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:43.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:43.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:43 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:43.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:44.030 INFO:journalctl@ceph.osd.6.smithi082.stdout:Apr 15 18:54:43 smithi082 bash[193631]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-6 2024-04-15T18:54:44.337 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:44.043+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:44.337 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:44.043+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:44.337 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:44.043+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:44.337 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:44.043+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:44.508 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:44 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:44.089+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:44.509 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:44 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:44.089+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:44.509 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:44 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:44.089+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:44.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:44.559+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:44.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:44.559+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:44.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:44.559+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:44.407+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:44.407+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:44.407+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:44.345+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:44.345+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:44.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:44 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:44.345+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:45.232 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.064+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:45.232 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.064+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:45.232 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.064+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:45.297 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:45.015+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:45.314 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:45.015+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:45.314 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:45.015+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:45.321 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:45.015+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:45.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.6.service' 2024-04-15T18:54:45.572 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:45.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:45.573 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:45.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:45.573 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:45.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:45.573 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:45.304+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:45.573 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:45.304+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:45.573 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:45.304+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:45.582 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:45.582 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-15T18:54:45.582 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-15T18:54:45.583 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.7 2024-04-15T18:54:45.867 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 systemd[1]: Stopping Ceph osd.7 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:45.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:45.580+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:45.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:45.580+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:45.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:45 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:45.580+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.873+0000 7f85d5a81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.873+0000 7f85d5a81700 -1 osd.7 931 *** Got signal Terminated *** 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:45 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:45.873+0000 7f85d5a81700 -1 osd.7 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:46 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:46.083+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:46 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:46.083+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:46.280 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:46 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:46.083+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:46.329 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:46.037+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:46.329 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:46.037+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:46.330 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:46.037+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:46.330 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:46.037+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:46.330 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:46.258+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:46.330 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:46.258+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:46.330 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:46.258+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:46.609 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:46.337+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:46.609 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:46.337+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:46.610 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:46.337+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:46.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:46.617+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:46.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:46.617+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:46.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:46 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:46.617+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:47.375 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:47.246+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:47.246+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:47.246+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:47.060+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:47.060+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:47.060+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:47.060+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:47.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:47.060+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:47.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:47 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:47.104+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:47.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:47 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:47.104+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:47.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:47 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:47.104+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:47.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:47 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:47.104+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.351808+0000 front 2024-04-15T18:54:25.351881+0000 (oldest deadline 2024-04-15T18:54:47.051834+0000) 2024-04-15T18:54:47.637 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:47.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:47.637 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:47.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:47.638 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:47.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:47.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:47.645+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:47.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:47.645+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:47.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:47 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:47.645+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:48.262+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:48.262+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:48.262+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:48.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:48.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:48.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:48.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:48.376 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:48.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:48.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:48 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:48.134+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:48.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:48 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:48.134+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:48.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:48 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:48.134+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:48.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:48 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:48.134+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.351808+0000 front 2024-04-15T18:54:25.351881+0000 (oldest deadline 2024-04-15T18:54:47.051834+0000) 2024-04-15T18:54:48.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:48.610+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:48.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:48.610+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:48.648 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:48.610+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:48.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:48.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:48.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:48.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:48.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:48.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:48.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:48 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:48.383+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:49.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:49.127+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:49.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:49.127+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:49.127+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:49.127+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:49.127+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:49.360+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:49.360+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:49.360+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:49.360+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:49.312+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:49.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:49.312+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:49.400 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:49.312+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:49.400 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:49.312+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:49.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:49.105+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:49.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:49.105+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:49.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:49.105+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:49.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:49 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:49.105+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.351808+0000 front 2024-04-15T18:54:25.351881+0000 (oldest deadline 2024-04-15T18:54:47.051834+0000) 2024-04-15T18:54:49.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:49.635+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:49.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:49.635+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:49.898 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:49.635+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:49.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:49 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:49.635+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:50.331+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:50.331+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:50.331+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:50.331+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:50.098+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:50.341 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:50.098+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:50.342 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:50.098+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:50.342 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:50.098+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:50.342 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:50.098+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:50.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:50 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:50.151+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:58.646840+0000 front 2024-04-15T18:53:58.646783+0000 (oldest deadline 2024-04-15T18:54:23.946859+0000) 2024-04-15T18:54:50.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:50 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:50.151+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.349585+0000 front 2024-04-15T18:54:08.349628+0000 (oldest deadline 2024-04-15T18:54:34.249392+0000) 2024-04-15T18:54:50.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:50 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:50.151+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:14.250311+0000 front 2024-04-15T18:54:14.387701+0000 (oldest deadline 2024-04-15T18:54:40.150142+0000) 2024-04-15T18:54:50.530 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:50 smithi082 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7[73238]: 2024-04-15T18:54:50.151+0000 7f85d2075700 -1 osd.7 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.351808+0000 front 2024-04-15T18:54:25.351881+0000 (oldest deadline 2024-04-15T18:54:47.051834+0000) 2024-04-15T18:54:50.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:50.348+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:50.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:50.348+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:50.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:50.348+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:50.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:50.348+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:51.074 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:50.679+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:51.074 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:50.679+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:51.074 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:50.679+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:51.074 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:50 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:50.679+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:51.352 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:51.081+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:51.352 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:51.081+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:51.352 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:51.081+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:51.352 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:51.081+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:51.352 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:51.081+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:51.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:51.359+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:51.359+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:51.359+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:51.359+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:51.366+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:51.366+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:51.366+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:51.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:51.366+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:52.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:51.696+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:52.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:51.696+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:52.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:51.696+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:52.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:51 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:51.696+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:52.186 INFO:journalctl@ceph.osd.7.smithi082.stdout:Apr 15 18:54:51 smithi082 bash[193898]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-7 2024-04-15T18:54:52.380 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:52.048+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:52.380 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:52.049+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:52.380 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:52.049+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:52.380 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:52.049+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:52.380 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:52.049+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:52.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:52.411+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:52.411+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:52.411+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:52.411+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:52.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:52.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:52.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:52.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:52.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:53.050 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:52.726+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:53.050 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:52.726+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:53.050 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:52.726+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:53.050 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:52 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:52.726+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:53.365 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.057+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:53.365 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.057+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:53.365 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.057+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:53.365 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.057+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:53.365 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.057+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:53.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:53.405+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:53.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:53.405+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:53.405+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:53.405+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:53.372+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:53.372+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:53.372+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:53.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:53.372+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:53.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.7.service' 2024-04-15T18:54:53.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:54:53.693 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-15T18:54:53.693 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-15T18:54:53.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.8 2024-04-15T18:54:54.071 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:53.762+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:54.071 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:53.762+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:54.071 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:53.762+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:54.071 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:53.762+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:54.072 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 systemd[1]: Stopping Ceph osd.8 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:54:54.072 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.975+0000 7fc8fac27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:54:54.072 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.975+0000 7fc8fac27700 -1 osd.8 931 *** Got signal Terminated *** 2024-04-15T18:54:54.072 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:53 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:53.975+0000 7fc8fac27700 -1 osd.8 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:54:54.388 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:54.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:54.388 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:54.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:54.388 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:54.078+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:54.388 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:54.079+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:54.388 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:54.079+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:54.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:54.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:54.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:54.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:54.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:54.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:54.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:54.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:54.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:54.395+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:54.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:54.395+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:54.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:54.395+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:54.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:54.395+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:55.079 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:54.777+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:55.079 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:54.777+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:55.079 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:54.777+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:55.079 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:54 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:54.777+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:55.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:55.086+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:55.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:55.086+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:55.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:55.086+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:55.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:55.086+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:55.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:55.086+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:55.754 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:55.421+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:55.754 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:55.421+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:55.421+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:55.421+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:55.456+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:55.456+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:55.456+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:55.755 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:55.456+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:56.125 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:55.761+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:56.125 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:55.761+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:56.126 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:55.761+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:56.126 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:55 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:55.761+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:56.398 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:56.133+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:56.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:56.133+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:56.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:56.133+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:56.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:56.133+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:56.399 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:56.133+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:56.418+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:56.418+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:56.418+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:56.418+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:56.409+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:56.409+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:56.730 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:56.409+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:56.731 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:56.409+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:57.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:56.737+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:57.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:56.737+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:57.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:56.737+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:57.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:56 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:56.737+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:57.435 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:57.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:57.435 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:57.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:57.435 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:57.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:57.435 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:57.360+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:57.436 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:57.162+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:57.436 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:57.162+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:57.436 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:57.162+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:57.436 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:57.162+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:57.436 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:57.162+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:57.767 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:57.442+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:57.767 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:57.442+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:57.767 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:57.442+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:57.767 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:57.442+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:58.108 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:57.774+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:58.108 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:57.774+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:58.108 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:57.774+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:58.108 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:57.774+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:58.108 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:57 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:57.774+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:51.043726+0000 front 2024-04-15T18:53:51.043709+0000 (oldest deadline 2024-04-15T18:54:16.243902+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:00.245647+0000 front 2024-04-15T18:54:00.245450+0000 (oldest deadline 2024-04-15T18:54:22.545493+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:08.347236+0000 front 2024-04-15T18:54:08.347148+0000 (oldest deadline 2024-04-15T18:54:33.046992+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:15.948116+0000 front 2024-04-15T18:54:15.948148+0000 (oldest deadline 2024-04-15T18:54:40.648173+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.549729+0000 front 2024-04-15T18:54:23.549617+0000 (oldest deadline 2024-04-15T18:54:47.049261+0000) 2024-04-15T18:54:58.379 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8[51239]: 2024-04-15T18:54:58.115+0000 7fc8f3213700 -1 osd.8 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:31.650731+0000 front 2024-04-15T18:54:31.650786+0000 (oldest deadline 2024-04-15T18:54:57.550626+0000) 2024-04-15T18:54:58.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:58.424+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:58.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:58.424+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:58.424+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:58.424+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:58.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:58.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:58.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:58.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:58.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:59.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:58.775+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:54:59.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:58.775+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:54:59.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:58.775+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:54:59.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:58.775+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:54:59.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:58 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:58.775+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:54:59.648 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:59.391+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:59.391+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:59.391+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:54:59.391+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:59.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:59.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:59.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:59.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:54:59.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:54:59.386+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:00.148 INFO:journalctl@ceph.osd.8.smithi159.stdout:Apr 15 18:54:59 smithi159 bash[187229]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-8 2024-04-15T18:55:00.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:59.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:00.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:59.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:00.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:59.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:00.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:59.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:00.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:54:59 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:54:59.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:00.648 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:00.429+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:00.429+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:00.429+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:00.429+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:00.429+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:00.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:00.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:00.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:00.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:00.649 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:00.387+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:01.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:00.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:01.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:00.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:01.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:00.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:01.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:00.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:01.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:00 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:00.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:01.494 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:01.435+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:01.435+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:01.435+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:01.435+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:01.435+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:01.420+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:01.420+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:01.420+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:01.420+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:01.495 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:01.420+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:01.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.8.service' 2024-04-15T18:55:01.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:55:01.747 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-15T18:55:01.747 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-15T18:55:01.747 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.9 2024-04-15T18:55:02.072 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:01 smithi159 systemd[1]: Stopping Ceph osd.9 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:55:02.072 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:01.863+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:02.072 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:01.863+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:02.072 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:01.863+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:02.072 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:01.863+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:02.073 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:01 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:01.863+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:02.398 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.079+0000 7f79e5c06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:55:02.398 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.079+0000 7f79e5c06700 -1 osd.9 931 *** Got signal Terminated *** 2024-04-15T18:55:02.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.079+0000 7f79e5c06700 -1 osd.9 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:02.430+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:02.466+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:02.466+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:02.466+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:02.843 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:02.466+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:02.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:02.466+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:03.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:02.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:03.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:02.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:03.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:02.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:03.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:02.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:03.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:02 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:02.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:03.481+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:03.481+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:03.481+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:03.481+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:03.481+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:03.865 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:03.454+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:03.866 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:03.454+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:03.866 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:03.454+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:03.866 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:03.454+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:03.866 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:03.454+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:04.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:03.872+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:04.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:03.872+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:04.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:03.872+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:04.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:03.872+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:04.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:03 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:03.872+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:04.856 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:04.528+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:04.528+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:04.528+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:04.528+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:04.528+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:04.477+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:04.477+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:04.477+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:04.477+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:04.857 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:04.477+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:05.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:04.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:05.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:04.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:05.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:04.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:05.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:04.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:05.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:04 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:04.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:05.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:05.574+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:05.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:05.574+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:05.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:05.574+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:05.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:05.574+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:05.843 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:05.574+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:05.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:05.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:05.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:05.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:05.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:05.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:05.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:05.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:05.844 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:05.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:06.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:05.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:06.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:05.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:06.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:05.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:06.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:05.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:06.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:05 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:05.850+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:06.804 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:06.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:06.804 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:06.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:06.804 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:06.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:06.804 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:06.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:06.804 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:06.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.465126+0000 front 2024-04-15T18:53:59.465156+0000 (oldest deadline 2024-04-15T18:54:25.364898+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:05.365557+0000 front 2024-04-15T18:54:05.365795+0000 (oldest deadline 2024-04-15T18:54:31.766077+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.067428+0000 front 2024-04-15T18:54:17.067516+0000 (oldest deadline 2024-04-15T18:54:43.468122+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:25.169494+0000 front 2024-04-15T18:54:25.169418+0000 (oldest deadline 2024-04-15T18:54:48.069165+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:33.970331+0000 front 2024-04-15T18:54:33.970228+0000 (oldest deadline 2024-04-15T18:54:59.770473+0000) 2024-04-15T18:55:06.805 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9[58910]: 2024-04-15T18:55:06.546+0000 7f79de1f2700 -1 osd.9 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:39.770957+0000 front 2024-04-15T18:54:39.770985+0000 (oldest deadline 2024-04-15T18:55:05.670828+0000) 2024-04-15T18:55:07.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:06.811+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:07.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:06.811+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:07.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:06.811+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:07.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:06.811+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:07.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:06 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:06.811+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:07.787 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:07.787 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:07.787 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:07.788 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:07.788 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:07.788 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:07.500+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:08.056 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:07.794+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:08.056 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:07.794+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:08.056 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:07.794+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:08.056 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:07.794+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:08.056 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:07 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:07.794+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:08.399 INFO:journalctl@ceph.osd.9.smithi159.stdout:Apr 15 18:55:08 smithi159 bash[187490]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-9 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.496+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.497+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.497+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.497+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.497+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:08.760 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:08.497+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:09.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:08 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:08.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:09.708 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:09.459+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:09.715 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.9.service' 2024-04-15T18:55:09.754 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:55:09.754 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-15T18:55:09.754 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-15T18:55:09.754 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.10 2024-04-15T18:55:10.041 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:09 smithi159 systemd[1]: Stopping Ceph osd.10 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:55:10.041 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.047+0000 7f53ff373700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:55:10.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:10.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:10.041 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:10.042 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:10.042 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:10.042 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:09 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:09.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:10.398 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.047+0000 7f53ff373700 -1 osd.10 931 *** Got signal Terminated *** 2024-04-15T18:55:10.399 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.047+0000 7f53ff373700 -1 osd.10 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:55:10.850 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:10.851 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:10.851 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:10.851 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:10.851 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:10.851 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:10.438+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:11.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:11.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:11.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:11.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:11.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:11.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:10 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:10.857+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:11.894 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:11.894 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:11.894 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:11.895 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:11.895 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:11.895 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:11.439+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:12.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:12.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:12.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:12.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:12.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:12.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:11 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:11.900+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:12.853 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:12.467+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:13.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:12 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:12.859+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:13.822 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:13.823 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:13.443+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.388029+0000 front 2024-04-15T18:54:48.388018+0000 (oldest deadline 2024-04-15T18:55:12.487906+0000) 2024-04-15T18:55:14.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:14.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:14.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:14.148 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:14.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:14.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:13 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:13.829+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:54:01.379610+0000 front 2024-04-15T18:54:01.379441+0000 (oldest deadline 2024-04-15T18:54:24.879484+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-04-15T18:54:07.680833+0000 front 2024-04-15T18:54:07.680869+0000 (oldest deadline 2024-04-15T18:54:31.780915+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:12.882237+0000 front 2024-04-15T18:54:12.882155+0000 (oldest deadline 2024-04-15T18:54:39.882712+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:23.383953+0000 front 2024-04-15T18:54:23.384064+0000 (oldest deadline 2024-04-15T18:54:49.283774+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:30.384952+0000 front 2024-04-15T18:54:30.384826+0000 (oldest deadline 2024-04-15T18:54:58.585229+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.086463+0000 front 2024-04-15T18:54:42.086422+0000 (oldest deadline 2024-04-15T18:55:06.686810+0000) 2024-04-15T18:55:14.807 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10[66086]: 2024-04-15T18:55:14.478+0000 7f53f795f700 -1 osd.10 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.388029+0000 front 2024-04-15T18:54:48.388018+0000 (oldest deadline 2024-04-15T18:55:12.487906+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:15.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:14 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:14.813+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.10.smithi159.stdout:Apr 15 18:55:15 smithi159 bash[187758]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-10 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:16.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:16.150 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:15 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:15.842+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:16.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:16 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:16.822+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:17.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.10.service' 2024-04-15T18:55:17.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:55:17.665 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-15T18:55:17.665 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-15T18:55:17.665 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 systemd[1]: Stopping Ceph osd.11 for 22c655ba-fb52-11ee-bc8f-c7b262605968... 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:17.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:17.900 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.853+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:18.399 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.957+0000 7f737604b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T18:55:18.399 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.957+0000 7f737604b700 -1 osd.11 931 *** Got signal Terminated *** 2024-04-15T18:55:18.399 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:17 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:17.957+0000 7f737604b700 -1 osd.11 931 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:19.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:18 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:18.862+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:20.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:19 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:19.864+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.159:6804 osd.8 since back 2024-04-15T18:54:54.294658+0000 front 2024-04-15T18:54:54.294696+0000 (oldest deadline 2024-04-15T18:55:19.594748+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:21.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:20 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:20.866+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.159:6804 osd.8 since back 2024-04-15T18:54:54.294658+0000 front 2024-04-15T18:54:54.294696+0000 (oldest deadline 2024-04-15T18:55:19.594748+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:22.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:21 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:21.824+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.159:6804 osd.8 since back 2024-04-15T18:54:54.294658+0000 front 2024-04-15T18:54:54.294696+0000 (oldest deadline 2024-04-15T18:55:19.594748+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-04-15T18:53:52.282876+0000 front 2024-04-15T18:53:52.282853+0000 (oldest deadline 2024-04-15T18:54:14.582811+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-04-15T18:53:59.884383+0000 front 2024-04-15T18:53:59.884249+0000 (oldest deadline 2024-04-15T18:54:26.284624+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-04-15T18:54:17.386977+0000 front 2024-04-15T18:54:17.387009+0000 (oldest deadline 2024-04-15T18:54:39.686940+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2024-04-15T18:54:24.888714+0000 front 2024-04-15T18:54:24.888738+0000 (oldest deadline 2024-04-15T18:54:49.489021+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-04-15T18:54:34.591079+0000 front 2024-04-15T18:54:34.591142+0000 (oldest deadline 2024-04-15T18:54:56.791308+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6820 osd.6 since back 2024-04-15T18:54:42.493733+0000 front 2024-04-15T18:54:42.493770+0000 (oldest deadline 2024-04-15T18:55:08.393537+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2024-04-15T18:54:48.394094+0000 front 2024-04-15T18:54:48.394069+0000 (oldest deadline 2024-04-15T18:55:14.294104+0000) 2024-04-15T18:55:23.149 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:22 smithi159 ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11[174551]: 2024-04-15T18:55:22.819+0000 7f737263f700 -1 osd.11 931 heartbeat_check: no reply from 172.21.15.159:6804 osd.8 since back 2024-04-15T18:54:54.294658+0000 front 2024-04-15T18:54:54.294696+0000 (oldest deadline 2024-04-15T18:55:19.594748+0000) 2024-04-15T18:55:23.899 INFO:journalctl@ceph.osd.11.smithi159.stdout:Apr 15 18:55:23 smithi159 bash[188035]: ceph-22c655ba-fb52-11ee-bc8f-c7b262605968-osd-11 2024-04-15T18:55:25.399 DEBUG:teuthology.orchestra.run.smithi159:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22c655ba-fb52-11ee-bc8f-c7b262605968@osd.11.service' 2024-04-15T18:55:25.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T18:55:25.438 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-15T18:55:25.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T18:55:25.679 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:55:26.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T18:55:27.174 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:55:28.291 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T18:55:28.520 INFO:teuthology.orchestra.run.smithi159.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T18:55:29.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:55:29.581 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:55:29.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T18:55:29.647 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-15T18:55:29.649 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi049/crash 2024-04-15T18:55:29.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T18:55:29.713 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T18:55:29.714 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2024-04-15T18:55:29.715 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi082/crash 2024-04-15T18:55:29.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T18:55:29.779 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T18:55:29.779 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-04-15T18:55:29.781 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi159/crash 2024-04-15T18:55:29.782 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T18:55:29.850 INFO:teuthology.orchestra.run.smithi159.stderr:tar: /var/lib/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T18:55:29.850 INFO:teuthology.orchestra.run.smithi159.stderr:tar: Error is not recoverable: exiting now 2024-04-15T18:55:29.852 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-15T18:55:29.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-15T18:55:29.996 INFO:teuthology.orchestra.run.smithi049.stdout:2024-04-15T18:20:00.000114+0000 mon.a (mon.0) 1564 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:55:29.997 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-15T18:55:29.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[SEC\]' /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-15T18:55:30.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]' /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-15T18:55:30.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[WRN\]' /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-15T18:55:30.195 INFO:teuthology.orchestra.run.smithi049.stdout:2024-04-15T18:20:00.000114+0000 mon.a (mon.0) 1564 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized 2024-04-15T18:55:30.195 INFO:tasks.cephadm:Compressing logs... 2024-04-15T18:55:30.196 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T18:55:30.237 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T18:55:30.240 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T18:55:30.266 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-15T18:55:30.273 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T18:55:30.274 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.y.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-15T18:55:30.280 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-15T18:55:30.281 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log 2024-04-15T18:55:30.281 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log 2024-04-15T18:55:30.282 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.a.log 2024-04-15T18:55:30.282 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T18:55:30.282 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-15T18:55:30.282 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T18:55:30.284 INFO:teuthology.orchestra.run.smithi159.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2024-04-15T18:55:30.284 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-15T18:55:30.288 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.6.log 2024-04-15T18:55:30.291 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.6.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.7.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.x.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.b.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-15T18:55:30.292 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-15T18:55:30.294 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.11.log 2024-04-15T18:55:30.294 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T18:55:30.296 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T18:55:30.296 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.10.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.8.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.c.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.9.log 2024-04-15T18:55:30.297 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T18:55:30.303 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.a.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.1.log: 87.7% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T18:55:30.339 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.9.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T18:55:30.393 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.4.log: /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.5.log: /var/log/ceph/cephadm.log: 93.2% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.x.log.gz 2024-04-15T18:55:30.414 INFO:teuthology.orchestra.run.smithi082.stderr: 82.4% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T18:55:30.442 INFO:teuthology.orchestra.run.smithi159.stderr: 91.3% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T18:55:30.467 INFO:teuthology.orchestra.run.smithi159.stderr: 82.4% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T18:55:30.510 INFO:teuthology.orchestra.run.smithi049.stderr: 94.6% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log.gz 2024-04-15T18:55:30.610 INFO:teuthology.orchestra.run.smithi082.stderr: 89.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T18:55:30.634 INFO:teuthology.orchestra.run.smithi082.stderr: 91.3% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T18:55:30.640 INFO:teuthology.orchestra.run.smithi159.stderr: 87.8% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T18:55:30.787 INFO:teuthology.orchestra.run.smithi049.stderr: 91.4% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T18:55:30.910 INFO:teuthology.orchestra.run.smithi082.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T18:55:30.914 INFO:teuthology.orchestra.run.smithi082.stderr: 93.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log.gz 2024-04-15T18:55:31.029 INFO:teuthology.orchestra.run.smithi159.stderr: 89.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T18:55:31.046 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log.gz 2024-04-15T18:55:31.050 INFO:teuthology.orchestra.run.smithi049.stderr: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T18:55:31.080 INFO:teuthology.orchestra.run.smithi049.stderr: 94.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log.gz 2024-04-15T18:55:31.760 INFO:teuthology.orchestra.run.smithi159.stderr: 93.8% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.6.log.gz 2024-04-15T18:55:31.760 INFO:teuthology.orchestra.run.smithi159.stderr: 94.5% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/objectstore_tool.7.log.gz 2024-04-15T18:55:31.798 INFO:teuthology.orchestra.run.smithi049.stderr: 82.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T18:55:32.002 INFO:teuthology.orchestra.run.smithi049.stderr: 89.8% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T18:55:32.451 INFO:teuthology.orchestra.run.smithi082.stderr: 87.8% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T18:55:32.756 INFO:teuthology.orchestra.run.smithi049.stderr: 91.4% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mgr.y.log.gz 2024-04-15T18:55:33.050 INFO:teuthology.orchestra.run.smithi082.stderr: 91.1% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.b.log.gz 2024-04-15T18:55:33.552 INFO:teuthology.orchestra.run.smithi159.stderr: 91.3% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.c.log.gz 2024-04-15T18:55:40.067 INFO:teuthology.orchestra.run.smithi049.stderr: 90.0% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-mon.a.log.gz 2024-04-15T18:57:28.392 INFO:teuthology.orchestra.run.smithi159.stderr: 94.1% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.8.log.gz 2024-04-15T18:58:59.469 INFO:teuthology.orchestra.run.smithi082.stderr: 92.4% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.6.log.gz 2024-04-15T18:59:07.145 INFO:teuthology.orchestra.run.smithi159.stderr: 93.7% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.11.log.gz 2024-04-15T18:59:22.469 INFO:teuthology.orchestra.run.smithi049.stderr: 92.7% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-15T18:59:30.699 INFO:teuthology.orchestra.run.smithi082.stderr: 92.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-15T18:59:34.956 INFO:teuthology.orchestra.run.smithi049.stderr: 92.2% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-15T18:59:47.847 INFO:teuthology.orchestra.run.smithi159.stderr: 92.3% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.9.log.gz 2024-04-15T19:00:13.117 INFO:teuthology.orchestra.run.smithi159.stderr: 91.6% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.10.log.gz 2024-04-15T19:00:13.119 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-15T19:00:13.119 INFO:teuthology.orchestra.run.smithi159.stderr:real 4m42.858s 2024-04-15T19:00:13.119 INFO:teuthology.orchestra.run.smithi159.stderr:user 6m15.516s 2024-04-15T19:00:13.119 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m15.575s 2024-04-15T19:00:37.522 INFO:teuthology.orchestra.run.smithi082.stderr: 91.9% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.7.log.gz 2024-04-15T19:00:58.966 INFO:teuthology.orchestra.run.smithi049.stderr: 91.5% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-15T19:01:54.784 INFO:teuthology.orchestra.run.smithi082.stderr: 91.5% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-15T19:01:54.786 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-04-15T19:01:54.786 INFO:teuthology.orchestra.run.smithi082.stderr:real 6m24.523s 2024-04-15T19:01:54.786 INFO:teuthology.orchestra.run.smithi082.stderr:user 8m3.587s 2024-04-15T19:01:54.786 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m19.992s 2024-04-15T19:02:40.617 INFO:teuthology.orchestra.run.smithi049.stderr: 91.5% -- replaced with /var/log/ceph/22c655ba-fb52-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-15T19:02:40.619 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-15T19:02:40.619 INFO:teuthology.orchestra.run.smithi049.stderr:real 7m10.362s 2024-04-15T19:02:40.620 INFO:teuthology.orchestra.run.smithi049.stderr:user 8m56.464s 2024-04-15T19:02:40.620 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m19.823s 2024-04-15T19:02:40.620 INFO:tasks.cephadm:Archiving logs... 2024-04-15T19:02:40.620 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi049/log 2024-04-15T19:02:40.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T19:04:17.126 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi082/log 2024-04-15T19:04:17.127 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T19:05:49.562 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi159/log 2024-04-15T19:05:49.563 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T19:06:56.500 INFO:tasks.cephadm:Removing cluster... 2024-04-15T19:06:56.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force 2024-04-15T19:06:56.816 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T19:06:57.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force 2024-04-15T19:06:58.107 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T19:06:58.942 DEBUG:teuthology.orchestra.run.smithi159:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22c655ba-fb52-11ee-bc8f-c7b262605968 --force 2024-04-15T19:06:59.438 INFO:teuthology.orchestra.run.smithi159.stdout:Deleting cluster with fsid: 22c655ba-fb52-11ee-bc8f-c7b262605968 2024-04-15T19:07:00.130 INFO:tasks.cephadm:Removing cephadm ... 2024-04-15T19:07:00.130 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T19:07:00.166 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T19:07:00.261 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T19:07:00.280 INFO:tasks.cephadm:Teardown complete 2024-04-15T19:07:00.281 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T19:07:00.290 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T19:07:00.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T19:07:00.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T19:07:00.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T19:07:00.459 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T19:07:00.467 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T19:07:00.468 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T19:07:00.470 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T19:07:00.473 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T19:07:00.488 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2024-04-15T19:07:00.490 INFO:teuthology.orchestra.run.smithi159.stderr:bash: ntpq: command not found 2024-04-15T19:07:00.491 INFO:teuthology.orchestra.run.smithi082.stderr:bash: ntpq: command not found 2024-04-15T19:07:00.509 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T19:07:00.509 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-04-15T19:07:00.510 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv01.front.sepia.ceph.com 3 7 377 47 -147us[ -150us] +/- 65ms 2024-04-15T19:07:00.510 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 112 -73us[ -80us] +/- 67ms 2024-04-15T19:07:00.510 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +277us[ +275us] +/- 74ms 2024-04-15T19:07:00.510 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T19:07:00.521 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T19:07:00.522 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-04-15T19:07:00.522 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 -125us[ -105us] +/- 65ms 2024-04-15T19:07:00.522 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 -87us[ -67us] +/- 68ms 2024-04-15T19:07:00.522 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +298us[ +318us] +/- 74ms 2024-04-15T19:07:00.522 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv01.front.sepia.ceph.com 3 8 377 115 -76us[ -53us] +/- 64ms 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -96us[ -96us] +/- 68ms 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +300us[ +300us] +/- 74ms 2024-04-15T19:07:00.523 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T19:07:00.524 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T19:07:00.533 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T19:07:00.533 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T19:07:00.542 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T19:07:00.669 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T19:07:00.773 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T19:07:00.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T19:07:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:01.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T19:07:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:01.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-15T19:07:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:01.068 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T19:07:01.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T19:07:01.087 INFO:teuthology.task.internal:Duration was 7477.861578 seconds 2024-04-15T19:07:01.088 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T19:07:01.097 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T19:07:01.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T19:07:01.100 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T19:07:01.102 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T19:07:01.183 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T19:07:01.191 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T19:07:01.216 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T19:07:01.516 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T19:07:01.516 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-04-15T19:07:01.517 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T19:07:01.548 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-04-15T19:07:01.548 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T19:07:01.593 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-04-15T19:07:01.593 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T19:07:01.653 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T19:07:01.654 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T19:07:01.663 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T19:07:01.667 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T19:07:01.811 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T19:07:01.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T19:07:01.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T19:07:01.832 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T19:07:02.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T19:07:02.399 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T19:07:02.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T19:07:02.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T19:07:02.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T19:07:02.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T19:07:02.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.581 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.584 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.619 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-04-15T19:07:02.619 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-04-15T19:07:02.625 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-04-15T19:07:02.651 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:02.668 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:02.689 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T19:07:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T19:07:02.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T19:07:02.719 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T19:07:02.720 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi049 2024-04-15T19:07:02.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T19:07:02.779 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi082 2024-04-15T19:07:02.780 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T19:07:02.826 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020/remote/smithi159 2024-04-15T19:07:02.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T19:07:02.881 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T19:07:02.882 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T19:07:02.884 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T19:07:02.887 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T19:07:02.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T19:07:02.937 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T19:07:02.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T19:07:02.968 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T19:07:02.968 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T19:07:02.970 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T19:07:02.972 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T19:07:02.985 INFO:teuthology.orchestra.run.smithi049.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 19:07 /home/ubuntu/cephtest 2024-04-15T19:07:02.987 INFO:teuthology.orchestra.run.smithi082.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 19:07 /home/ubuntu/cephtest 2024-04-15T19:07:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 19:07 /home/ubuntu/cephtest 2024-04-15T19:07:02.991 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T19:07:03.001 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T19:07:03.023 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T19:07:03.049 DEBUG:teuthology.nuke:shortname: smithi049 2024-04-15T19:07:03.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T19:07:03.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.009319', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpbeu/K9QPf6yYesDG5OyusKkoD0uIcLnn0mMGgpxjgXTzDbqOXkvYZ2OK30e2wC7nGabw2Z0inR0PMJyakmpY='} 2024-04-15T19:07:03.076 DEBUG:teuthology.nuke:shortname: smithi082 2024-04-15T19:07:03.076 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T19:07:03.097 DEBUG:teuthology.nuke:shortname: smithi159 2024-04-15T19:07:03.097 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T19:07:03.100 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.008178', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3V9nOZW6PFZFyIWiZcZE5DSGAtsz/oDq1c7zTcPDTyy2Ph2Iw6Sh1bZRoKE5isR3e7bmPV+yg2fGDlQnhcZLY='} 2024-04-15T19:07:03.118 INFO:teuthology.orchestra.console.smithi049:Power off 2024-04-15T19:07:03.119 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T19:07:03.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-14_20:00:15-rados-main-distro-default-smithi/7656020', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 16:51:17.006834', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3b44o1gqnrtqTFmqWhWiYplcsw6X/ndzmYAtJE/3ND1uuSSL8NDdR+sP7++axrtcqJj5ohdmpT3KNqD29+sYw='} 2024-04-15T19:07:03.144 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2024-04-15T19:07:03.144 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:03.166 INFO:teuthology.orchestra.console.smithi082:Power off 2024-04-15T19:07:03.166 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T19:07:03.175 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-15T19:07:03.188 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-04-15T19:07:03.188 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:03.202 INFO:teuthology.orchestra.console.smithi159:Power off 2024-04-15T19:07:03.202 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T19:07:03.211 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-04-15T19:07:03.225 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2024-04-15T19:07:03.226 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:03.247 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-04-15T19:07:07.177 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:07.212 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:07.248 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:07.288 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-15T19:07:07.322 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-04-15T19:07:07.357 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-04-15T19:07:11.293 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:11.323 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:11.358 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:11.404 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-04-15T19:07:11.432 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-04-15T19:07:11.467 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-04-15T19:07:11.505 INFO:teuthology.orchestra.console.smithi049:Power off completed 2024-04-15T19:07:15.436 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:15.468 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T19:07:15.548 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-04-15T19:07:15.577 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2024-04-15T19:07:15.650 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-04-15T19:07:15.678 INFO:teuthology.orchestra.console.smithi159:Power off completed 2024-04-15T19:07:15.783 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} duration: 7477.861577749252 failure_reason: '"2024-04-15T18:20:00.000114+0000 mon.a (mon.0) 1564 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 3001/39009 objects degraded (7.693%), 4 pgs degraded, 4 pgs undersized" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-04-15T19:07:15.784 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T19:07:15.867 INFO:teuthology.run:FAIL